Very preterm birth (VPT; <32 weeks of gestation) has been associated with impairments in memory abilities and functional neuroanatomical brain alterations in medial temporal and fronto-parietal areas. Here we investigated the relationship between structural connectivity in memory-related tracts and various aspects of memory in VPT adults (mean age 19) who sustained differing degrees of perinatal brain injury (PBI), as assessed by neonatal cerebral ultrasound. We showed that the neurodevelopmental consequences of VPT birth persist into young adulthood and are associated with neonatal cranial ultrasound classification. At a cognitive level, VPT young adults showed impairments specific to effective organization of verbal information and visuospatial memory, whereas at an anatomical level they displayed reduced volume of memory-related tracts, the cingulum and the fornix, with greater alterations in those individuals who experienced high-grade PBI. When investigating the association between these tracts and memory scores, perseveration errors were associated with the volume of the fornix and dorsal cingulum (connecting medial frontal and parietal lobes). Visuospatial memory scores were associated with the volume of the ventral cingulum (connecting medial parietal and temporal lobes). These results suggest that structural connectivity alterations could underlie memory difficulties in preterm born individuals.