Cerebral white matter lesions (WMLs) have been consistently related to cognitive dysfunction but the role of white matter (WM) damage in cognitive impairment is not fully determined. Diffusion tensor imaging is a promising tool to explain impaired cognition related to WMLs. We investigated the separate association of high-grade periventricular hyperintensities (PVHs) and deep white matter hyperintensities (DWMHs) with fractional anisotropy (FA) in middle-aged individuals. We also assessed the predictive value to cognition of FA within specific WM tracts associated with high-grade WMLs. One hundred participants from the Barcelona-AsIA Neuropsychology Study were divided into groups based on low- and high-grade WMLs. Voxel-by-voxel FA were compared between groups, with separate analyses for high-grade PVHs and DWMHs. The mean FA within areas showing differences between groups was extracted in each tract for linear regression analyses. Participants with high-grade PVHs and participants with high-grade DWMHs showed lower FA in different areas of specific tracts. Areas showing decreased FA in high-grade DWMHs predicted lower cognition, whereas areas with decreased FA in high-grade PVHs did not. The predictive value to cognition of specific WM tracts supports the involvement of cortico-subcortical circuits in cognitive deficits only in DWMHs.