Objectives: Although age-related brain changes are becoming better understood, midlife patterns of change are still in need of characterization, and longitudinal studies are lacking. The aim of this study was to determine if baseline fractional anisotropy (FA), obtained from diffusion tensor imaging (DTI) predicts volume change over a 4-year interval. Experimental design: Forty-four cognitively healthy middle-age adults underwent baseline DTI and longitudinal T1-weighted magnetic resonance imaging. Tensor-based morphometry methods were used to evaluate volume change over time. FA values were extracted from regions of interest that included the cingulum, entorhinal white matter, and the genu and splenium of the corpus callosum. Baseline FA was used as a predictor variable, whereas gray and white matter atrophy rates as indexed by Tensor-based morphometry were the dependent variables. Principal observations: Over a 4-year period, participants showed significant contraction of white matter, especially in frontal, temporal, and cerebellar regions (P < 0.05, corrected for multiple comparisons). Baseline FA in entorhinal white matter, genu, and splenium was associated with longitudinal rates of atrophy in regions that included the superior longitudinal fasciculus, anterior corona radiata, temporal stem, and white matter of the inferior temporal gyrus (P < 0.001, uncorrected for multiple comparisons). Conclusions: Brain change with aging is characterized by extensive shrinkage of white matter. Baseline white matter microstructure as indexed by DTI was associated with some of the observed regional volume loss. The findings suggest that both white matter volume loss and microstructural alterations should be considered more prominently in models of aging and neurodegenerative diseases. Hum Brain Mapp 35:2044–2054, 2014. © 2013 Wiley Periodicals, Inc.