Normal human aging is accompanied by progressive brain tissue loss and cognitive decline; however, several factors are thought to influence brain aging. We applied tensor-based morphometry to high-resolution brain MRI scans to determine whether educational level or physical activity was associated with brain tissue volumes in the elderly, particularly in regions susceptible to age-related atrophy. We mapped the 3D profile of brain volume differences in 226 healthy elderly subjects (130F/96M; 77.9 ± 3.6 SD years) from the Cardiovascular Health Study-Cognition Study. Statistical maps revealed the 3D profile of brain regions whose volumes were associated with educational level and physical activity (based on leisure-time energy expenditure). After controlling for age, sex, and physical activity, higher educational levels were associated with ∼2–3% greater tissue volumes, on average, in the temporal lobe gray matter. After controlling for age, sex, and education, greater physical activity was associated with ∼2–2.5% greater average tissue volumes in the white matter of the corona radiata extending into the parietal-occipital junction. Body mass index (BMI) was highly correlated with both education and physical activity, so we examined BMI as a contributing factor by including physical activity, education, and BMI in the same model; only BMI effects remained significant. This is one of the largest MRI studies of factors influencing structural brain aging, and BMI may be a key factor explaining the observed relationship between education, physical activity, and brain structure. Independent contributions to brain structure could not be teased apart as all these factors were highly correlated with one another. Hum Brain Mapp, 2010. © 2010 Wiley-Liss, Inc.