Previous studies have reported an association between exercise during youth and increased areal bone mineral density at old age. The primary aim of this study was to investigate if exercise during growth was independently associated with greater cortical bone size and whole bone strength in weight-bearing bone in old men. The tibia and radius were measured using both peripheral quantitative computed tomography (pQCT) (XCT-2000; Stratec) at the diaphysis and high-resolution pQCT (HR-pQCT) (XtremeCT; Scanco) at the metaphysis to obtain cortical bone geometry and finite element–derived bone strength in distal tibia and radius, in 597 men, 79.9 ± 3.4 (mean ± SD) years old. A self-administered questionnaire was used to collect information about previous and current physical activity. In order to determine whether level of exercise during growth and young adulthood or level of current physical activity were independently associated with bone parameters in both tibia and radius, analysis of covariance (ANCOVA) analyses were used. Adjusting for covariates and current physical activity, we found that men in the group with the highest level of exercise early in life (regular exercise at a competitive level) had higher tibial cortical cross-sectional area (CSA; 6.3%, p < 0.001) and periosteal circumference (PC; 1.6%, p = 0.011) at the diaphysis, and higher estimated bone strength (failure load: 7.5%, p < 0.001; and stiffness: 7.8%, p < 0.001) at the metaphysis than men in the subgroup with the lowest level of exercise during growth and young adulthood. Subjects in the group with the highest level of current physical activity had smaller tibial endosteal circumference (EC; 3.6%, p = 0.012) at the diaphysis than subjects with a lower current physical activity, when adjusting for covariates and level of exercise during growth and young adulthood. These findings indicate that exercise during growth can increase the cortical bone size via periosteal expansion, whereas exercise at old age may decrease endosteal bone loss in weight-bearing bone in old men. © 2014 American Society for Bone and Mineral Research.