A few studies have indicated that self-reported physical activity is associated with the risk of fractures in middle-aged and elderly men. We investigated whether objectively measured physical fitness in young adulthood was associated with the risk of low-energy fractures later in life in men. Aerobic capacity and isometric muscle strength were measured in 435,445 Swedish men who were conscripted for military service from 1969 to 1978. Incident fractures were searched in national registers. During a median follow-up period of 35 years (range, 11–41 years), 8030 subjects sustained at least one fracture, increasing the risk of death 1.8 times (95% CI, 1.6–2.0) during follow up. When comparing men in the lowest and highest decile of physical fitness, the risk of a fracture was 1.8 times higher (95% CI, 1.6–2.1) and that of hip fracture was 2.7 times higher (95% CI, 1.6–4.7). The risk of fracture was also 1.4 to 1.5 times higher when comparing the extreme deciles of muscle strength (p < 0.001 for all). In a subcohort of 1009 twin pairs, up to 22% of the variation in physical fitness and 27% to 39% of the variation in muscle strength was attributable to environmental factors unique to one twin; eg, physical activity. In conclusion, low aerobic capacity and muscle strength in young adulthood are associated with an increased risk of low-energy fractures later in life. © 2013 American Society for Bone and Mineral Research.