Low bone mineral density (BMD) increases fracture risk; how changes in BMD influence fracture risk in older men is uncertain. BMD was assessed at two to three time points over 4.6 years using dual-energy X-ray absorptiometry (DXA) for 4470 men aged ≥65 years in the Osteoporotic Fractures in Men (MrOS) Study. Change in femoral neck BMD was estimated using mixed effects linear regression models. BMD change was categorized as “accelerated” (≤−0.034 g/cm2), “expected” (between 0 and −0.034 g/cm2), or “maintained” (≥0 g/cm2). Fractures were adjudicated by central medical record review. Multivariate proportional hazards models estimated the risk of hip, nonspine/nonhip, and nonspine fracture over 4.5 years after the final BMD measure, during which time 371 (8.3%) men experienced at least one nonspine fracture, including 78 (1.7%) hip fractures. Men with accelerated femoral neck BMD loss had an increased risk of nonspine (hazard ratio [HR] = 2.0; 95% confidence interval [CI] 1.4–2.8); nonspine/nonhip (HR = 1.6; 95% CI 1.1–2.3); and hip fracture (HR = 6.3; 95% CI 2.7–14.8) compared with men who maintained BMD over time. No difference in risk was seen for men with expected loss. Adjustment for the initial BMD measure did not alter the results. Adjustment for the final BMD measure attenuated the change in BMD-nonspine fracture and the change in BMD-nonspine/nonhip relationships such that they were no longer significant, whereas the change in the BMD-hip fracture relationship was attenuated (HR = 2.6; 95% CI 1.1–6.4). Total hip BMD change produced similar results. Accelerated decrease in BMD is a strong, independent risk factor for hip and other nonspine fractures in men. © 2012 American Society for Bone and Mineral Research.