Epidemiological observations support a positive relationship between cardiovascular diseases (CVD) and osteoporosis, where cholesterol has been indicated to be a possible link. Only a few studies have investigated the relation between lipids and BMD, but the association remains unclear. We studied the relationship between serum lipids and BMD of the calcaneus. A cross-sectional population-based study was performed, based on data from the Longitudinal Aging Study Amsterdam, including 620 men and 635 women, 65–88 yr of age. BMD was measured by quantitative ultrasound (QUS), velocity of sound (VOS; m/s), and broadband ultrasound attenuation (BUA; dB/MHz). Models were adjusted for age, body mass index, physical activity, smoking, alcohol, diabetes mellitus, hypertension, testosterone, and 25-hydroxyvitamin D. No association was found between total cholesterol (TC) and QUS. Men and women in the highest quartile of high-density lipoprotein cholesterol (HDL-c) had a significantly lower QUS (men—VOS: β = −20.8, p = 0.00; BUA: β = −5.2, p = 0.02; women—VOS: β = −18.6, p = 0.00) compared with men and women in the lowest quartile. An even stronger positive association was seen between TC/HDL-c ratio and QUS (men—VOS: β = 21.8, p = 0.00; BUA: β = 5.5, p = 0.01; women—VOS: β = 19.2, p = 0.00; BUA: β = 3.6, p = 0.05). Our analysis shows that the lipid profile that is favorable in the prevention of CVD (i.e., high levels of HDL-c and low TC/HDL-c ratio) is unfavorable for QUS. These results indicate that HDL-c levels do not explain the association between osteoporosis and CVD.