Genes involved in sex hormone pathways are candidates for influencing bone strength. Polymorphisms in these genes were tested for association with heel quantitative ultrasound (QUS) parameters in middle-aged and elderly European men. Men 40–79 yr of age were recruited from population registers in eight European centers for the European Male Aging Study (EMAS). Polymorphisms were genotyped in AR, ESR1, ESR2, CYP19A1, CYP17A1, SHBG, SRD5A2, LHB, and LHCGR. QUS parameters broadband ultrasound attenuation (BUA) and speed of sound (SOS) were measured in the heel and used to derive BMD. The relationships between QUS parameters and polymorphisms were assessed using linear regression adjusting for age and center. A total of 2693 men, with a mean age of 60.1 ± 11.1 (SD) yr were included in the analysis. Their mean BUA was 80.0 ± 18.9 dB/Mhz, SOS was 1550.2 ± 34.1 m/s, and BMD was 0.542 ± 0.141 g/cm2. Significant associations were observed between multiple SNPs in a linkage disequilibrium (LD) block within CYP19A1, peaking at the TCT indel with the deletion allele associating with reduced ultrasound BMD in heterozygotes (β =−0.016, p = −0.005) and homozygotes (β = −0.029, p = 0.001). The results for BUA and SOS were similar. Significant associations with QUS parameters were also observed for the CAG repeat in AR and SNPs in CYP17A1, LHCGR, and ESR1. Our data confirm evidence of association between bone QUS parameters and polymorphisms in CYP19A1, as well as modest associations with polymorphisms in CYP17A1, ESR1, LHCGR, and AR in a population sample of European men; this supports a role for genetically determined sex hormone actions in influencing male bone health.