Previous studies suggest that male testosterone concentrations have declined over time. To explore this in a large US population, we examined testosterone and free testosterone concentrations in National Health and Nutrition Examination Surveys (NHANES) from 1988–1991 and 1999–2004. We also examined sex hormone-binding globulin (SHBG), estradiol, and androstanediol glucuronide (3α-diol-G) over the same period. Non-Hispanic white, non-Hispanic black, and Mexican-American men from 1988–1991 and 1999–2004 NHANES surveys who were ≥20 years old and had serum from morning blood draws were included in this analysis (1988–1991: N = 1,413; 1999–2004: N = 902). Testosterone, estradiol and SHBG were measured by competitive electrochemiluminescence immunoassays and 3α-diol-G was measured by enzyme immunoassay. Free testosterone was calculated using testosterone and SHBG values. Adjusted mean hormone concentrations were estimated using linear regression, accounting for NHANES sampling weights and design, age, race/ethnicity, body mass index, waist circumference, alcohol use and smoking. Differences in adjusted mean concentrations (Δ) and two-sided p-values were calculated; p < 0.05 was statistically significant. Overall, 3α-diol-G and estradiol declined between 1988–1991 and 1999–2004, but there was little change in testosterone, free testosterone, or SHBG (Δ: 3α-diol-G = −1.83 ng/mL, p < 0.01; estradiol = −6.07 pg/mL, p < 0.01; testosterone = −0.03 ng/mL, p = 0.75; free testosterone = −0.001 ng/mL, p = 0.67; SHBG = −1.17 nmol/L, p = 0.19). Stratification by age and race revealed that SHBG and 3α-diol-G declined among whites 20–44 years old (Δ: SHBG = −5.14 nmol/L, p < 0.01; 3α-diol-G = −2.89 ng/mL, p < 0.01) and free testosterone increased among blacks 20–44 years old (Δ: 0.014 ng/mL, p = 0.03). Estradiol declined among all ages of whites and Mexican-Americans. In conclusion, there was no evidence for testosterone decline between 1988–1991 and 1999–2004 in the US general population. Subgroup analyses suggest that SHBG and 3α-diol-G declined in young white men, estradiol declined in white and Mexican-American men, and free testosterone increased in young black men. These changes may be related to the increasing prevalence of reproductive disorders in young men.