The risk of hip fracture rises rapidly with age, and is notably higher in women. After falls and prior fragility fractures, the main clinically recognized risk factor for hip fracture is reduced bone density. To better understand the extent to which femoral neck density and structure change with age in each sex, we carried out a longitudinal study in subjects not treated with agents known to influence bone mineral density (BMD), to investigate changes in regional cortical thickness, as well as cortical and trabecular BMD at the mid-femoral neck. Segmental quantitative computed tomography (QCT) analysis was used to assess bone measurements in two anatomic subregions, the superolateral (superior) and inferomedial (inferior). A total of 400 older individuals (100 men and 300 women, aged 66–90 years) who were participants in the Age Gene/Environment Susceptibility-Reykjavik Study (AGES-Reykjavik), were studied. Participants had two QCT scans of the hip over a median follow-up of 5.1 years (mean baseline age 74 years). Changes in bone values during follow-up were estimated from mixed effects regression models. At baseline women had lower bone values in the superior region than men. At follow-up all bone values were lower in women, except cortical volumetric bone mineral density (vBMD) inferiorly. The relative losses in all bone values estimated in the superior region were substantially (about threefold) and significantly greater compared to those estimated in the inferior region in both sexes. Women lost cortical thickness and cortical vBMD more rapidly than men in both regions; and this was only weakly reflected in total femoral neck dual-energy X-ray absorptiometry (DXA)-like results. The higher rate of bone loss in women at critical locations may contribute materially to the greater femoral neck fracture incidence among women than men. © 2013 American Society for Bone and Mineral Research.