Cognitive abilities decline with age, but with considerable individual variation. The neurobiological correlate of this variation is not well described. Functional brain imaging studies have demonstrated reduced task-induced deactivation (TID) of the brain's default mode network (DMN) in a wide range of neurodegenerative diseases involving cognitive symptoms, in conditions with increased risk of Alzheimer's disease, and even in advanced but healthy aging. Here, we investigated brain activation and deactivation during a visual-motor task in 185 clinically healthy males from a Danish birth cohort, whose cognitive function was assessed in youth and midlife. Using each individual as his own control, we defined a group with a large degree of cognitive decline, and a control group. When correcting for effects of total cerebral blood flow and hemoglobin level, we found reduced TID in the posterior region of the DMN in the cognitive decline group compared to the control group. Furthermore, increased visual activation response was found in the cognitive decline group, indicating that the TID reduction was not exclusively due to overall impaired vascular reactivity. These results suggest a neurobiological basis for subclinical cognitive decline in late midlife, which includes TID alterations similar to the pattern seen in patients with AD and mild cognitive impairment. Hence, TID reduction might be suggested as an early marker for subtle cognitive decline in aging. Hum Brain Mapp 35:4488–4498, 2014. © 2014 Wiley Periodicals, Inc.