Aging affects both cognitive performance and the sleep-wake rhythm. The recent surge of studies that support a role of sleep for cognitive performance in healthy young adults suggests that disturbed sleep-wake rhythms may contribute to ‘age-related’ cognitive decline. This relationship has however not previously been extensively investigated. The present correlational study integrated a battery of standardized cognitive tests to investigate the association of mental speed, memory, and executive function with actigraphically recorded sleep-wake rhythms in 144 home-dwelling elderly participants aged 69.5 ± 8.5 (mean ± SD). Multiple regression analyses showed that the partial correlations of the fragmentation of the sleep-wake rhythm with each of the three cognitive domains (r = −0.16, −0.19, and −0.16 respectively) were significant. These associations were independent from main effects of age, implying that a unique relationship between the rest-activity rhythm and cognitive performance is present in elderly people.