Previous studies have suggested that obstructive sleep apnea (OSA) may be a risk factor for stroke. In this study, we assessed that OSA is an independent risk factor of silent cerebral infarction (SCI) in the general population, and in a non-obese population. This study recruited a total of 746 participants (252 men and 494 women) aged 50–79 years as part of the Korean Genome and Epidemiology Study (KoGES); they underwent polysomnography, brain magnetic resonance imaging and health screening examinations. SCI was assessed by subtypes and brain regions, and lacunar infarction represented lesions <15 mm in size in the penetrating arteries. Moderate–severe OSA was determined by apnea–hypopnea index ≥15. The results indicated that 12.06% had moderate–severe OSA, 7.64% of participants had SCI and 4.96% had lacunar infarction. Moderate–severe OSA was associated positively with SCI [odds ratio (OR): 2.44, 95% confidence interval (CI): 1.03–5.80] and lacunar infarction (OR: 3.48, 95% CI: 1.31–9.23) in the age ≥65-year group compared with those with non-OSA. Additionally, in the basal ganglia, OSA was associated with an increase in the odds for SCI and lacunar infarction in all age groups, and especially in the ≥65-year age group. In the non-obese participants, OSA was also associated positively with SCI in the ≥65-year age group, lacunar infarction in all age groups, and especially in the ≥65-year age group. There was also a positive association with the basal ganglia. Moderate–severe OSA was associated positively with SCI and lacunar infarction in elderly participants. Treatment of OSA may reduce new first-time cerebrovascular events and recurrences.