Women from high-risk families consider preventive measures for breast cancer including screening. Guidelines on screening differ considerably regarding starting age. We investigated whether age at diagnosis in affected relatives is predictive for age at diagnosis. We analyzed the age of breast cancer detection of 1,304 first- and second-degree relatives of 314 BRCA1, 164 BRCA2 and 244 high-risk participants of the Dutch MRI-SCreening study. The within- and between-family variance in the relative's age at diagnosis was analyzed with a random effect linear regression model. We compared the starting age of screening based on risk-group (25 years for BRCA1, 30 years for BRCA2 and 35 years for familial risk), on family history, and on the model, which combines both. The findings were validated in 63 families from the UK-MARIBS study. Mean age at diagnosis in the relatives varied between families; 95% range of mean family ages was 35–55 in BRCA1-, 41–57 in BRCA2- and 44–60 in high-risk families. In all, 14% of the variance in age at diagnosis, in BRCA1 even 23%, was explained by family history, 7% by risk group. Determining start of screening based on the model and on risk-group gave similar results in terms of cancers missed and years of screening. The approach based on familial history only, missed more cancers and required more screening years in both the Dutch and the United Kingdom data sets. Age at breast cancer diagnosis is partly dependent on family history which may assist planning starting age for preventive measures.