Recent cohort studies suggest that increased breast cancer risks were associated with longer smoking duration, higher pack-years and a dose-response relationship with increasing pack-years of smoking between menarche and first full-term pregnancy (FFTP). Studies with comprehensive quantitative life-time measures of passive smoking suggest an association between passive smoking dose and breast cancer risk. We conducted a study within the European Prospective Investigation into Cancer and Nutrition to examine the association between passive and active smoking and risk of invasive breast cancer and possible effect modification by known breast cancer risk factors. Among the 322,988 women eligible for the study, 9,822 developed breast cancer (183,608 women with passive smoking information including 6,264 cases). When compared to women who never smoked and were not being exposed to passive smoking at home or work at the time of study registration, current, former and currently exposed passive smokers were at increased risk of breast cancer (hazard ratios (HR) [95% confidence interval (CI)] 1.16 [1.05–1.28], 1.14 [1.04–1.25] and 1.10 [1.01–1.20], respectively). Analyses exploring associations in different periods of life showed the most important increase in risk with pack-years from menarche to FFTP (1.73 [1.29–2.32] for every increase of 20 pack-years) while pack-years smoked after menopause were associated with a significant decrease in breast cancer risk (HR = 0.53, 95% CI: 0.34–0.82 for every increase of 20 pack-years). Our results provide an important replication, in the largest cohort to date, that smoking (passively or actively) increases breast cancer risk and that smoking between menarche and FFTP is particularly deleterious.