Evidence is emerging that physical activity (PA) may improve overall survival after breast cancer diagnosis. However, the effect of PA on breast cancer recurrence and on cause-specific mortality is less investigated. We assessed the association of pre-diagnosis PA with recurrence, overall and cause-specific survival in a prospective cohort study in Germany including 3,393 non-metastatic breast cancer patients aged 50–74 years. Cox proportional hazards models were calculated adjusted for relevant prognostic factors. During a median follow-up of 5.6 years, 367 patients deceased. Overall mortality was significantly inversely associated with pre-diagnosis recreational PA. However, this effect was mainly attributed to deaths due to causes other than breast cancer. Multiple fractional polynomial analyses yielded a nonlinear association with markedly increased non-breast cancer mortality for women who did not engage in any sports or cycling in the years before the breast cancer diagnosis with a hazard ratio (HR, none vs. any) of 1.71, 95% confidence interval (1.16, 2.52). There were no further risk reductions with increasing activity levels. The association with breast cancer-specific mortality showed a similar dose-response but was far less pronounced with HR (none vs. any) = 1.22 (0.91, 1.64). In contrast, regarding cancer recurrence the dose-response was linear. However, this association was restricted to estrogen/progesterone receptor-negative (ER−/PR−) cases (pinteraction = 0.033) with HR (highest vs. no recreational PA) = 0.53 (0.24, 1.16), ptrend = 0.0045. Thus, breast cancer patients with a physically inactive lifestyle pre-diagnosis may decease prematurely irrespective of their cancer prognosis. Higher levels of exercise may reduce the risk of recurrence of ER−/PR− breast tumors.