We aimed to investigate if characteristics of contralateral breast cancer (CBC) are influenced by adjuvant radiotherapy for the first breast cancer. Using information from population-based registers and medical records, we analyzed two cohorts comprising all women with CBC diagnosed >3 months after their first cancer (809 patients in Stockholm 1976–2005 and 750 patients in South Sweden 1977–2005). We used Poisson regression to calculate risk of distant metastasis after CBC, comparing patients treated and not treated with radiotherapy for the first cancer. Logistic regression was used to estimate odds ratio (OR) of more aggressive tumor characteristics in the second cancer, compared to the first. For patients with CBC in Stockholm with <5 years between the cancers radiotherapy for the first cancer conferred a nearly doubled risk of distant metastasis [incidence rate ratio (IRR) = 1.91; 95% confidence interval (CI): 1.27–2.88], compared to those not treated with radiotherapy. This was replicated in the South Swedish cohort [IRR = 2.12 (95% CI: 1.40–3.23)]. In Stockholm, we found an increased odds that, following radiotherapy, a second cancer was of more advanced TNM-stage [OR 2.16 (95% CI 1.13–4.11)] and higher histological grade [OR = 2.00 (95% CI 1.08–3.72)] compared to the first, for patients with CBC with <5 years between the cancers. No effect on any of the investigated outcomes was seen for patients diagnosed with CBC >5 years from the first cancer. In conclusion, patients diagnosed with CBC within 5 years had worse prognosis and more aggressive tumor characteristics of the second cancer, if they had received radiotherapy for their first cancer, compared to no radiotherapy.