In observational studies of men with prostate cancer, men on endocrine treatment (ET) have had an increased risk of ischemic heart disease (IHD) and stroke. However, prostate cancer per se may increase risk of IHD and stroke and men on ET may have been at increased risk already prior to initiation of ET. We assessed the incidence of IHD and stroke in men with prostate cancer before and during different endocrine treatments. The hazard ratio (HR) of IHD and stroke in 39,051 men with prostate cancer vs. a matched control population without prostate cancer was assessed by use of Cox proportion hazard models. An increased risk was found among 30,883 men with prostate cancer who did not receive ET, with a HR of 1.08 (95% CI 1.00–1.18) for IHD and 1.10 (95%CI 1.00–1.21) for stroke. In 8,168 men who initiated ET during the observation period, the risk of IHD was significantly higher (p = 0.014), during ET (HR 1.40, 95% CI 1.17–1.67) compared with before initiation of ET (HR of 0.98, 95% CI 0.72–1.33), whereas no such increase was found for stroke. Regardless of treatment, men with prostate cancer had a small increase in risk of IHD and stroke and initiation of ET was associated with a further increase in risk of IHD. Our data underline the importance of a proper indication for ET because many men with low-risk prostate cancer currently receive ET.