Higher physical activity levels have been associated with a lower risk of developing various cancers and all-cancer mortality, but the impact of pre-diagnosis physical activity on cancer-specific death has not been fully characterized. In the prospective National Institutes of Health-AARP Diet and Health Study with 293,511 men and women, we studied prediagnosis moderate to vigorous intensity leisure time physical activity (MVPA) in the past 10 years and cancer-specific mortality. Over a median 12.1 years, we observed 15,001 cancer deaths. Using Cox proportional hazards regression, we estimated hazard ratios (HRs) and 95% confidence intervals (CIs) for MVPA with cancer mortality overall and by 20 specific cancer sites, adjusting for relevant risk factors. Compared to participants reporting never/rare MVPA, those reporting >7 hr/week MVPA had a lower risk of total cancer mortality (HR = 0.89, 95% CI 0.84–0.94; p-trend <0.001). When analyzed by cancer site-specific deaths, comparing those reporting >7 hr/week of MVPA to those reporting never/rare MVPA, we observed a lower risk of death from colon (HR = 0.70; 95% CI 0.57–0.85; p-trend <0.001), liver (0.71; 0.52–0.98; p-trend = 0.012) and lung cancer (0.84; 0.77–0.92; p-trend <0.001) and a significant p-trend for non-Hodgkins lymphoma (0.80; 0.62–1.04; p-trend = 0.017). An unexpected increased mortality p-trend with increasing MVPA was observed for death from kidney cancer (1.42; 0.98–2.03; p-trend = 0.016). Our findings suggest that higher prediagnosis leisure time physical activity is associated with lower risk of overall cancer mortality and mortality from multiple cancer sites. Future studies should confirm observed associations and further explore timing of physical activity and underlying biological mechanisms.