The aim was to investigate the associations between physical activity (PA), cardiorespiratory fitness (CRF) and intima media thickness (IMT) or stiffness. This was a population-based cross-sectional study (n = 336) of Danish adolescents [mean age (standard deviation, SD): 15.6 (0.4) years]. PA intensity was assessed objectively (ActiGraph model GT3X) and CRF using a progressive maximal bicycle test. Carotid IMT and arterial stiffness were assessed using B-mode ultrasound. In a multivariate analysis (adjusted for pubertal development and smoking status), CRF was inversely associated with measures of carotid stiffness (standard beta: −0.20 to −0.15, P < 0.05) in boys, but not in girls. No associations were observed between any of PA and IMT. Boys in the least fit quartile had significantly stiffer carotid arteries compared to the most fit quartile (difference between lowest and highest quartile ranging between 0.4 and 0.5 SD, P < 0.05). This difference in arterial stiffness between low and high quartiles was similar for moderate-to-vigorous PA (MVPA). Further adjustment for sedentary time attenuated the difference observed between quartiles MVPA slightly. Adiposity did not attenuate these differences. Our observations suggest that increasing CRF or MVPA in the least active group of the population may be beneficial for vascular health.