The variability in solar irradiance, the main external energy source of the Earth's system, must be critically studied in order to place the effects of human-driven climate change into perspective and allow plausible predictions of the evolution of climate. Accurate measurements of total solar irradiance (TSI) variability by instruments onboard space platforms during the last three solar cycles indicate changes of approximately 0.1% over the sunspot cycle. Physics-based models also suggest variations of the same magnitude on centennial to millennia time-scales. Additionally, long-term changes in Earth's orbit modulate the solar irradiance reaching the top of the atmosphere. Variations of orbital inclination in relation to the Sun's equator could potentially impact incoming solar irradiance as a result of the anisotropy of the distribution of active regions. Due to a lack of quantitative estimates, this effect has never been assessed. Here, we show that although observers with different orbital inclinations experience various levels of irradiance, modulations in TSI are not sufficient to drive observed 100 kyr climate variations. Based on our model we find that, due to orbital inclination alone, the maximum change in the average TSI over timescales of kyrs is ∼0.003 Wm−2, much smaller than the ∼1.5 Wm−2 annually integrated change related to orbital eccentricity variations, or the 1–8 Wm−2variability due to solar magnetic activity. Here, we stress that out-of-ecliptic measurements are needed in order to constrain models for the long-term evolution of TSI and its impact on climate.