We investigated one of the mechanisms of secular drift of Earth's obliquity, called “climate friction,” due to the feedback from the periodic variation of solar insolation and the glacial cycles. We estimated the change of the moment of inertia by utilizing Maxwell-type viscous Earth models and assumed linearity between insolation variation and ice sheet variation. We reproduced the feedback loop of insolation variation → ice sheet variation → change of the moment of inertia → change of the motion of the rotational axis of Earth, including the standard theoretical results of the quasi-periodic planetary perturbation. Our simulation resulted in a secular increasing drift of Earth's obliquity, which is induced by the time lag of ice sheet formation with respect to the solar insolation variation, and time lag of viscous deformation of the solid Earth with respect to the formation of ice sheets. We have estimated a secular increase of the obliquity 0.4°/10 m.y., which depends on the viscosity of the lower mantle and the degree of ice sheet development.