This paper develops a nonparametric controller with an internal model control (IMC) structure for the longitudinal speed tracking control of autonomous land vehicles by designing a proportional and internal model control (IMC) cascade (P-IMC) controller. An IMC architecture is employed in the inner control loop by establishing a nonparametric longitudinal dynamical model, whereas a P controller is designed for the outer control loop. An approach for estimating the terrain effects and compensating for the model errors is also introduced. The differences from other nonparametric controllers are discussed, and the stability of the P-IMC controller is analyzed and validated experimentally. The P-IMC controller is compared with the SpAM+PI to illustrate its advantages. The experimental results of autonomous all-terrain driving show the effectiveness of the P-IMC controller.