Despite abundant data on the early evolution of the Central Alps, the latest stage exhumation history, potentially related to relief formation, is still poorly constrained. We aim for a better understanding of the relation between glaciation, erosion and sediment deposition. Addressing both topics, we analysed late Pliocene to recent deposits from the Upper Rhine Graben and two modern river sands by apatite fission-track and (U-Th-Sm)/He thermochronology. From the observed age patterns we extracted the sediment provenance and paleo-erosion history of the Alpine-derived detritus. Due to their pollen and fossil record, the Rhine Graben deposits also provide information on climatic evolution, so that the erosion history can be related to glacial evolution during the Plio-Pleistocene. Our data show that Rhine Graben deposits were derived from Variscan basement, Hegau volcanics, Swiss Molasse Basin, and the Central Alps. The relations between glaciation, Alpine erosion, and thermochronological age signals in sedimentary rocks are more complex than assumed. The first Alpine glaciation during the early Pleistocene did not disturb the long-term exhumational equilibrium of the Alps. Recent findings indicate that main Alpine glaciation occurred at ca. 1 Ma. If true, then main Alpine glaciation was coeval with an apparent decrease of hinterland erosion rates, contrary to the expected trend. We suggest that glaciers effectively sealed the landscape, thus reducing the surface exposed to erosion and shifting the area of main erosion north toward the Molasse basin, causing sediment recycling. At around 0.4 Ma, erosion rates increased again, which seems to be a delayed response to main glaciation. The present-day erosion regime seems to be dominated by mass-wasting processes. Generally, glacial erosion rates did not exceed the pre-glacial long-term erosion rates of the Central Alps.