Increasing atmospheric CO2 concentration and related climate change have stimulated much interest in the potential of soils to sequester carbon. In ‘The Jena Experiment’, a managed grassland experiment on a former agricultural field, we investigated the link between plant diversity and soil carbon storage. The biodiversity gradient ranged from one to 60 species belonging to four functional groups. Stratified soil samples were taken to 30 cm depth from 86 plots in 2002, 2004 and 2006, and organic carbon contents were determined. Soil organic carbon stocks in 0–30 cm decreased from 7.3 kg C m−2 in 2002 to 6.9 kg C m−2 in 2004, but had recovered to 7.8 kg C m−2 by 2006. During the first 2 years, carbon storage was limited to the top 5 cm of soil while below 10 cm depth, carbon was lost probably as short-term effect of the land use change. After 4 years, carbon stocks significantly increased within the top 20 cm. More importantly, carbon storage significantly increased with sown species richness (log-transformed) in all depth segments and even carbon losses were significantly smaller with higher species richness. Although increasing species diversity increased root biomass production, statistical analyses revealed that species diversity per se was more important than biomass production for changes in soil carbon. Below 20 cm depth, the presence of one functional group, tall herbs, significantly reduced carbon losses in the beginning of the experiment. Our analysis indicates that plant species richness and certain plant functional traits accelerate the build-up of new carbon pools within 4 years. Additionally, higher plant diversity mitigated soil carbon losses in deeper horizons. This suggests that higher biodiversity might lead to higher soil carbon sequestration in the long-term and therefore the conservation of biodiversity might play a role in greenhouse gas mitigation.