The large number of abandoned quarries in many countries presents challenges for restoration of these extremely degraded habitats. To understand soil and plant development in these extreme habitats at a most critical stage of restoration, we evaluated the edaphic conditions and natural vegetation of three large quarries in southern China 3, 5, and 7 years following abandonment. Although soil fertility (organic matter, and N, P, K concentrations) did not differ significantly over a few years, it was much higher than would be expected from newly weathered soil and was comparable to that of the adjacent garden soil on level ground with no slope. This suggests that soil formation on the steep slopes of quarry cliffs is a secondary migration process rather than a primary weathering process. Vegetation cover increased from 10.6 to 18.6 and 23.4%, and species abundance increased from 8 to 11 and 12 species, and from 3 to 6 and 7 families. Plant species composition changed from predominantly annual and perennial herbaceous species to a more diverse community with drought-tolerant and heliophilous shrubs. The vegetation cover was highly positively correlated with soil depth and soil volume (p < 0.001), and also significantly correlated with soil organic matter, total N, and available N and P concentrations (p < 0.05). This suggests that vegetation succession is more limited by available soil volume than by soil fertility during the early stages of quarry restoration.