The early phases of plant colonization after the abandonment of agricultural land are a crucial starting point for a suite of successional mechanisms, such as priority effects, facilitation, and inhibition. Therefore, the first years of vegetation development substantially shape the trajectory of future vegetation recovery on former agricultural sites. Studies describing the abandonment of traditional, extensive agriculture are numerous, but what about intensively cultivated sites? This study covers the vegetation development in 47 plots in former maize fields in an intensive agricultural landscape in Belgium (Western Europe); fields were abandoned in 2001, 2002, and 2003. The woody canopy and the herbaceous understorey were recorded in 2006 (and in 2003 for the 2001 fields). The additional environmental variables measured were plot distance to seed trees and forest edges, soil nutrients, and soil pH. The colonization success of woody species in terms of abundance and species composition was highly variable between years, and presence–absence was hard to explain. However, the growth of the most abundant species (Salix caprea agg.) was positively related to a soil nutrient gradient. Whether or not a woody canopy did establish successfully was related to alternative trajectories of herbaceous vegetation development. Under a sparse canopy of woody species, fast-growing, opportunistic species became dominant. A denser canopy favored a higher percentage of bare ground, which may have offered suitable sites for the recruitment of additional shrub and tree species. Apparently, strong agricultural legacies can be both beneficial and detrimental to vegetation recovery following abandonment.