The vegetation cover during the non-cropping season could have important implications for the maintenance and recovery of soil fertility, as well as for biodiversity conservation in croplands. In this study, five fertilization regimes (control: non-fertilization; N: inorganic N fertilization; P: inorganic P fertilization; NPK: balanced fertilization with inorganic N, P and K; NPKM: balanced NPK plus farmyard manure) were conducted from 1981 in a double-rice (Oryza sativa L.)-cropping system in subtropical China. The effects of long-term fertilization were investigated on the weed growth, diversity and community structure during the fallow period. The results showed that, relative to the control, both inorganic fertilization alone (N, P and NPK) and NPKM in the rice-growing season significantly increased the weed density and biomass during the fallow period in the paddy field. There was no significant difference in the weed species richness (the number of species) among the treatments. Compared with the control, fertilization tended to reduce the weed diversity (Shannon's H′) and evenness (Shannon's E), especially in the N treatment. Long-term fertilization resulted in a significant shift in weed community's composition during the fallow period. The weed community's structure was affected by soil nutrients in the order P > N > K.