Phenotypic divergence between populations, i.e. how much phenotypes within a species vary geographically, is critical to many aspects of ecology and evolution, including eco-geographical trends, speciation and coexistence. Yet, the variation of divergence across species with different ecologies and distributions and the relative role of adaptive causes remains little understood. We predict that genetic control vs. phenotypic plasticity of traits, geographical distance and (assuming adaptation) environmental differences should explain much of the phenotypic variability between populations. We tested these predictions with body sizes of 1447 populations in 98 terrestrial vertebrate species. Population phenotypic variability differs strongly across species, and divergence increases with increasing levels of clade-typical phenotypic plasticity, the area covered by populations and body size. Geographical distance and environmental dissimilarity are similarly important predictors of divergence within species, highlighting a potential role for biotic and environmental conditions. Increased availability of phylogeographical and ecological data should facilitate further understanding of population divergence drivers at broad scales.