Tree growth response across environmental gradients is fundamental to understanding species distributional ecology and forest ecosystem ecology and to predict future ecosystem services. Cross-sectional patterns of ecosystem properties with respect to climatic gradients are often used to predict ecosystem responses to global change. Across sites in the tropics, primary productivity increases with temperature, suggesting that forest ecosystems will become more productive as temperature rises. However, this trend is confounded with a shift in species composition and so may not reflect the response of in situ forests to warming. In this study, we simultaneously studied tree diameter growth across the altitudinal ranges of species within a single genus across a geographically compact temperature gradient, to separate the direct effect of temperature on tree growth from that of species compositional turnover. Using a Bayesian state space modeling framework we combined data from repeated diameter censuses and dendrometer measurements from across a 1700-m altitudinal gradient collected over six years on over 2400 trees in Weinmannia, a dominant and widespread genus of cloud forest trees in the Andes. Within species, growth showed no consistent trend with altitude, but higher-elevation species had lower growth rates than lower-elevation species, suggesting that species turnover is largely responsible for the positive correlation between productivity and temperature in tropical forests. Our results may indicate a significant difference in how low- and high-latitude forests will respond to climate change, since temperate and boreal tree species are consistently observed to have a positive relationship between growth and temperature. If our results hold for other tropical species, a positive response in ecosystem productivity to increasing temperatures in the Andes will depend on the altitudinal migration of tree species. The rapid pace of climate change, and slow observed rates of migration, suggest a slow, or even initially negative response of ecosystem productivity to warming. Finally, this study shows how the observed scale of biological organization can affect conclusions drawn from studies of ecological phenomena across environmental gradients, and calls into question the common practice in tropical ecology of lumping species at higher taxonomic levels.