Body size of animals often increases with increasing latitude. These latitudinal clines in body size have interested biologists for over 150 years. However, the mechanisms that generate these clines in size are still unclear, though latitudinal gradients in temperature appear to play an important role. More importantly, many studies that examine latitudinal clines in body size and the mechanisms responsible for these clines use phenotypic data, confounding genetic (adaptive) and non-genetic (plasticity) sources of variation. Yet, most of these studies make adaptive conclusions based on phenotypic measures of size. Here I show the dangers of making adaptive inferences from phenotypic measures of size. In addition, I use a specific form of plasticity in body size of ectotherms, called the temperature–size rule, to illustrate how confusion about genetic and non-genetic contributions to phenotypic variation has hampered progress in understanding the evolution of latitudinal clines in size. Field-based measurements of body size can no doubt be influenced by plasticity, but demonstrating that latitudinal clines have a genetic basis is necessary to show that these patterns are adaptive.