Climate change is inducing changes in the phenological timings of organisms. Genetic diversity could influence phenological responses to climate change, but empirical evidence is very limited. We estimated the regional variation across Japan in flowering and leaf budburst dates of plants based on a dataset of phenological timings from 1953 to 2005. The observed plants' genetic diversities varied according to human cultivation. The within-species variations of phenological response to temperature as well as regional variations were less in the plant populations with lower genetic diversity. Thus, genetic diversity influences the variation in phenological responses of plant populations. Under increased temperatures, low variation in phenological responses may allow drastic changes in the phenology of plant populations with synchronized phenological timings. Our findings indicate that we should pay attention to maintaining genetic diversity of populations to alleviate changes in phenology due to future climate change.