Although loss of genetic variation is frequently assumed to be associated with loss of adaptive potential, only few studies have examined adaptation in populations with little genetic variation. On the Swedish west coast, the northern fringe populations of the natterjack toad Bufo calamita inhabit an atypical habitat consisting of offshore rock islands. There are strong among-population differences in the amount of neutral genetic variation, making this system suitable for studies on mechanisms of trait divergence along a gradient of within-population genetic variation. In this study, we examined the mechanisms of population divergence using QST–FST comparisons and correlations between quantitative and neutral genetic variation. Our results suggest drift or weak stabilizing selection across the six populations included in this study, as indicated by low QST–FST values, lack of significant population × temperature interactions and lack of significant differences among the islands in breeding pond size. The six populations included in this study differed in both neutral and quantitative genetic variation. Also, the correlations between neutral and quantitative genetic variation tended to be positive, however, the relatively small number of populations prevents any strong conclusions based on these correlations. Contrary to the majority of QST–FST comparisons, our results suggest drift or weak stabilizing selection across the examined populations. Furthermore, the low heritability of fitness-related traits may limit evolutionary responses in some of the populations.