Identifying conservation units below the species level is becoming increasingly important, particularly when limited resources necessitate prioritization for conservation among such units. This problem is exemplified with caribou, a mammal with a circum-Arctic distribution that is exposed to a broad spectrum of ecological conditions, but is also declining in many parts of its range. We used microsatellite markers to evaluate the suitability of existing intra-specific taxonomic designations to act as population units for conservation and contrasted this with landscape features that were independent of taxonomy. We also quantified the relationship between genetic differentiation and subpopulation size, a factor that has been under-represented in landscape genetic research. Our data set included three subspecies and three ecotypes of caribou that varied in population size by five orders of magnitude. Our results indicated that genetic structure did not correspond to existing taxonomic designation, particularly at the level of ecotype. Instead, we found that major valleys and population size were the strongest factors associated with substructure. There was a negative exponential relationship between population size and FST between pairs of adjacent subpopulations, suggesting that genetic drift was the mechanism causing the structure among the smallest subpopulations. A genetic assignment test revealed that movement among subpopulations was a fraction of the level needed to stabilize smaller subpopulations, indicating little chance for demographic rescue. Such results may be broadly applicable to landscape genetic studies, because population size and corresponding rates of drift have the potential to confound interpretations of landscape effects on population structure.