Gene flow and genetic drift are important factors affecting geographic variations in human phenotypic traits. In the present study, the effects of gene flow from an outside source on the pattern of within- and among-group variation of the Ainu from Sakhalin Island and three local groups of Hokkaido are examined by applying an R-matrix approach to metric and nonmetric dental data. The comparative samples consist of their ancestral and neighboring populations, such as the Neolithic Jomon, the subsequent Epi-Jomon/Satsumon, the Okhotsk culture people who migrated from Northeast Asia to the northeastern part of Hokkaido during a period 1600–900 years B.P., and modern non-Ainu Japanese. The results obtained by using the census population sizes of the regional groups of the Ainu as an estimate of relative effective population size suggest the possibility of an admixture between the Okhotsk culture people and the indigenous inhabitants in Hokkaido, at least in the coastal region along the Sea of Okhotsk. Such gene flow from Northeast Asian continent may have exerted an effect on the genetic structure of the contemporary Ainu. The present findings indicate that the population structure, as represented by genetic drift and gene flow, tend to be obscured in the results obtained by standard statistical methods such as Mahalanobis' generalized distance and Smith's MMDs. The present extension of the R-matrix approach to metric and nonmetric dental data provide results that can be interpreted in terms of a genetically, archaeologically, and prehistorically suggested pattern of gene flow and isolation. Am. J. Hum. Biol., 2010. © 2009 Wiley-Liss, Inc.