Assessing how genes flow across populations is a key component of conservation genetics. Gene flow in a natural population depends on ecological traits and the local environment, whereas for a livestock population, gene flow is driven by human activities. Spatial organization, relationships between farmers and their husbandry practices will define the farmer’s network and so determine farmer connectivity. It is thus assumed that farmer connectivity will affect the genetic structure of their livestock. To test this hypothesis, goats reared by four different ethnic groups in a Vietnamese province were genotyped using 16 microsatellites. A Bayesian approach and spatial multivariate analysis (spatial principal component analysis, sPCA) were used to identify subpopulations and spatial organization. Ethnic group frequencies, husbandry practices and altitude were used to create cost maps that were implemented in a least-cost path approach. Genetic diversity in the Vietnamese goat population was low (0.508) compared to other local Asian breeds. Using a Bayesian approach, three clusters were identified. sPCA confirmed these three clusters and also that the genetic structure showed a significant spatial pattern. The least-cost path analysis showed that genetic differentiation was significantly correlated (0.131–0.207) to ethnic frequencies and husbandry practices. In brief, the spatial pattern observed in the goat population was the result of complex gene flow governed by the spatial distribution of ethnic groups, ethnicity and husbandry practices. In this study, we clearly linked the livestock genetic pattern to farmer connectivity and showed the importance of taking into account spatial information in genetic studies.