Livestock guard dog (LGD) breeds from the Western Balkans are a good example of how complex genetic diversity pattern observed in dog breeds has been shaped by transition in dog breeding practices. Despite their common geographical origin and relatively recent formal recognition as separate breeds, the Karst Shepherd, Sarplaninac and Tornjak show distinct population dynamics, assessed by pedigree, microsatellite and mtDNA data. We genotyped 493 dogs belonging to five dog breeds using a set of 18 microsatellite markers and sequenced mtDNA from 94 dogs from these breeds. Different demographic histories of the Karst Shepherd and Tornjak breeds are reflected in the pedigree data with the former breed having more unbalanced contributions of major ancestors and a realized effective population size of less than 20 animals. The highest allelic richness was found in Sarplaninac (5.94), followed by Tornjak (5.72), whereas Karst Shepherd dogs exhibited the lowest allelic richness (3.33). Similarly, the highest mtDNA haplotype diversity was found in Sarplaninac, followed by Tornjak and Karst Shepherd, where only one haplotype was found. Based on FST differentiation values and high percentages of animals correctly assigned, all breeds can be considered genetically distinct. However, using microsatellite data, common ancestry between the Karst Shepherd and Sarplaninac could not be reconstructed, despite pedigree and mtDNA evidence of their historical admixture. Using neighbour-joining, STRUCTURE or DAPC methods, Sarplaninac and Caucasian Shepherd breeds could not be separated and additionally showed close proximity in the NeighborNet tree. STRUCTURE analysis of the Tornjak breed demonstrated substructuring, which needs further investigation. Altogether, results of this study show that the official separation of these dog breeds strongly affected the resolution of genetic differentiation and thus suggest that the relationships between breeds are not only determined by breed relatedness, but in small populations even more importantly by stochastic effects.