Polygynous mating results in nonrandom sampling of the adult male gamete pool in each generation, thereby increasing the rate of genetic drift. In principle, genetic paternity analysis can be used to infer the effective number of breeding males (Nebm). However, this requires genetic data from an exhaustive sample of candidate males. Here we describe a new approach to estimate Nebm using a rejection algorithm in association with three statistics: Euclidean distance between the frequency distributions of maternally and paternally inherited alleles, average number of paternally inherited alleles and average gene diversity of paternally inherited alleles. We quantify the relationship between these statistics and Nebm using an individual-based simulation model in which the male mating system varied continuously between random mating and extreme polygyny. We evaluate this method using genetic data from a natural population of highly polygynous fruit bats (Cynopterous sphinx). Using data in the form of mother–offspring genotypes, we demonstrate that estimates of Nebm are very similar to independent estimates based on a direct paternity analysis that included data on candidate males. Our method also permits an evaluation of uncertainty in estimates of Nebm and thus facilitates inferences about the mating system from genetic data. Finally, we investigate the sensitivity of our method to sample size, model assumptions, adult population size and the mating system. These analyses demonstrate that the rejection algorithm provides accurate estimates of Nebm across a broad range of demographic scenarios, except when the true Nebm is high.