Investigating the mating system of a population provides insight into the evolution of reproductive patterns, and can inform conservation management of threatened or endangered species. Combining behavioural and genetic data is necessary to fully understand the mating system and factors affecting male reproductive success, yet behavioural data are often difficult to collect for threatened species. In the present study, we use behavioural data and paternity analyses to characterize the mating system of a high density population of a long-lived, ancient reptile (tuatara, Sphenodon punctatus). We further investigate the phenotypic traits (including body size, body condition, tail length, and ectoparasite load) that affect male reproductive success. Our behavioural data reflect a seasonally monogamous system with low levels of polyandry and polygyny that are consistent with male mate guarding. Male reproduction is highly skewed (only 25–30% of males are successful), and body size is the primary predictor of male reproductive success. Based on the genetic data, multiple paternity was found in only 8% of clutches, and the results of the paternity analyses showed monandrous clutches from socially polyandrous females. Our behavioural and genetic results revealed complexities in female mating patterns that support the potential for cryptic female choice or sperm competition. This warrants further experimental investigation into the mechanisms underlying reptile fertilization and the disparities between social and genetic polyandry in wild populations. © 2009 The Linnean Society of London, Biological Journal of the Linnean Society, 2009, 98, 161–170.