In the human species, the two uniparental genetic systems (mitochondrial DNA and Y chromosome) exhibit contrasting diversity patterns. It has been proposed that sex-specific behaviours, and in particular differences in migration rate between men and women, may explain these differences. The availability of high-density genomic data and the comparison of genetic patterns on autosomal and sex chromosomes at global and local scales allow a reassessment of the extent to which sex-specific behaviours shape our genome. In this article, we first review studies comparing the genetic patterns at uniparental and biparental genetic systems and assess the extent to which sex-specific migration processes explain the differences between these genetic systems. We show that differences between male and female migration rates matter, but that they are certainly not the only contributing factor. In particular, differences in effective population size between men and women are also likely to account for these differences. Then, we present and discuss three anthropological processes that may explain sex-specific differences in effective population size and thus human genomic variation: (i) variance in reproductive success arising from, for example, polygyny; (ii) descent rules; and (iii) transmission of reproductive success.