The reproductive success of alternative mating behaviours may vary within and among populations in relation to environmental factors and demographic parameters. We used behavioural and genetic data to investigate how male density affects reproductive success of territoriality and sneaking in the European bitterling (Rhodeus sericeus, Cyprinidae), a freshwater fish that spawns on the gills of living freshwater mussels. Keeping the number of spawning sites constant, we manipulated male densities in laboratory and mesocosm experiments. We showed that sneaked fertilizations were common in R. sericeus, and that they increased significantly with male density. Territorial mating was almost 17 times more successful than sneaking at the lowest male density treatment, and still 2–3 times more successful at intermediate densities. However, both behaviours conferred the same fitness pay-off at the highest male density. While the success of territorial males declined with male density, the success of individual sneaking males remained constant across densities. Notably, the capacity of territorial males to outcompete sneakers by preoviposition sperm loading was the best predictor of male reproductive success, rather than aggression, body size or postoviposition ejaculation.