The size of the vertebrate brain is shaped by a variety of selective forces. Although larger brains (correcting for body size) are thought to confer fitness advantages, energetic limitations of this costly organ may lead to trade-offs, for example as recently suggested between sexual traits and neural tissue. Here, we examine the patterns of selection on male and female brain size in pinnipeds, a group where the strength of sexual selection differs markedly among species and between the sexes. Relative brain size was negatively associated with the intensity of sexual selection in males but not females. However, analyses of the rates of body and brain size evolution showed that this apparent trade-off between sexual selection and brain mass is driven by selection for increasing body mass rather than by an actual reduction in male brain size. Our results suggest that sexual selection has important effects on the allometric relationships of neural development.