A major focus of comparative neuroanatomy has been on whether the mammalian brain evolves in a concerted or a mosaic fashion. Workers have examined variation in the volume of different brain regions across taxa to test the degree to which selection is constrained by the timing of events in neural development. Whether a conserved neurogenetic program in the mammalian brain constrains the distribution of different cell types, however, has not yet been investigated. Here we tested for evidence of evolutionary constraints on the densities of different cell types in the primary visual cortex (V1) and the hippocampus in 37 primate and 21 carnivore species. Cellular densities in V1 and the hippocampus scale isometrically with respect to one another in carnivores, as predicted by the concerted evolution hypothesis. In primates, however, cellular distributions in the hippocampus and primary visual cortex show no correlations, which supports the hypothesis of mosaic brain evolution. We therefore provide evidence for the presence of constraints controlling the adult densities of different cell types in disparate regions of the mammalian brain, but also for specializations along the primate lineage. We propose that adaptations to modularity at the cellular level may carry a deep phylogenetic signal.