Ecological factors have been shown to be important for brain size evolution. In this comparative study among catarrhine primates, we examine two different ways in which seasonality may be related to brain size. First, seasonality may impose energetic constraints on the brain because it forces animals to deal with periods of food scarcity (Expensive Brain hypothesis). Second, seasonality may act as a selective pressure to increase brain size, as behavioral flexibility helps to overcome periods of food scarcity (Cognitive Buffer hypothesis). Controlling for phylogeny, we found a strong negative relationship between brain size (relative to body mass) and the degree of experienced seasonality, as estimated by the variation in net energy intake. However, we also found a significant positive relationship between relative brain size and the effect of so-called cognitive buffering, proxied by the difference between environmental seasonality and the seasonality in net energy intake actually experienced by the animals. These results show that both energetic constraints of seasonal habitats as well as cognitive buffering affect brain size evolution, leaving environmental seasonality uncorrelated to brain size. With this study we show the importance of simultaneously considering both costs and benefits in models of brain size evolution.