The number of group members in an animal society can have a major influence on group members’ life history, survival, and reproductive success. Identifying the factors that limit group size is therefore fundamental for a complete understanding of social behavior. Here, I examined the relationships between resource availability, social conflict, and group size in the coral-dwelling fish, Paragobiogon xanthosomus (Gobiidae). The size of the largest (breeding) female and the minimum size difference required for hierarchy stability strongly but not perfectly predicted maximum group size, suggesting that social conflicts and hierarchy structure set the upper limit on group size. Deviations in group size around the predicted maximum were explained by variation in average body size ratios but not by variation in coral size, suggesting that coral size does not directly influence group size. In contrast, coral size was a significant predictor of body size ratios, and possible explanations for this relationship are discussed. Group size may be limited by the social conflicts that characterize size-based dominance hierarchies. Ecological factors, namely coral size, may in turn play an indirect role via an effect on body size ratios.