Individuals obtain a range of benefits from aggregating with others. Evidence suggests that group size and the phenotypic characteristics of potential group mates are both important attributes influencing the grouping decisions of prospective group members. Their interaction, however, remains poorly understood. Here we investigate, in a series of dichotomous choice experiments, what happens when a preference with respect to group size is pitted against a preference for assorting on the basis of body size in swordtail fish. When controlled for body size, we found that swordtails preferred to associate with a larger shoal. When controlled for group size, we found that swordtails preferred to associate with similar-sized fish. However, when offered the choice between a single size-matched shoaling partner and four dissimilar-sized shoal mates, we found that females associated randomly. Our results suggest that, depending on context, group size and body size may be equally important in guiding the aggregating decisions of swordtails. Future studies will likely offer valuable insights into shoaling behaviour by considering how, and under what circumstances, different grouping preferences might be prioritized.