Sympatric divergence in traits affecting species recognition can result from selection against cross-species mating (reproductive character displacement, RCD) or interspecific aggression (agonistic character displacement, ACD). When the same traits are used for species recognition in both contexts, empirically disentangling the relative contributions of RCD and ACD to observed character shifts may be impossible. Here, we develop a theoretical framework for partitioning the effects of these processes. We show that when both mate and competitor recognition depend on the same trait, RCD sets the pace of character shifts. Moreover, RCD can cause divergence in competitor recognition, but ACD cannot cause divergence in mate recognition. This asymmetry arises because males with divergent recognition traits may avoid needless interspecific conflicts, but suffer reduced attractiveness to conspecific females. Therefore, the key empirical issue is whether the same or different traits are used for mate recognition and competitor recognition.