In this study, 4-month-old infants’ and adults’ spontaneous preferences for emotional and neutral displays with direct and averted gaze are investigated using visual preference paradigms. Specifically, by presenting two approach-oriented emotions (happiness and anger) and two avoidance-oriented emotions (fear and sadness), we asked whether the pattern of emotion–gaze interaction suggested by the shared signal hypothesis (SSH) would also be found with this paradigm. Both age groups demonstrated an ability to discern the approach- and avoidance-oriented emotions, matching them with direct and averted gaze, respectively. Nonetheless, infants showed a greater sensitivity for the congruent emotion-gaze combination in the approach-oriented emotions, while adults were equally sensitive to the gaze-expression congruence for both the approach- and avoidance-oriented emotions. In a follow-up experiment, infants showed no preference for direct or averted gaze in the context of neutral faces. We conclude that the SSH may have validity from infancy, gradually extending from approach-oriented emotions to avoidance-oriented emotions over the course of development.