Gaze following is a fundamental component of triadic social interaction which includes events and an object shared with other individuals and is found in both human and nonhuman primates. Most previous work has focused only on the immediate reaction after following another’s gaze. In contrast, this study investigated whether gaze following is retained after the observation of the other’s gaze shift, whether this retainment differs between species and age groups, and whether the retainment depends on the nature of the preceding events. In the social condition, subjects (1- and 2-year-old human children and chimpanzees) witnessed an experimenter who looked and pointed in the direction of a target lamp. In the physical condition, the target lamp blinked but the experimenter did not provide any cues. After a brief delay, we presented the same stimulus again without any cues. All subjects looked again to the target location after experiencing the social condition and thus showed a carryover effect. However, only 2-year-olds showed a carryover effect in the physical condition; 1-year-olds and chimpanzees did not. Additionally, only human children showed spontaneous interactive actions such as pointing. Our results suggest that the difference between the two age groups and chimpanzees is conceptual and not only quantitative.