Research suggests that averting gaze from an interlocutor can improve both children's and adults' performance in a range of cognitive tasks. With the present experiments, we investigated the effect of gaze aversion on adults' visual-spatial imagination, using a methodology adapted from Kerr (1987). Participants mentally kept track of a pathway through an imaginary matrix, while either maintaining eye-contact with the experimenter, closing their eyes, gazing at a static or a dynamic visual stimulus (in Experiment 1), or fixating an upright or inverted image of the experimenter's face (in Experiment 2). The results show that whereas maintaining eye-contact with another person disrupts accurate imagination of this pathway, averting gaze or looking at other visual stimuli does not. We conclude that gaze aversion benefits cognitive performance, not just by disengaging visual attention from irrelevant visual information, but also by interrupting social interaction processes involved in face-to-face communication.