Previous judgment studies of facial expressions of emotion in context have provided mixed results. These contradictory findings arise, in part, because of a lack of consideration of the different aspects of context with which facial expressions of emotion can be combined, and different meanings of face-context pairings vis-à-vis real-life phenomena. In this paper, we provide a framework of how faces and contexts combine and describe possible meanings of consistent and inconsistent face-context pairings. We also discuss methodological caveats that should be considered in the conduct of such judgment studies. We argue that future research in this area will benefit not from more studies of different combinations of stimuli created merely because we have the technological ability to create them, but instead from careful consideration of the ecological validity of the various types of combinations that can be produced, and their meaning.