The research examined whether subjects with hearing impairment would differ from normal hearing subjects in their ability to decode emotions from video stimuli (48 video takes in which two actors portrayed six different emotions). Studies in tactile and visual perception lead one to expect deficits, while there is also some evidence for compensation. Twenty-six subjects with hearing impairment and 26 matched normal hearing subjects participated (average age = 25.5 years; nine female, 17 male subjects in each group). Results indicate that in general subjects with hearing impairment were slightly less successful in decoding emotions from the visual stimuli than the normal hearing subjects. A comparison between highly (loss > 60–90 dBA) and moderately (loss about 30–60 dBA) impaired subjects on the other hand indicated poorer emotion decoding only for the moderately impaired group. Post-hoc analyses indicated that these effects were specific to males. Results are discussed with respect to compensation versus deficit, and with respect to issues of training.