Looking for somebody's face in a crowd is one of the most important examples of visual search. For this goal, attention has to be directed to a well-defined perceptual category. When this categorically selective process starts is, however, still unknown. To this end, we used magnetoencephalography (MEG) recorded over right human occipitotemporal cortex to investigate the time course of attentional modulation of perceptual processes elicited by faces and by houses. The first face-distinctive MEG response was observed at 160–170 ms (M170). Nevertheless, attention did not start to modulate face processing before 190 ms. The first house-distinctive MEG activity was also found around 160–170 ms. However, house processing was not modulated by attention before 280 ms (90 ms later than face processing). Further analysis revealed that the attentional modulation of face processing is not due to later, for example, back-propagated activation of the M170 generator. Rather, subsequent stages of occipitotemporal object processing were modulated in a category-specific manner and with preferential access to face processing.