In three-stimulus oddball studies, even typical deviant stimuli elicited a large P3a event-related brain potential (ERP) when target/standard discrimination was difficult. To investigate the underlying mechanisms, the effects of task difficulty on early deviant-related ERPs were assessed. Four visual stimuli defined by an orthogonal combination of task-relevant size (nontarget 80%, target 20%) and task-irrelevant luminance (standard 80%, deviant 20%) were presented randomly, where two task difficulties (easy, difficult) were defined by target/nontarget discriminability. An increase in task difficulty enhanced P3a as well as a posterior negativity (change-related negativity) and an anterior positivity (frontal positivity) elicited by deviant nontarget stimuli. These results suggest that attentional modulation of refractoriness-based rareness detection and an attention-triggering process underlie the P3a task-difficulty effect.