The amygdala is considered pivotal to processing emotional information [Dolan, 2002; Phelps and LeDoux, 2005], notably for threat-related stimuli [Amaral, 2002]. Projections from the amygdala are hypothesized to orchestrate adaptive behavioral and autonomic reactions [Amaral, 2002; LeDoux, 1996], modulate emotional face processing in extrastriate visual cortex [Morris et al., 1998; Noesselt et al., 2005; Rotshtein et al., 2001; Vuilleumier et al., 2004], and influence encoding of emotional stimuli into long-term memory [Dolan, 2002; Richardson et al., 2004; Smith et al., 2006]. Amygdala lesions abolish differential BOLD responses to fearful stimuli (vs. neutral) in extrastriate regions [Vuilleumier et al., 2004]. This latter observation accords with a causal role for the amygdala in mediating modulatory influences on sensory cortices [Amaral, 2003; Armony et al., 1998; LeDoux, 1996; Phelps and LeDoux, 2005]. However, the sluggish nature of the BOLD response means that fMRI data alone cannot reveal the time-course of remote amygdala influences. Such information is important to establish whether amygdala emotional responses precede or follow other perceptual and face-selective responses, such as those within visual cortex (see detailed discussion below).
Evoked related potential (ERP) studies in healthy volunteers have reported effects of threat-related stimuli (e.g. seen fearful faces) at several time windows. The earliest differential response for fearful versus neutral faces is found around ∼120 ms poststimulus onset [Eger et al., 2003; Eimer and Holmes, 2002; Pourtois et al., 2004, 2005a; Streit et al., 2003; van Heijnsbergen et al., 2007], corresponding to the visual P1 component that has been argued to reflect perceptual processing [Aru and Bachmann, 2009; Marzi et al., 2000; Thorpe et al., 1996; though see Brisson and Jolicoeur [ 2008]]. The P1 component is hypothesized to be generated in posterior occipito-temporal areas [Di Russo et al., 2002]. Emotional effects in this time window have also been reported for subliminal stimuli [Liddell et al., 2004]. The time window of this early scalp ERP effect is in line with initial response latencies of the amygdala shown by intracranial recording in both humans and monkeys [Gothard et al., 2007; Krolak-Salmon et al., 2004; Oya et al., 2002], though may have a different source. A second component, at around ∼200 ms for fronto-central electrodes, shows differential responses to threat stimuli depending on the intensity on the depicted emotion and its perception [Ashley et al., 2004; Eimer et al., 2008; Holmes et al., 2003; Leppanen et al., 2007; Liddell et al., 2005; Sprengelmeyer and Jentzsch, 2006]. This subsequent ERP effect may last up to 1 s poststimulus presentation [Eimer and Holmes, 2007]. Emotional modulations are also been reported in the human amygdala within this time range [Krolak-Salmon et al., 2004]. Finally, a late effect of emotions on ERPs is usually seen over central electrodes after ∼500 ms [Johansson et al., 2004; Keil et al., 2001; Kissler et al., 2006] and is associated with influences of emotion on episodic memory processes [Liddell et al., 2004; Maratos et al., 2000]. It remains entirely unknown whether projections from the amygdala contribute to these different ERP effects measured at the scalp. Specifically, it is unresolved how early in time amygdala influences on distant cortical regions can arise in humans and which (if any) threat-related ERP components reflect direct influences from the amygdala.
Two opposing perspectives have been advanced regarding the possible time-course of amygdala involvement in processing threat-related information. The first emphasizes that the amygdala receives information following processing by the corresponding sensory cortices and possibly after conscious perception by the observer [Pessoa et al., 2005a, b]. In accord with this, some neuroimaging studies in healthy humans have indicated that some aspects of the BOLD response in the amygdala depend on conscious perception of the stimulus and/or attention [Holmes et al., 2003; Pessoa et al., 2002, 2005a; Phillips et al., 2004]. According to this perspective, one might anticipate that amygdala function should influence threat-related ERPs relatively late in poststimulus onset, e.g., perhaps after ∼150 ms when the major effects of attention are typically observed [Di Russo et al., 2003] and when perception may become conscious [Thorpe et al., 1996], including for face expressions [Ashley et al., 2004; Eimer et al., 2008; Holmes et al., 2003; Leppanen et al., 2007; Liddell et al., 2005; Sprengelmeyer and Jentzsch, 2006].
An alternative perspective suggests that a rapid response, initiated by the amygdala, allows a fast modulation of sensory processing by the amygdala, especially for information containing threat-related signals [Compton, 2003; Dolan and Vuilleumier, 2003; LeDoux, 1996; Ohman et al., 2007; Ohman, 2005; Phelps and LeDoux, 2005; Vuilleumier, 2005]. This perspective receives support from neuropsychological studies of “blindsight” patients with lesions to striate and extrastriate cortex, who still retain some ability to respond to the presence of threat-related stimuli [de Gelder et al., 1999; Pegna et al., 2005] and can show preserved amygdala response during fMRI [de Gelder et al., 2005; Morris et al., 2001; Pegna et al., 2005]. Neuroimaging studies in healthy participants have also shown amygdala responses to subliminal threat stimuli [Carlsson et al., 2004; Morris et al., 1999] and to stimuli outside the main focus of attention [Vuilleumier et al., 2001]. From this perspective, one might anticipate that amygdala influences upon visual ERPs might emerge relatively early, possibly even for the first ERP effects produced by emotion expressions, i.e., for the P1 component at ∼120 ms. However, the actual time-course of amygdala contributions to threat-related ERPs has never been directly investigated in humans previously.
A further controversy is whether intact amygdala function is necessary for explicit perception of seen emotional expressions, particularly fear [Cristinzio et al., 2007; Graham et al., 2006; Rapcsak et al., 2000]. Neuropsychological research suggests that impairments in fear recognition may arise only after bilateral lesions to the amygdala [Adolphs et al., 1994], but this is not observed in all bilateral amygdala patients [Adolphs et al., 1999; Graham et al., 2006; Hamann and Adolphs, 1999]. Unilateral amygdala lesions are usually associated with intact explicit fear perception [Adolphs et al., 1994], though both unilateral right [Meletti et al., 2003] and left amygdala damage have been reported to have some impact on fear perception [Graham et al., 2006]. It is notable that most patients who show clear deficits in explicit perception of fear have extensive lesions not confined to the amygdala but impacting also on adjacent structures [Adolphs and Tranel, 2003; Adolphs et al., 2001; Anderson and Phelps, 2000; Anderson et al., 2000; Brierley et al., 2004; Broks et al., 1998]. Furthermore, several patients who show impairment in explicit perception of fearful expressions due to amygdala pathology were diagnosed with the rare, genetically determined, Urbach–Wiethe syndrome [Adolphs et al., 1994, 1995; Siebert et al., 2003]. Neuroimaging studies provide inconclusive evidence regarding the role of the amygdala in explicit recognition of expressions. Some studies suggest that amygdala activation to fearful stimuli arises primarily during explicit recognition tasks [Krolak-Salmon et al., 2004], but many others indicate that amygdala responsiveness does not depend entirely on explicit expression recognition [Bleich-Cohen et al., 2006; Critchley et al., 2000; Gothard et al., 2007; Vuilleumier et al., 2001].
The aims of our study were to determine the time-course of amygdala influences on fearful expression processing, by measuring ERPs to fearful versus neutral faces, and comparing groups of participants with or without amygdala damage. We were also interested in determining any impact of such amygdala damage on explicit fear perception from facial expressions. To address these questions, we combined neuropsychological and electrophysiological methods. We recorded ERPs to fearful and neutral facial expressions from unoperated patients that suffer from medial temporal lobe epilepsy (MTLE), with varying degrees of amygdala pathology, plus healthy volunteers. Our analysis focused on differential (fear minus neutral) effects of emotional expression on evoked responses, from early stimulus onset up to 600 ms. In an additional behavioral experiment, we measured explicit recognition of fearful expressions in the MTLE patients and healthy participants.
PROCEDURES AND METHODS
Participants included 17 MTLE patients, of whom 7 (4 female, 3 left handed, mean age 34.5 years, range 21–42 years) had structural pathology that included the amygdala (“MTLE-amygdala”) and 10 others (6 female, 2 left handed, mean age 37.7 years, range 23–48 years) with damage sparing the amygdala but affecting the hippocampus and/or other temporal regions (“MTLE-control”). None of the MTLE patients experienced any seizure for at least 24 h prior to the study. In addition, 13 healthy controls (7 female, 1 left handed, mean age 31.6 years, range 20–58 years) were recruited, most of them from among friends and relatives of the MTLE patients. None of the healthy controls had a clinical history of neurological or psychiatric illness. All participants gave written informed consent in accordance with local ethics.
The MTLE patients were recruited from a specialist epilepsy center and assigned to groups based on clinical diagnosis. This diagnosis was made by a clinician blind to our ERP and behavioral hypotheses. Note that all patients were diagnosed with unilateral brain pathology. Magnetic resonance (MR) T2-weighted images were acquired as part of the standard clinical diagnosis, as an indication for sclerosis [Bartlett et al., 2002]. T2 relaxation time above 92 ms within the amygdala indicates abnormal tissue [Bartlett et al., 2002]. Furthermore, the distribution of T2 signal within the healthy population is skewed and hence high T2 values, even below 92 ms (the clinical threshold), are rare in healthy individuals [Bartlett et al., 2002]. We used the structural T2 values as a further regressor in our ERP analyses, to perform quantitative tests of any relationship between the severity of amygdala pathology and observed ERP effects. Six MTLE-amygdala patients were classified as showing amygdala sclerosis (five left and one right). One MTLE-amygdala patient was diagnosed as having damage to the left amygdala based on assessment of the amygdala structure in her T1-weighted MR images. This patient did not suffer from sclerosis.
Importantly, the two patient groups did not differ significantly on clinical measures, including duration of epilepsy, seizure severity, treatment, or general intelligence (for details, see Table I), with any nonsignificant trends suggesting that, if anything, the MTLE-controls were slightly more impaired than the MTLE-amygdala group. Note that this tendency could only work against the effects we report below. From all the participants who underwent the ERP study, 10 healthy, 9 MTLE-control, and 7 MTLE-amygdala participants also completed a behavioral test of explicit perception for facial expressions (see below). Crucially, during the data collection, the experimenters were blind to the specific group-assignment of the patients. The participants were also naïve as to the aim of the experiment, but were debriefed at the end of the experiment.
Table I. Participant details
Onset Epi (years)
Means are given with ranges below in each cell where appropriate. There were no significant differences between patients groups on any clinical measure, except lesion (see T2 structural MR data in Fig. 2E scatter-plot, along x-axis there), and no significant difference between patient groups and healthy controls for age, gender, or handedness.
H, handedness; RH, right handed; H, hippocampus; HA, hippocampus + amygdala; TLE, temporal lobe epilepsy with exact lesion unspecified; MTS, medial temporal sclerosis
Epi, epilepsy. Note that the etiology of 1 MTLE-control was non-specific temporal epileptic, with no clear structural abnormality, 2 MTLE-amygdala had tumour in the amygdala. Medications refers to the total doses of drugs a patient receive.
Female = 5
RH = 7
LH (9); TLE (1)
Female = 4
RH = 5
LHA(5); LA (1); RA (1)
Female = 7
RH = 12
Behavioral Study—Explicit Fear Perception for Seen Faces
Expression perception was evaluated using a categorization task of prototypical [Ekman and Friesen, 1976] and ambiguous facial expressions from the “morphed-hexagon” stimulus set [Calder et al., 1996]. The latter stimuli were included as they are considered to depict more life-like expressions due to their relatively ambivalent nature and hence may be more sensitive for detecting any differences in explicit expression perception [Adolphs and Tranel, 2004].
We used 10 identities, posing six basic prototypical expressions (i.e., fear, surprise, happy, angry, disgust, and sad), taken from the Ekman and Friesen set [Ekman and Friesen, 1976]. Faces were cropped to exclude outline features and were presented on a gray background. Of these, six identities depicting the prototypical expressions were used to assess the perception of prototypical exemplars. Morphed images of the remaining four identities were used for the morphed-hexagon set. The morphed images were adapted from Calder et al., [ 1996]. There were five levels of morphing in steps of 20%. Pairing of expressions was based on a confusion matrix obtained from healthy volunteers [Calder et al., 1996]. The morphing between expressions followed the order: fear–surprise–happy–anger–disgust–sad–fear. Each identity was morphed within itself. A total of 120 morphs were used (4 identities × 6 expressions × 5 morph levels). Note that ambiguity arises because each morphed face conveys a mixture of expressions (see Fig. 1A for examples).
Each participant categorized the prototypical exemplars first: the six identities posing the six prototypical expressions. Each stimulus was presented once in random order with no time limit for a response. The written names of the six expressions corresponding to numbers (1–6) were presented below the face image. Assignment of names to numbers was random across participants. Observers were instructed to press the corresponding number (on a keyboard) for the expression that best described the image, as accurately and quickly as possible. No feedback was provided. Testing with the morphed-hexagon set followed the same procedure and was run subsequently. Each morph was presented twice in two separate sessions, with a short break in between (total of 240 morphs, 8 for each of the 30 levels).
Our analysis for the prototypical exemplars scored correct and incorrect responses to assess fear recognition. We used two approaches to analyze responses to the morphed expressions. The first analysis examined responses to morphs that included a degree of fear expression. A three-way mixed ANOVA was used to compute effect of group and morphing parameters on the fear responses. In the second analysis, we computed the participants' “bias” in perceiving fearful expressions. The morphed nature of these stimuli provides ambiguous expressions, allowing us to determine the tendency to read fear in the morphed expressions, rather than scoring the responses as correct/incorrect. This “fear bias” was estimated by computing the relative number of fear responses to all the morphed expressions. We concentrate on this measure rather than on absolute correct responses, as previous work indicates that amygdala damage usually results in positive judgment biases for the character and expressions of unfamiliar individuals [Adolphs et al., 1998]. Such a positive bias effect has also been observed in monkeys after amygdala damage [Amaral, 2003]. A one-way ANOVA was used to compare fear responses between the groups.
Stimuli and experimental procedure
Photographs of 10 individuals with fearful or neutral expressions, taken from the Ekman series [Ekman and Friesen, 1976], plus 20 photos of houses were used as stimuli. Houses were cropped to have an elliptical frame, while faces were cropped to exclude outline features within a similar frame. Stimuli were presented on a black background (Supporting Information Fig. S1A). A factorial design with two levels of expression (neutral or fearful) and two face orientations (upright or inverted), plus an additional condition of upright houses, was employed. The stimuli were presented in random order, each for 300 ms, with an interstimulus interval of 2,200 ms. Participants' task was to detect any immediate repetition of a stimulus (one-back task), and such repetitions arose with a ∼15% probability regardless of stimulus type.
The EEG experiment was divided into eight sessions. In each session, all stimuli were presented once (60 stimuli + ∼10 target repetitions). To keep the participant motivated and alert, they received feedback on their performance at the end of each epoch (i.e. accuracy and reaction times for detecting occasional immediate repetitions). The experiment started with a practice session, which was structured as an epoch and included presentation of all stimuli. This protocol enabled us to familiarize subjects with the set of stimuli used in the experiment and to minimize any novelty effects for the ERP signal. Observers were instructed to maintain fixation throughout the experiment, to avoid head movements and blinks, and to respond as quickly and accurately as possible. Stimuli were presented using Cogent1.24 (www.vislab.ucl.ac.uk/Cogent). Prior to the experiment, a photometer was used to measure the exact timing of each stimulus onset on the computer monitor, to ensure optimal synchronization of the stimulus onsets with the EEG trigger recording that was generated by the experimental computer.
Recordings were made with Ag–AgCl electrodes using the NeuroScan system and the 10–20 montage system. Twenty-three sites were recorded: Fpz, F7, F3, Fz, F4, F8, FC5, FC6, P7, C3, Cz, C4, P8, CP5, CP6, T5, P3, Pz, P4, T6, Ol, Or, and Oz, with linked-earlobe reference (see Supporting Information Fig. S2B). Horizontal EOG (HEOG) was recorded bipolarly from the outer canthi of both eyes. Electrode impedance was kept <5 kΩ. Amplifier bandpass was 0.1–40 Hz. EEG and HEOG were sampled with a digitization rate of 200 Hz.
The ERP analysis was performed using SPM5 (Wellcome Trust Centre for Neuroimaging, UCL, London) and Matlab7.1. The advantage of using SPM5 is that it provides a means to correct for familywise errors (FEW) based on the random field theory [Penny et al., 2003], while using established statistical methods to test for common and dissociated effects. It thus allows a comprehensive analysis method not restricted solely to specific electrodes or components of prior interest. Data preprocessing included epoching the data from −100 ms prestimulus onset to +600 ms poststimulus onset. We defined six event types: 2 facial expressions × 2 face orientations, houses, and immediate-repetition (target) trials. ERPs for each event were computed relative to the prestimulus baseline (−100 to 0 ms). An absolute threshold for artifact-removal was set to 70 μV to exclude events involving eye blinks, lateral or vertical eye movements, and any other artifacts causing distortions in the EEG. There were no significant differences between the groups in the quality of the EEG signal (see Supporting Information Table S1), and none of the patients had interictal discharges during the recording.
Importantly, all follow-up statistical analyses were performed as interactions of the within-subject and between-subject (group) factors. This approach entails that any nonspecific impact of group per se on the ERP signal, which might be due to abnormal brain structures, cannot by itself explain the more specific results we describe below. Our group comparisons did not assume equal variance.
SPM-ERP statistical analyses were initially implemented on 2D maps generated by a spatial linear interpolation of each ERP at each time point [Kiebel and Friston, 2004a, b]. We first aimed to replicate the known ERP face effects in all three groups, by contrasting faces versus houses at two time windows: 100–150 ms, i.e., covering the P1 time window [Liu et al., 2002], and 150–200 ms, i.e., encompassing the N170 [Bentin et al., 1996; George et al., 1996]. The main analysis then focused on expression effects between 100 and 600 ms poststimulus onsets using the following successive time windows: 100–150, 150–200, 200–250, 250–300, 300–400, 400–500, and 500–600 ms. To allow inferences at the population level, a second-level random-effects analysis was performed, where subjects were treated as random variables and the independent variable was the averaged differential effect size (the contrast image) for that window.
At each time window, we first computed an F test of the three-way interaction of group-by-facial-expression-by-orientation across the 2D interpolated images. When this interaction was significant, we next performed a separate ANOVA for each face orientation condition. Importantly, an effect of amygdala pathology on the ERPs was considered only if the MTLE-amygdala patients significantly (uncorrected P < 0.01 = FWE P < 0.08) differed from both the control groups (healthy and MTLE-control), thus allowing a replication of the effect, while also controlling for epileptic condition. Furthermore, to ascertain a causal influence of amygdala pathology on emotional responses at specific time windows, we also computed parametric statistics where the amygdala pathology was characterized quantitatively (based on the separately measured structural T2 signal intensity) rather than categorically. The prediction was that if the amygdala is directly involved in modulating ERPs to emotional faces, then more severe amygdala pathology would be associated with reduced emotional effects on the ERP.
In addition to test for any effects of facial expressions and orientation that were in common for all the three groups, we also computed a conjunction analysis (with intermediate null hypothesis [Friston et al., 2005]) for the two-way interaction of expression-by-orientation. This ensured that the reported common effects were evident in each group separately, with a strength that was larger than a minimal T value (minT, see “Results”). When significant conjoint interactions were observed, we computed separate statistical tests for each face orientation. Effects of expressions that were independent of epileptic condition or amygdala pathology were considered as common effects only if all three groups displayed this effect separately (at uncorrected P < 0.05).
To further verify our critical results, we also performed a complementary statistical analysis independent of the SPM approach (i.e. no longer on the 2D interpolated maps). High resolution time-bin analysis (5 ms) was performed to give a more fine-grained temporal characterization, at each electrode separately. Two-sample (not assuming equal variance) and one-sample t-tests were used to compute the reliability of differential fear versus neutral effects between and across groups, respectively, for successive time-bins. A threshold of P < 0.05 and a temporal extent of 20 ms were used for inferences and applied to the resulting statistical “maps” of ERP effects across the narrow time-bins.
Explicit fear perception
There were no significant differences between groups for correct explicit recognition of the prototypical fearful expressions (all P > 0.5). Proportion of accurate responses (mean ± SD) was as follows: healthy = 0.61 ± 0.23; MTLE-control = 0.61 ± 0.28; MTLE-amygdala = 0.52 ± 0.37. Mixed ANOVA was used to analyze participants' responses to the morphed stimuli that included some degree of fear expressions (see Fig. 1). The design included group as a between factor (e.g. healthy, MTLE-control, MTLE-amygdala) and two within factors: the expression that was morphed with fear (sad, or surprise) and percentage of fear in the morphed expression (i.e. 10, 30, 50, 70, or 90%). Neither amygdala damage nor epileptic condition affected the responses to the morphed fear expressions (all P > 0.1). More detailed behavioral results are reported in the Supporting Information Results.
Likewise, the fear-bias measure from the morphed-hexagon stimulus set indicated no significant differences between the three groups, i.e. healthy, MTLE-amygdala, MTLE-control (all P > 0.15). Out of a total of 240 ambiguous expressions, healthy participants categorized 16.7% ± 5 (SD) as fearful; for MTLE-control patients, this was 12.4% ± 6; and for MTLE-amygdala patients, 17.7% ± 5.8; see Figure 1B for individual performance.
One back task
In the one-back task performed during ERP recording (see “Procedures and Methods”), patients and healthy participants' responses did not differ (all Ps > 0.1; see Supporting Information for details and Supporting Information Fig. S1). Reaction times (RTs) were not affected by any of the experimental conditions, neither stimuli type nor group (all Ps > 0.1). There were also no differences in accuracy between faces and houses. Overall, all groups were more accurate in detecting repetitions of upright than inverted faces (F1,26 = 9.97, P < 0.01) and repetition of fearful than neutral faces (F1,26 = 5.1, P < 0.05), but these two factors did not interact (P > 0.1). The lack of RT differences between conditions and also of any main effects or interactions involving the group factor, even for accuracy, suggests that the ERP differences reported below could not have been confounded by task or attentional demands. We note also that our ERP analyses did not include the repeated trials, and so error-rate for those on target repetition trials could not affect the results for the nontarget trials' analyses.
Category-selective responses, faces versus houses
We first assessed general visual processing in the three participant groups. House stimuli were included as a control visual category to probe the processing of neutral nonsocial stimuli via ERPs, as compared with faces. Differential evoked responses to faces versus houses did not differ between the three groups (Supporting Information Results and Supporting Information Fig. S2). All three groups showed the expected N170 effect for faces compared with houses that peaked at electrode P8 (all Ps < 0.05), and also the delayed response to inverted faces compared with upright faces at this time window (all Ps < 0.05). Furthermore, healthy subjects and MTLE-amygdala patients also showed an earlier face versus house effect within the usual P1 time window (P < 0.05; Supporting Information Results and Supporting Information Fig. S2). There was no significant interaction of the P1 face effect with group at this time window (P > 0.1). These ERP results indicate that in our experiment neither amygdala pathology nor medical condition (i.e. presence of temporal lobe epilepsy) affected face-specific processing, as measured by face-versus-house or upright-versus-inverted-face ERP differences. Instead, significant effects of amygdala pathology upon ERPs, as reported below, were specific to the comparison of facial expressions.
Emotion effects: Fearful versus neutral faces in the early P1 time window
Most critically, at the time window of 100–150 ms, corresponding to the P1 component, we observed a significant three-way interaction of group-by-expression-by-face orientation, peaking at electrode Oz (Z = 2.32, P = 0.01, see Supporting Information Fig. S3A). Accordingly, we next analyzed ERPs for each face orientation separately. There were no significant expression and group effects for inverted faces (all Ps > 0.1). This indicates that the P1 expression effects reported below cannot be attributed to any low-level feature differences between expressions that would be shared between upright and inverted faces (see also Eimer and Holmes ). In contrast, for upright faces a significant interaction of group-by-expression was observed, involving lateral posterior electrodes on both sides, with a peak in vicinity to electrode Or (Z = 2.11, P = 0.018). This interaction reflected differences in the fear effect for MTLE-amygdala patients versus healthy subjects (peaking in vicinity to Or, Z = 2.6, P = 0.003; see Fig. 2), and importantly also for MTLE-amygdala versus MTLE-control patients (peaking in vicinity to P7, Z = 2.37, P = 0.009; see Fig. 2).
For the interaction described above, at this early time window, both the healthy group and the MTLE-control patients showed a significantly greater positivity for fearful than neutral faces. This emotional positivity enhancement was maximal over posterior electrodes, with a distributed scalp effect (Fig. 2A,B). For the healthy group, such an expression-effect on P1 was observed bilaterally at several posterior electrodes, maximal at Ol and Or (Z = 1.91, P = 0.028, see Fig. 2D); while for the MTLE-control patients, this effect predominated over the left-side and peaked at the nearby electrode T7 (Z = 2.34, P = 0.01). Direct comparison showed that these latter two groups did not differ significantly in the 100–150 ms time window.
By contrast, for the MTLE-amygdala patients, no electrode showed any reliable increased positivity to fearful versus neutral expressions, over the whole scalp, during the same time window (all Ps > 0.1). Thus, the increased positivity triggered by fearful expression in the 100–150 ms time window, which was found in both the healthy and MTLE-controls participants, was eliminated by pathology in the MTLE-amygdala group. A complementary analysis based on successive 5 ms time windows for each electrode separately (see Fig. 3A) revealed that a significant differential (fear-related) response between groups arose at occipital electrodes (e.g. Oz), starting at 120 ms poststimulus onset.
If the amygdala is indeed critical for modulating the early (100–150 ms, P1) visual processing of fearful faces, then in addition to the group effect described above, the actual severity of amygdala pathology in individuals should predict the degree of ERP attenuation of the fear effect in this time window. Severity of amygdala pathology was defined here based on the separately measured structural T2 relaxation time (see “Procedures and Methods”). We tested for any relation of the MR T2 signal with the magnitude of P1 modulation by emotion expression, using the same parametric approach used in a previous fMRI study of temporal sclerosis patients [Vuilleumier et al., 2004], but now applied to ERP data instead. As predicted, we found significant negative correlations of both the left and right amygdala T2 signal with the expression effect on ERPs (upright fearful minus neutral), which peaked in vicinity to electrode P7 (for left amygdala structural T2 signal: Z = 2.2, P = 0 014; for right amygdala structural T2 signal, Z = 1.54, P = 0.061). There were no significant differences between right and left amygdala effects (all Ps > 0.2). These correlations demonstrate that the worse the amygdala pathology, the greater the attenuation of the early P1 modulation by fearful versus neutral expressions in ERPs (Fig. 2E).
We note that these correlations were observed even for the right amygdala that showed only subclinical pathology according to MR T2 signal. This P1 correlation with the right amygdala seemed to be driven to a large extent by the single patient that had, radiologically, significant right amygdala damage, as removing this patient from the analysis reduced the extent of the correlation (r = −0.332, Z = 1.2, P = 0.13).
Emotion effects: Fearful versus neutral faces in the midlatency N1 and N2 time windows
In the two time windows that followed, 150–200 and 200–250 ms, there was no significant three-way interaction of group-by-expression-by-orientation (P > 0.1). Instead, the conjunction analysis revealed that all three groups showed in common an interaction between expression and orientation (Supporting Information Fig. S3B). These interaction effects across group overlapped in vicinity to electrodes FC5, FC6, and P4 (Z > 2.21, minT6 > 1.22). Testing each face orientation separately revealed that none of the groups showed a significant expression effects with inverted faces (all Ps > 0.1). However, all three groups showed an expression effect for upright faces that was primarily observed over the central frontal electrodes. The maximal effects common for the three groups arose in vicinity to electrode FC5 at 150–200 ms (Z = 2.64, P = 0.004) and at 200–250 ms (Z = 2.38, P = 0.009; see Fig. 4), with no significant group-by-expression interactions (all P > 0.1).
When considering each group separately for the 150–200 ms interval, the maximal effect was at electrode Cz in the healthy (Z = 2.77, P = 0.003), at T7 in MTLE-control patients (Z = 2.26, P = 0.012), and at F8 in MTLE-amygdala patients (Z = 2.16, P = 0.015), although similar effects were seen at the neighboring electrodes and overlapped between groups. Likewise, for the 200–250 ms interval, fearful minus neutral effects peaked at electrode F8 for healthy (Z = 3.66, P < 0.001), FC6 for MTLE-control in vicinity (Z = 2.2, P = 0.012), and F7 for MTLE-amygdala in vicinity (Z = 2.05, P = 0.02).
Unlike the earlier time window of 100–150 ms, the level of amygdala structural abnormality did not correlate with the fear effects for these two intervals, neither at 150–200 nor at 200–250 ms (all P > 0.1). The preserved fear effects during these time windows in both patient groups were further confirmed by testing successive 5 ms time windows for each electrode separately (Fig. 3B).
Emotion effects: Fearful versus neutral faces for later ERP components
At 250–300 ms, a three-way interaction of group-by-expression-by-orientation was observed, now in vicinity to electrode T7 (Z = 1.94, P = 0.025; see Supporting Information Fig. S3A). However, this interaction reflected epileptic condition rather than amygdala pathology in particular. For this time window, the two MTLE groups differed from the healthy group in their brain responses to facial expressions and orientations. As this was not clearly related to amygdala function in particular and so falls beyond the focus of the current research, we did not pursue further our analysis for this time window. We note also that the finer time-bin analysis (Fig. 3A) suggested only weak and marginal amygdala effects around 280 ms poststimulus onset, involving Or and P4 electrodes only. As these effects were marginal and did not produce a critical three-way-interaction nor a group-by-expression interaction when tested for upright faces only (all Ps > 0.1), we did not consider them further.
The next significant ERP effects were observed at 300–400 ms (Supporting Information Fig. S3B). Here all three groups showed a significant expression-by-orientation interaction, with a conjoint maxima peak in vicinity to F7 (Z = 2.68, P = 0.004, minT6 = 1.6). Further analysis revealed a significant conjoint effect of expression for upright faces, peaking in vicinity to F8 (Z = 1.92, P = 0.027) and F7 (Z = 1.67, P = 0.048), but no conjoint effect of expression with inverted faces (P > 0.05). All groups showed increased positivity for fearful compared to neutral expressions in upright faces, maximal in vicinity to F7 in healthy (Z = 2.51, P = 0.006), to P4 in MTLE-control (Z = 1.36, P = 0.087), and to F8 in MTLE-amygdala (Z = 1.42, P = 0.078) groups. We note that this effect was most reliable in the healthy participants, indicating that for this time window, epileptic condition rather than specific amygdala pathology might exert some (marginal) influence.
Finally, three-way interactions (group-by-expression-by-orientation) were also observed for the 500–600 ms time window, in vicinity to P8 (Z = 1.61, P = 0.05, see Supporting Information Fig. S3A). Separate analysis revealed, for upright faces only, a significant group-by-expression interaction in vicinity to P8 (Z = 2.54, P = 0.006). The MTLE-amygdala differed from the healthy at central posterior electrodes with a maxima at P8 (Z = 2.71, P = 0.003) and from the MTLE-control with a maxima also at P8 (Z = 3.11, P = 0.001 see Fig. 5). The two groups with intact-amygdala did not differ significantly (P > 0.1). During this late time interval, healthy and MTLE-control patients, but not MTLE-amygdala (P > 0.1), showed an increased positivity for fearful versus neutral upright expressions at several posterior central electrodes (healthy peaking in vicinity to Ol, Z = 2.8, P = 0.003; MTLE-controls peaking in vicinity to P4, Z = 2.48, P = 0.007). This outcome was further confirmed by our finer-grained temporal analysis using successive 5 ms time windows for each electrode separately (Fig. 3A), which confirmed a clear amygdala-dependent difference around 500–600 ms.
Moreover, the extent of separately measured, structural amygdala pathology (measured as the T2 MR signal) again correlated negatively with the size of the fear effect for the 500–600 ms window, with more severe amygdala damage leading to more pronounced attenuation of fear-related positivity. The conjoint effect of left and right amygdala damage peaked in vicinity to P8 (Z = 2.47, P = 0.007, minT16 = 1.46). The right amygdala parametric modulation of the upright fear > neutral responses during this time window peaked in vicinity to P4 (Z = 2.62, P = 0.004), while the parametric modulation due to left amygdala pathology peaked in vicinity to P8 (Z = 1.75, P = 0.04). In this late time window, frontal electrodes also showed some effect of right (but not left) amygdala pathology, with peak differences in vicinity to electrode F7 (Z = 2.45, P = 0.007). This differential effect of right amygdala pathology is intriguing given the subclinical pathology of MR signal for the right amygdala in all MTLE participants but one. However, similar to the effects for the earlier P1 effect (Fig. 2E), the present correlation with the right amygdala seemed to be substantially driven by this single patient with a radiologically evident right amygdala lesion (Fig. 5C), as removing this patient from the analysis reduced the extent of the correlation (r = −0.34, Z = 1.18, P = 0.12) for the late 500–600 ms time window.
In this late time window, an interaction between expression and orientation was also found to produce a bilateral conjoint effects across all groups, peaking in vicinity to F7 (Z = 2.16, P = 0.015; see Supporting Information Fig. S3B). Intriguingly, this conjoined frontal effect originated from an expression effect observed for inverted faces in particular. Here, all three groups showed increased negativity for inverted fearful compared to inverted neutral faces (Z = 1.77, P < 0.038), but not for upright faces (P > 0.1). This expression effect may reflect different visual processing demands for inverted faces, but for present purposes the main point is that this was not affected by amygdala pathology.
In this study, we measured EEG responses to threat stimuli in three groups (healthy controls, temporal lobe epilepsy patients with intact amygdale and with damaged amygdala). We show that the human amygdala has a significant role in modulating brain responses to threat-related facial expressions at specific, early and late, time points. Critically, we show that the impact of amygdala damage expresses itself at two specific time windows poststimulus onset: 100–150 and 500–600 ms. In addition, we observe differential responses to fearful versus neutral expressions in upright faces during an intervening time window (150–250 ms) that remain intact despite amygdala damage. We propose that these latter findings may relate to the fact that explicit perception of fearful expressions was preserved in our MTLE-amygdala participants, as shown by their behavioral responses. By contrast, both early and late ERP responses to fearful faces were selectively disrupted by amygdala damage, but preserved in MTLE patients with intact amygdala, as well as for healthy controls. Taken together, the results indicate multiple processing stages or routes for fearful face stimuli, which vary in their timing and their dependence on normal amygdala function. Our data reveal for the first time that amygdala pathology disrupts an early and a late cortical processing stage for fearful faces while sparing an intermediate stage of emotion-related processing.
The earliest ERP effect specific to upright fearful versus neutral faces here was observed in healthy and MTLE-control groups within ∼100–150 ms, emerging as a “fear” enhanced positivity at ∼120 ms after stimulus onset, over posterior electrodes. By contrast, within this same time window, an abnormal ERP response in the MTLE-amygdala group was observed, corresponding to a loss of this early fear-induced positivity (Figs. 2 and 3A). Moreover, the attenuation of this effect was larger in patients with more severe amygdala structural pathology, suggesting a direct causal role for the amygdala. This early ERP expression effect corresponds with the well-established latency and spatial distribution of the P1 component, which is thought to arise from extrastriate visual cortex [Di Russo et al., 2002, 2003]. Therefore, these results provide new evidence that processing of fearful faces in visual cortex is susceptible to a rapid modulation by the amygdala. This supports previous suggestions that the amygdala provides rapid and “reflexive” feedback to cortical processing, to promote an effective response to potential threat [Amaral, 2002; Dolan, 2002; LeDoux, 1996; Phelps and LeDoux, 2005; Vuilleumier, 2005].
A second abnormal ERP fearful response in the MTLE-amygdala group was expressed in a much later time window, 500–600 ms poststimulus onset (i.e., corresponding with the late-P3, Figs. 3A and 5). Emotion-related ERP effects in this time-range have been associated with possible arousal responses [Kissler et al., 2006], while others have linked them to enhanced episodic memory-related processing for emotional stimuli [Maratos et al., 2000]. This latter interpretation might appear potentially at odds with the behavioral results reported here, e.g., no effect of amygdala pathology on the one-back task. However, we note that the late-P3 ERP emotional memory effect has previously been discussed in the context of long-term episodic memory processing [Maratos et al., 2000] rather than in the context of short-term working memory as required by the one-back task used here. In support of the proposal that the amygdala may be involved with episodic emotional memory processing, it has been shown that the severity of structural amygdala abnormality in MTLE patients predicts mnemonic impairments related to emotional stimuli [Richardson et al., 2004], i.e., a loss of the usual long-term memory advantage for emotional relative to neutral stimuli. A wide literature suggests a crucial involvement of amygdala in emotional memory processes [Dolan, 2002; LaBar et al., 1995]. Our new ERP results in the 500–600 ms (“late P3”) time window may relate to this.
Taken together, our early (P1) and late (P3) amygdala-damage effects on expression-related ERPs indicates that the amygdala directly affects both perceptual–attentional and mnemonic processing. P1 modulations have often been reported to relate to attention-related enhancements of processing in visual cortex [Di Russo et al., 2003; Holmes et al., 2003], and such visual effects might account not only for better detection of emotional stimuli in attention tasks [e.g., Phelps and LeDoux, 2005; Vuilleumier, 2005] but also potentially contribute to strengthening subsequent memory traces for emotional material [Adolphs et al., 2005; Buchanan et al., 2006; Talmi et al., 2007]. Our findings that amygdala damage also affected a later distinct ERP component (late P3), associated with memory, provides a novel line of evidence for amygdala influences on learning and memory formation, on top of any earlier attentional effect.
In contrast to the abnormal early and late ERP responses, the MTLE-amygdala patients demonstrated normal explicit perception of fearful expressions. This was observed for prototypical Ekman expressions [Ekman and Friesen, 1976], and also when using a more sensitive measure with the morphed-hexagon expressions [Calder et al., 1996]. These findings suggest that normal functioning of the amygdala is not a prerequisite for explicit fear recognition, consistent with some other reports [Adolphs et al., 1995; Graham et al., 2006; Rapcsak et al., 2000]. Furthermore, amygdala pathology did not affect performance in the one-back task performed during EEG recording that is presumably based on short-term or working memory. All three groups showed more accurate responses to fearful compared with neutral expressions in the one-back task, and this was unaffected by amygdala damage.
Likewise, we also observed preservation of fearful expression effects on ERPs, specifically during the 150–250 ms (and to a lesser degree the 300–400 ms) time windows that intervened between the early and late abnormalities. A fear effect during this time window was observed for the MTLE-amygdala participants, just as for the healthy group and the MTLE-controls (Figs. 3B and 4). An enhanced positivity in response to fearful as compared to neutral faces has been found between 130 and 200 ms after stimulus onset in several previous ERP studies (see Eimer and Holmes , for a review). This effect is assumed to be linked to the explicit detection of fearful faces and typically shows a fronto-central scalp distribution, similar to what was observed in the present experiment during the 150–250 ms time interval for all three groups.
In keeping with the explicit behavioral judgments and performance of the one-back task, these aspects of the ERP data indicate that not all fear-related processes were disrupted in MTLE-amygdala patients. We surmise that the preserved ERP and behavioral effects might be linked, pointing to a processing stage at around 150–250 ms poststimulus onset mediating both explicit fear perception and emotional effects on working memory. This hypothesis accords with previous ERP data in healthy participants that tentatively linked fearful ERP effects at this time window to explicit processing of expression [Mikhailova and Davydov, 1999], and showed that these ERP responses to expressions depend on attention to the stimulus [Eimer and Holmes, 2007]. In any case, the preserved fear effect for this time-period in MTLE-amygdala subjects certainly demonstrates that not all processing of fearful faces depends on contributions from the amygdala, suggesting instead that there are multiple pathways for processing emotional information [Amaral, 2002; LeDoux, 1996; Vuilleumier, 2005], each involving differential neural structures and importantly operating with a specific time course.
Although our ERP data revealed that amygdala pathology affected fear processing in two distinct time windows, no deficits were evident in the particular behavioral measures used in this study (explicit recognition of expressions, or performances in a one-back memory task). This raises the question of what is the possible functional significance of these ERP effects. We speculate that the early ERP expression effect (around the P1) might constitute a marker for a signal that acts to direct processing resources toward potential threat in the environment, in accord with previous imaging data on P1 responses to emotional stimuli [e.g., Pourtois et al., 2004, 2005b]. This neural marker could possibly also prepare the autonomic response system to react to potential threat in the environment [Critchley et al., 2005; Williams et al., 2004]. We further suggest that this signal might facilitate the explicit perception of expression but is not necessary for it. Therefore, we predict that amygdala damaged patients should show impaired autonomic responses to threat stimuli and reduced attention-capture by threat-related stimuli (see Anderson and Phelps [ 2001]). The late ERP effect (500–600 ms) may relate to the facilitation of emotional influences on encoding into long-term memory. Thus, the same amygdala patients should show a loss of the usual enhancement in long-term memory for emotional items (see Adolphs et al. [ 2000] and Richardson et al. ). Further research is needed to establish the exact behavioral significance of the different effects of amygdala damage on ERPs to various classes of emotional stimuli, in different perceptual and memory task conditions. But for present purposes, the present ERP results clearly establishes an impact of amygdala damage on the brain response to fearful versus neutral faces at distinct time windows, while showing other brain responses remain intact at intervening time windows.
Finally, in accordance with previous literature [Rotshtein et al., 2001], our data suggest that during the first 600 ms poststimulus onset, the amygdala was involved only in processing fearful expressions for upright but not inverted faces. Effects of fearful expression with inverted faces were observed at 500–600 ms, but crucially those were not affected by amygdala pathology. This particular late effect may correspond to a delay in expression recognition that has been reported as typical for inverted faces [McKelvie, 1995]. Further research is needed to gain a better understanding of the fear effect in inverted faces. In addition, it is interesting to note that medial temporal lobe pathology, irrespective of amygdala involvement or sparing, produced some distinct effects on expression processing at 250–300 ms. This dissociation between general MTLE effects, versus amygdala-specific effects, emphasizes the necessity of including a control group that shares etiology and medical history with the neurological group of main interest (MTLE-control vs. MTLE-amygdala, respectively), as done here, to better isolate the specific relationship between brain structures and function.
In conclusion, our ERP data show for the first time that the amygdala makes distinctive contributions to processing of fearful faces in at least two distinct time windows, both early (100–150 ms) and late (500–600 ms). We demonstrate a reduction of these effects due to amygdala damage, in direct parametric proportion with the structural severity of the pathology. Conversely, we show that at an intervening time window (150–250 ms), the effects of fearful expression on brain responses were preserved despite amygdala pathology. We conclude that threat-related stimuli are processed within multiple neural stages and pathways, some dependent on the amygdala and others not, each with a particular time-course. More generally, our study illustrates how combining the lesion approach with EEG recording can uncover the specific time-points at which the damaged area normally contributes to influence information processing in distant cortical regions that survive the lesion.
We thank S. Kennett, A. Quayle, and U. Noppeney for help, plus the patients and their relatives for participating.