SEARCH

SEARCH BY CITATION

Keywords:

  • Emotion;
  • Depression;
  • Startle blink

Abstract

  1. Top of page
  2. Abstract
  3. Method
  4. Results
  5. Discussion
  6. References

Prior work suggests that major depression is associated with abnormal startle blink responses; however, only chronic or recurrent depression appears to be associated with this effect. The current study tested this hypothesis directly by examining whether recurrent major depression accounted for the anomalous startle seen in major depression using a sample of 515 female twins from the Minnesota Twin Family Study. Blink responses recorded at the age of 20 were examined in relation to number of episodes of depression prospectively assessed from ages 11 to 20. Results showed that only subjects who had experienced multiple episodes of depression showed abnormal startle responses. Subjects who had experienced just one episode of depression in their lifetime did not differ from controls. This lends additional support to the idea that recurrent depression may have a different etiological basis than nonrecurrent depression.

The startle blink reflex, the modulation of which is known to be mediated directly by the amygdala (Davis, Walker, Miles, & Grillon, 2009) has been frequently used to measure physiological reactivity in several mood and anxiety disorders (Grillon & Baas, 2003; Lang & McTeague, 2008; Vaidyanathan, Patrick, & Cuthbert, 2009). Indeed, initiatives to map out the neurobiology of psychiatric disorders such as the Research Domain Criteria (RDoC; Insel et al., 2010) sponsored by the National Institute of Mental Health (NIMH) have noted that the blink reflex is a particularly good example of such neurobiological circuitry. Relative to neutral stimuli, individuals in the general population show an attenuated blink response in the context of pleasant stimuli, and an augmented one during unpleasant stimuli (Vrana, Spence, & Lang, 1988). Thus, as the valence of the foreground image changes from pleasant to unpleasant, the blink response shows a linear increase in magnitude. Deviations from this linear pattern have been associated with abnormalities in emotional processing in various psychiatric disorders (Grillon & Baas, 2003; Lang & McTeague, 2008; Vaidyanathan et al., 2009). Major depression, in particular, appears to be associated with a lack of modulation of the blink response by picture type, or even increased response to pleasant pictures (Allen, Trinder, & Brennan, 1999; Forbes, Miller, Cohn, Fox, & Kovacs, 2005; Kaviani et al., 2004; Taylor-Clift, Morris, Rottenberg, & Kovacs, 2011). However, a close read of this literature reveals that those with mild to moderate depression (as measured by the Beck Depression Inventory; Allen et al., 1999), those with low levels of anhedonia (as measured by the Hospital Anxiety and Depression Scale; Kaviani et al., 2004), or those with relatively fewer episodes of depression since childhood (Forbes et al., 2005) evinced normal linear patterns of startle responses. In each of these studies, abnormal startle was evident in depressed groups that showed the greatest pathology. Allen et al. (1999) even found that severely depressed subjects showed an increased startle response to pleasant pictures relative to neutral pictures. Similarly, Taubitz, Robinson, and Larson (2013) reported that those with high levels of dysphoria showed attenuated startle during early phases of unpleasant picture viewing relative to nondysphoric subjects. Others have reported that individuals with comorbid depression and anxiety (Taylor-Clift et al., 2011) also showed a flat pattern of startle, compared to those with anxiety problems alone or to controls; however, work from other labs shows that this may not always be the case (e.g., Allen et al., 1999; Forbes et al., 2005).

Thus, taken as a whole, while there is general support for the idea that startle blink abnormalities are related to depression, the picture appears more nuanced. What is clear from the studies reviewed above is that mild depression is not associated with abnormal blink responses. However, given the diversity of approaches used to subtype major depression, which involve different questionnaires, number of episodes, age of onset, anhedonia, comorbidity, etc., it is unclear what particular property of depression is related to abnormal blink responses. The present study was designed to examine the possibility that abnormal startle is associated with recurrent depression. Recurrent depression is more debilitating than nonrecurrent depression and is often associated with various correlates reflecting severity, such as early age of onset and comorbidity, which in turn have been associated with startle anomalies. This focus on recurrence is further motivated by the possibility that recurrent depression may be etiologically different from nonrecurrent depression, a hypothesis that receives substantial support from research in other domains.

For example, unlike nonrecurrent depression, which occurs at greater rates in women, rates of recurrence do not differ between the two sexes (Coryell, Endicott, & Keller, 1991; Kessler, McGonagle, Swartz, Blazer, & Nelson, 1993; Kovacs, Obrosky, & Sherrill, 2003). Similarly, recurrent depression appears to be comorbid more often with dysthymia (chronically depressed mood lasting at least 2 years, but with fewer symptoms than depression) than nonrecurrent depression (Barkow et al., 2003; Warner, Weissman, Fendrich, Wickramaratne, & Moreau, 1992). Klein, Kotov, and Bufferd (2011) have noted that first-degree relatives of individuals with chronic depression have higher levels of depressive personality traits as opposed to nonchronic forms of depression. Similarly, other comprehensive studies such as those by Klein and colleagues (1999) or the Sequenced Treatment Alternatives to Relieve Depression (STAR*D; Zisook et al., 2007) have found that those who have younger ages of onset for depression tend to report greater rates of recurrence, comorbidity, and impairment. A meta-analysis by Sullivan, Neale, and Kendler (2000) found that recurrence is a good predictor of familial aggregation of depression. Reviews by other researchers (Burcusa & Iacono, 2007; Wichers, Geschwind, van Os, & Peeters, 2010) have also noted that, despite recurrence being a common phenomenon, its causes are unclear, and there is no concrete evidence that prior episodes of depression can somehow leave a scar or kindling effect such that they cause some permanent change and increase the future likelihood of additional depressive episodes. In sum, as in the startle literature, the overarching theme here is that depression with repeated episodes has different correlates in numerous domains, as compared to nonrecurrent depression.

Such findings regarding the differences between recurrent and nonrecurrent depression across a variety of domains, in conjunction with the ambiguous findings in the startle literature, logically leads to the hypothesis then that startle response abnormalities in relation to depression occur in the context of recurrent depression, and not single episode or nonrecurrent depression. The current study was undertaken to test this hypothesis directly in a large sample of female twins from the general population who were assessed longitudinally. More specifically, it was hypothesized that abnormal startle modulation among those with major depression would be accounted for by individuals with two or more episodes of depression; in other words, this group should show abnormal startle blink reactivity when compared to both those with just one lifetime episode of depression and control subjects who have never been depressed. As a corollary, controls and those with just a single lifetime episode of depression should not differ significantly from each other in terms of their startle blink responses. Other characteristics of the two depressed groups such as early age of onset and comorbid internalizing psychopathology were also examined to further understand if such variables helped differentiate them as well.

Method

  1. Top of page
  2. Abstract
  3. Method
  4. Results
  5. Discussion
  6. References

Participants

Participants for the current study consisted of female twins from the younger age cohort of the Minnesota Twin Family Study (Iacono, Carlson, Taylor, Elkins, & McGue, 1999) who completed a laboratory assessment at the time of their age 20 follow-up (N = 589). They ranged in age between 19.58 and 23.09 years (M = 20.9; SD = .55) at the time of assessment. Written informed consent was obtained from all subjects.

Stimulus Materials and Design

The experimental procedure utilized for the current study was the affective picture startle paradigm in which participants view a series of emotional (pleasant and unpleasant) and neutral pictures, while auditory startle probes are delivered intermittently during and between the viewing of these pictures. Subjects viewed a total of 27 pictures from the International Affective Picture System (IAPS; CSEA-NIMH, 1999), nine each of pleasant, neutral, and unpleasant. Three photographs from each of the three valence conditions were presented during each third of the task in pseudorandom order, with each participant viewing the same sequence of pictures.1 Each picture was displayed for 6 s, while auditory startle probes were randomly delivered between 2 to 5 s after the onset of the picture. Probes consisted of 105 decibels, 50-ms bursts of white noise, with almost instantaneous rise and fall times. They were delivered during 18 of the 27 pictures (i.e., during six trials of each picture type). Another six probes were delivered during the intertrial interval (ITI) between pictures, which varied randomly between 10 and 15 s. These ITI probes were presented between 5 and 10 s before the onset of the next slide. Hardware and software constraints that were present at the time startle data collection was initiated in the early 1990s precluded data collection during ITI trials; to maintain consistency in data collection procedures due to the longitudinal nature of the study, this practice was continued in later waves of data collection as well. All participants sat in a padded chair during the experiment, and viewed the pictures on a computer monitor at a distance of approximately 95 cm.

Physiological Measures

Startle blink responses were recorded by means of two sensors placed on the orbicularis oculi muscle, with one directly below the right eye and the other slightly lateral to it. A single computer controlled stimulus delivery and data collection, using custom software. A Grass Neurodata 12 acquisition system was used for data acquisition. All data were filtered at low and high frequency settings of 100 and 1000 Hz (6 dB down), and digitized to 12 bits resolution at 1000 Hz. Impedances for the sensors were kept below the standard 20 kΩ.

Electromyogram (EMG) data reduction was performed in MATLAB (version 7.8, Mathworks Inc.) using custom scripts. Prior to any digital processing, instances of amplifier clipping were identified and marked for deletion (2.4% of all trials). The raw EMG data were rectified and smoothed forward and backward to prevent any phase shift using a third-order low-pass filter with a cutoff frequency of 40 Hz. As in literature by Blumenthal et al. (2005), startle blink magnitude was defined as the maximum value of the signal occurring 20 to 150 ms following the probe relative to a 50-ms preprobe baseline. To control for trials contaminated with equipment or physiological artifact, trials in which the variance of the baseline (-50 ms to stimulus onset) or poststimulus (stimulus onset to 150 ms) periods exceeded three standard deviations from the sample mean were excluded (3.5% of nonclipping trials). This excluded a total of approximately 5.9% of all trials across all subjects. Subjects who had data for less than a third of their trials (i.e., less than 7 out of 18 trials), or without a valid value for any of the valence conditions (because of rejected trials) were also excluded from analyses (N = 11). In addition, 19 subjects were eliminated for a variety of reasons such as equipment failure, being intoxicated at the time of the session, and other factors that might adversely affect the data. These criteria resulted in 559 subjects with usable startle data.

The blink response for the first trial for each subject was excluded from all analyses as preliminary analyses revealed that all subjects showed a disproportionately large response to this trial relative to other trials even after dropping all bad trials, (t[518.91] = −6.65, p < .001 for trials 2 and above compared to trial 1; cf. Benning, Patrick, & Iacono, 2005). Individuals differ greatly in the magnitude of their raw blink amplitudes for reasons believed to be associated with muscular rather than neurobiological factors. Thus, a common approach used in the startle literature is to standardize blink responses within each individual by first subtracting the mean response from all trial responses and then dividing each of these by the standard deviation of all responses (i.e., z score; Bradley, Codispoti, Cuthbert, & Lang, 2001; Levenston, Patrick, Bradley, & Lang, 2000). This resulted in standardized blink magnitude scores with a mean of 0 and a standard deviation of 1 for each participant. Trial-level scores for each subject were then averaged by valence yielding an average blink response for each subject for pleasant, neutral, and unpleasant conditions.

Diagnostic Measures

Clinical assessment

Participants were interviewed for symptoms of depression and other DSM-IIIR disorders by carefully trained interviewers with a Bachelor's or Master's degree in psychology or a related field using the Structured Clinical Interview for DSM-IIIR (SCID; Spitzer, Williams, Gibbon, & First, 1992), with each twin interviewed by a different interviewer. Interviewer training included an apprenticeship period, written examination, and a proficiency test. Feedback about interview quality is provided on an ongoing basis from diagnosticians who review audiotaped interviews. Interviews were subsequently reviewed by pairs of students with advanced graduate training in clinical psychology, who reached a consensus concerning the presence or absence of each symptom. A team never included the person who had conducted the clinical interview, and each team was blind to the identity of the interviewee. In addition, a different team was assigned to different family members. Several hundred interviews were reviewed separately by independent and blind teams to assess reliability, and the kappa coefficient for major depression was .82. Major depression was scored as present if all criteria were met as specified in the DSM-IIIR, the diagnostic system in place when participants were first assessed at age 11.

Subjects were assessed for depression, as well as other DSM disorders, once every 3 years, starting at the age of 11 (i.e., 11, 14, 17, and 20). Subjects were queried about the number of depressive episodes they had experienced at the 17- (the number of lifetime episodes till then) and 20- (number of depressive episodes in the past 3 years) year-old assessments. These were then summed to get a total of the number of episodes of depression all subjects had experienced in their lifetime as a whole. Subjects were also asked about the age of onset of their first episode of depression at the 17- and 20-year-old assessments. Given the longitudinal focus of the current study, individuals with missing information regarding number of depressive episodes were eliminated from the analyses (N = 44). This led to a final sample of 515 with diagnostic and startle blink data out of which 361 individuals reported never having experienced a single episode of depression up to the age of 20 (i.e., controls), 93 reported experiencing just one episode up to that point, and 61 reported two or more episodes in their lifetime. Out of this, complete startle data was available for 140 MZ twin pairs (i.e., 280 individuals) and 38 MZ singletons, and 85 DZ twin pairs (i.e., 170 individuals) and 27 DZ singletons.

Anxiety disorders assessed included simple phobia, social phobia, panic disorder, agoraphobia without panic disorder, generalized anxiety disorder, and posttraumatic stress disorder (PTSD).2 Since the study participants were recruited from the general population, to ensure that subjects were experiencing clinical levels of anxiety, assessments were designed to focus on the identification of those with significant impairment. To parallel the time span covered by the number of episodes of depression variable, a lifetime anxiety comorbidity variable indexing the number of lifetime anxiety disorders experienced by a subject was computed (i.e., yes/no diagnosis for each of the anxiety disorders summed). Forty-seven subjects had at least one comorbid lifetime anxiety disorder. Three individuals had missing information regarding some of the anxiety diagnoses; consequently, the number of lifetime anxiety diagnoses for these persons could not be computed.

Personality assessment

Since personality traits such as negative and positive emotionality have been concurrently and prospectively associated with depression, (see Klein et al., 2011, for a review), attrition analyses using personality ratings were undertaken to ensure that subjects varying in risk for depression were not more likely to drop out of the study or have unusable data. Mothers rated the personality of both twins using the Multidimensional Personality Questionnaire (MPQ; Tellegen & Waller, 2008) at the intake data collection session when the twins were approximately 11 years old. Briefly, the MPQ is a broadband personality inventory that has three major factor scores—positive emotionality, which assesses the tendency to experience positive emotions such as well-being, optimism, and social connectedness; negative emotionality, which indexes emotions such as stress reactivity, isolation from others, and aggressive tendencies; and constraint, which measures proclivities towards impulsive, risk-taking, unconventional, and rule-breaking behaviors. Out of the final sample of 515, depending on which of the three major MPQ scales was examined, 429–435 subjects had personality ratings. Attrition analyses comparing the subjects who had been eliminated for missing diagnostic data and/or noisy startle data versus those who had been retained revealed no significant differences in parental ratings of their personality: positive emotionality: F(1,20) = .02, p = .89; negative emotionality: F(1,29) = 2.05, p = .16; constraint: F(1,28) = .97, p = .22.

Data Analysis

All analyses were undertaken as three-level linear mixed models (LMMs), consisting of both fixed and random effects, to account for the within-subject correlation structure of startle magnitudes and nonindependence of twins at the two higher levels, respectively. The general setup of the LMMs was as follows. In LMMs that included them, groups were modeled as fixed effects. A random effect of picture valence was included at the level of individual subjects in order to account for the observed within-subject correlations in blink magnitude. The covariance matrix of random effects was unstructured, corresponding to a multivariate analysis of variance (MANOVA) structure for the repeated measures. A second random effect of picture valence was included at the level of twin pairs to account for twin correlations in blink magnitude. This covariance matrix was also unstructured, allowing blink magnitudes and twin correlations within and across valence conditions to be freely estimated. LMMs were estimated using restricted maximum likelihood (REML), which corrects a small bias in maximum likelihood estimation (cf. Bryk & Raudenbush, 1992). LMMs were estimated in SAS PROC MIXED (SAS for Windows 9.3). Statistical significance was evaluated at p ≤ .05. The general structure of the equations used for LMMs was as follows, using the notation of Bryk and Raudenbush (1992) for convenience and grouping like terms:

  • display math

where i = 1,2,3 represents valence condition, j = 1,2 represents individual within pair, and k = 1, … , N represents twin pair. Yijk is startle magnitude, γ000 is the sample intercept, γ100 is the valence fixed effect (i.e., sample means), r1jk and u10k are random slopes at the individual and twin level (levels 2 and 3, respectively), and eijk is within-subject error. The individual-level random slope, r1jk, accounts for within-subject dependence (correlation) in startle magnitude across valence conditions, while the twin-level random slope, u10k, accounts for differences among pairs in the pattern of correlations.

The following LMMs were undertaken. To confirm that basic modulatory effects of emotion on startle reactivity were present in the sample as a whole, a LMM was performed in which startle blink amplitude served as the dependent variable and picture valence (pleasant, neutral, unpleasant) served as the within-subject (repeated) independent variable. Next, to test the specific hypotheses posited earlier, three pairwise LMMs were undertaken where picture valence (pleasant/neutral/unpleasant) served as the within-subject independent variable and the various depressed groups (controls vs. nonrecurrent, nonrecurrent vs. recurrent, and controls vs. recurrent) served as the between-subjects measures in the respective LMMs. As supplementary analyses, to examine whether the recurrent and nonrecurrent depressed groups differed in other respects, age of onset of the first episode of depression and lifetime anxiety comorbidity were added as covariates to the LMMs at the subject level.

Results

  1. Top of page
  2. Abstract
  3. Method
  4. Results
  5. Discussion
  6. References

When all participants were collapsed into a single group, the sample demonstrated the expected pattern of startle reactivity, with blink responses to pleasant pictures significantly less than unpleasant pictures (main effect of valence: F(2,273) = 12.95, p < .01; linear contrast: F(1,277) = 18.12, p < .001; quadratic contrast: F(1,272) = 11.79, p < .01).

Impact of recurrence

As hypothesized, the two depressed groups (recurrent vs. nonrecurrent) differed significantly from each other in terms of startle modulation patterns (main effect of picture: F(2,238) = 4.18, p < .05; interaction: F(2,34) = 4.36, p < .05). The recurrent group showed increased startle responses to both pleasant and unpleasant pictures (see Figure 1), while the nonrecurrent group evinced the expected linear pattern of responses. Similarly, those with recurrent depression also differed significantly from controls as well (main effect of picture: F(2,534) = 5.64, p < .01; interaction: F(2,34) = 3.32, p < .05). However, as predicted, the nonrecurrent group did not differ from controls with regard to startle modulation (main effect of picture: F(2,540) = 11.45, p < .0001; interaction: F(2,84) = 0.59, p > .05).

figure

Figure 1. Startle blink response magnitude as a function of depression recurrence. To ease interpretation, z-scored blink response values multiplied by 100 are depicted on the y axis.

Download figure to PowerPoint

Further post hoc analyses revealed that the group that had experienced only one episode of depression tended to have a later age of onset for their first episode of depression relative to the 2+ episodes group: F(1,17) = 22.96, p < .001; 1 episode: M (SE) = 17.66 years (.22); 2+ episodes: M (SE) = 15.68 years (.35).

Given the literature suggesting the possible influence of comorbid anxiety, an additional post hoc test was conducted by adding the number of lifetime anxiety diagnoses as a covariate to the LMM contrasting the recurrent versus nonrecurrent depressed groups. Results revealed that it had minimal impact: Main effect of picture type: F(2,238) = 4.05, p < .05; recurrent versus nonrecurrent by picture interaction: F(2,34) = 3.86, p < .05; Effect of presence/absence of lifetime internalizing disorders, F(1,5) = 2.59, p > .05.

Of interest is how the startle pattern evident in the recurrent group may vary as a function of the number of episodes. Figure 2 provides a visual depiction of the relationship between the number of episodes and startle patterns that was evident among those in the recurrent group and includes the mean startle response of all participants who did not have recurrent depression as a reference. As can be seen among those with recurrent episodes, with the possible exception of the small group with 4–5 episodes, there is a general tendency for blink responses to pleasant trials to be equal to or greater than unpleasant trials, and this is true no matter the number of lifetime episodes. Contrary to this, startle responses to pleasant trials are less than those to unpleasant trials among the nonrecurrent subjects comprising the reference group. Hence, the startle pattern observed in the recurrent group seems to be a general feature of those in this group that varies little by the actual number of episodes experienced.

figure

Figure 2. Startle blink response magnitudes in the recurrent depressed group as a function of the number of lifetime depressive episodes. For comparison, the heavy bolded line represents the mean startle response pattern of all those with nonrecurrent depression, i.e., those with no depression or with no more than a single episode combined into one group. To ease interpretation, z-scored blink response values multiplied by 100 are depicted on the y axis.

Download figure to PowerPoint

Discussion

  1. Top of page
  2. Abstract
  3. Method
  4. Results
  5. Discussion
  6. References

The current study examined the association between recurrent and nonrecurrent depression and startle blink reflex reactivity in a large sample of female twins from the general population. Findings revealed abnormal startle responses in relation to major depression, but only among subjects who experienced recurrent (i.e., two or more) episodes. While prior studies have shown a mix of different types of abnormal responses in depression, with some suggesting a flat startle pattern or lack of modulation between picture types (e.g., Kaviani et al., 2004; Taylor-Clift et al., 2011), others have noted an increased response to pleasant pictures relative to neutral in severely depressed subjects (e.g., Allen et al., 1999). The results presented in the current study were more like the latter, suggesting perhaps that subjects with multiple episodes of depression were reacting or potentiating to pleasant pictures almost as if they were unpleasant pictures. Comorbid anxiety diagnoses did not affect the results presented here, though comorbid anxiety symptoms have not always consistently demonstrated a relation to startle reactivity (e.g., Allen et al., 1999; Forbes et al., 2005).3

Thus, overall, these findings are in line with prior startle studies on depression and studies investigating other characteristics of recurrent depression in the larger literature—all of which suggest that this form of depression may have a unique etiological basis as compared to nonrecurrent depression. Indeed, other researchers (Parker et al., 2010) have argued precisely this point and noted that melancholia, typically characterized by recurrent episodes, should be classified as a separate disorder in DSM-5. Should this be the case, it would have significant implications for refining classification systems that are currently in place, which treat major depression as a unitary disorder. It is worth noting, however, that the fact that individuals who have experienced just one lifetime episode of depression do not show abnormal startle does seem to be evidence against the scar theory, it is also possible that the scar effect may appear only after the experience of recurrence (i.e., scarring requires at least two episodes).

Future research will have to focus more on distinguishing the etiology of recurrent from nonrecurrent depression. It is likely that some of those in the nonrecurrent depressed group, and even some of those in the control group, will develop recurrent depression as they further pass through the age of risk. However, despite the possible misassignment of these individuals to other than the recurrent group, the recurrent group was nevertheless found to differ from the other two. A second limitation is the fact that the current study utilized only females; future studies will have to evaluate whether these findings generalize to males.

Despite these shortcomings, the strengths of the current study include its large population-representative sample with neurobiological data and its longitudinal nature. As noted by the DSM-5 architects (Andrews et al., 2009), integrating results of this sort across methodologies and longitudinally will be key to defining and understanding the structure of mental disorders in general. Given its worldwide prevalence and its impact on occupational and psychosocial functioning (Bromet et al., 2011; Kessler, 2012; Murray & Lopez, 1996), major depression does indeed deserve more study.

Footnotes
  1. 1

    All female subjects viewed the following IAPS slides in the following order: 5500, 8370, 1390, 3000, 8210, 2800, 8080, 7090, 3140, 8490, 7060, 2410, 3010, 7050, 8501, 3120, 9140, 4510, 3170, 6150, 8200, 4660, 7500, 3220, 3180, 8030, 7130. Due to subject relatedness and the longitudinal nature of the study, to maintain consistency within families (within twin pairs and between the twins and their parents) as well as to maintain consistency over time, counterbalancing the order of the slides across successive participants and assessments was not feasible. Thus, all subjects viewed the same sequence of pictures.

  2. 2

    Obsessive-compulsive disorder (OCD) was not included as part of the Minnesota Center for Twin and Family Research assessment battery due to its relatively rare prevalence in the population at large.

  3. 3

    One of the reviewers for the current manuscript pointed out that pleasant trial amplitudes were marginally greater than neutral trials among control subjects. One plausible explanation for this effect might be due to the fact that startle responses to the first trial were excluded from all analyses, which happened to be the startle blink response to a neutral picture. Since the stimulus order across all subjects was the same, this meant that the largest blink response to neutral pictures was excluded across all subjects—a part of which was artificially large due to its being the first trial consistently, but a part of which might have contained some level of ‘true’ response as well. Doing so would thus have artificially lowered the neutral response slightly below the pleasant response, but not significantly so.

References

  1. Top of page
  2. Abstract
  3. Method
  4. Results
  5. Discussion
  6. References