Towards a valid, reliable measure of sleep effort

Authors


Dr Niall M. Broomfield, Department of Psychological Medicine, Gartnavel Royal Hospital, University of Glasgow, 1055 Great Western Road, Glasgow G12 0XH, UK. Tel.: 0141 211 3904; fax: 0141 357 4899; e-mail: nmb2v@clinmed.gla.ac.uk

Summary

A frequent clinical observation is that patients with insomnia strive to control their sleep. However, sleep is an involuntary physiological process, which cannot be placed under full voluntary control. Therefore, direct, voluntary attempts to control sleep may actually exacerbate and perpetuate insomnia. To date, no reliable scale has been available to test this hypothesis directly. Moreover, while sleep effort is a core International Classification of Sleep Disorders – Revised criterion for psychophysiological insomnia, clinicians lack a reliable measure with which to assess the construct. In this initial scale validation study, we present psychometric data for the Glasgow Sleep Effort Scale based on a relatively small but representative sample of patients with insomnia and good sleepers. The clinical and research value of the new scale is discussed and future research directions are described.

Introduction

Sleep is a fundamental life process, much like eating or drinking. Presumably therefore, inability to sleep is significantly threatening (Broomfield et al., 2005; Harvey, 2002). This may explain the frequent clinical observation that patients with insomnia often strive to control their sleep. As sleep is an involuntary physiological process, which cannot be placed under full voluntary control, any effort to control sleep is likely to fail. Effort to sleep may, therefore, represent a key perpetuating factor in insomnia.

We conceptualize the construct of sleep effort as multicomponent, comprising core behavioural and cognitive elements, and describe here an initial validation study of the Glasgow Sleep Effort Scale (GSES), a new self-report scale designed to measure effort.

This initial validation study is important and timely for several reasons. First, as already indicated, the behaviour and cognition of insomnia patients often involve heightened sleep effort. This may express itself as performance anxiety about sleep, a need for control over sleep, and/or trying too hard to sleep. Interestingly, and we think this is important, good sleepers tend towards the opposite. When asked what they do to fall asleep, good sleepers usually: (i) appear bewildered and (ii) report not consciously delivering any behaviour. This supports good sleep as passive and effortless, and implicates effort as disruptive (Espie, 2002). Reliance on clinical observation to inform psychological theory testing is of course a critical tradition across psychological research studies (Salkovskis, 2002) and insomnia research is no exception (Tang and Harvey, 2003).

Secondly, consensus expert opinion has for some time recognized sleep effort as a relevant aetiological factor. The International Classification of Sleep Disorders – Revised (ICSD-R) and its predecessor (ICSD), both cite ‘trying too hard to sleep’ as a core factor leading to learned sleep preventing associations in psychophysiological insomnia (American Sleep Disorders Association, 1990, 1997). Yet we lack the means to address, diagnostically, the presence or otherwise of effort. Several existing scales integrate relevant single items [e.g. The Dysfunctional Beliefs and Attitudes about Sleep (DBAS) scale (Morin et al., 1993); Sleep Disturbance Questionnaire (Espie et al., 1989)]. This suggests acknowledgement of the aetiological relevance of sleep effort. Yet no dedicated measure is available.

Thirdly, previous work suggests sleep effort can discriminate good and poor sleepers. Kohn and Espie (in press) compared good sleepers with primary insomniacs and depressed insomniacs, and measured sleep effort using a pilot version of the GSES, which had not at that time been subject to psychometric evaluation. Effort was significantly higher among the two poor sleeping groups and regression analyses selected GSES as the best discriminatory measure between the two insomnia groups and good sleepers. A separate, initial validation study on the psychometrics of the GSES is now needed.

Fourthly, there is evidence that paradoxical intention therapy, an empirically supported insomnia treatment (Chesson et al., 1999; Morin et al., 1999) improves sleep by minimizing sleep effort. Insomnia patients allocated to 14-night paradoxical intention therapy (relax at lights out, keeping eyes open) showed, relative to controls, reduced effort and sleep improvement (Broomfield and Espie, 2003). This indicates that sleep effort is sensitive to change following psychological therapy and may act to maintain insomnia. Again, the measure of effort used was a pilot GSES not subject to psychometric evaluation. A valid reliable effort measure would allow identification of insomnia sufferers particularly suited to this therapeutic approach.

Fifthly, experimental data from the ironic cognitive control literature (Wegner, 1994) support effort as an insomnia-maintaining factor. Good sleepers directed to fall asleep urgently, under high mental load (listening to marching music) show significantly longer sleep latencies relative to good sleepers directed to sleep urgently under low mental load (relaxing music). Although requiring replication in insomnia patients, these data suggest effort to sleep, particularly under experimental stress conditions, drives wakefulness in good sleepers (Ansfield et al., 1996).

Finally, there is an outstanding need for a valid, reliable measure of sleep effort. The present initial validation study represents an important first step in this process. We report psychometric data for the GSES based on small, but representative samples of good sleepers and individuals with insomnia. As we believe effort impacts sleep initiation both at sleep onset and at times of sleep interruption throughout the night, the sample selected here includes initial and maintenance insomnia patients.

Method

Development of the pilot GSES

A pilot GSES not subject to psychometric evaluation has been used in two previous studies (Broomfield and Espie, 2003; Kohn and Espie, 2005). This pilot GSES was originally developed following discussions involving the authors of these papers. An exhaustive content analysis of relevant existing scales was completed, which resulted in a working model of sleep effort, integrating seven core components of sleep effort (Fig. 1). Each component was assigned a single item, forming the pilot GSES (Appendix 1), which we now evaluate. All items in GSES are valenced in the same way. And instructions make reference to sleep ‘in the past week’. This sets up GSES as a present state rather than trait measure. Insomnia patients sleep well at certain times and on certain nights, a point reflected in formal diagnostic criteria (e.g. ICSD-R).

Figure 1.

A preliminary working model of sleep effort.

GSES field testing and psychometric evaluation

Participants

In order to examine GSES psychometric properties, insomnia patients and good sleepers were recruited. All participants were 18 years or older, with no upper age limit. The majority of insomnia participants were general practitioner referrals to a large treatment trial, the results of which are reported elsewhere (Espie et al., 2004). The remainder were obtained via university e-mail, as were the good sleepers.

All insomnia participants were screened using a standard in-house protocol (copy available on request) based on the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV; American Psychiatric Association, 1994) criteria for primary insomnia. This was completed at home, and considered sleep complaint (chronicity, number of poor nights sleep, latency, wake time after sleep onset), daytime functioning, and substance use and mood data. Sleep-disruptive medical conditions (e.g. chronic pain) and suspected other sleep disorders were also assessed. Follow-up telephone interviews clarified detail.

Inclusion criteria were a complaint of significant difficulty initiating and/or maintaining sleep, occurring 4+ nights per week for minimum 6 months, daytime impairment (concentration difficulty, fatigue) and a score >5 on the Pittsburgh Sleep Quality Index (PSQI) (Buysse et al., 1989). This cut-off discriminates good from poor sleepers adequately (Backhaus et al., 2002). Participants who reported taking ‘prescribed medications to assist sleep’ were included. Data on specific medication sort were not gathered. Patients with known sleep-disruptive medical conditions, suspected other sleep disorders (respiratory, muscular, neurological, circadian) such as depression, or substance abuse problems were excluded.

Good sleepers had no sleep disturbance complaint, failed to meet DSM-IV criteria for primary insomnia, agreed with the statement ‘I am a good sleeper’, described their sleep pattern in a typical month and this month as ‘good’ or ‘very good’, and scored <5 on the PSQI. These conservative criteria ensured a representative good sleeper cohort.

Measures and procedure

To examine psychometric properties, all participants completed the GSES, the Hospital Anxiety and Depression Scale (HADS; Zigmond and Snaith, 1983) and the DBAS scale (Morin et al., 1993). Scales were mailed out immediately following the telephone screening and completed at home.

The HADS-A (anxiety subscale) is a reliable, widely used measure of clinical anxiety. DBAS is a reliable, widely used measure of beliefs and attitudes regarding sleep. Inclusion of these supported concurrent validity analysis, by allowing an examination of whether GSES scores (i) merely represented elevated general anxiety and (ii) were associated with a known cognitive measure of sleep.

Results

Participants

A total of 89 insomnia patients and 102 good sleepers met criteria. Mean age of insomnia participants was 50.67 years, SD = 16.17 (56 were female). Mean PSQI score was 12.10, SD = 3.60 (range 5–18). Mean age of good sleepers was 34.08 years, SD = 15.98 (52 were female). Mean PSQI score was 1.96, SD = 1.30 (range 0–5).

Internal consistency

Cronbach's α was 0.77 using the insomnia patient group. Item-deletion alphas reveal the stability of a psychometric measure when individual items are systematically eliminated. For the GSES, item-deletion alphas remained high, with little variation between values (mean α = 0.75, range 0.74–0.76). Typically, a criterion of 0.80 is deemed acceptable (Nunnaly and Bernstein, 1994). Mean corrected item–total correlation was 0.64 (range 0.49–0.73). The item-by-item correlation matrix for insomnia patients showed a range of values (r = 0.15–0.58) (Table 1).

Table 1.  Glasgow Sleep Effort Scale (GSES) item-by-item correlation matrix
GSES item1234567
11.00      
20.441.00     
30.340.241.00    
40.380.360.511.00   
50.270.150.400.511.00  
60.370.290.520.560.421.00 
70.400.330.510.580.390.571.00

Concurrent validity

Anxiety subscale scores from the HADS (Zigmond and Snaith, 1983) were available for 68 insomnia patients. For these, the association between GSES and HADS-A was non-significant (r = 0.19, P > 0.1, NS).

DBAS scores were available for 64 insomnia patients. For these individuals, there was also a significant moderate positive correlation between GSES and DBAS (r = 0.50, P < 0.0001).

For all participants, a weak positive correlation was observed between GSES and age (r = 0.22, P < 0.005). This is a small significant correlation, accounting for approximately 4% variance.

Discriminant validity

The ability of GSES to discriminate insomnia patients (n = 89) from good sleepers (n = 102) was investigated. Mean total GSES score for insomnia patients was 7.06, SD = 3.58; and 1.22, SD = 1.35 for good sleepers.

The GSES readily discriminated the groups (t = −15.27, d.f. = 189, P < 0.0001). Mean scores for individual GSES items differed significantly between groups (Table 2). Distribution examination indicated limited overlap between highest scoring good sleepers and lowest scoring insomnia patients (Fig. 2).

Table 2.  Mean Glasgow Sleep Effort Scale (GSES) item scores for insomnia (I) and good sleeper (GS) participants
 Mean item score
GSItP-value
1. I put too much effort into sleeping when it should come naturally0.121.04−10.970.000
2. I feel I should be able to control my sleep0.561.13− 6.270.000
3. I put off going to bed at night for fear of not being able to sleep0.070.71−7.340.000
4. I worry about not sleeping if I cannot sleep0.251.17−10.070.000
5. I am no good at sleeping0.031.27−16.600.000
6. I get anxious about sleeping before I go to bed0.040.70−7.940.000
7. I worry about the consequences of not sleeping0.161.03−10.260.000
Figure 2.

Box and whiskers plot demonstrating the ability of the Glasgow Sleep Effort Scale to discriminate good sleepers and insomniacs.

Given the significant association between GSES and age, to be conservative discriminant validity between groups was considered using analysis of covariance (ancova), with age as covariate. GSES still readily discriminated the groups, with age partialled out [F(1,182) = 228.52, P < 0.0001].

Impact of medication

Sleep medication can affect perception of sleep. Data on whether insomnia patients used or did not use ‘prescribed medication to assist sleep’ were available for 70 of 89 insomnia patients (78.6%). Thirty reported non-use, 40 reported medication use. Independent t-test analysis indicated no significant differences between users and non-users on GSES score (t = −0.25, d.f. = 68, P > 0.8, NS).

Sensitivity and specificity

The sensitivity of a scale is the probability that an individual with the condition will be correctly classified as having the condition. Specificity is the probability that a person without the condition will be properly classified as not having the condition (Fleiss, 1981). A cut-off score of 2 correctly identified 93.3% of insomnia patients and 87.3% of good sleepers. Only seven of 89 insomniacs are not identified using this cut-off (Table 3a). Using 3 as the cutoff, 82.2% of insomnia patients and 92.2% of good sleepers were correctly identified. Only 16 of 89 insomniacs were not identified (Table 3b).

Table 3.  Identification of insomniacs and good sleepers
 IGS
(a) Using GSES cut-off of 2
  Correct8213
  Incorrect789
(b) Using GSES cut-off of 3
  Correct738
  Incorrect1694

Following Bossuyt et al. (2003), predictive value and likelihood ratio data were also calculated (Table 4). Positive predictive and likelihood ratios were high. This suggests GSES predicts the presence of insomnia well.

Table 4.  Diagnostic test results for the Glasgow Sleep Effort Scale score >2
  1. Range values are given within the parenthesis.

Positive predictive value0.87 (0.78–0.92)
Negative predictive value0.94 (0.87–0.97)
Likelihood ratio for insomnia+7.3 (4.39–12.20)
Likelihood ratio for good sleepers−0.08 (0.04–0.17)

Finally, using a cut-off score of 2, participants were split into two groups: ‘low effort’ (score of 0, 1 or 2), and ‘high effort’ (score 3 or more). Differences between these groups on the PSQI were then analysed. High effort participants scored higher on the PSQI (mean = 11.01, SD = 4.61; t = −15.93, d.f. = 189, P < 0.0001) relative to low effort participants (mean = 2.38, SD = 2.59).

Principal component structure of GSES

Data from 159 insomnia patients and 120 good sleepers were included in a principal component analysis (PCA) with varimax rotation. To maximize numbers, the sample used was generated from participants described here, and from two previous studies (Broomfield and Espie, 2003; Kohn and Espie, in press). One principal component was found (eigenvalue = 4.38) accounting for 62.6% of total variance. Each item loaded significantly on this factor (mean α = 0.79, range 0.64–0.85).

Discussion

In this initial validation study, GSES psychometric evaluation was completed with 89 insomnia patients and 102 good sleepers. Field testing revealed highly acceptable data. Internal consistency and item-by-item correlation data were adequate. Item–total correlations were satisfactory, indicating no single-scale item accounts for the majority of variance. A cut-off score of 2 correctly identified 92.1% of insomnia patients and 87.3% of good sleepers, with excellent likelihood ratios at this cut-off. GSES adequately discriminated insomnia patients from good sleepers, supporting previous work (Kohn and Espie, in press). Although sleep medication can impact perception of sleep, hypnotic users did not differ from non-users on GSES. The GSES HADS-A association was non-significant, suggesting the scale is not simply measuring anxiety. Factor analysis revealed one component accounting for 62% of variance and a moderate correlation was observed with DBAS suggesting substantial unique variance (75%). This sets GSES apart from existing cognitive insomnia measures. Although requiring replication with a much larger and more carefully defined sample, these data implicate GSES as a valid, reliable measure with considerable potential for quantifying sleep effort.

Once fully validated using a larger sample, GSES should serve several purposes for the sleep medicine specialist. It will identify if effort is a relevant concern in a given insomnia case. It will also provide a useful baseline cognitive screen for dysfunctional beliefs, which may then be targeted as part of cognitive restructuring. And it will denote potential patient suitability for single-component paradoxical intention therapy. This therapy as noted improves sleep by reducing effort (Broomfield and Espie, 2003), so insomnia patients scoring high on GSES may be good responders. Research is needed on this.

Following larger scale validation, general practitioners could also use GSES to identify a core symptom of psychophysiological insomnia in sleep-disturbed individuals. The scale is short, easily administered and appears to show, at least on this initial sample reasonable sensitivity, specificity and discriminant validity. Whereas we would predict patients with other types of sleep disturbance, e.g. sleep apnoea or circadian disorders would not score highly. Research examining GSES with other sleep disorder groups is needed.

Finally, the validation of GSES will allow research into how effort perpetuates insomnia. GSES appears to provide the means to measure effort reliably, so comparison between studies is possible. Further work on putative mechanisms underpinning paradoxical intention therapy is also now possible. The evidence base for paradoxical intention therapy as a single-component therapy is established (Chesson et al., 1999), but our understanding of how it promotes sleep change is poor. The answer will come from research using GSES to clarify the role of paradoxical intention therapy in unwinding inhibitory sleep effort.

Several limitations of this study merit consideration. First, sample size was relatively small for a validation study, and insomnia participants were obtained from a large treatment trial and from a university-based population. Recruited participants were neither asked to record sleep data to confirm diagnosis of primary insomnia nor was polysomnography (PSG) employed to rule out occult primary sleep disorders. Diagnosis was made on the basis of a sleep complaint present 4+ nights per week for 6 months, and a PSQI score >6. No inter-rater check on diagnosis was made, no data on insomnia severity and type (initial versus maintenance) were collected, and clinical interviews were not employed for screening. Future GSES research will benefit from such detail and from additional screening methods, e.g. structured interview methodology, single-night PSG (Edinger et al., 2004) and inter-rater reliability data. Test–retest data should also be collected.

Furthermore, to test the validity of GSES, correlation was only made to other self-report scales (HADS, DBAS). Objective sleep assessments are needed to confirm external scale validity. This could be performed on a smaller sample. Related to this, HADS, DBAS and medication data were only available from the majority of insomnia patients and data on specific hypnotic medication type were not recorded.

Also, the insomnia and good sleeper samples differed significantly on age. Unmatched samples are not unusual in insomnia scale research however (e.g. Buysse et al., 1989; Fichten et al., 1998), probably because of epidemiology factors (Mellinger et al., 1985). As sleep effort is likely to play a role in insomnia maintenance throughout the lifespan, research with age-matched younger and with age-matched older good and poor sleepers will be desirable. Crucially, while in our sample age and GSES were significantly associated, ancova demonstrated adequate GSES discriminant validity with age partialled out.

Importantly, the majority of insomnia patients (83%) were physician-referred. So the cohort is a clinical insomnia rather than university-based population. Notably also, good sleepers were not merely defined by absence of sleep complaint. They also described themselves as good sleepers, and reported their sleep as good or very good both typically, and in the last month. This exceeds research diagnostic guidelines (Edinger et al., 2004), and is a strength rarely seen in studies that use ‘good’ sleepers to compare.

Finally, GSES item derivation relied on an empirically untested model of effort. It was, however, our intention to do this. The approach of setting testable hypotheses extracted from clinical experience in order to inform theory is defendable, and has assisted our understanding and treatment of a range of psychological disorders in recent years (Salkovskis, 2002). Indeed, this should be employed more to study insomnia (Tang and Harvey, 2003). The working model we present requires empirical clarification using experimental methods, and GSES will assist with this. Carefully designed single casework is needed for tracking insomnia patients who fit the model. Related research questions include whether it is possible to induce sleep effort experimentally in good sleepers? And what is the impact of experimentally induced sleep effort on insomnia patients? This research will extend our understanding of effort in insomnia, and provide important groundwork for further study of paradoxical intention therapy and putative mechanisms.

To summarize, this validation study suggests that GSES shows good psychometric properties. It is a useful and novel measure of clinical assessment and change, which should screen out psychophysiological insomnia in sleep-disturbed individuals and aid identification of insomnia patients suited to psychological therapies.

Ancillary