- Top of page
Impairment because of narcolepsy strongly limits job performance, but there are no standard criteria to assess disability in people with narcolepsy and a scale of disease severity is still lacking. We explored: (1) the interobserver reliability among Italian Medical Commissions making disability and handicap benefit decisions for people with narcolepsy, searching for correlations between the recognized disability degree and patients’ features; (2) the willingness to report patients to the driving licence authority and (3) possible sources of variance in judgement. Fifteen narcoleptic patients were examined by four Medical Commissions in simulated sessions. Raw agreement and interobserver reliability among Commissions were calculated for disability and handicap benefit decisions and for driving licence decisions. Levels of judgement differed on percentage of disability (P < 0.001), severity of handicap (P = 0.0007) and the need to inform the driving licence authority (P = 0.032). Interobserver reliability ranged from Kappa = −0.10 to 0.35 for disability benefit decision and from Kappa = −0.26 to 0.36 for handicap benefit decision. The raw agreement on driving licence decision ranged from 73% to 100% (Kappa not calculable). Spearman’s correlation between percentages of disability and patients’ features showed correlations with age, daytime naps, sleepiness, cataplexy and quality of life. This first interobserver reliability study on social benefit decisions for narcolepsy shows the difficulty of reaching an agreement in this field, mainly because of variance in interpretation of the assessment criteria. The minimum set of indicators of disease severity correlating with patients’ self assessments encourages a disability classification of narcolepsy.
- Top of page
Occupational problems are among the major psychosocial consequences of narcolepsy (Broughton et al., 1981; Daniels et al., 2001; Kales et al., 1982; Teixeira et al., 2004) and include ‘statistically significant deleterious effects upon performance, promotion, earning capacity, fear of or actual job loss and increased disability insurance’ (Broughton et al., 1981). Up to 80% of narcoleptic patients report falling asleep at work (Kales et al., 1982) and the rate of patients who had lost or left a job because of narcolepsy ranges from 36% (Daniels et al., 2001) to 52% (Teixeira et al., 2004). The rate of unemployment among patients with narcolepsy reaches 59% (Dodel et al., 2004) and the high socioeconomic burden of the illness includes the indirect costs of early retirement because of the disease (Dodel et al., 2004).
Despite the fact that impairment because of narcolepsy strongly limits job performance, criteria to rate the degree of disability in narcolepsy have only been suggested in Germany (Mayer, 1996; Mayer et al., 1993; Rühle and Mayer, 1998). The American Medical Association’s Guides to evaluation of permanent impairment (Andersson and Cocchiarella, 2001) provide criteria for assessing impairment because of ‘sleep and arousal disorders’, but not specifically for narcolepsy. Thus, there are no worldwide criteria to assess disability in people with narcolepsy, and even a scale of disease severity is lacking.
In Italy, the Social Security System has a non-contributory disability programme with benefits that cover all disabled citizens. Eligibility for economic and other benefits depends on the percentage of disability, ranging from 34% to 100%; people over 65 years are eligible only for non-economic benefits such as prostheses and aids for a percentage of disability ≥34% and exemption from National Health System co-payment if a full disability is recognized.
In order to orient the rating of the main disabling condition, the Italian Ministry of Health published a new Table of disability percentages for disabling diseases in 1992. If a disabling disease is not included in the above table, the disability degree has to be inferred by analogy within the frame of the general principle of rating the percentage of disability proportionately to the permanent ‘reduction of work capacity’.
In 1992 an Italian act introduced benefits for handicapped people, defined as ‘a person who has a stable or progressive physical, mental or sensory impairment which causes difficulties in learning, relationships or work integration resulting in social disadvantage or emargination’. They consist of non-economic benefits that are granted according to the severity of the handicap. If a severe handicap has been ascertained, the benefits are given to parents or relatives living with and looking after a seriously handicapped person (the right to choose the workplace nearest to his/her residence and prohibition against transfer, entitlement to 3 days of work permit every working month).
Both disability and handicap conditions are ascertained by ad hoc appointed Medical Commissions. In case of doubts on the patient’s fitness to drive at the time of the assessment, Medical Commissions are required to inform the competent driving licensing authority who will refer patients to a Local Commission for medical examination. If a patient is deemed unfit to drive, the driving licence can be revoked or provisionally suspended.
At present Medical Commissions do not have standardized criteria for disability and handicap assessment of narcolepsy, which is not included in the Ministerial table. In addition, Local Commissions’ policy recommendations do not provide for narcolepsy, and in most of the cases the patient’s driving licence is suspended.
According to confidential data provided by the Associazione Italiana Narcolettici (Italian Narcolepsy Association -AIN), people with narcolepsy usually encounter major difficulties in gaining disability and handicap benefits, and differences in evaluation have been noted in different geographical areas. These findings are in agreement with reports of patients from all over Italy referred to our Sleep Centre that has a multidisciplinary team for the study of narcolepsy, comprising experts in legal medicine and psychologists. This uncertainty concerns patients and could hamper their trust in the Social Security System. Worried about a simultaneous refusal of benefits and suspension of driving licence, many patients prefer not to apply for social benefits.
The first objective of our study was to explore the interobserver reliability among Medical Commissions making disability and handicap benefit decisions for people with narcolepsy and the Medical Commissions’ willingness to report them to the driving licence authority. The second objective was to find a correlation among Medical Commissions ratings of disability degree and patients’ clinical/polysomnographic features. As the interobserver reliability among Medical Commissions making disability benefit decisions has never been investigated, the third aim of this study was to evaluate the role played by possible sources of variance in agreement, in view of the ethical implications of such decisions.
- Top of page
Of the sixteen patients enrolled in the study, one dropped out and fifteen were examined by the four Medical Commissions. The main demographic, clinical and psychosocial features of patients (reported in Table 1) were consistent with previous Italian data on a large (n = 108) population of narcoleptic patients (Vignatelli et al., 2004), suggesting that the sample reflected the whole spectrum of the disease.
Table 1. Main demographic, clinical and psychosocial features of patients included in the study
|N (%)||15 (100.0)||9 (60.0)||6 (40.0)|
| Mean (SD)||48.7 (18.8)||43.6 (17.6)||56.5 (19.1)|
| Median (range)||40 (30–78)||39 (30–74)||61 (33–78)|
|Marital status, n (%)|
| Unmarried||6 (40)||4 (44)||2 (33)|
| Married||7 (47)||4 (44)||3 (50)|
| Divorced/widowed||2 (13)||1 (12)||1 (17)|
|Education, n (%)|
| Elementary||5 (33)||1 (11)||4 (66)|
| Middle school||4 (27)||3 (34)||1 (17)|
| High school||2 (13)||1 (11)||1 (17)|
| University||4 (27)||4 (44)||–|
|Occupation, n (%)|
| Employed||7 (47)||5 (56)||2 (33)|
| Unemployed||2 (13)||2 (22)||–|
| Housewives||3 (20)||–||3 (50)|
| Retired||3 (20)||2 (22)||1 (17)|
|Lost year at school, n (%)||6 (40)||4 (44)||2 (33)|
|Change job for the disease*, n (%)||5 (38)||2 (22)||3 (75)|
| Range||1–10 times||1–10 times||1–2 times|
|Application for social benefits, n (%)||7 (47)||4 (44)||3 (50)|
|Driving licence, n (%)|
| In possession||9 (60)||7 (78)||2 (33)|
| Suspended||3 (20)||1 (11)||2 (33)|
| Never applied for||3 (20)||1 (11)||2 (33)|
|Accidents (all life)|
| Work accidents*||6 (46)||4 (44)||2 (50)|
| Car/motorcycle accidents†||10 (77)||6 (66)||4 (100)|
| Domestic accidents||4 (27)||1 (11)||3 (50)|
|Age at onset (years)|
| Mean (SD)||25.3 (13.2)||24.4 (15.0)||26.7 (11.1)|
|Disease duration (years)|
| Mean (SD)||24.1 (19.1)||20.2 (14.4)||29.8 (24.9)|
|Diagnostic delay (years)|
| Mean (SD)||14.9 (16.1)||11.1 (14.6)||20.7 (17.8)|
| MSL, median (range)||184 s (68–642)||282 s (90–642)||138 s (78–606)|
| SOREMPs, median (range)||3 (1–5)||4 (2–5)||3 (1–5)|
|Cataplexy, n (%)||12 (80)||7 (78)||5 (83)|
|Nap per day|
| Median (range)||2 (1–5)||2 (1–5)||2 (1–5)|
|Therapy for narcolepsy, n (%)|
| No drug||2 (13)||1 (11)||1 (17)|
| Modafinil||7 (47)||5 (56)||2 (33)|
| Modafinil + anticataplectics||6 (40)||3 (33)||3 (50)|
|ESS, mean score (SD)||14.7 (5.7)||15.0 (6.7)||13.7 (4.3)|
|BQS, mean score (SD)||7.8 (4.5)||7.9 (5.4)||7.7 (3.0)|
|IQC, mean score (SD)||12.1 (7.0)||10.0 (7.3)||15.2 (5.9)|
|Zung, mean score (SD)||32.9 (10.8)||31.6 (13.9)||35.0 (3.4)|
|SF-36 PCS, mean score (SD)||44.2 (10.2)||45.5 (13.9)||42.5 (4.6)|
|SF-36 MCS, mean score (SD)||43.9 (10.4)||45.9 (13.2)||41.3 (5.9)|
|Associated medical disorders, n (%)|
| Diabetes||1 (7)||1 (11)||–|
| Visual disorders||1 (7)||–||1 (17)|
| Respiratory disorders||3 (20)||2 (22)||1 (17)|
| Acoustic disorders||1 (7)||–||1 (17)|
| Limb limitations||2 (13)||1 (11)||1 (17)|
The level of judgement differed among the four Medical Commissions on percentage of disability (P < 0.001), severity of handicap (P = 0.0007) and need to inform the driving licence authority (P = 0.032) (see Table 2).
Table 2. Descriptive statistics of Medical Commissions’ (MC) level of judgements before and after the training procedure
|Before the training procedure|
| Percentage of disability|
| Median (range)||46 (0–90)||60 (0–100)||46 (0–100)||70 (46–100)||<0.001|
| Judgement on handicap, n (%)|
| No handicap|| 2 (13)|| 3 (20)|| 8 (53)|| 0||0.0007|
| Handicap||12 (80)|| 6 (40)|| 7 (47)||10 (67)|
| Severe handicap|| 1 (7)|| 6 (40)|| 0|| 5 (33)|
| Information to DLA, n (%)||15 (100)||12 (80)||11 (73)||15 (100)||0.032|
|After the training procedure|
| Percentage of disability|
| Median (range)||46 (0–90)||60 (0–100)||67 (46–100)||75 (46–100)||<0.001|
| Judgement on handicap, n (%)|
| No handicap|| 2 (13)|| 3 (20)|| 4 (26)|| 0||0.028|
| Handicap||12 (80)|| 6 (40)||10 (67)|| 9 (60)|
| Severe handicap|| 1 (7)|| 6 (40)|| 1 (7)|| 6 (40)|
| Information to DLA, n (%)||15 (100)||12 (80)||10 (67)||15 (100)||0.015|
The raw agreement (Table 3) on disability decision ranged from 20.0% to 53.4% between each pair of Medical Commissions; the interobserver reliability ranged from Kappa = −0.10 to 0.35 (‘fair agreement’); overall consensus was never reached on any patient. The raw agreement on handicap decision ranged from 13.3% to 60.0% and the interobserver reliability ranged from Kappa = −0.26 to 0.36 (‘fair agreement’). The raw agreement on driving licence decision ranged from 73% to 100% (Kappa not calculable).
Table 3. Raw agreement and interobserver reliability (Kappa) on disability and handicap benefit decisions between each pair of Medical Commissions (MC)
| ||Disability benefit decision||Handicap benefit decision|
|Raw agreement (%)||Kappa|| 95% CI||Raw agreement (%)||Kappa|| 95% CI|
The following categories of questions recurred differently among the Medical Commissions: functional status and autonomy (P = 0.0001), and occurrence of accident of any type (P = 0.0010). A qualitative analysis of answers given by patients to the same questions did not disclose major differences.
The following categories of grounds of disability recurred differently among Medical Commissions: disease severity (P < 0.0001), reduction of work capacity (P = 0.002), occurrence of accident of any type (P = 0.001).
Spearman’s correlation among Medical Commissions’ percentages of disability ranged from rho 0.80 (P < 0.001) to rho 0.92 (P < 0.001). Spearman’s correlation between percentages of disability and patients clinical/polysomnographic features showed a correlation with age of patients, naps per day, EDS (by means of Bologna Questionnaire on Sleepiness), cataplexy and SF-36 Physical and Mental Components Summaries in all or almost all Medical Commissions (Table 4). No correlations were found with mood status and mean sleep latency at MSLT.
Table 4. Spearman’s correlation between percentages of disability and patients’ clinical/polysomnographic features (statistically significant correlations in bold)
| ||Percentage of disability|
|Rho (P)||Rho (P)||Rho (P)||Rho (P)|
|Age||0.60 (0.018)||0.68 (0.005)||0.60 (0.017)||0.66 (0.007)|
|Age at onset||0.45 (0.095)||0.45 (0.096)||0.53 (0.043)||0.35 (0.18)|
|Disease duration||0.19 (0.49)||0.27 (0.32)||0.17 (0.55)||0.33 (0.23)|
|MSL||0.24 (0.38)||0.34 (0.22)||0.34 (0.21)||0.25 (0.36)|
|Nap per day||0.54 (0.036)||0.47 (0.073)||0.24 (0.40)||0.61 (0.016)|
|ESS||0.39 (0.14)||0.28 (0.30)||0.26 (0.35)||0.44 (0.10)|
|BQS||0.65 (0.008)||0.67 (0.006)||0.42 (0.12)||0.71 (0.003)|
|IQC||0.79 (0.001)||0.75 (0.001)||0.57 (0.026)||0.71 (0.003)|
|Zung||0.14 (0.61)||0.37 (0.17)||0.067 (0.81)||0.19 (0.49)|
|PCS||−0.58 (0.028)||−0.68 (0.007)||−0.33 (0.24)||−0.64 (0.015)|
|MCS||−0.56 (0.039)||−0.60 (0.022)||−0.68 (0.008)||−0.57 (0.033)|
After the training procedure (Table 2), the level of judgement on disability, handicap and the need to inform the driving licence authority still differed. Only one Medical Commission statistically increased its percentages of disability (median from 46 to 67, P = 0.008). The interobserver reliability substantially did not change either for disability decision (Kappa ranged from −0.10 to 0.43) or handicap decision (Kappa ranged from −0.05 to 0.44).
- Top of page
Our study explored the agreement on social benefit decisions for people with narcolepsy. As suggested by patients’ reports, we found a low interobserver reliability among Medical Commissions on both disability and handicap benefit decisions, a result that was not improved by the training procedure on narcolepsy or by additional information.
Despite different assessment procedures and methods, it is well known that it is difficult, in practice, to determine what constitutes disability and work incapacity (Stattin, 2005). Even if interobserver reliability has been recommended in all the fields of medicine (Simel and Rennie, 1997), to our knowledge few studies have addressed medical decisions in the social field. Elder et al. (1994) found a low level of agreement (overall Kappa = 0.24) on ill-health retirement decisions on a series of 10 simulated requirement requests considered by 35 occupational physicians during a conference. The authors attributed the result to a wide variation in interpretation of the criterion, i.e. ‘permanent incapacity’, used for ill-health retirement. In an analogous study by Davies et al. (2004) (39 participants, 16 cases) Kappa ranged from −0.33 to 0.5. The study by Fleten et al. (2004) assessed whether modified working conditions might reduce ongoing sick leave on the basis of certificate-mediated information of general practitioners. They found an interobserver reliability among consultants that never exceeded slight agreement (Kappa ≤ 0.20) and concluded that ‘medical certificates include little information on factors important for predicting a potential sick-leave reduction by modified working conditions’.
Our study was the first to explore interobserver reliability in social benefit decisions for a single sleep disorder, and confirms the difficulty in reaching a consensus on such medical judgements.
Although the four Medical Commissions failed to agree on which benefits to award to patients, the excellent correlation among the disability percentage ratings of all four commissions showed their skill in discriminating different degrees of disability because of narcolepsy. Moreover, the fairly good correlation among percentages of disability stated by the Medical Commissions and the severity of symptoms scored by questionnaires could indicate that patients’ self-assessments of symptoms are reliable and the Medical Commissions share a minimum set of clinical indicators of narcolepsy severity:
While the correlations found for cataplexy severity (measured by the Italian questionnaire on cataplexy) and the impact on quality of life (expressed by Physical and Mental Component Summaries from the SF-36) were largely expected, the Medical Commissions’ judgments on EDS correlated only with the numbers of naps per day and with the Bologna questionnaire on sleepiness, but not with the other subjective measure of sleepiness (ESS) or values of MSLT.
This is not surprising, considering not only that objective and subjective tools measure different aspects of sleepiness, but also that subjective sleepiness is multifaceted and includes various domains (Kim and Young, 2005; Rinaldi et al., 2001). In particular, apparently incongruous studies respectively showed a moderate (Johns, 1994), weak (Blaivas et al., 2007; Briones et al., 1996; Chervin et al., 1997; Olson et al., 1998; Sangal et al., 1999) or no correlation (Benbadis et al., 1999; Chervin and Aldrich, 1999) between ESS score and MSLT latencies. Others found a low correlation between ESS and the latencies at the Maintenance of Wakefulness Test (Sangal et al., 1999). Finally, Kim and Young (2005) showed that different scales reflect different aspects of subjective sleepiness.
With the caution arising from this preliminary study, our data suggest that also from the medico-legal point of view EDS is a complex phenomenon, and further efforts are needed to identify the most reliable and useful scales (or techniques) to assess the disabling weight of sleepiness in narcolepsy and other hypersomnias.
The Medical Commissions’ propensity to report all or almost all the examined patients to the driving licence authority was noteworthy. This finding indicates that Medical Commissions de facto fill the gap in national regulations about driver licensing in narcolepsy. In fact, Italian legislation, as well as the prevalent law in another seven (Austria, Denmark, Germany, Greece, Ireland, Luxembourg, Portugal) out of 15 European member states before the 2004 enlargement, only applied the European Community regulation of the driving licence, i.e. Council Directive 91/439/EEC dated 29th July 1991. Annex III to the directive does not mention either narcolepsy, or any other disease associated with EDS among the diseases or impairments which can affect the ability to drive. Conversely, the seven remaining countries (Belgium, France, Finland, Netherlands, Sweden, United Kingdom, Spain) implemented a supranational law with specific regulations for driver licensing in narcolepsy (Ingravallo et al., 2005). However, these regulations scarcely detail the methods required to evaluate either disease severity (i.e. questionnaires or laboratory exams) or eventual improvements after treatment, and also show a widespread variability in the criteria adopted to evaluate the fitness to drive and the licence period of validity (Ingravallo et al., 2005). This heterogeneity is also shared by the regulations of different states in the USA (Pakola et al., 1995) and likely reflects the lack of evidence-based supporting procedures for the evaluation of narcoleptic patients’ ability to drive. While convergent studies confirm that MSLT cannot predict the driving risk in narcoleptic patients (Aldrich, 1989; Findley et al., 1995), further research is needed to confirm the reliability of driving simulator tests (Findley et al., 1995; Kotterba et al., 2004) or alternative neurophysiological instruments (Maintenance of Wakefulness Test).
The further aim of our study was to investigate the sources of disagreement on disability benefit decisions. In accordance with the conceptual framework outlined by Shrout et al. (1987) and Lipton et al. (1993), we explored the variance in criteria, information and interpretation.
The criterion variance was in theory excluded a priori, because all raters had to refer to the same criterion, i.e. ‘reduction of work capacity’. As regards information variance, we monitored this source of variance rather than removing (i.e. through a standardized interview) to avoid an unreal setting. The standardization of sessions and the randomization of patients theoretically restricted this source of variance to the variability in Medical Commissions’ questions (question variance) and patients responses (response variance). Response variance did not seem substantial, whereas question variance could have reflected the different background and operative routine of each Medical Commission. A variance in interpretation of the criterion ‘reduction of work capacity’ was suggested by significant differences in three to four categories of grounds of disability. Different raters’ opinions on this topic could also account for the double-sided results of a poor agreement in disability benefit decisions and an excellent correlation among Medical Commissions in disability percentages rating.
Overall, these findings demonstrate that Medical Commissions agreed on who are the most severe and the mildest patients but did not agree on what are the minimum and the maximum benefits a patient with narcolepsy could gain, as indicated by the mismatch in application of the available range of disability percentage: only two Medical Commissions applied it in full, whereas one Medical Commission reckoned no patient had a full disability and for another Medical Commission all subjects were at least more than 45% disabled.
We speculate that this lack of agreement suggests that certain distinctive features of narcolepsy prevent a univocal appraisal of its consequences on work capacity. Firstly, narcolepsy characteristically not only limits patients’ working capacity, but also restricts their range of occupations to jobs which do not entail periods of physical inactivity or boredom, or risk of sleep-related accidents. Secondly, as the various symptoms are in part paroxysmal and modulated by circumstances and emotions, patients experience a kind of ‘intermittent disability’, often with an individual time-schedule of daytime sleep. Lastly, people with narcolepsy are probably unfit for current ‘standard’ work, consisting of an office job that is not appropriate for people with EDS, and they are not helped by new technologies that may play a major role in integrating other disabled workers. These features, differentiating narcolepsy from many other disabling diseases, strongly suggest that most of the criteria and procedures adopted to assess disability and ill health retirement are inadequate to evaluate people with narcolepsy. Thus, in order to reduce inconsistency and avoid social inequality, a guidance is required for assessing disability because of narcolepsy.
Although the multifaceted nature of EDS and the lack of direct biological measures of sleepiness hamper any objective evaluation of the social consequences of EDS on patients’ lives, our study indicates that Medical Commissions shared a minimum set of indicators of narcolepsy severity and that patients’ self assessments of narcolepsy symptoms seem reliable. A standardized multidisciplinary procedure (including experts in sleep medicine, legal medicine and occupational medicine) should devise a disability classification of narcolepsy based on the clinical severity of symptoms.