Edited by: Hans-Uwe Simon
Reliability of EP3OS symptom criteria and nasal endoscopy in the assessment of chronic rhinosinusitis – a GA2LEN study
Version of Record online: 17 NOV 2010
© 2010 John Wiley & Sons A/S
Volume 66, Issue 4, pages 556–561, April 2011
How to Cite
Tomassen, P., Newson, R. B., Hoffmans, R., Lötvall, J., Cardell, L. O., Gunnbjörnsdóttir, M., Thilsing, T., Matricardi, P., Krämer, U., Makowska, J. S., Brozek, G., Gjomarkaj, M., Howarth, P., Loureiro, C., Toskala, E., Fokkens, W., Bachert, C., Burney, P. and Jarvis, D. (2011), Reliability of EP3OS symptom criteria and nasal endoscopy in the assessment of chronic rhinosinusitis – a GA2LEN study. Allergy, 66: 556–561. doi: 10.1111/j.1398-9995.2010.02503.x
- Issue online: 24 FEB 2011
- Version of Record online: 17 NOV 2010
- Accepted for publication 11 October 2010
- reproducibility of results;
- validation studies
To cite this article: Tomassen P, Newson RB, Hoffmans R, Lötvall J, Cardell LO, Gunnbjörnsdóttir M, Thilsing T, Matricardi P, Krämer U, Makowska JS, Brozek G, Gjomarkaj M, Howarth P, Loureiro C, Toskala E, Fokkens W, Bachert C, Burney P, Jarvis D. Reliability of EP3OS symptom criteria and nasal endoscopy in the assessment of chronic rhinosinusitis – a GA2LEN study. Allergy 2011; 66: 556–561.
Background: The European Position Paper on Rhinosinusitis and Nasal Polyps (EP3OS) incorporates symptomatic, endoscopic, and radiologic criteria in the clinical diagnosis of chronic rhinosinusitis (CRS), while in epidemiological studies, the definition is based on symptoms only. We aimed to assess the reliability and validity of a symptom-based definition of CRS using data from the GA2LEN European survey.
Methods: On two separate occasions, 1700 subjects from 11 centers provided information on symptoms of CRS, allergic rhinitis, and asthma. CRS was defined by the epidemiological EP3OS symptom criteria. The difference in prevalence of CRS between two study points, the standardized absolute repeatability, and the chance-corrected repeatability (kappa) were determined. In two centers, 342 participants underwent nasal endoscopy. The association of symptom-based CRS with endoscopy and self-reported doctor-diagnosed CRS was assessed.
Results: There was a decrease in prevalence of CRS between the two study phases, and this was consistent across all centers (−3.0%, 95% CI: −5.0 to −1.0%, I2 = 0). There was fair to moderate agreement between the two occasions (kappa = 39.6). Symptom-based CRS was significantly associated with positive endoscopy in nonallergic subjects, and with self-reported doctor-diagnosed CRS in all subjects, irrespective of the presence of allergic rhinitis.
Conclusion: Our findings suggest that a symptom-based definition of CRS, according to the epidemiological part of the EP3OS criteria, has a moderate reliability over time, is stable between study centers, is not influenced by the presence of allergic rhinitis, and is suitable for the assessment of geographic variation in prevalence of CRS.
Chronic rhinosinusitis (CRS), a disease defined as chronic inflammation of the nose and paranasal sinuses, has a considerable impact on morbidity and quality of life. There are varying estimates of disease prevalence based on a limited amount of data (1–4), and to date, no pan-European epidemiological study has been undertaken. The paucity of comparable and reliable data is in part related to the lack of uniformly accepted diagnostic criteria for CRS. Although a number of guidelines and consensus documents have been developed, considerable differences in diagnostic criteria and the lack of an accepted gold standard diagnosis make it difficult to make comparisons.
Upper airway diseases present with a variable pattern of common symptoms such as nasal obstruction and discharge, making the epidemiological diagnosis of CRS difficult to differentiate from allergic and nonallergic rhinitis based on symptomatic grounds only. Moreover, not all patients presenting with symptoms meeting CRS criteria have evidence of disease if diagnosis is complemented with nasal endoscopy and CT. The 2007 EP3OS guideline (5) incorporates symptomatic, endoscopic, and radiologic criteria in the clinical diagnosis of CRS. However, as nasal endoscopy and CT are difficult to apply in large-scale epidemiological studies, the EP3OS document defines CRS by symptoms only, when used in epidemiological studies. The repeatability and the validity of the EP3OS criteria have not yet been validated extensively.
Recently, the Global Allergy and Asthma European Network of Excellence (GA2LEN) initiated a large epidemiological study comprising a postal survey (the GA2LEN Survey) followed by a case–control study (the GA2LEN Survey Follow-Up), on allergy, asthma, and upper airway disease across Europe. In this study, diagnosis of CRS is based on a questionnaire for symptoms forming part of the EP3OS diagnostic criteria. The current study aims to validate this by reporting the repeatability of the epidemiological EP3OS symptom criteria, and by describing the relationship of symptom criteria and self-reported doctor-diagnosed CRS with findings from nasal endoscopy.
In a first cross-sectional phase (the GA2LEN Survey), 11 participating centers sent a questionnaire by mail to a random sample of at least 3000 subjects aged 15–75 years, with up to three attempts to elicit a response. Samples were identified by random sampling from a population-based local sampling frame.
The questionnaire was newly developed for the diagnosis of CRS (Table 1). A positive diagnosis of CRS was based on symptoms as defined in the 2007 EP3OS epidemiological criteria (Table 1); additionally, subjects were asked if a doctor had ever told whether the subject had CRS (further referred to as ‘self-reported doctor-diagnosed CRS’). Asthma was defined as reporting ‘having ever had asthma’ and at least one of the following symptoms in the last 12 months: (i) wheeze or whistling in the chest; or (ii) waking up with chest tightness, shortness of breath or an attack of coughing. Allergic rhinitis was defined by the self-reported history of ‘nasal allergy’.
|EP3OS criteria for diagnosis of CRS|
|Presence of two or more of the following symptoms|
|Nasal blockage, obstruction or congestion|
|Nasal discharge (either anterior or posterior nasal drip)|
|Facial pain or pressure|
|Reduction or loss of smell|
|One of which should be blockage or discharge|
|Symptoms should be present during >12 weeks without complete resolution|
|Endoscopic signs of|
|Mucopurulent discharge, primarily from middle meatus, and/or|
|Edema or obstruction primarily in middle meatus|
|CT changes: mucosal changes within the ostiomeatal complex and/or sinuses|
|For assessing CRS as per EP3OS|
|Has your nose been blocked for more than 12 weeks during the last 12 months?|
|Have you had pain or pressure around the forehead, nose or eyes for more than 12 weeks during the last 12 months?|
|Have you had discolored nasal discharge or discolored mucus in the throat for more than 12 weeks during the last 12 months?|
|Has your sense of smell been reduced or absent for more than 12 weeks during the last 12 months?|
|Has a doctor ever told you that you have chronic sinusitis|
|Do you have any nasal allergies, including hay fever?|
In a second phase (the GA2LEN Survey Follow-Up), each center invited 120 randomly selected subjects with asthma, 120 with CRS, 40 with asthma and CRS, and 120 with neither asthma or CRS for a clinical study visit with further investigations among which a questionnaire including the same questions as those described earlier for the postal survey.
In two centers (Ghent and Amsterdam), each participant in the follow-up phase was invited to undergo nasal endoscopy. Nasal endoscopy was performed, blinded to symptom status, by otorhinolaryngology specialists or residents using routine clinical rigid 30° endoscopes. An endoscopy positive for rhinosinusitis was defined, based on the EP3OS criteria, as presence of polyps, presence of edema in the middle meatus, or presence of thick purulent discharge in the middle meatus, at either nasal side.
All data available to the coordinating center that had undergone full quality control by November 1st 2009 were included in this analysis. The prevalences of CRS, each of the symptoms of CRS, asthma and allergic rhinitis in the survey and follow-up were estimated using data only from participants who had taken part in both. As the sample in the follow-up phase was selected based on disease in the survey sample (and therefore had higher prevalences of asthma, CRS and both compared to the general population), prevalence estimates were standardized, for both CRS and asthma, to the original sampled population by using inverse sampling probability weights. The standardized difference in prevalence of disease between the two phases was estimated for each center and as an overall estimate (6). Variation of this difference between centers was estimated (Wald chi-square test for heterogeneity), and the I-squared heterogeneity measure was computed (7). Absolute repeatability (8), standardized to account for the high prevalence of asthma, CRS and both in the follow-up phase, and Cohen’s kappa (κ) statistics were derived, with confidence intervals calculated using the delta method with the normalizing transformation log (1 − κ).
The odds ratios of having CRS symptoms by endoscopy results or by self-reported doctor-diagnosed CRS were derived and tested with Pearson chi-square test. To assess whether these associations were similar in subjects with and without current allergic rhinitis (defined as self-reported nasal allergy or hay fever, plus sneezing, runny or blocked nose in the absence of a cold in the last 12 months), analyses were stratified by current allergic rhinitis, the Breslow-Day test was used to test for interaction, and the Mantel–Haenszel weighted odds ratio was calculated. Binomial confidence intervals according to Clopper and Pearson were calculated around proportions. All statistical analyses were carried out using stata Version 11 (StataCorp, College Station, TX, USA) and spss Version 15 (SPSS Inc., Chicago, IL, USA).
Eleven centers in seven countries provided data from baseline and follow-up surveys to the coordinating center by November 2009. One center which had not yet completed the study was excluded. A total of 36 790 subjects had completed the postal questionnaire, and 1700 subjects had been seen in the follow-up clinical visit. In this group, 652 were controls, 469 had asthma but no sinusitis, 411 had sinusitis but no asthma, 168 had asthma and sinusitis. Of these, 50.1% were women, the median age was 48.7 years (IQR 36.8–59.6 years), and the median time between postal survey and clinical visits was 287 days (IQR 205–359 days).
Results are based on the subjects who had taken part in both phases of the study (n = 1700). Table 2 shows the standardized difference in prevalence between the two study phases, the absolute repeatability (standardized for disease prevalence), and the unstandardized kappa statistic, for CRS, asthma, allergic rhinitis, and some of their related symptoms. Figure 1 illustrates the standardized difference in prevalence between the two study phases for the outcomes CRS, asthma, and allergic rhinitis in each of the participating centers.
|Outcome||N||Standardized prevalence in survey (%)||Standardized difference in prevalence||Absolute repeatability||Kappa|
|Diff. (%)||95% CI||P||Tests for heterogeneity|
|CRS||1700||9.3||−3.0||−5.0 to −1.0||0.003||0.0||0.570||91.8||39.6|
|Blocked nose||1687||13.3||1.3||−4.8 to 7.3||0.680||52.5||0.026||84.7||45.4|
|Pain or pressure||1691||7.2||−3.1||−5.0 to −1.2||0.002||50.5||0.034||93.0||35.7|
|Discolored nasal discharge||1687||6.8||−1.5||−2.7 to −0.3||0.012||45.6||0.057||93.3||33.5|
|Reduced sense of smell||1683||6.9||−1.0||−1.9 to −0.1||0.034||55.3||0.018||95.4||53.0|
|Doctor-diagnosed CRS||1685||2.9||5.7||0.2 to 11.2||0.040||52.0||0.028||92.3||48.8|
|Asthma||1700||8.0||1.3||−0.1 to 2.7||0.066||41.5||0.082||96.5||82.3|
|Wheezing with SOB||1602||13.3||−3.8||−9.3 to 1.8||0.180||58.9||0.001||89.6||54.6|
|Nasal allergy||1618||41.8||0.1||−1.8 to 2.0||0.900||8.5||0.360||92.0||72.8|
The prevalence of symptom-based CRS, estimated from the second phase, was lower than that obtained in the first phase (−3.0%; 95% CI −5.0 to −1.0%), and this difference was similar in all centers (I2 = 0). Standardized absolute repeatability of symptom-based CRS was 91.8%, and the unstandardized kappa was 39.6. All of the individually reported symptoms that contributed to the symptom-based definition of CRS showed a pattern similar to that of CRS.
The prevalence of self-reported doctor-diagnosed CRS was lower than symptom-based CRS, with a marginally higher kappa (48.8). The standardized difference in prevalence showed an overall increase in prevalence in the second phase, with significant heterogeneity between centers (I2 = 52.0; P = 0.028). By comparison, the prevalence of wheezing with breathlessness showed a nonsignificant (P = 0.18) fall between the two study phases with significant variation between centers (I2 = 58.9; P = 0.0095). The unstandardized kappa (54.6) showed a moderate agreement. The prevalence of asthma showed no significant standardized difference between the baseline and clinical follow-up, with nonsignificant variation of this between centers (I2 = 41.5; P = 0.082). The reporting of a history of ‘hay fever or nasal allergies’ showed no significant difference in prevalence between the two study phases, a standardized repeatability similar to that for CRS, and an unstandardized kappa (72.8) indicating good agreement.
Association of symptoms with endoscopy and self-reported doctor-diagnosed CRS
Three hundred and forty-two participants in Ghent and Amsterdam underwent nasal endoscopy. Table 3 shows the associations of symptom-based CRS with endoscopy and self-reported doctor-diagnosed CRS, stratified for current allergic rhinitis. Overall, 61.7% (95% CI: 50.3–72.3%) of symptom-positive subjects had a positive endoscopy, and 38.0% (32.3–44.1%) of symptom-negative subjects had a positive endoscopy. Of positive endoscopies, 33.6% (26.0–41.7%) had CRS symptoms, and 83.9% (77.9–88.8) of negative endoscopies had no CRS symptoms. A total of 31.4% (21.8–42.3) of symptom-positive and 11.1% (7.7–15.4%) of symptom-negative subjects had a self-reported doctor-diagnosed CRS. Symptom-based CRS was significantly associated with a positive endoscopy (OR 2.62; 95%CI [1.57–4.39]; P < 0.001) and with middle meatal purulent secretions and middle meatal edema. The association of symptom-based CRS to a positive endoscopy was stronger in subjects without current allergic rhinitis (OR 3.78; P < 0.001) compared to subjects with allergic rhinitis (OR 1.45; P = 0.437), and the Mantel–Haenszel-corrected OR was comparable with the uncorrected OR (OR 2.41 [1.43–4.05], P < 0.001) (Table 3). The Breslow-Day test showed no significant differences between odds ratios of each subgroup. Symptom-based CRS was associated with a self-reported doctor-diagnosed CRS (OR 3.67 [2.03–6.60], P < 0.001). This association was not modified by the presence of allergy (adjusted OR 3.62 [1.97–6.63], Breslow-Day P = 0.871).
|Crude odds ratio||Subjects without current AR||Subjects with current AR||Breslow-Day test||Mantel–Haenszel adjusted odds ratio|
|OR||95% CI||P (χ2)||OR||95% CI||OR||95% CI||P||Adj. OR||95% CI||P|
|Middle meatal findings|
The revised EP3OS consensus document provided diagnostic criteria for CRS in 2007, and we have applied these criteria in a two phase, multicenter, questionnaire-based cross-sectional epidemiological study, the GA2LEN Survey and the GA2LEN Survey Follow-Up Endoscopic findings characteristic of CRS (as defined by the EP3OS criteria), and the reporting of doctor-diagnosed CRS was used to assess the validity of the reported symptoms for defining CRS in this setting.
We used three parameters to assess the reliability of the CRS questionnaire: standardized difference in prevalence, standardized absolute repeatability, and unstandardized kappa statistic. When using general population surveys to describe between center differences in chronic disease prevalence, a prevalence estimate that is stable over time is needed, even though individual changes (disease incidence and disease remission) may be occurring within the population. The absence of change in prevalence implies that, at population level, the number of subjects who are asymptomatic in the first phase but have symptoms in the second phase is equivalent to the number of those with symptoms in the first phase who report no symptoms in the second phase. We observed a decrease in prevalence of CRS between the two occasions. We also observed a decrease in prevalence for ‘wheezing with breathlessness’, a commonly used symptom question in respiratory epidemiology. The magnitude of the difference for CRS was equivalent to that seen for ‘wheezing with breathlessness’ and most importantly showed no variation between centers. This means that there is no evidence that the broad interpretation of geographic variation in prevalence of disease using this instrument will be affected (that is, the error is constant across populations).
Absolute repeatability was high for all questions, and to some extent this is not surprising as within subject agreement for low prevalence conditions is likely to be solely because of the chance. Unstandardized repeatability (Cohen’s κ) was fair to moderate for CRS questions and for symptom-based CRS definition, whereas it was moderate to good for asthma and nasal allergy. Cohen’s kappa is a widely accepted measure to assess chance-corrected agreement (9) but it has been argued that in questionnaire development for assessing symptoms in population-based studies (where the prevalence of the outcome is low) survey items should not be rejected on the basis of kappa alone (8). Other parameters should be considered, including change in prevalence and measures of validity against clinical criteria.
Development of instruments suitable for the epidemiological investigation into CRS is hampered by the lack of an easily measurable gold standard definition of disease. We compared symptom criteria to endoscopy and to self-reported doctor-diagnosed CRS, which are assumed to be highly specific, but not sensitive, for CRS. We demonstrated significant associations of the symptom criteria with positive endoscopy and doctor-diagnosed CRS. Of subjects who had positive symptoms, 62% had a positive endoscopy, whereas 38% of symptom-negative patients had a positive endoscopy. As patients in this study were required to have chronic symptoms in the last 12 months but not necessarily at the time of endoscopy, we expect that a small proportion of endoscopy-negative patients may have had a positive endoscopy during active symptoms and vice versa. To our knowledge, this study is the first to document endoscopy in asymptomatic subjects.
In line with work of Stankiewicz (10) investigating CT and endoscopy in CRS patients, we observe that only a proportion of symptom-positive patients had a positive endoscopy. However, in that study, only 29% of participants had a positive endoscopy, while 62% of our symptom-positive subjects had a positive endoscopy. This difference could be explained by less strict symptom criteria, and the exclusion of nasal polyp patients and patients with purulence on rhinoscopy. In a large hospital-based study in Istanbul, Tahamiler et al. (11) report that in 768 patients with CRS fulfilling the EP3OS symptom criteria, 31.3% of allergic patients and 24.7% of nonallergic patients had a positive nasal endoscopy. This is a much smaller proportion than in our study (respectively 58% and 65%), but the reason for the difference is unclear, as this study used even less strict criteria for positivity of endoscopy.
In the diagnosis of CRS, controversy exists whether or not to corroborate positive symptoms with endoscopy and CT (12). The EP3OS criteria propose a confirmation by either CT or endoscopy. As it is not possible to include CT in epidemiological studies involving healthy subjects, we can only hypothesize that some of our participants with positive symptoms but negative endoscopy may have had radiographic evidence of disease. In fact, in a study comparing CT and endoscopy using a proprietary scoring system in CRS patients (13), 65% of endoscopy-negative patients had radiographic evidence of disease. In another study (10), this proportion was 36%. Extrapolating these data to our population, we can estimate that 76–87% of our symptom-based CRS diagnoses would be confirmed by endoscopy or CT had both been available.
The study by Tahamiler suggests that the association of symptom-based CRS with objective markers of disease is not greatly influenced by the presence of allergic rhinitis. However, there is overlap in the symptoms associated with each condition, particularly for nasal obstruction and rhinorrhoea (14–16). Therefore, we might expect a weaker association of symptom-based CRS with objective markers of disease in subjects with allergic rhinitis. We addressed this question by stratifying our analyses for current allergic rhinitis. The strength of the association of a positive endoscopy with CRS symptoms was weaker in the presence of allergic rhinitis, although we found no statistically significant evidence for this (Breslow-Day test for interaction, P = 0.074). However, it has been shown that the statistical power for testing interaction is too low in many epidemiological studies (17). Although our observations could be explained by an overlap of CRS and allergic rhinitis symptoms, endoscopic findings such as edema can also be present in both diseases. This may account for a high proportion (49.3%) of positive endoscopies in CRS-negative allergic rhinitis patients. In contrast to endoscopy, symptom-based CRS was associated with self-reported doctor-diagnosed CRS, irrespective of the presence of allergic rhinitis. Taken together, these findings suggest that a symptom-based definition of CRS is stable, irrespective of the presence of allergic rhinitis, and that positivity of the endoscopic criteria may be influenced by the presence of allergic rhinitis. Further research on the specificity of symptom criteria and endoscopy in relation to radiologic changes is warranted.
We have for the first time assessed the reliability of the symptom-based EP3OS definition for epidemiological diagnosis of CRS. Our findings suggest that a symptom-based definition of CRS has a moderate reliability over time, is stable between study centers, is not influenced by the presence of allergic rhinitis, and is suitable for the epidemiological assessment of geographic variation in prevalence of CRS.
All authors are members of Workpackage 1.2.1 of the Global Allergy and Asthma European Network and are supported by the Sixth European Union Framework program for research, contract no. FOOD-CT-2004-506378.
PT, RN, RH, and DJ analyzed the data and wrote the manuscript. WF, CB, PB, and DJ conceived and supervised the study. All authors collected data and critically revised the manuscript.