Edited by: Hans-Uwe Simon
Prospective validation of ‘Allergy-Control-SCORETM’: a novel symptom–medication score for clinical trials
Article first published online: 24 JAN 2011
© 2011 John Wiley & Sons A/S
Volume 66, Issue 5, pages 629–636, May 2011
How to Cite
Häfner, D., Reich, K., Matricardi, P. M., Meyer, H., Kettner, J. and Narkus, A. (2011), Prospective validation of ‘Allergy-Control-SCORETM’: a novel symptom–medication score for clinical trials. Allergy, 66: 629–636. doi: 10.1111/j.1398-9995.2010.02531.x
- Issue published online: 7 APR 2011
- Article first published online: 24 JAN 2011
- Accepted for publication 23 November 2010
- allergic disease;
- medication scores;
- symptom scores
To cite this article: Häfner D, Reich K, Matricardi PM, Meyer H, Kettner J, Narkus A. Prospective validation of ‘Allergy-Control-SCORETM’: a novel symptom–medication score for clinical trials. Allergy 2011; 66: 629–636
Background: Combined symptom and medication scores (SMS) are recommended as primary endpoints in clinical trials. Several SMS have been created, but none has been formally validated.
Objective: To evaluate the validity of the ‘Allergy-Control-SCORE© (ACS)’, a novel instrument to assess patient’s allergy severity by recording symptoms and rescue medication.
Methods: One hundred and twenty-one consenting subjects (age 18–65 year), including 81 patients with allergic rhino-conjunctivitis and/or asthma and 40 healthy controls, participated in the study. They recorded daily nasal, eye, and lung symptoms using a 4-point scale (none, mild, moderate, and severe) and use of anti-symptomatic medication. Pollen counts were monitored during the study period. Symptom and medication scores values were compared to global allergy severity, quality of life, and allergy-related medical consultations. Feasibility was tested through a questionnaire on comprehensibility, easiness of use, and completeness. Retest reliability was assessed by testing consistency, in relation to pollen exposure, and for values recorded during each of 2 consecutive weeks.
Results: Convergent reliability analysis indicated a highly significant correlation between ACS© and global allergy severity (P < 0.0001), quality of life (P < 0.0001), and allergy-related medical consultations (P < 0.0001). Scores were highly related to pollen counts. Allergy-Control-SCORE© showed a good retest reliability (r = 0.81; P < 0.0001) and discriminated extremely well between patients with allergy and healthy controls (6.1 ± 4.8 vs 0.2 ± 0.5; t = 10.82; P < 0.0001) with a sensitivity of 97% and a specificity of 87%. Study participants evaluated the feasibility of the SMS as excellent.
Conclusions: Allergy-Control-SCORE© is a valid and reliable instrument to assess allergy severity in clinical trials and observational studies of respiratory allergic diseases.
New treatments for allergic rhino-conjunctivitis and asthma are frequently proposed and tested for safety and clinical efficacy (1–4). The need for well-performed clinical trials for respiratory allergies is therefore growing (5). The use of meta-analyses investigating the role of SIT and drugs for treating respiratory allergies has increased in the last decade (6–10). Unfortunately, study heterogeneity greatly restricts the assembly of data from different trials and impairs the reliability of the outcomes of meta-analyses (6). This heterogeneity is attributable in part to the great variety of scoring systems used to measure disease severity (6). Indeed, there is still no universally accepted system to measure symptoms. Many scales have been created, based on the impact of disease on the daily life (quality of life, symptom-free days, medication-free days, visual-analog scales, etc.) (1, 6). The most frequently used approaches are symptom scores (SS), medication scores (MS), and combined symptom–medication scores (SMS) (1). Symptom and medication scores are recommended to measure the primary outcome of clinical trials on respiratory allergies (11), and their use is proposed by international regulatory agencies, such as the European Medicines Agency (EMA) (12).
Unfortunately, no SMS has been validated so far and there is still no global consensus as to which SMS should be adopted. In an attempt to fill this gap, we elaborated an SMS defined Allergy-Control-SCORE© (ACS), developed this score over years, used it successfully in different clinical trials (13–15), and designed a study to acquire information on the validity of this optimized Score. Allergy-Control-SCORE© covers drugs used in clinical trials and observational studies. This novel tool has now been validated by assessing reproducibility, discrimination capacity, and feasibility in healthy controls and patients with respiratory allergies.
Materials and methods
Patients (age 18–65 years) were recruited from those attending the outpatient clinic departments of the ‘Dermatologikum Hamburg’, Germany, between June 21 and August 17, 2008. Inclusion criteria were the following: (1) atopic sensitization (SPT positivity to at least one of the following allergens: grass, rye, mugwort pollen, and house dust mites) (Allergopharma, Reinbek, Germany); (2) current clinical manifestations of allergic rhino-conjunctivitis and/or asthma because of exposure to one of the four allergenic sources listed above; (3) expected natural exposure to the relevant allergenic source during the study period. Controls were non-atopic volunteers with a negative history for IgE-mediated allergies. The following exclusion criteria were applied to both patients and controls: (1) current use of systemic or nasal corticosteroids, inhalative corticosteroids >400 μg budesonide or >500 μg beclomethasondipropionate; (2) long-term prophylactic use of anti-allergic medication with constant dose; (3) current treatment with specific immunotherapy; (4) food allergy; (5) clinically relevant rhinitis/rhino-conjunctival or respiratory symptoms related to other unidentified causes, (6) vasomotor, drug-induced or other kinds of nonallergic rhinitis/rhino-conjunctivitis, (7) febrile infections or inflammation of the respiratory tract, (8) irreversible secondary alterations of the upper and lower airways (e.g., emphysema and bronchiectasis).
The study was designed as a noninterventional observational controlled study. Patients and healthy controls completed a questionnaire on demographic and clinical parameters at recruitment. Scores of individual symptoms and individual medications were documented according to the ACS© on a daily basis during the pollen season over a period of 2 weeks in patients with allergy and 1 week in healthy controls, respectively. On the same day, both patients and controls were also asked to complete a self-administered questionnaire for a global assessment of severity. At the end of the first week, patients completed a validated questionnaire for the quality of life (RQLQ) (16). A ‘Global Assessment of Severity of Allergy’ (GA) was scaled (Rating Scale 1–10) through a Visual Analogue Scale by the patients themselves. A questionnaire on the feasibility of the ACS© was given to each patient at the end of the study period. Individual participants started their study period at different times during the season, so that both study groups included subjects that were exposed to either higher or lower pollen counts.
The characteristics of Allergy-Control-SCORE© are listed in Table 1, elaborated according to the GA(2)LEN taskforce Guidelines (17). The symptom score (SS) was recorded using diaries in which subjects documented on a daily basis the severity of various allergy symptoms scaled according to EMA Guidelines (12). The scale ranges from 0 to 3: 0 = absent (no sign/symptom evident); 1 = mild (sign/symptom clearly present, but minimal awareness; easily tolerated); 2 = moderate (definite awareness of sign/symptom that is bothersome, but tolerable); 3 = severe (sign/symptom that is hard to tolerate; causes interference with activities of daily living and/or sleeping). For each day, the sum of the values of ten allergy symptoms was calculated. These included ocular (itching, tear flow, and redness), nasal (sneezing, itching, running, and blockage), and bronchial (cough, wheezing, and dyspnoea) symptoms.
|Author||Kettner J., Narkus A.|
|Target||To objectively monitor severity of allergic rhinitis conjunctivitis ± allergic asthma|
|Population||Adult and adolescent patients with allergic rhinitis/conjunctivitis ± allergic asthma|
|Existing translations||German, Polish|
|Number of items||10 symptoms and 745 drugs|
|Tool dimensions||See methods|
|Scaling of items||Score points|
|Scoring of items||0–60 (global)|
|List of items||See methods|
|Minimal important difference||To be determined|
|Performed trials||Validation and use in different clinical trials|
|Copyright||Allergopharma Joachim Ganzer KG|
|Contact information||Häfner D. Department Clinical Research Allergopharma J. Ganzer KG, Hermann-Körner-Str. 52 21465 Reinbek e-mail: firstname.lastname@example.org|
Patients also documented the allergy medication needed. All allergy medications for treating related symptoms were scored for each patient and each available day in the diary. The whole list of commercially available preparations (n = 745) of the relevant categories has been taken into account in the SMS. The categories taken into consideration include nasal and ocular anti-histamines and glucocorticoids, nasal decongestants, nasal cromoglicic acid and salts, systemic antihistamines, glucocorticoids and their combinations, bronchodilators, inhaled corticosteroids, and their combinations, leukotriene receptor antagonists, and xanthines. Drugs not considered by international Guidelines for treating allergic rhino-conjunctivitis plus/minus mild asthma were not included (e.g., anti-IgE). According to the maximum SS, the total number of ‘score points’ (SP) which could be achieved by one patient in 1 day is 30. Thus, the maximum SP achieved by intake of medication is also 30 SP, subdivided into the three sub-scores for nose (max. 12 SP), eyes (max. 9 SP), and lungs (max. 9 SP), respectively. Each drug was scored considering pharmacological action (according to the corresponding ATC code), expected impact on symptoms, route of administration, the dose taken, and the duration of effect. Each medication score (MS) was balanced for the respective weight on symptoms and within the maximum score of each organ system. Thus, scoring of medication cannot yield a higher value than symptoms (Table 2).
|Medication||No. of admin||SP* per admin||Nose SP||Eyes SP||Lungs SP||Total SP|
|Prednisolone 5 mg||4||4||6.4||4.8||4.8||16|
The SMS were simply obtained by adding the daily medication score to the daily symptom score.
Pollen counts (grasses, rye, and mugwort) were derived from the European pollen information database (EAN, Siegfried Jäger, Vienna, Austria) between June 21 and August 17, 2008. Pollen exposure was graded as ‘None’, ‘Weak’, ‘Moderate’, and ‘Strong’ according to German Meteorological Service definition for each week of assessment. A ‘nonconstant’ pollen count was defined as a significant change of the respective pollen count from no or weak pollen count to moderate or strong pollen count (or vice versa), or a change from moderate to strong pollen count (or vice versa) within the 2 weeks of assessment for each patient. A ‘high’ or ‘low’ exposure to grass pollen was defined as exposure to ‘at least’ and ‘less’ than 20 grass pollen grain/m3, respectively, for at least 4 days in a week (18).
Background and demographic characteristics of subjects were summarized for both groups. Continuous variables were summarized by sample size, mean, median, standard deviation, and range. Discrete variables were summarized by frequencies and percentages. Missing daily SMS values were replaced by the last-observation-carried-forward (LOCF) method. For calculating the area under the curve (AUC) in the Receiver Operating Characteristics (ROC) analysis, missing values were replaced by interpolation. A maximum of 25% missing SMS values were allowed. Data management and statistical assessment were performed using the statistics program spss Version 15.0.
Performance of the ACS©
The data set was analyzed to measure the convergent reliability, discrimination capacity, retest reliability, and feasibility of ACS© as follows.
Handling of missing values. Missing SMS values were obtained by linear interpolation. Regarding all other parameters, the method LOCF was applied.
Convergent reliability. The convergent reliability of the ACS© was tested by comparing the average SMS value of week 1 with the following four parameters: (1) Global Assessment of Severity of Allergy (Rating Scale 1–10); (2) Quality of Life (RQLQ); (3) number of medical consultations because of the allergy within the last 12 months; and (4) the number of nonproductive days because of allergic rhino-conjunctivitis and/or asthma within the last 12 months. If data were normally distributed, Pearson’s product-moment correlations were calculated for each of the criteria 1–4. In case of abnormally distributed data, nonparametric test procedures (rank correlations) were performed. A significant positive correlation (α < 0.05, two-tailed) was considered as evidence for convergent validity. This corresponds to a correlation of r = 0.30 which is defined as a medium effect and hence can be regarded as a considerable correlation (19).
Discrimination capacity. Discrimination capacity of the ACS™ was assessed by comparing the average SMS values of the allergy (first week) and control group, respectively. Discrimination capacity was assumed to be good if the SMS value in the allergy group was significantly higher (α < 0.05, two-tailed) than the SMS value of the control group (20). Significance testing was performed with the t-test for independent samples if data were normally distributed. If this was not the case, data were analyzed using the Mann–Whitney U-test. Sensitivity and specificity were determined and a ROC-curve was depicted to calculate the difference between groups.
Retest reliability. The retest reliability was determined, for the patient group only, by analyzing the consistency between the SMS values of first week with that of the second week. Depending on the distribution characteristics of the data, the calculation was a product-moment correlation (normal distribution) or a rank correlation (non-normal distribution).
Feasibility. Feasibility was tested by using a feasibility sheet to assess the comprehensibility, easiness of use, and completeness of the SMS. Patients were also asked how much time they spent completing the diary, and they could give any comments to the diary.
Population: sociodemographic and clinical data
A total of 122 adults (82 patients and 40 controls) were asked to participate in the study. Of the 82 patients, 81 fulfilled the in/exclusion criteria and agreed to participate in the study and 80 completed the entire study. All 40 control subjects agreed to participate in the study and completed it. Mean age was 30.4 ± 9.7 years in the patient group and 35.5 ± 9.1 years in the control group (t-test, P > 0.050). The proportion of women was significantly higher in the patient group compared to the control group (61/81 [75%] vs 22/40 [45%]; χ2-test, P = 0.036). There was a predominance of caucasians in both groups (77/81 [97.1%] vs 37/40 [92.5%]; χ2-test, P > 0.050). All patients but one were affected by allergic rhinitis (80/81, 98.8%), 69/81 (85.2%) by allergic conjunctivitis, 31/81 (38.3%) by allergic asthma, and 9/81 (11.1%) by atopic dermatitis. Grass pollen sensitization was by far the most frequently observed pollen sensitization (67/81, 83%), followed by sensitization to dust mites (57/81, 73%), rye grass (53/81, 65%), and mugwort (26/81, 33%).
Convergent reliability of the SMS
The average measure of each of the criteria is also reported. A statistically significant (P < 0.0001, two-tailed) positive correlation was observed between the ACS© and the Global Assessment of the Severity of Allergy, the Quality of Life, and the number of medical consultations due to allergic disease, but not with the number of nonproductive days due to allergic rhino-conjunctivitis and/or asthma within the last 12 months. However, only seven patients (7/80, 9.9%) had lost at least one productive day due to allergy, so that there was a very low variability in this parameter (Table 3).
|Criterion||n||M ± SD||r*||P|
|SMS (units)||80||6.1 ± 4.8||–°||–|
|Global assessment of severity of allergy (rating scale 1–10)||80||3.6 ± 1.6||0.705°||<0.0001|
|Quality of Life (RQLQ total score)||79||1.9 ± 1.1||0.693°||<0.0001|
|Medical consultations because of allergy in the last 12 months||78||1.7 ± 2.5||0.389^||<0.0001|
|Nonproductive days because of allergy in the last 12 months||80||0.6 ± 2.5||0.040^||0.7240|
Discrimination capacity of the ACS©
The patient group showed a significantly higher mean SMS in comparison with the control group (6.1 ± 4.8 vs 0.2 ± 0.5; t-test, two-tailed, t = 10.82, P < 0.0001) (Fig. 1A, B). SMS separated excellently between patient and control group. The best possible SMS-cutoff value with a sensitivity of 97% and a specificity of 87% was 0.5 (Fig. 1C). In other words, 97% of the allergic patients with an SMS of 0.5 or higher could be assigned correctly to the allergy group. The same analysis has been repeated separately for SS and MS and provided similar results among participants with ‘high’ or ‘low’ pollen exposure (data not shown).
Retest reproducibility of ACSTM over time
In the patient group, the average SMS values of week 1 and week 2 were almost identical (week 1 = 6.1 ± 4.8; week 2 = 6.1 ± 5.1). The SMS values of the first and second week, considered at individual level, correlated significantly (r = 0.81, P < 0.0001, two-tailed). The retest reproducibility of SMS was analyzed separately for the two sub-groups of patients exposed either to constant or to variable pollen counts, respectively. The reproducibility of the SMS was higher in the sub-group of 45 patients for whom pollen counts were constant (5.5 ± 4.1 vs 5.8 ± 4.5; r = 0.825, P < 0.0001) than in the sub-group of 35 patients for whom pollen counts were not constant (6.8 ± 5.6 vs 6.4 ± 5.8; r = 0.795, P < 0.0001). Medication scores and symptom scores showed a high correlation coefficient when tested separately (Table 4), and independently of low or high pollen exposure.
|Item||Low pollen exposure||High pollen exposure|
|Week 1 x ± SD||Week 2 x ± SD||r||P||Week 1 x ± SD||Week 2 x ± SD||r||P|
|Symptom scores (SS)||4.5 ± 3.2||4.6 ± 3.5||0.86||<0.0001||6.1 ± 4.4||5.4 ± 5.0||0.87||<0.0001|
|Medication scores (MS)||1.2 ± 2.3||1.6 ± 3.1||0.56||<0.0001||2.5 ± 2.7||2.0 ± 2.7||0.74||<0.0001|
|Symptom–medication scores (SMS)||5.8 ± 4.2||6.2 ± 4.8||0.77||<0.0001||8.6 ± 6.7||7.4 ± 6.7||0.87||<0.0001|
Feasibility of the patient’s diary
The mean time to complete the patient diary was 4.3 ± 3.9 min. Patients stated on average that completing the diary was ‘not strenuous at all’ and ‘not unpleasant at all’. Altogether, patients appraised the content of the patient diary as ‘very easy’ to understand. Patients did not have to think very long about the single questions (average value 0.9 ± 0.8 min) and appraised the comprehensibility of the instruction as ‘very easy’. The instruction on the application of medication was estimated as ‘easy’ (0.8 ± 0.8) (Table 5).
|Item||n||x ± SD||Min||Max||Median|
|How much time did it take to complete the patient diary? [min]||80||4.3 ± 3.9||0.5||30.0||3.0|
|How strenuous was it to respond to the diary? (0 = not at all, 4 = very much)||80||0.3 ± 0.5||0.0||2.0||0|
|How unpleasant was it to answer to the questions? (0 = not at all, 4 = very much)||80||0.1 ± 0.5||0.0||4.0||0|
|I appraised the content of the patient diary altogether: (0 = very easy, 4 = very difficult)||80||0.5 ± 0.6||0.0||2.0||0|
|How long did you have to think to answer the single questions? (0 = not long at all, 4 = very long)||80||0.9 ± 0.8||0.0||3.0||1.0|
|How comprehensible is the instruction on filling out the diary? (0 = very easy, 4 = very difficult)||80||0.4 ± 0.6||0.0||2.0||0|
|How comprehensible is the instruction on the application of medication? (0 = very easy, 4 = very difficult)||79||0.8 ± 0.8||0.0||3.0||1.0|
Dependence of ACS© variations to pollen counts variations
The pollen counts of Poaceae in the two aerobiology stations were monitored during an 8-week period (Fig. 2A). The first 3 weeks and the last 5 weeks were characterized by a ‘high’ and a ‘low’ exposure to grass pollen, respectively. Impact of exposure to grass pollen on average weekly SMS was analyzed in patients sensitized to grass pollen during weeks of ‘high’ (n = 18) or ‘low’ (n = 48) pollen counts (Fig. 2B). The weekly average SMS was higher, although not significantly, in the group of patients exposed to ‘high’ pollen counts. Receiver operating characteristics analysis confirmed that the discriminatory capacity of the SMS was slightly lower, but still excellent even in the group of patients exposed to a low concentration of grass pollen (Fig. 3A and B).
This study evaluated the validity of the ACS©, a symptom-medication-score (SMS) assessing overall severity of respiratory allergy and the influence of medication. ACS© is reliable, reproducible, and feasible because: (1) the convergent reliability analysis showed, highly significant correlations with Global Assessment of Allergy Severity, a validated measure of the Quality of Life and the number of medical consultations due to allergy within the past year; (2) it discriminated significantly between patient and control groups (P < 0.0001), both in periods of high and low pollen exposure (3) it showed good retest reliability, especially under constant pollen exposure; (4) it had an excellent sensitivity (97%) and a very good specificity of (87%) in discriminating between patients and healthy controls, even in periods of low pollen concentration; (5) it responded well to weekly variations in pollen counts; and (6) patients’ assessment was good. Thus, the Allergy-Control-SCORE© is considered as a reliable and valid instrument evaluating the severity of symptoms of respiratory allergies. To our knowledge, ACS© is the first tool in its category that considers the recommendations of the EMA Guideline on the clinical development of products for specific immunotherapy for the treatment of allergic diseases (12) and which is thoroughly validated through an observational study.
Peculiarities of the ACS©
Allergy-Control-SCORE© includes a full list of relevant drugs. During a clinical trial of SIT in respiratory allergy, usually only permitted and pre-defined rescue medication can be used. The ACS© is a tool designed for use in daily practice and real-life situations, i.e., without any limitation in drug administration. Furthermore, ACS© was designed to balance impact of symptoms and drugs.
Discrimination capacity of the ACS©
Obviously, an SMS must be efficient in discriminating between patients and healthy volunteers during peak pollen season. This efficiency is challenged during low exposure to allergens, when symptoms are perceived by the patient as an unpleasant experience. Allergy-Control-SCORE© showed good discrimination capability during a period of low pollen exposure.
Monitoring allergic diseases is crucial for self-management of any chronic condition. Disease monitoring must be simple and quick; otherwise the patient’s compliance is poor. Interestingly, missing data in the validation study were minimal (not shown). Almost all patients considered the ACS© to be an extremely easy and fast tool.
The combination of SS and MS
The ACS© combines SS and MS by summing up symptoms and medication. Retest reliability and discrimination power were excellent for the symptom score and medication score separately, thus providing reassurance that both components are performing well. Among the different SMS so far available, the combined use of the ‘Average Rhinitis Rhinoconjunctivitis Total Symptom Score’ (ARTSS) and the ‘Average Rescue Medication Score (ARMS)’ has been specifically designed considering WAO recommendations (11). It has valuable power to demonstrate treatment efficacy in trials of specific immunotherapy. However, this tool lacks weighting of rescue medication and does not balance the use of rescue medication vs symptoms (21). Moreover, ARMS cannot be used in observational studies because only the predefined medication is scored. The ACS© scores each medication individually, considering efficacy, mode of action, mode of administration, duration of action, and it balances medication vs symptoms.
Allergy-Control-SCORE© is already available in German, English, Polish, and other languages. Questions used are easily translated to other languages. The medication score does not require a cross-cultural validation because use of ATC codes will guarantee that even country-specific therapies can be coded. We anticipate that validation of ACS© in many different languages will not require additional work and further studies.
In conclusion, this study shows that ACS©™ is a valid and reliable tool, for assessing and monitoring allergy severity. It considers both, symptoms and allergy medication. The structure of the SMS is robust enough for using it in clinical trials and daily clinical practice. Therefore, the ACS©™ may be considered as a gold standard for clinical trials and observational studies as promoted by the World Allergy Organization. The adoption of one such standard would remove a major hurdle for performing meta-analyses of trials in respiratory allergies.
- 7Allergen immunotherapy for asthma. Cochrane Database Syst Rev 2003;4:CD001186., , .
- 11Recommendations for standardization of clinical trials with Allergen Specific Immunotherapy for respiratory allergy. A statement of a World Allergy Organization (WAO) taskforce. Allergy 2007;62:317–324., , , , , et al.
- 12Guideline on the clinical development of products for specific immunotherapy for the treatment of allergic diseases (CHMP/EWP/18504/2006).
- 15The effects of short-term immunotherapy using molecular standardized grass and rye allergens compared with symptomatic drug treatment on rhinoconjunctivitis symptoms, skin sensitivity, and specific nasal reactivity. Otolaryngol Head Neck Surg 2005;133:538–543., , , , , .
- 17The Global Allergy and Asthma European Network. Recommendations for assessing patient-reported outcomes and health-related quality of life in clinical trials on allergy: a GA(2)LEN taskforce position paper. Allergy 2010;65:290–295., , , , , et al.
- 19Statistical power analysis for the behavioral sciences. Hillsdale, NY: Erlbaum, 1988..
- 20Testaufbau und Testanalyse. Weinheim: Psychologie Verlags Union, 1998., .