SEARCH

SEARCH BY CITATION

Keywords:

  • asthma;
  • questionnaire design;
  • rhinitis;
  • validation studies

Abstract

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References

To cite this article: Fonseca JA, Nogueira-Silva L, Morais-Almeida M, Azevedo L, Sa-Sousa A, Branco-Ferreira M, Fernandes L, Bousquet J. Validation of a questionnaire (CARAT10) to assess rhinitis and asthma in patients with asthma. Allergy 2010; 65: 1042–1048.

Abstract

Background and aim:  The Control of Allergic Rhinitis and Asthma Test (CARAT) was developed to be used in the concurrent management of these diseases, as recommended by the Allergic Rhinitis and its Impact on Asthma (ARIA) guidelines. However, it was necessary to statistically identify and remove redundant questions and to evaluate the new version’s factor structure, internal consistency and concurrent validity.

Methods:  In this cross-sectional study 193 adults with allergic rhinitis and asthma from 15 outpatient clinics in Portugal were included. The CARAT questionnaire was reduced using descriptive analysis, exploratory factor analysis and internal consistency. Spearman’s correlations were used to compare the CARAT scores with a medical evaluation and other measures of control, including the Asthma Control Questionnaire and symptoms’ visual analogue scales. The performance against physician rating of control was summarized using the area under the curve (AUC) from receiver operating characteristic analysis. In addition, CARAT was compared with the physician’s decision to reduce, maintain or increase treatment.

Results:  The reduced version has 10 questions and 2 factors (CARAT10). The Cronbach’s alpha was 0.85. All correlation coefficients of CARAT10 and factors with the different measures of control met the a priori predictions, ranging from 0.58 to 0.79. The AUC was 0.82. For the physician’s decision groups of reduce, maintain or increase treatment, the mean (IC95%) scores of CARAT10 were 24 (21.4;26.6), 21 (19.4;21.9) and 15 (13.6;16.5), respectively.

Conclusion:  CARAT10 has high internal consistency and good concurrent validity, making it useful to compare groups in clinical studies.

Abbreviations
ACQ5

Asthma Control Questionnaire

ACSS

Asthma Control Scoring System

ACT

Asthma Control Test

ARA

allergic rhinitis and asthma

ARIA

Allergic Rhinitis Impact on Asthma guidelines

ATAQ

Asthma Therapy Assessment Questionnaire

AUC

area under the curve

CARAT

Control of Allergic Rhinitis and Asthma Test

CARAT10

10-question version of the Control of Allergic Rhinitis and Asthma Test

CARAT17

17-question version of the Control of Allergic Rhinitis and Asthma Test

EQ-5D

EuroQol Questionnaire

ROC

receiver operating characteristic

VAS

visual analogue scale

Allergic rhinitis and asthma (ARA) are prevalent allergic respiratory diseases that represent significant losses in the patient’s quality of life and work/school performance, as well as an important socioeconomic burden. The Allergic Rhinitis Impact on Asthma guidelines (ARIA) guidelines recommend the optimal control of these diseases as the primary goal of their treatment (1, 2). A combined approach to the management of upper and lower airways disease has been extensively proposed (1, 3, 4).

Several tools have been developed to assess the patients’ quality of life and symptoms’ severity for asthma and for rhinitis, but these questionnaires rarely assess both diseases simultaneously (5–9). More recently, asthma control questionnaires have also been proposed as an important tool to improve asthma outcomes (10–12). However, a tool for the concurrent assessment of ARA control was not available. This has led us to the development of the Control of Allergic Rhinitis and Asthma Test (CARAT) (13).

CARAT is a self-administered questionnaire to measure the degree of control of ARA in adult patients with a previous diagnosis of these diseases.

A high quality methodological approach to the development of a questionnaire is needed for accuracy and usefulness of the tool. The CARAT project has three phases: (i) the construction of an ARA control assessment tool, (ii) a cross-sectional study to reduce the number of questions and evaluate the questionnaire’s internal consistency, factor structure and concurrent validity and (iii) a longitudinal study to assess its reproducibility, predictive validity and responsiveness. The first phase of the CARAT project included an item generation process based on related literature, followed by a web-supported consensus process and a feasibility study. It generated a 17-question version of CARAT (CARAT17), which must be evaluated to remove statistically redundant questions. A shorter version would also reduce the completion time allowing the use of CARAT in a wider range of clinical settings. Furthermore, the evaluation of the questionnaire’s psychometric properties is necessary.

We aimed at describing the second phase of CARAT project, in which we establish the final version of the CARAT questionnaire by reducing the number of questions through an analytical process and evaluate its cross-sectional internal consistency and validity.

Methods

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References

Study design and setting

This cross-sectional item-reduction study was conducted in 15 allergy or respiratory outpatient clinics, mostly hospital based, from the Portuguese regions of Norte, Centro, Lisboa, Alentejo and Açores. At each centre, several physicians could participate and were asked to include at least eight patients each. The study was conducted in the last trimester of 2008.

Participants

To ensure a broad usage of the questionnaire, all patients between 18 and 70 years of age, with a medical diagnosis of asthma and allergic rhinitis and at least 6 months of follow-up at the clinic, were eligible.

The study was approved by the local ethics committee, and each patient gave his/her written informed consent.

Data collection

In addition to CARAT17, data collection included instruments to assess control and quality of life, as well as a medical evaluation. Regarding control, patients answered the Asthma Control Questionnaire (ACQ5) (10) and three visual analogue scales (VAS) concerning the presence of any airways symptoms, bronchial symptoms and nasal symptoms. Health-related quality of life was assessed by the EuroQol Questionnaire (EQ-5D) (14). The medical evaluation was carried out by the attending physician, who was blinded to the results of the questionnaires. The physician registered the patient’s rhinitis severity and control, asthma severity and control, known allergies and current medication (data not shown), as well as his decision regarding treatment (increase, maintain or decrease treatment).

Statistical analysis

Descriptive statistics

The population was described using standard descriptive statistical techniques. The statistical analysis was carried out using spss 17.0 (SPSS Inc., Chicago, IL, USA). The level of significance was set at P < 0.05.

Item reduction

An exploratory factor analysis using principal components and varimax rotation was performed to identify the number of different factors within the questionnaire and the possible statistical redundancy of questions. We applied standard procedures of exploratory factor analysis with respect to the number of variables used in the analysis and the selection of number of factors. Internal consistency analysis was performed simultaneously and was taken into account for the item reduction. An item was considered redundant if one of the following occurred: responses over 90% in a single category of a variable; considerable cross-loading (>0.300 in more than one factor); low item-total correlation (<0.400); increased Cronbach’s alpha if the item was deleted.

Description of CARAT10

The new version of the questionnaire has 10 questions (CARAT10). The time frame to which it refers is the previous 4 weeks. Seven questions address the frequency of symptoms, one sleep impairment, one activities limitation and the last one addresses the need for more medication (Additional file 1). The response options for all questions are 4-point Likert scales. The first 9 questions were scored from 0 (the response option reflecting complete absence of control) to 3 points. The question regarding medication increase had 3 response options (‘Never’– 3 points, ‘Less than 7 days’– 2 points, ‘More than 7 days’– 0 points) and an option ‘I’m not taking any medication’ which was also attributed 3 points. The questionnaire’s score was the sum of all questions. The range of possible scores for CARAT10 is 0–30, 0 being the complete absence of control.

The scores of CARAT10 were compared among categories of patients’ characteristics, namely gender, age, severity of asthma and of rhinitis, using t-tests or anova where appropriate. For this purpose, age was categorized in quartiles.

Evaluation of CARAT10

We studied the new reduced version’s (CARAT10) discriminative properties – the ability to distinguish people at a single point in time. The internal consistency was assessed with Cronbach’s alpha. The concurrent validity was studied using Spearman’s correlation coefficients between the reduced version or its factors and control assessment instruments and the physician’s assessment. A priori predictions for the correlation coefficients of the new version with the other control measures were as follows: (i) 0.6–0.8 with ACQ5; (ii) 0.6–0.8 with the symptoms VAS; (iii) 0.4–0.6 with the physician’s assessment. Scatter plots were used to present these correlations.

Receiver operating characteristic (ROC) analyses were conducted to evaluate the performance of CARAT10 against physician rating of control. For this, we derived dichotomous variables from the physician ratings of control as described by Nathan et al. (11). ROC curves for CARAT10 and for each of factors of the questionnaire were plotted, and the areas under the curves (AUC) were computed.

The CARAT10 scores in the three groups of physician’s treatment decision (reduce, maintain or increase treatment) were compared using the Kruskal–Wallis test. An error bar graph was used to present this comparison.

Results

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References

This study included 193 patients, assessed by 34 physicians in 15 outpatient clinics. The sample’s characteristics are described in Table 1.

Table 1.   Description of the patients included (n = 193). *0–10, 0 being absence of control; #1–10, 1 being no disturbance; $0-100, 0 being the worst imaginable health state
  1. ACQ5, Asthma Control Questionnaire; EQ-5D, EuroQol Questionnaire; VAS, visual analogue scale.

Gender n (%) 
 Male62 (32.1)
 Female131 (67.9)
Age mean (SD) years37.5 (13.84)
Physician assessment
 Severity n (%)
  Asthma
   Intermittent36 (20)
   Mild persistent52 (29)
   Moderate persistent81 (45)
   Severe persistent12 (6)
  Rhinitis
   Intermittent44 (27)
   Persistent116 (73)
   Mild81 (51)
   Moderate/severe78 (49)
 Control* median (p-25-p75)
  Asthma7 (4–8)
  Rhinitis8 (5–9)
 Treatment decision n (%)
  Reduce11 (6)
  Maintain107 (58)
  Increase67 (36)
ACQ5 score median (p25-p75)0.8 (0.2–1.75)
  <0.5 n (%)70 (36)
  0.5–1.5 n (%)67 (35)
  >1.5 n (%)55 (29)
VAS# median (p25-p75)
  All symptoms4 (2–6)
  Bronchial/pulmonary symptoms4 (2–7)
  Nasal symptoms3 (2–6)
EQ-5D %with any problem
  Mobility13
  Self-care   2.1
  Usual activities  23.0
  Pain/discomfort  37.7
  Anxiety/depression  39.8
EQ-5D VAS$ median (p25-p75)75 (60–90)

Overall, 0.58% of all questions were not answered. The items with most missings were ‘Nasal pruritus’ and ‘Throat symptoms’ (4 missings each – 2.1%).

The item reduction began with the elimination of the item ‘Hospitalization’, a dichotomic variable that had 99% of answer ‘No’. Then, using exploratory factor analysis and internal consistency analysis, a 10-question version of the questionnaire (CARAT10) was obtained. It has 2 factors – one with 4 items, the other with 6 items (Table 2).

Table 2.   Factor analysis of items in the initial* and final solutions. Loading to a factor of each itemis represented in bold and grey background, and the items with cross-loading are represented in bold italic Thumbnail image of

CARAT10 had a mean (standard deviation) score of 18.8(6.85), with a minimum score of 2 (n = 2) and a maximum score of 30 (n = 5).

In the internal consistency analysis, the Cronbach’s alpha was 0.89 for CARAT17 and 0.85 for CARAT10, with 0.76 for Factor 1 and 0.86 for Factor 2.

The scores of CARAT10 among categories of gender, severity of asthma and severity of rhinitis were different; while for age categorized in quartiles, the scores were not statistically different (Table 3).

Table 3.   Comparison of CARAT10 scores according to different patients’ characteristics
 CARAT 10P
Mean (sd)Min–Max
Gender0.02
 Male20.4 (6.27)2.0–30.0 
 Female18.0 (7.0)2.0–30.0
Age in years (quartiles)0.39
 <2719.0 (5.0)10.0–28.0 
 27–3518.8 (6.5)3.0–30.0 
 36–4719.9 (7.5)2.9–30.0 
 >4717.5 (7.8)2.0–30.0 
Physician assessment 
 Severity
  Asthma<0.001
   intermittent22.8 (4.5)11.0–30.0 
   mild persistent18.9 (6.9)2.0–29.0
   moderate persistent17.5 (7.1)3.0–30.0
   severe persistent15.6 (6.9)2.0–27.0
  Rhinitis
   intermittent21.7 (5,8)9.0–30.00.004
   persistent18.1 (7.2)2.0–30.0 
   mild19.8 (6.0)2.0–30.00.006
   moderate/severe16.9 (7.5)2.0–30.0 

All the correlation coefficients of CARAT10 and its factors with ACQ5, symptoms VAS and the physician’s assessment met the a priori predictions (Table 4, Fig. 1) and were statistically significant (P < 0.001). Also significant was the correlation between CARAT10 and the EQ-5D VAS, with a Spearman correlation coefficient of 0.48.

Table 4.   Spearman’s correlations of CARAT10, its factors and external measures of asthma and allergic rhinitis control. A priori predictions for the correlation coefficients of the new version with the other control measures were as follows: 1) 0.6–0.8 with ACQ5; 2) 0.6–0.8 with the symptoms VAS; 3) 0.4–0.6 with the physician’s assessment. The correlations between the measures of similar domain are shown in bold
 ACQ5 scoreVisual analogue scalePhysician assessment
All symptomsBronchial/pulmonary symptomsNasal symptomsAsthma controlRhinitis control
  1. ACQ5, Asthma Control Questionnaire.

CARAT10−0.710.69−0.69−0.610.570.52
Factor 1 (rhinitis related)−0.38−0.53−0.450.670.320.58
Factor 2 (asthma related)0.79−0.620.70−0.420.620.36
image

Figure 1.  Scatter plots showing the correlations between CARAT10 and its factors with the all symptoms’ visual analogue scale (VAS), the nasal symptoms’ VAS, the bronchial/pulmonary symptoms’ VAS and Asthma Control Questionnaire (ACQ5).

Download figure to PowerPoint

Receiver operating characteristic curves of CARAT10 score and its factors are shown in Fig. 2. The areas under the curve for CARAT10, factor 1 and 2 are, respectively, 0.82, 0.80 and 0.82.

image

Figure 2.  Receiver operating characteristic curves of CARAT10 score against physician rating of control (A), factor 1 against physician rating of rhinitis control (B) and factor 2 against physician rating of asthma control (C). The areas under the curves are, respectively, 0.82, 0.80 and 0.82.

Download figure to PowerPoint

A statistically significant difference (P < 0.001) was found in the CARAT10 scoring in the 3 groups of the physician’s treatment decision (reduce, maintain or increase treatment). The comparison is shown in Fig. 3.

image

Figure 3.  Error bar comparing the scoring of CARAT10 in the three groups of the physician’s treatment decision. The comparison of the 3 groups was statistically significant using the Kruskal–Wallis test (P < 0.001).

Download figure to PowerPoint

Discussion

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References

This article describes the development and evaluation of a short 10-question version of the Control of Allergic Rhinitis and Asthma Test (CARAT10). It is the second phase of a project that aims to the creation and validation of a tool to simultaneously assess the control of both diseases. A series of methodological steps have been followed to assure the final quality of the questionnaire (15).

The reduction in the initial version of the tool (CARAT17) was necessary to remove statistical redundancy which would affect the scoring and interpretation of the questionnaire, maintaining good validity and reliability. In the exploratory factor analysis, two independent factors were extracted from the final reduced version, and the items’ allocation matched exactly the initially proposed theoretical allergic rhinitis and asthma subdomains (Table 4).

The new version of the questionnaire was shown to have good discriminative properties – internal consistency and concurrent validity. CARAT10’s internal consistency was very similar to that of CARAT17, supporting the item reduction process. The CARAT10 internal consistency was satisfactory (16) for the questionnaire and for each of the subdomains. Moreover, it has similar or better internal consistency than questionnaires used for assessing asthma control, which ranges between 0.72 and 0.84 (11, 17). This level of internal consistency seems to support the concept of a questionnaire assessing the control of allergic rhinitis and asthma simultaneously (1, 18).

For the evaluation of the CARAT10’s concurrent validity and considering the lack of a gold standard, we used different ‘external measures’ that take into account both the patients’ and physicians’ views. All correlations met the a priori predictions and are similar or better than those of widely used asthma questionnaires (10, 11, 17). Symptoms VAS were recently proposed as an adequate instrument to measure the severity of allergic rhinitis (19). A good correlation was observed between the CARAT10 allergic rhinitis subdomain and a nasal symptoms VAS. The correlation between CARAT10 asthma subdomain and ACQ5 was similar to that of previously observed between the Asthma Control Scoring System (ACSS) and ACQ5 (17) and between the Asthma Control Test (ACT) and the Asthma Therapy Assessment Questionnaire (ATAQ) (20). Correlation with the physicians’ assessment was as good as could be expected and greater than that of ACT (11). In a similar manner, the AUC that summarizes the performance against the physician rating of control was better for CARAT10 when compared with ACT published results (0.82 and 0.77, respectively).

A limitation of this study is the lack of comparison with objective tests such as lung function. Nevertheless, several ‘external measures’ were used in the evaluation of concurrent validity. Moreover, longitudinal validation studies to assess CARAT10’s evaluative properties, to establish a scoring system and the minimal clinical important difference are necessary. These studies are currently in progress. As with any questionnaire, validation will be required for the use of CARAT10 in specific populations and settings (21).

In summary, this study, with more than 30 physicians and a large sample of patients with ARA from different clinical settings, showed that CARAT10 questionnaire has high internal consistency and construct validity, making it useful to compare groups in clinical studies.

Acknowledgments

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References

We thank the Portuguese Allergology and Clinical Immunology Society, the Portuguese Pneumology Society and the Portuguese Asthma Patients Association for their collaboration in the project, We thank the physicians participating in the data collection: Dr. Carlos Mendonça, Dr. Carlos Neto, Dr. Celso Pereira, Dr. José Ferraz Oliveira, Dr. Francisca Carvalho, Dr. José Ferreira, Dr. José Moreira Silva, Prof. Luís Borrego, Dr. Mário Miranda, Dr. Paulo Santana, Dr. Rui Silva, Dra. Ana Arrobas, Dra. Ana Romeira, Dra. Ângela Gaspar, Dra. Arminda Guilherme, Dra. Cristina Lopes Abreu, Dra. Elisa Pedro, Dra. Emília Faria, Dra. Eunice Castro, Dra. Filomena Neves, Dra. Maria Graça Castel-Branco, Dra. Inês Lopes, Dra. Isabel Rosmaninho, Dra. Joana Caiado, Dra. Marianela Vaz, Dra. Paula Pinto, Dra. Sara Prates, Dra. Susana Oliveira, Prof. Luis Delgado, Prof. Doutor Bugalho Almeida, Prof. Lia Fernandes, Prof. Luís Barata. We thank the Directors of the centers for making possible the data collection and the patients for their time and help. Supported by an unrestricted grant from Merck Sharp & Dohme Portugal.

Authors contributions

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References

JAF is responsible for the CARAT project and participated in all stages and tasks, LNS participated in the analysis and collaborated in the process of writing the manuscript draft, ASS participated in data collection and analysis and collaborated in the process of writing the manuscript draft, MMA participated in study design and reviewed the manuscript; LFA participated in study design, led the data analysis and reviewed the manuscript, LF, MBF and JB provided critical review during the project and reviewed the manuscript.

References

  1. Top of page
  2. Abstract
  3. Methods
  4. Results
  5. Discussion
  6. Acknowledgments
  7. Authors contributions
  8. Conflicts of interest
  9. References