Development and validation of an electronic version of the Rhinoconjunctivitis Quality of Life Questionnaire


Professor Elizabeth Juniper MCSP, MSc
20 Marcuse Fields
West Sussex PO18 8NA


Background:  As clinicians and pharmaceutical companies move from paper versions of health status questionnaires to electronic versions, it cannot be assumed that adaptations to other media will produce valid data.

Aims:  The aims of this study were to (1) adapt the Rhinoconjunctivitis Quality of Life Questionnaire [RQLQ(S); standardized version], for the Palm Treo 650, (2) test the device for ease and accuracy of understanding and (3) examine the validity of the electronic version by comparing it with the original paper version of the RQLQ(S).

Methods:  Seventy adults with current rhinoconjunctivitis symptoms completed the electronic and paper versions of the RQLQ(S). They were randomized to complete either the paper or the electronic version first. After a 2-h break, they completed the other version.

Results:  Concordance between paper and electronic versions for the overall RQLQ(S) score was acceptable with an intraclass correlation coefficient of 0.95 and there was no evidence of bias (P = 0.13). Concordance for the seven individual domains ranged from 0.86 to 0.94. A small but significant bias was observed in the activity and sleep domains (P = 0.02). Completion times were quicker with paper (4.1 vs 4.9 min, P < 0.0001). About 51% of patients preferred electronic, 17% preferred paper and 31% had no preference.

Conclusions:  This electronic version of the RQLQ(S) was easy for patients to use and the concordance between paper and this version on the Palm Treo 650 provides evidence of the validity of this electronic version.

The 28-item Rhinoconjunctivitis Quality of Life Questionnaire (RQLQ) was developed to measure the functional problems that patients with rhinoconjunctivitis experience in their daily lives. Both the original RQLQ and the standardized version [RQLQ(S)], in which all activities are generic, have been fully validated. They give concordant data and have similar and strong measurement properties (1, 2).

Previous validation studies have assessed the paper versions of these questionnaires. In this study, the RQLQ(S) was adapted by the PHT Corporation (2 chemin Louis-Herbert, 1213 Petit-Lancy, Geneva, Switzerland) and the developer of the RQLQ(S) (EFJ) for electronic data collection using a hand-held device, the Palm Treo 650. First, we tested the electronic version for ease of use and for accuracy of understanding of the instructions, questions and response options. We then conducted a validation study to examine the concordance (agreement and bias) of patients’ responses between the original paper version of the RQLQ(S) and the electronic version. We also compared completions times, completion rates and need for assistance and we asked patients which method of data collection they preferred.


Development of the electronic version

The PHT Corporation worked closely with the developer's team (EFJ and BAJ) to format the RQLQ(S) for the Palm Treo 650. Individual questions and their response options, which appear together in the paper version, could not be fitted on one screen of this hand-held device and this necessitated changes in question wording and instructions. During this process, care was taken to maintain the conceptual accuracy of the questions and the interval properties of the response option scale. Scrolling up or down was not permitted. Prior to the validation study, the electronic version underwent a cognitive debriefing in 10 patients to ensure ease and accuracy of understanding. During this phase, several improvements were made. The final English version was culturally adapted into Danish by the MAPI Research Institute (Lyon, France) prior to the validation study in Denmark. MAPI had done the original cultural adaptation of the paper version for Denmark.


Seventy Danish-speaking adults (17–65 years) with current symptoms of rhinoconjunctivitis participated in the study. They represented a wide range of rhinoconjunctivitis severity, academic ability, age and gender. Patients were excluded if they had another illness that might have had an impact on health-related quality of life.

Participants attended the clinic on a single day. They were randomized either to complete the paper or electronic version first. There was a 2-h interval between the two completions during which patients carried out a nonmedical activity (e.g. read, TV, etc.). During this interval they did not talk to clinic staff or discuss their clinical condition. After completing both the paper and electronic versions, patients were asked which version they preferred and they were asked to identify any difficulties they experienced with either version. The clinic staff recorded the length of time that it took patients to complete each version and whether any additional help was requested.

Overall and domain scores of the RQLQ(S) for the electronic and papers version were compared using a paired t-test. Concordance between the two methods was examined with an intraclass correlation coefficient (ICC). It was decided a priori that concordance between electronic and paper would be considered acceptable if the ICC for the overall RQLQ(S) score was 0.95 or greater. This is in keeping with the concordance observed in previous validation studies of electronic devices (3) and for test–retest reliability of the paper version (2).


For the overall RQLQ(S) score, the concordance between paper and electronic versions reached an acceptable ICC of 0.95 and there was no evidence of any bias (P = 0.13, Table 1). Similar results were found for the following domains: nonrhinitis symptoms, practical problems, nose symptoms, eye symptoms and emotional function. However, concordance was not quite as good as for the overall score (ICC = 0.94–0.86). In the activities and sleep domains, there was a small but significant bias (P = 0.02) with the differences in mean scores being −0.15 and 0.14, respectively. On average, patients took just under a minute longer to complete the electronic version (Table 2). Fifty-one per cent preferred the electronic version, 17% the paper and 31% had no preference. Completion rates were 100% for electronic and 99.6% for paper. No patients asked for additional help and they identified no problems.

Table 1.   Comparison between electronic and paper version of the Rhinoconjunctivitis Quality of Life Questionnaire [RQLQ(S)]
  Electronic (mean ± SD) Paper (mean ± SD)DifferencePaired t-test (P-value)Pearson correlation coefficientIntraclass correlation coefficient
RQLQ overall2.32 ± 0.892.37 ± 0.94−
Activities3.04 ± 0.923.20 ± 1.00−
Sleep1.91 ± 1.491.77 ± 1.370.140.020.940.93
Nonrhinitis symptoms2.03 ± 1.082.11 ± 1.19−
Practical problems2.90 ± 1.32.99 ± 1.42−
Nasal symptoms2.92 ± 1.162.99 ± 1.22−0.060.320.900.90
Eye symptoms2.39 ± 1.492.45 ± 1.52−0.060.350.940.94
Emotional function1.48 ± 1.151.47 ± 1.21−0.010.880.860.86
Table 2.   Completions rates, times and preferences
  1. *31% had no preference.

Percentage of patients completing questionnaire10099.6>0.05
Completion time (mean ± SD)4.90 ± 1.484.07 ± 1.97<0.001
Percentage preferring each device*5117 


The results of this study suggest that this particular electronic version of the RQLQ(S) has shown adequate concordance between paper and electronic with no evidence of bias for the overall score. However, for individual domains, the concordance is not quite so good and there was a small but significant bias in the sleep and activities domains. This suggests that it would probably be unwise to change between different methods of data collection within a clinical trial and, when used in a clinic, it would be imprudent to compare previous paper results with new electronic results within an individual patient. The reason for the bias going in opposite directions in the two domains is unclear.

With a large screen electronic version of the Asthma Quality of Life Questionnaire, where instructions, questions and their response options could be presented on a single screen, the concordance between paper and electronic versions was higher (ICC = 0.99) and there was no evidence of bias (3). The test–retest reliability of the paper version of the RQLQ(S), with a 1 week time separation between completions, gave an ICC = 0.96 which was slightly better than the present study where only 2-h separated paper and electronic. Therefore, the results of the present study show that different electronic versions of validated paper questionnaires may show different degrees of agreement with the original instrument and they emphasize the importance of ensuring that all new electronic versions of questionnaires undergo a validation study.


We thank Dr Valdo Arnera and his colleagues at the PHT Corporation for their close collaboration and innovation to overcome the difficulties of getting long questions each with complex 7-point responses on to a small hand-held device in a manner that yields valid data, is easy for patients to use and is adaptable to other languages. We also thank Dr Torben Balchen at Cyncron in Copenhagen, Denmark for conducting the validation study.