The assessment of rheumatoid arthritis and the acceptability of self-report questionnaires in clinical practice

Authors


Abstract

Objective

To assess the acceptability of self-report questionnaires (SRQ) in clinical practice and to understand the value that rheumatologists give to various assessment methods in rheumatoid arthritis.

Methods

Rheumatologists who completed a training course in the use of SRQ in clinical trials and clinical practice used the SRQ in their practices. Six months later 221 rheumatologists completed a survey regarding their experiences in assessing rheumatoid arthritis and in the use of SRQ.

Results

Prior to the start of the program, 18% of rheumatologists used self-report questionnaires, 6 months later, 48% were using SRQ in their practices. Rheumatologists who did not use questionnaires placed less value on all assessment methods, and particularly on questionnaire assessments of function and pain. They also were more likely to report that questionnaires were difficult to use, not accepted by staff, were too long, and that they had limited staff. Rheumatologists who used the questionnaires reported none of these difficulties and were satisfied with the benefits provide by SRQ. When assessment measures were ranked, rheumatologist ranked ACR 20, radiography, and erythrocyte sedimentation rate/c-reactive protein as the least important in both clinical trials and in clinical practice, and they rated swollen and tender joint counts followed by SRQ as the most useful assessment tools.

Conclusion

SRQ are well received by rheumatologists, and following a training program almost 50% continued to use SRQ in their practices. Those who used questionnaires were generally more positive about assessments and had little difficulty in the technical aspects of administration, scoring, and interpretation.

INTRODUCTION

Although patient self-report questionnaires play a central role in all randomized clinical trials (RCTs) and observational studies, published reports indicate that they are used infrequently in clinical practice (1). A number of reasons have been suggested for the limited use, including insufficient exposure to questionnaires in training and practice, difficulty in interpretation, difficulty in administration and scoring, and a lack of information considered useful by the rheumatologist.

It has also been suggested that questionnaires do not provide a useful benefit to clinicians. One previous study provided rheumatologists with the results of the Arthritis Impact Measurement Scales (AIMS) (2, 3) at 3-month intervals over a period of one year (4). Clinicians generally reacted favorably to the information, but no changes in clinical practice or other measurable benefits were discernable. It was suggested that “The failure to demonstrate objective benefits of health status reports in this study may be due to physician unfamiliarity with health status scores [or] failure to link the report with an office visit … ” However, the design of this study differed significantly from that of usual clinical practice.

Surprisingly, in reviewing the literature of clinical measures and questions, we found no data regarding the value that clinicians attribute to assessment measures, including measures such as joint counts, laboratory tests, questionnaires, radiographs, and American College of Rheumatology (ACR) improvement criteria (5). In 1999, with the support of Aventis Pharmaceuticals, we initiated a training program for rheumatologists that was followed by an opportunity to use questionnaires in an actual clinical setting. At the conclusion of the program we reviewed the experience of the rheumatologist participants and asked the following questions in the context of questionnaire use in clinical practice for rheumatoid arthritis (RA): 1) How do rheumatologists rate assessment methods, including self-report questionnaires? 2) Is there a difference in the perceived value of assessment methods in RCTs compared with clinical practice? 3) Will rheumatologists use questionnaires? 4) Is there a difference in the rating of usefulness for various assessments for those using questionnaires in practice compared to those not using questionnaires? 5) In addition to ‘usefulness,’ are there other reasons for the difference between users and non-users of questionnaires?

METHODS

In 1999 and 2000, four meetings, sponsored by Aventis Pharmaceuticals, were held for rheumatologists in different locations in the US. All US rheumatologists on the Aventis list of rheumatologists were invited, and invitations were not related to participation in clinical trials. There was a goal of attracting approximately 100 rheumatologists to each of the four meetings. The purpose of the meetings was to provide information about self-report questionnaire assessments in RA in the context of clinical trials, observational studies, and clinical practice. In addition to lectures, participants were educated on the psychometric aspects of questionnaires, and on how to use questionnaires in practice, including information on administration, scoring, and interpretation. Finally, participants were provided with questionnaire forms to be used in their practices. The self-report questionnaire administered to patients was a subset of questions from the CLINHAQ (6, 7), including the Health Assessment Questionnaire (HAQ) (8), and visual analog scales (VAS) for patient global, pain, fatigue, and gastrointestinal symptom severity. The CLINHAQ questionnaire can be found at www.arthritis-research.org.

Physician participants were urged to use the questionnaires in their practices and either to score the questionnaires themselves or use an automated scoring system. The automated system was made available in which questionnaires completed by patients could be faxed to the study coordinating center (National Data Bank for Rheumatic Diseases, Wichita, KS) where they were scored electronically. The results were faxed back to the physicians and were presented in numeric and graphic formats, with interpretative data included.

Six months after the completion of the program, the participants were surveyed regarding their experience using the questionnaires and aspects of the assessment of RA. In the followup questionnaire, we asked the physicians about the value they gave to clinical, laboratory, radiographic, self-report questionnaires, and ACR improvement criteria (5) using 4- and 5-point Likert scales (9).

Statistical methods.

Data were analyzed using Stata statistical software version 7.0 (10). In various analyses physicians were grouped according to previous responses. Differences between two groups of physicians for categorical variables was assessed by the Kornbrot test, which tests the equality of distributions for matched pairs of observations where the data are ordinal, using the Wilcoxon matched-pairs signed-ranks test. Ordered logistic regression was employed when 3 groups of physicians were evaluated.

RESULTS

Three hundred eight-six rheumatologists participated in the educational programs, and 221 (57.3%) completed the 6-month followup evaluation. Responders and nonresponders (physicians who did not complete the followup evaluation) did not differ by sex (chi-square 0.074, P = 0.785), but responders were slightly older than nonresponders (51.1 years versus 48.8 years), t = 2.34, P = 0.020. The response rate for the 6.6% of rheumatologists over age 65 did not differ from the younger rheumatologists (χ2 = 0.586, P = 0.440). Thirteen percent of physicians were university based, but the response rates did not differ according to whether physicians were in a solo, group, health maintenance organization, university, or government practice (χ2 [5df] = 2.184, P = 0.823). Subsequent analyses were confined to the 221 rheumatologists who answered questions on the followup questionnaire.

Table 1 describes the intent of rheumatologists to use questionnaires in the 6-month period following the educational course. Five percent did not plan to use a questionnaire, 51% planned to consider using the questionnaire, 43% indicated they would use a questionnaire (including 18% already using questionnaires), and 1% did not answer the question. In Table 2 we combined the three positive categories of Table 1 (planned to definitely use questionnaires or already using questionnaires in 10% or 50% of their patients) into a “definitely will use” category for ease of interpretation. Table 2 details the actual use of questionnaires by the physicians. In subsequent analyses we dropped the “other” category for simplicity of interpretation. Overall, 43% of responding rheumatologists reported using questionnaires in their practices at the followup assessment (Table 2) compared with 18% prior to the educational program (Table 1).

Table 1. Intent of 221 rheumatologist to use questionnaires in practice*
CategoryFrequency (%)
  • *

    Assessment at initial educational session.

Did not expect to use a questionnaire11 (5.0)
Planned to consider using a questionnaire113 (51.1)
Planned to definitely use a questionnaire55 (24.9)
Already using in more than 10% of patients19 (8.6)
Already using in more than 50% of patients21 (9.5)
No response2 (0.9)
Table 2. Actual use of questionnaires compared with intention to use
Prior usage plansCategories of use*
Never usedStarted and stoppedUsingOtherTotal
  • *

    As assessed ∼6 months after educational program.

Not to use721111
 63.6%18.2%9.1%9.1%100.0%
Consider using4634294113
 40.7%30.1%25.7%3.5%100.0%
Definitely will use111564595
 11.6%15.8%67.4%5.3%100.0%
Total64519410219
 29.2%23.3%42.9%4.6%100.0%

Valuation of assessment methods: comparison between actual users and non-users of questionnaires as to their value in clinical practice and randomized controlled trials.

Rheumatologists who used questionnaires (column 4 of Table 2) valued questionnaire and non-questionnaire assessment methods more highly than the physicians who never used questionnaires or who started using questionnaires then stopped using them (columns 2 and 3 of Table 2) (in RCTs: MANOVA F = 1.6293 P = 0.038 and in clinical practice: multivariate analysis of covariance F = 4.174 P < 0.000) for the items in Table 3. This difference between physician groups was stronger for clinical practice than RCTs.

Table 3. Rating of rheumatoid arthritis assessment methods in randomized trials and clinical practice by rheumatologists who do and do not use questionnaires*
 Use questionnairesDon't use questionnairesAll rheumatologists
  • *

    Ratings are mean scores for each item for the question, “Please rate the importance of these measures.” Codes are 1 = rarely important, 2 = sometimes important, 3 = usually important, 4 = almost always important. ACR = American College of Rheumatology; ESR/CRP = erythrocyte sedimentation rate/c-reactive protein.

  • Indicates that use and non-use groups differ significantly at P < 0.05.

Randomized clinical trials   
 Swollen joint count3.53.43.5
 Tender joint count3.43.33.4
 Patient global3.53.33.3
 Functional status questionnaires3.53.13.3
 Pain scale3.43.13.2
 Physician global3.33.23.2
 ACR703.23.13.2
 ACR503.23.13.1
 ESR/CRP3.03.03.0
 Radiography3.12.93.0
 ACR203.02.82.9
Patient clinical care   
 Swollen joint count3.33.43.4
 Tender joint count3.23.33.3
 Patient global3.23.33.2
 Functional status questionnaires3.33.13.2
 Pain scale3.32.62.9
 Physician global2.82.72.7
 ACR703.32.22.7
 ACR502.52.52.5
 ESR/CRP2.22.12.2
 Radiography2.12.02.1
 ACR202.01.91.9

We next examined differences between ratings for the same assessment method in RCTs and clinical care for each item of Table 3 by computing difference scores between RCTs and clinical care assessment scores, and then applying ordered logistic regression to the difference. In all instances there was no difference between the group that had never used questionnaires and those who said they used them and stopped. Therefore these groups were combined into a single “non-use” for easier interpretation, as shown in Table 3. Table 3 demonstrates that the physician groups differed only in their assessment of the value of patient rating of pain and patient self-report of function, with questionnaire users rating these assessment methods more highly.

Comparative valuation of assessment methods.

Column 3 of Table 3 presents the relative ranking of the assessment methods for all physicians. The relative rankings were the same for RCTs as for clinical care, although the assessments were rated to be slightly more important in RCTs. Of interest, rheumatologists rated the ACR20 as the least useful of all measures. But they also ranked erythrocyte sedimentation rate/c-reactive protein (ESR/CRP) and radiography at the least useful end of the spectrum in RCTs as well as in clinical care.

At the upper end of the rankings, swollen and tender joint counts were thought important by both groups of physician, but questionnaire users rated self-report items as considerably more important than did non-questionnaire users. Within the context of clinic care, questionnaire users ranked functional status questionnaires and pain scales as first and second among the rankings.

Problems and benefits in the use of questionnaires.

Participant rheumatologists were also asked to rate their experience with questionnaires in the practice setting. As with their overall valuation of questionnaire items, non-users and users of questionnaires differed in their appraisal of questionnaires in actual practice. As shown in Figure 1, in comparison to users, non-users found the questionnaires hard to use, indicating that they were too long and were not accepted by staff. In addition, they indicated that they did not have staff to administer the questionnaires. Although non-users attitudes were neutral to in disagreement that questionnaires were not as good as other measures, were hard to interpret, hard to score, hard for the patient to complete, of little benefit, and were not taught to them, in all of these respects they differed substantially from users who disagreed with these statements.

Figure 1.

Assessment of self-report questionnaires by 221 rheumatologists 6 months after completion of a training program and use of self-report questionnaires in clinical practice. Too long = questionnaire too long; No staff = no staff to help; Not accepted = not accepted by staff; Hard to use = difficult to use; Not as good = objective tests are better; Hard interpret = difficult to interpret; Hard to score = difficult to score; Hard patient = not accepted by patient; No benefit = of no benefit; Not taught = no opportunity to learn about questionnaires.

DISCUSSION

The results of this study provide some insight into the value rheumatologists place on assessment methods in rheumatoid arthritis. Although the ACR20 criterion has become the standard of assessing treatment efficacy, rheumatologists rated it as the least effective of all measures in interpreting the results of clinical trials. Joint examinations, on the other hand, were rated as the most effective measures by the combined rheumatologists group. These impressions, however, are contrary to published data regard the ability of the various tests to distinguish active treatment from placebo (5, 11). In addition, when individual tests are considered in scientific studies, questionnaire data are the most effective measures in identifying treatment effect (11, 12).

How then is it possible for the 221 rheumatologists to arrive at their assessment? A possible interpretation is the ACR20 criterion is a weak determinant of efficacy, and one that clinicians may believe is clinically unimportant (13). The ACR20 is also a multidimensional measure that is not easily interpretable in a common clinical metric. In addition, the ACR20 is a measure of change rather than patient status, such that patients may improve but still have active, uncontrolled disease; therefore the measure may be perceived as unsatisfactory and unsatisfying. The joint count measures, by contrast, are recognizable to clinicians, representing an understandable metric that is part of their daily professional life.

Questionnaire measures provide a different perspective. Rheumatologists who did not use or continue to use questionnaires rated them as being less useful than did questionnaire users. Since self-report questionnaires are among the most useful of measures in RCTs, a fact that was not recognized by the non-users, it seems probable that education in rheumatology training programs and in the medical literature has not kept up with the scientific research.

It is also of interest that laboratory tests and radiographs were not found to be as important as the other measures in RCTs as well as in clinical care. The reason for this is not entirely clear. Radiography plays a small part in clinical care, and although it may be an important marker for distant patient outcome, it has little to do with day-to-day care of the RA patient. Laboratory tests are helpful measures in the assessment of disease activity and the prediction of long-term outcomes. Perhaps the fact that for most rheumatologists laboratory test results are not available at the time of the patient encounter may play a role in this relative down-ranking of the ESR and CRP.

When it comes to the use of patient questionnaires in clinical practice, those who did not use questionnaires had a lower opinion of their value. But this group also indicated considerably more difficulty in the use of questionnaires. As shown in Figure 1, they found them more difficult to use and to be accepted in their clinics. No one to help in administration and/or scoring of the questionnaires was another attributed problem. Although the questionnaires were short, comprised of the HAQ and a few VAS scales, non-users often found the questionnaires to be too long. By contrast, users did not find difficulties, and thought questionnaires were helpful in their practices.

Studies such as this that examine attitudes do not provide firm answers. It is not known whether non-users of questionnaires found them difficult to use because they were uncomfortable with patients' self-report information or because of intrinsic physical and staff difficulties in their clinics. It has been observed that such problems can be overcome easily if rheumatologists support questionnaire use and find them valuable (1). We suggest that some of the misconceptions regarding the use of questionnaires in RCTs can be overcome by further education, perhaps as evidenced by the increase in use among rheumatologists in this study from the 18% prior to the educational program to 43% at the end of the assessment period. It is also likely that support from rheumatology training programs and from educational programs that familiarize clinicians with the meaning and interpretation of questionnaires will go a long way in reversing some of the misinformation that surrounds them; and it is particularly important that rheumatologists in training have continuous exposure to these clinical assessment tools.

Ancillary