Agreement between patient report and medical record review for medications used for rheumatoid arthritis: The accuracy of self-reported medication information in patient registries

Authors


Abstract

Objective

With the growth in patient registries in rheumatic disease research, it is important to validate the collected information. We examined the convergent validity of self-reported medication use for rheumatoid arthritis (RA).

Methods

In the setting of the Brigham Rheumatoid Arthritis Sequential Study (BRASS), a large registry of patients with RA, we examined the agreement between patients' self-report of current and past RA medication use and information from medical records. For a sample of patients in BRASS, these 2 sources of information were compared using the kappa statistic as well as the percent agreement.

Results

The 91 patients selected for assessment were typical of a prevalent RA cohort: >80% were women and the mean disease duration was 16 years. The agreement for current medication use was excellent, ranging from 0.71 for sulfasalazine to 0.96 for methotrexate. However, for past medication use agreement was lower, ranging from 0.13 for methotrexate to 0.74 for aurothioglucose. The weighted kappa for cumulative oral glucocorticoid dose was 0.67.

Conclusion

Self-report of current medication use and cumulative oral glucocorticoid dose appears to have moderate to excellent validity. However, self-report of past medication use may not be valid.

INTRODUCTION

Patient registries are flourishing in the rheumatic diseases. Registries can facilitate collection of comprehensive longitudinal data on carefully defined cohorts of patients and were developed to study diseases such as rheumatoid arthritis (RA), systemic lupus erythematosus, and osteoarthritis (1–3). Many, but not all, registries rely on patient self-report for much of the data. Although patient self-report has been reported by several groups to be an excellent source of information for pain and function (4, 5), others have noted the poor accuracy of self-reported diagnosis (6), and the accuracy of patients' report of medication use is less clear. Studies from other areas of medicine suggest that some patients do not accurately report long-term use of blood pressure–lowering treatments as well as many other prolonged treatments (4–8). To our knowledge, there are no prior studies examining the validity of self-reported medication use for RA. The validity of self-report medication questionnaires in the rheumatic diseases must be examined before there is widespread adoption of this method.

We conducted a study assessing the agreement between self-report and medical records for RA treatments, including disease-modifying antirheumatic drugs (DMARDs) and oral glucocorticoids in patients enrolled in an RA registry. Agreement between these 2 sources of similar information allows one to examine their convergent validity, but not the criterion validity because neither source can be considered the gold standard for medication use. Several better data sources of medication use, such as pharmacy claims or pill bottle cap diaries, were not available to the investigators.

METHODS

Study protocol.

The study population consisted of patients with RA who received their care at Brigham and Women's Hospital. All eligible patients were enrolled in BRASS (Brigham Rheumatoid Arthritis Sequential Study), a longitudinal RA patient registry. At the time that we initiated BRASS, 583 patients had been enrolled. Every 6 months, patients in BRASS complete questionnaires that include items regarding functional status, quality of life, mental health, fatigue, comorbid conditions, medications for RA and other conditions, and resource utilization (9). The baseline questionnaire asks about the patient's past RA medications, duration of use, and reasons for stopping. A trained research assistant helped patients answer questions on the baseline questionnaire.

Because of our interest in the agreement between self-reported medication use and the medical record, we restricted the current analyses to patients who had received care for the duration of their RA at Brigham and Women's Hospital. Thus, patients had to have been seen at Brigham and Women's Hospital within 2 years of their original RA diagnosis, reducing the eligible sample size to 102 patients. Eleven patients had incomplete records (with obvious gaps in records) and were excluded from this analysis, leaving a final sample size of 91.

All of the patients in BRASS gave written informed consent to have their medical records reviewed. The current protocol was reviewed by the Partners Healthcare Institutional Review Board.

Data collection.

We used 2 sources of data for the present study: the BRASS baseline patient questionnaire and the medical record. First, we abstracted RA medication information from the BRASS questionnaire as noted above. These data included both current and past use of DMARDs and oral glucocorticoids. For past DMARDs, patients were asked about the duration of use and reasons for stopping. For the current medications, patients were asked about the duration of use and current dose. Detailed questions about oral glucocorticoid use were included, such as duration of use in months and average dose in milligrams over the duration of RA. In addition, standard RA instruments were completed including the Disease Activity Score in 28 joints (DAS28), an index of disease activity, and the Multidimensional Health Assessment Questionnaire (MDHAQ), a functional status scale (9, 10).

The second source of information was the outpatient Brigham and Women's Hospital medical record, which is kept primarily in electronic format. The electronic medical record includes physicians' notes and a medication list, and both were consulted during data collection. The medical record was used as the primary source of information when it specifically mentioned starting or stopping a medication of interest. In other instances, physicians' notes stated that the medication list should be consulted for a patient's current treatments, and we did so in such cases.

A research assistant (AL) reviewed the electronic medical record of each patient. This research assistant was trained by a board-certified rheumatologist (DHS), and 20 charts were reviewed by both the research assistant and the rheumatologist. The interrater reliability (kappa) was 0.96. Each rheumatology visit note was read to find the following information: current RA medications, reasons for discontinuing a given medication, and duration of use. The duration was calculated by examining physicians' notes to determine the start and stop dates. To calculate the cumulative oral glucocorticoid dose, we created a monthly calendar with doses and then summed the monthly doses.

Statistical analysis.

We first described the population using means and proportions. Then, the kappa statistic was calculated using standard methods (11). The kappa statistic cannot be accurately calculated when there are zero observations in a given table cell. In such cases, the percent agreement was also calculated by dividing the sum of the observations that agreed by the total observations. Furthermore, we calculated the prevalence-adjusted and bias-adjusted kappa to assess the effect of the imbalance in observations (12). The agreement for cumulative oral glucocorticoid dose was calculated by quartile of dose. Because 4 categories were used, we calculated a weighted kappa, where greater weight is given to cells on or closer to the diagonal of agreement. In addition, we graphically examined the correlation between patient report and medical record of cumulative glucocorticoid dose. All analyses were performed using SAS software, version 9 (SAS Institute, Cary, NC).

RESULTS

The characteristics of the 91 patients included in the analysis are shown in Table 1. The majority of the patients were college-educated, older women with an average RA duration of 15.7 years. Two-thirds of patients had positive rheumatoid factor or anti–cyclic citrullinated peptide antibodies. The mean DAS28 scores were moderately high, and the MDHAQ scores were <1.

Table 1. Characteristics of patients from the Brigham Rheumatoid Arthritis Sequential Study cohort selected for medical record review*
CharacteristicValue
  • *

    Values are the number (percentage) unless otherwise indicated. CCP = cyclic citrullinated peptide; DAS28 = Disease Activity Score 28-joint count; MDHAQ = Multidimensional Health Assessment Questionnaire.

Total number91
Female sex76 (83.5)
Age, mean ± SD years57.3 ± 14.3
Highest level of education 
 Did not graduate from high school4 (4.4)
 Graduated high school but not  college24 (26.4)
 Graduated college63 (69.2)
Rheumatoid arthritis duration,  mean ± SD years15.7 ± 13.7
Rheumatoid factor or CCP positive57 (62.6)
DAS28, mean ± SD score4.3 ± 1.5
MDHAQ, mean ± SD score0.5 ± 0.9

Self-reports of current medication use had a moderate to strong agreement with reports in the medical record (Table 2). Most patients reported taking at least 1 DMARD. Kappa statistics were excellent, ranging from 0.71 for sulfasalazine to 0.96 for methotrexate. For the less commonly used medications, the kappa statistic was unreliable because of zero cell counts. For these drugs, it is useful to consider the percent agreement, which was very high, ranging from 97% to 100%.

Table 2. Agreement between self-report and medical record review for current medication use*
MedicationSelf-reportMedical record reviewUnadjusted kappa (95% CI)Percent agreementAdjusted kappa (95% CI)
YesNoYesNo
  • *

    95% CI = 95% confidence interval.

  • The adjusted kappa is both prevalence and bias adjusted (see ref.12).

  • Kappa could not be calculated because of empty cells.

Hydroxychloroquine296230610.92 (0.84, 1.00)970.93 (0.86, 1.00)
Methotrexate444744470.96 (0.90, 1.00)980.96 (0.90, 1.00)
Sulfasalazine4877840.71 (0.40, 1.00)970.93 (0.86, 1.00)
Leflunomide98210810.94 (0.83, 1.00)990.98 (0.94, 1.00)
Etanercept137812790.95 (0.86, 1.00)990.98 (0.94, 1.00)
Infliximab2893880.79 (0.40, 1.00)990.98 (0.94, 1.00)
Anakinra0910911001.00
Adalimumab3883881.001001.00
Cyclosporine0910911001.00
Azathioprine0910911001.00
Aurothioglucose0910911001.00
Auranofin0910911001.00
Penicillamine0910911001.00
Rituximab0910911001.00
Cyclophosphamide0910911001.00

Agreement was considerably less robust for past medication use (Table 3). Kappas were relatively low, ranging from 0.13 for methotrexate to 0.74 for aurothioglucose. For example, 56 patients were noted to have taken methotrexate based on the medical record review, but only 14 self-reported prior use on the questionnaire. Kappa was unreliable in cases of low cell counts, and the percent agreement, which was quite high for many medications, could be considered instead. We also calculated the prevalence-adjusted, bias-adjusted kappas, which were often higher than the unadjusted kappas.

Table 3. Agreement between self-report and medical record review for past use of medications*
MedicationSelf-reportMedical record reviewUnadjusted kappa (95% CI)Percent agreementAdjusted kappa (95% CI)
YesNoYesNo
  • *

    95% CI = 95% confidence interval.

  • The adjusted kappa is both prevalence and bias adjusted (see ref.12).

  • Kappa could not be calculated because of empty cells.

Hydroxychloroquine276458330.35 (0.21, 0.49)640.27 (0.08, 0.47)
Methotrexate147756350.13 (0.02, 0.24)49−0.01 (−0.21, 0.19)
Sulfasalazine147722690.66 (0.47, 0.84)890.78 (0.65, 0.91)
Leflunomide78417740.53 (0.29, 0.77)890.78 (0.65, 0.91)
Etanercept28916750.19 (−0.04, 0.41)850.69 (0.54, 0.84)
Infliximab3886850.65 (0.29, 1.00)970.93 (0.86, 1.00)
Anakinra0910911001.00
Adalimumab1904870.39 (−0.14, 0.92)970.93 (0.86, 1.00)
Cyclosporine190091990.98 (0.94, 1.00)
Azathioprine190091990.98 (0.94, 1.00)
Aurothioglucose4874870.74 (0.39, 1.00)980.96 (0.90, 1.00)
Auranofin190190−0.01 (−0.03, 0.01)980.96 (0.90, 1.00)
Penicillamine0910911001.00
Rituximab289091980.96 (0.90, 1.00)
Cyclophosphamide190091990.98 (0.94, 1.00)

Few patients reported the reasons for discontinuation of past medications used for RA (Table 4). The number of patients whose medical record or self-report provided a duration of use was much smaller than the number who appeared to have used a given medication. Also, the number of patients with a clearly defined reason for discontinuation was much smaller than the number of patients who reported using a medication in the past. No formal tests of agreement were calculated for the data in Table 4.

Table 4. Numbers of patients reporting duration of past medication use and reasons for discontinuation*
MedicationNo. who reported duration of past useNo. who reported reasons for discontinuation
Medical recordSelf-reportMedical recordSelf-report
  • *

    This table represents the number of patients whose medical record or self-reports give a duration of use and/or a reason for discontinuation.

Hydroxychloroquine57251615
Methotrexate571357
Sulfasalazine221399
Leflunomide17742
Etanercept16221
Infliximab6322
Anakinra0000
Adalimumab4111
Cyclosporine0100
Azathioprine0100
Aurothioglucose4402
Auranofin1101
Penicillamine0000
Rituximab0100
Cyclophosphamide0101

To assess the agreement between the cumulative glucocorticoid dose from self-report and medical records, we examined quartiles of dose. The agreement between the quartiles of cumulative glucocorticoid dose from the patient self-report and medical record review was moderate (Table 5), with a weighted kappa of 0.67. The correlation between the actual cumulative doses was assessed graphically (Figure 1). Pearson's correlation coefficient was 0.59 (P < 0.001).

Table 5. Agreement between questionnaire and chart review for quartiles of cumulative glucocorticoid dose*
 Quartile 1Quartile 2Quartile 3Quartile 4
  • *

    Quartiles across the horizontal refer to chart review and quartiles down the vertical refer to self-report. Row percentages sum to 100%. Counts represent the number of patients in each ranked category (total = 91). κ (weighted) = 0.67 (95% confidence interval 0.56, 0.78).

  • Agreement between the 2 data sources.

Quartile 127 (90%)0 (0%)1 (3%)2 (7%)
Quartile 22 (10%)7 (35%)10 (50%)1 (5%)
Quartile 30 (0%)1 (6%)11 (65%)5 (29%)
Quartile 40 (0%)3 (13%)6 (25%)15 (62%)
Figure 1.

Correlation between patient report of cumulative glucocorticoid dose (grams) versus the cumulative dose derived from review of the medical record. Pearson's correlation coefficient was 0.59 (P < 0.001).

DISCUSSION

Patient registries combine substantial clinical and biologic data in a longitudinal record. Much of the self-reported information is collected using measures with well-understood reliability and validity; however, this is not the case for self-reported medication use. We found relatively strong agreement between these 2 data sources for current medication use, but weaker agreement for past use. Duration of past use and reasons for discontinuation were not consistently reported; therefore, we did not attempt to judge their agreement.

It is interesting to contrast the relatively poor agreement for past RA medications with the moderate agreement for glucocorticoids. Most of the lack of agreement for DMARDs stemmed from low patient self-report of prior use. Medical records revealed higher rates of past use than did self-reports for most medications. This finding suggests that many patients cannot recall individual DMARD use. Patients may have better recall of glucocorticoid use because these agents are such a common part of the treatment regimen for RA. In addition, the questionnaire included 3 separate questions for oral glucocorticoid use, prompting better recall: it first asked about any use of oral glucocorticoids, then about duration of use, and finally about typical daily dose. Patients were given answer categories for duration and dose.

Although our findings suggest that patient report of current DMARDs and oral glucocorticoid use has strong convergent validity, i.e., agreement with another source of similar information, we are unable to comment on the criterion validity, agreement with a gold standard (13). Information from pharmacies on prescription filling would be a more accurate source of drug data than the medical record, but we did not have access to this source in our study population (14). Our findings also suggest that self-report of cumulative glucocorticoid dose has moderate convergent validity compared with the medical record. However, self-report of past medication use (which medications, duration of use, and reasons for stopping) should be viewed with skepticism. This seems to stem mostly from a lower frequency of self-report of prior medication use than what we gathered from the medical record.

In our study, we compared 2 common sources of medication information, self-report versus the medical record. Neither of these are the criterion standard for medication use. Many researchers consider electronic pill cap monitors to be a gold standard, but these also can be problematic when patients use weekly pill boxes and not the pill bottle enclosed by an electronic cap (15). Pharmacy claims are a useful source of prescription medication information, but this data source does not capture whether patients actually use their medications, just that they fill prescriptions (14). Self-report of medication use has been shown to correlate with medication adherence as measured by electronic monitors in some settings, but not in others (16, 17). In fact, self-report of antihypertensive medication use has been found to be in very poor agreement with pharmacy claims (7).

There are several limitations to our methods. Our study population included only patients participating in a longitudinal cohort at 1 academic medical center; therefore, the generalizability of our findings is unclear. Similar studies should be conducted in other populations that include a broader range of patients. In addition, we tested only the RA treatment questionnaire that was part of the BRASS baseline questionnaire. It is possible that other medication questionnaires would facilitate a more accurate collection of medication data. We encourage other investigators to test alternative instruments.

Patient registries using self-reported information will provide new and important information about rheumatic diseases. Many of the scales used to collect such data have been validated and provide valid and reliable information. Drug utilization information should be collected using actual records of prescription filling whenever possible. When these data are not available, self-report of current RA treatments appears valid, and reports of cumulative oral glucocorticoid dose may also be useful. Further work comparing self-reported RA treatments with prescription filling records or electronic monitors is required. It would also be useful to broaden studies of the validity of self-reported RA treatment to include more demographically diverse populations.

AUTHOR CONTRIBUTIONS

Dr. Solomon had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study design. Solomon, Licari, Weinblatt, Maher, Shadick.

Acquisition of data. Solomon, Licari, Weinblatt, Maher, Shadick.

Analysis and interpretation of data. Solomon, Stedman, Weinblatt, Shadick.

Manuscript preparation. Solomon, Weinblatt, Shadick.

Statistical analysis. Solomon, Stedman.

Ancillary