Validity of self-reported rheumatoid arthritis in a large cohort: Results from the Black Women's Health Study

Authors


Abstract

Objective

To evaluate the positive predictive value (PPV) of 3 case definitions of rheumatoid arthritis (RA) based on self-reported data on RA diagnosis and use of arthritis medications, and to determine whether a validated screening survey would increase the PPVs in the 3 groups.

Methods

Medical records and physician checklists were reviewed for confirmation of an RA diagnosis among a sample of Black Women's Health Study participants who reported incident RA and were categorized according to reported medications: disease-modifying antirheumatic drugs (DMARDs) (n = 102), nonsteroidal antiinflammatory drugs (NSAIDs) (n = 100), and no arthritis medications (no meds) (n = 101). PPVs for confirmed RA were calculated for each of the medication groups, both overall and according to the results of the screening survey.

Results

The PPVs of confirmed RA were 76%, 61%, and 29% in the DMARDs, NSAIDs, and no meds groups, respectively. After exclusion of women who reported other rheumatic conditions or who reported taking only prednisone, the PPV increased in the DMARDs group to 88%, but little improvement was seen in the other groups. The PPVs increased somewhat according to results of the screening survey for the DMARDs group (92% for positive screen versus 85% for negative screen; P = 1.00), and increased substantially for the NSAIDs group (89% versus 38%, respectively; P = 0.03), but only 43% of participants completed the survey.

Conclusion

We found that self-report of RA, along with self-reported DMARDs, is a useful case definition for identifying confirmed RA. The validated screening survey could be useful for identifying cases of confirmed RA in some, but not all, medication groups.

INTRODUCTION

Rheumatoid arthritis (RA) is a chronic, systemic, inflammatory disease of unknown etiology that affects ∼1% of adults (1) and often results in severe pain and disability (2–6). The prevalence of RA increases with age, and it is a disease that is 2–3 times more likely to affect women than men (7).

Because incident RA is relatively uncommon, hospital-and clinic-based epidemiologic studies often face difficulties in enrolling adequate numbers of cases. Large, population-based studies could advance knowledge of the etiology of RA, but they often rely on self-reported diagnosis information. The positive predictive value (PPV) of self-reported RA has been studied in various populations, including the elderly and disabled, and was found to be low, ranging between 21% and 34% (8–10), possibly due in part to confusion with other forms of arthritis, such as osteoarthritis. In two large, population-based studies that investigated risk factors for RA, the Nurses' Health Study and the Iowa Women's Health Study, only 6–7% of self-reported RA diagnoses could be confirmed by medical record review (11, 12). To overcome the validity concerns associated with self-reported RA, studies have relied on medical record review to confirm the diagnosis. However, the number of potential cases lost may be substantial because of the difficulty in obtaining consent from study participants to release medical information (11–13). In addition, selective consent to record review could introduce bias in risk factor studies. Therefore, a valid RA case definition based on self-reported information would be very valuable for large, questionnaire-based epidemiologic studies.

The primary objective of the present study was to evaluate the PPVs of 3 case definitions of RA among African American women, each based on self-reported data on RA diagnosis and use of arthritis medications. A secondary objective was to determine whether supplemental self-reported information based on responses to a validated screening survey increased PPVs within the 3 case groups.

MATERIALS AND METHODS

Population.

The Black Women's Health Study (BWHS) is a prospective study of 59,000 African American women throughout the US that began in 1995. The participants were recruited through mailings to Essence magazine subscribers (a general readership magazine targeted to African American women), mailings to members of African American professional organizations, and mailings to friends and relatives of early participants (14). The 1995 baseline questionnaire obtained information on demographic characteristics, reproductive and medical histories, and lifestyle characteristics such as cigarette smoking and diet. Biennial followup questionnaires mailed to all participants elicited information on disease occurrence and selected exposures, with ≥80% of participants responding in each followup cycle.

Sample.

Subjects for the present study included a sample of BWHS participants who reported that they had been diagnosed with RA for the first time in the 2 years preceding either the 1999 or the 2001 followup questionnaire. The sample was categorized, based on reported medications, into 3 hierarchical groups: disease-modifying antirheumatic drugs (DMARDs), nonsteroidal antiinflammatory drugs (NSAIDs), and no reported RA medications (no meds). The smallest medication group, DMARDs (n = 101), set the sample size for the 2 other medication groups. The 3 groups were selected as follows.

Self-reported RA and use of DMARDs.

All women who reported RA and use of DMARDs in 1999 (n = 60) or 2001 (n = 41) were included in this group. DMARDs include methotrexate, azathioprine, D-penicillamine, sulfasalazine, etanercept, gold sodium thiomalate, cyclophosphamide, infliximab, hydroxychloroquine, leflunomide, glucocorticoid steroids (prednisone), and gold.

Self-reported RA and use of NSAIDs only.

All women who reported RA and use of NSAIDs in 2001 (n = 70) and a random sample of women who reported RA and use of NSAIDs in 1999 (n = 31), but who did not report any DMARDs, were included in this group. NSAIDs include indomethacin, etodolac, diclofenac sodium, nabumetone, sulindac, piroxicam, celecoxib, rofecoxib, ibuprofen, naproxen sodium, and ketoprofen.

Self-reported RA and no use of DMARDs or NSAIDs.

A random sample of women who reported RA and no arthritis medications in 2001 (n = 101) was included in the no meds group. Because a sufficient number of women reported RA and no medications in the 2001 questionnaire, a sample of women who reported RA and no medications in 1999 was not needed.

One participant in the NSAIDs group was subsequently found to have reported using prednisone, and was moved into the DMARDs group, resulting in 102 subjects in the DMARDs group, 100 subjects in the NSAIDs group, and 101 subjects in the no meds group. The participants ranged from 27 to 73 years of age, with an average age of 48 years.

Screening questionnaire and case confirmation.

All women in the sample were mailed a form for consent to release medical information. The consent form requested permission to obtain copies of medical records and to send a checklist to her rheumatologist and/or other physician who could provide information regarding her RA. The checklist consisted of the American College of Rheumatology (ACR; formerly the American Rheumatism Association) criteria for RA (15). The consent form provided an option to consent only to the physician checklist, and not the release of medical records.

All women were also mailed an RA screening survey that has been validated in African American women. The survey consists of the RA portion of the questions from the Connective Tissue Disease Screening Questionnaire (16, 17) and elicits information on the year and month of RA diagnosis, year of symptom onset, medications used to treat RA, RA symptoms, and results of the rheumatoid factor (RF) test. A positive screen on the survey was defined as a report of ≥4 of the following 6 symptoms included in the questionnaire: morning stiffness for ≥1 hour and present for ≥6 weeks; swelling of ≥3 joints for ≥6 weeks; swelling of wrist, metacarpophalangeal, or proximal interphalangeal joints for ≥6 weeks; symmetric joint swelling; rheumatoid nodules; and a positive test result for RF (16).

Both the consent form and the screening survey provided the participant the opportunity to refute her initial report of physician-diagnosed RA. Women who did not respond to the initial mailing of the consent form and screening survey were sent up to 2 additional mailings and were then telephoned. Physicians who did not respond to the initial medical records/checklist request were sent 1 additional request and were then telephoned. Physicians who had still not responded were then mailed or faxed a modified checklist, which asked for confirmation or refutation of an RA diagnosis and a list of any RA medications prescribed for the subject.

Case definitions.

The ACR criteria for RA (15), documentation of an RA diagnosis, year of diagnosis, and year of symptom onset were abstracted from medical records (reviewed by a trained rheumatologist [TEM] blinded to the subjects' reported medications) and physician checklists for all subjects for whom medical information was received. Three case definitions were used to classify subjects as having definite, probable, or clinical RA. Subjects were classified as having definite RA based on documentation of ≥4 of 7 ACR criteria (15). Limiting cases of RA to only those who meet 4 criteria may exclude those with less severe disease (7, 18). Therefore, probable RA was defined as those who met 3 ACR criteria, along with those classified as having definite RA. Finally, because the ACR criteria may not be well documented in the medical records, participants with an RA diagnosis mentioned in their medical records or physician checklist, or who met ≥3 ACR criteria, were defined as having clinical RA.

Statistical analyses.

Response rates to the medical records release form and RA screening questionnaire were calculated for each of the 3 study sample groups. Among those with medical records or physician checklists, the PPVs for those classified as confirmed RA (definite, probable, clinical) were calculated for each study sample group. PPVs were also calculated according to the results of the RA screening survey (positive or negative screen) within each study sample group. Many rheumatic diseases have a high degree of symptom overlap and are treated with prednisone. The accuracy of self-reported RA may be improved by excluding subjects who reported other rheumatic conditions or whose only reported medication was prednisone. Therefore, we additionally calculated the PPVs of confirmed RA in the 3 study sample groups, excluding those who reported other rheumatic conditions (systemic lupus erythematosus [SLE], myositis, dermatomyositis, mixed/undifferentiated connective tissue disease, scleroderma, CREST [calcinosis, Raynaud's phenomenon, esophageal dysmotility, sclerodactyly, telangiectasias] syndrome, Sjögren's syndrome, and sarcoidosis), and in the DMARDs group, excluding those whose only reported DMARD was prednisone. The homogeneity of the PPVs of definite, probable, and clinical RA was tested across the 3 sample groups. In addition, pairwise comparisons of the PPVs of the 3 sample groups were made for definite, probable, and clinical RA. The utility of the RA screening survey was also evaluated by comparing the PPVs of definite, probable, and clinical RA for those who screened positive versus negative within each sample group. Chi-square tests of proportions, or Fisher's exact tests for cells with <5 subjects, were used for all comparisons.

RESULTS

Frequencies and percentages of subjects' responses to medical records release requests and RA screening surveys, and of physicians' responses to medical records requests, are displayed in Figure 1. Approximately half of the DMARDs group consented to release their medical information, whereas only one-third of the NSAIDs group and one-fifth of the no meds group were willing to release information. Three subjects (3%) in the DMARDs group refuted their initial report of RA, whereas 21 (21%) and 15 (15%) refuted their diagnosis in the NSAIDs and no meds groups, respectively. There was no response to the medical records release or RA screening survey from 37% of the subjects in both the DMARDs and NSAIDs groups, and approximately half of the no meds group did not respond. Among those who consented to the release of their medical information, medical records or physician checklists were received for 88% of the DMARDs group, 70% of the NSAIDs group, and 67% of the no meds group. The final proportions of those for whom medical information was received in each sample group was 41% for the DMARDs group, 23% for the NSAIDs group, and 14% for the no meds group.

Figure 1.

Flow chart of subjects' responses to medical records release requests and the RA screening survey, and physicians' responses to medical records requests. DMARDs = disease-modifying antirheumatic drugs; NSAIDs = nonsteroidal antiinflammatory drugs; RA = rheumatoid arthritis; * = RA screening surveys were received from all subjects for whom medical records release forms were received; ** = a modified physician checklist was sent to nonresponding physicians (n = 48), which included only information on an RA diagnosis and medications (no American College of Rheumatology criteria for RA); modified physician checklists were received for 5, 4, and 3 participants in the DMARDs, NSAIDs, and no meds groups, respectively.

Table 1 describes the frequencies and PPVs of confirmed RA by sample group among those for whom medical records or physician checklists were received. Table 2 displays the P values for tests of homogeneity of proportions across the 3 sample groups, and pairwise comparisons of PPVs for definite, probable, and clinical RA, excluding those who reported prednisone only or other rheumatic conditions. Because the modified physician checklist did not include the ACR criteria, the number of subjects for whom medical information was obtained was higher for clinical RA than for definite or probable RA. The PPV of definite RA was highest for the DMARDs group (62%), intermediate for the NSAIDs group (53%), and lowest for the no meds group (27%).

Table 1. Frequencies and PPVs of definite, probable, and clinical RA by sample group among those for whom medical information was received*
Sample group Definite RAProbable RAClinical RA
Information received, no.No.PPV, %No.PPV, %Information received, no.No.PPV, %
  • *

    PPVs = positive predictive values; RA = rheumatoid arthritis; DMARDs = disease-modifying antirheumatic drugs; NSAIDs = nonsteroidal antiinflammatory drugs.

  • For clinical RA, response to the modified physician checklist increased the number of subjects for whom medical information was received.

DMARDs3723622568423276
 Excluding prednisone only3322672370383079
 Excluding other rheumatic conditions3022732377342985
 Excluding prednisone only and other rheumatic conditions2821752279322888
NSAIDs1910531158231461
 Excluding other rheumatic conditions189501056221359
No meds1132732714429
 Excluding other rheumatic conditions1033033013431
Table 2. P values by Fisher's exact tests or chi-square tests of homogeneity of proportions across the 3 sample groups, and pairwise comparisons of PPVs of definite, probable, and clinical RA excluding those who reported prednisone only or other rheumatic conditions*
ComparisonDefinite RAProbable RAClinical RA
  • *

    See Table 1 for definitions.

  • By chi-square test.

Homogeneity0.030.020.0005
DMARDs vs. no meds0.020.020.0004
DMARDs vs. NSAIDs0.080.100.02
NSAIDs vs. no meds0.430.250.16

The exclusion of women who had used only prednisone, or who had other rheumatic conditions, increased the PPV in the DMARDs group (75%), but only slightly affected the NSAIDs (50%) and no meds (30%) groups (for homogeneity, P = 0.03). Using the case definition of probable RA did not substantially increase the PPVs. After exclusions, PPVs for probable RA were 79%, 56%, and 30% in the DMARDs, NSAIDs, and no meds groups, respectively (for homogeneity, P = 0.02). The PPV of clinical RA was 76% in the DMARDs group, 61% in the NSAIDs group, and 29% in the no meds group. After exclusions, the PPV increased substantially in the DMARDs group to 88%, but little improvement was seen in the NSAIDs or no meds groups (for homogeneity, P = 0.0005).

The PPV of confirmed RA was higher in the DMARDs group than the no meds group for all case definitions (definite RA: 75% versus 30%, P = 0.02; probable RA: 79% versus 30%, P = 0.02; clinical RA: 88% versus 31%, P = 0.0004). Higher PPVs were found in the DMARDs group compared with the NSAIDs group for all case definitions (definite RA: 75% versus 50%, P = 0.08; probable RA: 79% versus 56%, P = 0.10; clinical RA: 88% versus 59%, P = 0.02). In addition, higher PPVs were found in the NSAIDs group compared with the no meds group for all case definitions.

The performance of the RA screening survey is displayed in Table 3. Within the DMARDs group, the proportion of cases classified to have definite, probable, or clinical RA who screened positive was somewhat higher than that of subjects who screened negative, although no statistically significant differences were found (e.g., clinical RA: 92% versus 85%; P = 1.00). Within the NSAIDs group, the proportion of cases classified to have definite, probable, or clinical RA who screened positive was substantially higher than that of subjects who screened negative (e.g., clinical RA: 89% versus 38%; P = 0.03). Within the no meds group, the numbers were too small to make any valid comparisons of the RA screening survey results.

Table 3. Frequencies, PPVs, and Fisher's exact tests of definite, probable, and clinical RA by screening survey results within sample groups, excluding those who reported using prednisone only and having other rheumatic conditions*
Sample group    Definite RA    Probable RA  Clinical RA
Medical information received, no.No.PPV, %PNo.PPV, %PMedical information received, no.No.PPV, %P
  • *

    See Table 1 for definitions.

  • For clinical RA, response to the modified physician checklist increased the number of subjects for whom medical information was received.

DMARDs           
 Positive screen98890.378890.631211921.00
 Negative screen191368 1474 201785 
NSAIDs           
 Positive screen86750.156750.1998890.03
 Negative screen10330 440 13538 
No meds           
 Positive screen21501.001501.0031331.00
 Negative screen8225 225 10330 

DISCUSSION

We found that the accuracy of a self-reported diagnosis of RA varies within groups defined by reported medications. Although RA could not be confirmed among a high proportion of women who reported taking no arthritis medications or who reported taking NSAIDs, most women who reported taking DMARDs were confirmed to have RA. This self-reported case definition improved after excluding women whose only reported DMARD was prednisone and women who reported other rheumatic conditions.

Our finding that the validity of self-reported RA improves substantially with reported use of DMARDs is similar to that of a validation study of self-reported RA among postmenopausal participants in the Women's Health Initiative (13). Walitt et al reported that the PPV of self-reported RA increased from 14.7% to 62.2% when use of DMARDs was also reported, but did not evaluate other medication groups, such as NSAIDs (13).

As was expected, the proportion of cases confirmed as clinical RA was greater than the proportions confirmed by ACR criteria. Although confirmation of RA by documentation of the ACR criteria has become the gold standard, there are several problems with its use in this and other large epidemiologic studies. First, the ACR criteria were developed primarily as a research tool, not as a means of clinically diagnosing patients (15). Although physicians use the ACR criteria as a guide, patients are frequently diagnosed and treated for RA without meeting 4 criteria. It is, therefore, possible for subjects to have a physician diagnosis of RA but not meet the criteria for definite RA. In addition, the ACR criteria may not be well documented in the medical record. For example, among the subjects for whom we received both medical records and physician checklists (n = 7), an average of 4 criteria were documented in the checklist, whereas an average of only 1 criterion was documented in the medical record. Last, because patients often see more than one physician (i.e., the family physician, a rheumatologist), the medical records may be located in more than one facility, increasing the difficulty of obtaining a complete record.

The RA screening survey has been found to be useful in identifying RA cases in previous research (16), and has been shown to increase the PPV of self-reported RA beyond that of self-reported use of DMARDs (13). Walitt et al reported a PPV of 82% among those who reported RA along with DMARDs and a positive screen on the Connective Tissue Disease Screening Questionnaire, compared with a PPV of 62% among those who reported only RA and DMARDs. We found that among women who reported RA and use of DMARDs, the screening survey provided little additional information to help identify cases of clinical RA. The Connective Tissue Disease Screening Questionnaire did improve the PPV in our study from 68% to 89% for definite RA, which is likely a more similar outcome to that used by Walitt et al (13). The survey did appear to identify cases in the BWHS among those who reported use of NSAIDs, but the numbers in the current study were too small to be definitive. A drawback of using the survey is that only 43% of women completed it; relying on the survey for validation would, therefore, result in an appreciable loss of cases. Further research should be conducted to determine the usefulness of the RA screening survey in ascertaining cases as a third step after self-reported RA and NSAIDs.

A limitation of our study is the low response to medical records requests, which reduced statistical power and introduced the potential for significant selection bias. Of utmost concern is the large percentage of subjects in the DMARDs group for whom no medical information was received (59%), and the possibility that they, in fact, do not have RA. The subjects were given the opportunity to refute their initial report of RA, however. In addition, when questioned, BWHS participants cited confidentiality concerns as their reason for refusal to allow record review. We found little difference between those for whom medical information was received and those for whom no medical information was received in age in years (48 versus 46; P = 0.48) or smoking (62% versus 58% ever smoked; P = 0.70), and although not statistically significant, a higher percentage of those for whom medical information was received had graduated from college compared with those for whom no medical information was received (48% versus 38%; P = 0.33). In addition, women for whom no medical information was received but who responded to the RA screening survey (n = 16) were compared with the women for whom medical information was received (n = 42), and they were found to have no differences in mean number of reported RA symptoms (2.6 versus 2.6; P = 0.91). A similar response rate was found for a study of SLE in the BWHS (19), and even in a study of female health professionals who reported connective tissue disease, response to requests for medical records was only 28% (20). Difficulties in obtaining medical information in this and other studies illustrate the disadvantage of validating cases through medical record review and underscore the need for an RA case definition based on self-reported information.

Medical information (i.e., medical records and physician checklists) was required to confirm or disconfirm a subject's report of an RA diagnosis. Therefore, PPVs were calculated only among women for whom we received medical information. Women who refuted their initial report of RA were not included in the calculations because it was unclear whether the initial report of RA or the refutation of the diagnosis was accurate, and we had no medical information to confirm either. It is possible that these women do not have RA, which would lower the PPVs in the sample groups. For clinical RA, the PPV would slightly decrease in the DMARDs group to 80%, but would substantially decrease in the NSAIDs (30%) and no meds (14%) groups.

We were unable to estimate the number of unreported RA cases in our study population. Ideally, medical records would have been obtained for a sample of women who did not report RA to determine the proportion of cases missed by relying on self-reported diagnoses. However, ascertaining medical records on such a sample was not feasible due to the difficulty of obtaining consent to release medical information among a group of subjects who have not reported the diagnosis under study. Based on previous research, the number of missed cases is likely to be small (10), reducing the likelihood of considerable bias in risk factor analyses using a case definition based on self-reported information.

Self-reported RA together with self-reported use of DMARDs was the best case definition, but its use would introduce some misclassification. Women who self-reported RA and were true cases, but did not report use of DMARDs, would be excluded. On the other hand, inclusion of women who self-reported RA and DMARDs but do not have RA would include some false-positives. However, the small number of false-positive RA cases in the DMARDs group would result in a very high specificity and, assuming that misclassification is nondifferential, a highly specific case definition would result in negligible bias of the effect estimate (21). For example, assuming an incidence rate of RA of 45/100,000 person-years and 200,000 person-years of followup over a 4-year period, we would expect ∼90 true cases of RA in the BWHS (n ≈ 59,000). Based on the best case definition of RA in this study (self-reported RA and DMARDs, excluding those who reported prednisone only and other rheumatic conditions), we would classify 63 subjects as having RA. A PPV of 88% would result in 55 correctly classified clinical RA cases, providing a sensitivity of 61% (55 of 90) and a specificity of ≥99% (58,902 of 58,910). With a 50% exposure in the BWHS cohort, disease misclassification would bias a true risk ratio of 2.0 to 1.9, and a true risk difference of 0.001 to 0.0006.

The gold standard for confirmation of an RA diagnosis in research is a clinical examination or medical record review for the ACR criteria, which are not feasible in large, geographically dispersed epidemiologic studies as is suggested by the low response rate in this and other studies. Previous research has indicated that the PPV of self-reported RA is low (8–10, 19) but improves with additional information, such as reported medications. We confirmed that self-report of RA, along with DMARDs, is a valid case definition for identifying clinical RA, and is sufficient for use after excluding those who report other rheumatic conditions and prednisone as their only DMARD. This case definition for RA based on self-reported information may improve and advance research on RA in large, epidemiologic studies.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Dr. Formica had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design. Formica, McAlindon, Rosenberg.

Acquisition of data. Formica, Rosenberg.

Analysis and interpretation of data. Formica, McAlindon, Lash, Demissie, Rosenberg.

Ancillary