SEARCH

SEARCH BY CITATION

Introduction

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES

Opportunistic infections such as tuberculosis have been associated with the use of tumor necrosis factor α (TNFα) antagonists such as infliximab and etanercept (1). These drugs are increasingly used for the treatment of rheumatoid arthritis (RA), Crohn's disease (CD), and a variety of other inflammatory conditions. Case reports and small case series have raised additional concerns for an increased risk of other opportunistic infections such as coccidiomycosis, histoplasmosis, and Pneumocystis jiroveci among infliximab and etanercept users (2–4). Other serious adverse events (AEs) that have also been reported following TNFα antagonist use include aplastic anemia, hematologic malignancies, and lupus-like syndromes. However, these conditions are relatively uncommon in the US, and there are few available data sources large enough to allow meaningful comparison of incidence rates in populations exposed to TNFα antagonists versus those unexposed.

Administrative data sources such as the claims databases of large health insurers are often used to study AEs related to particular drug exposures (5). The large sample sizes, generalizability, and cost-effective data collection of these databases make them a useful tool for research. The positive predictive values (PPVs) of certain single or combinations of International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) administrative codes are very high (e.g., >90% for acute myocardial infarction), suggesting that costly and labor-intensive case-by-case medical record review may not be needed for validation for some medical outcomes (6). In light of the highly variable success of validating claims data for various conditions and preliminary reports quantifying the risk of serious AEs associated with biologic agents using administrative data end points without medical record confirmation (7–9), we were interested in examining the validity of a similar approach using a different data source. We sought to determine the PPV of ICD-9-CM diagnostic codes corresponding to various rare, serious AEs identified in administrative data among patients with RA and CD using medical record review as the gold standard. We also were interested in determining whether the PPV of the AE diagnoses differed between patients receiving TNFα antagonists versus those not receiving these agents, and we hypothesized that detection of AEs in patients receiving these agents (who may also see their physicians more frequently) might be affected by detection bias or other sources of confounding. If so, we hypothesized that AE confirmation rates would be higher among patients treated with TNFα antagonists than among those receiving other immunosuppressive medications for RA/CD.

Patients and Methods

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES

Patients with RA and CD were identified using medical and pharmacy administrative claims data from a large, geographically diverse US health care organization between January 1, 1998 and December 31, 2002. Patients were defined as having RA and CD using a single ICD-9 diagnosis code (714.x and 555.x, respectively) during the study period or the prior 6 months. Patients with RA were assigned an index date based on their first exposure to etanercept or infliximab (the exposed cohort) or based on the third filled prescription for methotrexate (the comparator cohort). Patients with CD were assigned an index date based on their first exposure to infliximab (the exposed cohort) or the date of the third filled prescription for methotrexate, 6-mercaptopurine, azathioprine, or prednisone >10mg/day (the comparator cohort). We required at least 3 filled prescriptions to increase the likelihood that the diagnoses of patients with RA and CD were valid and that the severity of illness warranted ongoing immunosuppressive therapy. Patients with any claim for human immunodeficiency virus disease, organ transplant, or solid tumor malignancies were excluded.

To maximize sensitivity, the presumptive AEs of interest (shown in Table 1) were identified using ≥1 diagnosis codes on any type of claim after the index date. These claims included not only face-to-face physician encounters, but also claims for diagnostic tests including laboratory or radiologic studies. We attempted to identify patients with a new AE of interest by excluding individuals if their administrative data indicated that they had been treated for the same condition in the prior 6 months. Presumptive AE cases were confirmed or refuted through medical record review using an evidence-based, pilot-tested abstraction instrument. Trained nurse abstractors completed the instrument after being directed to the physician office or hospital location associated with the medical claim for the AE diagnosis. Approximately one-quarter of the records corresponding to the AE claim could not be obtained because the record was unavailable (e.g., practice closure, desired records could not be located). In these cases, the claims data were examined in the dates proximate to the AE claim to identify an alternative medical record. Similarly, when the suspected AE appeared on a claim for laboratory or diagnostic procedures (∼20% of the total) rather than face-to-face encounters with a physician, an alternative record for a face-to-face physician encounter was identified using the proximate claims data. Cases that did not meet the prespecified evidence-based case definitions but that had clinical information available from medical record abstraction were evaluated by the physician investigators. The additional clinical information was used to confirm or refute the suspected diagnosis independent of the case definition, and discordance among the physician investigators was resolved by consensus.

Table 1. Confirmation status of 46 claims-identified suspected serious adverse events by medical record review*
 ConfirmedNot confirmedTotal
  • *

    One abstraction request was declined. Positive predictive value of administrative claims data (8/45) = 18%.

  • Latent tuberculosis infection was not considered confirmed active tuberculosis disease.

Aplastic anemia2911
Non-Hodgkin's lymphoma257
Lupus-like syndrome066
Active Mycobacterium tuberculosis01414
Pneumocystis jiroveci011
Histoplasmosis303
Coccidioidomycosis022
Cryptococcus101
Total83745

Results

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES

The study population consisted of 1,682 patients with RA and CD exposed to infliximab or etanercept and 3,145 individuals with RA and CD exposed to comparator disease-modifying antirheumatic drugs (DMARDs). The mean ± SD age of the 2 cohorts was 50 ± 14 years and 48 ± 16 years, respectively. Approximately two-thirds of both cohorts were women.

Using the administrative data for these 4,827 persons, we identified 46 patients with presumptive AEs of interest. Patients had a mean of 2.6 claims for the condition of interest, and they were enrolled in the health plan for a mean of 14 months after the condition was first identified in the claims data. Twenty of the 46 persons had only 1 claim for the suspected AE, and in many of these cases, the claim represented a rule-out diagnosis. Of the 46 medical records requested, 45 were abstracted; 10 were located in an inpatient setting and the remaining 35 were abstracted from an outpatient physician office setting. The results of the medical record review for each condition are shown in Table 1. The PPV of the claims data for confirmed AE cases was 18% (95% confidence interval 9–33%). For some conditions, such as active tuberculosis, none of the presumptive cases were confirmed.

The PPVs of various subgroups of the claims-based case identification strategy are shown in Table 2. The PPV of suspected AEs identified during inpatient hospitalizations was higher compared with outpatient settings. Similarly, the PPV of suspected AEs with >1 claim was slightly higher than those with only 1 claim. There was no significant difference between the mean number of claims for confirmed AEs (n = 4.0) compared with the mean number of claims for unconfirmed AEs (n = 2.3) (data not shown).

Table 2. Positive predictive values (PPVs) of various partitions of the claims-based case identification strategy*
 PPV (%)95% CI
  • *

    95% CI = 95% confidence interval.

  • The desired medical record was the one associated with the diagnosis claim found in the administrative data.

Treatment setting  
 Inpatient hospitalization4012–74
 Outpatient visit113–27
Number of diagnosis claims  
 >1249–45
 Only 1101–32
Desired medical record abstracted  
 Yes227–44
 No; alternate medical record  abstracted143–35

Among many potential reasons that cases were not confirmed, we identified several that were applicable to our case finding strategy. For AEs that shared a diagnostic name with a more common condition, medical coding mistakes appeared to have caused false positive findings. For example, most of the suspected cases of aplastic anemia were not confirmed (9 of 11), and the patient instead had another form of anemia. For tuberculosis, the most common reason for lack of confirmation was that the claim represented a rule-out diagnosis. Although we excluded patients with AEs of interest identified in the administrative data in the 6 months prior to the index date, even for some confirmed cases, the medical record indicated that the infection was a prevalent condition and not a new medical problem requiring treatment.

The confirmation rates of the suspected AEs in Table 1 are shown in Table 3 according to biologic exposure status. There were no significant differences in the AE confirmation rates between patients receiving biologic agents and those receiving DMARDs (10% versus 25%; P = not significant). Of note, the confirmation rate of the RA and CD diagnoses was higher for individuals treated with biologic agents than for those receiving comparator DMARDs (95% versus 71%; P < 0.05) (data not shown).

Table 3. Confirmation of suspected adverse events (AEs) in abstracted medical records by tumor necrosis factor α antagonist exposure status*
 AE diagnosis confirmed, no.AE diagnosis not confirmed, no.Confirmation rate, %
  • *

    RA = rheumatoid arthritis; CD = Crohn's disease; DMARD = disease-modifying antirheumatic drug. P values comparing confirmation rates for patients with RA/CD treated with tumor necrosis factor α antagonists versus those treated with comparator DMARDs.

  • The suspected AEs are listed Table 1.

AE confirmation for RA/CD patients receiving infliximab or etanercept21910
AE confirmation for RA/CD patients receiving comparator DMARDs61825

Discussion

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES

The usefulness of administrative claims data to identify serious AEs potentially associated with TNFα antagonist use has received only limited attention (8). Among a cohort of almost 5,000 patients enrolled in a US managed care plan over 4 years, we demonstrated that a case identification strategy searching for ≥1 ICD-9-CM codes representing AEs possibly associated with TNFα antagonist use had low PPV (18%) to identify confirmed AEs. Case confirmation rates did not significantly differ by whether patients were exposed to TNFα antagonists or not, although study power was limited to address this hypothesis. Our results also suggest that identifying suspected AEs using only administrative data may, depending on the algorithm used, substantially overestimate the rates of at least some AEs.

We identified several possible reasons for the lack of confirmation of suspected AEs of interest. Although the use of a single diagnostic code to detect an outcome of interest maximized sensitivity, it also contributed to the low PPV of our case finding approach. However, no other selection criteria using claims data (inpatient rather than outpatient claims, >1 claim, or mean number of claims) had a PPV higher than 50%. Although we abstracted only 1 medical record that sometimes did not contain all the information necessary to confirm or refute the diagnosis of the AE of interest, even full access to the entirety of a patient's medical record is often found to be an imperfect gold standard (7).

There are only a few studies that have examined the validity of administrative and electronic medical record data to identify serious infections. Using the UK General Practitioner Research Database, Metlay and Kinman identified cases of pneumococcal pneumonia using diagnosis codes for pneumonia and found that of 89 medical records reviewed, community-acquired pneumonia was diagnosed by the treating physician in only 62% of cases; pneumococcal pneumonia was confirmed in only 26% of cases (10). In a methodologic study of 1,376 women enrolled in a large US managed care organization, pharmacy data were used to identify suspected infections, and a physician-diagnosed infection in the outpatient medical records plus microbiologic data were used as the gold standard (11). Using a set of decision rules applied to the pharmacy data, a PPV of ∼60% was obtained. Another study evaluated 96 postoperative surgical site infections among 4,086 procedures (12). An algorithm integrating inpatient codes, outpatient codes, filled antibiotic prescriptions, and duration of surgical procedure had a PPV of 81%. In contrast to our cohort, patients in these studies did not have inflammatory diseases and were not receiving concomitant immunosuppressive agents, factors that may mimic or mask the signs and symptoms of active infection. The higher PPVs found in studies identifying common AEs such as bacterial infections are consistent with the understanding that the PPV of any diagnostic test is related to the prevalence of the condition. Requiring a very specific diagnosis, such as pneumococcal pneumonia, also lowers PPV. The prevalence of infections in these studies (between 2% and 10%) was far greater than the prevalence of the end points in our cohort (0.2% prevalence for all conditions combined). Our results are similar to those of a Canadian study of 996 patients treated with biologic agents that examined the risk for tuberculosis over 32 months (8). Among these individuals, only 2 of the 13 persons with suspected tuberculosis based on administrative data were subsequently treated for tuberculosis, suggesting that the majority of the suspected cases did not represent active tuberculosis disease.

An important strength of our study was the patient population, which was enrolled in a large, geographically diverse US health plan and received TNFα antagonists or other immunosuppressives in real-world settings. Moreover, we conducted a comprehensive, nationwide medical record abstraction. However, we were limited to abstracting only 1 record per patient; abstracting additional medical records per patient may have increased our case finding yield. We also did not use the medical claims data in conjunction with pharmacy data to identify suspected AEs, which has been shown to be useful in identifying conditions such as active tuberculosis for which multiple drugs are often prescribed (13). However, we confirmed no cases of active tuberculosis, and it is possible that the use of pharmacy data would improve the sensitivity of our approach, which used only diagnosis codes. We also acknowledge that the results we found using a US patient cohort may differ compared with cohorts of patients from geographic areas with higher prevalence rates of some of the AEs we examined (e.g., active tuberculosis).

In conclusion, claims data are useful as an initial screen to identify uncommon adverse outcomes associated with TNFα antagonists and DMARDs. However, for rare events and those requiring highly specific diagnoses, the low PPV of ≥1 ICD-9 codes to identify confirmed outcomes highlights the need for medical record confirmation as a second step to improve specificity and PPV. Although a more sophisticated claims-based algorithm without medical record review might achieve better positive predictive values than what we observed, the rarity of these AEs suggests that a highly sensitive, first-stage search strategy such as the one we used is desirable so as not to miss cases and that a second, more specific case validation stage also is required.

AUTHOR CONTRIBUTIONS

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES

Drs. Curtis and Saag had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Study design. Curtis, Martin, Saag, Kramer, Shatin, Allison, Braun.

Acquisition of data. Curtis, Martin, Shatin, Braun.

Analysis and interpretation of data. Curtis, Martin, Saag, Patkar, Kramer, Shatin, Allison, Braun.

Manuscript preparation. Curtis, Martin, Saag, Patkar, Kramer, Shatin, Allison, Braun.

Statistical analysis. Curtis, Martin, Shatin, Braun.

Project management. Patkar.

Federal funding. Shatin.

Acknowledgements

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES

We thank Drs. Nigel Rawson, Aparna Mohan, and Larry Moreland for their contributions to this project.

REFERENCES

  1. Top of page
  2. Introduction
  3. Patients and Methods
  4. Results
  5. Discussion
  6. AUTHOR CONTRIBUTIONS
  7. Acknowledgements
  8. REFERENCES
  • 1
    Keane J, Gershon S, Wise RP, Mirabile-Levens E, Kasznica J, Schwieterman WD, et al. Tuberculosis associated with infliximab, a tumor necrosis factor α-neutralizing agent. New Engl J Med 2001; 345: 1098104.
  • 2
    Bergstrom L, Yocum DE, Ampel NM, Villanueva I, Lisse J, Gluck O, et al. Increased risk of coccidioidomycosis in patients treated with tumor necrosis factor α antagonists. Arthritis Rheum 2004; 50: 195966.
  • 3
    Seddik M, Meliez H, Seguy D, Viget N, Cortot A, Colombel JF. Pneumocystis jiroveci (carinii) pneumonia following initiation of infliximab and azathioprine therapy in a patient with Crohn's disease [published erratum in Inflamm Bowel Dis 2005;11:705]. Inflamm Bowel Dis 2004; 10: 4367.
  • 4
    Velayos FS, Sandborn WJ. Pneumocystis carinii pneumonia during maintenance anti-tumor necrosis factor-α therapy with infliximab for Crohn's disease. Inflamm Bowel Dis 2004; 10: 65760.
  • 5
    Shatin D, Rawson N, Stergachis A. UnitedHealthGroup. In: StromB, editor. Pharmacoepidemiology. 4th ed. West Sussex: John Wiley & Sons; 2005.
  • 6
    Kiyota Y, Schneeweiss S, Glynn RJ, Cannuscio CC, Avorn J, Solomon DH. Accuracy of Medicare claims-based diagnosis of acute myocardial infarction: estimating positive predictive value on the basis of review of hospital records. Am Heart J 2004; 148: 99104.
  • 7
    West S, Strom B, Poole C. Validity of pharmacoepidemiologic drug and diagnosis data. In: StromBL, editor. Pharmacoepidemiology. 4th ed. West Sussex: John Wiley & Sons; 2005.
  • 8
    Choquette D, Einarson T, Leombruno J. Opportunistic infections with anti-TNF agents: estimating incidence from pharmacy and physician claims databases. Arthritis Rheum 2005; 52: S34950.
  • 9
    Smitten A, Simon T, Hochberg M, Suissa S. Rates of infection in rheumatoid arthritis [abstract]. Arthritis Rheum 2004; 50: S4789.
  • 10
    Metlay J, Kinman J. Failure to validate pneumococcal pneumonia diagnoses in the general practice research database [abstract]. Pharmacoepidemiol Drug Saf 2003; 12: S1634.
  • 11
    Leveille SG, Gray S, Black DJ, LaCroix AZ, Ferrucci L, Volpato S, et al. A new method for identifying antibiotic-treated infections using automated pharmacy records. J Clin Epidemiol 2000; 53: 106975.
  • 12
    Sands K, Vineyard G, Livingston J, Christiansen C, Platt R. Efficient identification of postdischarge surgical site infections: use of automated pharmacy dispensing information, administrative data, and medical record information. J Infect Dis 1999; 179: 43441.
  • 13
    Yokoe DS, Coon SW, Dokholyan R, Iannuzzi MC, Jones TF, Meredith S, et al. Pharmacy data for tuberculosis surveillance and assessment of patient management. Emerg Infect Dis 2004; 10: 142631.