Performance Characteristics of a Methodology to Quantify Adverse Events over Time in Hospitalized Patients

Authors

  • Paul J. Sharek,

    1. Division of General Pediatrics, Department of Pediatrics, Lucile Packard Children's Hospital and Stanford University School of Medicine, 700 Welch Rd., Suite #227, Palo Alto, CA 94304
    Search for more papers by this author
    • Address correspondence to Paul J. Sharek, M.D., M.P.H., Division of General Pediatrics, Department of Pediatrics, Lucile Packard Children's Hospital and Stanford University School of Medicine, 700 Welch Rd., Suite #227, Palo Alto, CA 94304; e-mail: psharek@lpch.org. Gareth Parry, M.Sc., Ph.D., is with the Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Cambridge, MA. Donald Goldmann, M.D., is with the Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Divisions of Immunology, Infectious Diseases, and Epidemiology, Harvard School of Public Health, Cambridge, MA. Kate Bones, Roger Resar, M.D., and Frances A. Griffin, R.R.T., M.P.A., are with the Institute for Healthcare Improvement, Cambridge, MA. Andrew Hackbarth, is with the Pardee RAND Graduate School, RAND Corporation, Santa Monica CA. Dale Rhoda, M.A.S., M.S., M.P.P., and Cathy Murphy, are with the Centers for Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC. Christopher P. Landrigan, M.D., M.P.H., is with the Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Harvard Medical School, Boston, MA.

  • Gareth Parry,

    1. Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Cambridge, MA
    Search for more papers by this author
  • Donald Goldmann,

    1. Institute for Healthcare Improvement, Department of Pediatrics, Harvard Medical School, Divisions of Immunology, Infectious Diseases, and Epidemiology, Harvard School of Public Health, Cambridge, MA
    Search for more papers by this author
  • Kate Bones,

    1. Institute for Healthcare Improvement, Cambridge, MA
    Search for more papers by this author
  • Andrew Hackbarth,

    1. Pardee RAND Graduate School, RAND Corporation, Santa Monica CA
    Search for more papers by this author
  • Roger Resar,

    1. Institute for Healthcare Improvement, Cambridge, MA
    Search for more papers by this author
  • Frances A. Griffin,

    1. Institute for Healthcare Improvement, Cambridge, MA
    Search for more papers by this author
  • Dale Rhoda,

    1. Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC
    Search for more papers by this author
  • Cathy Murphy,

    1. Public Health Research and Evaluation, Battelle Memorial Institute, Durham NC
    Search for more papers by this author
  • Christopher P. Landrigan

    1. Departments of Medicine, Brigham and Women's Hospital and Children's Hospital Boston, Harvard Medical School, Boston, MA.
    Search for more papers by this author

Abstract

Objective. To assess the performance characteristics of the Institute for Healthcare Improvement Global Trigger Tool (GTT) to determine its reliability for tracking local and national adverse event rates.

Data Sources. Primary data from 2008 chart reviews.

Study Design. A retrospective study in a stratified random sample of 10 North Carolina hospitals. Hospital-based (internal) and contract research organization–hired (external) reviewers used the GTT to identify adverse events in the same 10 randomly selected medical records per hospital in each quarter from January 2002 through December 2007.

Data Collection/Extraction. Interrater and intrarater reliability was assessed using κ statistics on 10 percent and 5 percent, respectively, of selected medical records. Additionally, experienced GTT users reviewed 10 percent of records to calculate internal and external teams' sensitivity and specificity.

Principal Findings. Eighty-eight to 98 percent of the targeted 2,400 medical records were reviewed. The reliability of the GTT to detect the presence, number, and severity of adverse events varied from κ=0.40 to 0.60. When compared with a team of experienced reviewers, the internal teams' sensitivity (49 percent) and specificity (94 percent) exceeded the external teams' (34 and 93 percent), as did their performance on all other metrics.

Conclusions. The high specificity, moderate sensitivity, and favorable interrater and intrarater reliability of the GTT make it appropriate for tracking local and national adverse event rates. The strong performance of hospital-based reviewers supports their use in future studies.

Ancillary