Meta-analysis of clinical trials as a scientific discipline. II: Replicate variability and comparison of studies that agree and disagree

Authors

  • Thomas C. Chalmers,

    1. Clinical Trials Unit, Mount Sinal School of Medicine of CUNY, 1 Gustave L. Levy Place, New York. NY 10029, U.S.A.
    2. Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115. U.S.A.
    Search for more papers by this author
  • Jayne Berrier,

    1. Clinical Trials Unit, Mount Sinal School of Medicine of CUNY, 1 Gustave L. Levy Place, New York. NY 10029, U.S.A.
    2. Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115. U.S.A.
    Search for more papers by this author
  • Henry S. Sacks,

    1. Clinical Trials Unit, Mount Sinal School of Medicine of CUNY, 1 Gustave L. Levy Place, New York. NY 10029, U.S.A.
    2. Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115. U.S.A.
    Search for more papers by this author
  • Howard Levin,

    1. Clinical Trials Unit, Mount Sinal School of Medicine of CUNY, 1 Gustave L. Levy Place, New York. NY 10029, U.S.A.
    2. Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115. U.S.A.
    Search for more papers by this author
  • Dinah Reitman,

    1. Clinical Trials Unit, Mount Sinal School of Medicine of CUNY, 1 Gustave L. Levy Place, New York. NY 10029, U.S.A.
    2. Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115. U.S.A.
    Search for more papers by this author
  • Raguraman Nagalingam

    1. Clinical Trials Unit, Mount Sinal School of Medicine of CUNY, 1 Gustave L. Levy Place, New York. NY 10029, U.S.A.
    2. Department of Health Policy and Management, Harvard School of Public Health, 677 Huntington Avenue, Boston, MA 02115. U.S.A.
    Search for more papers by this author

Abstract

The replicate variability of meta-analyses of controlled clinical trials has been assessed as a measure of scientific precision. 46 of 91 known meta-analysis papers were divided into 20 cohorts of studies of the same therapies. Ten cohorts contained meta-analyses with different statistical conclusions; 14 contained differing clinical conclusions with a wider spread than the statistically differing studies. Possible causes of variability, such as different trials included, different policies regarding the inclusion of non-randomized and unpublished trials, and different statistical methodologies, were not obvious causes of differing conclusions. Further work in this area should include multivariate analyses in order to explore possible interactions in the factors accounting for the variability found in replicate meta-analyses.

Ancillary