How study design affects outcomes in comparisons of therapy. II: Surgical

Authors

  • James N. Miller,

    1. Center for Science and International Affairs, John F. Kennedy School of Government, Harvard University, Cambridge, MA 02138, U.S.A.
    Search for more papers by this author
  • Dr. Graham A. Colditz,

    Corresponding author
    1. Channing Laboratory, Department of Medicine, Harvard Medical School and Brigham Women's Hospital, 180 Longwood Avenue, Boston, MA 02115, U.S.A.
    • Channing Laboratory, Department of Medicine, Harvard Medical School and Brigham Women's Hospital, 180 Longwood Avenue, Boston, MA 02115, U.S.A.
    Search for more papers by this author
  • Frederick Mosteller

    1. Technology Assessment Group, Department of Health Policy and Management, Harvard University School of Public Health, 677 Huntington Avenue, Boston, MA 02115, U.S.A.
    Search for more papers by this author

Abstract

We analysed the results of 221 comparisons of an innovation with a standard treatment in surgery published in six leading surgery journals in 1983 to relate features of study design to the magnitude of gain. For each comparison we measured the gain attributed to the innovation over the standard therapy by the Mann—Whitney statistic and the difference in proportion of treatment successes. For primary treatments (aimed at curing or ameliorating a patient's principal disease), an average gain of 0.56 was produced by 20 randomized controlled trials. This was less than the 0.62 average for four non-randomized controlled trials, 0.63 for 19 externally controlled trials, and 0.57 for 73 record reviews (0.50 represents a toss-up between innovation and standard). For secondary therapies (used to prevent or treat complications of therapy), the average gain was 0.53 for 61 randomized controlled trials, 0.58 for eleven non-randomized controlled trials, 0.54 for eight externally controlled trials, and 0.55 for 18 record reviews. Readers of studies evaluating new treatments, particularly for primary treatments, may consider adjustment of the gain according to the study type.

Ancillary