Methodology Review

New treatments compared to established treatments in randomized trials

  1. Benjamin Djulbegovic1,*,
  2. Ambuj Kumar2,
  3. Paul P Glasziou3,
  4. Rafael Perera4,
  5. Tea Reljic5,
  6. Louise Dent6,
  7. James Raftery7,
  8. Marit Johansen8,
  9. Gian Luca Di Tanna9,
  10. Branko Miladinovic2,
  11. Heloisa P Soares10,
  12. Gunn E Vist11,
  13. Iain Chalmers12

Editorial Group: Cochrane Methodology Review Group

Published Online: 17 OCT 2012

Assessed as up-to-date: 24 MAR 2010

DOI: 10.1002/14651858.MR000024.pub3


How to Cite

Djulbegovic B, Kumar A, Glasziou PP, Perera R, Reljic T, Dent L, Raftery J, Johansen M, Di Tanna GL, Miladinovic B, Soares HP, Vist GE, Chalmers I. New treatments compared to established treatments in randomized trials. Cochrane Database of Systematic Reviews 2012, Issue 10. Art. No.: MR000024. DOI: 10.1002/14651858.MR000024.pub3.

Author Information

  1. 1

    USF Health Clinical Research, University of South Florida, USF Clinical Translational Science Institute, Dpts of Medicine, Hematology and Health Outcome Research, USF and H. Lee Moffitt Cancer Center, Tampa, Florida, USA

  2. 2

    University of South Florida, Center for Evidence Based Medicine and Health Outcomes Research, Tampa, Florida, USA

  3. 3

    Bond University, Centre for Research in Evidence Based Practice, Gold Coast, Queensland, Australia

  4. 4

    University of Oxford, Department of Primary Care Health Sciences, Oxford, UK

  5. 5

    Center for Evidence Based Medicine and Health Outcomes Research, University of South Florida, Department of Internal Medicine, Tampa, Florida, USA

  6. 6

    Southampton General Hospital, University of Southampton Clinical Trials Unit, Southampton, Hampshire, UK

  7. 7

    University of Southampton, NIHR Evaluation, Trials and Studies Coordinating Centre, Southampton, Hampshire, UK

  8. 8

    Norwegian Knowledge Centre for the Health Services, Global Health Unit, Oslo, Norway

  9. 9

    Sapienza University of Rome, Public Health and Infectous Diseases, Rome, Italy

  10. 10

    Mount Sinai Medical Center, Miami Beach, Florida, USA

  11. 11

    Norwegian Knowledge Centre for the Health Services, Prevention, Health Promotion and Organisation Unit, Oslo, Norway

  12. 12

    James Lind Initiative, Oxford, Oxfordshire, UK

*Benjamin Djulbegovic, USF Clinical Translational Science Institute, Dpts of Medicine, Hematology and Health Outcome Research, USF and H. Lee Moffitt Cancer Center, USF Health Clinical Research, University of South Florida, 12901 Bruce B. Downs Boulevard, CMS 3057, Tampa, Florida, 33612, USA. bdjulbeg@health.usf.edu. benjamin.djulbegovic@moffitt.org.

Publication History

  1. Publication Status: New
  2. Published Online: 17 OCT 2012

SEARCH

 

Abstract

  1. Top of page
  2. Abstract
  3. Plain language summary

Background

The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments.

Objectives

Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results.

Search methods

We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010).

Selection criteria

Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans.

Data collection and analysis

RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival.

Main results

The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I2 = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy).

Authors' conclusions

Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.

 

Plain language summary

  1. Top of page
  2. Abstract
  3. Plain language summary

New treatments versus established treatments in randomized trials

Random allocation to different groups to compare the effects of treatments is used in fair tests to find out which among the treatment options is preferable. Random allocation is only ethical, however, if there is genuine uncertainty about which of the treatment options is preferable. If a patient or their healthcare provider is certain which of the treatments being compared is preferable they should not agree to random allocation, because this would involve the risk that they would be assigned to a treatment they believed to be inferior. Decisions about whether to participate in randomized trials are made more difficult because of the widespread belief that new treatments must inevitably be superior to existing (standard) treatments. Indeed, it is understandable that people hope that this will be the case. If this was actually so, however, the ethical precondition of uncertainty would often not apply. This Cochrane methodology review addresses this important question: "What is the likelihood that new treatments being compared to established treatments in randomized trials will be shown to be superior?" Four cohorts of consecutive, publicly funded, randomized trials, which altogether included 743 trials that enrolled 297,744 patients, met our inclusion criteria for this review. We found that, on average, new treatments were very slightly more likely to have favorable results than established treatments, both in terms of the primary outcomes targeted and overall survival. In other words, when new treatments are compared with established treatments in randomized trials we can expect slightly more than half will prove to be better, and slightly less than half will prove to be worse than established treatments. This conclusion applies to publicly funded trials as we did not include studies funded by commercial sponsors in our analysis.The results are consistent with the ethical preconditions for random allocation – when people are enrolled in randomized trials, the results cannot be predicted in advance as there is genuine uncertainty about which of the treatments being compared in randomized trials will prove to be superior.