Potential conflict of interest: Nothing to report.
Apply sound scientific methodology to the evaluation of new medical treatments†
Article first published online: 25 JUL 2011
Copyright © 2011 American Association for the Study of Liver Diseases
Volume 54, Issue 2, page 739, August 2011
How to Cite
Wiesner, R. H. (2011), Apply sound scientific methodology to the evaluation of new medical treatments. Hepatology, 54: 739. doi: 10.1002/hep.24375
- Issue published online: 25 JUL 2011
- Article first published online: 25 JUL 2011
- Accepted manuscript online: 21 APR 2011 07:51AM EST
- Manuscript Accepted: 9 APR 2011
- Manuscript Received: 7 APR 2011
To the Editor:
A recent HEPATOLOGY article by Lindor and Lindor1 suggests that we need to reconsider the status of randomized controlled clinical trials as the gold standard of evidence for evaluating new medical treatments. They suggest that observational studies for the evaluation of medical therapy are faster and cheaper and are more easily performed, and they ultimately lead to faster approval, a reduction of medication costs, and better patient care. However, Lindor and Lindor ignore the well-known fallibility of observational studies that have supported the therapeutic value of a treatment without evidence from well-developed randomized trials. There are many notable contemporary examples of widely used treatments that were accepted on the basis on observational studies and were subsequently proved to be ineffective or harmful when controlled clinical trials were performed. These treatments include the following:
- 1High-dose chemotherapy with bone marrow transplantation for metastatic breast cancer. This treatment produced high rates of response in phase 2 trials but was proved to be no more effective than standard chemotherapy and was more toxic.2–4
- 3Erythropoietin for the treatment of anemia due to chronic kidney disease. Observational studies had shown increased survival and higher hemoglobin levels; however, randomized trials were unable to show such benefits and found an association with increased rates of death, nonfatal heart attacks, strokes, and heart failure.7
- 4Early initiation of hemodialysis versus late initiation. A number of observational studies suggested that starting dialysis early could improve the survival and quality of life of patients. Although this practice was widely employed by nephrologists throughout the United States with substantial financial advantages, a recent randomized controlled trial has found that the early initiation of dialysis is not associated with improvements in survival or clinical outcomes.8
In each of these contemporary examples, observational studies suggested a beneficial effect that was widely promoted in medicine but was proven false by a randomized controlled trial.
Even more bothersome is the recent report by Zuckerman et al.,9 who reported that the US Food and Drug Administration (FDA) recalled 113 medical devices between 2005 and 2009 because they were found to pose a high risk to patients and were not rigorously studied before they were cleared for sale by the FDA. Most of these devices were approved only on the basis of observational studies and were not subjected to sound scientific methodology.
Although observational studies often are cheaper, quicker, and less difficult to perform, we should not lose sight of the simple fact noted by Pocock and Elbourne10: “ignorance calls for careful experimentation.” This means high-quality randomized controlled trials and not observations that reflect personal choices or beliefs that often rely on bogus assumptions and mathematical arguments to prove what is not true to be true. In this era of money-based medicine and FDA laxity, when the ethics of pharmaceutical companies are continually being questioned and the costs of treatment continue to skyrocket, let us continue to evaluate health care interventions by the most scientifically sound and rigorous methods available.
- 1The value of observational research in liver disease. HEPATOLOGY 2011; 53: 1-3., .
- 2Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med 1991; 324: 781-788., , , , , , et al.
- 3Conventional-dose chemotherapy compared with high-dose chemotherapy plus autologous hematopoietic stem-cell transplantation for metastatic breast cancer. Philadelphia Bone Marrow Transplant Group. N Engl J Med 2000; 342: 1069-1076., , , , , , et al.
- 4False Hope: Bone Marrow Transplantation for Breast Cancer. Oxford, United Kingdom: Oxford University Press; 2007., , , .
- 5A controlled trial of arthroscopic surgery for osteoarthritis of the knee. N Engl J Med 2002; 347: 81-88., , , , , , et al.
- 6Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial. JAMA 2002; 288: 321-333., , , , , , et al.
- 7Food and Drug Administration. Public advisory: erythropoiesis-stimulating agents including Procrit, Epogen, and Aranesp.
- 8A randomized controlled trial of early versus late initiation of dialysis. N Engl J Med 2010; 363: 609-619., , , , , , et al.
- 9Medical device recalls and the FDA approval process. Arch Intern Med; doi:10.1001/archinternmed. 2011.30., , .
- 10Randomized trials or observational tribulations? N Engl J Med 2000; 342: 1907-1909., .
Russell H. Wiesner M.D.*, * Mayo Clinic, Rochester, MN.