Statistics-based research – a pig in a poke?
Article first published online: 11 AUG 2011
© 2011 Blackwell Publishing Ltd
Journal of Evaluation in Clinical Practice
Special Issue: Philosophy of Evidence Based Medicine
Volume 17, Issue 5, pages 862–867, October 2011
How to Cite
Penston, J. (2011), Statistics-based research – a pig in a poke?. Journal of Evaluation in Clinical Practice, 17: 862–867. doi: 10.1111/j.1365-2753.2011.01717.x
- Issue published online: 23 SEP 2011
- Article first published online: 11 AUG 2011
- Accepted for publication: 14 June 2011
- causal inference;
- frequentist statistics;
- randomized controlled trials
Much of medical research involves large-scale randomized controlled trials designed to detect small differences in outcome between the study groups. This approach is believed to produce reliable evidence on which the management of patients is based. But can we be sure that the demonstration of a small, albeit statistically significant, difference is sufficient to infer the presence of a causal relationship between the drug and the outcome?
A study is claimed to have internal validity when other explanations for the observed difference – namely, inequalities between the groups, bias in the assessment of the outcome and chance – have been excluded. Despite the various processes that are put into place – including, for example, randomization, allocation concealment, double-blinding and intention-to-treat analysis – it remains doubtful whether the groups are equal in terms of all factors relevant to the outcome and whether bias has been excluded. As for the exclusion of chance, not only may inappropriate statistical tests be used, but also frequentist statistics has been subjected to serious criticisms in recent years that further bring internal validity into question.
But the problems do not end with the flaws in internal validity. The philosophical basis of large-scale randomized controlled trials and epidemiological studies is unsound. When examined closely, many obstacles emerge that threaten the inference from a small, statistically significant difference to the presence of a causal relationship between the drug and the outcome.
Given the influence of statistics-based research on the practice of medicine, it is of the utmost importance that the flaws in this methodology are brought to the fore.