Today we salute the Editors!
The Editors recently highlighted the issue of research that can be ‘ephemeral…and destined to the dustbin of irreproducible results’. They acknowledged the critical responsibility of Editors and Journals to identify those papers that form the foundation upon which others can build (Bosenberg et al., 2013).
The issue of irreproducibility of the majority of preclinical scientific reports has deservedly received considerable attention of late. Several groups have indicated that we have a serious problem, particularly in the ‘top-tier’ journals, with between ~75 and 90% of publications unable to be reproduced (Begley and Ellis, 2012; Booth, 2011; Prinz et al., 2011). This finding was also confirmed by scientists representing both Novartis and Astra Zeneca during their presentations at the American Association of Cancer Research annual meeting in 2012. This is a widespread, systemic problem involving many laboratories and multiple journals. It is likely attributable to the intense pressure to publish.
The Editors of this Journal have sought to address this challenge head on with new author guidelines. The challenge is to strike appropriate balance: we certainly do not want to be so rigid that we stifle real innovation, but real innovation has to reproducible for it to be real.
Some of the changes that are being introduced require authors to provide data to support the validation of reagents including antibodies, siRNA, and small molecule inhibitors. This is a major advance as many irreproducible publications overlook this critical step. Too frequently, the effects of a multipotent small molecule inhibitor are attributed to the authors' favorite molecule, ignoring the other targets that could be equally responsible. Similarly, antibodies that detect multiple antigens are used, illegitimately, for immunohistochemistry, and the results again attributed to the authors' favorite antigen. Now, the controls for those experiments will be required.
The Editors are also demanding that observations be confirmed in more than one cell line. This is another advance, as studies in a single cell line tell us little or nothing about a more general process. The Editors want to see inclusion of positive and negative controls and a statement as to how many time experiments have been repeated. Data selection is also specifically addressed with a requirement for a statement as to how many experiments showed a similar result and how many did not.
It would seem unnecessary to have these as specified requirements for publications, as these would typically be regarded as essential and routine elements of standard scientific practice. Yet frequently in the biological journals, experiments are presented that lack these critical elements. The changes introduced by these Editors represent real progress!
Like this Journal, Nature has recently announced new guidelines for authors. They too are to be congratulated. That journal will provide more space for methods, and authors will need to ‘provide precise characterization of key reagents such as cell lines and antibodies’ (Nature, 2013). However, while it is true that ‘exploratory investigations cannot be done with the same level of statistical rigor as hypothesis-testing studies’ (Nature, 2013), there is little acknowledgment of the important difference between the two and the degree of confidence that can be ascribed as a consequence. Further, there is still an inadequate focus on the need for scientific rigor to underpin any result, statistically significant or not. In fact, our current scientific system appears to have evolved to place more value on the more speculative, less reliable ‘exploratory investigations’ than on those studies that confirm or refute a hypothesis. The latter in particular are poorly valued: those who document the invalidity of a published piece of work seldom get a welcome from journals, funding agencies, conference organizers, even as money and effort are wasted on false and irreproducible ‘exploratory investigations’.
The focus on ‘exploratory investigations’ is perhaps understandable, as we like to be scientifically titillated with a new, exciting idea even if it does not stand the test of time. However, these irreproducible ‘exploratory investigations’ represent the bulk of publications in biology journals. While they may have some value, the ideas that are robust are more likely to move the field forward. Perhaps, the exploratory investigations in top-tier journals that typically generate the excitement, the press releases, the speaker invitations, and the grants should be labeled for what they are ‘Exploratory Studies’.
One of the most troubling aspects of our analysis, which spanned a decade (Begley, 2013; Begley and Ellis, 2012), was the realization that leading investigators were unable to reproduce their own results in their own laboratory when experiments were performed blinded. It is, after all, easier to obtain the desired result in the absence of blinding! Simply demanding experiments be performed by blinded investigators would remove much of the investigator bias. Although this would represent the single most important step toward improving the reliability of published studies, this appears to be something that journals and investigators are loathe to directly address. The Editors here have partially addressed this, as authors are ‘encouraged’ to perform experiments that have a subjective endpoint in a blinded fashion. Experiments involving immunohistochemistry, cell counting, and tumor measurements are specifically mentioned. Although this is a step in the right direction, the Editors could have gone one step further and made this a requirement just as it is for publications in the top-tier clinical journals.
In fact, when contemplating these new instructions, it is salutary to realize that many of the same issues, lack of blinding, exclusion of data at the whim of the investigator, lack of appropriate statistical testing, concurrent controls, and requirement for a prespecified hypothesis, were dealt with in the clinical literature several decades ago. It is routine for a first-class clinical publication to have addressed each of these. It is also routinely accepted that the original data may be subject to review by those who paid for the clinical study or by regulatory authorities. Unfortunately, the preclinical arena lags a long way behind. In fact, some argue that such requirements would stifle preclinical research: the notion of independent review of the primary data is seen as particularly affronting. The contrary argument is that change is essential given our poor record of generating reliable ‘exploratory investigations’. Their current value is comparable to that of a single case report.
Clearly, these changes will demand more effort on the part of investigators. That is as it should be. It is essential that the authors shoulder the burden for greater rigor. The investigators and their institutions are ultimately responsible for addressing the issues of scientific sloppiness and the consequent irreproducible data. The investigators, not editors, not reviewers, are principally responsible. Unfortunately, despite this, it is evident that many investigators can subjugate their key responsibility to obtain a sought-after publication.
These changes should also make it more difficult to have a paper accepted in PCMR and should increase the likelihood that studies published in this Journal will stand the test of time. Conversely, journals that do not embrace these type of changes will be recognized as such, and their reputation should be impacted accordingly.
Although the journal now has a minimum set of standards that must be met, there are still many ways that investigators can obfuscate data, so it is obscured. One such way is to present data as a percentage of control. This allows an investigator to include experiments that should really have been rejected. It is impossible for an editor or reviewer to recognize inclusion of such experiments. Similarly, a result can be statistically significant although of little biological significance: an investigator understands the nuances of their system better than any reviewer and is therefore better able to recognize the difference between the two. Equally, the reviewer cannot know how many undeclared post hoc analyses were performed to find the result that is presented as the final story. The busy reviewer cannot be expected to check all the citations to see which are incorrectly cited to support an illegitimate claim. It is the investigator who should plot cell growth on a logarithmic not a linear axis (cell growth is an exponential function after all!). Of course, stringent guidelines will not prevent lies and fraud.
Although the changes embraced in this Journal represent an important step forward, there are additional systemwide challenges that remain to be addressed. There is little or no consequence for publishing a poor-quality, irreproducible paper: the focus of review committees and promotion committees is typically on the journal where the paper is published, not on the quality of the work. Unfortunately, there is also little value placed on training students and postdoctoral fellows. Students and postdocs need to be trained and closely supervised, so that they understand the need for controls, for replication, for objective reporting, and how to evaluate data. The groups that can primarily effect changes in these areas are the funding agencies, grant review committees and promotions committees. It is hoped that these groups will soon follow the lead set by these Editors.
So again, congratulations to the Editors of PCMR. This is a real step forward!