Pubcreds: Fixing the Peer Review Process by “Privatizing” the Reviewer Commons

Authors


Abstract

The peer review system is breaking down and will soon be in crisis: increasing numbers of submitted manuscripts mean that demand for reviews is outstripping supply. This is a classic “tragedy of the commons,” in which individuals have every incentive to exploit the “reviewer commons” by submitting manuscripts, but little or no incentive to contribute reviews. The result is a system increasingly dominated by “cheats” (individuals who submit papers without doing proportionate reviewing), with increasingly random and potentially biased results as more and more manuscripts are rejected without external review. Because this is a classic tragedy of the commons, we propose a classic solution: privatizing the commons. Specifically, we propose that instead of being free to exploit the reviewer commons at will, authors should have to “pay” for their submissions using a novel “currency” called PubCreds, earned by performing reviews. We discuss how this simple, powerful idea could be implemented in practice, and describe its advantages over previously proposed solutions. While our proposal may seem radical, doing nothing will lead to a system in which external review becomes a thing of the past, decision-making by journals is correspondingly stochastic, and the most selfish among us are the most rewarded.

Introduction: the peer review crisis

Peer review is a fundamental and essential component of the scientific process, providing some quality control over what science is funded and published, and helping assure publication of articles in appropriate journals (Wagner 2006). Authors submit funding proposals and manuscripts reporting research findings, and these are evaluated by reviewers drawn from the community of their peers. Individual scientists participate in the process as both author and reviewer.

Increasing numbers of manuscripts submitted to an ever-growing number of journals are putting increasing demands on the peer review process (Riisgard et al. 2001, Hochberg et al. 2009, Holyoak 2009, McPeek et al. 2009). Receiving in one week five or more requests to review manuscripts is not uncommon. Editors speak of greater and greater difficulty soliciting sufficient numbers of quality reviews. Indeed, some leading journals require that editors nominate 10 potential reviewers, in order to stand a good chance of securing three reviews. Coupled with the limited direct incentives to review, the demand for reviews is beginning to outstrip supply (note, however, that there is little quantitative information about long-term trends). It is possible that the average quality of reviews will decline because reviewers have less time to spend per manuscript, if this is not already happening. The peer review system is under significant pressure due to this “tragedy of the reviewer commons” (Hochberg et al. 2009).

Funding agencies also have to obtain sufficient numbers of useful peer reviews, often in a relatively short time. Some have taken action that suggests they too are experiencing increasing difficulty soliciting peer reviews (e.g., the establishment of the Natural Environment Research Council, UK, Peer Review College). For the remainder of this article we focus on the peer review of manuscripts submitted for possible publication in scientific journals. Many of the points we make and solutions we suggest are, however, likely to be equally applicable to the review of funding applications.

Here we address the problem of obtaining sufficient quality and timely peer reviews. We outline the solutions so far proposed, propose and discuss in detail our new solution, and briefly explore how it could be implemented.

Proposed solutions

Attempts to solve the problem of obtaining high-quality, timely reviews have included appeals to altruism and professional ethics (Hochberg et al. 2009, McPeek et al. 2009). We are sceptical about the short- or long-term efficacy of such appeals. First, many academics probably already consider themselves quite altruistic. Second, one has to consider the reports of increasing pressures on academics' time, and associated effects on the time available for altruism. Lastly, if altruism and professional ethics were sufficiently powerful, the “tragedy of the reviewer commons” (Hochberg et al. 2009) would never have developed in the first place.

In his original paper identifying the tragedy of the commons, Garrett Hardin (1968) argued that appeals to altruism were futile, and that the only solution was “mutual coercion, mutually agreed to.” Hardin's suggestion underpins solutions based on providing incentives to review and/or disincentives to submit. Suggestions have included penalties for declining requests to review (Hauser and Fehr 2007), charging authors a fee to submit, which is then used to pay referees (see the online comments associated with Hauser and Fehr [2007]), and rewarding referees in various nonmonetary ways, such as the listing of “top reviewers.”

Although well motivated and likely to have some positive effect, these suggestions all have practical problems. Reviewers may have good reason to decline requests to review (e.g., because they already agree to review many manuscripts), so that penalizing them will be unfair. Charging submission fees and paying for reviews is attractive in that, in principle, authors could pay for their submissions by doing a proportionate amount of refereeing. However, a fee-to-submit system would disadvantage authors who lack the means to pay, might require exorbitant payments in order to attract referees who would not otherwise agree to serve, likely would cause authors to avoid journals charging submission fees, and would require frequent currency exchange due to the international nature of science. Finally, many journals already name and praise their referees in print, and there is no evidence that such token gratitude (which costs little to provide) affects referee behavior. Other signs of gratitude include invitations to join Editorial Boards.

Another suggestion is to increase the number of available reviewers, for example by encouraging graduate students to do more reviewing than at present (Riisgard et al. 2001, Hochberg et al. 2009). Although we agree that reviewing is good training, we do not believe there is substantial training benefit to students in asking them to do more than occasional reviews. Many graduate students are already overworked and poorly paid, and make relatively few submissions. It seems unfair to ask them to do the refereeing that should be done by their more established colleagues who do most of the submitting. Other methods for increasing the number of available reviewers include increasing the size of editorial boards (Holyoak 2009). Larger numbers of handling editors might increase the number and diversity of reviewers who are asked to review.

Demand on the peer review system could be reduced somewhat by sharing of reviews among journals. If a manuscript is rejected from one journal, the reviews carried out for that journal could be used by any journal that subsequently considers the manuscript. This helps ensure that authors revise their manuscripts appropriately, and also cuts down on duplication of effort. Sharing of reviews occurs already on a rather ad hoc basis; at least one ecology journal allows authors of manuscripts previously rejected by leading general-science journals to provide the previous reviews. A formalized system of sharing reviews among journals could work alongside the solution that we propose below.

Finally, we have heard the view expressed that the current system should be maintained as it is not broken, and is, like democracy, the worst system except for all the others. But if the peer review system is not broken now, it could well be soon, and even if the system doesn't break down entirely, the quality of reviews is at serious risk.

Many leading journals attempt to treat the symptoms of the problem rather than the underlying causes, for example by rejecting many manuscripts without review. This increasingly popular practice has the undesirable side effect of increasing the randomness of decision making. The decisions of even highly competent editors are inevitably more stochastic than decisions aided by the larger sample of expertise and opinion provided by multiple external reviews. One of us recently had a manuscript rejected without review by one leading journal, only to have essentially the same manuscript reviewed and accepted by a second, equally leading journal, with the best reviews he or any of his co-authors had ever received! Further, the increased perception of stochasticity arising from doing away with external reviews has the perverse effect of encouraging authors to quickly resubmit rejected manuscripts to other journals, and to submit to highly selective journals in the hopes of “getting lucky,” thereby increasing pressure on the peer review system. Finally, it seems likely that at some point the flood of submissions will become too large for editorial boards to handle in a timely manner even without seeking external reviews, at which point the peer review system truly will be broken. Rather than relying on a “treatment” that makes the underlying “disease” even worse, we believe proactive efforts should be made to improve on the existing system of peer review.

What is the ideal peer review system?

In order to improve the system, it is helpful to envision an ideal system. In an ideal system, how much reviewing would an individual do? We and others suggest that authors should review about three times as many manuscripts as they submit (Riisgard et al. 2001). First, this would ensure sufficient reviews in the “commons” for the submissions made (about three reviews per submission). Second, it would ensure that an individual's contribution to the peer review system (reviews provided) balances the demand being placed on the system (reviews received). Individuals who make more submissions should do more reviewing; everyone does their fair share. But how to achieve or even approach this ideal if appeals to altruism and professional ethics are ineffective, proposed incentive schemes unworkable (Hauser and Fehr 2007), and penalty systems unfair (Riisgard et al. 2001)? We suggest taking advantage of modern technology to privatize the reviewer commons.

The majority of journals now use online systems to handle both submissions and reviews. These systems could be used to link number of reviews with number of submissions. We propose that authors “pay” for their submissions with credits, called PubCreds, “earned” by doing reviews. Submission of a manuscript costs three PubCreds, while a completed peer review pays one PubCred. Every individual would have an account held in the central “PubCred Bank.” Their account would be credited when they carry out a peer review, and debited when a manuscript is submitted. Individuals could view their account balance and transaction history on the PubCred Bank web site. We suggest that the PubCred Bank also log requests to review that have been declined, and the reason for declining (the reasons for this are explained below). Critically, submission of a manuscript to a journal would be possible only if an individual's account balance contained sufficient PubCreds (we discuss this amount below).

Such a system would create a direct link between an individual's reviews and submissions. Specifically, one has to do three reviews for every submission. Authors are free to make as many submission as they like—so long as they do a corresponding amount of reviewing. Each author only has access to the reviewer commons by the amount he or she contributes to it. This is effectively a means of privatizing the reviewer commons. Alternatively, our proposal can be thought of as simply making overexploitation of the reviewer commons impossible. Furthermore, individuals who attempt to publish smallest acceptable units (the “salami slicer syndrome”; Hochberg et al. 2009), or who resubmit rejected manuscripts without appropriate revision (Riisgard et al. 2003), would be free to do so, but would pay an appropriate price.

Getting the details right

The core idea just outlined leaves unspecified many details about how the PubCred Bank would operate. Without wishing to diminish the importance of these details, we believe the core idea is sufficiently simple and powerful to be robust to many different choices about the operational details. Next we suggest how some of the operational details could work in practice, while emphasizing that our suggestions could be modified while maintaining the core idea.

  1. Starting up the PubCred Bank would require careful consideration and monitoring. We suggest that, prior to startup, the bank could be used simply to gather information for a period of time, with payments for reviews and charges for submission logged, but no limits on the account balance required for submission (i.e., unlimited overdrafts available). This information could be used by individuals to gauge their contribution to the peer review process. This information also could be used to judge the scale of the problem (what fraction of authors are “cheaters” exploiting the peer review commons, and to what degree [Hochberg et al. 2009], to calibrate the exact amounts paid and charged, and to make any required modifications.

  2. Individuals early in their careers need to be allowed time to earn PubCreds. This could be implemented by allowing individuals to earn PubCreds by reviewing for some period of time (say, one year) before beginning charges for submissions. Alternatively, all authors could be charged, with a limited overdraft facility allowing for payment for submissions during an individual's early career stage (see point 5 below).

  3. Any author on multiauthored manuscripts should be permitted to pay part or all of the submission fee, so long as the authors collectively pay the entire fee. All that matters is that, collectively, the author(s) of each submission do enough reviewing to cover the cost that they, as a group, create in the reviewer commons.

  4. If reviews are already available, for example from a previous submission of the manuscript to a different journal, the author is not charged for these reviews a second time. This would allow the PubCred Bank to work alongside any increase in the sharing of reviews among journals.

  5. There may occasionally be good reasons why, even collectively, the authors of a manuscript have not earned enough PubCreds to pay for a submission. Perhaps the sole author is a graduate student, or an academic who happens not to have been asked to review for a long time. We suggest that all authors have a limited “overdraft” facility on their PubCred accounts. We imagine that few if any academics will need to go into debt by more than 10–15 PubCreds, even at the beginning of their academic careers. A period of data collection prior to startup could determine whether this is correct (see point 1).

    An alternative approach to allowing all authors a limited overdraft facility would be to allow authors who lack PubCreds a very small number of free submissions per year (say, one or two), with reviewers and handling editors still being paid. Analogously, many journals owned by scientific societies forego page charges on one submission per year from society members who certify that they cannot afford the page charges. In order to avoid undermining the entire system, the limit on free submissions would need to be set very low, and could not be applied on a per journal basis (i.e., authors lacking PubCreds would be allowed one or two free submissions per year in total, not one or two per journal).

  6. The academic community should decide if the balance and turnover of all PubCred accounts is publicly viewable, so that anyone could see which individuals do more than their fair share of reviewing and editing, and which individuals do less. We suspect that this might provide a powerful informal incentive for authors not to run up overdrafts. This kind of naming-and-shaming might be viewed as over-aggressive, but has the potential to create self-policing, and might even do away with the need to enforce overdraft limits centrally.

  7. Reviewers providing late, superficial, sloppy, or inappropriate reviews should receive no PubCreds for doing so. The handling editor would decide whether a review was too late, superficial, or sloppy to be useful, and therefore to earn a PubCred. We believe that most handling editors are sufficiently frustrated by the frequent provision of extremely brief, cursory reviews that they would have no hesitation in refusing credit for such reviews.

  8. We suggest that handling editors should receive 0.5 PubCreds for every manuscript they handle. In our experience, editorial decision making is not a trivial task, but is not as time consuming on average as providing an external review, at least when the reviews are of good quality.

  9. If a manuscript is rejected without review, 2.5 of the 3 PubCreds paid to submit the manuscript should be returned to the author (the remaining 0.5 “pays” for the editor's decision; see point 8). We emphasize that, if our proposal works, it should no longer be difficult to find willing, careful reviewers, and so journals could return to their traditional practice of sending a large majority of manuscripts out for review. One of the main goals of our proposal is to eliminate the perceived need to reject a large fraction of manuscripts (>60% at some leading ecology journals) without review.

  10. Authors should not be charged for submitting invited revisions, and referees and handling editors should not be paid for reviewing invited revisions. In our view, a referee who agrees to review a manuscript should agree to review an invited revision, on pain of not being paid any PubCreds (payment would be made following the final decision on the ms). Review and handling of revisions typically is much less time consuming than review and handling of new manuscripts, which justifies our suggestion here.

  11. Invited manuscripts (contributions) should be paid for just like any other manuscript, if they are to be handled and reviewed in the same way.

  12. Journals would not be able to compete for submissions by offering reduced submission fees or free submissions, or compete for referees by offering higher payments. Any journal participating in the system would be obliged to charge, and pay, the same PubCred fees.

  13. Some journals/editors maintain a blacklist of people whom they will not ask to review papers. This may be because these individuals always refuse to review, are too slow or unreliable, provide unhelpfully brief reviews, and/or make inappropriate statements. Perhaps the PubCred Bank could change the behavior of some reviewers. Otherwise, blacklisted individuals would have to rely on PubCreds earned by co-authors.

The challenge of implementation

Setting up such a cross-journal reviewing and submission market was infeasible just a decade ago, but is eminently feasible now due to the pervasive use of electronic submission and review systems. These systems make possible automation of a large part of the running of the market. Many journals already collect and save the key data (the names of authors, reviewers, and requested reviewers) required to implement our proposal. The obstacles to implementation are not technological.

While our proposal could be implemented by journals individually (e.g., Ecology could pay “Ecology-PubCreds” to its own reviewers and handling editors, and accept only “Ecology-PubCreds” in payment for submission to Ecology), but we believe this would be of limited value at best. Requests to review often do not come from the same journals to which an author wishes to submit in the near future. Accordingly, many authors who wish to submit to, say, Ecology, may have difficulty accumulating Ecology-PubCreds. This situation would be bad for Ecology (which presumably wants to attract good manuscripts, regardless of authorship), as well as for the authors. And reviewers who do not plan to submit to Ecology in the near future may well decline requests to review for Ecology, because they have no plans to spend the resulting Ecology-PubCreds. In practice, a single journal would be unlikely to implement its own “currency” in this way, for fear of authors abandoning that journal in favour of journals operating under current rules.

We argue that prospective authors owe a duty of service to the field as a whole, not to any particular journal. Authors should ideally be able to use PubCreds earned from reviews for any journal to pay for submissions to any journal (a “common currency”). Under our system, reviews for all journals would earn PubCreds, and so no journal would need to fear any difficulty in finding referees. The more journals that participated, the better the system would work and the more attractive it would become for other journals to join.

Perhaps the two greatest obstacles to implementation are financial and organizational. Who should or even could cover the very real development, maintenance, and improvement costs of the PubCred Bank? And could publishers, journals, authors, and reviewers all agree to and support the new system?

First, it is worth noting that that journals already pay for their own online manuscript handling systems. The additional costs of creating the PubCred Bank and systems for receiving and transmitting information from and to these manuscript handling systems might not be large. (However, we have insufficient information to gauge these costs.) It is possible that, in time, the whole of these costs could be paid for by advertising. The central PubCred web site likely would be very frequently and reliably visited by all academics, which might make it attractive to advertisers looking to reach academics. One can imagine the PubCred Bank run as a not-for-profit organization, with any net profits from advertising used to promote and fund science.

Second, many journals are owned, jointly owned, or published on behalf of scientific societies by a small number of publishers, and almost all the journals published by a given publisher use a single online manuscript-handling system. In practice, only a few major publishers would need to come to an agreement as to how to share costs and revenues associated with developing, maintaining, and using the PubCred Bank. It might even be feasible for some major publishers and/or societies to implement the Bank unilaterally. For instance, if Wiley-Blackwell were to implement the Bank, using Wiley-PubCreds (earnable from, and payable to, only journals published by Wiley-Blackwell), many of the leading journals in ecology and evolution (including Ecology Letters, Evolution, Journal of Evolutionary Biology, Molecular Ecology, Oikos, and the five journals of the British Ecological Society) would be included. Many authors and reviewers would not want to give up submitting to and reviewing for all Wiley-Blackwell journals, and so a “Wiley-PubCred Bank” should not lack for “customers.” However, the best solution is for multinational investment in developing the PubCred Bank, with commitments from as many journal owners and publishers as possible, rather than a piecemeal approach to implementation.

Once costs and organizational issues were addressed, implementation could be rapid. An experienced web developer estimated two to three person-months to implement the PubCred Bank web site and the systems for receiving information from the variety of web-based submission and review systems already in existence.

Helping reviewers

We originally conceived this privatization of the peer-review system to ensure the timeliness and quality of reviews, and to prevent the tragedy of the reviewer commons. However, we believe that the PubCred Banking system would have beneficial side effects. It would allow individual academics to better gauge how many reviews they conduct, by measuring the number of reviews against the number of manuscripts submitted. While the system would be built on the idea that submission should be paid for in terms of reviews, there is nothing to prevent the altruistic among us from racking up positive PubCred balances. Indeed, such a centrally quantified contribution, logged by the PubCred Bank, could lead to more direct recognition of reviewing activities by employers (Perrings 2006).

Individuals reviewing too much could be seen as creating a problem, by monopolizing PubCreds and effectively preventing other people from submitting their manuscripts. In our view, formal mechanisms that prevent willing reviewers from carrying out reviews are not a good or necessary response to this potential problem. Competition among academics for reviewing tasks is perhaps not something to be discouraged. In practice, we suspect that few individuals are likely to accumulate large PubCred balances. Under the current system, a major reason some altruistic individuals are asked to do more than their fair share of reviewing is because other individuals decline requests to review. Under the PubCred system, many fewer requests to review will be declined, and so the altruistically inclined will not be asked to review so frequently. Further, if PubCred balances were publicly viewable, we suspect that editors would often request reviews from individuals with small PubCred balances, as these individuals are likely to agree to review.

Conclusions

A perfect system seems unlikely. Some manuscripts might remain difficult to find appropriate reviewers for, perhaps due to the small number of qualified people. Individuals in positions of power could coerce others to do their reviewing, in return for authorship. People are likely to find ways to cheat, and the PubCred Banking system may have to be modified in response.

But potential drawbacks to our proposed system must be weighed against the actual drawbacks of the current system, which are widely recognized and increasingly serious. Over the last 10–15 years, ecology has increasingly shifted from a system in which papers were rarely rejected without review for reasons of content, to a system in which this is rapidly becoming the norm. The current system has the virtue of making rapid decisions, but also risks compromising the perceived and/or real quality of the decisions being made. Ecologists at least need to have a discussion about whether rejection without review, decided quickly by one person, should be the most common outcome of submission. And are we satisfied with a system that can reward the most selfish among us? And what happens if the flood of submissions becomes too much for editorial board members to cope with in a timely manner even without obtaining external reviews?

Our proposal may seem radical, but it is merely a technologically straightforward way of bringing about the ideal or nearly ideal peer review system we believe all academics would like to have. This kind of change is necessary in order to ensure that the peer review system can continue to perform its essential functions as well or better than it has in the past. It was an ecologist who first recognized the tragedy of the commons. Surely ecologists can solve one.

Acknowledgments

Marcel Holyoak, Mark McPeek, Hans Ulrik Riisgard, Peter Morin, and Liz Ferguson provided useful comments about our ideas. O. L. Petchey is a Royal Society University Research Fellow.

Ancillary