This paper provides an extended guide to reviewing for ESPL in particular and geomorphology in general. After a brief consideration of both how we choose reviewers and why we hope that reviewers will accept, I consider what makes a fair and constructive review. I note that we aim to publish papers with the rigour (r) necessary to sustain an original and significant contribution (q). I note that judging q is increasingly difficult because of the ever-growing size of the discipline (the Q). This is the sense in which we rarely have a full appreciation of Q, and our reviews are inevitably going to contain some bias. It is this bias that cannot be avoided (cf. Nicholas and Gordon, 2011) and makes the job of ESPL's Editors of critical importance. With this in mind, I identify six elements of a good review: (1) an introductory statement that explains your assessment of your competences in relation to the manuscript (r and Q); (2) a summative view of the originality and significance of the manuscript (q) in relation to Q: (3) a summative view of the methodological rigour of the manuscript (r); (4) identification and justification of any major concerns; (5) identification of any minor issues to be corrected if you think the manuscript merits eventual publication; and (6) note of any typographical or presentation issues to be addressed although this latter activity is also an editorial responsibility. In addition, I note the importance of a constructive review, grounded in what is written in the manuscript, justified where appropriate and avoiding reference to personal views as far as is possible. I conclude with a discussion of whether or not you should sign your review openly and the importance of reviewer confidentiality. Copyright © 2012 John Wiley & Sons, Ltd.
This article is based upon a contribution invited by Jens Turowski at a session on reviewing in geomorphology at the European Geosciences Union in April 2011, specifically designed for younger geomorphologists. While Mike Ellis drew upon his experience of grant reviews at the US National Science Foundation, I was asked to contribute material on how to review a manuscript for a geomorphological journal. In preparing the material for this session, I realized that I was encoding into routine a series of practices that I look for in the reviews we receive at Earth Surface Processes and Landforms (ESPL). In looking across reviewing practices more widely, across a range of science journals, I detected a common theme: a tendency to see that reviewing is something that we typically learn by osmosis; and a number of journals have flagged the need to make sure that reviewers (as well as authors) receive proper training (Isaacs, 2004; Shugan, 2007; Moizer, 2009). Given the importance of publication to authors, the time that authors invest in preparing their manuscripts, and the critical role that reviewers play in defining the corpus of knowledge that constitutes our discipline, drifting into reviewing practice by osmosis carries risks: that our submitted manuscripts, and their authors, might not get the attention that they deserve; and, most seriously, that reviewing practices acquired through experience are not subject to the proper scrutiny and formation that they require. Thus, and with the support of ESPL's Advisory Board, I have prepared this short manuscript on how to review a manuscript in geomorphology. A shorter version of the manuscript is available in the form of a webinar (powerpoint presentation with oral commentary) through the journal's home pages. As such, the manuscript also represents a formal statement on ESPL's policy and practice in relation to reviewing manuscripts.
Why should I review manuscripts?
There is a surprising debate about why we should review manuscripts. Rarely are we paid. Most of our reviewers are known only in terms of their identity to members of the ESPL Editorial Board, because of our policy relating to confidentiality (see below). Recognition is generally hidden. Reviewing receives variable credit in terms of performance evaluation, promotion and the wider job market: it is negligible in some countries; more significant in others; the returns are few. An activity that is unpaid, hidden and with few returns (Moizer, 2009), but still required (Lane, 2011) seems characteristic of some kind of 19th century workhouse activity. Recourse to rational economic explanations of the motivations for reviewing in the face of these ambiguous incentives (Moizer, 2009) suggests that most of us would avoid reviewing; and in rather bleak assessment, Moizer observes, ‘once the formalities are observed, the referee is free to indulge him or herself. They might consider the common good, but personal interests must inevitably also play a part’ (2009, 292).
I disagree with such an interpretation. Finding reviewers is hard, but at ESPL we still manage to deliver over 1000 reviews to authors per year. The quality of the vast majority of those reviews is excellent when judged against the kinds of criteria that make a good review and which are outlined below. I think that the community still shares a sense that reviewing is one of the ultimate privileges of academic life, one where we are being asked to provide advice on whether or not a piece of science has met those community standards that we set for whether or not a manuscript should become a permanent record of our knowledge. It is very rare for a published manuscript to be retracted, or even corrected, and so reviewing is the hurdle that academic work must pass in order for it to become ‘permanent’. Reviewing makes a difference. There is an implicit reward in being invited to review; a reward made because an Editor thinks you will make this difference. All of ESPL's reviewers are active researchers, and so it might also be argued that we review because of a second implicit: a notion of co-operative exchange; by reviewing, you obtain an implicit right to have your own work reviewed. We might add that reviewing carries benefits in that we get to learn about exciting research well before the rest of the community. This is something that must be cautioned as reviewers are strongly constrained by how much they can do with this knowledge without breaching ethical concerns related to the ownership of intellectual property (Anonymous, 2008), especially in relation to work that is, in due course, not published.
However, I think that all of these reasons hide a deeper and core reason for reviewing. Evaluating a manuscript, the mental exercise of providing constructive criticism, is one of the inherent pleasures of being an academic, provided we, as Editors, have filtered the very worst of manuscripts out of the system before you see them, and guide you through formalities (Moizer, 2009) that encourage those reviews to be fair and constructive (see below).
What happens when a manuscript is submitted to ESPL?
The act of submitting a manuscript to ESPL involves signing up to a series of rules regarding submission. These rules cover conflicts of interest, authorship, data ownership, plagiarism, redundant publishing and copyright. They also relate to minimum standards in relation to presentation of the manuscript. Thus, the manuscript is screened by our Assistant Editor to make sure it conforms to our submission rules and the Managing Editor confirms its fit to the journal. If it does fit, an Associate Editor is assigned to the manuscript. They will read the manuscript. Provided that they are convinced that the manuscript is of a sufficient standard in terms of quality and rigour (see below), they will aim to appoint two reviewers to the manuscript.
How do we choose our reviewers?
Evidence suggests that there are few if any good predictors of what makes a good reviewer (Black et al., 1998). It would be valuable if we could rely solely on previous experience of those who have undertaken good reviews for us, as such information has been shown to make a positive difference in other fields (Godlee et al., 1998). Likewise, again in other fields, active researchers and younger researchers have been shown to produce better reviews (Godlee et al., 1998). Relying on previous experience would seem to be a safe bet, but that would not produce enough reviewers and, of course, it is not a sustainable strategy over the timescale of decades (we would see a progressive attrition of our reviewer pool). This is why choosing our reviewers is often about choosing people of whom we may have little reviewer experience, including younger academics, and where, therefore, there can be an element of serendipity. It makes training our reviewers even more critical a task.
In general, if you have been invited to review a manuscript it is because we think that you are an active researcher with specialist expertise in the topic of the manuscript, such that you are able to provide us with a reliable judgement on what the manuscript contains. This expertise might not relate to the entire manuscript. Especially with some of the more innovative manuscripts that we get, we may need an expert in one element of the manuscript (e.g. a methodology) but not necessarily one who can cover all of a manuscript. Indeed, we are aware that someone who is an expert on a method may not be the best person to evaluate a manuscript's originality and significance (Shugan, 2007). We are also convinced that you can provide us with a review that will be of value to the decision we have to take. We do not know of any serious conflicts of interests between you and the manuscript or its authors. For instance, we would view someone as being in a conflict of interest if they were reviewing a manuscript authored or co-authored by someone in the same University. Likewise, we would tend to avoid approaching someone who it is clear to us has collaborated recently with an author of the manuscript. Things can get a little grey and so we do use our judgement (e.g. does the same national research institute count as the same institution?).
Authors are required to nominate possible reviewers when they submit their manuscript. These are extremely useful but perhaps not for the reason that the community might imagine. We may use one of the authors' suggestions, although we would normally look for at least one reviewer who has not been nominated by the author and sometimes we do not use author nominees at all. Interestingly, we find author nominated reviewers to be as critical if not more critical than reviewers not nominated by authors. More importantly, though, the nominated reviewer can provide us with a foothold into the wider community that might be relevant for peer review. We do use internet resources, including bibliographic databases, to find reviewers, and a foothold into a network of scientists can sometimes be very valuable. Of course, nominated reviewers, databases, our experience, etc. are all critical, but so are new reviewers. Finding these is harder and if you would like to review for ESPL, you have just finished a PhD or are a post-doc, then we would welcome hearing from you. Please send a short CV to email@example.com.
In summary, our choice of reviewers will reflect: (1) the topic of the manuscript; (2) the methods that have been used; (3) the published work that has been referred to in the manuscript; (4) a reviewer's track record of reviewing for the journal, unless they are a new reviewer; and (5) who we have asked to review recently (we try to avoid asking the same person for more than 2 or 3 reviews per year).
What should I consider when deciding whether or not to accept?
The first and most important question is whether or not you agree with our assessment that you are appropriate to review the manuscript in terms of your expertise (Moizer, 2009). We do make mistakes, and if you receive an abstract that you feel does not match your expertise it is better to decline to review it. Remember, however, that the ultimate reviewers of the journal are its readers and sometimes this means that the best reviewers are those who understand the bigger picture and who can then take a broader perspective on a manuscript. Note that if you have accepted a manuscript, it is very well worth reading the manuscript straight away so as to confirm that your decision to accept does hold. Sometimes, abstracts are not as reliable as they might be in relation to the content of a manuscript. It is better to know when you accept a manuscript that you no longer think you have the expertise rather than waiting until the manuscript is due.
The second question is whether or not you have the time to do it. It is much more valuable to decline a review that cannot be completed in the time available (or a little more than that, as we can always extend this time a little) than to accept a review and not to do it. We stop searching for reviewers for a manuscript when we have two who have agreed to review and this is for good reason. Our pool of reviewers is finite: if we lock in more than we will eventually need, we will lock them out of being available to review for the wider community (Lane, 2011). However, if one of those two reviewers does not deliver, we have a problem. We have a target time to first decision (i.e. the time from when we accept a manuscript into our review system) of 50 days: 7 days for pre-review assessment; 28 days for review; and up to 15 days for Associate Editor recommendation and Managing Editor decision. Delivering on this means that any delay by a reviewer is a problem.
The third is whether or not you think you have a conflict of interest with the authors, in which case you should decline to review the manuscript. As Editors, we are often ‘in the dark’ about these. For instance, we may not know of existing or planned collaborations. We may not know about personal situations. We may not know of entrenched disagreements, planned changes of personnel between institutes, etc. Thus, in answering this question there is a simple guide: do you think you could produce a fair and constructive review? If yes, then you should accept. If, in answering yes, you do have information that might be perceived to be a conflict of interest, then it is best to let us know before accepting, so that we can make an informed assessment.
What do you mean by a ‘fair’ review?
It is worth observing that there is no such thing as an unbiased review. We are all biased by the particular knowledge we have and the particular methods that we think are most appropriate for our domain of research. We all have preferences. We all differ in the extent to which we can put our biases to one side. As all reviews are biased to some degree, we insist on all manuscripts being carefully considered by at least two reviews and at least one Associate Editor and the Managing Editor. It is worth introducing here a conceptual analysis, introduced by Ellison (2002) (see also Moizer, 2009): q – r theory. Academic papers can be evaluated along two dimensions: r, their rigour, including the quality of their argument and presentation; and q, their inherent interest. Ellison's argument is that our own experience of being reviewed forces us to apply standards to our reviews of others' work, one that inevitably grows our emphasis on r, even to the detriment or loss of q: papers that have an inherent interest lose to those that are less interesting but ever more rigorously presented (see also Benos et al., 2007). While I have no reason to find that this argument either holds or does not hold for ESPL, it strikes me that it overlooks the necessary correlation between q and r. However interesting a manuscript might be, we have to be certain that its interest is sustained by the data and analyses upon which it is based, as well as how those data and analyses are presented. However rigorous a manuscript may be, we have to be certain that it is saying something of interest to the discipline. We need our reviewers to consider both q and r. Here is the problem: judging r is practically more straightforward than judging q, because the latter is not only dependent upon a reviewer's knowledge of the manuscript itself, but the much wider field within which it sits. If we define that field as Q, the proliferation of journals and the advent of new communication systems, coupled to the growth of research activities globally over the last 5 decades, means Q is becoming larger, and so the ratio q:Q is falling (under the assumption that the capacity for our appreciation of q to increase is finite). Given the growth in pressure to publish both in general and at ESPL, judging q becomes even more important: we cannot publish papers simply because they make it on r alone. Hence, q now matters more in our evaluations, but the parameter that allows us to evaluate q effectively is a high q:Q, which itself is declining. This problem has been bemoaned by many in terms of rejection rates that ever increase (Moizer, 2009), and confirmed empirically in studies that have shown that the decision to reject can be, in some instances, a lottery: there are famous examples of research that has profoundly changed what we know, even though that research was declined publication on one or more occasions (Shugan, 2007). I'll consider the question of rejection below. But, what my discussion emphasizes is that we are all biased reviewers because we do not have a complete knowledge of Q. Hence, a fair review relates the manuscript's q to the reviewer's understanding of Q, not forgetting r. As noted above, even though a review may be biased, there are some kinds of bias which are simply not acceptable.
How do I decide what my summative recommendation should be?
By reading the manuscript and then not worrying unduly about this question. We do not make decisions using summative recommendations (e.g. ‘accept’, ‘minor revision’, etc.) but rather the detailed comments provided by the reviewer along with our own reading of the manuscript. Further, the decision we take is an editorial decision and not a reviewer's decision. The categories that we provide can and do mean different things to different reviewers. Thus your summative recommendation is simply that – a recommendation that we will consider carefully. We do give our reviewers an indication of the eventual decision we make: if our decision is different to that which you gave us, you should not be unduly alarmed and it certainly should not be taken as a criticism of your recommendation. Rather, it reflects that we take our decisions in the round.
But, you still ask for a summative recommendation, and so what do the categories mean?
There are four primary categories a reviewer can choose from. First, we have accept: a manuscript does not need any further revision, even minor topographical changes; this is a category that is sometimes appropriate if you are asked to consider a revised manuscript but is very rare for the first draft of a manuscript. If there is any further work required in the manuscript, even if these are only typographical decisions, then minor revision is the recommendation. We handle manuscripts revised after a recommendation of minor revision very quickly, with decisions generally taking under a week if the revisions have been undertaken thoroughly.
Second, we have minor revisions: the manuscript needs some revision, normally in relation to matters of clarification, expression or presentation; but there is no doubt about the quality of the science that is presented (r), its originality (q) and its significance (q). Minor revisions could be quite significant in terms of the amount of work required. The boundary with major revision is really regarding whether or not the science in the manuscript is rigorous, original and significant. If work were required on any of these, then major revision is appropriate.
Third, we have major revision: the manuscript falls short in some way in relation to its rigour (r), its originality or its potential significance (q), requiring significant additional work. This could be a substantial addition of literature, the re-analysis of data, changes to the representation or interpretation of data, modification of the discussion or a rethink of the Conclusions. For first decisions at ESPL (i.e. decisions taken the first time we consider a manuscript), major revision is our most common recommendation. Given the kinds of changes required, we would normally expect a manuscript revised after recommendation of major revision to be sent for further external review. But, by choosing major revision we are sending a signal to the authors: we think that the manuscript might make it, on the basis of the information that we have. Very occasionally, major revision produces a revised manuscript that, after review, has flaws that have been made apparent by revision. Even if an author has carefully followed the reviewer and Editor instructions, the eventual outcome can still be a rejection.
Fourth, we have reject, which applies to 40 to 50% of manuscripts submitted to ESPL. This may seem high but reflects the fact that we receive many more submissions than we can publish. The category is appropriate if the manuscript has little of originality or significance (it fails on q) and/or has serious flaws in relation to method and to data (r) that mean that even if the manuscript were resubmitted, these would be difficult to address. We ask that reject recommendations are fully justified.
There is a fifth category, that we do not allow reviewers to choose but which we, as Editors, can use: reject, but invite a new submission. There are, in essence, three types of manuscript that get this recommendation: (a) ones that have an interesting idea (i.e. q) but lack the supporting data; additional data need to be collected before the manuscript is likely to sustain the idea (i.e. r); (b) ones that have interesting data (possibly r), but the context for the work and the interpretation of the data are some way from being worthy of further consideration even with a major revision (the q is not clear); and (c) ones that are very poorly presented (there is neither q nor r as far as we can see, but we can't see very far). Normally, the Managing Editor or Associate Editor would have filtered out (c) before we send a manuscript to review, but there can be occasions when the expert assessment of a manuscript reveals poor levels of explanation, especially in relation to methodology. We don't allow Reviewers to choose this fifth category as it is very much a summative and hence an Editorial judgement, that we take in the round.
One thing should be clear, however. Your summative judgement is valuable but, of most importance, are the written comments that you provide on a manuscript, the reasons for you reaching your recommendation.
What do you value in a written recommendation for authors?
There are six elements to a good review: (1) an introductory statement that explains your assessment of your competences in relation to the manuscript (r and Q) and noting any areas where you do not feel that you have the right expertise; (2) a summative view of the originality and significance of the manuscript (q) in relation to Q: (3) a summative view of the methodological rigour of the manuscript (r); (4) identification of any major concerns that you have with the manuscript, and justification of those concerns; (5) identification of any minor issues to be corrected if you think the manuscript merits eventual publication (if you are going to recommend minor revision or major revision); and (6) note of any typographical or presentation issues to be addressed (if you are going to recommend minor revision or major revision), although this latter activity is also an editorial responsibility. Nicholas and Gordon (2011) provide a valuable introduction to how to approach reviewing and then writing a review.
In addition: (a) your comments should be related to what is written in the manuscript; (b) points made should be grounded in matters of fact, and supported by evidence that is available to the authors; (c) you should acknowledge what is good (its r and its contribution to Q) as well as what is bad; (d) you should avoid any reference to personal views unless these are justified; (e) it is important to be clear about what is required to improve the work, even if this might still not make the manuscript publishable; and (f) it must be written constructively.
Your comments should be as long as you think appropriate. But note that a review that is too long may be just as inappropriate as a review that is too short. You are judging the quality of a piece of work as written by the authors, and neither acting as a co-author to improve a manuscript ready for submission to a journal (Shugan, 2007) nor as a supervisor directing a student as to the paper that should be really written. The author(s) should be free to write the article that they want to write (Shugan, 2007; Moizer, 2009). The authors should do the work in addressing the concerns that you have and so your focus should be on the concerns that you have, and a steer as to what might be done, not the doing of the work itself. Sometimes, authors query what is being asked of them, and it is an Editor's responsibility, not a reviewer's responsibility, to provide clarification.
Crucially, your review has to be constructive. There are two dimensions here. First, we commission approximately 700 reviews per year at ESPL. Of these, I would estimate that for between 1 and 2% (somewhere between 5 and 15) we dismiss reviews received because the review is written in a form that is unfairly hostile to a piece of work. This is not the same as a review that is incorrectly critical of a piece of work and we find, at editorial level, that criticism to be unfounded. Rather, it is where the review is written in a way that is not appropriate. Some Editors suggest excising comments from reports (e.g. Sheppard, 2000; West, 2002). We do not. Rather, we dismiss the review, no longer use that reviewer and appoint a new reviewer. The authors will still receive the dismissed review but with a clear confirmation of the actions that we have taken. Second, criticism is what we want, in the more narrow definition of the term: skilful analysis of the manuscript and its contents; rather than analysis that is censorious or fault finding (Isaacs, 2004). However, a review that has a skilful analysis can be delivered in a way that appears censorious through its delivery, which may make a major difference both to more junior authors (Sheppard, 2000), but also to how much faith, as Editors, we have in the reviewer (see also Anonymous, 2011). My sense is that (on the basis of approaching 1200 ‘major revision’ or ‘reject and resubmit’ decisions since January 2008) a constructively written review is more likely both to result in a revised manuscript and a manuscript that has been thoroughly revised in relation to what is being asked. Geomorphological science can take a lot of time, especially when it involves long and arduous fieldwork, and that is before the effort put into analysing data and writing a manuscript. Some commentators have bemoaned high rejection rates (e.g. 90%, Moizer, 2009) and manuscripts that are returned for revision but never revised contribute to these rates. Taken together, these imply that we have a duty to avoid rejecting potentially publishable work through including in our reviews an implicit discouragement.
Should I make all my comments available to authors?
ESPL, as with most journals, allows reviewers to make confidential comments that cannot be seen by authors. Also, authors do not see the recommendation you make unless you include it in your written comments for the authors something that we prefer that you avoid, as with other journals (West, 2002). This is because you, as a reviewer, do not decide the fate of a paper. There are some authors who expect the reviewers' judgement to be final and who become very concerned when a reviewer's recommendation diverges from the editorial decision taken on a manuscript. A reviewer's judgement cannot be final as the decision to publish a manuscript is not a reviewer's one but an editorial one, albeit informed by what reviewers think. At submission of a review, the reviewer has only seen their own assessment of a manuscript, and will also not have seen the other set of manuscripts that are being evaluated. As we make clear in our statement to authors, we cannot publish all manuscripts that reach the required standard in terms of r, and sometimes the effective evaluation of q is only possible by comparison with other manuscripts that have been across the Editors' desk. Finally, it can be quite confusing to an author to read a lenient review report from a reviewer whose confidential comments are more critical, found to be of substance by the Editor, and cause them to make a decision that is more negative than those comments suggest (Turka, 2009; Anonymous, 2011).
Hence, we generally advise that it is best to make all your review available to authors except your summative recommendation (e.g. ‘major revision’); and that confidential comments should be restricted to matters of a sensitive nature, a self-critical assessment of your own ability (in terms of Q and r) to review the manuscript and any additional justification of your summative recommendation that you feel is necessary.
What should I do if I find something of ethical concern in a manuscript?
ESPL, as with many journals, has a policy statement concerned with what we call ‘irregularities’. They are defined and detailed on the ESPL web-site (http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1096-9837/homepage/ForAuthors.html). Central to our approach is guidance issued by the Committee on Publication Ethics (COPE), which is a publishing industry organization that ESPL is signed up to. Reviewers do provide a role here in spotting concerns such as plagiarism and redundant publishing. If you have concerns then they should be raised in the confidential comments that reviewers can provide. All concerns are investigated and we have specialist software to help us do this.
What happens with my review?
The Associate Editor assigned to your manuscript will read your review. He or she will also read the manuscript, the other review(s) and any other information provided by the author at submission. The objective then is to reach a summative recommendation, which may or may not follow your recommendation. This is an important step because the decision made is a comparative one, especially in relation to originality and significance; and because it is inevitable that how we, as reviewers, assess the same manuscript will vary between reviewers.
If we have reviews that are very different or a review that we do not feel is substantiated by the written comments, then we may seek a third and independent review. Normally, we let authors know about the resultant delay. However, if there is a difference between reviews, then the Associate Editor or the Managing Editor may be able to balance them to reach a more rapid outcome without the need for a further review.
Finally, the Managing Editor will look at the manuscript, all reviews and the Associate Editor recommendation so as to reach a final decision. In this final decision, reviewer comments are very important in helping to define the revisions that will be asked for.
Should I sign my review openly?
This is hotly debated. In an ideal world, we would all welcome critical reviews as a means of improving our work. In theory, openly signing (unmasking) a review would probably make us write more constructive reviews because we are openly accountable to them (Moizer, 2009). But, the empirical evidence is very mixed. McNutt et al. (1990) found that Editors handling unmasked reviews tended to find them more constructive and courteous and authors receiving unmasked reviews tended to find them fairer. In other studies: unmasking may lead many of the critical elements of manuscript evaluation to be scored more highly than masked reviews, if not the scoring of the manuscripts as a whole (McNutt et al., 1990); does not appear to lead to better reviews when evaluated quantitatively from the perspectives of Editors (van Rooyen et al., 1999a, 1998); does not lead to any significant change (increase or decrease) in the detection of intentionally introduced errors (Godlee et al., 1998); does not necessarily cause authors to rate the quality of the reviews they receive more highly (van Rooyen et al., 1999a); but did significantly increase the probability that a reviewer would decline to review (van Rooyen et al., 1999a). There seems to be no clear empirical case for unmasking on the basis of empirical evidence (van Rooyen et al., 1999b) and possibly some evidence that finding reviews where reviewer identities will be unmasked may be more difficult (Benos et al., 2007).
Whether or not these questions translate into geomorphic science is perhaps academic and even overlooks the social reality of the review process. Reviews are steps towards a decision on a manuscript being taken (although not the final decision) and at least some of these decisions will involve rejection, which may have material outcomes for an author or authors (e.g. promotion; tenure). We must also not forget that reviews exist to be critical and reviews are critical steps in deciding when a manuscript should be made available to the wider world as a peer-reviewed piece of work. Think about this in terms of Popperian falsification. Type 1 errors (rejecting the null hypothesis when it is true) are more serious than Type 2 errors (accepting the null hypothesis when it is false) because a null hypothesis that has not been falsified should be subject to ongoing testing, and so can always be open to later falsification. A Type 1 error is finite. The same is true of manuscripts but we have choice over our null hypothesis: (1) all manuscripts should be published (the null) unless review shows otherwise; or (2) all manuscripts should be rejected (the null) unless review shows otherwise. Some argue that (1) should prevail (Isaacs, 2004) but once a manuscript is accepted and published, there are very few and very limited situations that can lead to it being retracted (examples might include fabrication or plagiarism). However, a manuscript that is rejected, or which has to be revised, can still be reworked for later submission. Thus, rejection of a manuscript that should be published is less serious (albeit still very serious, Shugan, 2007) than acceptance of a manuscript that should be rejected. We need reviews to be critical so as to reject the null (rejection) hypothesis and to make the finite decision to accept.
Unfortunately, the problem with unmasking is that you cannot control the effects of a critical review. From the communications that I have seen, confidentially, over the last four years, I can confirm that there is some disharmony in our community both between individuals and between research groups. The symptoms of this disharmony may be enabled by masked reviews, which allow an individual to hide behind their anonymous assessment. But one of the causes of disharmony is knowledge of the identity of a critical reviewer. For a more senior scientist, having your identity known may be of less importance. For a junior scientist it is potentially very serious. Generally, therefore, we recommend that you: (a) write a fair review; (b) retain your anonymity; and (c) let the Editors, whose names are published in the journal, ‘take the flak’ for the decisions that they, and not you, take. We can often see the symptoms of community conflict, revealed in differences of opinion between reviewers, reviews that are not constructive or well substantiated by comparison with our own reading of the manuscript or in the kinds of references being used in the paper. As Editors, we do not simply act as secretaries, passing reviews to authors. We evaluate those decisions and we are perfectly able to see when a review is unfairly critical, poorly justified or unnecessarily unconstructive and we make sure that the authors understand this. There should be no need to break anonymity in a system of fair reviewing and active editorial control.
The very worst position to take is to only sign your reviews if they are positive – this is profoundly dishonest to the community.
Should I get other advice when I am completing a review?
This is a surprisingly easy question to answer and, in general, the answer is no.
First, in general, only an appointed reviewer should review a manuscript. The invitation to review a manuscript involves a confidential agreement between the Editor and the reviewer. The Editor has to control who reviews a manuscript because only the Editor has all the knowledge that the authors have provided them at submission. There can be very good reasons for some individuals not being involved in the review of a particular manuscript and these may not be known to the reviewer. Further, we have to be sure that all manuscripts get equitable treatment and are not subject to significantly more (or to less) review than others. So, you should not ‘co-review’ a manuscript.
Second, you may feel that there is good reason for you seeking advice from a colleague who has expertise in a particular area that you do not have. Again, this is hard to do without breaching the confidentiality between Editor and Reviewer. It is much better to review the manuscript and to flag areas of concern that you have with your review to the Editor, and to suggest your colleague as a third reviewer should the need arise.
Third, if you are producing a review for the first time, perhaps the second, you could consult with a supervisor or a colleague. But, it is important that if you do this, your consultation is over the review and not the manuscript you are reviewing. The colleague should not be made aware of the identity of the authors (i.e. leave their names off the review at first) and the focus should be on the nature of your review, and not the manuscript that is being reviewed.
This paper contains a reflection on reviewing for ESPL as well as some guidelines and principles that we ask our reviewers to adopt. Attached to the manuscript (Supplementary Online Material) is a webinar that provides a succinct guide on how we ask our reviewers to review manuscripts. I hope that this is something that we can use as a community to make sure that our reviews provide fair and constructive assistance to the ESPL editorial team. For the majority, this article is academic because the reviews that they provide are fair and constructive. To those, I extend my full thanks and hope that you are able to continue to assist us.
The ESPL Associate Editors provided very valuable comments on an earlier draft of this manuscript.