The art of reviewing a paper

Authors


Abstract

Key content

  • As with all skills, the art of reviewing a scientific paper can be acquired, refined and perfected.
  • It is not only a compliment to be invited to review an article but the knowledge acquired in the process, along with the continuing professional development credits gained, makes it a valuable opportunity for the reviewer, quite apart from it being a service to the journal and the scientific community.
  • This article describes the basic principles and responsibilities of reviewing a medical paper.
  • A considered peer review gives constructive feedback to the authors, enhances the credibility and value of the journal and is a service to medical science.

Learning objectives

  • To understand the basic principles of reviewing a paper.
  • To be aware of the responsibilities of a reviewer.
  • To know how to write feedback for authors and the editor.

Ethical issues

  • Plagiarism can be committed by the reviewer or author.
  • Does the allegiance of the reviewer lie with the journal or the author?

Introduction

Despite the publication of peer reviewed research articles being the gold standard for the advancement of medical science, there is almost no formal training for peer reviewers. As with all skills, the art of reviewing a paper can be refined over time with experience, but refining the art of reviewing requires a clear, logical and indeed ethical approach to the art. Editors usually choose reviewers on the basis of expertise in a particular area. It is not only a compliment to be invited to review an article but the knowledge acquired in the process, along with the personal accumulation of continuing professional development (CPD) credits, also makes it a valuable activity personally and for the wider medical and research community. By following a certain format most clinicians – even the least scientific – can become proficient reviewers.

The peer review process has multiple aims. Primarily it is to advise editors, helping them to ensure that their journals are publishing papers of appropriate quality that are up to date and evidence-based. However, the reviewer also owes a duty of care to the authors, to give them detailed and fair feedback to help them improve the quality of their work.

This article describes the authors' approach to the basic principles and responsibilities of reviewing a medical paper.

Why review?

Reviewing an article requires time and a considerable amount of thought. The researchers are likely to have spent a considerable amount of time from conception of the idea, through obtaining ethics approval, applying for funding, carrying out the study, and ultimately, writing up and presenting the work. The reviewer is coming to it cold and in a matter of hours can make or break the years of work that have preceded the review! The first principle of reviewing must therefore be that the paper addresses an area of research where you have at least some knowledge of the literature and the subject. The review process is laborious and, of course, beyond CPD points, there is no monetary remuneration. However, the advantages to the reviewer are considerable:

  • It increases the reviewer's knowledge and experience.
  • It helps improve the quality of the reviewer's own work.
  • It can stimulate the reviewer into a new avenue of research.
  • The reviewer will acquire a wealth of bibliography.
  • It looks good on the reviewer's curriculum vitae.
  • There is an altruistic benefit to the scientific community and ultimately to patients.

The art of reviewing can be divided into three stages: accepting the invitation, reading the paper and writing the review.

Stage 1: Accepting the invitation to review a paper

An invitation to review an article should be seen as recognition of your standing in the research community and, therefore, an honour (cup half full). If you see it as an unwelcome burden on your time (cup half empty), however, you should probably decline the invitation. There are certain rules for acceptance that you should apply:

  • Consider declining the invitation if the article is not in your area of expertise. Going beyond your limits may not only harm your reputation and credibility but is not fair to the authors or to the journal.
  • Consider declining the invitation if you (or others) might perceive a conflict of interest: are you doing similar work at your institution? If so, a recommendation for rejection might be perceived as stalling the paper until your own work is published. Have you any previous or present connections with the authors? If so, accepting a paper might be seen as favouritism.
  • Confidentiality is essential and the article should not be discussed or previewed by others at your invitation without the editor's permission. If there is an area of the article that you are unsure about discuss this with the editor and seek the journal's guidance as to whether this aspect of the article requires a separate review. For example, you might need help with the statistics or a technical aspect or procedure described in the paper. If in doubt, ask the editor. Do not seek help without permission.
  • Timeliness is appropriate. If you feel that you will not be able to complete the review in the time frame suggested by the journal then decline the invitation to review. Your delay may mean that someone else will publish similar findings before the authors' work can complete the review process and get into print.
  • If you do agree to review a paper, always do a thorough job.

Stage 2: The review process

Having agreed to review a paper, how should you proceed? There is no single correct approach as each of us gradually develops a technique that we find most efficient. However, there are certain basic guidelines. Our practice would be to read the paper at least three times: the first pass is to develop a feel for what the authors are aiming to do and for their findings. Before the second pass you should establish the aims of the paper and think through how you might have approached the question in hand and then re-read the paper thoroughly for the second time to understand the author's approach thoroughly. By the end of the second read through you should clearly understand the aims of the paper, its methodology, the results and its conclusions. Finally, read the paper a third time as you write your review. You may well need to re-read the paper and your review a fourth time before submission but the time taken to do this reduces with experience. Allow 4–5 hours for each paper.

Before starting to write your review, consider the journal that you are reviewing for. Is it a journal that is broad in its outlook or is it very tightly focused on a specific area: is the paper relevant? The broader the remit of the journal the greater must be the impact of the paper to justify its publication. The READER acronym[1] can be helpful for a reviewer to summarise what to look for: Relevance, Educational value, Applicability, Discrimination (is it valid?), Evaluation (check the quality), and Reaction (how do you feel about it?). Most journals include guides or forms to guide the reviewers to follow a schematic approach. This typically includes: scientific reliability; originality of the work; whether the paper fits within the scope of the journal; the clarity of the research question and whether it has been answered and results interpreted in a methodical manner with a clear mention of strengths and weaknesses. While maintaining the quality of the scientific and technical content of the paper, the general presentation and language have to be of an appropriate standard – something that is rather more subjective than the rest of the review process.

Most papers follow the IMRaD (Introduction, Methods, Results and Discussion) structure.[2] The title and abstract are stand alone components of a paper that convey their own story.

Title

The title should summarise in a very few words what the paper is about without being sensational. The title should probably be commented on both first and last in the review process: at the start one must understand what the title suggests the paper is about and at the end one must be clear that the paper has actually achieved what the title said it would.

Abstract

We all know that an abstract is a stand alone passage that summarises the paper. However, it is surprising how many authors simply cannot effectively précis the ten pages of their paper into one paragraph. It has to be accurate and tie in with the text of the paper. The process is made easier where the journal has a structured abstract format. Whether structured or unstructured the abstract should not understate or overstate what is in the paper. A sensible author should remember that it is the abstract that catches the potential reader's eye: a bad abstract will have the potential reader move on to another paper.

Introduction

The introduction should set the scene. Writing an introduction is a bit like writing a story. Does the introduction tell the story of how a variety of strands of research have led to the point where the current piece of work had to be done? It should describe the thought process of the author from the conception of the question, through the hypothesis, to end in a clearly defined aim or objective. The reviewer is looking for that clarity of thought. The research question should be sensible and relevant. At the end of this section, as the reviewer, you must ask whether the author has a genuine question based on logic and evidence, and if they have presented that evidence completely and in an unbiased way.

Method

This section essentially explains how the research question was answered. It is useful, as the reviewer, to take a moment to sketch out what you would do to answer the question being posed. Then check your thoughts against the appropriateness of the author's design and techniques. Depending on the clinical category of the study the design will of course vary.[3, 4] In general, clinical studies deal with therapy, diagnosis and screening, causation and prognosis. The preferred study design where the effectiveness of treatments, drugs, procedures or interventions is being tested would be a randomised, double-blind, placebo controlled trial (RCT). By contrast, the preferred design for studies that measure the validity and reliability of a diagnostic and screening test would be a cross-sectional survey comparing a new test with an established gold standard. Case–control studies and cohort studies are typically best for studies investigating the causation of illness. Whenever a disease is very rare, case reports might actually provide an acceptable method of study. Longitudinal cohort studies are perhaps best used for studies investigating the prognosis of illnesses.

In an RCT the inclusion and exclusion criteria, randomisation of the groups, and whether the study group was representative of the population must be determined. There should be proper accounting of participants, blinding of investigators, equal treatment of groups and the referee should be alert to the presence of any bias. Power calculations are mandatory to prevent a type 2 error. Type 1 error, statistical significance and clinical significance have to be correctly interpreted.

There are guidelines on how to report the findings of different research study methods (Table 1).[5, 6] These include a specific set of items in the form of a checklist, flow diagram or text to describe what was done and found in the study. The Consolidated Standards Of Reporting Trials (CONSORT) Statement[7] offers a standard and evidence-based way for authors to prepare and report RCT findings transparently, aiding their critical appraisal and interpretation. The latest version (2010) consists of a 25-item checklist and a participant flow diagram of progress through the trial. Likewise, the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Statement[8, 9] was developed by the STROBE initiative, an international collaboration of epidemiologists, methodologists, statisticians, researchers and journal editors, with the aim of assisting authors when writing up analytical observational studies, to support editors and reviewers when considering such articles for publication and to help readers when critically appraising published articles. This reporting guideline includes a checklist of 22 items that are considered essential for good reporting of observational studies. In essence, the reviewer must ensure that the authors have designed and executed their study in as rigorous and watertight a manner as possible. Deficiencies may not critically flaw the paper and the reviewer must guide the editor in this respect.

Table 1. Reporting guidelines for specific types of studies[5], [6]
Type of study Guideline
Randomised controlled trialCONSORT (Consolidated Standards Of Reporting Trials)[7]
Observational studiesSTROBE (STrengthening the Reporting of OBservational studies in Epidemiology)[8], [9]
Systematic reviewPRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)[10]
Economic analysisBritish Medical Journal Economic Evaluation Working Party[11]
Review of observational studiesMOOSE (Meta-analysis of Observational Studies in Epidemiology)[12]
Non-randomised evaluationTREND (Transparent Reporting of Evaluations with Nonrandomized Designs)[13]
Diagnostic accuracy studiesSTARD (STAndards for the Reporting of Diagnostic accuracy studies)[14]
Qualitative studyCOREQ (Consolidated criteria for reporting qualitative research)[15]
Genetic association studiesSTREGA (STrengthening the REporting of Genetic Association Studies)[16]
Tumour prognostic studiesREMARK (REporting recommendations for tumor MARKer prognostic studies)[17-19]
Quality improvement studiesSQUIRE (Standards for QUality Improvement Reporting Excellence)[20]

Results

The results should be simple, straightforward, well presented and credible. Fundamentally, they should answer the research question. A balance of text, tables and figures is important. While too many tables or figures can clutter the paper, too few can make the paper difficult to read. The tables/figures have to be relevant, enhance the paper with appropriate use of titles, legends and abbreviations and they should not duplicate the information in the text. More recently, some journals have offered to publish additional tables and figures online if not in the paper copy of the journal. The data in the results obviously have to be correct but reviewers have no way of checking this although they can raise concerns with the editor. The analysis/interpretation of results and statistical analysis might be complicated and may require a separate statistical review. When you reach the end of the results section reflect on whether the author has actually presented the findings logically and clearly and whether you understand them. If as an invited reviewer and expert in the field you do not, how could the average reader be expected to?

Discussion

The temptation for an author is to represent the results in a more verbose style. That of course is not the purpose of a discussion. The results should be snappily summarised and the authors should point out any deficiencies in their own methodology and explain them. They should then put the research into the context of the existing literature. Finally, the authors are allowed to speculate: they can suggest what they would like to do next (safe in the knowledge that they will be well on with that before anyone else even gets to read their ideas, let alone ‘borrow’ them!).

Conclusions

Finally, are the conclusions accurate? Are they actually supported by the evidence generated in the study? Are the implications for practice and future research clearly stated? There is a tendency for authors to get carried away at this point and claim that all sorts of things have now been proven as a result of their research: the honest proven truth can, surprisingly often, be quite far removed from the paper's conclusions!

Acknowledgements

Any contribution to the paper should be acknowledged if the contributors are not named as co-authors.

Conflict of interest

The authors have to declare a conflict of interest if any exists.

Funding

Any funding must be mentioned: it might have conflict of interest implications.

References

The references must be in the correct format for the journal. However, the referee must ask if the number is appropriate? Are there too many self-references? Are there missing citations? Are the references up to date? It is just too disappointing to find that the most recent reference is already 10 years old.

Stage 3: Writing feedback

Most journals request separate feedback to the authors and to the editors. As the reviewer your thoughts should be communicated in as constructive a manner as possible (sometimes an art in itself). Feedback to editors helps them decide on the suitability of the paper for the journal so be succinct and systematic.

Feedback to the author

This can be one of the most difficult tasks for the reviewer. The feedback should be as constructive as possible, even though sometimes this is not the reviewer's knee-jerk response to the paper. Tact and respect should be maintained with the aim of leaving the author in a positive frame of mind even if you are recommending rejection. The quality of the paper may be poor but the author is likely to have spent some considerable time reaching the point of submission. This might be their first paper, their first language may not be English and their senior authors may not have contributed much more than a signature. The aim is, therefore, to encourage the author to improve even if the paper is not of the quality to be published in its present format.

Where a journal does not give an actual layout for the report, our suggestion would be to include the following elements.

Summary

Write your concise summary of the paper as you see it. Clearly state in your opinion what the authors have tried to achieve and what new contribution the paper may (or may not) be making to the scientific literature. This shows that you have understood the paper; it also helps the editor to stay in focus, especially where there is a controversial manuscript, and reassures the authors that you have understood what they are trying to communicate.[20] It is very easy for authors to enter into a protracted disagreement with a reviewer and this is often due to misinterpretation of information on one or both ‘sides’. Convey to the editor that you do understand what sort of paper you are reviewing, for example, the type of study, review or commentary.

Ethical aspects

Sometimes reviewers forget that they have a responsibility to the author as well as to the editor. Authors, admittedly sometimes after some time for reflection, appreciate detailed feedback. Editors like their journal to publish work that is up to date and evidence-based. In addition, the reviewer must confirm that the study has satisfactory ethical approval and has a duty to alert the editor if they have any concerns in this respect. Reporting plagiarism, or the possibility of plagiarism to the editor, is another of the reviewer's responsibilities: editors have access to powerful programmes that can determine if plagiarism is present. This is a serious offence and the editor will have to decide whether or not to make a referral to the General Medical Council or the equivalent relevant professional body.

The Committee on Publication Ethics has recently published the ‘COPE Ethical Guidelines for Peer Reviewers’,[21] that journals are now attaching to reviewer invitation letters and editorial policies.

General comment

A general commentary on overall quality, originality, methodology and write-up provides important feedback to the authors. A good piece of research badly presented is very different from a bad piece of research beautifully presented.

Strengths

Listing the strengths of the manuscript makes the review well rounded and even if the paper is rejected the author is more easily convinced that the review was not one-sided and that the work was considered fairly.

Specific comments

A list of specific comments would include the literature review, design and execution of the work, the technical content, writing style and presentation. You should include an action list of specific comments that the authors may wish to address, whether the paper is ultimately accepted or rejected by this particular journal. Point out any conflicting statements or ideas, any parts that are difficult to understand and where improvements in style can be made. Do not be afraid to carry out your own literature search and quote this in your comments: this increases your credibility. If the study is not original, as claimed, cite the earlier study. Comment on tables, figures, flowcharts and any illustrations: are there too few or too many? Are they actually relevant? Do not forget the references, which have to be in the required format for the particular journal.

Beware of using too informal a style or humour: this might not be appreciated and is quite likely to be taken the wrong way by a disappointed and possibly disgruntled author. Being objective and tactful is the safest route.

By quoting the page, paragraph and line numbers both the author and the editor will know exactly what you are referring to in any one point. This will help the editor to determine if the authors have addressed the points raised in any subsequent correspondence. Technically, typos, spelling and grammar errors should be left for the editor but if they are persistent and glaring it does no harm to mention them. It is unacceptable and disrespectful to submit a carelessly written paper.

Feedback to the editor

This section essentially includes all the comments to the author and ‘for the editor's eyes only’ comments that will help the editor to make a decision. In this day and age it is not unheard of (although thankfully rare) for an author to take legal action against a referee: your comments to the editor will be recoverable – remember that! Any conflict of interest on your part must be highlighted.

The final comments to the editor should explain to them if in your opinion the work is original, clinically relevant, valid and reliable scientifically. The strengths and weaknesses with possible remediable measures should be recorded clearly. Do remember, though, that it is the editor's job to accept or reject the paper, not yours.

Conclusion

The review process can be a burden but all of us involved in medical practice and research have a moral duty to carry out the reviewer's role. Reviewers must remember that they have responsibilities both to the author and editor to ensure quality control but in a fair and appropriate way. This article is only a rather crude generic guide to the art of reviewing – innovative reviewers will, of course, have their own modifications and additions to the process to perfect their personal approach to the review process.

At all times remember that confidentiality is crucial: the unpublished paper is a privileged document and should not be misused, cited or even discussed before publication without the editor's express permission. Peer review can be thought provoking and will stimulate new ideas for the reviewer but be conscious not to use any privileged information in your own projects. If in doubt talk to the editor: DO NOT contact the authors directly, even if you know them.

Contribution of authors

JJ conceived, designed, drafted the article and revised it critically for important intellectual content. NM revised the article critically for important intellectual content. Both authors approved the final version for publication.

Disclosure of interests

The authors have no conflict of interest to declare.