SEARCH

SEARCH BY CITATION

Keywords:

  • blinding;
  • double-blind method;
  • journals;
  • nursing;
  • peer review;
  • publication bias;
  • survey

Abstract

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Title. Blinding in peer review: the preferences of reviewers for nursing journals.

Aim.  This paper is a report of a study to assess the beliefs and preferences of reviewers for nursing journals about blinding of authors to reviewers, reviewers to authors, neither or both.

Background.  Blinding of author and reviewer names in the manuscript review process has been of interest to nursing editors, but reports that are based on data rather than simply opinion concern the editorial practices of biomedical rather than nursing journals. There has been no study of nursing journal reviewer beliefs and preferences related to blinding.

Method.  A descriptive web-based survey was conducted. The sample included 1675 anonymous reviewers, recruited through 52 editors of nursing journals from their review panels. Data were collected in 2007.

Findings.  Double-blinding of reviews was the most common method reported. Ninety per cent of respondents reported that the papers they received to review did not include author names. When author names were blinded, 62% of reviewers could not identify the authors of papers; another 17% could identify authors ≤10% of the time. Double-blinding was the method preferred by 93·6% of reviewers, although some identified some advantages to an unblinded open review process.

Conclusion.  Nursing journal reviewers are generally very satisfied with double-blinding and believe it contributes to the quality of papers published. Editors or editorial boards interested in a more open review process could consider alternatives such as offering authors and reviewers the option to unblind themselves. Simply announcing that the review process will henceforth be unblinded would probably lead to loss of reviewers.


What is already known about this topic

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References
  •  Most literature on blinded peer review comes from the biomedical literature and primarily from the United States of America.
  •  Blinding of author and reviewer names in the manuscript review process has been of interest to nursing editors, but reports that are based on data rather than simply opinion concern the editorial practices of biomedical rather than nursing journals.

What this paper adds

  •  Nursing journal reviewers surveyed identified double-blinding as the most common and preferred mode of review.
  •  Reviewers believed peer review is important to ensure high-quality research.
  •  Nursing reviewers did not want either to know authors’ names or to have reviewers’ names revealed to authors, but the reviewers were able to identify some advantages to more open review.

Introduction

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Since the mid-1990s, much has been written about blinding (concealing the identity of reviewers or authors) of peer reviews in the biomedical literature. Blinding is often assumed to improve the objectivity of reviews, but there is little consistency in its use across professions. Although in nursing journals double-blinding (blinding reviewer to author identity and author to reviewer identity) has been the norm, in other biomedical journals the procedure is often single-blind, concealing reviewers’ names from authors but not the reverse.

Recent studies have provided evidence of medical journal practices and reviewer responses and some information about nursing editors’ opinions, but little is known about what reviewers for nursing journals believe about blinding. The results reported here are a part of a larger exploratory study of numerous aspects of reviewing for nursing journals. A full description of the methods, sample, and issues of experience, time investment and motivators for reviewing is in press (Kearney et al. 2008). Other topics covered in the study include the preparation and training of reviewers, issues relevant for international journal reviewers, and the ethics of reviewing. The purpose of this portion of the study was to ascertain the beliefs and preferences of reviewers for nursing journals about blinding of author and/or reviewer names.

Background

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Topics covered in the literature related to blinding in peer review include whether blinding changes the quality of reviews (Justice et al. 1998, van Rooyen et al. 1998, Walsh et al. 2000, Jefferson et al. 2002a,Jefferson et al. n.d.). A second topic in the literature is whether there is bias or difference in scores assigned to manuscripts, by blinded vs. unblinded reviewers (Fisher et al. 1994, Smith et al. 2002, Ross et al. 2006). A third set of writings about blinded review are less easily categorized: whether a policy of blinded peer review changes citation frequency for journals (Laband & Piette 1994), announcements of trials of open peer review (Smith 1999, Dougherty 2004), musings by editors and commentators on whether blinded peer review should be abandoned or taken up (Davidoff 1998, Baggs 1999, Malone 1999, Tilden 2002, Mason 2003, Steers 2006), reports of editors’ and reviewers’ preferences about blinding (Kearney & Freda 2005, Regehr & Bordage 2006), and reviews of the research on various aspects of peer review including blinding (Godlee 2002).

Historical perspectives on peer review

In 1986, the Journal of the American Medical Association announced that it would hold periodic research conferences on ‘subjects surrounding the publication of science relevant to clinicians’ (Rennie 1998, p. 214). The journal then published themed issues in 1990, (263, 9 March), 1994 (272, 13 July), 1998 (280, 15 July) and 2002 (287, 5 July) featuring papers from those conferences. This opportunity to present and publish stimulated a number of authors to take up the challenge to conduct research in this arena.

Blinded peer review and quality

Based on review papers (Godlee 2002, Jefferson et al. 2002a, Jefferson et al. n.d.) and our review of the literature, six studies were identified that involved assessment of quality of peer reviews and how it is related to blinding (McNutt 1990,Godlee et al. 1998, Justice et al. 1998, van Rooyen et al. 1998, 1999b [two reports on the same study], van Rooyen et al. 1999a, Walsh et al. 2000). In one of these studies (Walsh et al. 2000), signed reviews were judged to be of higher quality, more courteous and took longer to complete; in a second study (McNutt 1990), the quality of reviews was higher when reviewers were blinded to authorship, and signing reviews had no effect on quality. In the other five studies, no effect on quality of review for either type of blinding was found. Studies of other methods to improve quality of reviews, such as feedback to reviewers from editors, submission checklists and reviewer training (Callaham et al. 2002, Jefferson et al. 2002a, 2002b) have been unsuccessful in identifying ways to improve quality.

Blinded peer review and bias

Although most researchers have assessed effect of blinding on quality of review, there are four studies of the impact of blinding on bias (Fisher et al. 1994, Smith et al. 2002, Ross et al. 2006, Budden et al. 2008). Budden et al., Fisher et al. and Ross et al. all reported evidence of bias in unblinded vs. blinded reviews. Budden et al. identified bias against women as first authors in unblinded review; Fisher et al. found that blinded reviewers provided a more unbiased review and Ross et al. found bias favouring authors from the United States of America (USA), other English-speaking countries, and prestigious academic institutions. J. A. Smith et al., on the other hand, found no consistent evidence of bias related to unblinding.

Is blinding possible?

One challenge related to studying blinded vs. unblinded reviews is that at times reviewers are able to identify authors or the author group, even when the identity of the author is blinded to them. In six studies, reviewers were asked if they could identify the authors (Yankauer 1991, Fisher et al. 1994, McNutt 1990, Cho et al. 1998, Justice et al. 1998, Katz et al. 2002). The percentages of reviewers able to identify authors in blinded studies ranged from 27% (McNutt 1990) to 46% (Fisher et al. 1994).

Preferences about blinding

There have been two studies of preferences about blinding. Reviewers (= 838) for the journal Medical Education reported a preference (68%) for concealing author names and (72%) for concealing reviewer names. When given choices of single blinding of author, single blinding of reviewer, optional single blinding, double-blinding, no blinding, or other, 50% preferred double-blinding, with no other option receiving more than 15% endorsement (Regehr & Bordage 2006). Similarly, in a survey of 88 editors of nursing journals, Kearney and Freda (2005) found 80% preferred double-blinding. The most common reasons for preferring double-blinding given by both the reviewers and editors in these two studies were reduction of bias by facilitating more objective reviews and protecting both authors and reviewers from ‘professional or personal consequences’ (Kearney & Freda 2005, p. 448). No assessment of the preferences of reviewers for nursing journals has been published.

The study

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Aim

The aim of the study was to assess the beliefs and preferences of reviewers for nursing journals about blinding of authors to reviewers, reviewers to authors, neither or both.

The research questions were: What form of blinding is most common in nursing journals? Can authors’ names be successfully blinded? What are nursing journal reviewers’ preferences about types of blinding? How do nursing journal reviewers rate peer review in insuring quality? What do reviewers see as advantages and disadvantages of openly posted or public reviews?

Design

The methods for this descriptive correlational study, which was part of a larger study of reviewers for nursing journals, have been presented in more detail elsewhere (Kearney et al. 2008).

Sample

A group of 52 editors of nursing journals, accessed through the International Academy of Nursing Editors listserv and conference attendance lists and from the International Network of Doctoral Education in Nursing list of international journal editors, sent invitations to participate to their review panels. The editors primarily represented nursing journals, but those represented included six without nurse or nursing in their titles: Advances in Skin & Wound Care, Air Medical Journal, Disaster Management & Response, Family & Community Health, Home Health Care Management & Practice and Journal of Midwifery & Women’s Health.

Data were collected in 2007. There were 1675 valid responses by reviewers to the survey as a whole; another 25 provided only demographic data. The total number who received the survey is unknown, as some editors did not state the size of their review panel. The ns provided throughout the paper indicate the number who responded to the question being discussed. Respondents chose to answer some questions but not others, and the number of respondents particularly decreased for the open-ended responses.

Data collection

Instrument

The questionnaire was developed by the authors of this paper based on the literature, their experience as editors, and recommendations from other nursing editors. The questionnaire was entered into SurveyMonkey™ (Portland, Oregon, USA), a software platform that can be accessed from a web page addresses emailed to recipients, answered within the questionnaire, and returned anonymously. The instrument consisted of 69 fixed-choice and open-ended questions on topics including preferences related to blinding of reviews, the topic of this paper. The survey request was sent only once to potential respondents in an email.

Ethical considerations

The study was approved as exempt by two institutional review boards. The completion and the return of the questionnaire demonstrated consent.

Data analysis

The data were analysed using SPSS™ (SPSS Inc., Chicago, IL, USA). For this aspect of the larger study, descriptive statistics and content analysis, assessing for most frequent responses, were undertaken. Content analysis of responses to the open-ended items focused on aggregating responses into categories with similar content.

Validity and reliability

The questionnaire was pilot tested with 18 reviewers for readability, face validity, time to completion, clarity of questions, feasibility and comprehensiveness. After pilot testing, modifications were made to shorten the questionnaire and to reword and clarify some questions and the types of responses needed.

Results

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Sample

The survey respondents were primarily women (91%), nurses (90%), doctorally prepared (68%) and reviewing primarily for research journals (57%). Although a majority were from the USA (74%), reviewers came from 44 countries, including 8% from the United Kingdom, 4% from Canada and 3% from Australia. The remaining 11% were from the Middle East, Europe, South America, Asia, Africa, the south Pacific and Eastern European countries. In response to a question about professional background, 1541 listed nursing, but 127 listed ‘other,’ primarily in these categories: bioethics, biostatistics, education, health services, law, medicine, midwifery, pharmacy, psychology, public health, respiratory therapy, social worker and sociology.

What form of blinding is most common in nursing journals?

Reviewers were asked, ‘When you receive manuscripts from this journal are you given the names of authors?’ and ‘Do authors receive the names of reviewers of their manuscripts?’ The most common form of blinding cited by respondents was double-blinding: 96% (= 1580/1646) reported that they did not see authors’ names when they received a manuscript to review, and 70% (= 1105/1652) believed that authors were not given their names. The lower value for the second response is explained by the 31·6% (= 522) who reported that they did not know if their names were revealed to authors.

Can authors’ names be successfully blinded?

Reviewers were asked, if they did not receive authors’ names, ‘What percentage of the time can you identify the authors anyway?’ For this group, the blinding of author names was generally successful: 62% (= 512/821) of those who did not receive author names who responded to this question reported that they never could identify authors, and another 17·5% (= 144) reported that they could identify ≤10% of the time.

What are reviewers’ preferences about types of blinding?

Not only was double-blinding most common, but in response to the question ‘Do you believe double-blinding (in which identities of both author and reviewer are unknown to each other) should continue as the norm in nursing journals?’: 93·6% (= 1350/1443) endorsed it as their preference into the future.

Is blind review appropriate?

Whether blind review was appropriate was asked very specifically. Respondents were asked their agreement with the statement ‘Reviewers should be informed of the identities of authors of a manuscript’: 81·8% (= 1317/1610) disagreed or strongly disagreed. In responding to the statement ‘Authors should be informed of the identities of the reviewers of their manuscripts’, a similar number disagreed or strongly disagreed: 76·5% (= 1231/1608). Over 1000 respondents chose to provide open-ended answers when asked to ‘Explain your answer’ to each of the two statements.

Reviewer unblinding to author names

In response to the item about reviewers knowing the identity of authors, by far the most common comments (64%) related to keeping reviewers blinded to prevent reviewer bias and maintain objectivity. Three sample comments are:

As a reviewer, I focus on the manuscript and not the author(s)…I am a steward of the journal’s reputation for scholarliness, fairness and autonomy. If I need to do [this] for the journal, I cannot know who the author(s) is,

I enjoy reviewing work and not being swayed by who the author is…If I knew who the authors were and what organizations they were from, I would be biased in my review, even though I would do my best not to be,

Anonymity ensures the integrity of the system and provides younger faculty more opportunity to have their work appear in reputable journals.

Other arguments in support of blinding author names included avoidance of interpersonal conflict and avoidance of political issues. A substantial number of respondents (103) indicated that it would be unnecessary to unblind (e.g. ‘The name of the individual is not as important as the quality of the article’ and ‘It would not matter to me’).

Arguments in favour of transparency or unblinding author names included professionalism, improved communication, accountability and fairness. Example comments included: ‘I just think we should be able to be open and honest’, ‘I would much rather know the identity of the author because then I can have a conversation with them as well as their ideas’, ‘If I knew an author was new to the field, I would include more mentoring suggestions’ and ‘Professionals should be able to talk to each other’.

Several reviewers suggested that it would be useful to know the author credentials without disclosing names (e.g. ‘They [reviewers] should know the degrees and current role of the writer to indicate they have the necessary expertise’). Another suggestion from several reviewers was that unblinding of authors could be optional, the choice of the author.

Author unblinding to reviewer names

When reviewers were asked to indicate agreement or disagreement with the statement ‘Authors should be informed of the identities of the reviewers of their manuscripts’, comments were more divided. The most common comments (171) related to unblinding being unnecessary (e.g. ‘I cannot see the benefit of this to the author or to the review process’). Concerns about bias again were raised, in the context of reviewers being unwilling to be honest if this were done (e.g. ‘If the author does not know the names of reviewers, it provides an opportunity for reviewers to be more forthcoming’ and ‘I worry that reviewers who are more junior risk incurring the ire of senior researchers if they criticize a manuscript’). Interpersonal issues, most commonly the possibility of resentment, confrontation or reprisal from angry authors, were fairly prominent in responses (‘Authors could harbour ill feelings against a reviewer’ and ‘Reviewers do not want the potential to be hassled by writers regarding their review decisions’). Some reviewers also indicated that they would be less likely to review (‘No quicker way to lose a reviewer than to lose her anonymity’).

Arguments in favour of unblinding reviewer identity included fairness, better communication and professionalism. Sample comments included: This will help give some accountability to the reviewers to respect the author’, ‘It may be helpful for the reviewer and author to talk…if there are questions about the feedback’ and ‘It [unblinding] seems to require the reviewer to take more accountability for what one says in a review’. As in unblinding of author names, the suggestion was made to provide credentials only or to make the decision not to be anonymous a choice of the reviewer.

Advantages and disadvantages of openly posted or public reviews

Respondents were asked, ‘Some journals publish the reviews of manuscripts on a website when an article is published. Would you agree to have your reviews posted or made accessible to the public?’. Only 14·7% (= 234/1588) were willing to have a review posted with their name attached to it, although 47·4% (= 752) would be willing to have a review posted anonymously.

Advantages

Reviewers were asked, ‘Which of the following are advantages of making reviews open to the public?’. They could mark any or all of five possible advantages, which many did (Table 1). For the 87 responses to ‘Other’, content analysis demonstrated that by far the most frequently discussed advantage (= 51; 58·6%) related to use of open review as an educational opportunity, for novice researchers, authors, reviewers, or students, or as a learning tool even for those with more experience. As one respondent put it, ‘Writing reviews requires knowledge and skill – all reviewers and prospective reviewers can learn about this skill by reading a variety of them’. The only other advantage with more than four responses was open posting of reviews as a forum for debate, dialogue or scholarly discourse; as one said, ‘Provides a forum for increased debate, rather than taking published work at face value’.

Table 1.   Reviewers’ reports of advantages of open review (N = 1675)
 n (%)
  1. 1–5 response choices provided in survey.

1. Makes the review process more transparent856 (51·1)
2. Opportunity to compare my review with the other reviews827 (49·4)
3. Opportunity to observe how the manuscript   was changed based on reviews796 (47·5)
4. Biases and conflicts of interest are more readily apparent637 (38·0)
5. Provides career advancement for reviewers as a form of scholarly work469 (28·0)
No advantages134 (8·0)
Other (wrote open-ended response)87 (5·2)
Disadvantages

Respondents were also asked, ‘Which of the following are disadvantages of making reviews open to the public?’. They could indicate any of four possible advantages (Table 2), and give an open-ended response. Responses about disadvantages were more varied than those about advantages. The most frequent responses (= 43) involved concerns about authors. Participants worried that unpleasant reviews could harm authors, particularly when those comments related to earlier versions of the manuscript and might no longer be relevant. As one respondent said:

Table 2.   Reviewers’ reports of disadvantages of open review (N = 1675)
 n (%)
  1. 1–4 response choices provided in survey.

1. Reviewer may not be as candid or rigorous1241 (74·10)
2. Reviewer’s identity may be disclosed925 (74·10)
3. More difficult to write a review for a general audience906 (54·10)
4. Prefer to address my comments to the author and editor only885 (52·80)
No disadvantages13 (0·01)
All of the above10 (0·01)
Other (wrote open-ended response)111 (6·6)

Authors should not have the flaws in early drafts of their work exposed. Good reviews hel[p] authors improve the quality of their writing. No one needs to know about any deficiencies in the earlier versions of the manuscript.

Respondents indicated that fear of exposure could lead authors to be reluctant to resubmit or ever to try to publish again. The second most frequent category of responses (= 33) related to problems for reviewers. These comments included the need to spend more time polishing reviews for publication, and that there could be loss of collegiality and/or retribution against a reviewer who wrote an unfavourable review. This all could lead to reluctance, and from some respondents, refusal to review under such conditions (e.g. ‘I would not review for such a journal’). A related issue raised (= 4) was that the quality of reviews could suffer, with an emphasis on form over substance, increasing bias and leading to loss of objectivity. One respondent believed that, ‘The purpose of the review is to improve the article and prepare it for publication. I think this changes the purpose of the review, and I think only positive reviews would be published’.

The third most frequent category (= 14) related to the need for voluntariness by both author and reviewer and the privacy of both. Concern was expressed that the review process is not intended to be a public process, but a communication among author, reviewer and editor (as one respondent noted, lacking its own peer review).

Discussion

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Study limitations included that the respondents were a convenience sample. Only those with Internet access who were comfortable reading and responding in English, and who were willing to devote the time to completing the survey participated.

Our findings on perceptions of reviewers about blinded reviews for nursing journals match similar findings in other disciplines. The most common form of blinding nursing journal reviewers encountered was double-blinding. Their preference was to maintain that double-blinding, as found for reviewers for Medical Education (Regehr & Bordage 2006) and editors of nursing journals (Kearney & Freda 2005). This paper is the first to provide open-ended responses with the rationale for these preferences. The percentage of times reviewers thought they could identify authors in our study (only 10% thought they could identify ≥10% of the time) was less than in previous work (17% McNutt et al. 1994, to 46%Fisher et al. 1994). This difference may be related to the diversity of the scope of nursing research, number of journals in the area and differences in the education of reviewers for nursing journals.

Despite mixed empirical evidence on the relationship of bias and blinding (Fisher et al. 1994, Smith et al. 2002, Ross et al. 2006), our respondents believed that blinding decreased bias, as did reviewers and editors in other studies (Kearney & Freda 2005, Regehr & Bordage 2006). As in those same two studies, our respondents thought that blinding decreased the potential for interpersonal conflict, encouraged honest and fair assessments on the part of reviewers, and protected junior reviewers from senior authors.

Conclusion

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Our results suggest that nursing journal reviewers are generally very satisfied with double-blinding and believe it contributes to the quality of papers published. Editors or editorial boards interested in a more open review process could consider alternatives such as blinding names but revealing credentials or offering authors and reviewers the option to unblind themselves. There was recognition by the reviewers that there are advantages to unblinding (Table 1), but simply announcing that the review process will henceforth be unblinded would likely lead to loss of reviewers.

Acknowledgements

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

Appreciation is extended to Valerie Parham-Thompson for web-based survey management, to Tanya Ostrogorsky for transformation of the database and to Alison Dezolfi for content analysis of two open-ended questions.

Author contributions

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References

JB, MB, MD, MF and MK were responsible for the study conception and design, performed the data collection, made critical revisions to the paper for important intellectual content, provided statistical expertise and supervised the study. JB performed the data analysis and was responsible for the drafting of the manuscript. JB, MK and MD provided administrative, technical or material support.

References

  1. Top of page
  2. Abstract
  3. What is already known about this topic
  4. Introduction
  5. Background
  6. The study
  7. Results
  8. Discussion
  9. Conclusion
  10. Acknowledgements
  11. Author contributions
  12. References