Peer review in a changing world: An international study measuring the attitudes of researchers

Authors


Abstract

This large-scale international study measures the attitudes of more than 4,000 researchers toward peer review. In 2009, 40,000 authors of research papers from across the globe were invited to complete an online survey. Researchers were asked to rate a number of general statements about peer review, and then a subset of respondents, who had themselves peer reviewed, rated a series of statements concerning their experience of peer review. The study found that the peer review process is highly regarded by the vast majority of researchers and considered by most to be essential to the communication of scholarly research. Nine out of 10 authors believe that peer review improved the last paper they published. Double-blind peer review is considered the most effective form of peer review. Nearly three quarters of researchers think that technological advances are making peer review more effective. Most researchers believe that although peer review should identify fraud, it is very difficult for it to do so. Reviewers are committed to conducting peer review in the future and believe that simple practical steps, such as training new reviewers would further improve peer review.

Introduction

Each year more than 1.3 million learned articles are published in peer reviewed journals (Björk, Roos, & Lauri, 2009). Peer review is critical to the integration of these new research findings (Campbell, 2006). The authority of peer review is so preeminent that the research community generally views with scepticism any research that appears in the public domain, such as in the media, which has not first been published in a peer reviewed journal. Such is the importance of peer review that according to Ziman (1968, p. 111), it is “the lynchpin about which the whole business of science is pivoted.”

Peer review has a long history. Review by peers has been a part of scientific communication since the appearance of the first journals in the 1660s. The Philosophical Transactions of the Royal Society is accredited as being the first journal to introduce peer review (Zuckerman & Merton, 1971, pp. 68–69). However, in spite of its long legacy, it is only since Second World War that peer review has been formalized across much of the scholarly literature as noted by Rennie (2003).

The expansion of the global research community and the year-on-year increase in the number of papers published means dependency upon peer review has grown. However, as the dependency has increased, so, too, has the volume of those studying and questioning peer review's effectiveness. In this research study, we sought to determine the opinion of a large number of researchers on the effectiveness of the current peer review system, specifically, whether or not potential weaknesses identified in the literature are considered weaknesses by the broader community.

To provide appropriate context for this study, it is important to review the literature and outline some of the more significant developments in peer review in recent years.

In its simplest form, peer review is the evaluation of an author's manuscript by selected reviewers who make recommendations to the journal's editor as to whether or not the manuscript should be accepted, revised prior to publication, or rejected. The reviewer is invited to make observations on the quality, originality, and importance of the work. In her systematic review of editorial peer review, Weller (2001, p. xii) states that “ ‘the goal of the process is to ensure that the valid article is accepted, the messy article cleaned up, and the invalid article rejected,’ ” thereby ensuring that the article made available to the reader is quality controlled. As noted by Shatz (2004, p. 1), peer review also benefits the author, who is certified by the process: “Careers are often made or destroyed by the process.”

Both Weller (2001) and Shatz (2004) provide the most comprehensive overviews of editorial peer review. Each examines whether or not peer review is fulfilling its function. While Weller identifies a number of benefits and weaknesses of the system, it is Shatz who is possibly the more critical of the two. Shatz focuses on bias as a key weakness of peer review, and suggests a useful typology for the different types of bias: ad hominem, affiliational, and ideological and aesthetic.

The work of Wennerås and Wold (1997) identify ad hominem gender bias as a fault in the peer review system, but ultimately they believe such bias can be overcome. Meadows (1998) also examines the issue of gender bias, but believes it is not substantiated by studies in the literature. Instead, he identifies affiliational bias as the main issue, specifically the preference given to “prestigious” authors' whose work is reviewed with more regard for the author than their paper. Peters and Ceci (1982) had previously identified affiliation bias when they noticed the predisposition of reviewers to reject papers that did not originate from prestigious institutes.

To tackle some of these biases, different forms of peer review have emerged as alternatives to traditional peer review. Single-blind peer review, which is where the author is known to the reviewer but the reviewer is unknown to the author, is generally accepted as the traditional and most common form of peer review in scholarly publishing, as noted by Ware (2011).

A relatively well-established alternative form of peer review is double-blind reviewing. This is where both the reviewer and author remain unknown to one another. It is perceived as avoiding some but not all of the biases, notably ad hominem bias, as identified by Wennerås and Wold (1997), and affilitational bias, as identified by Peters and Ceci (1982) and Meadows (1998).

Open peer review is an alternative form of peer review that has emerged more recently and, to some extent, mitigates against ad hominem and ideological bias. Open peer review, in which both the reviewer and the author are known to one another, increases transparency and encourages honest open responses. In addition, this approach is also believed to prevent “malicious” comments and control plagiarism. The British Medical Journal adopted open peer review in 1999 as the preferred form of review after a series of trials conducted by the journal. Smith (1999), the editor of the journal at the time, reported the decision in 1999. Open peer review has developed into an umbrella term that covers a number of initiatives with varying degrees of openness. The majority of BioMed Central publications have adopted open peer review (http://www.biomedcentral.com/about/peerreview). They have taken openness one step further and encourage reviewers to sign their reports so that they can be published alongside the article. Moreover, open peer review is not limited to the biomedical field, the journal Atmospheric Chemistry and Physics uses a two-stage approach to open peer review that includes public commentary and the posting of the reviews alongside the article with or without their name, dependent upon the reviewer's preference (Pöschl, 2010).

Ideological bias is perhaps the most difficult form of bias to prevent. Horrobin (2001) argues that peer review is so fundamentally flawed that it is holding back innovative science. He speculates that because of the conformist nature of peer review, those in the field of psychopharmacology are still pursuing themes initiated in the 1950s. Spier (2002) explores this theme and considers that most peer review processes are ideologically conservative in nature and stifle innovation.

Possibly the most critical assessment of peer review has come from those in the medical field. Jefferson, Rudin, Folse, and Davidoff (2007) assessed 28 studies of peer review in biomedicine to determine whether peer review fulfils its role as gatekeeper. They assert that there is little evidence available to “support the use of editorial peer review as a mechanism to ensure quality of biomedical research” (p. 14).

In response to the perceived shortcomings of peer review, the most radical recent initiative to emerge has been to limit the role of peer review and to eliminate its use as a tool to assess “importance.” PLoS ONE was the first journal to remove importance as a criterion for review. The editors declare on their site that “too often a journal's decision to publish a paper is dominated by what the Editor/s think is interesting.” Instead, all papers submitted to PLoS ONE that are judged to be technically sound are published in the journal. As Patterson (2010), PLoS' director of publishing, states: “The editors and peer reviewers make no judgment as to the potential impact of the work.” Since the emergence of PLoS ONE, other journals such as Sage Open (http://www.sagepub.com/journalsProdDesc.nav?prodId=Journal202037), BMJ Open (http://bmjopen.bmj.com/site/about/reviewerguidelines.xhtml), and Biology Open (http://bio.biologists.org/site/misc/aims_scope.xhtml) have emulated PLoS ONE's approach. Each offers a very similar form of peer review. As PLoS ONE does, they use the “wisdom of the crowd” to help assess importance. They all provide article-level metrics and the opportunity for readers to post comments after publication.

Recently both Souder (2011) and Ware (2011) have taken a fresh look at the status of peer review. Souder suggests that as an expert-based quality control mechanism, the peer review system is likely to be a better alternative to the increasingly aforementioned populist online communities for preserving the ideals of scientific inquiry. Ware (2011) asserts that contrary to the perception that peer review is in crisis, the emergence of alternative forms of peer review indicate that it is at its most diverse.

Although alternative forms of peer review have evolved to tackle issues of bias, it is less clear what effect, if any, they will have upon fraud. High-profile cases of fraud and plagiarism have brought the debate about the efficacy of peer review to a wider audience, attracting greater public attention. Such incidences include Jan Hendrik Schon, tipped to be a Nobel Prize winner, who published a series of fraudulent papers that were withdrawn from Nature, Science, Physical Review, and Applied Physics Letters (Lerner, 2003). More recent examples include the Wakefield article published in the Lancet (Wakefield et al., 1998), which was subsequently withdrawn in 2010 (Rose, 2010), and the falsification of data by Hwang Woo-suk (http://www.nature.com/news/2009/091026/full/4611181a.html).

Because of the pivotal role that peer review plays and the negative perceptions surrounding peer review, there have been a number of studies that examined the attitudes of researchers towards peer review. See Table 1 for a summary of the key studies and their outcomes.

Table 1. Previous surveys that have examined peer review
AuthorsName of studyCoveragePopulation characteristicsTopics coveredFindingsType of researchSample sizeResponse rateDate
Harley, Acord, Earl-Novell, Lawrence, and Judson

Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines.

University of California principally but drawn from 45 U.S. institutes.Scholarly staff drawn from a range of research intensive N. American institutes.

Scholarly communication needs of researchers as both producers and users of research. Capabilities of traditional and emerging models including peer review.

Peer-reviewed journals are key to tenure and promotion decisions. Peer review is critical filter for time-pressed faculty in determining what to read.In-depth interviews. Focus groups.160n/a2010 & 2007
Resnik, Gutierrez-Ford, and Peddada

Perceptions of ethical problems with scientific journal peer review: an exploratory study.

Health Sciences at US Gov't institute the National Institute of Environmental Health SciencesPublished researchers at U.S. Health institute.Ascertain perceptions of the journal peer-review system.Their survey identified the most frequent problems as incompetent reviews and bias (including delaying publication, and inclusion of unnecessary references).Print/e-mail after training sessions28350.9%2008
Ware and MonkmanPeer review in scholarly journals: perspective of the scholarly community an international study.GlobalAuthors, reviewers, and editors.Attitudes to peer review, its benefits, types of peer review thought effective, alternative approaches.Peer review is widely supported by academics. Improves the quality of the published paper. Double blind preferred. Postpublication review seen as a useful supplement.Online survey3,0407.7%2008
University of California Office of Scholarly Communication

Faculty attitudes and behaviours regarding scholarly communication: Survey findings from the University of California.

University of CaliforniaScholarly staff at single institute.

Measure faculty attitude across a range of issues, including promotion processes, copyright, emerging forms of publication, peer review, and dissemination.

Scholars want to maintain current publishing outlets. In respect to peer review, respondents showed broad and strong loyalty to peer review as the primary means of ensuring the quality of published works.Online survey1,11823%2007
Nicholas and Rowlands

The changing

 scholarly

 communication

 landscape:

 an international

 survey of senior

 researchers.

GlobalAuthorsAttitudes to scholarly publishing generally, including open access publishing, institutional repositories and peer review.Authors' experience of the peer review process was highly positive. The community attaches considerable value to the function of peer review in regulating the quality of what is published: 96% indicated it is “very” or “quite important.”Online survey5,5137.2%2006
Nicholas, Jamali, Huntington, and RowlandsIn their very own words: authors and scholarly journal publishing.GlobalAuthorsAttitudes to scholarly publishing generally. Authors were encouraged to provide open-ended comments that were analysed using textual analysis software.Results covered the peer-review system, copyright, journal prices, big deals. In respect to peer review, most comments were supportive, but highlighted some perceived weaknesses, bias, time-consuming.Online survey3,7873.5%2005
Swan and BrownAuthors and electronic publishing: What authors want from the new technology.

Global. Strong focus on business and management.

AuthorsObtain views on electronic publishing of learned journals.

Print version of journals more important than electronic version. Continuing availability of back volumes important strong concerns about pricing of electronic journals. Publication speed through not a priority. Preprint and reprint archives are used very little. Peer-review process is valued: 94% believe it is important.

Online survey1,2469%2003
Swan and Brown

What authors want: the ALPSP research study on the motivations and concerns of contributors to learned journals.

United KingdomAuthorsFactors authors take into consideration when deciding where to submit. Concerns of authors about the publishing process. Expectation of authors for the future of scholarly publishing.Reputation most important criteria when deciding where to publish. Authors' concerns include copyright, publication delays, and peer review. Two thirds of authors satisfied with the peer review and half consider it to be an obstacle. Reasons include delays by the reviewer, superficial reviews, and unnecessary hostile reviews.Unspecified2,50021.7%1999

Although some of the above-mentioned studies are authoritative and comprehensive, such as Ware and Monkman's (2008), others are curtailed by geographic spread, subject scope, or type of respondent. Many of these studies are now a little out of date. We intend to build upon these earlier works, by conducting a global study across all subjects that not only examined some of the critical issues at the heart of peer review but also assessed the reaction of the research community to new forms of evaluation that have emerged in recent years.

We examine the attitudes towards peer review from three perspectives: (a) to capture the opinions of respondents as researchers generally, in other words, as an author, reader, or reviewer; (b) from the perspective of researchers as authors; and (c) from the perspective of those responsible for completing peer review, the reviewers themselves. It was also important to identify whether attitudes or experience varied by discipline, age, or geography. The specific questions we wished to answer for each group are outlined below:

Researcher Perspective

  • Are researchers satisfied with peer review system offered by scholarly journals?
  • Do researchers believe that peer review plays an important role in scholarly publishing?
  • Are researchers satisfied with the current peer review system?
  • Is the current peer review system the best that can be achieved?
  • Is peer review sustainable in the context of an expanding number of papers?
  • What is the purpose of peer review? How well does it meet these objectives?
  • What form of peer review is considered to be most effective by researchers and why?
  • Are there acceptable alternatives to peer review?

Author Perspective

  • How long did the author's last article take to be peer reviewed, and were they happy with the time taken?
  • How did peer review improve the author's last article? Was the improvement substantive?

Reviewer Perspective

  • What motivates reviewers?
  • What improvements do reviewers believe could be made to the peer review system?
  • How often do reviewers decline to review and why do they decline?
  • How long do reviewers typically spend reviewing a paper?
  • What are the attitudes of reviewers to incentives? Do they wish to be rewarded, if so what is their preferred reward?

Methodology

Forty thousand published researchers were randomly selected from over a million article records contained in the Thomson Reuters author database. Each researcher was sent an e-mail invitation to complete the survey, and a reminder was sent after a week if they had not already responded. The survey was in field between July 28, 2009, and August 11, 2009. Preliminary results from this study have been reported earlier (Mulligan & Raphael, 2010).

In total, 4,037 responses were received, making it one of the largest international surveys of researcher attitudes towards peer review. The response rate of 10% raises a question common to all surveys, specifically online surveys. Are there any systematic differences between the invited and responding populations? Determining nonresponse bias is difficult and there is always a possibility of nonresponse bias. However, it is worth noting that the response rate on this study compares favourably to previous studies in this field (see Table 1). The country of origin of the researcher was not included in the sample supplied by Thomson Reuters; thus, it is not possible to compare sample distribution against respondent distribution. The geographic distribution of respondents broadly reflects the geographic distribution of the world's research output in terms of articles as reported in Research Trends (http://www.researchtrends.com/issue8-november-2008/geographical-trends-of-research-output/), although it is worth noting that North America is overrepresented (5%), Asia, which included the main publishing countries of Japan, China, South Korea, and Taiwan, was slightly overrepresented (3%), while Europe was underrepresented. Regional classifications used in the analysis are taken from the United Nations Statistics Division (http://unstats.un.org/unsd/methods/m49/m49.htm) (see Table 2).

Table 2. Distribution by region
RegionFrequencyPercent
North America1,40034.7
Asia1,12127.8
Europe92322.9
Latin America and the Caribbean2917.2
Oceania1884.7
Africa1142.8
Total4,037100.0

It is worth noting that because the sample was drawn from the Thomson Reuters database of authors there is likely to be a bias towards the experience of those who have published and reviewed in higher quality, international, and English-language journals.

The number of respondents by discipline can be seen in Table 3. The broad categories bring together similar subjects. Some subject areas make up a relatively small proportion of the scholarly literature and did not have sufficient responses to be analysed independently. For example, there were only 53 responses for those based in the arts and humanities. Rather than exclude this group from the analyses because the numbers were too low to be analysed, it was decided that respondents in this subject would be combined with responses from the most closely related area social sciences and economics. We recognize there may be differences in approach to peer review within the arts and humanities when compared with the subjects with which it has been grouped, and this should be borne in mind when considering responses for this combined group.

Table 3. Distribution by subject
SubjectFrequencyPercent
Agriculture and biological sciences53113.2
Arts and humanities, social sciences, and economics41710.3
Astronomy and physics3508.7
Biochemistry, genetics and molecular biology, immunology, and microbiology3629.0
Chemistry and chemical engineering3107.7
Mathematics and computer sciences and IT2566.3
Earth and planetary sciences and environmental sciences3167.8
Electrical, electronic engineering, engineering, and technology42310.5
Materials sciences2025.0
Medicine and allied health and nursing61615.3
Neurosciences1784.4
Pharmacology, toxicology, and pharmaceutics761.9
Total4037100.0

Respondents are also diverse in terms of their age, position, and type of organization with which they are associated. Institute type ranged from hospitals and commercial organizations to research institutes and universities (see Tables 4, 5, and 6).

Table 4. Distribution by position
PositionFrequencyPercent
Head of department/senior management57314.2
Senior researcher and middle management1,41135.0
Researcher and staff member1,40734.9
Other (please specify)64616.0
Total4,037100.0
Table 5. Distribution by age
AgeFrequencyPercent
Younger than 2250.1
22 to 25431.0
26 to 351,00224.8
36 to 451,29832.1
46 to 5595423.6
56 to 6550712.6
Older than 652115.3
Prefer not to say170.4
Total4,037100
Table 6. Distribution by institute type
Institute typeFrequencyPercent
University or college2,65665.8
Hospital or medical school40210.0
Industry or commerce1112.7
Research institute65516.2
Government1533.8
Other601.5
Total4,037100.0

The questionnaire used filter questions to select researchers with the relevant experience to answer a specific set of questions. For example, only those that had completed a review within the last 12 months were allowed to progress to the section on reviewer experience.

The results included in this article represent the key findings as identified by the authors. During the process of preparing this article, the raw data were analysed according to a variety of different demographic variables: subject, geography, age, and researcher experience (see Tables 2, 3, 5, and 7). It is impossible to include all the different analyses in this article. Instead, we have chosen to focus on those elements that reveal patterns of difference that we think the reader would find most interesting. The survey instrument is included in the Appendix.

Table 7. Distribution by number of articles published
No. of articles published in career to dateFrequencyPercent
1–555614%
6–1051013%
11–2071818%
21–501,07527%
51–10071418%
More than 10046411%
Total4,037100%

As part of this research, respondents were asked why they thought a particular form of peer review was considered effective. While not all of these free text comments can be included, a number of comments have been included to reflect the variety of opinion.

Unless otherwise indicated, results are reported as percentages of respondents agreeing with a statement. Significance tests were run at the 95% confidence level (p < 0.05). Comparisons between a subgroup and the overall group are made using z tests of proportion; the z value is reported whenever a z test has been used. To compare results with earlier studies, we used z tests for two independent samples (p < 0.05) and reported z values. The margin of error in this study for all 4,037 responses assuming maximum variance in the data (50%) is ±1.5% (p < 0.05). The reviewer subset mentioned above is based on 3,597 responses; this gives a margin of error ±1.6% (p < 0.05). Throughout this report, the specific margin of error for each variable is reported.

Results

Satisfaction Levels for the Peer Review System Used on Scholarly Journals

Overall satisfaction with the peer review system used by scholarly journals is relatively high at 69% (±1.4%, p < 0.05, n = 4,037) (see Figure 1). Moreover, the level of satisfaction has increased by 4 percentage points (z = 3.6, p < 0.05) because a similar question was asked by Ware and Monkman (2008). Few (9%) were dissatisfied or very dissatisfied. However, when we examine satisfaction through the prism of demographic variables, differences start to emerge. Those in chemistry (77%, z = 3.1, p < 0.05), materials sciences (76%, z = 2.2, p < 0.05), and earth and planetary sciences (77%, z = 3.1, p < 0.05) are most satisfied. Astronomers and physicists (65%, z = 2.0, p < 0.05) and those in the arts and humanities, social sciences, and economics (64%, z = 2.3, p < 0.05) are least satisfied (see Figure 1). There are few differences by age (see Figure 2), but there are by region. Those in Asia (75%, z = 6.0, p < 0.05) and Africa (82%, z = 3.0, p < 0.05) are more satisfied with the system, while slightly less are satisfied in Europe (65%, z = 3.0, p < 0.05) and North America (65%, z = 4.0, p < 0.05; see Figure 3).

Figure 1.

Satisfaction with peer review by subject. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Figure 2.

Satisfaction with peer review by age. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Figure 3.

Satisfaction with peer review by region. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

The Importance and Sustainability of Peer Review

Researchers' general attitudes towards peer review (see Figure 4) show that they believe peer review plays a vital role in scholarly publishing, 84% (±1.1%, p < 0.05, n = 4,037) and believe that “without peer review, there is no control in scientific information.” However, it is clear that most researchers believe there could be improvements to the process. Just 32% (±1.4%, p < 0.05, n = 4,037) say that the current peer review system is the best we can achieve. This percentage increases with age, ranging from 25% (z = 5.8, p < 0.05) for the “under 36s” to 40% for the “over 65s” (z = 2.5, p < 0.05; see Figure 5). However, the evidence shows that a minority of researchers (30% ±1.4%, p < 0.05, n = 4,037) believe that “peer review in journals needs a complete overhaul.” Few respondents (19%, ±1.2%, p < 0.05, n = 4,037) believe that peer review is unsustainable because there are too few reviewers. Although there are some small differences by subject, those in mathematics and computer sciences are more concerned by its sustainability (25%, z = 5.8, p < 0.05). When reviewers were asked whether they enjoy reviewing and will continue to review, 86% (±1.1%, p < 0.05, n = 3,597) said they would (see Figure 12).

Figure 4.

Attitudes towards peer review by subject. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Figure 5.

Attitudes towards peer review by age. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

An examination of the statements by subject shows a number of significant differences. Materials sciences researchers have the most confidence in peer review in its present form; 45% (z = 4.06, p < 0.05) believe “the current peer review system is the best we can achieve,” while just 22% (z = 10.1, p < 0.05) of those in arts and humanities, social science, and economics have the same opinion (see Figure 4).

The Purpose of Peer Review

To better understand the perceived purpose of peer review, researchers were requested to indicate the extent to which they thought a given objective “should” be the purpose of peer review as well as the extent to which peer review is able to deliver on that objective. These objectives are shown in Table 8.

Table 8. Objectives of peer review
ALLAgriculture/Biological SciencesArts & Humanities/Social Sciences/EconomicsAstronomy/PhysicsBiochemistry, Genetics & Molecular Biology/Immunology/MicrobiologyChemistry/Chemical EngineeringMathematics/Computer Sciences & ITEarth & Planetary Sciences/Enironmental SciencesElectrical/Electronic Engineering/Engineering & TechnologyMaterials SciencesMedicine & Allied Health/NursingNeuro-sciencesPharmacology, Toxicology, and Pharmaceutics
  1. Note. Scores marked with * denote a significant difference between the subgroup and the overall score at 95% confidence levels.
That is selects the best manuscripts for the journalshould is able85.584.095.0*75.1*88.184.586.784.283.083.787.389.978.9
60.864.264.053.7*57.765.559.462.757.064.461.759.053.9
Determines the orginality of the manuscriptshould is able92.089.9*91.190.995.0*92.994.593.094.189.190.991.692.1
64.364.457.1*64.364.664.262.967.764.569.863.568.072.4
Improves the quality of the published papershould is able94.394.996.489.7*96.192.692.296.291.3*93.696.6*94.994.7
77.582.5*78.272.0*76.875.274.682.9*75.773.378.279.273.7
Ensures previous work is acknowledgedshould is able80.578.385.6*79.181.581.081.683.584.2*76.774.5*82.684.2
54.153.561.4*52.348.1*51.958.261.4*58.251.546.9*56.259.2
Determines the importance of findingsshould is able83.679.7*83.775.7*87.3*86.585.586.485.685.684.979.881.6
58.360.653.7*49.4*57.761.055.963.061.561.959.153.964.5
Detects plagiarismshould is able81.180.677.2*79.181.284.582.885.8*86.5*83.776.8*77.080.
37.736.928.5*38.936.541.951.2*41.545.4*46.5*28.9*29.8*39.5
Detects fraudshould is able78.974.8*73.9*80.380.184.8*81.382.384.2*84.274.5*77.072.4
32.731.318.0*32.630.141.0*50.4*42.4*40.9*43.6*23.7*21.9*28.9

Ninety-four percent (±0.7%, n = 4,037, p < 0.05) felt that improving the quality of a published paper “should” be the purpose of peer review, and 77% (±1.3%, n = 4,037, p < 0.05) agreed that peer review “is able” to do this. Most researchers (86% ± 1.1%, n = 4,037, p < 0.05) believe that peer review selects the best manuscripts for the journal. The number of researchers agreeing that this is the purpose of peer review is particularly high among those in the arts, humanities, social sciences, and economics, at 95% (z = 5.593, p < 0.05).

The biggest gaps between expectation and reality occur for detecting fraud and plagiarism. Eighty-one percent (±1.2%, n = 4,037, p < 0.05) expect peer review to detect plagiarism, but just 38% (±1.5%, p < 0.05) feel that the current system is able to do this. Similarly, 79% (±1.3%, p < 0.05, n = 4,037) would like peer review to detect fraud, compared with 33% (±1.5%, p < 0.05, n = 4,037) who feel it is successful in this aspect. However, a review of comments to the question “How can peer review be improved for authors?” suggests that neither of these two aspects were foremost in researchers' minds.

There were some notable differences in priorities when breaking the data down by subject area. The astronomy/physics community shows some distinct differences of opinion in several areas. Their expectations of peer review are the lowest for three of the attributes (manuscript selection, improves the quality of the manuscript, and determining importance). However, the gap between expectation and what is felt to be the current capabilities of the peer review system is often less than in other areas.

Mathematics, computer sciences, and IT respondents are most confident in peer review's ability to detect fraud and detect plagiarism (50.4%, z = 5.14, p < 0.05 and 51.2%, z = 6.56, p < 0.05, respectively). Least confident are those in the arts, humanities, social sciences, and economics, with scores of 18% (z = 8.97, p < 0.05) and 28.5% (z = 1.94, p < 0.05), respectively.

Effectiveness of Different Types of Peer Review

When commenting on the efficacy of different types of peer review most (76%, ±1.3% p < 0.05, n = 4,037) indicated that double-blind peer review was the most effective method, with 45% (±1.5%, p < 0.05, n = 4,037) believing that single blind was effective. Twenty percent (±1.2%, p < 0.05, n = 4,037) of respondents rated open peer review as effective (See Figure 6). It is worth noting that the percentage agreeing that double-blind peer review is effective is 5% higher than in the 2007 peer review study (t = 4.7, p < 0.05) by Ware and Monkman (2008). When these scores are broken out by subject area of the respondent, there are some distinct differences between subgroup and overall group.

Figure 6.

Types of peer review thought to be effective by subject. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Single-blind peer review is perceived as least effective by those within the arts and humanities, social science, and economics (26%, z = 8.32, p < 0.05) and medicine and allied health/nursing (32%, z = 7.13, p < 0.05). These two groups are the strongest advocates of double-blind peer review: 87% (z = 5.83, p < 0.05) for arts and 83% for medicine (z = 4.51, p < 0.05). Double-blind peer review is thought to be least effective by those in the astronomy/physics community (66%, z = 4.53, p < 0.05), who have a significantly higher preference for single blind than the group as a whole (53%, z = 3.14, p < 0.05) compared with 45%.

Respondents were also asked to consider the effectiveness of usage statistics in the absence of peer review and supplementing peer review with postpublication evaluation. We did not specify a particular type of usage statistic and did not identify whether usage statistics would be normalized. Fifteen percent (±1.1%, p < 0.05, n = 4,037) believed that usage statistics instead of formal peer review would be effective. This is highest among materials scientists (24%, z = 3.67, p < 0.05) and lowest among those in the arts and humanities and the social sciences (9%, z = 3.62, p < 0.05). Supplementing peer review with postpublication review was considered effective by 47% (±1.5%, p < 0.05, n = 4,037) more in medicine (54%, z = 4.87, p < 0.05) thought it effective, while the most sceptical respondents were in chemistry (35%, z = 3.68, p < 0.05).

Reasons Why Different Types of Peer Review Were Considered Effective

Respondents indicated why they thought a particular form of peer review was effective or not effective. Below are some comments made by respondents, which have been extracted to reflect the balance of opinion.

Single-blind peer review—why it is considered effective

Researchers perceive that it eliminates bias, encourages forthright opinion, and allows the reviewer to focus on the quality of the manuscript:

A junior referee may freely review the paper of a powerful author or his working group without being afraid to experience disadvantages in his own career. (aged 46–55 years in medicine from Germany)

I'm not sure I would volunteer as easily, or be as honest, if the authors knew who I was. If I didn't like the article but knew the people, it would probably bias me. (aged 36–45 years in electrical & electronic engineering from the United States)

Single-blind peer review—why it is NOT considered to be effective

Those researchers who did not believe this was effective think it is biased against less prestigious institutes and developing countries, with too much respect/credibility given to established authors. There is a perception that it allows competitors to delay research under the auspices of review.

Basically because this is the type of peer review in which many kinds of bias are most prone to occur, including bias against papers produced in less prestigious institutes (or geographic regions), against young or rather unknown researchers, or against an author to whom a reviewer is unsympathetic for various reasons (including non-scientific ones). (aged 36–45 years in microbiology from Brazil)

Invites pandering and corruption of the process – becomes a venue for revenge and favors – terrible idea! (aged 36–45 years in social sciences from the United States)

Double-blind peer review—why it is considered effective

Researchers believe it eliminates bias, encourages forthright opinion, and allows the reviewer to focus on the quality of the manuscript:

Acceptance or rejection should be completely dependent upon the quality of the work, not the authors. Reviewers are human and inherently subject to bias. By minimizing those parameters that contribute to bias, we increase the likelihood of an objective review. (aged 56–65 years in biochemistry from the United States)

Academic integrity and publication ethics can be achieved by double-blind review as the reviewing process will be solely focusing on material submitted and research findings. (aged 26–35 years in computer science/IT from Malaysia)

It is clear in our discipline (poli sci) that when double-blind was implemented in 1969 that the authors who were published changed, with many supposedly high quality scholars no longer able to get their work in print. (aged 26–35 years in social sciences from the United States)

Double-blind peer review—why it is NOT considered to be effective

Researchers believe you can never truly hide the identity of the author–citations, subject, or style will give the author away. Others feel that knowing the author is important for better understanding the paper and helps identify plagiarism.

Because the authors of a manuscript almost always will be known to the reviewers, simply based on the citation pattern. For this to work, we would need to write our articles in a very different way than we do today. (aged 56–65 years in biological sciences from the United States)

The authors should be known to better detect “autoplagiarism.” (aged 26–35 years in engineering and technology from Canada)

Open peer review—why it is considered effective

Researchers believe it ensures that reviewers are honest, more thoughtful, and less likely to be vitriolic in their evaluation:

Anonymous reviewers are too often cowards, who take inappropriate and unsubstantiated pot-shots. Making reviewers sign their reviews keeps them honest. … (aged 36–45 years in biological sciences from the United States)

We generally give more thought to documents we sign. The author and reviewer engage in scientific debate. (aged older than 65 years in earth and planetary sciences in the United States)

Open peer review—why it is NOT considered to be effective

It encourages reviewers to be less critical, is likely to put off younger reviewers, gives the author the opportunity to influence the reviewer, and encourages dispute between reviewers and authors.

A completely open process would inhibit reviewers from being completely honest about a paper, especially when it is of poor quality. It could especially inhibit junior reviewers who might worry about angering a senior researcher in the field. (aged 36–45 years in social sciences from the United States)

Because I know very few scientists that would accept to serve as referees under these conditions. (aged 36–45 years in physics from Croatia)

Open peer review and published review – why it is considered to be effective

In addition to ensuring that reviewers are honest, more thoughtful, and less likely to be vitriolic in their response, publishing names and reports helps the reader decide on the quality of the work and encourages dialogue.

Because it would put much more pressure on the reviewers to be fair and thorough. Reviewers would be held responsible publicly for the articles that they reviewed: real accountability is what is missing from the current system. (aged 26–35 years in medicine and allied health from Canada)

Because this would preclude review reports based on personal prejudice, which sometimes is the case, particularly if newer research challenges previous “accepted” opinions. If the reviewer knew that his/her report would be published, a more science based approach is more likely. (aged over 65 years in medicine and allied health from Norway)

Open peer review and published review—why it is NOT considered to be effective

Respondents felt that some cultures might be more adversely affected by having their reports made public, probably due to the nature of hierarchies in those societies, and others believe that the reports will be uncritical.

Because in this case, it will be difficult for him/her to critically give comments to the author in some culture. (aged 36–45 years in economics from China)

Because people may be reluctant to identify a poor piece of work if it were from a leader in the field. (aged 36–45 years in neuroscience from the United Kingdom)

Supplementing review with postpublication—why it is considered to be effective

Respondents felt that this encourages dialogue, broadens the scope of the comments beyond two or so reviewers, and allows subsequent short-comings to be linked directly to the article.

A typical peer-reviewed paper is read by a handful of people prior to publication. This sample may be biased and/or someone may simply be having a bad day. Postpublication review would add an additional round of checks and balances—the authors would get credit for the publication, but errors would not be allowed to live on. (aged 26–35 years in social sciences from Hong Kong)

Errors and ambiguities often escape the review process. Traditionally, such shortcomings are identified in subsequent publications, but it would be good to have these errata associated with the source of the original publication. (aged 36–45 years in computer science/IT from Canada)

Supplementing review with postpublication—why it is NOT considered to be effective

Researchers worry about a lack of editorial control and that it would lead to endless debate, while others prefer this type of discussion to take place at conferences.

1) There is no quality control on online readers 2) there is no editor functioning as moderator 3) detailed discussion will confuse non-specialist readers. (aged 36–45 years in biochemistry from Germany)

In my opinion, this would generate a mess – an endless sequence of comments and responses. (aged 56–65 years in earth and planetary sciences from the Czech Republic)

Replacing peer review with usage statistics—why it is considered effective

Researchers believe that usage statistics would be faster, likely more objective, better suited to deal with the large volumes of papers, and would not preclude “negative” results papers that are equally valid.

In this age, a good paper will be found and used by many. The peer review process is very painful and often ineffective because reviewers frequently do not offer useful feedback and/or are ill qualified to review the material they are given. (aged 36–45 years in engineering and technology from the United States)

Sometimes statistics can more objective than the reviewer. (aged 36–45 years in microbiology from Turkey)

Replacing peer review with usage statistics—why it is NOT considered effective

Researchers are concerned about a delay in meaningful statistics and that papers are not improved and science will become a popularity contest. Moreover, there is a belief that in a world of usage statistics and there will not be allowance for subject differences in respect to download behaviour.

A popular paper does not make a good scientific paper. The tabloids sell more newspapers than the broadsheets–does this mean they contain better journalism? Also, there is so much information to read that there needs to be some check on what is correct and what is not so we do not waste our time reading papers that are fundamentally flawed. (aged 26–35 years in electrical and electronic engineering from Australia)

Fashionable topics usually get more downloaded or citations. That does not necessarily mean a paper is good or bad. Also number of citations across different topics cannot be compared. (aged 36–45 years in physics from India)

Authors' Experiences of Peer Review

Researchers have different expectations of peer review depending upon their relationship with the process. For example, a researcher as an author has different needs and expectations than a reviewer. The length of peer review is an important consideration for authors, as noted by Grainger (2007). Forty three percent of respondents (±1.5%, p < 0.05, n = 4,037) felt that the time it took to the first decision for their last article was slow or very slow (see Figure 7). This dropped slightly to 42% (±1.5%, p < 0.05, n = 4,037) when respondents evaluated the time to reach a final decision (See Figure 8).

Figure 7–8.

Perception of the speed of peer review. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Splitting the time taken for review by subject highlights some differences (Figure 9). Astronomy/physics and chemistry/chemical engineering had just 15% of reviews taking longer than 6 months (z = 6.9, p < 0.05), compared with 31% for the overall group. The comparable figure for the arts and humanities, social science, and economics was 59% (z = 13.0, p < 0.05).

Figure 9–10.

Perception of the speed of peer review by subject. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

However, researchers' expectations are attuned to the speeds of their own field as can be seen in Figure 10, which shows their perception of speed. Here, the scores are more evenly balanced and there is little statistical variance across the groups. Electrical/electronic engineering, engineering, and technology is the only group that substantially deviates from the average, with 6% (z = 2.9 p < 0.05) believing that the reviewing time was “very fast” or “fast.” Perhaps most surprising is that arts and humanities, who have the slowest time, have the joint highest percentage of researchers saying that the time taken was very fast or fast (12%).

Areas of Recent Article Most Improved by Peer Review

Ninety-one percent (±0.9%, p < 0.05, n = 4,037) of respondents agreed that the review process improved the quality of the last paper they published.

Peer review is considered to be more effective in improving some sections of the article than others. Respondents felt that the discussion section is most likely to be improved by peer review, with 91% (±0.9%, p < 0.05, n = 4,037) recording some level of improvement (see Figure 11). Improvements were least likely for the statistics section, although it is likely that the “no improvement” category included researchers who did not have a statistical section. Oceania recorded the lowest levels of improvement across the regions for all areas except discussion. Asia shows the highest level of improvement for language/readability, reflecting the challenges of translating articles for English language journals.

Figure 11.

Areas of the article improved during peer review by region. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Reviewer Motivations and Improvement Areas

For the peer review process to remain sustainable, it is important that the reviewer experience does not deter repeated participation. While an author has the potential reward of a published paper at the end of a review, it is perhaps more difficult to identify the gains for reviewers. In spite of the lack of an obvious reward, the vast majority of respondents (86%, ±1.1%, p < 0.05, n = 3,597) indicated that they enjoy reviewing and will continue to review (see Figure 12).

Figure 12.

Improvement areas and sustainability of peer review by age. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

An analysis of this statement by age group shows the likelihood of agreement drops with increased age. Of those aged younger than 36 years of age, most agree (89%, z = 2.9, p < 0.05), while fewer of those aged 56 to 65 years agree (80%, z = 3.9, p < 0.05). Predictably, perhaps, the younger age group also has the highest agreement across all of the age groups on the statement “there is generally a lack of guidance on how to review papers” (60%, z = 3.4, p < 0.05). The level of agreement decreases for each subsequent increase in age group to the lowest level of agreement for those older than 65 years of age (46%, z = 2.9, p < 0.05). The majority of respondents (68%, ±1.5%, p < 0.05, n = 3,597) feel that formal training would improve the quality of reviews, while 73% (±1.5%, p < 0.05, n = 3,597) believe that technological advances in the previous 5 years have made it easier to complete a more thorough review.

Reasons for Declining to Review

The majority of respondents (61%, ±1.6%, p < 0.05, n = 3,597) had declined to review on at least one occasion in the preceding 12 months. The average number of declines was 2.2 per annum (see Table 9). When asked for the main reasons for declining a review, the majority (58%, ±1.6%, p < 0.05, n = 3,597) stated it was because the paper was outside their area of expertise (see Figure 13).

Figure 13.

Reasons for declining to review by subject. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Table 9. During the last 12 months, how many times have you declined an invitation to review?
Range (for declines)Frequency% of baseReverse cum. %Range mid- valueRange * freq
01,41339%61%00
1–21,32637%63%1.51989
3–559417%83%42376
6–101875%95%81496
11–20481%99%15.5744
21–50251%99%35.5887.5
More than 5040%100%95380
Total3,597100% 7872.5
Average no. of declines2.2

The paper being outside the reviewer's expertise proved to be a particular problem in the area of materials sciences, with 82% (z = 2.9, p < 0.05) stating this as one of the main reasons. Being too busy with work was the second most frequently mentioned reason (49%, ±1.6%, p < 0.05, n = 3,597), while prior reviewing commitments came third (30%, ±1.6%, p < 0.05, n = 3,597). This latter reason was a particular issue in the arts and humanities, social science, and economics (43%, z = 6.2, p < 0.05).

Time Spent Reviewing

In respect to time spent reviewing a manuscript, medicine, allied health and nursing arts and humanities reviewers spent less time reviewing than any other group; 35% (z = 4.8, p < 0.05) 45% (z = 2.5, p < 0.05) spent 6 hours or more on the last paper they reviewed compared with 53% across all groups (see Table 10). However, it is worth noting that arts and humanities also had the highest proportion of people expecting to do between 6 and 10 reviews per annum (Figure 14).

Figure 14.

Number of reviews completed per annum by subject. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Table 10. The amount of time spent reviewing
SubjectHours spent reviewing
0–56–1011–2021–3031–5051–100100+
Agriculture and biological sciences41%37%13%5%2%2%1%
Arts and humanities, social sciences, and economics55%31%9%2%1%1%1%
Astronomy and physics42%28%15%6%5%3%1%
Biochemistry, genetics and molecular biology, immunology, and microbiology50%29%12%3%3%1%1%
Chemistry and chemical engineering52%24%11%5%5%2%1%
Mathematics and computer sciences and IT29%27%20%9%12%3%0%
Earth and planetary sciences and environmental sciences29%32%21%7%7%1%2%
Electrical, electronic engineering, engineering, and technology39%28%16%7%5%3%2%
Materials sciences44%30%11%5%6%3%1%
Medicine and allied health and nursing65%22%8%2%2%1%0%
Neurosciences55%31%9%3%1%1%0%
Pharmacology, toxicology, and pharmaceutics46%25%16%4%4%3%1%
All47%29%13%5%4%2%1%

Effect of incentives on the Willingness to Review

Respondents were asked whether certain incentives would make them more or less likely to review for a journal. Initiatives aimed at garnering recognition and increasing transparency in peer review are generally not popular among reviewers. Just over half of reviewers (51%, ±1.6%, p < 0.05, n = 3,597) said they would be less likely to review if the reviewers' names are revealed to the author (see Figure 15). Forty-five percent (±1.6%, p < 0.05, n = 3,597) of reviewers indicated they are less likely to review if the reviewers' names are published alongside the article. The reviewers reports being published alongside the articles are most likely to discourage reviewers, with 58% (±1.6%, p < 0.05, n = 3,597) indicating they would be less likely to review. Comments made by reviewers in respect to these initiatives suggested a fear of retribution from authors whose paper they may have been critical of or for which they had recommended rejection. Payment in kind by the journal (e.g., free subscription, waiver of colour, or other publication charges etc.) was the option most likely (51%, ±1.6%, p < 0.05, n = 3,597) to encourage future reviews.

Figure 15.

Percentage of reviewers that thought a type of incentive was more or less likely to make them review. (Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.)

Discussion

The results from this study suggest that researchers believe that peer review remains an essential element of scholarly publication. The majority are satisfied with the peer review system used by scholarly journals. Furthermore, the vast majority believe that without formal peer review there is no control in scientific communication. In spite of this, few, just under a third, believe it is the best we can achieve. Although the majority do not appear to be in favor of replacing peer review, many believe it could be improved and there seems to be some agreement on suggested areas for improvement. Almost half believe that supplementing peer review with some form of postpublication commentary would be beneficial. However, it is worth noting that journals such as PLoS ONE already provide researchers the ability to post comments. The limited number of comments on such journals suggests that in practice there may be some constraints on this form of commentary.

When alternatives for peer review are posited, for example, replacing peer review with general usage statistics, support was not particularly strong. However, we have to be cautious in our interpretation; if we had tested usage statistics as a supplement to peer review instead of a replacement of peer review, the response may have been more positive.

In respect of the purpose of peer review, researchers believe that its most important function is to improve the quality of research published. This is closely followed by the need to determine the originality of papers as well as to select the best papers for publication. The ability of peer review to deliver on these objectives can lag behind expectation by 15 or more percentage points. Most do believe that peer review is able to deliver on its most important objective of improving papers, and there is little variance across subjects in this belief. Moreover, when researchers considered their most recent published article, 91% indicate that their paper was improved by peer review, particularly the discussion section. Studies elsewhere also suggest that peer review improves the quality of published research (Goodman, Berlin, Fletcher, & Fletcher, 1994).

The gap between expectation and the ability of peer review to deliver on its objectives is greatest when the research community considers the issues of fraud and plagiarism. Notwithstanding this gap and some high profile cases of fraud, generally researchers do not believe fraud to be prevalent. Very few respondents mentioned fraud as an issue when asked how peer review might be improved. Data recently released by the Web of Science tend to support this position. Van Noorden (2011) reports that the number of articles retracted in 2011 is approximately 400. While this figure is higher than previous years, estimates suggest that only 44 (11%) will be due to fabrication or falsification. Considering that over 1.3 million articles are published per annum (Bo-Christer, Annikki, & Mari, 2009) and the year-on-year increase in the number of articles is approximately 3% to 4% (Mabe & Amin, 2002), this represents a very small proportion.

When considering the practicalities of detecting fraud, short of repeating experiments, which is impractical if not impossible for reviewers, it is difficult to see how peer review can achieve these objectives effectively. It is conceivable that the increasing availability of data alongside the article will deter the submission of fraudulent work, as reviewers will be able to check the data. However, it is unlikely to deter the determined fraudster. Nonetheless, a recent report by the U.K. government's Science and Technology Select Committee suggests that protocols should be established to minimize fraud (http://www.%20publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/85602.htm). Publishers have developed tools such as CrossCheck, which help to spot plagiarism. As Van Noorden (2011) asserts, it is likely that tools such as CrossCheck have influenced the increases in retractions.

Taking researchers as a whole, there is broad agreement about the purpose and ability of peer review to deliver: however, one size does not fit all. The desire to publish results quickly in some fields has led to the development of preprints. A preprint is an early version of the author's manuscript prior to peer review that is posted onto a preprint server. While not formally peer reviewed, preprints are often categorized, commented on, and endorsed by moderators, and so there is some form of evaluation.

Preprint servers are available in a number of disciplines, but there is a particularly strong tradition of using preprint servers in the physics and astronomy community. The preeminence of the preprint repository arXiv is likely to have an influence on the perceptions within this community upon peer review. Those in astronomy and physics indicate they value peer review less. They consistently believe that peer review is less likely to be able to select the best manuscripts, improve the paper, and determine the importance of findings and least likely to believe it currently fulfils these objectives. However, it would be wrong to conclude that those in physics and astronomy do not value peer review. Many of the papers that appear in arXiv go on to be published in peer reviewed journals (Mine, 2009). Clearly, many researchers consider the final version of the article in the peer reviewed journal to be valuable.

Reviewers believe that maintaining anonymity makes peer review more effective and double-blind review is most preferred. Across all subjects this preference is high, but is particularly strong among those in medicine, allied health, and nursing, as well as those in the arts and humanities, social sciences, and economics. Comments from researchers suggest there is a stronger suspicion of bias in these fields that may partly explain this preference. However, in spite of the popularity of the double-blind peer review, many identified weaknesses in this form of peer review that are difficult to overcome. In some research areas, particularly those that are more specialized, researchers believe the nature of the work itself would reveal the identity of the authors (or at least the group) behind the study. Work by Van Rooyen et al. (1999) showed that reviewers could identify the authors in about a quarter to a third of cases. It is quite feasible that such a figure would be higher today given the advances in search technology in the interceding years. Moreover, in Van Rooyen's study, when blinding was successful there was no evidence that the quality of the review was improved.

Open peer review is a broad term and in this study we examined two forms of open peer review. The first is the narrower, where authors, but not readers, know the identity of the reviewers. Few (20%) thought that this form of peer review was effective. The second and more transparent form of peer review, which included the publication of reviewer's report along with the reviewer's name, was perceived as more effective (25%). As previously noted open peer review is more common in the biomedical field. Thus, it is a little surprising to observe that respondents in the medical and life science fields would consider open peer review to be no more effective than other disciplines. Reviewer comments suggest there is a degree of scepticism regarding open peer review, notably because it encourages reviewers to be less than forthright. Nature's evaluation of their experiment with open peer review in 1996 tends to support this position (Campbell, 2006), as does a recent research study measuring the effect of open peer review on the BMJ by Van Rooyen and colleagues. This study indicated that open peer review deters reviewers and does not improve the quality of reviews submitted (Van Rooyen et al., 2010).

Few thought that peer review was unsustainable because of lack of willing reviewers. Moreover, the vast majority of reviewers believe reviewing is an important part of being a researcher: 86% enjoy reviewing and are committed to reviewing in the future. Concerns about the engagement of younger researchers are likely to be misplaced, as more young reviewers than any other age group say they are happy to review in the future.

Recommendations

Technological advances during the 5 years prior to this study have made it easier for reviewers to complete reviews that are more thorough. Moreover, the results suggest that some simple innovations will help further improve the efficiency and quality of peer review.

  • Over half of reviewers spend 6 hours or more reviewing a single paper. Greater triaging of papers by editorial staff prior to peer review would help ensure that manuscripts meet minimum standards, which would likely positively affect this timeframe.
  • The most frequently mentioned reason for declining to review is that the paper was outside the expertise of the reviewer. The enrichment of thesauri and improvements in the controlled vocabularies within publisher systems should improve the matching of articles to potential reviewers.
  • To ensure that reviewers offer appropriate advice as well as identify the best papers, researchers believe that formal training of reviewers would be beneficial.
  • Clearer guidelines and consistent review templates would improve standards of reviewing. Those younger than 36 years of age indicate they are most likely to benefit from such measures.
  • The introduction of incentives, such as payment or payment in kind, and/or bestowing greater public recognition, specifically an acknowledgment in the journal, is likely to encourage more researchers to participate in peer review.

Future Research

It is recommended that future studies explore why there are differences between certain subject fields and examine the attitudes of researchers towards new initiatives in peer review, including the following:

  • “Impact free” peer review introduced on journals, such as PLoS ONE and BMJ Open, specifically the role of reader comments and ratings in identifying high quality research.
  • The value of social media, notably reference managers such as Mendeley and social bookmarking tools and alternative metrics (altmetrics) that could be used to assess the quality of published research, as postulated by Taraborelli (2008) and examined by Priem and Hemminger (2010).
  • The effectiveness of third-party review sites such as PaperCritic (http://www.papercritic.com/) and Peerevaluation (http://www.peerevaluation.org/), which utilize crowd sourcing to complete peer review.
  • Finally, there is the proliferation of data to consider. Increasingly, the data upon which research papers are based are being made available publicly. Is it feasible or indeed desirable that such data be reviewed?

Conclusion

While researchers recognize that peer review is imperfect, it appears that most believe it is the most effective mechanism for ensuring the reliability, integrity, and consistency of the scholarly literature. It is quite possibly an overused quotation, but Winston Churchill's rationale for continuing with democracy still applies to peer review: “Democracy is the worst form of government except all those other forms that have been tried” (U.K. Parliament, 1947 [http://hansard.millbanksystems.com/commons/1947/nov/11/parliament-bill#column_206]).

Clearly peer review does not meet all expectations, specifically in respect to fraud. As Weller (2001) articulates, it is evident that while peer review cannot guarantee that work is not fraudulent or always of high quality, it does, however, make it far more likely. It is not a question of peer review being reinvented, but evolving. There is clearly some dissatisfaction with peer review, but rather than being “in crisis” or “broken,” as suggested by Myers (2009), the opposite could be argued, namely, that peer review is going through a period of vibrant innovation and renewal.

Appendix: Appendix

Sense About Science – Peer Review Survey 2009

i1 – Introduction

Sense About SciencePeer Review SurveyWelcome to the 2009 Peer Review Survey, and thank you for participating. Your responses will be used to identify trends from comparison with the 2007 Peer Review Survey of authors and reviewers and to explore some new issues that are likely to affect editors, publishers, reviewers and authors in the next few years.This survey should take about 10 minutes – please do complete the whole survey as this will enable us to derive the most value from your responses.

q2 – Screener

To begin with, please indicate the number of academic peer-reviewed papers (articles):a) You have published in your career to date:

  • ○ 0 (1)
  • ○ 1–5 (2)
  • ○ 6–10 (3)
  • ○ 11–20 (4)
  • ○ 21–50 (5)
  • ○ 51–100 (6)
  • ○ more than 100 (7)

q3 – Screener 2

b) You have reviewed in the last year:

  • ○ 0 (1)
  • ○ 1–5 (2)
  • ○ 6–10 (3)
  • ○ 11–20 (4)
  • ○ 21–50 (5)
  • ○ more than 50 (6)
STOPScreened – Thank you
Thank you for your assistance. Unfortunately, as you have indicated that you have not published any peer-reviewed papers, we shall not be able to make use of your responses.

q4 – Peer review generally

Overall, how satisfied are you with the peer review system used by scholarly journals?

  • ○ Very Satisfied (1)
  • ○ Satisfied (2)
  • ○ Neither satisfied nor dissatisfied (3)
  • ○ Dissatisfied (4)
  • ○ Very Dissatisfied (5)

q5 – Peer review generally 2

Please indicate the extent to which you agree with the following statements

Check one box in each row

Strongly Agree (1)Agree (2)Neither Agree nor Disagree (3)Disagree (4)Strongly Disagree (5)Don't Know/Not Applicable (6)
The current peer review system is the best we can achieve (1)
Peer review is a concept understood by the public (2)
Peer review is a concept well understood by the scientific community (3)
Peer review in journals needs a complete overhaul (4)
Peer review is holding back scientific communication (5)
Peer review is biased against authors who are from developing countries (6)
Scientific communication is greatly helped by peer review of published journal papers (7)
Peer review is unsustainable because there are too few willing reviewers (8)
Without peer review there is no control in scientific communication (9)
It is reasonable that journal editors evaluate and reject a proportion of articles prior to external peer review (10)

q6 – Effectiveness

For research papers published in your field, to what extent do you agree that the following types of peer review are/would be effective?

Please check one box in each row

Strongly Agree (1)Agree (2)Neither Agree nor Disagree (3)Disagree (4)Strongly Disagree (5)Don't Know/Not Applicable (6)
Single-blind peer review (where the authors' names are known to the reviewer but the reviewers' names are not known to the author) (1)
Double-blind peer review (where neither the author nor the reviewers are known to the other) (2)
Open peer review (where the authors and reviewers are known to each other, but the reviewers' reports are kept private to themselves and the journal) (3)
Open & published peer review (where the authors and reviewers are known to each other and additionally the reviewers' signed reports are openly published alongside the paper) (4)
Supplementing review with post-publication review (where online readers comment on and/or rate the paper following publication) (5)
Peer review could in principle be replaced by usage statistics (for instance the number of times a paper is downloaded) or citation data to identify good papers (6)

q7new – Follow up verbatim

You <Pipe text for Agreed/ Disagreed rating for one type of peer review> that the following would be an effective type of peer review <Pipe: Type of peer review> Why do you say that?

q9 & q10 – objectives

There are some differences on what individuals believe peer review should do, is currently able to do, and how well it meets these objectives.

To what extent do you agree or disagree that the following objectives should be the purpose of peer reviewTo what extent do you agree or disagree that peer review is currently able to do the following?
Strongly Agree (1)Agree (2)Neither Agree nor Disagree (3)Disagree (4)Strongly Disagree (5)Don't Know/Not Applicable (6)Strongly Agree (1)Agree (2)Neither Agree nor Disagree (3)Disagree (4)Strongly Disagree (5)Don't Know/Not Applicable (6)
That it selects the best manuscripts for the journal (1)
Determines the originality of the manuscript (i.e. novel and new) (2)
Improves the quality of the published paper (3)
Ensures previous work is acknowledged (4)
Determines the importance of findings (5)
Detects plagiarism (6)
Detects fraud (i.e. results that are falsified) (7)
Other (8)____________

q11 – Revision

In this next section we are going to ask you some questions about the last peer-reviewed paper that you had accepted for publication. Did you have to revise the paper?

  • ○ Yes (1)
  • ○ No (2)

q12 – Revision 2

Do you think that the peer review process improved the quality of the paper?

  • ○ Yes (1)
  • ○ No (2)

q13 – Aspects

Which aspects were improved and to what extent?

Please rate using the scale 1–5, where 1 = no improvement and 5 = substantial improvement

1 – no improvement (1)2 (2)3 (3)4 (4)5 – substantial improvement (5)
Language or readability (1)
Methodology (2)
Results/Analysis (3)
Statistics (4)
Presentation (figures/tables etc) (5)
References (identified missing or inaccurate references (6)
Discussion (7)

q14 – Length of peer review

Approximately how long did the peer review process take?

1 week or less (1)2–3 weeks (2)1 to 2 months (3)3 to 6 months (4)More than 6 months (5)
From submission to first decision? (1)
For any revision stages that you undertook? (2)
For submission to final acceptance by the journal? (3)

q15 – Opinion of speed

How did you feel about the reviewing time:

Very slow (1)Slow (2)Acceptable (3)Fast (4)Very fast (5)
From submission to first decision? (1)
From submission to final acceptance by the journal? (2)

q16 – Rejection

To how many journals (if any) did you submit your last paper before it was accepted?

  • ○ 0 (1)
  • ○ 1 (2)
  • ○ 2 (3)
  • ○ 3 (4)
  • ○ 4 (5)
  • ○ 5 (6)
  • ○ 6 (7)
  • ○ 7 (8)
  • ○ 8 (9)
  • ○ 9 (10)
  • ○ 10 or more (11)

q17 – How could peer review be improved

How do you believe peer review could be improved for authors?

q18 – Reasons for reviewing

Experience as a journal reviewerIn this section we consider your role as a reviewer (sometimes called a referee). Please indicate the extent to which you agree that the following reasons describe why you review.

 Strongly Agree (1)Agree (2)Neither Agree nor Disagree (3)Disagree (4)Strongly Disagree (5)Don't Know/Not Applicable (6)
I like playing my part as a member of the academic community (1)
I enjoy seeing new work ahead of publication (2)
I want to reciprocate the benefit gained when others review my papers (3)
I enjoy being able to help improve a paper (4)
I believe it will enhance my reputation or further my career (5)
It will increase my chances of being offered a role on the journal's editorial team (6)
I will gain personal recognition from reviewing (7)
It is an opportunity to build a relationship with the Editor (8)
It will increase the likelihood of my future papers being accepted (9)

q19 – Reviewing generally

Please indicate the extent to which you agree with the following

 Strongly Agree (1)Agree (2)Neither Agree nor Disagree (3)Disagree (4)Strongly Disagree (5)Don't Know/Not Applicable (6)
There is generally a lack of guidance on how to review papers (1)
I enjoy reviewing and will continue to review (2)
With technological advances it is easier to do a more thorough review now than 5 years ago (3)
Formal training of reviewers should improve the quality of reviews (4)

q20 – Declined review

During the last 12 months, how many times have you declined an invitation to review?

  • ○ 0 (1)
  • ○ 1–2 (2)
  • ○ 3–5 (3)
  • ○ 6–10 (4)
  • ○ 11–20 (5)
  • ○ 21–50 (6)
  • ○ more than 50 (7)

q21 – Reasons for declining

Thinking of the last time you declined an invitation to review, what were the main reasons for declining?

Please select up to a maximum of three reasons

  • □ Too busy doing my own research, lecturing etc (1)
  • □ Personal reasons (e.g. holiday, sickness) (2)
  • □ Too many prior reviewing commitments (3)
  • □ Paper was outside my area of expertise (4)
  • □ Proposed deadline was too short to conduct a thorough review (5)
  • □ Journal was not on my preferred list of journals (6)
  • □ Poor scientific quality of the paper (7)
  • □ Poor quality English of the paper (8)
  • □ I have not declined a reviewing invitation recently enough to recall (9)
  • □ Conflict of interest (10)
  • □ Other (11)____________

q22 – Time taken to review

Now thinking specifically about the last paper you agreed to review, how much actual time did you devote to the review? (in hours)

q23 – Weeks spent

Approximately, what was the time between your acceptance of the invitation to review and the delivery of your report?

  • ○ 1 week (1)
  • ○ 2 weeks (2)
  • ○ 3 weeks (3)
  • ○ 4 weeks (4)
  • ○ 5 weeks (5)
  • ○ 6 weeks (6)
  • ○ 7 weeks (7)
  • ○ 8 weeks or longer (8)
  • ○ Unable to say, too long ago (9)

q24 – Who reviewed

The last paper you agreed to review was:

  • ○ Reviewed just by myself (1)
  • ○ Reviewed by a junior member(s) of my research group (2)
  • ○ Reviewed by a junior member(s) of my research group but under my overall direction and supervision (e.g. as part of their training) (3)
  • ○ Reviewed mostly by me, but colleagues contributed (4)
  • ○ Other (6)____________

q25 – Likelihood

Reviewers' opinions on peer reviewPlease say whether the following would make you more or less likely to review for a journal:

 Much less likely (1)Less likely (2)No difference (3)More likely (4)Much more likely (5)
Your name as the reviewer disclosed to the author only (1)
Your name being published alongside the paper as one of the reviewers (2)
Your signed report being published with the paper (3)
Acknowledgement in the journal (4)
Accreditation (CME/CPD points) (5)
Payment in kind by the journal (e.g. Free subscription, waiver of colour or other publication charges, free offprints etc) (6)
Payment by the journal (e.g. receive monetary amount) (7)

q26 – Payment follow up

Would you still want payment if the following was true? Author pays the fee

  • ○ Yes (1)
  • ○ No (2)

q27 – Payment follow up 2

Funding body pays the fee

  • ○ Yes (1)
  • ○ No (2)

q28 – Payment follow up 3

Publisher/Society covers the cost

  • ○ Yes (1)
  • ○ No (2)

q29 – Maximum papers to review

What is the maximum number of papers that you would be prepared to review in a year?

  • ○ 1–2 (1)
  • ○ 3–5 (2)
  • ○ 6–10 (3)
  • ○ 11–15 (4)
  • ○ 16–20 (5)
  • ○ 21–50 (6)
  • ○ more than 50 (7)

q30 – Timeliness

Do you submit your reviews on time?

  • ○ Always (1)
  • ○ Frequently (2)
  • ○ Sometimes (3)
  • ○ Rarely (4)
  • ○ Never (5)

q31 – Improvement for reviewers

How could the peer review process be improved for reviewers?

q32 – Technologies

About youPlease indicate which of the following technologies you use.

Select all that apply

  • □ RSS newsreader (1)
  • □ Blogs – I regularly read blogs (2)
  • □ Blogs – I have my own blog or actively comment on others' blogs (3)
  • □ Wiki – I have contributed to a work related wiki (4)
  • □ Social bookmarking services such as Del.icio.us, Connotea or CiteULike (5)
  • □ Handheld e-mail devices such as smartphones (6)
  • ○ None of the above (7)

q33 – Organisation

Which of the following best describes your organisation?

  • ○ University or College (1)
  • ○ Hospital or Medical School (2)
  • ○ Industry or Commerce (3)
  • ○ Research Institute (4)
  • ○ Government (5)
  • ○ Other (6)

q34 – Age

Which of the following categories does your age fall into?

  • ○ Under 22 (1)
  • ○ 22 to 25 (2)
  • ○ 26 to 35 (3)
  • ○ 36 to 45 (4)
  • ○ 46 to 55 (5)
  • ○ 56 to 65 (6)
  • ○ Over 65 (7)
  • ○ Prefer not to say (8)

q35 – Gender

And please could you indicate your gender?

  • ○ Female (1)
  • ○ Male (2)
  • ○ Prefer not to say (3)

q36 – Country

In which country do you live?

  • ○ Afghanistan (1)
  • ○ Albania (2)
  • ○ Algeria (3)
  • ○ American Samoa (4)
  • ○ Andorra (5)
  • ○ Angola (6)
  • ○ Anguilla (7)
  • ○ Antarctica (8)
  • ○ Antigua and Barbuda (9)
  • ○ Argentina (10)
  • ○ Armenia (11)
  • ○ Aruba (12)
  • ○ Australia (13)
  • ○ Austria (14)
  • ○ Azerbaijan (15)
  • ○ Bahamas (16)
  • ○ Bahrain (17)
  • ○ Bangladesh (18)
  • ○ Barbados (19)
  • ○ Belarus (20)
  • ○ Belgium (21)
  • ○ Belize (22)
  • ○ Benin (23)
  • ○ Bermuda (24)
  • ○ Bhutan (25)
  • ○ Bolivia (26)
  • ○ Bosnia and Herzegovina (27)
  • ○ Botswana (28)
  • ○ Brazil (29)
  • ○ British Indian Ocean Territory (30)
  • ○ BRUNEI (31)
  • ○ Brunei Darussalam (32)
  • ○ Bulgaria (33)
  • ○ Burkina Faso (34)
  • ○ Burundi (35)
  • ○ Cambodia (36)
  • ○ Cameroon (37)
  • ○ Canada (38)
  • ○ Cape Verde (39)
  • ○ Cayman Islands (40)
  • ○ Central African Republic (41)
  • ○ Chad (42)
  • ○ Chile (43)
  • ○ China (44)
  • ○ Christmas Island (45)
  • ○ Cocos (Keeling) Islands (46)
  • ○ Colombia (47)
  • ○ Comoros (48)
  • ○ Congo (49)
  • ○ Cook Islands (50)
  • ○ Costa Rica (51)
  • ○ Cote d'Ivoire (52)
  • ○ Croatia (53)
  • ○ Cuba (54)
  • ○ Cyprus (55)
  • ○ Czech Republic (56)
  • ○ Denmark (57)
  • ○ Djibouti (58)
  • ○ Dominica (59)
  • ○ Dominican Republic (60)
  • ○ East Timor (61)
  • ○ Ecuador (62)
  • ○ Egypt (63)
  • ○ El Salvador (64)
  • ○ Equatorial Guinea (65)
  • ○ Eritrea (66)
  • ○ Estonia (67)
  • ○ Ethiopia (68)
  • ○ Falkland Islands (Malvinas) (69)
  • ○ Fiji (70)
  • ○ Finland (71)
  • ○ France (72)
  • ○ French Guiana (73)
  • ○ French Polynesia (74)
  • ○ French Southern Territories (75)
  • ○ Gambia (76)
  • ○ Georgia (77)
  • ○ Germany (78)
  • ○ Ghana (79)
  • ○ Gibraltar (80)
  • ○ Greece (81)
  • ○ Greenland (82)
  • ○ Grenada (83)
  • ○ Guadeloupe (84)
  • ○ Guam (85)
  • ○ Guatemala (86)
  • ○ Guinea-Bissau (87)
  • ○ Haiti (88)
  • ○ Heard Island and McDonald Islands (89)
  • ○ Holy See (Vatican City State) (90)
  • ○ Honduras (91)
  • ○ Hong Kong (92)
  • ○ Hungary (93)
  • ○ Iceland (94)
  • ○ India (95)
  • ○ Indonesia (96)
  • ○ Iran (Islamic Republic of) (97)
  • ○ Iraq (98)
  • ○ Ireland (99)
  • ○ Israel (100)
  • ○ Italy (101)
  • ○ Jamaica (102)
  • ○ Japan (103)
  • ○ Jordan (104)
  • ○ Kazakhstan (105)
  • ○ Kenya (106)
  • ○ Kiribati (107)
  • ○ North Korea (108)
  • ○ Kuwait (109)
  • ○ Kyrgyzstan (110)
  • ○ Lao People's Democratic Republic (111)
  • ○ LAOS (112)
  • ○ Latvia (113)
  • ○ Lebanon (114)
  • ○ Lesotho (115)
  • ○ Liberia (116)
  • ○ Libyan Arab Jamahiriya (117)
  • ○ Lithuania (118)
  • ○ Luxembourg (119)
  • ○ Macau (120)
  • ○ Madagascar (121)
  • ○ Malawi (122)
  • ○ Malaysia (123)
  • ○ Maldives (124)
  • ○ Mali (125)
  • ○ Malta (126)
  • ○ Martinique (127)
  • ○ Mauritania (128)
  • ○ Mauritius (129)
  • ○ Mexico (130)
  • ○ Micronesia (Federated States of) (131)
  • ○ Monaco (132)
  • ○ Mongolia (133)
  • ○ Montserrat (134)
  • ○ Morocco (135)
  • ○ Mozambique (136)
  • ○ Myanmar (137)
  • ○ Namibia (138)
  • ○ Nauru (139)
  • ○ Nepal (140)
  • ○ Netherlands (141)
  • ○ Netherlands Antilles (142)
  • ○ New Caledonia (143)
  • ○ New Zealand (144)
  • ○ Nicaragua (145)
  • ○ Niger (146)
  • ○ Nigeria (147)
  • ○ Niue (148)
  • ○ Norfolk Island (149)
  • ○ Norway (150)
  • ○ Oman (151)
  • ○ Pakistan (152)
  • ○ Palau (153)
  • ○ Panama (154)
  • ○ Papua New Guinea (155)
  • ○ Paraguay (156)
  • ○ Peru (157)
  • ○ Philippines (158)
  • ○ Pitcairn (159)
  • ○ Poland (160)
  • ○ Portugal (161)
  • ○ Puerto Rico (162)
  • ○ Qatar (163)
  • ○ Reunion (164)
  • ○ Romania (165)
  • ○ RUSSIA (166)
  • ○ Russian Federation (duplicate2)
  • ○ Rwanda (168)
  • ○ Saint Helena (169)
  • ○ Saint Kitts and Nevis (170)
  • ○ Saint Lucia (171)
  • ○ Saint Vincent and the Grenadines (172)
  • ○ Samoa (173)
  • ○ Sao Tome and Principe (174)
  • ○ Saudi Arabia (175)
  • ○ Senegal (176)
  • ○ Serbia and Montenegro (177)
  • ○ Seychelles (178)
  • ○ Sierra Leone (179)
  • ○ Singapore (180)
  • ○ Slovakia (181)
  • ○ Slovenia (182)
  • ○ Solomon Islands (183)
  • ○ Somalia (184)
  • ○ South Africa (185)
  • ○ SOUTH KOREA (186)
  • ○ Spain (187)
  • ○ Sri Lanka (188)
  • ○ Sudan (189)
  • ○ Suriname (190)
  • ○ Swaziland (191)
  • ○ Sweden (192)
  • ○ Switzerland (193)
  • ○ Syrian Arab Republic (194)
  • ○ Taiwan (195)
  • ○ Tajikistan (196)
  • ○ TANZANIA (197)
  • ○ Thailand (198)
  • ○ THE NETHERLANDS (duplicate1)
  • ○ Togo (200)
  • ○ Tonga (201)
  • ○ Trinidad and Tobago (202)
  • ○ Tunisia (203)
  • ○ Turkey (204)
  • ○ Turkmenistan (205)
  • ○ Turks and Caicos Islands (206)
  • ○ Uganda (207)
  • ○ Ukraine (208)
  • ○ United Arab Emirates (209)
  • ○ United Kingdom (210)
  • ○ United States (duplicate3)
  • ○ United States Minor Outlying Islands (212)
  • ○ Uruguay (213)
  • ○ USA (211)
  • ○ Uzbekistan (215)
  • ○ Vanuatu (216)
  • ○ Venezuela (217)
  • ○ Viet Nam (218)
  • ○ Virgin Islands (219)
  • ○ VIRGIN ISLANDS (US) (220)
  • ○ Virgin Islands, British (221)
  • ○ Wallis and Futuna (222)
  • ○ Yemen (223)
  • ○ Zambia (224)
  • ○ Zimbabwe (225)
  • ○ Palestinian Territory,Occupied (226)
  • ○ Moldova, Republic of (227)
  • ○ Marshall Islands (228)
  • ○ Macedonia, The Former Yugoslav Republic of (229)
  • ○ Liechtenstein (230)
  • ○ Korea, Republic of (231)
  • ○ Guyana (232)
  • ○ Guinea (233)
  • ○ Gabon (234)
  • ○ Faroe Islands (235)
  • ○ Zanzibar (236)
  • ○ Tokelau (237)

q37 – Subject field

Which of the following best describes your field of research?

  • ○ Agriculture (1)
  • ○ Arts & Humanities (2)
  • ○ Astronomy (3)
  • ○ Biochemistry, Genetics and Molecular Biology (4)
  • ○ Biological Sciences (5)
  • ○ Chemical Engineering (6)
  • ○ Chemistry (7)
  • ○ Computer Sciences / IT (8)
  • ○ Earth & Planetary Sciences (9)
  • ○ Economics (10)
  • ○ Electrical/Electronic Engineering (11)
  • ○ Engineering & Technology (12)
  • ○ Environmental Sciences (13)
  • ○ Immunology (14)
  • ○ Microbiology (15)
  • ○ Materials Science (16)
  • ○ Mathematics (17)
  • ○ Medicine & Allied Health (18)
  • ○ Nursing (19)
  • ○ Neuroscience (20)
  • ○ Pharmacology, Toxicology and Pharmaceutics (21)
  • ○ Physics (22)
  • ○ Social Science (23)

q38 – Position

What is your position within your organisation?

  • ○ Head of Department/Senior Management (1)
  • ○ Senior Researcher/Middle Management (2)
  • ○ Researcher/Staff Member (3)
  • ○ Other (please specify) (4)____________

Ancillary