SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Secrecy in Peer Review
  6. Conclusion
  7. References

The subject of peer review has come under increased scrutiny partially due to high profile cases of fraud and other misinformation appearing in academic journals. At the same time, new publishing technologies have made it possible to experiment with new forms of peer review, while the “Open Archives Initiative” has the potential to allow authors to bypass the traditional peer review process entirely. In this paper, I examine some rationales given for the need for secrecy in peer review, especially as it relates to the perceived need for anonymity on the part of reviewers and sometimes even authors. I will propose a basic framework for secrecy and transparency as it relates to the peer review process of academic journals.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Secrecy in Peer Review
  6. Conclusion
  7. References

In December 2006, there was a debate about peer review featured prominently on the website of the journal Nature. In addition to a series of articles discussing various systems of peer review; its effect on the quality and value of research; research ethics; new technical implementations of peer review; and a series of individual perspectives and reflections on the subject; it includes an experiment in “open” peer review. Authors were given the option of posting their manuscripts for open peer review at the same time they submitted their work for the more traditional, confidential peer review process. For authors who elected to participate in the experiment in open peer review, comments on their research were solicited from qualified members of the public. Those who wished to comment could do so on the moderated forum on the Nature website. Unlike traditional peer review, anonymous comments were not allowed. The self-selected reviewers were required to give both their name and institutional affiliation. The stated purpose of this experiment was to “measure the level of participation among authors and the quality of comments received by members of the specialist community who are not the selected peer reviewers of the manuscripts concerned.” (Nature website, 2006).

Despite some initial enthusiasm, the four month experiment met with mixed results. Only five percent of authors chose to post their papers on the open server for comments, perhaps due to the fear of being “scooped”. Except for a handful of relative highly commented on papers, there was a dearth of substantive comments despite statistics that showed a high level of web traffic.

Biology Direct is an example of a journal that has gone beyond the experimentation phase, and formally implemented a system of open peer review. An online journal that began publication in January 2006, the editorial policy requires that reviewer comments, along with the author's response, are published alongside the article. The main hurdle to authors to overcome in order to qualify for publication is that three members of the editorial board must become sufficiently interested in the research to agree to act as reviewers or to solicit outside reviewers. Once this requirement is met, the policy even allows for publication of papers that have received skeptical or critical reviews, which has already occurred on at least one occasion. (Koonin et. al. 2006) Early results have found that this policy has led to the publication of solid, “business as usual” paper, along with more unconventional and hypothetical research.

These are just two examples of journals that are experimenting with new forms of peer review. During the past decade, the process of peer review has increasingly been called into question. The growth of the Internet and the Open Archives Initiative have given authors the opportunity to self-publish, and therefore potentially sidestep the traditional peer review process. There have been high profile cases of research being retracted from some of the most prominent scientific journals due to fabrications on the part of one or more authors, leading to skepticism about its role as a gate keeping mechanism. The process of peer review itself is viewed as cumbersome and slow. It increases the amount of time to publication, which is a particularly significant issue for medical and scientific developments. At the same time, the adoption of online submission systems has inflated the number of articles being submitted to journals in general and added to the workload of editors and reviewers. These and other causes have led to a general sense of disillusionment among the scholarly community about the institution of peer review. There has been evidence of an increasing openness to discussing and experimenting with new models, which has been greatly facilitated by new publishing technologies.

In this paper, I will concentrate on the role that secrecy plays in peer review. There have been increasing calls for transparency in the peer review process, up to and including publishing the reviewer's name and comments next to the research they have reviewed. There are many tensions inherent in peer review, since it performs a gate keeping function for scholarly research. Since the stakes are perceived to be so high, the question of secrecy versus transparency has emerged as a key issue. In particular, the role of the anonymous reviewer who seemingly has the power to decide what will and what will not appear in scholarly journals has increasingly been called into question.

Background

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Secrecy in Peer Review
  6. Conclusion
  7. References

Peer review is a way of establishing control over what is published in scholarly or academic journals, and of promoting the legitimacy of sanctioned research within the professions and disciplines, and among the public at large. The practice of peer review is believed to be an important means of elevating scholarly discourse. One of the main goals of peer review is to ensure that research is performed according to sound principles. It can be argued that by ensuring the reliability and validity of research, peer review has the effect of improving the quality of research in general. Therefore, peer review can be linked with modernization because it advances scientific and technological progress. Peer review is also linked to empiricism and secularism, especially in the physical, social and life sciences.

Peer review differentiates learned or academic journals from technical or popular journals. Peer review originated with medical and scientific journals, and has spread to the social sciences and humanities. The peer review process is also used for other purposes, including evaluating research grants and tenure applications for academic faculty. These uses are outside the scope of this paper.

Journal editors have exercised some form of editorial control since The Philosophical Transactions of the Royal Society began publishing research in the biological sciences in 1665. This has evolved into a peer review system that is well defined, though many variations have emerged. (Weller 2001). The role of the editor or editorial board, especially, varies in the peer review process. The editor may accept or reject a manuscript without review (for example, if the subject matter is outside the scope of the publication), or may elect to send the manuscript to the reviewers. The reviewer (sometimes called the referee) may advise the editor to accept the manuscript as is or to accept it with revisions. If dissatisfied with the manuscript, or if it outside the scope of the journal, the reviewer may advise that it be rejected. In some cases, the author may be encouraged to resubmit a rejected manuscript once suggested improvements are made.

Once the review process is complete, the editor often has the prerogative to accept or reject the manuscript, even if may mean overriding the reviewers. Some journals also require that the editor grade the reviewer on the quality and usefulness of their comments. Reviewer comments may be transmitted verbatim to the author, or they may be transmitted to the author only after are summarized by the summarized by the editor. In some instances, the author may receive only the final verdict. Traditionally, reviewer comments have not been seen by anyone other than the editor, the author, and perhaps the other reviewers.

Secrecy is often a component of peer review. The editorial process may be implemented in such as way that the identities of those involved are shielded. However, secrecy is not an absolute requirement. Some journals, such as the Harvard Business Review, have an editorial policy where the identities of reviewer and authors are known. There are medical journals that have also adopted an editorial policy where the reviewers are known to the authors. Yet, many journals continue to have a policy of anonymous peer review, where the reviewer is not formally known to the author, and sometimes not to other reviewers. In blind (or double-blind) peer review, neither the author or reviewer are supposed to be informed of each other's identity. The diagram below describes various levels of transparency that may be used in peer review. Some of the variables include whether the author identity is known to the reviewer; whether the reviewer identity is known to the author; how reviewers are selected; who makes the final decision regarding publication; and finally, who sees the reviewer's comments -the editor, the author, or even the public at large.1

Table 1. Levels of Transparency in Peer Review
 Author IdentityReviewer IdentityWho Selects the Reviewers?Who Decides on Publication?Who sees comments?
AnonymousKnownNot knownEditorial boardEditor or ReviewersEditor, sometimes author
BlindNot knownNot knownEditorial boardEditor or ReviewersEditor, sometimes author
OpenKnownKnownEditorial boardEditor or ReviewersEditor, author, sometimes the public

Peer review policies may evolve over time in a single journal. Sometimes, a change in policy may coincide with a new editorial team. For example, by examining the front and back matter of a peer reviewed historical journal called Church History, it appears that blind peer review was adopted as late as 1998, at which time prospective authors were asked to omit their names and any identifying material from their submissions. The change in policy shortly followed a move the journal made from the Divinity School of the University of Chicago to the Divinity School of Duke University. It also coincided with a call for greater diversity of research methods and subject matter, including research into the history of the church outside the West. Perhaps the editors anticipated that by implementing blind review, there was a possibility that new authors would be selected to appear in the journal's pages?

Among its defenders, peer review is seen as more important than ever, especially from the encroachment of popular and non-academic websites that disseminate information on scientific and health-related breakthroughs, which is traditionally their terrain. For example, in 2002, a U.K.-based charitable trust called “Sense About Science” urged researchers to explicitly explain the value of peer review when presenting their work in public forums, such as interviews and radio talk shows. One study found that 75 percent of the public did not know what constitutes peer review. (Butler 2004) One misconception of peer review is that it acts as a sort of academic jury where experts meet to weigh the evidence and vote on the findings. (Drake 2006)

If the image that peer review has in the public imagination is one of a jury, then recent high profile cases of mistakes, fraudulent research, or hoaxes appearing in peer reviewed journals have led to increasing questions about the jury's competency. To cite one high profile example, in December 2005, Woo Suk Hwang, a Korean researcher, retracted his paper concerning the cloning of embryonic stem cells that was originally published in the June 17, 2005 issue of the peer-reviewed journal, Science. The chain of events that led to the revelation that evidence had been falsified started “in early December on a Korean Web site, [when] an anonymous writer, who claims to be a life scientist, pointed out duplications in some of the photographs of ES [Embryonic Stem] cells published in the 2005 paper.” (Normille 2005) The paper is still available on the Science website though it is marked “retracted”. Another study that has generated considerable controversy is one that seemed to prove the effectiveness of intercessory prayer during pregnancy, published in the Journal of Reproductive Medicine. (Kwang et. al. 2001) Incidents such as these have led to questions regarding peer review's effectiveness as a gate keeping mechanism, though some of its' defenders maintain that its' purpose has never been to weed out deliberate fraud.

Secrecy in Peer Review

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Secrecy in Peer Review
  6. Conclusion
  7. References

Anonymity is associated with both positive and negative implications for the individual, as well as for the larger society. Some of the beneficial effects on the individual may include encouraging candor, honesty, impartiality, and even a sense of idealism. On a societal level, anonymity may enhance the free flow of information (especially in closed or tyrannical situations) and promote freedom of expression. (Weicher 2006) The negative consequences of anonymity may include a lack of accountability, credibility, and transparency. This can lead to the nondisclosure of conflicting interests and an overall decrease in the likeliness to correct mistakes or abuses of power. (Brin 1998) In this paper, I will explore the role that secrecy and anonymity play in the peer review process, and how it has been challenged (and defended) during the past decade.

There are many stated and unstated beliefs and assumptions surrounding the practice of peer review. One assumption is that reviewers are supposed to be motivated by a sense of service to the profession or discipline of which they are a member. For example, an opinion piece from the new editorial team at The Archives of Internal Medicine cited peer review as one of the “pillars” that allows the medical profession to regulate itself and to serve the public good, or at the very least “first, do no harm.” (Emanuel 2005) According to the editors, peer review depends upon reviewer impartiality, adherence to accepted standards of scientific merit, and a commitment to altruism.

Even though reviewers may be motivated by altruism, they are often asked to expend a significant amount of effort for few tangible returns. Reviewers usually do not receive financial compensation. In addition, they may fail to receive any type of public credit or recognition for their work. The editors of The Archives of Internal Medicine attempted to combat this disincentive by publicly recognizing top reviewers, and perhaps most significantly, providing letters of acknowledgement designed to be used for career advancement. Similarly, BMJ (formerly the British Medical Journal) rewards its reviewers with token payments, free subscriptions, annual parties, and certificates. Still, a recent editorial found this to be insufficient recognition for the work of peer reviewers, and encouraged the development of a more formal and substantial system of incentives. (Groves 2006)

Public recognition of reviewers is possible, in part, because BMJ, along with other medical journals, has already adopted s a system of open peer review. For BMJ, this means that the reviewer is known to the author. If accepted, the reviewer's name is also published alongside the article. (Bloom 2006) The BMJ transparency policy also requires that details of funding be revealed, as well as any relationships among the authors or researchers and the funders.

Many journals outside of medicine still practice anonymous or blind reviewer. Though ideally, reviewers are motivated by a sense of service and altruism, there have been repeated questions raised concerning reviewer bias, reviewer negligence, and reviewer favoritism and self-interest. (Campanario 1998a) Anonymity intensifies these concerns because it is viewed as a means by which individuals can hide or escape the consequences of their actions. For example, in a recent “laundry list” of complaints about peer review, the author cites “insulting, unhelpful or unqualified reviewers”, as well as “reviewers who express prejudices, have conflicts of interest or have not read the paper carefully or thoroughly enough.” (Sieber 2006)

This type of complaint has been echoed in many other forums. In an article in the Guardian Education Pages, the author claims that “anonymity has provided a mask behind which petty jealousies, envy, spitefulness, rivalry and intellectual sectarianism has flourished.” (Back 2000) It has allowed the development of a culture that lacks an “ethics of measured critique” where criticism can be delivered in a constructive and tactful manner. Instead, adopting the role of a tough critic has become a measure of intellectual “authenticity.” A related concern is that since reviewers are, by necessity, judging the research of others in the same field of expertise, they are also competing with them for scarce research dollars. There is the sense among some that a self-perpetuating cycle of unethical behavior has led to a breakdown in the system. (Goodstein 1996)

Among its' defenders, it is believed that blind peer review has the potential to raise the level of research by encouraging impartiality on the part of reviewers, since they do not know the identity of the authors (though in rarified research circles, it is sometimes possible to guess correctly.) A 1997 study by researchers from the British Medical Journal tried to find out if there was evidence for this belief. It found there was no significant difference between the reviewers of signed and unsigned research in terms of review quality, whether an article was recommended for publication, or the turnaround time. (Goldbeck-Wood 1997) It is not clear whether any of the authors were actually known to the reviewers “in real life” and if so, what type of difference this may have made. However, the results seem to suggest that whether they know the author's identity or not has little effect on how reviewers conduct their work, at least for the medical journal being studied.

On the other hand, shielding the identity of reviewers from authors is believed to result in increased candor, and therefore higher quality reviews. Another study from the British Medical Journal in 1999 found that reviewers were less likely to accept assignments if they know their names would be revealed to those being reviewed (Van Rooyen et. al 1999). Probably not surprisingly, it also was found that authors prefer to know the identities of those who are commenting on and judging their research. In general, the study found little difference in the quality of the reviews of identified versus anonymous reviewers. The article advances the case for open peer review in the interest of “increased accountability, fairness, and transparency.”

Conclusion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Secrecy in Peer Review
  6. Conclusion
  7. References

During the last decade, technological changes have contributed to the debate surrounding peer review. Technology alone usually does not cause a cultural shift, but it can magnify existing tensions. Major conflicts that surround peer review are the need for secrecy versus the desire for transparency, as well as legitimacy and gate keeping concerns.

Secrecy has been viewed as a prerequisite for candor and disinterest on the part of reviewers. In addition, secrecy is employed in blind peer review in an attempt to reduce reviewer bias. The underlying belief is that when the identity of the author is hidden, the reviewer is more likely to judge each work on its' own merits, with impartiality and candor. The need for reviewer secrecy, especially, has been called into question during the last decade. This questioning is at least partially due to unhappiness with the quality and content of reviewer comments. Other reasons for dissatisfaction have been the sense that anonymous peer review is used to enforce “disciplinary orthodoxy” (Shwartzmann 1997), advance political agendas (Friedman and Elihu 2005, Gropp 2005), hide conflicts of interest, and discourage risk-taking in general. There is even research that suggests that chance plays a larger role in e whether articles are accepted than might have been thought. (Neff and Olden 2006)

The Internet plays a role in the debate over peer review in more than one way. First, it enables editors to experiment with peer review and reach out for new sources of peer comments. For example, the ability to post a manuscript on a moderated forum and invite public comments, as Nature magazine has done, would have not been possible before the widespread use of the Internet. At the same time, online submission systems have inflated the number of papers that are potentially the subject of peer review (and increasing the number of rejections), thereby further stressing the system.

In addition, the culture of the Internet encourages the transparency of information, (though somewhat paradoxically individual privacy is highly valued.) Peer review's traditional role of gate keeping can be viewed as running counter to the notion of the Internet as a marketplace of ideas or an intellectual proving ground. The idea has been advanced that by posting research on the Internet, its' shortcomings or flaws will be exposed by the many pairs of eyes reviewing the work. This could conceivably reduce or eliminate the need for formal peer review. This idea is similar in concept to the Open Software Initiative, where by making software programs public, they become better. According to the OSI website:

The basic idea behind open source is very simple: When programmers can read, redistribute, and modify the source code for a piece of software, the software evolves. People improve it, people adapt it, people fix bugs. And this can happen at a speed that, if one is used to the slow pace of conventional software development, seems astonishing. (Open Source website 2006)

Ideas like this can be viewed as a challenge to the authority of those who have a stake in the peer review process. When authors chaff at the idea of two or three anonymous reviewers judging their work, the Internet provides them with other options. This includes self-publishing through the Open Archives Initiative or other venues; or re-submitting their manuscripts (made relatively easy with the aid of online submission software) until it is accepted somewhere else. Without the growth of the Internet, the underlying conflicts surrounding peer review might not have become as prominent over the past decade. It is apparent that this is a highly sensitive and complex issue for those involved in research and scholarship, especially the question of secrecy as manifested in anonymous peer review. The debate seems particularly heightened in the sciences, perhaps due to a sense that the stakes are higher as researchers compete for grants and other sources of funds from government and business. If history is a guide, social sciences and humanities journals may follow the lead of the sciences. In any case, it seems likely that at least some journals will adopt policies that enable increased transparency for authors, reviewers, scholarly peers, and the readership at large.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Background
  5. Secrecy in Peer Review
  6. Conclusion
  7. References
  • Nature Peer review: Frequently asked questions (FAQ). in Nature [database online]. 2006. Available from http://www.nature.com.jerome.stjohns.edu:81/nature/peerreview/faq.html.
  • On the web: Open peer-review debate. 2006. Nature 441, (7097) (06/29/): xiiixiii.
  • Peer review on trial. 2006. Nature 441, (7094) (Jun 8): 668.
  • Quantified: Peer review. 2006. Nature 441, (7094) (06/08/): xiiixiii.
  • Three cheers for peers. 2006. Nature 439, (7073) (Jan 12): 118.
  • The value(s) of editorial peer review. 2006. On the Horizon 14, (3): 121129.
  • Flaws in a study of peer review. 2004. The Chronicle of Higher Education 51, (16) (Dec 10): A.39.
  • Science and technology: Publish and perish; theoretical physics. 2002. The Economist 365, (8299) (Nov 16): 104.
  • Methodical progress. 1997. The Economist 344, (8036) (Sep 27): 89.
  • Top scientists reveals student's fraud … raising doubts about efficacy of peer review. 1997. The CQ Researcher 7, (1) (Jan 10): 14.
  • Albanese, Mark. 2006. Three blind mice—might make good reviewers. Medical Education 40, (9) (09//): 828830.
  • Algase, Donna L. 2006. Reviewing peer review. Research and Theory for Nursing Practice 20, (3) (Fall): 179.
  • Alkass, Sabah, Mark Mazerolle., and Frank Harris. 1998. Rigour in research and peer-review: A reply. Construction Management and Economics 16, (2) (Mar): 139.
  • Altman, Lawrence K. 2006. For science's gatekeepers, a credibility gap. New York Times 155, (53567) (05/02/): F1F7.
  • Armstrong, J. Scott. 1996. Publication of research on controversial topics: The early acceptance procedure. International Journal of Forecasting 12, (2) (Jun): 299.
  • Back, Les. 2000. The devil you know: Academic reviewers should not be anonymous, says les back. The Guardian, Jul 18, 2000.
  • Bloom, Theodora. 2006. Online frontiers of the future of peer review. Nature.
  • Brin, David. 1998. The transparent society: Will technology force us to choose between privacy and freedom? New York: Basic Books.
  • Brysbaert, Marc. 1996. Improving the journal review process and the risk of making the poor poorer. The American Psychologist 51, (11) (Nov): 1193.
  • Butler, Declan. 2004. Academics seek to cast peer review as a public service. Nature 430, (6995) (Jul 1): 7.
  • Campanario, Juan Miguel. 1998. Peer review for journals as it stands Today—Part 1. Science Communication 19, (3) (March): 181211.
  • Campanario, Juan Miguel. 1998a. Peer review for journals as it stands today–part 2. Science Communication 19, (4) (Jun): 277.
  • Chapman, Peter M. 2006. When is peer review excessive? Examples of peer review hell. Vol. 12 Taylor & Francis Ltd.
  • Couzin, Jennifer. 2006. … and how the problems eluded peer reviewers and editors. Science 311, (5757) (Jan 6): 23.
  • DeAngelis, Catherine D., and Phil B. Fontanarosa. 2006. Thank you, JAMA peer reviewers and authors. JAMA: The Journal of the American Medical Association 295, (10) (Mar 8): 1171.
  • Emanuel, Linda L., and Philip Greenland. 2005. Peer review and professionalism at the archives of internal medicine. Archives of Internal Medicine 165, (22) (12/12/): 25592560.
  • Enserink, Martin. 2001. Peer review and quality: A dubious connection? Science 293, (5538) (Sep 21): 2187.
  • Estrada, Carlos, Adina Kalet, Wally Smith, and Marshall H. Chin. 2006. How to be an outstanding reviewer for the journal of general internal medicine and other journals. JGIM: Journal of General Internal Medicine 21, (3) : 281284.
  • Eugene Koonin, Laura Landweber., David Lipman and Ros Dignon. 2006. Reviving a culture of scientific debate. can ‘open peer review’ work for biologists? Biology direct is hopeful.
  • Feder, Toni. 1999. As OSTP readies conduct guidelines, physicists begin to think more about research ethics. Physics Today 52, (4) (Apr): 62.
  • Fenn, Peter. 1997. Rigour in research and peer review. Construction Management & Economics 15, (4) (07//): 383385.
  • Fine, Mark A. 1996. Reflections on enhancing accountability in the peer review process. The American Psychologist 51, (11) (Nov): 1190.
  • Fontanarosa, Phil B., Richard M. Glass, and Catherine D. DeAngelis. 2000. Thanking authors, peer reviewers, and readers—constancy in a time of change. JAMA: The Journal of the American Medical Association 283, (15) (Apr 19): 2016.
  • Friedman, Lee, and Elihu D. Richter. 2005. Conflicts of interest and scientific integrity. International Journal of Occupational and Environmental Health 11, (2) (Apr-Jun): 205.
  • Giles, Jim. 2006. Journals submit to scrutiny of their peer-review process. Nature 439, (7074) (01/19/): 252252.
  • Goldbeck-Wood, Sandra. 1997. Blinding reviewers to authors' identity does not improve quality. British Medical Journal 315, (7111) (Sep 27): 766.
  • Goldsmith, Lowell A., Elizabeth N. Blalock, Heather Bobkova, and Russell P. Hall III. 2006. Picking your peers. Vol. 126 Nature Publishing Group.
  • Goodstein, David. 1996. ‘Every scientist I know has stories of being treated unfairly by anonymous referees.’. The Chronicle of Higher Education 42, (47) (Aug 2): B5.
  • Griffin-Sobel, Joyce P. 2005. The status of peer review. Vol. 9 Oncology Nursing Society.
  • Grivell, Les. 2006. Through a glass darkly: The present and the future of editorial peer review. EMBO Reports 7, (6) (Jun): 567.
  • Gropp, Robert E. 2005. Politics and peer review. Bioscience 55, (9) (Sep): 734.
  • Groves, Tricia. 2006. Quality and value: How can we get the best out of peer review? Nature.
  • Guterman, Lila. 2004. Peer-reviewed paper on pregnancy and prayer may soon be retracted. The Chronicle of Higher Education 50, (41) (Jun 18): A. 16.
  • Hampton, Tracy. 2005. Biomedical journals probe peer review. JAMA: Journal of the American Medical Association 294, (18) (11/09/): 22872288.
  • Hartley, James. 2006. Reading and writing book reviews across the disciplines. Journal of the American Society for Information Science and Technology 57, (9) (Jul): 1194.
  • Henderson, Albert. 2001. Undermining peer review. Society 38, (2) (Jan/Feb): 47.
  • Hurst, J. R., E. C. Howard, and J. A. Wedzicha. 2005. Reviewer selection: Author or editor knows best? Vol. 60 BMJ Publishing Group.
  • Jefferson, Tom. 2005. Peer review and publishing: It's time to move the agenda on. The Lancet 366, (9482) (Jul 23-Jul 29): 283.
  • Kennefick, Daniel. 2005. Einstein versus the physical review. Physics Today 58, (9)(Sep): 43.
  • Kumashiro, Kevin K. 2005. Thinking collaboratively about the peer-review process for journal-article publication. Harvard Educational Review 75, (3) (Fall): 257.
  • Kwang Y. Cha, M. D., Daniel P. Wirth, J. D., Wirth M. S., and Rogerio A. Lobo, M. D. 2001. Does Prayer Influence the Success of in Vitro Fertilization-Embryo Transfer? Report of a Masked, Randomized Trial. Journal of Reproductive Medicine. 46, (9).
  • Lehmann, Donald R. 2003. Finding important findings. Journal of Business Research 56, (1) (Jan): 89.
  • Levenson, Richard L., Jr. 1996. Enhance the journals, not the review process. The American Psychologist 51, (11) (Nov): 1191.
  • Levine, Joseph M. 2005. Intellectual history as history. Journal of the History of Ideas 66, (2) (Apr): 189.
  • Marris, Emma. 2006. Journal lays bare remarks from peer reviewers. Nature 439, (7 2710 077) (Feb 9): 642.
  • Marris, Emma. 2006. Should journals police scientific fraud? Nature 439, (7076) (Feb 2): 520.
  • Merriman, Joyce A. 2006. Peer review ensures continued AFP quality. American Family Physician 73, (5) (Mar 1): 751.
  • Miller, C. Chet. 2006. Peer review in the organizational and management sciences: Prevalence and effects of reviewer hostility, bias, and dissensus. Academy of Management Journal 49, (3) (Jun): 425.
  • Monastersky, Richard. 2002. The emperor's new science: French TV stars rock the world of theoretical physics. The Chronicle of Higher Education 49, (12) (Nov 15): A.16.
  • Morrison, Jill. 2006. The case for open peer review. Medical Education 40, (9) (09//): 830831.
  • Neff, Bryan D., and Julian D. Olden. 2006. Is peer review a game of chance? Bioscience 56, (4) (Apr): 333.
  • Noble, David W., Eugene Garfield., Marie McVeigh, and Marion Muff. 2006. Preventing scientific Fraud/IN RESPONSE. Annals of Internal Medicine 145, (6) (Sep 19): 472.
  • Normile, Dennis, Gretchen Vogel., and Constance Holden. 2005. STEM CELLS: Cloning researcher says work is flawed but claims results stand. Science 310, (5756) (December 23): 18861887.
  • Paustenbach, Dennis J., David S. Egilman, and Susanna Rankin Bohme. 2006. Scientific method Questioned/In reply. International Journal of Occupational and Environmental Health 12, (3) (Jul-Sep): 290.
  • Peek, Robin. 2003. Could peer review be wrong? Information Today 20, (4) (Apr): 17.
  • . Ready, Tinker. 2006. Plagiarize or perish? Nature Medicine 12, (5) (May): 494.
  • Regehr, Glenn, and Georges Bordage. 2006. To blind or not to blind? what authors and reviewers prefer. Medical Education 40, (9): 832839.
  • Ross, Joseph S., Cary P. Gross, Mayur M. Desai, and Yuling Hong. 2006. Effect of blinded peer review on abstract acceptance. JAMA: The Journal of the American Medical Association 295, (14) (Apr 12): 1675,
  • Rossiter, John R. 2003. Qualifying the importance of findings. Journal of Business Research 56, (1) (Jan): 85.
  • . Runeson, Goran, and Martin Loosemore. 1999. Gate-keepers or judges: Peer reviews in construction management. Construction Management and Economics 17, (4) (Jul): 529.
  • Scarpa, Toni. 2006. Peer review at NIH. Science 311, (5757) (Jan 6): 41.
  • Schwartzman, Roy. 1997. The forum: Peer review as the enforcement of disciplinary orthodoxy. The Southern Communication Journal 63, (1) (Fall): 69.
  • Sieber, Joan E. 2006. How can we research peer review? Nature.
  • Smit, Christian. 2006. Peer review: Time for a change? Bioscience 56, (9) (09//): 712713
  • Smith, Richard. 1997. Peer review: Reform or revolution? British Medical Journal 315, (7111) (Sep 27): 759.
  • Stout, David E., James E. Rebele, and Thomas P. Howard. 2006. Reasons research papers are rejected at accounting education journals. Issues in Accounting Education 21, (2)(May): 81.
  • Susan van Rooyen, Fiona Godlee., Stephen Evans, Nick Black, Richard Smith. 1999. Effect of open peer review on quality of reviews and on reviewers' recommendations: A randomised trial. BMJ 318: 2327.
  • Teaf, Christopher M., and Barry L. Johnson. 2006. Fraud and deception in publication of scientific research: Is there a solution? HERA's policy change. Human and Ecological Risk Assessment 12, (4) (Aug): 623.
  • Vastag, Brian. 2006. Cancer fraud case stuns research community, prompts reflection on peer review process. Journal of the National Cancer Institute 98, (6) (Mar 15): 374.
  • Ware 3b9, Mark. 2005. Online submission and peer-review systems. Learned Publishing 18, (4) (October): 245250.
  • Weicher, Maureen. 2006. [Name withheld]: Anonymity and its implications. Paper presented at Proceedings of the American Society for Information Science & Technology, Austin, TX.
  • Yiotis, Kristin. 2005. The open access initiative: A new paradigm for scholarly communications. Information Technology & Libraries 24, (4) (120): 157162.
  • Youngner, Julius S. 1998. The scientific misconduct process: A scientist's view from the inside. JAMA: The Journal of the American Medical Association 279, (1) (January 7): 6264.