Overburdening of peer reviewers: A multi-stakeholder perspective on causes and effects

Peer review of manuscripts is labour-intensive and time-consuming. Individual reviewers might feel themselves overburdened with the amount of reviewing they are requested to do. Aiming to explore how stakeholder groups perceive reviewing burden and what they believe to be the causes of a potential overburdening of reviewers, we conducted focus groups with early-, mid-, and senior career scholars, editors, and publishers. By means of a thematic analysis, we aimed to identify the causes of overburdening of reviewers. First, we show that, across disciplines and roles, stakeholders believed that the reviewing burden is distributed unequally across members of the academic community, resulting in the overburdening of small groups of reviewers. Second, stakeholders believed this to be caused by (i) an increase in manuscript submissions; (ii) inef ﬁ cient manuscript handling; (iii) lack of institutionalization of peer review; (iv) lack of reviewing instructions and (v) inadequate reviewer recruiting strategies. These themes were assumed to relate to an inadequate incentive structure in academia that favours publications over peer review. In order to alleviate reviewing burden, a holistic approach is required that addresses both the increased demand for and the insuf ﬁ cient supply of reviewing resources.

. Being overwhelmed with reviewing workload, reviewers might decline to review more often and editors might face difficulties in recruiting knowledgeable reviewers. Also, reviewers might review in a haste and fail to detect errors in manuscripts (Elsevier & Sense about Science, 2019;Nicholas et al., 2015). Innovations aimed at alleviating reviewing burden are currently being tested. One strand of innovation includes artificial intelligence and machine learning for increasing efficiencies in the manuscript handling process and for alleviating reviewing workload. This includes aiding manuscript handling, such as automatically identifying reviewers based on manuscript contents, assessing performance or conflicts of interest of reviewers and checking whether references and the manuscript structure meet journal policies (BioMed Central and Digital Science, 2017;Frontiers, 2018). It has also been suggested that automation could aid or replace review processes as such. This comprises plagiarism checks, identifying fraudulent behaviour and using language processing to extract key findings of a manuscript and placing these in context with existing work (BioMed Central and Digital Science, 2017;Frontiers, 2018;Heaven, 2018). A further strand of innovation is described as 'open peer review', which encompasses different ways in which peer review can be opened (Ross-Hellauer, 2017). First, this can include inviting the public to contribute or sharing review reports amongst reviewers to facilitate co-reviewing, thereby potentially distributing workload more equally and leveraging synergies. Second, this can include revealing author and reviewer identities and publishing review reports and author responses, in order to recognize reviews as scholarly outputs and attribute these to their authors (Ford, 2013;Ross-Hellauer et al., 2017). Incentivizing peer review to motivate more scholars to agree to review describes another innovation that aims at alleviating reviewing burden. Incentives can be material in the form of reviewer payments, discounts on book purchases or fee waivers for publications. Incentives can also be non-material, including giving credit for and displaying reviewing activities (Ravindran, 2016).
The acceptance of these innovations within the academic community determines their success. Depending on how they perceive the reviewing burden and what they believe to be the underlying mechanisms that cause a potential overburdening of reviewers, stakeholders, including authors, reviewers, editors and publishers, might vary in their acceptance of potential solutions.
This currently presents a knowledge gap as no research qualitatively explores and compares stakeholders' perceptions of reviewing burden. There are mathematical models that estimate the overall sustainability of the peer review system (Kovanis et al., 2016) and standardized surveys that gauge overburdening of reviewers as one reason why scholars decline to review (Brannon et al., 2016;Djupe, 2015;Mulligan et al., 2013;Publons, 2018;Tite & Schroter, 2007;Ware, 2008aWare, , 2008b. Publishers and editors usually are not represented in these surveys and junior scholars are mostly under-represented with few notable exceptions (eLife, 2018;Jamali et al., 2020;Rodríguez-Bravo et al., 2017). Qualitative analyses of stakeholder perceptions are another strand of research (Glonti et al., 2019;Harley et al., 2010;Severin & Chataway, 2020;University of Tennessee & CIBER Research Ltd., 2013). This research is not focused on reviewing burden, but indicates that stakeholders generally differ in their perceptions of peer review challenges and solutions. For example, Zaharie and Osoian (2016) explored the potential of incentives for improving peer review engagement and found that senior scholars believed reviewing to be a reciprocal duty and hence did not expect to receive any rewards, whilst junior scholars perceived reviewing as a means of career advancement and appreciated being mentioned on the journal website (Zaharie & Osoian, 2016). In addition, few surveys gauge stakeholder attitudes on innovations aimed at alleviating reviewing burden (Besançon et al., 2020;Publons, 2018;Ross-Hellauer et al., 2017;Tite & Schroter, 2007;Warne, 2016).
To our knowledge, no study comprehensively investigates how stakeholders perceive the burden that is placed on reviewers and the causes of a potential overburdening. This is important as considerable resources are dedicated to innovative review methods aimed at alleviating reviewing burden (Birgit & Edit, 2017). We conducted focus groups with stakeholders involved in academic publishing, including junior-to senior-career scholars, reviewers, editors and publishers, to explore what their expectations towards peer review were and what they believed to be important challenges in the peer review system. A range of different challenges was discussed. Because overburdening emerged as a theme of central importance in stakeholder responses, this paper focuses on reviewing burden as a challenge in peer review.

METHODS
By means of focus groups, we explored how stakeholders involved in peer review perceive current challenges in peer review and how these could be addressed, with a particular focus on the burden that is placed on reviewers. Methods are described

Key points
• All stakeholder groups agreed that unequal distribution of the reviewing workload is leading to a small group of reviewers feeling overwhelmed.
• Editors and scholars agree that the burden on reviewers is increased by both lack of good instructions and inadequate recruiting strategies.
• Lack of instruction may lead to an overestimation of the workload required from reviewers and disincentive them from accepting invitations to review.
• Inefficient manuscript handling is blamed for increasing the review burden, particularly sending out articles that should have been rejected immediately.
• Both early and senior career scholars agree that reviewers are overwhelmed with low-quality manuscripts. below. A detailed account is published elsewhere (Severin & Chataway, 2020).

Sampling
By means of maximum variation sampling (Breen, 2006), we selected participants who covered early-to senior scholars, reviewers, editors and publishers, and all academic disciplines, including social sciences, humanities, natural sciences and life sciences. We recruited participants from different sources, including university staff websites, journal editorial board websites, professional network websites and academic social networks. We contacted participants via email and provided them with a consent form and information sheet containing information about study aims, procedures, rules of conduct and confidentiality measures.

Data collection
We held focus groups workshops in March to June 2019. Where possible, we structured workshops by stakeholder group to allow for similarity in participants' experiences. Workshops were 2-hour long and involved 3 to 7 participants each. Before each workshop, we restated study aims, procedures as well as confidentiality measures and obtained informed consent. We moderated discussions by means of a semi-structured topic guide, which was based on a review of the literature and further refined following a pilot workshop. We asked participants to discuss what they expected from peer review and what they believed to be important challenges in the peer review system. We audiorecorded all discussions, imported audio files to NVivo 12 and transcribed these. An assistant took notes (Severin & Chataway, 2020).

Data analysis
We analysed transcripts thematically by exploring patterns and themes in relation to reviewing burden. Following an approach published elsewhere (Severin & Chataway, 2020), this was done in two steps. The first step included developing a preliminary codebook, which was driven by our research questions (A. S. and J. C.). In a second step, A. S. read and reread the transcripts and coded their topics. A. S. coded topics already entailed in the codebook whilst allowing new topics to emerge. A. S. repeated the coding until saturation across reviewing burden and potential causes was reached, defined as the point where no additional information was forthcoming from coding (Ando et al., 2014;Severin & Chataway, 2020). A. S. updated and revised the codebook continuously. Where codes emerged in a repeated pattern, they became a theme.

RESULTS
A total of 37 participants were recruited for seven workshops (Table 2). This included five early-career researchers, four midcareer researchers, 17 senior researchers (of which 13 researchers also held an editorial position), eight publishers and three editors who did not hold a position at a research institution. The groups of senior career scholars and editors are in large part overlapping as most recruited senior career scholars held an editorial position and because most editors held an academic position. Because both groups showed no differences in their perceptions, we refer to them as one stakeholder group (Severin & Chataway, 2020).
When stakeholders were asked what they perceived to be important challenges in the peer review system, they agreed that the volume of manuscripts that need to be reviewed has increased by a rate that threatens the academic community's ability to supply the reviewing resources necessary to address its own demand for peer review.
As the volume of submissions that potentially will need to be reviewed is increasing, the time allocated to each individual reviewer is increasing. This strains the community.

Stakeholder group
Early-career scholars (n = 5), mid-career scholars  Across disciplines and roles, stakeholders believed that this imbalance between the demand for and the supply of reviewing resources potentially results in an overburdening of reviewers ( Fig. 1). When asked to elaborate, publishers, editors and senior scholars explained that the reviewing workload would actually be distributed unequally across members of the academic community, with small groups of reviewers carrying a disproportionate share of the overall workload, causing them to be overwhelmed and feeling overworked.

Effects
The overburdening of reviewers was perceived to be problematic for a number of reasons. To publishers, editors and scholars with editorial positions, the overburdening of reviewers became most visible in difficulties in recruiting reviewers for their journals. In contrast, junior scholars did not mention this theme as an important consequence of the overburdening of reviewers.
Junior-to senior-career scholars and editors feared that reviewers occasionally might conduct superficial quality control.
They assumed that, given their overwhelming workload, reviewers might not always have the time to read manuscripts thoroughly and might therefore fail to detect crucial errors in manuscripts.
The thing is that [scholars review] under huge time pressures because they have a full time job doing something else. They are squeezing this in to their full time job […] and therefore the temptation is that they will do a light quick easy job reviewing manuscripts.
[…] If they had more time, they would do a good job. I am sure they would, but very often, they do not have the time, so they will cut corners from time to time. (Senior lecturer, computer sciences) Junior-to senior scholars and editors perceived the overburdening of reviewers to cause delays in the review process as reviewers might not be able to stick to agreed timelines for submitting reviewers.
My main point is that reviewers are on time. If they agree on a timeframe then they should comply with this timeframe. If this is three weeks or a month, or two months, if they comply with this timeframe, I am satisfied as an editor. Problematic are reviewers that are not corresponding to the time frame. (Professor and editor-in-chief, political science) Whilst junior scholars were concerned that delayed reviewers would slow down their research, mid-and senior scholars showed some understanding for delayed reviewers.
Reflecting upon their own workload, they explained that delays in submitting reviews were natural as scholars might not always be able to balance heavy reviewing workloads with existing academic responsibilities.
In contrast to scholars and editors, publishers did not mention superficial quality control or delays in the review process as important consequences of the overburdening of reviewers.

Causes
Different themes were deduced in our analyses that stakeholders believed to be causes of the overburdening of reviewers. These related to five key themes: (i) an increase in manuscript submissions; (ii) inefficient manuscript handling; (iii) lack of institutionalization of peer review; (iv) lack of reviewing instructions and FIGURE 1 Word cloud of keywords. Keywords discussed in relation to overburdening of reviewers, represented by frequency (larger text indicates greater frequency) using NVivo 12.
(v) inadequate reviewer recruiting strategies. These themes were assumed to have an impact either upon the demand for or on the supply of reviewing resources, causing an unequal distribution of reviewing workload and the overburdening of small groups of scholars. As shown in Fig. 2, these themes were also believed to be partly interrelated and to mutually reinforce each other.

Increase in manuscript submissions
Across disciplines and roles, stakeholders believed that growing numbers of manuscripts submitted for publication were one of the main drivers of the overburdening of reviewers. Whilst publishers did not speculate about the causes of the increase in manuscript submissions, scholars and editors presented a number of factors they believed to be responsible. Across disciplines, these stakeholders felt that inadequate incentive structures were one of the main drivers for the increase in manuscript submissions. It was believed that, as the criteria for academic hiring and promotion as well as for research funding allocate on increasingly focus on scientific publications, scholars are inclined to invest their time in publishing their own research rather than in reviewing the work of their peers.
Reviewing relies on goodwill. Frankly, I am surprised that it works even to the extent that it does now. Because the selfish narrowing incentives are so far into the other direction, to publishing instead of reviewing.
[…] So it is sad that the incentive structure is not quite well aligned with peer review. (Professor and editorial board member, mathematics) Particularly stakeholders from the social sciences and the humanities expressed concerns about the 'publish-or-perish' culture of academia where scholars have to publish work to advance their career.

Inefficient manuscript handling
Across disciplines, stakeholders identified inefficient manuscript handling processes as a further cause of the overburdening of reviewers. First, this was related to inadequate editorial triage.
Given the increase of manuscript submissions, stakeholders expected editorial review to filter out manuscripts that are unlikely to survive the review process. By checking formality requirements, running plagiarism checks and evaluating if manuscripts meet minimum quality standards and the scope of a journal, the editorial team should decide whether manuscripts are forwarded into peer review or outright rejected. Stakeholders believed that too many unsuitable manuscripts are sent to review and concluded that editorial triage currently fails to regulate demand for peer review. Reflecting on their roles as reviewers, junior-to senior-career scholars related this to the quality of manuscripts. They argued that peer review is overwhelmed with low-quality manuscripts.
Personally, I am surprised with the low quality of manuscripts that are sent to peer review. More papers should be outright rejected. (Professor and editor, literature studies) Scholars stated that this would mean avoidable work for them. Perceiving it an annoyance to review manuscripts of poor quality, some scholars stated that they started declining to review more often. Publishers, who expected editorial triage to assess the suitability of manuscripts for their particular journals, reported that too many papers are reviewed that are out of the scope of their journals.
I think it is a real challenge to try and match the editorial review with the aims and objectives of publication […]. So authors would be surprised in many cases when their article is accepted or rejected based on the stated aims of the journal […]. I think the big message is […] that it is problematic for publishers trying to get a handle on a focus for a particular community for their journal to meet the needs of that particular community. (Publisher, cross-disciplinary) Insufficient editorial triage was rationalized in different ways.
Editors and senior scholars with editorial roles explained that, as their responsibilities were delegated to editorial teams and software systems, their oversight and ability to filter out manuscripts that should not enter peer review have been reduced considerably. Reflecting upon their experience as authors, junior scholars explained that sometimes authors purposively submit manuscripts that are not yet publishable, hoping that they will be improved through peer review.
Yes, I do that sometimes. I submit premature manuscripts.
I am collecting anonymous reviews just because I would like to get some feedback on my work. (Postdoc, political sciences) As editors fail to filter out such manuscripts, reviewers would have to review these manuscripts, increasing their workload. Second, inefficient manuscript handling was related to peer review not being re-used when a manuscript has been rejected by one journal and is then submitted to another journal. Usually, when a rejected manuscript is submitted to elsewhere, editors recruit reviewers to assess the manuscript again and new demand for peer review is created, irrespective of the fact that a manuscript might have been thoroughly assessed before.
The same journals ask the same people to review a paper that has been rejected in one place and then goes to somewhere else. […] peer review does not get carried forward. There is a lot of inefficiencies. (Publisher, crossdisciplinary) Third, stakeholders pointed out that basic manuscript processing steps are still performed manually, even though they could be automated. Examples mentioned by stakeholders included identifying reviewers and correspondence with reviewers and authors, assessing conflicts of interest of reviewers and checking the manuscript structure meets journal policies.
These inefficiencies were believed to create additional albeit avoidable work for reviewers and editors, thereby further overwhelming the academic community.

Lack of institutionalization
There was cross-disciplinary agreement amongst stakeholders that, because reviewing is not institutionalized, scholars face difficulties to balance reviewing with already existing responsibilities.
Particularly scholars in the social sciences and humanities criticized that, even though reviewing serves crucial functions in scholarly publishing, it is neither part of their employment contract with the university nor included in research grants.
At the moment, what is weird is that there is a massive academic service that is really important for the whole [Scholars] review under huge time pressure because they've got a full time job doing something else, so they're squeezing it in. (Senior lecturer, computer sciences) As scholars are requested to review growing numbers of manuscripts, they might not find the time to accommodate all requests to review and hence decline to review.
Senior scholars and editors added that, because reviewing remains unrecognized as a scholarly output in the academic career trajectory, scholars would further be discouraged from reviewing. They elaborated that inadequate incentive structures exacerbate inequalities in the distribution of reviewing burden.
Because the criteria for academic hiring and promotion as well as for funding allocation put most weight on publications, scholars who still have to secure tenure would be inclined to publish their own research without reviewing an adequate number of their peers' manuscripts.
Of course, reviewing goes into Research Excellence Framework statement […], but it is sort of weirdly disconnected […] I think it would matter if there were a way that reviewing would be taken into account in all the actual things that matter to the academic trajectory, right? So, in the context of funding, the career and the tenure, and all these things. Because that is the thing that really matters to academics (Professor and editor, philosophy)

Lack of reviewing instructions
Senior scholars with editorial positions, editors and publishers considered unclear reviewing instructions to be a further cause of the overburdening of reviewers. Stakeholders shared that reviewers might not always be well informed about the reviewing instructions of a journal. This was considered problematic because reviewers might not be able to correctly predict the amount of work it takes to review a manuscript. They might either underestimate the workload and be overburdened with the task, or overestimate it and decline the request to review, making it difficult for editors to recruit sufficient numbers of reviewers.
There is a perception around that reviewing is a very diffi- Senior scholars with editorial positions, editors and publishers further believed that, because particularly inexperienced scholars might overestimate the effort involved in reviewing and decline to review, the unequal distribution of the reviewing workload would be intensified. It was also explained that in absence of clear reviewing guidelines, particularly inexperienced reviewers would assume what is expected of them. This might create redundant work as reviewers either assess aspects in detail that do not feed into publication decisions or fail to review aspects that are relevant, requiring editors to request additional comments.

Inadequate reviewer recruiting strategies
Irrespective of their discipline, editors, mid-to senior-career scholars and publishers believed inadequate reviewer recruiting to be a driver of the overburdening of reviewers. They believed that, as only certain groups of scholars are recruited to review, reviewing workload would be distributed unequally across members of the academic community. Publishers in particular reported that they had the impression that authors located in Asian countries submit growing numbers of manuscripts but are less often invited to review than reviewers located in high-income countries, particularly North America and Europe. Publishers speculated further that, because many journals consider having published previously as a requirement for being qualified as a reviewer, editors might not recruit early-career scholars, even though they would be willing and capable to provide thorough reviews.
I think a lot of journals would consider having published previously as a condition for being qualified to be prepared to review. This is some arbitrary level of previous publications or years of experience as a research group leader for instance (Publisher, life sciences) In addition, unclear reviewing criteria might intensify difficulties in recruiting early-career scholars. Because reviewing criteria are not always clear to potential reviewers, inexperienced scholars might overestimate the effort involved in reviewing and decline to review. It was concluded that, as editors and publishers currently fail to recruit authors located in Asian countries and early-career scholars as reviewers, the reviewing workload is skewed geographically and demographically. Junior scholars did not identify reviewer recruitment as an important driver of reviewer overburdening.

DISCUSSION AND CONCLUSION
By means of qualitative focus groups, this study provided an indepth exploration of how stakeholders involved in peer review, including early-, mid-, and senior career scholars, reviewers, editors, and publishers, perceived the burden that is placed on reviewers and what they believed to be the drivers of a potential overburdening of reviewers. It was also important to examine whether stakeholder perceptions differed depending on their relationship with the review process.
One key finding of this study was that across roles and disciplines stakeholders believed the reviewing workload to be unequally distributed amongst members of the academic community. Thereby, this study revealed a more differentiated account of reviewing burden than most anecdotal reports, which describe the entire academic community as being overwhelmed by its own demand for peer review (Alberts et al., 2008;Arns, 2014;Stahel & Moore, 2014). Stakeholders perceived that small groups of scholars would carry a disproportionate part of the overall reviewing workload, which might result in individual reviewers being overwhelmed and potentially overworked. This finding confirms mathematical models of the overall sustainability of the peer review system, which revealed that a small number of researchers handles a large share of the overall reviewing workload (Kovanis et al., 2016;Publons, 2018).
As a further key finding, this study showed that the unequal distribution of reviewing workload is caused by an imbalance between the demand for and the supply of reviewing resources.
The underlying causes of this imbalance were related to the incentive structure of academia. Stakeholders believed that, as the criteria for academic hiring and promotion as well as for the allocation of research funding prioritise publications over peer review, scholars might be inclined to publish their own research without reviewing an adequate number of their peers' manuscripts in return. A further key finding was that the causes of reviewer overburdening are interrelated and, in part, mutually reinforce each other. Such complex interdependencies stress the need for adopting a holistic approach in alleviating reviewing burden. This means that in order to alleviate reviewing burden, it is necessary to change the overall incentive structure in academia. This could be done by acknowledging peer review as a scholarly output in academic hiring and promotion decisions as well as the allocation of research funding, whilst making it part of scholars' employment contracts with their institution and including it in research grants. The overall reviewing burden could further be alleviated by automating basic manuscript processing steps, such as identifying suitable reviewers or assessing conflicts of interest of reviewers, by making reviews portable across journals and publishers and by improving reviewing guidelines and reviewer training.
Finally, depending upon their experiences and their relationship with the review process, stakeholders put different weight on the effects and causes of reviewer overburdening. Junior scholars' experiences with the review process were mainly limited to their roles as authors or reviewers, which led them to focus on potential consequences of the overburdening of reviewers, mostly in relation to superficial quality control and delays in the review process. In contrast, publishers, editors and senior scholars with editorial roles drew upon their experiences in managing journal business operations, handling manuscripts and recruiting reviewers. Doing so, they were able to take a more holistic perspective and share their insights into potential causes of the overburdening of reviewers. Having said that, the stakeholder groups agreed that there is a sense of urgency about the effects of an unequal distribution of reviewing workload.
This means that it might be challenging but not impossible to find solutions that are acceptable to the wider community.
We recognize a number of limitations in our study. First, there are limitations related to our sampling approach. Because participants were recruited using purposive maximum variation sampling, there might have been selection biases in how we selected participants. We tried to alleviate this by means of pre-defined recruitment criteria. Further, because focus groups were face to face, stakeholders who lived far from the workshop location were less likely to join than stakeholders within close proximity were.
We compensated travel costs to reduce geographical biases.
Nonetheless, stakeholders based in other countries than Switzerland or England were not represented in this study. Because academic publishing differs geographically (Collyer, 2018;Severin & Chataway, 2020), the generalizability of results was limited. Moreover, due to limited resources, the size of our sample was comparatively small, which might have limited our capability to comprehensively depict the causes of reviewer overburdening from the viewpoints of all relevant stakeholders. Second, because the validity of self-reported attitudes might suffer from inaccuracies in recollection, erroneous perceptions, incapability to answer correctly, and socially desirable answering (O'Sullivan, 2008), there might be inconsistencies between what stakeholders stated to be the drivers of overwhelmed reviewers and what they actually believed to cause overburdening. Further, there might be discrepancies between subjective stakeholder perceptions and reality. One example includes the fact that stakeholders believed submissions of manuscripts to have increased considerably, but failed to acknowledge that additional authors, that is, available reviewers, might also have entered the system, meaning that the article output per author might not have grown substantially. Finally, qualitative studies always includes some degree of subjectivity as the researcher's experience and judgement influence how data are collected, analysed and interpreted. To mitigate subjectivity, we based our analysis and interpretation on a codebook and give exemplary participant quotes (Severin & Chataway, 2020).
To our knowledge, this study is the first to explore how different stakeholder groups experience the reviewing burden that is placed on scholars and where they identify the causes for a perceived overburdening. Having identified underlying mechanisms of the overburdening of reviewers, this study aids understanding reviewing burden as an important challenge in the current peer review system. Based on this understanding, potential solutions can be developed and implemented.