Improving peer‐review by developing reviewers' feedback literacy

There is a need to train journal peer reviewers to provide professional, constructive, and actionable feedback: i.e., develop their feedback literacy. Journals and publishers can improve the way they support peer reviewers' feedback literacy by raising awareness and providing guidance and exemplars of good practice. Existing online peer‐review training resources developed by major publishers only focus on content of feedback but neglect the socio‐emotional aspect of feedback. Resources to develop peer reviewers' feedback literacy should be formulated by adopting knowledge‐based, skills‐based, and community‐based approaches.


INTRODUCTION
Peer-review is a central and integral process of academic publishing. It serves an important function of ensuring the quality of journal publications and advancing the development of research fields (Rowland, 2002). Useful and constructive feedback from peer reviewers helps authors improve their manuscripts. On the contrary, ill-conceived and cynical feedback can be detrimental to both scholarship and wellbeing of academics (Silbiger & Stubler, 2019). Therefore, it is crucial to develop peer reviewers' feedback literacy, which refers to their capacity, disposition, and skills to provide constructive, professional, and actionable feedback to authors. Despite the importance of feedback literacy in scholarly peer-review, there is insufficient attention given to the training and professional development of peer reviewers, who comprise researchers at different career stages and with varying experiences in peer-review. It is also worth noting that there is a growing number of journal peer reviewers who are early career researchers (McDowell et al., 2019). Peer-review, because of its often double-blind nature, is done 'behind-the-scenes', making the process more mysterious. Although there are existing guidelines on most journals' and publishers' websites for peer reviewers, I can speak from experience as an associate editor of two international refereed journals, an active peer reviewer and author, and an early career researcher, that there is a lack of professional development opportunities for journal peer reviewers, especially in relation to how peer reviewers ought to give feedback. In this article, I introduce the notion of feedback literacy to the academic publishing community and suggest ways to develop feedback literacy of peer reviewers. I will then review existing online training resources provided by major publishers for peer reviewers to underscore the deficient attention paid to developing peer reviewers' feedback literacy. Finally, I offer recommendations for improving these online resources.

WHAT REALLY IS FEEDBACK?
Feedback is not just a product but a complex process (Winstone & Carless, 2019). Giving feedback and motivating recipients to engage with feedback require expertise and experience. Therefore, it is of paramount importance to develop feedback literacy of journal peer reviewers. Feedback literacy has been an embryonic yet exciting field of research in assessment in higher education with a plethora of conceptual and empirical works published in the past 2 years. In a nutshell, to feedback recipients, feedback literacy refers to their abilities, capacities, and dispositions of individuals to cognitively, emotionally, and behaviourally engage with feedback. From the perspective of feedback givers, feedback literacy concerns their abilities to provide feedback which facilitates recipients' engagement with feedback. In their seminal work and focusing on students, Carless and Boud (2018)  Relating to peer-review, academics' feedback literacy is much less discussed in the literature. In a recent publication, Gravett et al. (2020) reported on a collaborative autoethnography on academics' feedback experiences as authors engaging with critical feedback from journal peer reviewers, arguing that academics are often placed in a situation similar to the one experienced by students in an educational context. While publications on feedback literacy mainly focus on teaching and learning, I would argue that the notion is a very relevant one to journal peer reviewers. In the subsequent section, I would refer to existing feedback literacy models to underscore the similarities and differences of feedback literacies of students/teachers and journal peer reviewers. My key message is that it is pressing for journal peer reviewers to develop their feedback literacy because they face more constraints than teachers and students in the feedback process. Suggestions on how to develop peer reviewers' feedback literacy will also be provided.

DEVELOPING FEEDBACK LITERACY OF JOURNAL PEER REVIEWERS
Understanding the nature of feedback Carless and Boud (2018) contended that students need to appreciate and understand the usefulness of feedback and their roles in the feedback process in order to engage with feedback meaningfully. Likewise, journal peer reviewers have to understand the purpose of giving feedback to authors and the kinds of feedback useful to authors. In the context of peer-review, feedback is often viewed as written information which provides judgement and evaluation of manuscripts. For instance, focusing on the content of feedback, Silbiger and Stubler (2019) defined unhelpful peer-review feedback as 'any statement that is unethical or irrelevant to the nature of the work' (p. 2). It is undeniable that peerreview feedback needs to include comments on the strengths and weaknesses of submissions, preferably with suggestions. However, in addition to the 'what to comment' question in peerreview feedback, it is equally important to consider the question 'how to comment'. It is often the case that journal peer reviewers perceive feedback as a written report of comments; very few, in my personal view, consider feedback as a dialogic process (Chong, 2018a). When conceptualizing feedback as a dialogic process and considering the 'how' in feedback, there are several important questions for peer reviewers to ponder over: • How can I determine feedback foci?
• How can I provide feedback that is actionable?
• How can I provide feedback that is specific? • How can I provide feedback that is manageable?
• How can I provide feedback that facilitates authors' reflections?
One way to draw peer reviewers' attention to the dialogic nature of feedback is for journals or publishers to explicitly state their expectations and definitions of 'useful feedback'. It is a common practice to include information about how manuscripts will be assessed on journals' websites, in submission systems, and peer-review invitation emails. These guidelines are useful for peer reviewers to focus their feedback information on areas valued by journals. Similar guidelines or checklists can be included to help peer reviewers envisage what good feedback looks like, for example, by using the questions above or including short descriptions of characteristics of 'effective feedback' based on the aforementioned questions.

Managing emotions
Feedback is a dialogic process. Like other kinds of dialogue, effectiveness of feedback dialogues is contingent upon the relationship between interlocutors. Feedback literature has suggested that a trusting relationship is vital to feedback engagement (Chong, 2018b). Equally important, it is important for feedback givers to cater for the emotional needs of feedback recipients, for instance, through hedging criticisms and incorporating evidencebased praises (Rowe, 2016). Nevertheless, it is not to say that feedback givers should attempt to please the recipients at the expense of giving honest and critically constructive feedback. It is worthwhile to consider how feedback can be framed to be textfocused rather than person-focused (Hattie & Timperley, 2007). Applying such understanding to feedback in peer-review, I believe it is crucial for peer reviewers to contemplate on the affective effect of their feedback on the authors. Authors, especially junior researchers, have devoted a lot of time into writing up a manuscript; many of them may have very limited experience submitting their works to journals. Regardless of the quality of the manuscript, their effort and intention to contribute to the field should at least warrant our respect and appreciation, which needs to be explicitly shown in our feedback. It must be noted that, however, it is difficult for peer reviewers to develop a trusting and professional relationship with authors which leads to positive feelings because they face more constraints than teachers when giving feedback. Table 1 summarizes the limitations of peer-review feedback when compared to feedback given in an educational setting.
To develop trust and consider the emotional responses of authors, peer reviewers may want to take into consideration the following when giving feedback: • How can I show respect and appreciation to the author explicitly?
• How can I direct my feedback to focus on the manuscript but not the author?
• How can I gain the author's trust by giving feedback in a professional manner?
• How can I clarify my position as a peer reviewer and acknowledge the limitations of my perspective?
What can journals and publishers do to raise peer reviewers' awareness of the socio-emotional aspect of feedback? Similar to my earlier suggestions, the simplest way is to incorporate the above questions into the checklist sent to peer reviewers who have agreed to review a manuscript as a reminder. Sometimes, peer reviewers may not read the instructions and reminders in emails or on the websites; therefore, I think it is important to have these questions included in the manuscript submission system. Some journals include evaluation checklists in the system for peer reviewers to complete in addition to finishing their reports. In a similar vein, peer reviewers can be asked to respond to some questions related to the tone and wordings in their feedback before they can successfully submit their reviews (e.g., the above questions can be rephrased into yes/no questions: 'Is my feedback directed towards the manuscript but not the author?'). In this way, it will help ensure peer reviewers are attentive to the style and formality of their feedback. Furthermore, some journal editors create short videos introducing the scope of their journals which are uploaded to the journals' websites. These videos can also include a part on explaining what good peer-review feedback means; in some Identity of feedback receiver Teachers know their students and usually have at least a semester of time to get to know them better. Since teachers can identify needs and personalities of their students through observing their behaviours or talking to them, it is easier for teachers to develop appropriate student-centred feedback strategies.
In most cases, peer-review in my discipline (education) is double-blind, meaning that the identities of the peer reviewers and authors are hidden. Certainly, there is good reason to do that, for example, to uphold the integrity and objectivity in the review process. However, when it comes to developing relationship, it becomes more difficult (but not impossible) because there is no mechanism in the peer-review system for the peer reviewers and authors to get to know each other.

Impact of feedback
Teacher feedback is not always high-risk. Teachers can give feedback informally through conversations with students. In these more relaxing situations, students are less likely to feel defensive and may be more ready to take on board comments.
Feedback by peer reviewers is always high-stakes. As much as we intend to make the peer-review process developmental, it is undeniable that peer reviewers play the role of gatekeepers who need to judge the quality of manuscripts and recommend a decision to the editor. It may be more challenging for the authors to engage in an honest conversation with the peer reviewer; instead, they may be more inclined to accept all suggestions by the reviewers because their primary objective is to get their work published.
cases, it may be helpful to mention some of the 'don'ts', including how peer reviewers should refrain from making personal accusations or patronizing the authors. The advantage of including such information in a video is that editors can adopt a personal or case study approach to share their own experiences or invite some experienced and successful peer reviewers to share some tips on giving feedback using appropriate tones and wordings.

Making judgement
Making judgement refers to an individual's ability to determine the quality of a piece of work based on a set of assessment standards or evaluation criteria (Tai et al., 2018). In educational context, research shows that it is difficult to develop students' understanding of assessment standards because it is regarded as a kind of tacit knowledge which can hardly be explained but demonstrated (Sadler, 2010). Accordingly, feedback researchers have suggested various ways to develop students' evaluative judgement, including the use of exemplars (Chong, 2019), co-creation of rubrics (Fraile et al., 2017), and peer assessment (Falchikov & Goldfinch, 2000). In journal peer-review, it is equally if not more important to train peer reviewers to make accurate evaluative judgement because peer reviewers act on behalf of the journal's editor and editorial board to determine the suitability and quality of manuscripts. While general guidelines are often provided on journals' websites or in journals' invitation emails, these are often vague and unclear with wordings which can evoke multiple interpretations (e.g., The submission should make an original and substantial contribution to the field). It is especially difficult for novice peer reviewers or junior researchers with limited publishing experience to understand a journal's standards because peer-review is often done individually with little support.
I hope the following questions could help peer reviewers understand journal-specific standards: • What are the expectations of the journals as stipulated in the author guidelines (this is especially important when a journal has multiple sections)?
• How are these guidelines translated into practice as exempli-  In my opinion, this is an aspect of feedback literacy which many journals focus on. A lot has been done to promote evaluative judgement of peer reviewers, most notably and commonly through asking peer reviewers to refer to information on the journal websites (e.g., scope and aim, instructions for authors).
However, more can be done to make such information more accessible to peer reviewers. For example, as suggested previously, by making it a compulsory step for peer reviewers to complete an evaluation checklist based on journals' submission guidelines before they submit their reports. This is especially important to journals which have multiple sections, and guidelines differ from one section to another. Another issue peer reviewers face is the ability to understand submission guidelines, which are sometimes written in ambiguous language. As an associate editor of a language education journal, the editors-in-chief and I have recently decided to share some examples of good publications as exemplars to help peer reviewers understand the journal's expectations and standards. With the consent of the publisher, we are able to make a few exemplary publications open access and we place them at the bottom of the 'instruction for authors' page.
Peer reviewers, especially new ones, are encouraged to refer to them.

DEVELOPING PEER REVIEWERS' FEEDBACK LITERACY
Professional development of journal peer reviewers is not unheard of. In fact, there are peer reviewer training resources developed by major international publishers. These online resources usually take the form of short online courses. In this section, I attempt to provide an overview of the peer reviewer training resources prepared by Wiley, Taylor & Francis, Elsevier, Sage, and Publons (Table 2). As a disclaimer, I registered to the online training programmes (except Taylor & Francis because it is a live event and Sage because it does not require registration) and my analysis is based on my browsing of some of the content of these resources. Since this is an opinion piece, it is not my intention to provide a thorough and in-depth analysis of these resources nor do I aim to evaluate the usefulness of these resources.

Content
The five peer reviewer training resources resemble each other closely especially with regard to their content. Despite structuring the resources slightly differently, all of them cover important areas of peer-review such as the peer-review process, the role and responsibility of peer reviewers, and the features and structure of a reviewer report. All these training resources seem to have a lopsided focus on developing peer reviewers' holistic understanding of the peer-review process. Much less attention, however, is paid to developing peer reviewers' ability and capacity to give effective feedback.

Features
The most common features included in these five resources are videos and slides. Some unique features can be found on some platforms. For instance, Taylor & Francis conduct live training for peer reviewers, and they include case studies and invite peer reviewers to share their experiences. They also connect junior peer reviewers with journals and offer one-on-one support, providing feedback on some of their reviewer reports. Other unique features include a space for reacting to the video content (Elsevier), exercises to check understanding (Publons), mentorship (Publons), exemplars of peer-review reports (Publons), and links to external resources (Sage).

Relevance to feedback literacy
All the five platforms include resources on feedback in peerreview. These resources focus almost exclusively on the content of feedback, advising reviewers on the structure of a reviewer report, the focus of feedback for each section of the manuscript.
The platform developed by Wiley seems to address the relational and emotional aspect of feedback by introducing good practices of giving feedback which concern not only the clarity, credibility, and structure of feedback but also appropriateness of feedback. Advice such as writing in an objective tone and striking a balance between criticisms and constructive feedback is provided in the form of descriptive paragraphs.

RECOMMENDATIONS FOR ONLINE PEER-REVIEW TRAINING RESOURCES
Based on the above, I would like to propose alternative approaches and new features which publishers and journals can add to their online training resources to facilitate the development of feedback literacy of the next generation of peer reviewers.

A knowledge-based approach
This is probably the existing approach adopted by all the major publishers. The online resources and courses developed by leading publishers focus on helping new peer reviewers understand the process of peer-review, the roles of different stakeholders, and the features of reviewer reports. Useful features to include can be recorded presentations on good practices of peer-review feedback, podcasts interviewing not only journal editors but novice/experienced reviewers (see a podcast (https://www.bera.ac. uk/media/the-bera-ecr-network-presents-navigating-peer-reviewbehind-the-scene) I did with Shannon Mason for British Educational Research Association), resource banks collating useful materials about giving peer-review feedback (see a website

A skills-based approach
This approach is less common in the existing online peer-review courses. A skills-based approach to developing feedback literacy of peer reviewers focuses on the practicalities of giving feedback (the 'how' I have been mentioning). This can be realized through the development of a 'feedback bank' which stores samples of peer-review feedback or excerpts of reviewer reports (see Publon's example to include feedback samples from open peerreview journals) and annotations by journal editors regarding the strengths and weaknesses of the feedback. In this way, less experienced peer reviewers can have a solid understanding of what good peer-review feedback looks like. Another way is to incorporate skills-based components into the existing online courses. Again, using authentic peer-review feedback as exemplars, participants can be asked to evaluate the ways feedback is given.

A community-based approach
This is probably the least practised approach nowadays. A community-based approach to developing peer reviewers' feedback literacy aims to gather a group of likeminded people and different stakeholders in the peer-review process. This could be a formal community or an informal community. A formal community can be initiated by a journal's editor and editorial board or through a publisher's central training scheme. For example, the journal Higher Education Research & Development (Taylor & Francis) has a very well-established programme to nurture junior peer reviewers. This includes shadowing by a more experienced peer reviewer. Its sister journal, Advancing Scholarship & Research in Higher Education, operates a group-based peer-review process. Led by an editorial board member, peer reviewers work in groups and draft individual reports. Then these reports are discussed in a group meeting and consensus is reached. Looking ahead, it may be beneficial to consider initiating a mentorship programme at the journal level and make it one of the responsibilities of editorial board members to coreview with new reviewers on some manuscripts as a mentor. In some cases, it may even worth pondering over the possibility of having an editor or senior editorial board members responsible for overseeing professional development of peer reviewers.
Informal communities can also be extremely useful to bring together peer reviewers to discuss issues related to feedback in peer-review. Given its informal nature, the communities will not be limited by disciplinary boundaries and could serve as excellent platforms to promote interdisciplinary dialogues on feedback practices. Social media platforms may be a suitable means to host such communities. For an example, see an online peer-review community recently set up by Shannon Mason and me on Twitter (@Scholarly_Peers (https://twitter.com/Scholarly_Peers)).

CONCLUSION
In this opinion piece, I have introduced the notion of feedback literacy and identified its relevance to professional development of journal peer reviewers. I have discussed attributes of feedback literate peer reviewers which include (1) the consideration of not only the 'what' but 'how' when giving feedback; (2) the capacity to address the emotional and relational aspects of feedback; (3) the ability to make accurate and evidence-based judgement about manuscripts. I have also provided specific suggestions for journals and publishers to help peer reviewers become aware of not only what information should be included in feedback but how feedback information can be presented in a way which shows support and respect to authors. In the second half of this article, I have provided an overview of five online peer-review training resources developed by leading publishers and analysed their relevance to developing feedback literacy of peer reviewers.
Finally, looking ahead, I suggested three directions for publishers and journals to develop peer reviewers' feedback literacy through enriching the content in their online training programmes.