Lost in reviews: Looking for the involvement of stakeholders, patients, public and other non‐researcher contributors in realist reviews

The involvement of non‐researcher contributors (eg, stakeholders, patients and the public, decision and policy makers, experts, lay contributors) has taken a variety of forms within evidence syntheses. Realist reviews are a form of evidence synthesis that involves non‐researcher contributors yet this practice has received little attention. In particular, the role of patient and public involvement (PPI) has not been clearly documented. This review of reviews describes the ways in which contributor involvement, including PPI, is documented within healthcare realist reviews published over the last five years. A total of 448 papers published between 2014 and 2019 were screened, yielding 71 full‐text papers included in this review. Statements about contributor involvement were synthesized across each review using framework analysis. Three themes are described in this article including nomenclature, nature of involvement, and reporting impact.


| INTRODUCTION
Researchers are increasingly being held to account to involve non-researcher contributors in their research and to provide accurate reporting of the nature of this involvement. 1,2 As a result, research teams increasingly structure the research process into a variety of advisory, expert or stakeholder meetings in order to account for alternative perspectives within the design, production and dissemination of research.
Evidence synthesis is one approach that has been quick to involve contributors such as stakeholders (eg, content experts or clinicians). However, it has been slower to navigate the more explicit involvement of patients and public. 3,4 Evidence synthesis is an umbrella term for numerous ways of reviewing and synthesizing secondary data. In addition to the mainstream Cochranestyle systematic reviews and meta-analyses, this now includes a range of alternative methods 5 in which researchers make interpretative and value-laden judgments throughout the research process. There are a number of opportunities throughout the process therefore, for individuals other than the immediate research team to contribute.
One theory-driven approach to evidence synthesis is realist reviewing. Realist reviews are a form of evidence synthesis that take into account the complexities of an intervention or innovation, producing causal explanations about what works, for whom and in what contexts. 6 We have recently conducted several realist reviews about healthcare organization and design [7][8][9][10][11] that have involved a range of contributors. We became aware that in our reviews we approached this differently-both in terms of who was involved and how. This observation made us curious as to whether this was the norm. Hence, we decided to more systematically identify the ways in which nonresearcher contributor involvement has been used within existing healthcare realist reviews. For the remainder of this article, we use the term "contributor involvement" to capture non-researcher involvement including (but not limited to) stakeholders, patients and the public, decision and policy makers, experts and lay contributors. Of particular interest for this review was the way in which patient and public involvement (PPI) was reported (or not).
PPI refers to research carried out with and by members of the public. 12 Originating in the 1990s, PPI policies aimed to democratize medicine and challenge the dominant authority of and disillusion with healthcare decisionmaking. 13 PPI is often described as transforming the way in which healthcare research is undertaken. It is hailed as a means to improve efficiency and social accountability and to balance power dynamics through a democratic dialogue. 13 However, a long-standing issue with PPI is the limited evidence available about both reporting the involvement of patients and the public (ie, how and why) and reporting the impact of PPI on research. 14,15 Guidance on involving patients and the public within realist reviews has received much less attention than that in other forms of evidence synthesis. 16,17 2 | METHODS This review examines the ways in which contributor involvement is described within healthcare realist reviews published since 2014. We chose to select articles published from 2014 onwards because prior to this date there were no expected standards to which realist reviews could be held up to. The Realist And MEta-narrative Evidence Syntheses: Evolving Standards (RAMESES) were published in 2014; despite not setting any standards for involving contributors, they acted as an international standard for undertaking realist reviews more broadly. 18 The purpose of this review was to (1) describe the ways in which contributors have been involved in realist reviews, with a particular focus on PPI and (2) document how involvement has been reported. This review responds to the research question, "in what ways have contributors been involved in healthcare realist reviews and how has this been reported?" and relatedly, "in what ways are patients and the public involved (or not) in healthcare realist reviews?"

Highlights
Evidence synthesis methods often involve a number of different non-researcher contributors throughout the research project lifecycle. There are varied and multiple ways of involving contributors. Realist review approaches do not always make contributor involvement explicit, in particular patient and public involvement.
This review synthesizes the ways in which contributors have been involved in realist reviews, including patients and the public. Based on our study findings, we emphasize a clear need to identify who contributors are, when they are involved and how, so that their impact is not lost in future realist reviews.
For readers both inside and outside of the realist research community, we encourage reflection, collaboration, and development of further guidelines that help to recognize contributor involvement in a number of different forms and structures. support of an information specialist (NR). Our initial search strategy was kept broad in order to locate as many realist reviews as possible within healthcare. We limited the search to English language and papers published from 2014 to 2019; conference abstracts were excluded. An example of our search strategy can be found in Table 1.

| Study selection
Using a systematic approach, 448 papers published between 2014 and 2019 were screened initially by RA using title and abstract. Papers were included for full-text screening according to the following criteria: • Peer-reviewed articles • Full and completed realist reviews (ie, not protocols, not systematic reviews) • Undertaken in any healthcare setting, in any country.
A total of 155 papers were included for full-text screening. These papers were read in full and included if they met the following criteria: • Included any key words associated with contributor involvement such as: stakeholder(s), PPI/contribution, expert(s) panel/group, and advisory panel/group.
We used these terms to capture the breadth of ways contributors have been described in reviews, and to account for the possibility of involvement not formally labeled as PPI. Papers referring to any of these terms were included. This step yielded a total of 71 papers making active reference to contributor involvement. All papers selected for inclusion were then double-screened by JR ( Figure 1).

| Data analysis
Data were analyzed by RA, JR, and SP using framework analysis. 19 Framework analysis follows a six-stage process including familiarization, coding, developing an analytical framework, application of the analytical framework, charting the data in a framework matrix, and interpreting the data. 19,20 It is a rigorous, systematic, and transparent process for data management and its flexible, iterative nature means that it is well-suited to our review process. 21 Our framework analysis matrix can be found in the supplementary material. To help guide our analysis, and as part of an iterative process between the authors, we asked the questions set out in Table 2 of our data.

| FINDINGS
Of the 71 papers included in this review, 16 came from North America and five from Australia. The remaining 50 papers came from Europe, with the majority originating in the UK (n = 36). We present three categories from the analysis of contributor involvement in healthcare realist reviews: (1) nomenclature; (2) nature of involvement; and (3) reporting impact.

| Nomenclature
A range of terminology was used to describe contributor involvement (including PPI) in realist reviews. We have categorized this involvement as part of our data synthesis using the terms and delineations provided in the articles. This includes experts/ expert/reference panels/groups (typically comprising academics, content experts and at times service user representatives) (n = 25); stakeholders (typically comprising commissioners, policy-makers, service providers, and key informants) (n = 32) and advisory groups (typically comprising clinicians, academics, and service providers) (n = 14).

| Patient involvement
Of the 71 papers included, four made direct acknowledgment to patient involvement but did not label it as such. 22

| Public involvement
Of the 25 papers referring to "experts," seven referred specifically to service users who were positioned as members of the public. [26][27][28][29][30][31][32] The only difference between these seven papers is seen in the work of Pearson et al, 30 who made use of the term "peer researchers" for service users involved in their advisory group.

| Nature of involvement
Contributors termed "stakeholders" (eg, policy-makers and service providers) tended to be involved during the initial stages of realist reviews. This included consultations, [33][34][35] informal stakeholder interviews to help inform initial programme theories, 36,37 and workshops held at the beginning of the research project. 38,39 In the studies drawing on "expert" or "advisory panel/ groups" (eg, academics, clinicians), individuals were involved in the development of research questions, the sharing of relevant literature and aspects of data extraction or analysis. 40,41 Conversely, papers reporting the involvement of patients generally described the role of contributors as providing expertise on experiential knowledge to inform programme theory development, 22 or advice throughout the project. 23 McNeil et al 24 involved 17 patients as stakeholders at a one-day workshop used to generate understanding and assumptions associated with patient engagement to clarify the scope of their review. They also held another workshop with 11 patients to discuss their findings. Papoutsi et al 25 held four meetings, each of about two-hours in length across the entire project life cycle.

| Reporting impact
A number of reviews provided a short commentary on the impact of contributor involvement. This reporting of impact included the ways in which a particular group or meeting may have helped to shape emergent findings, refine final programme theories or enhance the conceptual clarity of a review's mechanisms. [42][43][44] Generally, this type of reporting was brief but descriptive in detail. For other reviews, the ability to report on impact was likely to be contingent upon the point at which contributors were involved in the various stages of the realist review. For example, the input of experts may come at a later stage in the project, that is, when findings and knowledge generated from the review are disseminated, post-project completion, as opposed to having an immediate effect. 45 Other reviews have discussed contributor involvement but then not commented on the impact of this in relation to their review. 46,47 One exception to the above descriptions was De Weger et al, 48 who integrated the reflections of contributors throughout the results section of their review. By and large, however, what counted as impactas "worthy" of being reported-appeared to be dependent on an individual review team's judgment, increasing variation of reporting across realist reviews.

| DISCUSSION
This review of 71 papers explores the ways in which contributors, including patients and the public, are involved in realist reviews. The majority of included papers demonstrate that a range of contributors are involved, but involvement is not always clearly reported. Findings from this review indicate that contributors, particularly patients and public, are incorporated into already existing structures or groups including stakeholder, expert, or advisory panels. This may, at a later point (ie, reporting stage), make it challenging for researchers to identify the exact nature of involvement or contribution made, specifically by patient and public contributors, if required or asked to by funders, journal editors or other stakeholders. Indeed, extant research indicates that contributor involvement in research more broadly, particularly patient involvement, is often not acknowledged or may be inconsistently accounted for at best. 1 This is significant given the increasing drive by bodies, such as the United Kingdom's National Institute for Health Research, to both encourage and account for contributor involvement. The fact that contributor involvement may get "lost" in reviews is problematic, particularly so when approaches that foster openness and reciprocity are called for to support contributor involvement in research. 49 Findings from this review also suggest that the point at which contributors are involved in realist reviews appears to differ across projects. Advice on contributor involvement in other forms of evidence synthesis such as traditional systematic reviews has been published by the organization INVOLVE. 12 Involvement might include: (a) defining the scope of the review; (b) assisting with keywords for literature searches; (c) suggesting relevant literature; (d) appraising literature; (e) interpreting findings; (f) disseminating results. 12,50,51 In realist reviews, contributors play an additional role in helping to devise and reshape programme theories as the study progresses. This demonstrates that, for realist reviews, contributor involvement may in part be connected to specific project milestones. As such, contributors may be expected to provide input at a higher level of conceptual complexity. It may therefore not be necessary or appropriate to expect or invite their involvement across an entire project cycle, as advocated for in other types of evidence syntheses and research. 1,12,49 Inviting involvement at key stages of a realist review respects both the expertise as well as the boundaries of that expertise within any given stakeholder, expert or advisory group. It is, however, unclear how to do this in a way that is ethical (since literature reviews do not require an ethics approval, there is no guidance on handling ethical dilemmas that may arise in using contributors' statements), and transparent about the provenance of interpretations without burdening the text of a review and without introducing rankings of expertise that the realist approach does not endorse. What is clear is that to support contributor involvement in realist reviews, researchers need to create clear ways to communicate what realist reviews are and how they might differ from other forms of synthesis so that contributors and funders understand what they are supporting and how.
Finally, findings from this review demonstrate that, as with other forms of contributor involvement in evidence synthesis and research more broadly, reporting of involvement is poor. 1,[52][53][54][55][56] This limits the ability to articulate contributor impact because of an absence of detail about which tasks they have involved in. 52,53 The decision-making process involved in reporting contributor involvement (or not) could be for a number of reasons. For example, word count restrictions in research publications may be one reason or there may be pragmatic constraints such as time, funding, research agendas, and researcher experience. 54,55 In order to move away from current constraints, Price et al 1 indicate a need for journals, funders, and research institutions to work together to support the reporting of contributor involvement through, for example, standardized reporting measures. However, a blanket approach to reporting contributor involvement may not be an appropriate solution when different types of research have different requirements or needs from their contributors, unless these can be shaped around a set of essential and desirable principles that act as guidelines as opposed to rules. 56 For realist reviews, in particular, there is a lack of guidance or standards available on both undertaking and reporting contributor involvement. The realist research community could reflect on this and develop ways to consider contributor involvement, in particular of patients and the public. Working towards more concrete guidance on both the nature and reporting of involvement in realist reviews is one way of moving forwards. This may include drawing on existing frameworks employed in other modes of evidence synthesis that help to articulate tasks and roles for contributor involvement. 52,53 As a result of the findings presented above, and in collaboration with two patient representatives, we generated a series of prompts for both researchers and contributors involved in a realist review. These prompts are available online 57 and recognize the need for and purpose of contributor involvement to differ between projects. These prompts, which are supported by our review's findings, could form a foundation and stimulus for future research to develop recommendations on the role of contributors, such as patients and the public, in realist reviews, and provide guidance to support researchers in their future collaboration with contributors.

| Strengths and limitations
Whilst all the full-text articles included in this study were double-screened, the initial screening of title and abstract was single screened by one reviewer. We did not conduct any quality appraisal of the included reviews. Had we found sufficient variability in the quality of reporting of contributor involvement across reviews, it would have been informative to consider if this correlated with other parameters of higher review quality. However, since the reporting of involvement varied across all reviews, it was of no direct value to the goals of the study to carry out individual quality appraisals.
We did not explicitly include additional contributors such as patients or the public in our review as the research required the skills of trained individuals with knowledge of data screening, extraction, and analysis, skills, which were largely present in our research team. However, during this review, we did feel a need to reflect on the role of patients and their perspective in terms of contributing to evidence syntheses and, as indicated, developed with two patient representatives a series of reflective prompts to support contributor involvement in realist reviews.

| CONCLUSION
This review has synthesized the terminology, nature, and reported impact of contributor involvement across healthcare realist reviews published in the last five years. Whilst the majority of included articles demonstrate that a range of contributors are involved in realist reviews, this synthesis has highlighted the variation across reviews and under-reporting of contributor involvement. In particular, PPI is not always made distinct from other types of involvement. The limited guidance available to realist reviewers regarding contributor involvement hinders this aspect of research transparency and knowledge generation, which is a crucial part of contributor involvement, particularly for realist reviews and programme theory development. As a result, there is now a clear opportunity to shape the role of contributor involvement, including PPI, in realist reviews by developing guidance to support researchers in their future collaborations. news/blog/the-role-or-not-of-patients-and-the-public-inrealist-reviews