Mainstreaming public involvement in a complex research collaboration: A theory‐informed evaluation

Abstract Introduction There is an extensive literature on public involvement (PI) in research, but this has focused primarily on experiences for researchers and public contributors and factors enabling or restricting successful involvement in specific projects. There has been less consideration of a ‘whole system’ approach to embedding PI across an organization from governance structures through to research projects. Objective To investigate how a combination of two theoretical frameworks, one focused on mainstreaming and the other conceptualizing quality, can illuminate the embedding of positive and influential PI throughout a research organization. Methods The study used data from the evaluation of a large UK research collaboration. Primary data were collected from 131 respondents (including Public Advisers, university, NHS and local government staff) via individual and group interviews/workshops. Secondary sources included monitoring data and internal documents. Findings CLAHRC‐NWC made real progress in mainstreaming PI. An organizational vision and infrastructure to embed PI at all levels were created, and the number and range of opportunities increased; PI roles became more clearly defined and increasingly public contributors felt able to influence decisions. However, the aspiration to mainstream PI throughout the collaboration was not fully achieved: a lack of staff ‘buy‐in’ meant that in some areas, it was not experienced as positively or was absent. Conclusion The two theoretical frameworks brought a novel perspective, facilitating the investigation of the quality of PI in structures and processes across the whole organization. We propose that combining these frameworks can assist the evaluation of PI research.


| INTRODUC TI ON
Public involvement (PI) is an integral element of much health research, often required by funders. It is promoted as a means of quality improvement: increasing research relevance, and improving study participant recruitment and retention. 1 It is also advocated on ethical and political grounds, promoting values of justice and fairness and increasing democratic accountability for public funds. 2 Previous research, primarily focused on PI in specific projects, has highlighted the following as contributing to 'successful' PI: clear systems to recruit and support public contributors; targeted communication to raise public awareness about opportunities available 2 ; resources to ensure appropriate recruitment 3 and recognize the input of public contributors, 4-6 involvement from early stages in the research process 7 ; and the ability to tailor roles to both project requirements and the needs of public contributors. 8 Training for public contributors has also been identified as important, particularly where they take on technical roles. 5 Greater impact has been reported where goals for PI have been defined early and are reflected in implementation plans. 7 Studies also suggest that dedicated co-ordination and support roles in research teams facilitate the embedding of PI in projects. 3,4,6 A recurrent research finding is that researchers need to build and nurture relationships with public contributors so they feel supported and part of the research team. 3,5,8,9 This requires researchers to have both time and a positive attitude about the value of PI. 2 Training for researchers to address differences in knowledge and experience of PI and opportunities to reflect and share best practice have been advocated. 2,3,8 More reflective practice may also enable researchers to provide feedback on the value of public contributions, facilitating their learning and development and motivating further involvement. 10,11 In addition to project-specific barriers and enablers, there is widespread agreement that effective PI in research requires a whole-system approach. Some studies have pointed to elements of this 'systems' approach suggesting, for example, that for the benefits to be maximized public contributors need to be involved throughout the research process (from priority setting to dissemination of findings) and research leaders need to be committed to/champions for PI. 2,5,6,12 Other factors shown to impact on the quality of PI in decision-making in general and research in particular include opportunities for cross-organization learning, investment in dedicated involvement infrastructure and an 'involvement culture'. 5,6,8,[13][14][15] Specific elements of organizational culture identified as contributing to successful PI include the following: organizational commitment to learning and changing in response; non-hierarchical collaboration between professionals and public contributors; and staff behaviour reflecting mutual recognition and respect. 9 Notwithstanding these important insights, knowledge about how a 'whole system' approach to embedding and sustaining high-quality PI across research organizations is patchy and potentially important areas, such as PI in organizational governance, are particularly neglected. This paper aims to contribute evidence in this area by presenting findings from an evaluation of the approach taken by a large research collaboration to embedding PI across all organizational levels, processes and activities. Two complementary theoretical frameworks informed the study: an established mainstreaming framework 16 with a strong organizational focus and the Involvement Cube 17 focusing on aspects of the quality of PI. The rationale for selecting these is discussed further below.

| S TUDY S E T TING AND DE S I G N
The evaluation was based in the English Collaboration for Leadership in CLAHRCs were expected to involve the public in their work. 18 However, evaluation of their experience is limited. In the first wave CLAHRCs, only three projects investigated how PI was 'enacted' and how roles developed as part of wider evaluations 9,12,19,20 whilst a more recent study used an action research approach to look explicitly at enablers and barriers to PI in research in a CLAHRC. The evaluation was primarily qualitative to enable in-depth investigation 21 with content analysis of internal documents and analysis of routine monitoring data undertaken to varying degrees across the four components. It was conducted by teams of academics and Public Advisers. In addition, a panel of six Public Advisers contributed to the study design and the interpretation and dissemination of findings. Importantly, the academics leading and conducting the K E Y W O R D S community involvement, evaluation, mainstreaming, Public Adviser, public involvement, theoretical framework evaluation had research roles in CLAHRC-NWC and no responsibility for strategic delivery of, or support for, PI.
In total, data were obtained from 131 individuals: semi-structured face-to-face interviews (n = 58) and group interviews/workshops (n = 73). These included staff from CLAHRC-NWC's NHS, local government, university and not-for-profit partners; Public Advisers; and professional interns (Appendix S1). Information sheets and consent forms emphasized that participation was voluntary. Ethical approval was ob- Interviews for the neighbourhood programme evaluation focused on involvement of residents and asked if their experience had impacted on other activities they were involved in. The volume and detail of data therefore varied across components but together provided a 'thick' picture of PI across CLAHRC-NWC.
All interviews were recorded and transcribed. A combination of deductive and inductive approaches was used to develop an coding frame, based initially on existing theory around mainstreaming and PI, as the analysis evolved additional codes were added. 22 Researchers familiarized themselves with the data by reading the transcripts, noting new codes. The final coding frame was uploaded to Excel, systematically applied to all transcripts with the first 20 tested by two researchers. CLAHRC-NWC policies and strategies and Steering Board minutes were reviewed to identify references to PI, and routine PI monitoring data were utilized. Data were coded into a set of analytical charts, which were studied to identify common or divergent perspectives and discussions within the team iden-

| THEORE TIC AL FR AME WORK S
CLAHRC-NWCs aspirations for PI had two dimensions. The first was to incorporate 'the public, including patients, service users and carers and members of communities of interest and place across all of our work.' In effect, PI was to be mainstreamed across the organization, that is 'accepted as normal by most people'. 23 To assess progress towards meeting, this aim we drew on the gender main- In selecting these two theoretical frameworks, we were mindful of Edelman and Barron's argument that evaluating PI as if it were a complex intervention can result in too narrow a focus. 25 The diversity of theoretical and methodological approaches used in evaluations of PI in research has helped address this risk. However, two alternative approaches most relevant for our study are underpinned by an intervention perspective: Normalization Process Theory 26 developed to assess how complex social interventions are embedded and sustained in practice and Realist Evaluation 27 developed to assess whether 'a policy works, for whom, in what circumstances'. In contrast, the gender mainstreaming framework has an explicit focus on organizational cultural and structural change. We wished to test the added value of combining this framework with one focused on drivers of quality PI.

| Cultural change: adopting the terminology and sharing a vision
The first stage of the mainstreaming process requires the development of shared understandings about PI to be articulated and Importantly, PI terminology evolved over time. Initial documentation reveals deliberations about the meaning of, and distinction between, 'engagement' and 'involvement'. Over time 'involvement' was used more consistently to reflect the aspiration for all the collaboration's activities to be conducted 'with' or 'by' members of the public. The early title of 'public volunteer' was also discarded in favour of 'Public Adviser' to give more formal status to their role, and the Public Reference Panel was renamed the Adviser Forum to 'signal that it was an inclusive space for all Public Advisers' (int16-University-CC).

| Putting the policy in place-organizational processes and infrastructure for involvement
Two public contributors took a leading role in producing the initial policies to support and promote PI throughout the collaboration.

| Implementation: public involvement in practice
For PI to be mainstreamed, we would expect to find evidence of It's not a rubber stamping this sub-committee… and I know that we get quite a rough time sometimes and they do give projects back and they ask for more information and they ask for more public engagement.
They do point out gaps and sometimes things have been round more than once before they get through to the Steering Board.  …before it was very much right, we've developed a project let's go out to consultation, we will have a group of the public, we will consult with them and then we will take that information back and work on it. So, the PI is over and done in one meeting maybe.  (grp3-Adviser-CC) However, routine monitoring data revealed that four years in, PI was still not equally embedded throughout CLAHRC-NWC. Some project leads argued that the time required to recruit and support Advisers could be prohibitive, whilst other respondents felt where there was limited PI it largely reflected a lack of commitment by some senior staff rather than organizational barriers: Do people really believe that this [PI] will improve the quality and relevance of their research? My personal view is that, though they wouldn't admit it, they don't because if they did they would be doing it better. (int16-University-CC)

F I G U R E 2 Public involvement in CLAHRC-NWC
Additionally, it was suggested that CLAHRC-NWC's requirements were the only reason some projects had recruited Public Advisers, leading to tokenistic practices: It was so frustrating to know that all the work had been done and then to have to be really civil and polite. I think they just put you there then because you tick the box. (grp2-Adviser-CC)

| The quality of involvement: experience and influence
Alongside mainstreaming PI across the organization, CLAHRC-NWC aspired to deliver involvement that was experienced as positive and influential. The findings point to progress as well as continuing challenge within the 'quality' domains of the Involvement Cube.

| Valuing diverse knowledges and innovating involvement approaches
Staff across partner organizations reported a growing understanding of the value that Public Advisers could bring to their projects. One We work as a team with other members of the group.
That's important, that we all complement one another rather than just accepting the fact that we are there.   …we have been able to get more involvement of public members and that has been beneficial. It's been challenging but it's been beneficial as well. Challenging because of understanding the level that they are coming from, to adjust what we do and say to make sure that it's relevant and helpful to those individuals as well.

| Strengthening the public voice through support and communication
(int8-University-CC) However, language was sometimes experienced as excluding. One Adviser suggested that researchers should '…tone down the academic side, it frightens ordinary members' (grp8-Adviser-NR) whilst another stressed that 'communication is key and getting the messages right is key and I think sometimes the jargon that's used makes it really difficult'. (grp11-University-PPP). Additionally, poor communication and insufficient information had left some Advisers feeling their voice was weak and out of touch with their research project: Nothing has happened in the past 12 weeks, well not that I'm aware of. This is another frustration -it would be rewarding if they could drop an email and say well, we realise we've got you on hold you know, could you just be aware of this this and this. (grp14-Adviser-PPP)

| Strengthening the public voice by building skills and knowledge
There was a widespread view that members of the public needed access to development opportunities so that they could contribute fully to their chosen activities. As one Adviser highlighted: [Name] told us as an advisory panel we are going to present, so it's given us some ideas how to present.
We prepared a PowerPoint presentation, so it's given me as an Adviser some confidence to present my work, to disseminate our work as a team in front of all the [project] participants.
(grp15-Adviser-PPP) For some, skills development, combined with growing confidence had strengthened the Advisers' voice, enabling them to become more involved in project activities: One Adviser was saying that they wanted to see all the Adviser comments from [project] …I think that person wouldn't have made that comment a few years ago. I don't think they would have had the confidence, but I don't think they would necessarily have had the skills to be able to evaluate the Advisers' input.
(int2-NHS-CC) Skills development for professionals was also important.
Advisers, for example, contributed to training on construction of role descriptions which, as one university-based respondent suggested, increased researchers' understanding of 'added value and the skills' (grp16-University-CC) that Public Advisers could bring.

| The Adviser Forum-collective involvement and influence
Advisers' experience of participation in the Adviser Forum and its subgroups demonstrated the extent and influence of PI in the strategic governance of CLAHRC-NWC. As one Adviser noted: The Public Reference Panel [Adviser Forum] was actually developed by the public so we were part of discussing the fees, involved in the protocols and plans, the actual welcome packs; we were part of everything.

| D ISCUSS I ON
The research reported here aimed to explore the extent to which positive and influential PI had been embedded throughout a large research collaboration. Some potential limitations of the study should be noted. The research was conducted by internal teams but several steps were taken to reduce potential bias 29 : Only Popay was involved in CLAHRC-NWC governance or had any responsibility for implementing PI policy. She provided methodological advice and contributed to the interpretative process but was not involved in conduct of the research or data analysis. Additionally, team members did not interview people they had previously worked with; the initial coding frame was based on existing theory, which also informed the analysis; data extraction from a subset of transcripts was undertaken by two researchers; and the credibil- The collaboration, however, had not fully achieved their aspirations. In some themes and projects, there was limited PI and at times the experience was less positive. A key factor here was the lack of commitment from some senior staff, communicated to their colleagues, that time invested in PI was time well-spent.
The evaluation adopted a novel theoretical approach, combining two frameworks, one focused on mainstreaming (not used previously in this field) and the other on quality. It moved away from the project-based design prominent in the field and sought to represent perspectives from different 'constituent' groups across the organization. In this way, we hoped to contribute to the more reflective approach to evaluating PI in research that Boivin has recently argued will help produce a more 'nuanced' understanding of how benefits can be maximized and potential risks avoided. 30 Informed by the two frameworks, the findings illuminate the dynamic nature of PI throughout a research organization and the factors driving change at individual, project, programme and organizational levels. The mainstreaming framework focused the analysis on progress and challenges in three key organizational domains in which change was needed if PI was to become embedded across the collaboration. They also reveal that evidence of policy implementation does not necessarily mean that involvement is experienced as positive and influential by members of the public or staff.
The Involvement Cube provided a theoretical lens through which to assess these 'quality' dimensions in terms of public and professional knowledge exchange; diversity of involvement opportunities; strength of the public voice; and the collaboration's willingness to act on the public's input.

| CON CLUS ION
Examining the evolution of institutional culture alongside policies, infrastructure and procedures has not been a feature of much previous PI research which has often focused at the project level, sometimes looking outwards towards organizational processes that enable PI in projects but not at the vision, structures and processes of the organization as a whole. As Moser and Moser contend, having a clear institutional stance is vital for mainstreaming success, contributing to '…a long-term transformation process for the organization in terms of attitudes, 'culture', goals and procedures'. 16 Our findings suggest that these elements are all critical to making progress in embedding PI across research organizations but as an evaluative tool, this framework alone is not enough. Combining it with the Involvement Cube enabled a more nuanced and qualityfocused examination of PI across the CLAHRC-NWC, suggesting it is a useful combination of theoretical approaches to apply to future PI research.

ACK N OWLED G EM ENTS
The authors wish to thank the Public Advisers who contributed as part of our Evaluation Subgroup for their contributions and Dr Emma Halliday for her valuable comments.

CO N FLI C T O F I NTE R E S T S
The author(s) declared no potential conflicts of interest with respect to the research, authorship and/or publication of this article.

DATA AVA I L A B I L I T Y S TAT E M E N T
Due to confidentiality, and the nature of the consent obtained, the qualitative interview transcripts cannot be shared. For further information related to this data set, please contact the corresponding author.

S U PP O RTI N G I N FO R M ATI O N
Additional supporting information may be found online in the Supporting Information section.