Implementing public involvement throughout the research process—Experience and learning from the GPs in EDs study

Abstract Background Public involvement in health services research is encouraged. Descriptions of public involvement across the whole research cycle of a major study are uncommon and its effects on research conduct are poorly understood. Aim This study aimed to describe how we implemented public involvement, reflect on process and effects in a large‐scale multi‐site research study and present learning for future involvement practice. Method We recorded public involvement roles and activities throughout the study and compared these to our original public involvement plan included in our project proposal. We held a group interview with study co‐applicants to explore their experiences, transcribed the recorded discussion and conducted thematic analysis. We synthesized the findings to develop recommendations for future practice. Results Public contributors' activities went beyond strategic study planning and management to include active involvement in data collection, analysis and dissemination. They attended management, scrutiny, planning and task meetings. They also facilitated public involvement through annual planning and review sessions, conducted a Public Involvement audit and coordinated public and patient input to stakeholder discussions at key study stages. Group interview respondents said that involvement exceeded their expectations. They identified effects such as changes to patient recruitment, terminology clarification and extra dissemination activities. They identified factors enabling effective involvement including team and leader commitment, named support contact, building relationships and demonstrating equality and public contributors being confident to challenge and flexible to meet researchers' timescales and work patterns. There were challenges matching resources to roles and questions about the risk of over‐professionalizing public contributors. Conclusion We extended our planned approach to public involvement and identified benefits to the research process that were both specific and general. We identified good practice to support effective public involvement in health services research that study teams should consider in planning and undertaking research. Public Contribution This paper was co‐conceived, co‐planned and co‐authored by public contributors to contribute research evidence, based on their experiences of active involvement in the design, implementation and dissemination of a major health services research study.


| BACKGROUND
Public involvement across the research cycle is encouraged to improve the relevance, quality and accountability of health and care research. [1][2][3] Many research funders expect evidence of involvement in developing study proposals and in planning how involvement will occur throughout a funded study. 2,4,5 However, there continue to be barriers to developing and undertaking impactful research with patients and public members. [6][7][8][9] Conceptual models of public involvement in research highlight the different levels of involvement from none to full control of research. 10,11 Involvement and partnership require power to be shared within research teams. 12 Some public contributors report negotiating their influence by using their experience, leading to queries of professionalization. 13 Others argue that communication skills sensitize research teams to patient experiences, a role that goes beyond simply presenting patients' encounters with health care services. 14 To address concerns about the quality and effectiveness of public involvement processes, several guidelines, standards and value statements have been published, most prominent of which is the UK Standards for Public Involvement. [15][16][17] Researchers and policy experts also recommend that public involvement in research is thoroughly assessed, to build an evidence base and inform future practice. 15,18 The GRIPP 2 reporting checklist is one tool for ensuring that public involvement is consistently described in published papers requiring, as a minimum, a summary of public involvement activity. 19 The checklist does not consider the quality or effects of reported involvement. Detailed accounts of how public involvement is undertaken within research studies remain limited 20 and usually focus on specific involvement roles. 18,[21][22][23][24] Descriptions of public involvement across the whole research cycle of a major research study are uncommon. 25 More evidence about the context, process and effects of public involvement in research is needed. 21,26 There are divergent views over the value of, and approaches to, assessing effects of public involvement, [27][28][29] including defining core features, activities and mechanisms 30,31 and encouraging feedback and mutual learning. 32,33 We undertook a large research study (realist evaluation) involving three phases of data collection and analysis over 4 years.
Our study evaluated models of care in which general practitioners work in or alongside Emergency Departments at UK hospitals. 34 Our successful application to the National Institute of Health Research (NIHR) Health Service and Delivery Research (HS&DR) programme (project 15/145/04) included a description of proposed public involvement in delivering the study. Our objectives for public involvement were to ensure that our research remained relevant to health service users by including public and patient experiences and perspectives throughout research planning, implementation and dissemination. We also determined that our research should be acceptable, understandable and feasible, to be of the highest quality.
These objectives informed our plan for public involvement. This plan was developed with two public co-applicants who also had extensive input throughout planning the research design.
In this paper, we describe how we involved public contributors in our study. We consider how this involvement evolved from the way we envisaged at the study outset, the experiences of all co-applicants collaborating in the research and present lessons for future practice. 19 This first-hand account of public involvement 22

| Initial public involvement approach
Before planning our research, we recruited two public contributors They shared responsibility for authoring plans for public involvement in the study, including reviewing the proposed budget. They drafted the plain English summary. They were named research co-applicants on the outline and full applications.
In our study, we proposed the following approach to public involvement. We committed to continuing our relationship with the two public contributors we named as co-applicants on the study. We intended that they would maintain an equal status among all coapplicants within the research management team with full opportunity to contribute to strategic and operational decisions throughout the research. We also proposed they would join subgroups to plan data collection, review early findings, co-plan stakeholder events and lead the public strand of our dissemination activities. At their suggestion, we expected them to host pre-meetings to brief and support public and patient members attending our two stakeholder events.
In addition, we planned to recruit two different individuals through the HCRW Public Involvement Community 35 to join our Steering Committee as public contributors and providing independent oversight of our study. Finally, we intended to recruit seven public and patient members to attend two stakeholder events alongside equal numbers of other stakeholder groups (Table 1).
We calculated a budget (0.5% of total research bid) to provide honoraria and reimburse all expenses. We named an academic co-

| Data collection
To understand how we implemented or diverged from our initial public involvement plan, team members made comprehensive notes throughout our study. We documented public involvement activities in our meeting minutes and any reported effects on study processes.
We made a timeline of public involvement activities in our study by compiling a table recording the activity and effects.
To understand experiences and effect of public involvement in the study, we held a group interview (Supporting Information: Appendix 1) with co-applicants attending a regular (online) study management meeting during the last 6 months of the funded study H. responded to questions and comments at the end of the interview.
With participants' consent, the discussion was audio-recorded and transcribed using a facility within Zoom software. We encouraged interview respondents to be frank in their answers and invited them to send comments by email to a third party if they felt uncomfortable at any point. Through discussion, we generated themes and interpretation, taking a critical stance to test and confirm findings. [39][40][41] We then brought together all data about implementation processes, experiences and effects to consider what we had learnt about involvement in our study. We identified what contributed to effective involvement and prepared recommendations for implementing public involvement, based on our findings.  We convened five forums and ensured public involvement at all stages (Table 1). Effects on delivering the research In many instances, the contributions made by public contributors were in team meetings where their views were sought and heard, playing a part in debate and collective decision-making. In these situations, it was not generally possible to determine whose opinion or contribution caused, or was the consequence of, which decision. In other instances, we could describe the role and outcomes of public involvement in the study. In

Role
Public contributors played an active role in the process of planning and implementing public involvement throughout the study. They decided how the involvement processes, summarized in the research application, were undertaken in practice. They held discussions at Public Involvement team meetings and undertook an annual Public Involvement review that initiated additional activities to extend the role of public contributors and strengthen the practice.

Effects on processes of public involvement
Public involvement in the study was more extensive than proposed in the research proposal and funding application. The Funder described it as 'a fine example of embedding public and patient involvement (PPI) throughout the course of the research' after receiving routine study reports. Activities that were extra to those proposed in the original study proposal included: 1. planning and managing PPI input through annual review meetings and regular contact between the PPI coordinator and public contributors, 2. collecting data on effects of public involvement during the study, 3. conducting an audit of public involvement, against the UK standards for public involvement and 15 4. monthly research updates from the core team, at the request of public contributors, to keep wider team members abreast of study progress.
In we were able to make suggestions, and they were welcomed and incorporated, where possible, and we were given real and not token roles. (PPI 01)

| What made it work?
Respondents identified factors that enabled perceived effective public involvement in the study: 1. Commitment across the co-applicant team to public involvement, led by the Chief Investigator.

Having a university staff member to lead public involvement and
to support individuals in their role.
3. Providing public contributors with support and training in the skills required to undertake the role.
4. Including public contributors as co-applicants, equal at meetings and visible alongside other team members, which made them integral to conducting and delivering the research.

5.
Having regular face-to-face meetings (and online from 2020), building up a relationship within the team so that people felt they could work together. 6. Public contributors being able to work to researcher deadlines, flexible and prompt in responding to requests for input.
7. Public contributors being confident enough to articulate their views including challenging other team members.

| Challenges
Managing study resources within budget and timescales while also meeting rising interest in and expectations of public involvement was a challenge for researchers. Tight deadlines, staff availability and fixed costs meant that research staff struggled to work with public contributors as frequently as, or when, they wished.

| Summary of findings
In this study, we had a structured approach to involving public contributors that enabled them to have roles in project delivery and dissemination and in how public involvement was undertaken. Public contributors were actively involved throughout the study, taking on more research roles than was envisaged when the study was designed. This enabled changes to study processes, such as patient recruitment and dissemination. It made research co-applicants more aware of patient views and outcomes when reporting results. They identified factors that facilitated this positive public involvement but also challenges when expectations exceeded resources.

| Strengths and limitations
Data capture and analysis of public involvement in this study were led by the public contributors. This insider approach has been used by researchers and public contributors in other studies describing and reflecting on experience of public involvement in research [49][50][51] and is encouraged for generating informed assessment of complex factors. 15,16,19,[52][53][54][55][56] Nevertheless, we sought to minimize the risk of bias from potential conflict of interest, while capturing relevant data to address our study aim, in line with existing evidence. Our study records provided prospective data to capture real-time information and minimize recall bias. We used a public involvement effects framework to structure data collection and analysis. 37 We encouraged interview respondents to be frank in their answers and invited them to send comments by email to a third party if they felt uncomfortable expressing views verbally. One respondent did challenge the meeting consensus on some occasions and others did identify challenges. No emailed views were received.

| Comparison with the existing literature
Best research practice is to involve public contributors in developing research from the earliest feasible stage and plan their involvement in the funded study. 57 We believed that we had a robust and coherent plan that avoided tokenism. 58 Co-applicants believed that this Despite the increase in public involvement, there continue to be few reports of such integrated involvement in decision-making throughout a research study. 59,60 This placed more pressure on team resources to respond to additional involvement, despite having a well-costed proposal including a PPI budget. Funders need to acknowledge the cost of genuine public involvement and accept that these may result in bigger research budgets. 15,36 Otherwise, research teams risk cutting other costs that are necessary to delivery of high-quality and timely research, to appear competitive in the contested research application process, risking tokenistic involvement.
Extending public involvement in our study generated questions over whether these public contributors risked being too integrated within the research team and losing their outsider status. The notion that public contributors become too professional has been reported previously, but this view is not shared by the individuals who become involved in research. [61][62][63] Researchers and user-researchers report that experience differs from professionalism; the role requires skills that develop with practice and can be passed between contributors. [64][65][66] That this interpretation persists may suggest that trust continues to be shaky among the research community and that individuals remain uncertain about the process and robustness of involvement. It may also reflect the practical challenges of delivering work to deadlines and with competing priorities, so that the requirement to involve public members and respond to their views becomes too demanding. 63 The public contributors in our study said that they remained very conscious of their different skills and perspectives, including their health and care experiences. 67 There is a long-standing debate on who should be a public contributor. 68 The current emphasis on including less-heard voices overlaps these issues.
People new to public involvement must gain enough knowledge and skills to make their voices heard. Power is recognized to rest with the research community and with it the responsibility to empower people and facilitate communication and interaction. 61,63 To overcome the competing demands on researchers, a Public Involvement Coordinator within a team can support patients and public members to be effective in their role. 15 We included someone in this role within our team from the outset of project Our account is a rare example of the story of public involvement in a research study that describes the actual activities that our contributors undertook. 26,63 With systematic record keeping and comprehensive meeting minutes, we were able to record specific changes because public contributors were involved in our study. We showed that they affected patient recruitment, interpretation and dissemination of findings. They also affected the context for discussions, providing new information or reassurance to support decision-making and changing the awareness of research, clinical and policy study members so that the study gained a stronger focus on patient outcomes as it progressed. 70 The interrelationship between tangible and intangible effects suggests the importance of integrating public members in a team so that they are part of ongoing discussions and decisions. 61 The challenges of recording effects are well reported, 49,50 but can be overcome.

| Implications for practice
Our study shows that there are some actions that support effective public involvement in health services research. Based on lessons from our experience, we make the following recommendations to support research teams to plan and undertake public involvement in largescale research studies. These recommendations may also be helpful for research teams when evaluating processes and effects of public involvement.
1. Ensure that public involvement is embedded from the earliest stage to allow influence in project design and implementation and to demonstrate equality within partnership from the outset.
2. Agree a team commitment to public involvement and document this through meeting documents and minutes.
3. Appoint a Public Involvement Coordinator, funded by the study budget, with the skills and responsibility to lead public involvement and support public individuals in their role.
4. Provide access to relevant and timely support and training so that public members have the skills and confidence to undertake their role.
5. Integrate public involvement in study delivery by including public members as equal members within research management structures and processes.
6. Make public involvement a standing item on meeting agendas to ensure that public contributors have an opportunity to raise issues above contributions made to other discussions.
7. Build a relationship within the research team to establish trust for collaborative working.
8. Clarify structures and ways of working so that public members and researchers have clear and realistic roles and expectations.
9. Review public involvement structures and processes during the study so that new opportunities can be identified and taken and additional support needs can be met. 10. Instigate simple data-gathering of public involvement processes and outcomes and report these so that experiences can inform the emerging evidence base.

| CONCLUSION
Public involvement in health services research is growing, but remains inconsistent and poorly reported. Using the skills, commitment and resources in our team, we extended our planned approach to public involvement in our research study. We identified benefits to the research process that were both specific and general. These led to changes in patient recruitment, data interpretation and dissemination, but also refocused the study to include more information on patient outcomes. Study co-applicants reported positive experiences but also identified challenges, including matching rising expectations to available study resources. We identified good practice to support effective public involvement in health services research that study teams should consider in planning and undertaking research. | 2481