Using the Public Involvement Impact Assessment Framework to assess the impact of public involvement in a mental health research context: A reflective case study

Abstract Background We assess the utility of the Public Involvement Impact Assessment Framework (PiiAF) as a resource to support research teams in assessing the impact of Public Involvement across diverse research and public involvement (PI) contexts. PiiAF was developed in response to a well‐documented growth in Public Involvement in health research in the United Kingdom that demands a more sophisticated evidence base to demonstrate its impact. Design We used a reflective case study approach drawing on contemporaneous meeting notes, PiiAF website resources and retrospective reflections to describe how PiiAF helped us to develop an impact assessment plan of the PI in a university‐based mental health research centre. Discussion We consider key aspects of our experiences of using PiiAF as a tool to help us design an impact assessment of PI, interpret these experiences with reference to relevant theory and research and share insights that may be useful to other teams considering using PiiAF. Conclusion These insights include understanding the commitment of time and effort required to develop effective PI impact assessment plans; the flexibility of PiiAF and its ability to be used in a range of research and PI contexts; and the advantages of involving all stakeholders (including the public) in the development of an PI assessment plan.

took place, PI is primarily focused on involving people with lived experience of mental health problems and their relatives.

| Background to the development of PiiAF
The growth of PI in health research in the United Kingdom in particular is well documented; however, some concerns about the sophistication of the evidence regarding its impact have been articulated. 2,3 As the field matures, debates around PI reflect both the complexity of the underpinning theoretical and conceptual issues along with the practical challenges of involving members of the public in health research contexts. There is broad agreement that the public have a right to be involved in research related to health conditions or issues related to them. [4][5][6] However, there are also advantages to developing an evidence base regarding the impact of PI on a range of potential outcomes. 1 Firstly, by exploring the relative effectiveness of different PI approaches, we can begin to understand the mechanisms by which PI can impact on research and consequently develop more effective involvement strategies, increasing the value of PI for both the public and researchers. Secondly, health research funders require detailed justification and costing for all elements of research including PI. Evidence to demonstrate the effectiveness and value for money of PI would strengthen the rationale for this funding.
Many models and frameworks exist to describe the different ways in which PI can happen [7][8][9] and to assess its quality. 10 Reviews of the PI literature have concluded that despite the limited nature of the evidence base, and lack of explicit conceptualizations of the PI underpinning the research, a wide variety of PI impacts can be identified. 2,11 But there are fewer attempts to demonstrate how the impact can be directly assessed (for exceptions see 2,3 ). PiiAF addresses some of these concerns. It seeks to support people to identify the types of impact they might reasonably expect for PI in specific research projects/programmes provides a new framework for assessing PI impacts and guidance on how to do this.
PiiAF is based on research carried out by a multidisciplinary team of researchers and members of the public across the United Kingdom, PiiAF conceptualizes PI in research as a complex social process and acknowledges the significance of the personal and organizational contexts in which it takes place as well as the difficulties associated with attributing specific impacts to PI in general or to specific aspects of the PI process. 1 Consequently, PiiAF promotes an explicit values-based approach to PI and its assessment that focuses amongst other things, on understanding the motivations for researchers and members of the public to engage in PI, which is considered important in increasing the likelihood of effective PI and positive outcomes.
PiiAF does not therefore provide a quick-fix tool or measure of PI impact, but rather supports research teams (including public advisers) through a developmental process, the aim of which is to design a plan to assess PI impact that will meet the needs of their specific project.
PiiAF was primarily intended to be used by research teams for PI impact assessment planning in the initial development of single research project funding proposals. The framework however was also designed to be flexible and to be applied to the range of different contexts in which PI in research happens. This could include, for example, plans to assess the impact of PI priority setting in programme grant applications.
Two members of the team that developed PiiAF, (FL and MC) along with 2 other colleagues wanted to test the utility of PiiAF in developing a plan to assess the impact of PI in a mental health research centre. In the next section, we briefly describe the structure of PiiAF and then go on to provide an overview of the research context in which we used PiiAF.

| The PiiAF framework
PiiAF is structured into 2 parts. Part 1 (see Figure 1) highlights 5 elements that research teams might need to consider when planning their PI impact assessment: values associated with PI in research, approaches to PI, research focus and study design, practical issues shaping PI in research and identifying feasible Impacts. A series of key issues and questions accompany each element, inviting research teams to engage in an assessment process.
The PiiAF and resources to support its use can be found at the following website. * Responses to the issues and questions raised in each element of phase 1 can be captured and recorded using the Record Card which is a key PiiAF resource and can be obtained from the website * ( Figure 2).
Once completed, the record card can be used in Part 2 of the PiiAF process (see Figure 3) which guides research teams through 4 steps to create an impact assessment plan. These include working out why you want to assess the impact of PI and who should be involved in this work; developing your intervention theory (ie, showing how your approach to PI will lead to the desired impacts); identifying how the potential context of your research might affect the impact *http://piiaf.org.uk/ of PI; and finally utilizing all these features to formulate an impact assessment plan including the design of the assessment.
In the next section, we describe the research context in which we used the PiiAF process discussed above.

| The Spectrum Centre for Mental Health Research
The Spectrum Centre for Mental Health Research was set up in 2008 at Lancaster University in the United Kingdom. At the time of the work reported here, the core funded team consisted of 2 professors, 3 lecturers, 2 part-time service user researchers (SUR) and a part-time administrator. There were also additional doctoral and post-doctoral fellows and research associates within the team who were funded externally. The Spectrum Centre focuses on developing and evaluating psychosocial approaches to supporting people with severe mental health problems and their support networks (see Box 1).

| Study aims
Our aim was to capture our experience of using PiiAF to develop a plan to assess the impact of the range of methods of involving users of mental health services in the research carried out in the Spectrum centre and to share key aspects of this experience, which we hope, will be of use to others wishing to use PiiAF. To achieve this, we used a Reflective case study method.

| Methods used for capturing and reporting on our experiences of using PiiAF
We convened a Steering Group (SG) to oversee the use of PiiAF in the Spectrum Centre, which included FL (Co-Director of Spectrum) and MC (Research Associate working in the same Division as Spectrum) who were both involved in the development of PiiAF, RL the SUR working in Spectrum mentioned earlier and finally AP, a Spectrum volunteer with a background in psychiatry and sociology who had no previous involvement with PiiAF. At initial meetings of the SG, we decided who should be involved in developing the PI impact assessment plan for the centre and familiarized ourselves with PiiAF and the processes involved in using it. The SG The following 4 sections of the paper draw on data recorded by the authors during the PiiAF process. Using the Marks-Maran & Rose critical reflection framework, we will (i) describe our experience of using the PiiAF; (ii) critically reflect on key moments in the PiiAF process; (iii) locate our experience of using PiiAF in the context of existing PI impact assessment and literature; and finally (iv) share some insights for others who may wish to use the PiiAF in future.

| Description of our application of the PiiAF Process
The PiiAF process has 2 parts. Below we describe the activities that we undertook during Part 1 and Part 2 of the process.

| Using Part 1 of PiiAF
The main tasks in Part 1 of PiiAF are to identify the range of issues that researchers could consider when planning their PI and thinking about its impact. The SG intended to capture the values of as many Spectrum Box 1 Public Involvement in the The Spectrum Centre The "public" involved in The Spectrum Centre work are primarily people with mental health problems (most commonly referred to as "service users") as well as members of their support networks including carers, relatives and peer groups. Frontline clinical staff and health service commissioners are also key stakeholders in The Spectrum Centre. The work reported here focused specifically on developing a plan to assess the impact of involvement of service users in the Spectrum Centre's work. were different from those who did not complete the tool, we are satisfied that we got a wide enough range of responses to make it a valuable exercise. Furthermore, the PiiAF process is intended to be iterative and flexible thus allowing the voices of those not heard during this initial process to be captured subsequently. It is likely that some of those who attended also filled in the audit tool, the workshop provided an opportunity to elaborate on these responses. The responses were analysed thematically by the SG, and the identified themes were captured on the record card. See Table 1.  3. PI impact on the interpretation of findings.

PI impact on the dissemination of findings.
Once the SG had identified the PI impacts that we were potentially interested in exploring and, prompted by the framework, we thought about how the Spectrum Centre approach to PI could lead to the desired impacts. This led us to specify a number of intervention theories (see Table 2 below). Once the intervention theory or pathway between our approach to PI, specific methods for involvement and the desired impacts were identified the two members of the SG who work for the Spectrum Centre were asked to consider how the context in which PI in research takes place might influence its process and impacts and to capture this in writing. The SG was asked to consider how the context in which PI in research takes place might influence its process and impacts and to capture this in writing.
This exploration of context and its influence on PI process and impact suggested that some key features of PI in the Spectrum Centre would be likely to lead to positive impacts. PI

is embedded within the ethos of the centre;
2. is integral to or considered throughout the research process; 3. happens in different ways through SURs, the Advisory Panel, the establishment of project specific Service User Reference Groups, online and through a variety of outreach work and events.
Some potential barriers to maintaining PI were also identified especially around ongoing funding of PI. In addition, there was some concern about the wider institutional context, which currently does not fund PI as a matter of routine and organizational procedures that can make remunerating PI difficult. Other structural challenges include a lack of formally recognized or supported career structures for SUR Box 2 Issues that were raised in the follow-up workshop that were additional to existing audit tool responses included

Values
Public involvement (PI) was seen as a way of giving a voice to service users; as a means of making research more relevant and acceptable and as a way of reducing stigma associated with bipolar disorder.

Approaches to PI
The importance of PI happening throughout the research process and using a range of different approaches was highlighted.
Research focus and study design Some aspects of the mental health research context were experienced as paternalistic, so the importance of PI in counteracting this was emphasized.

Practical issues
A range of practical issues including financial resources, the national research policy context and pathways of engagement with the Spectrum Centre were identified.

Identifying impacts
There were mixed messages about whether PI impacts had been assessed. One key impact that emerged was the importance of foregrounding lived experience of bipolar in research.
or personal development strategies. It could be argued, however, that awareness of these potential barriers means that the centre can be an advocate for change in institutional processes and attempt to mitigate in a limited way the potential challenges faced by members of the public engaging in PI.
Phase 4: Formulating the assessment questions and designing the assessment Phase 4 focuses upon identifying the specific impact assessment questions to be addressed and identifying the data that could be collected. The PiiAF guidance provides some tables that can help with this that are illustrated below. Table 3 contains the impact assessment questions we generated using a resource provided with the PiiAF guidance. 1 Next, the PiiAF process invites research teams to consider how the intervention theory and research questions can inform decisions around impact assessment design and data collection. Table 4 below shows how we used the intervention theory and impact assessment plan and the impact assessment questions from Tables 2 and 3 to identify indicators, measures and sources of data.
As part of the development of PiiAF, a database of studies with detailed descriptions of their methods and the tools that were used to assess the impact of PI was produced. † Using this searchable database, we were able to identify a number of existing studies that had explored some of the impacts we were interested in assessing. [18][19][20][21][22] Being signposted to these studies helped us to think about how we could develop an impact assessment that engaged with the existing evidence rather than starting from scratch.

| Critical Reflection on Key Moments of our use of PiiAF
In the section above, we have provided a detailed description of how we used PiiAF, we will devote the remainder of the paper to a critical reflection on this experience to identify learning that might be helpful to others who are considering using PiiAF. The following sections are structured according to the key moments in the PiiAF assessment process identified in the critical reflection template completed by the 4 members of the SG.

| Audit Tool
Use of the audit tool developed by the SG and our subsequent identification of themes from the responses received from centre staff and members of the public involved in the centre's research were generally considered to have been useful. SG members could see that something tangible was happening and we could begin to understand how PiiAF would work. There was some discussion during the development of the audit tool about whether we should provide example answers as a guide. Our concern that they may bias the kind of responses we received led them to be omitted, but the potential benefits of helping respondents understand the task were noted.

| The workshop
Although the workshop was generally thought to have gone well and achieved its aim of gaining a greater understanding of the rather telegraphic data provided by the audit tool, there were challenges in running this. Public involvement is an emotive subject, and some of the participants (mental health service users) had powerful stories to tell, including at times, their own traumatic life experiences that led them to be in the position of being a service user. At times, the discussion was shaped by the need for people to share their experiences in detail, rather than ad- we felt that it was important to give people time and space to discuss or air any strong feelings evoked by the topic. It would have been inappropriate to close down such discussions to focus exclusively on the questions to be answered. We do not see it as problematic that such discussions took place, but rather consider it to have been part of the process.

| Updating the record card
The record card which we used to record the themes identified from the centre staff and members of the public involved in the audit process and the workshop (described above) worked well as a physical prompt

| Working through Part 2
All phases of Part 2 were felt to make sense in their own right, but there was some concern about how Part 1 fitted with Part 2 and as a consequence we did not necessarily follow the order of the framework as set out in the guidance. However, the PiiAF guidance says that it can be used in any order and iteratively, so this was not a problem.
For example, we discussed the purpose of the impact assessment and who should be involved (key elements of Part 2) during early meetings of the SG before we had even begun to engage with Part 1 of the framework. During Part 2, in particular, we felt that some of the terminology and concepts did not feel user-friendly, for example, some SG members were unsure about what an intervention theory was. We identified a larger number of potential impacts to assess than we anticipated, and to a certain extent, this overwhelmed us. We also became distracted by thinking about how we might actually put the impact assessment plans into practice and tried to identify potential sources of funding. However, as we began to complete the tables to identify the assessment questions and data collection methods, the process began to feel more tangible and got back on track.

| Overall reflection on process
We found the process of using PiiAF useful. Part  amongst members of the centre, encouraging us to identify how the context in which we were working enabled or constrained PI and making us aware of the wide range of potential PI impacts that could be assessed. Part 2 felt less tangible and user-friendly initially and required a deeper engagement although we felt more confident with the process when we began to populate the tables to identify the intervention theory and develop the impact assessment plan.
However, as the work involved in Part 2 requires some research expertise, it is perhaps not surprising that it is more challenging than the general discussions triggered by Part 1.

| Discussion of our experiences of using PiiAF in the context of previous research
PI in research is characterized as a complex social activity. [1][2][3] Attempting to develop an impact assessment plan for PI in a research centre rather than an individual research project adds an additional layer of complex and time-intensive social activity-groups with a pre-existing commitment to, and valuing of, genuine and effective PI in research will perhaps be more willing to make this time available.
This might account for the well-documented difficulties associated with developing a sophisticated evidence base. 2,3,9,23 As PI in research is so firmly located within multiple interpersonal, social, research and cultural contexts (eg, that of members of public involved, the researchers, the wider community, funders) it is unsurprising that there have been calls to make greater links between process and context variables and PI impacts. 2,3 The use of the PiiAF record card in Part 1 to capture key points from the discussions of the contextual complexity and the development of an intervention theory in Part 2 (although difficult conceptually) that made explicit our thinking about how the PI process would lead to the hoped-for impacts supported us in making exactly these kinds of links between process and context and PI impact. In our experience, PiiAF represents a way of negotiating the complexity without making the process of assessing impact so simple that it lacks validity or sensitivity to the specific context in which it is being used.
By not being prescriptive about the kind of evaluation design that research teams should use when developing an impact assessment plan, PiiAF supports the development of an elaborated understanding of the variables that will make positive impacts of PI in research more likely.
The backgrounds of the members of the SG variously included service user research, clinical research and social constructionist qualitative research, and we feel that by using PiiAF, we would have been able to develop a plan that would have been acceptable within any of these paradigms.

| Future action-Some Insights from using PiiAF
Research teams will find their own ways of using PiiAF that will be contingent upon the particular context of their PI and their research, their expertise, resources and time availability. However, there are some things that we have learnt from using PiiAF that other research teams might find useful and we set out 10 insights below. It is important to note that these are based on one case example of our experience of using PiiAF. They are offered in the hope they will stimulate further exploration of the process of developing ways to assess the impact of PI. They do not necessarily reflect the views of the team that developed PiiAF.

1.
PiiAF is a complex process rather than a quick-fix measure, but this allows greater flexibility and is more likely to lead to effective impact assessment plans.

2.
The context in which we used PiiAF was important. The Spectrum Centre already had PI systems and processes in place that we could draw on to implement the stages of using PiiAF. Research teams that do not have this will need to consider how they will support PI and involve members of the public in the planning and/ or implementation of an impact assessment. It is possible to use PiiAF without a supporting infrastructure for PI in place, but the process is likely to take longer whilst time is taken for recruitment of members of the public and protocols for working are established.

3.
We had decided at an early stage that PI in the development of

4.
We did not always do the tasks of PiiAF in the order presented, but the guidance does clearly state that users can use it in any order and iteratively. We did tasks as they cropped up in our discussions or when it made sense to look at them. For example, we had discussions about the purpose of the impact assessment and who would be involved in the first meeting of the SG, but the PiiAF guidance suggests that this discussion would be best taking place at the beginning of Phase 2. This did not seem to cause any problems in the process, and we suggest that research teams use PiiAF flexibly as indeed it was designed to be used.

5.
Filling in the record card with key points arising from discussions prompted by the PiiAF questions was a helpful process in and of itself. Research teams that are interested in improving the quality of their PI without necessarily assessing its impact might find it useful to go through Part 1 along with the PiiAF Draft Standards for Good Practice ‡ in PI resource on the PiiAF website.

6.
We found it quite a difficult process to use the record card in Part 2 and would in retrospect like to have taken more time to think about how our learning from Part 1 could be linked to the challenges we had to address in Part 2. Part of the challenge was that we generated a lot of ideas about the kinds of impacts we would like to explore. This may not be a consequence of using PiiAF and may have happened in any focused discussion of likely impacts.
Given the subsequent PiiAF stages of developing intervention theories and impact assessment plans, we went through a process of narrowing our focus on a small number of impacts, but it would be helpful to have some practical guidance about how to prioritize ideas. As we were not in a position to implement an impact assessment plan, reducing the number of impacts to be assessed was a hypothetical question for us. We included most of the impacts that we identified in the figures presented earlier to demonstrate the different kinds of impacts that could be assessed to maximize the utility of this paper as a learning resource. The small amount of narrowing down we did was informed by the values and impacts that we had made explicit in Part 1 of the process of using PiiAF. We suggest that research teams factor in time to think about how they might identify which impact assessment ideas they would like to take forward.

7.
We struggled with intervention theory both conceptually and practically and seemed to spend a lot of time on this. PiiAF understands this difficulty and that it requires research expertise and provides some resources to support this. After going through a process of reflection, we recognize that the intervention theory is potentially crucial to making explicit the context and involvement process and their links with impacts. Although perhaps not immediately intuitive, it is worth persevering with this step in the process to ensure that these links are made and also to check that the PI approach is appropriate to the desired impacts.

8.
During development of the impact assessment plan, we encountered problems conceiving plans that would allow us to attribute impact to PI (rather than anything else) especially given the type of research carried out in The Spectrum Centre which tends to be multidisciplinary, pragmatic research employing a wide range of methodologies. This, as already discussed, is a challenge common to PI assessment and was made even more challenging because The Spectrum Centre already has an established programme of PI activity, making it difficult to identify ways to compare research processes that did involve PI with those that did not. Whilst PiiAF did not provide any easy answers to this, it helped us to identify some of the challenges.

9.
PiiAF was developed by a multidisciplinary team with a strong social science background and is primarily aimed at research teams that include professional researchers. All of the members of our team struggled to understand some of the terminology, for example, as discussed "intervention theory," "normative values." Therefore, from the outset, it is helpful to establish a shared understanding of some of the key words and concepts in PiiAF and encourage people to ask for clarification regarding any terms that are not familiar to them.

| CON CLUS ION
The aim of this paper was to provide a case example of using PiiAF to develop an impact assessment plan for PI in a mental health research centre. We chose to use a reflective case study so that we could share key aspects of our experience and the practical steps involved.
We have shared what we learnt from this process in the hope this may be of use to other teams considering the use of PiiAF and also those who are unfamiliar with it or had not thought about using it.
The insights gleaned from this process presented above need to be considered alongside the limitations of our study design.
Firstly, we were using PiiAF in a different way than was originally intended-in the context of a mental health research centre considering its existing PI, rather than during the planning stage of a single research project or programme. We feel that the flexibility built into PiiAF meant that we were able to apply it, albeit with some minor adjustments in this context. Secondly, our SG contained people who were already familiar with PiiAF so we perhaps did not use the Piiaf guidance as fully as we might. However, we think the guidance will be invaluable to those who are less familiar with the developing several PI impact assessment plans. Our next step is to implement these plans and evaluate the impact of PI in our work.

ACK N OWLED G EM ENTS
We would like to acknowledge the work of the PiiAF Study Group in developing the PiiAF and the contribution made by members of the study's Public Advisory Group and Advisory Network. We would also like to thank Tamara Rakic for her comments on an earlier version of this paper.

CO N FLI C T O F I NTE R E S T
Three of the authors (MC, JP and FL) were part of the original research team who developed the PiiAF.

APPENDIX 1
Initial protocol discussed at first meeting of the Steering Group for using PiiAF within The Spectrum Centre How will the activities associated with these 2 parts be achieved • Part 2: What will the impact assessment plan look like and who will develop it?
We anticipate that: • Part 1: To collect information about public involvement (PI) values, anticipated impacts, PI approach we will use regular steering group discussions, including mostly face-to-face but possibly telephone. We will liaise with members of the centre to obtain information. Frequency of SG meetings to be 3 or 4 weekly initially but will adjust according to the demands of the project • Part 2: To develop the impact assessment plan. Work will be focused within the SG-assess the need for frequency of meetings How will the impact assessment plan be implemented? To implement the assessment plan, we will need to identify sources of funding. The feasibility of this will be assessed as the project proceeds Will we evaluate the PiiAF-based impact assessment? This will depend on obtaining funding (above) Who will be involved (small group, whole team?) • Developing the impact assessment plan • Carrying out the impact assessment • Evaluating the impact assessment?
Initially, the SG will be involved in pulling together the information needed to develop an impact assessment plan. Part 1 will involve centre members of staff and service users in providing their responses to. Developing the impact assessment plan (Part 2 of PiiAF process) will involve SG members only. Carrying out/evaluation impact assessment will depend upon obtaining funding How long will it take? This is hard to gauge. There is no funding for this project, so depends upon the availability of SG members to both attend meetings and undertake pieces of work outside of the meetings. Aim for 6 mo initially but review as the project progresses Some involvement is informed by peoples' experiences of mental health professionals and the mental health system. Perception that mental health system can be infantilizing or paternalistic-important that involvement counteracts this Mental health system/professionals also influence recruitment to studies and can act as gatekeepers. In turn, this shapes the impact that PI can have in that a SUR representing the study makes it more credible than if researchers represent it (Continues)