Reflections, impact and recommendations of a co‐produced qualitative study with young people who have experience of mental health difficulties

Abstract Background There is limited evidence of genuine equal partnership where power is shared with young people with mental health difficulties throughout all research stages, particularly in data collection and analysis. Objective To describe how our qualitative study, exploring young peoples’ perceptions on the feasibility of using technology to detect mental health deterioration, was co‐produced using principles of co‐production, whilst reflecting on impact, challenges and recommendations. Methods Young people with experience of mental health difficulties were appointed and then worked with researchers throughout all research stages. The study was evaluated against the five principles of co‐production. Reflections from researchers and young people were collected throughout. Results Seven young people formed an initial Young People's Advisory Group (YPAG); three became co‐researchers. Reflection was key throughout the process. Sharing power became easier and more evident as trust, confidence and mutual respect grew over time, particularly after a safe space was established. The safe space was crucial for open discussions, and our WhatsApp group enabled continual communication, support and shared decision‐making. The resulting co‐produced topic guide, coding framework, thematic map, papers and presentations demonstrated significant impact. Conclusions To our knowledge, this is the first qualitative mental health study to be co‐produced using the principles of co‐production. Our rigorous assessment can be utilized as an informative document to help others to produce meaningful co‐produced future research. Although co‐production takes time, it makes significant impact to the research, researchers and co‐researchers. Flexible funding for spontaneous suggestions from co‐researchers and more time for interview training is recommended.


| BACKG ROU N D
Patient and public involvement (PPI) is defined as research with and by patients, rather than to, for or about them. 1 In recent years, there has been a significant shift in PPI in research, moving away from tokenism, towards more meaningful involvement and co-production. 2 This has been facilitated by changes in research infrastructure and the growing patient voice. Increasingly, funding bodies have made it compulsory for prospective applicants to declare how patients, carers or the public have and will be involved in the proposed research. Likewise, some journals also require researchers to add a PPI statement 3,4 to their manuscript.
This has led to research organizations creating dedicated PPI resources within their institutions.
Patients, carers and members of the public can be co-researchers and equal partners throughout the whole research process: ideas generation, design, applying for funding, ethics, management, data collection, analysis, evaluation and dissemination. 5 Some peer-reviewed papers have discussed the benefits, costs and learning of co-production, 6,7 and some larger studies have found positive impact on findings (eg shared decision-making in mental health services). 8,9 However, most studies have not been evaluated against NIHR INVOLVE's principles of co-production. There are five main principles: sharing power, including all perspectives and skills, respecting and valuing the knowledge of all those working together, reciprocity, and building and maintaining relationships. 10 Critically, whilst these principles help guide our work, they do not tell us how to co-produce research, the main challenges that might arise, or how to overcome these challenges. 11 INVOLVE have recently produced two reports that showcase selected studies against the principles, and on how to co-produce research. 12,13 However, none are focused on young people with experience of mental health difficulties. Indeed, there is limited evidence of genuine equal partnership where power is shared throughout project stages in mental health research.
The systematic evaluation of PPI started almost ten years ago, 14 but it is only recently that capturing the impact of involvement has been formalized. For example, The BMJ and the National Institute for Health Research (NIHR) now suggest researchers use the GRIPP2 checklist 15 to report on PPI. Numerous papers have shown value in involving people from differing population groups, through various research stages and in different research designs. [16][17][18][19] However, there is an ongoing debate in the field as to if and how the impact of PPI should be captured or measured. Many of the existing tools are fairly prescriptive and lead to more quantitative measurements of who and how people were involved, rather than qualitative reflection (from all parties), on what has changed due to the involvement. PPI is ultimately about an interaction between researchers and patients or members of the public, which is dynamic, context specific and dependent upon many uncontrollable factors. 20 There is also a significant research gap in working with young people throughout all stages of a mental health research project and evaluating this involvement. We recently conducted a qualitative study that explored the acceptability and feasibility of technologies to detect mental health deterioration in young people with current mental health difficulties, where young people were research partners. 21 This research topic was informed by the McPin Foundation Priority Setting Partnership (PSP) on children and young people's mental health, in partnership with James Lind Alliance (JLA). 22 The PSP was rigorous process where children, young people, teachers, parents, mental health professionals and researchers came up with and prioritized research questions. Our study was therefore co-produced from start to end, ensuring it was appropriately designed, delivered and impactful. 23 The aim of this article is to: (a) describe how our study was co-produced following the principles of co-production; and (b) report on reflections that capture the impact on the research, researchers and co-researchers. The main challenges that arose and advice on how to overcome them in similarly co-produced studies are also described. This paper was reported using the GRIPP2 checklist (Appendix 1). 15

| Appointment of young co-researchers
The opportunity to work on our project 21

| Design and ethics
Initially, the YPAG reviewed the research question and protocol and co-created the interview topic guide, information sheet and consent form ( Figure 1) through two face-to-face meetings and further email exchanges.

| Roles of involvement
The seven YPAG members were asked to choose which area of the research they would next like to be involved in (a) data collection and/or data analysis, (b) management, and (c) dissemination. All YPAG members were assigned the roles they requested-two were assigned to management, three to data collection and analysis and two to dissemination. However, three YPAG members dropped out from management and data collection and analysis, before roles commenced due to deteriorating health and potential breach of the expense policy. This breach is wider than the project and for legal reasons cannot be discussed in this paper.
All roles were re-advertised but just one new person applied (resulting in five YPAG members) and was appointed two months after the project started. This person was involved in management, data collection, analysis and dissemination. The three young people assigned to data collection and analysis became embedded in the research project and were subsequently involved in management and dissemination and as such, described as co-researchers ( Figure 1).

| Training
Three co-researchers were trained in conducting interviews with people with mental health difficulties and coding interview transcripts. The training involved two, one-day (9:30 am-5 pm), face-toface sessions. Co-researchers were paid for their time (£150 each per session). Disclosure and Barring Check (DBS) were sought and approved for all co-researchers. A full breakdown of the training process is provided in Appendix 2.

| Data collection
The team co-produced a working model that described the best approach for co-researchers to interview participants in this study. As a team we decided on a three-stage approach: (a) co-researcher to shadow the lead researcher (LD) (with the participant's consent); (b) co-researcher to then conduct the interview with LD present, and (c) co-researcher to conduct the interview alone. Each co-researcher was debriefed separately about what went well and what could be improved from both the perspective of the lead researcher and the co-researchers, and there was time for questions. The co-researcher and researcher were present in six interviews. Co-researchers shadowed two, conducted three with the researcher present and conducted one alone, with the researcher in the next room for support if needed. A psychiatrist was available on site for safeguarding support throughout.

| Analysis
Unfortunately, one of the co-researchers informed the lead researcher that they could no longer be involved in the remainder of the project because of personal reasons. Two co-researchers (CC and JJ) continued and analysed three transcripts each which were also coded by two researchers (LD and ML); the lead researcher (LD) was contactable for support. The co-researchers coded their associated interview transcripts when possible. We co-produced a thematic coding framework during a 2-hour virtual meeting (Skype) which we then used to analyse the rest of the transcripts. We then co-produced a thematic map over a further two virtual meetings: (a) the researchers wrote the themes and sub-themes on a white board and we discussed these as a group; (b) all suggestions from co-researchers were added to the board; (c) we again discussed these as a group; (d) we added, refined or removed themes/sub-themes/links; and finally; (e) reached consensus on final themes and sub-themes.

| Dissemination
Four presentations (two posters and two oral) were co-presented at national and international conferences. [25][26][27][28] The lead researcher (LD) produced presentation drafts and co-researchers made comments and changes. This approach was utilized because of the limited amount of money dedicated to this level of dissemination involvement (four outputs) at grant projection and study planning stage. The co-researchers were also in employment or education, which made it more difficult for co-researchers to complete the first drafts which required substantial time commitment. We had a further 2-hour meeting to decide together who would present each section and to practice the oral presentation. Both co-researchers (CC and JJ) and the young person originally assigned to dissemination (KP), also contributed to two papers (the current paper and the main qualitative paper) as co-authors. The two co-researchers (CK and JJ) also reflected on, and evaluated, each stage of involvement and the support given during the project using questionnaires, free-text responses and face-to-face discussion.

| Principles of co-production
Our study is described against the principles of co-production (Box 1) across the research stages, from design to dissemination.

| Sharing of power
Trust and rapport were developed between the members of the team by taking time early in the research process to get to know each other better. The researchers and PPI lead began by sharing their own experiences of mental health difficulties to help create a safe space. The co-researchers then shared their own experiences.
Moreover, our shared values and complementary working approaches (eg hard working, passionate for the subject, active listeners, empathetic) helped develop this rapport.
Sharing power and decision-making between the young co-researchers and lead researcher became more substantial after each research stage, as our trust and confidence in each other increased; co-researchers' ideas were listened to and acted on. For example, the young people took the lead in developing the topic guide questions.
On suggestion of the co-researchers, we carried out a lunchtime meet and greet at the clinical site for the research team to meet potential study participants, to encourage them to take part. One of the co-researchers suggested we contact each other by WhatsApp and led on creating the group. This enabled everyone to be more easily involved in decision-making, as contact could be made quickly and the information readily available and all stored in one searchable thread.
I felt very involved with the decision-making progress throughout the project. The decision-making process was democratic, and we have often had the opportunity to review things before they are finalised and suggest changes. I think that the team, handled discussions very well and we were able to work well to achieve a good result.
Being a part of this team has been wonderful and extraordinary, and "empowered" is not a strong enough word for how I feel. Thank you for letting me be involved -this project has completely transformed how I view the world of research and technology, and I will be forever grateful :-).
(JJ) I found the experience very rewarding, working closely with the co-researchers to analyse the qualitative data provided me with alternative analytical perspectives. I believe this led to a more comprehensive and contextualised interpretation of the data, which would not have been possible without that collaboration. (ML)

| Including all perspectives and skills
Our YPAG involvement strategy aimed to include people of varied experiences in both mental health and technology. We were successful in appointing a diverse group of young people aged between

| Respecting and valuing knowledge
In our case, respecting and valuing each other and therefore the knowledge of all of us working on the project came naturally, due to our similar values and characteristics. We were aware that we all came from differing, but equally valuable perspectives and created a safe space: agreeing on meeting principles; openly sharing personal experiences; and spending time to get to know each other before/ after each meeting. There were strong support and communication as we talked via virtual meetings (Skype/FaceTime), a WhatsApp group, emails, and in person-in places and at times that suited everyone. We tried to accommodate every person's needs and preferences (eg around university and working hours). For example, some people could not attend one face-to-face meeting; therefore, a separate meeting was arranged for them. We also recognized team

| Reciprocity
Both the researchers and co-researchers benefited from co-produc-

| Building and maintaining relationships
We worked hard to build and maintain relationships by allowing time to build trust. Initially, we met everyone face-to-face and started each meeting with an informal period where we talked about our breaks, weekends and latest news. This helped the group to bond on a personal level, feel comfortable to speak up and break down any predefined power differentials. This was particularly apparent with the co-researchers because responding quickly to each other through WhatsApp helped to show that we were all prioritizing the project and each other. These supportive relationships have continued past the lifespan of the project. Between March 2018 and July 2019, there were 494 WhatsApp messages and 23 images sent between us. In contrast, this was not the case for the other young people who were not involved in the data collection and analysis stages. This was because role activity was intermittent and although they were given ad hoc project updates, there was not as much time to build as strong relationships.
A new co-researcher was recruited two months after the project started due to another potential co-researcher's health deteri-

| Reporting reflections that capture the impact on the research, the researchers and co-researchers
PPI impact was demonstrated in the design, data collection, analysis and dissemination stages. The co-researchers co-created the topic guide. The resulting guide was written in a way that young people could understand, was appropriate for the audience and community validity was evident. 18 In the data collection, co-researchers developed rapport with the interviewee quickly (eg interviewee 1, 'you know what I mean!'). They made the interview feel like a conversation, humanizing the person and the situation, and understanding specific terminology. Consequently, they used appropriate probing on certain areas because they knew what the interviewee meant (eg 'your Bipolar, go back [to talk] about the mania'). In contrast, the lead researcher would have missed this. However, personal experiences sometimes impacted on the level of probing. For example, sometimes a co-researcher used leading questions because of their mutual mental health experience; they were invested in the conversation. This mainly happened in the first interview, when the researcher shadowed them and this improved each interview, after the self-reflection and feedback sessions.
The impact of working with the co-researchers to co-produce a transcript framework and thematic map was evident in the data analysis. Figures 2 and 3 display the thematic map prior to (Figure 2), and after ( Figure 3) co-researcher input (Purple font reflects co-researcher comments/changes). For example, co-researchers changed the specific 'negative impact of peer support' sub-theme to include both negative and positive impacts. Significant changes were made to wording of themes and sub-themes and theme linkage. Sub-themes were merged or separated based on the co-researchers' perspectives.
For example, issues in reaching out were changed to considerations of reaching out, to reflect both positive and negative consequences.
Involvement of co-researchers has delivered impact through dissemination at two important levels. Firstly, they contributed to dissemination of this research at international and national conferences and in international journal publications, and secondly, they shared their experience of the PPI process. This was shared from the front line, to clinical and academic audiences at conferences to public audiences that were young or new to PPI, and who may have wanted to follow in their footsteps.

| Main findings
To our knowledge, this is the first paper to describe the process of co-producing a qualitative mental health research study with young people, in line with the principles of co-production. The relationships created and maintained throughout the project were vital to the study's success. Indeed, involving the same coresearchers throughout the whole project, and not intermittently, translated into meaningful involvement and true co-production.
Their ability to develop rapport quickly with interviewees, probing deeper in interviews and understanding of terminology, was evidential impact on the research. However, our different experiences and backgrounds meant that at times we had conflicting priorities. For example, the lead researcher's priorities were as follows: rigorously collecting data that was needed to answer the research question; sticking to the topic guide (not going off-track); and not asking leading questions to the participants. Although the co-researchers were trained and aimed to do this, they had a greater tendency to reassure the participant and share their personal experiences, which could cause them to lead the participants and potentially affect the integrity of the data. This was managed by debriefing the co-researchers on the need to balance building rapport with efficient data collection.
Furthermore, true co-production takes time. Every effort was made to co-produce the project in its entirety and to address power imbalances between the researcher and co-researchers. Sharing F I G U R E 2 Section of thematic map before co-researcher involvement power was made more difficult by the fact the lead researcher had to continue to maintain overall responsibility of the project including organizing PPI activity, resources, planning logistics for the interviews, meetings and events, as well as remaining responsible for the co-researchers' safety, finances and potential errors. This was unavoidable in an academic environment and with the budget available.
Moreover, building trust and relationships, collective decision-making and support, demanded dedication from both the researcher and co-researchers and took up more time/budget than was originally predicted. Despite best efforts to plan, interactions with people and tasks will either (a) take up more time than expected; or (b) will require additional funds due to unexpected delays or events. Nevertheless, reciprocity was evident throughout, as co-researchers gained new knowledge, skills and recognition from the University, PPI community and their peers. The researchers gained valuable experience of co-producing a research project, overcoming challenges and also gained recognition from the academic and clinical communities.
INVOLVE has reported on six studies against the principles of co-production. 12,13 To our knowledge, no other qualitative study involving young people with mental health difficulties has evaluated their study against the five principles of co-production. 10 However, a recent narrative review found 17 health research studies that involved patients as co-researchers throughout the research process. 9 All of these used qualitative methodology but only six took place in the UK and only three related to mental health. Moreover, only one of these studies had patient involvement or co-production as its primary focus. Worryingly, the other studies that did centre on this (but from other topic areas, for example cancer and learning disabilities) had issues with transferability, relevance of findings and strength of impact. There is a limited number of studies that involved young co-researchers (18-25 years). The review authors argue that studies that do involve them, do not help advance knowledge.
Whilst we agree that working with co-researchers is challenging because of our reflections above, our study used a rigorous assessment against the principles of co-production that captured the impact on a research project, but also on the researcher and co-researchers themselves. Co-producing our study with young people offered a deeper understanding of the findings and terminology that would not have been achieved without them.

| Future co-production recommendations
Building on other recommendations, 6,7 we have several recommendations that can be applied to co-production with young people who have experience of mental health difficulties, and more widely. We would recommend offering appropriate clinical support for young people with experience of mental health difficulties and involving several co-researchers, in case of drop out (eg due to situation or health changes). Whilst there was group consensus of the need for the researcher to act in a 'leader' role (ie to have overall coordination and oversight of the project), we suggest potential honorary researcher status could be given to the co-researchers, which could mean they could take on, or share, this role with the researcher, depending on budget and co-researcher's desire/capacity/time. We also suggest considering young people to co-design and co-deliver the training to co-researchers to further reduce power restraints. A flexible budget is needed for meaningful co-production. We suggest using the INVOLVE calculator to predict PPI costings but also have extra contingency fund for tasks not originally costed. 21 In the future, flexibility should also be acknowledged by funders, reviewers and governing bodies as currently this is not the case.
For example, some suggestions by young people with lived experience were only put forward after the project had been funded and started (eg the meet and greet event). In addition to the training schedule (Appendix 2), the co-researchers in particular recommend more practice in conducting interviews. For example, it would be helpful to provide the opportunity for the co-researchers to practice the interviews on a different day with feedback (as we did not monitor if they practiced with friends) or provide a researcher to give feedback (Box 2). A refresher session could also be offered (with questions and answers), to increase their confidence and skills. Furthermore, we recommend capturing all reflections, dates, notes and feedback within an electronic notebook (eg Evernote).

| Limitations
We evaluated our work against the five principles of co-production and provided reflections from the co-researchers (CC and JJ) and researchers (LD and ML). Involving only one male co-researcher may have negatively impacted upon perspectives across both genders. The psychiatrist's reflections were also not reported. This was mainly because building relationships and sharing power was primarily between the co-researchers and lead researchers. In the future, we will attempt to record interactions between all team members throughout the research process. The article also provides recommendations that are primarily addressed to those undertaking mental health and qualitative research; hence, transferability might be limited. Nevertheless, most recommendations are centred on achieving the five principles which are universal. Additional work is needed to evaluate co-production in quantitative and mixed methods studies, across different populations and areas other than mental health.

ACK N OWLED G EM ENTS
We would like to thank all participants from our original qualitative study who took time out of their lives to talk to us. We would also like to thank the four other young people with experience of mental health difficulties who worked with us during the course of the study. Finally, we would like to thank The McPin Foundation for their advice and support in relation to patient and public involvement. contributions to the acquisition and interpretation of data. All authors have been involved in drafting manuscript or revising it critically for important intellectual content and approved the final manuscript.

E TH I C A L A PPROVA L
Ethical approval was not needed for this reflection point within the qualitative mental health study. It should not be read as a research paper.

DATA AVA I L A B I L I T Y S TAT E M E N T
Data sharing is not applicable to this article as no new data were created or analysed in this study.  [12][13][14][15] A PPE N D I X 2

S TE P S TA K E N TO TR A I N A N D S U PP O RT YO U N G PEO PLE WITH PA S T M E NTA L H E A LTH D I FFI CU LTI E S I N CO N D U C TI N G I NTE RV I E WS A N D CO D I N G TR A N S CR I P T S
Step Detail Training preparation (for session 1 and 2) • Pre-read documentation on the project prepared and sent to co-researchers • Doodle poll set-up, sent out and training dates confirmed • Rooms booked and catering ordered for all day (coffee/tea/water-AM/PM, lunch, cakes and healthy snacks) • Presentation and key points about interviewing prepared by lead researcher (LD) and approved by PPI lead (ALJ) Training session 1 (Conducting interviews) • Initial introductions, learning objectives and expectations for the day were discussed • Discussed everyone's background and skills and a brief recap on the research project was conducted • Main considerations for interviewing people with current mental health conditions in an interactive session were discussed • Researchers performed role plays of conducting an interview (eg including potentially challenging situations) to help the co-researchers to gain tools and confidence • Co-researchers took part in role play and took turns as interviewee, interviewer or shadowing the interview (gave specific feedback, alongside feedback from the researchers) • Questions and answers session conducted Feedback, actions and homework • Researchers/co-researchers gave feedback via email and held questions and answers sessions • Co-researchers were given take away materials including notes, extra tips and reflection from the training day (sheet to refer to) • Co-researchers were asked to spend time revising the topic guide and practising interviewing with a friend • Each co-researcher was given certificate of attendance • Researcher was available to be contacted with any questions During the interviews • Co-researcher shadowed researcher in real interview; then, researcher shadowed co-researcher • Researcher and clinician were onsite for any support for co-researchers before/during/after interviews, including a debrief after each interview with self-reflection and feedback from researcher • Researcher was contactable at all times throughout interview days and in-between interviews Training session 2 (Coding transcripts) • Reflection on being a co-researcher and interviewing people was discussed • An initial impression of the data collected in the interviews was discussed • Went through working example of a thematic analysis using previous qualitative study • Introduction to basic thematic analysis and practice session on first two steps (1) familiarization and (2) initial coding a transcript (matched to an interview they have conducted when possible) was conducted • On-site support from lead researcher, PPI lead and PhD student was provided • Questions and answers session and feedback Feedback, actions and homework • Researchers/co-researchers give feedback via email and hold a Q and A session • Take away materials given to co-researchers including quick guide to thematic analysis • Each co-researcher given certificate of attendance • Co-researchers asked to code another two transcripts each • Researcher was available to answer any questions

Quote 1
'The support was really good overall. I felt like a valued member of the team and I was well looked after' (CC)

Quote 2
'I enjoyed the initial session of proofreading the topic guide and information sheet as a group. It felt like my edits were valuable and it was really rewarding to see that something I said or contributed ended up being in the final product' (JJ)

Quote 3
'…I think it's an enjoyable and interesting experience to get involved in a project like this from any background. I think that in a sense it gives more context to one's experiences, and it does to some extent make the things that one might have experienced feel like they at least have a use in the world. In addition to this I feel I've gained useful people and teamwork skills and also training in research methods that will be useful to me in my degree and hopefully elsewhere in my career' (CC)

Quote 4
'I have nothing but good things to say about the process and the team that we have worked in the process has been amazing and informative and overall a great experience for me. Although the process definitely, brought up some memories for me, I would not describe this as being negative, in some ways it just provided a new perspective on my own experiences and those of others. In addition, I felt that it was useful to be able to apply what I have learnt from my own experiences, and Lindsay was good at being aware of the potential impact that doing the research might have on us and giving me a positive perspective on reflections'. (CC)