‘I felt like a human being’—An exploratory, multi‐method study of refugee involvement in the development of mental health intervention research

Abstract Background Great advancements have been made in patient and public involvement (PPI), including the development of guidance on how to conduct, report and evaluate PPI. Despite these efforts, the evidence base remains relatively weak. A substantive methodological development is required. This is particularly important for vulnerable groups within society, for whom PPI can be challenging but has the potential to play a transformative role in shaping research. Objectives To describe the group dynamic characteristics and immediate impact of PPI from the user representatives’ perspective in a case study of refugee involvement in the development of mental health intervention research. To pilot and methodologically appraise the Active Involvement of Users in Research Observation Schedule and Questionnaire. Design The Active Involvement of Users in Research Observation Schedule and Questionnaire were administered together with a focus group discussion. Setting ‘Refugee Advisors’ were involved in the development of a randomized controlled trial protocol evaluating a brief group intervention for refugee children experiencing symptoms of post‐traumatic stress in Sweden. Results The multi‐method approach demonstrated good feasibility. There were clear examples of how the advisors influenced research development. The advisors described a perceived impact on the research, equality and acceptance, and knowledge gain. A sense of appreciation and empowerment was also interpreted. However, potential issues relating to the relevance of contributions and use of an interpreter were identified. Discussion and conclusion The methodological approach piloted in this study offers a promising, rigorous way to evaluate PPI. The research tools require further refinement and validation.


| INTRODUC TI ON
Patient and public involvement (PPI) refers to an active partnership between patients and/or members of the public and researchers. 1 Involvement is distinct from participation in research: patients and/ or the public contribute to the research process as advisors, and possibly as co-researchers. 1 Involving 'vulnerable' groups within society, such as refugees, in research development can be challenging, but has the potential to play a transformative role in shaping research. Workshops with various stakeholders including PPI representatives can be an effective way to establish priority research areas that are meaningful to patients, clinicians and researchers alike. 2  It can help to inform sensitive and efficient recruitment strategies and, importantly, support truly informed consent. This can be particularly complex given the various languages potential refugee participants may use. It is common for researchers to exclude certain populations on linguistic grounds, a practice which may reflect on research outcomes and in turn may affect the provision of services to linguistically diverse populations. 3 Language, culture, religion, social norms and experiences of oppression may also make it difficult to obtain accurate responses to research questions. 4 The format of the research is another important consideration. Interviews may raise suspicion, given the long interviews as part of the asylum process, and focus groups could stimulate tension if participants come from different backgrounds. 4

| Standardization and evaluation of PPI
Thorough guidance on how to conduct and report PPI, such as INVOLVE (see invo.org.uk) and the Guidance for Reporting Involvement of Patients and the Public checklists (GRIPP 5 & GRIPP2 6 ), has been developed in an effort to standardize the process. Moreover, PPI evaluation frameworks have been developed to guide PPI impact assessment, such as the Public Involvement Impact Assessment Framework (PiiAF; see piiaf.org.uk). There is clear support for the importance of evaluating the impact of PPI 7 and reviews of the literature indicate a growing research interest. 8,9 Yet, despite these efforts, the evidence base remains relatively weak. An international effort is required to improve the PPI evidence base, as the majority of evaluative literature comes from the UK. 8,9 Substantive methodological development is needed, including methods for assessing impacts both qualitatively and quantitatively. Evaluative data on PPI are often brief, narrative descriptions, which reflects the lack of robust tools specifically developed to assess PPI. 8 Attempts to quantitatively assess the impact of PPI have been carried out, but have been critiqued for the lack of sufficient attention to (a) the context in which involvement takes place and (b) the way it is carried out. 10 Yet, a fine balance is required as scientific approaches designed around a specific context are difficult to extrapolate beyond that given context. 10 Staley 10 maintains the value of evaluating the impact of PPI but calls for more detailed accounts of PPI experiences; she argues that researchers' accounts of their experience potentially provide a source of insight and learning to influence others.
There is a strong tendency to report on the beneficial influence PPI has on research and the research process, with few papers reporting on negative influence. 8 Costs associated with PPI and coproduction could include adverse effects on the research itself, the research process, personal and professional risks for researchers and stakeholders and risks to the wider cause of scholarship. 8,11 The positive reports could indicate the benefits of PPI far outweigh the challenges, or it may indicate publication bias. 8 It could be that methodological bias is also contributing.
Behavioural observation, the systematic recording of behaviour by an external observer, could bring great insight to the factors and agents that contribute to successful PPI. Yet, this appears to be an underexplored research area. A semi-structured observational approach would allow for mechanistic elements of PPI to be coded against a set protocol and for the context-specific details to be recorded in the form of observational notes, and thus also providing a detailed experiential account. This approach would strike the aforementioned balance between context and extrapolation called for by Staley. 10 By utilizing an observational approach whereby the observer is not part of the social exchange being observed, the aspect of reciprocity (i.e. the practice of exchange for mutual benefit) can be observed. The structure of reciprocity in exchange affects the solidarity of bonds that arise from exchange. 12 If PPI is to represent a partnership approach to developing and conducting research, then high levels of reciprocity are important. Incorporation of the theory of reciprocity into the evaluation of PPI can be further strengthened by a triangulation of data by which researchers and PPI representatives are given the opportunity to directly reflect on the PPI both in terms of the mechanisms and perceived impact, in addition to being observed.

| Evaluation of PPI group dynamics
Group dynamics are the interpersonal processes that determine how group members relate to and engage with each other and what the group achieves. 13 There are several influential aspects to consider when studying group dynamics, including communication processes and interaction patterns; interpersonal attraction and cohesion; social integration and influence; and power and control. 13 (Figure 1). Derived from group process literature, the framework acknowledges the reliance on group effectiveness in achieving short-term and long-term participatory research goals. It is proposed that, in order to progress to research outputs that are both influenced by and meaningful to user representatives and researchers alike, the process of meeting and discussing the research must facilitate meaningful involvement.
Collaboration is dual-faceted in that both the content of the problem to be solved and the relational aspects of the group need to be managed; and more successful groups are more responsive to one another. 16 Table 1 for the full coding list and descriptions. There is an accompanying paper-based assessment form that allows attendees to independently and anonymously grade the meeting on a Likert scale from 1 (Not at all) to 5 (A lot) on a list of items that correspond to those on the observation pro-forma.

| AC TIVE INVOLVEMENT OF US ER S IN RE S E ARCH OBS ERVATI ON SCHEDULE AND QUE S TIONNAIRE
The core components of group dynamics 13,14 were used to guide the construction of the evaluation instruments (Table 1). Previous findings from PPI evaluations were used to place the group dynamic components in context. For instance, the communication processes and interaction patterns attribute of 'Linguistic barriers to advisor participation' was derived from user representative reports of inaccessible research language, 18 as well as the potential for multi-lingual group meetings. 'Reference to advisors' expertise', considered an attribute of social integration and influence, was included in the protocol due to reports of assumed lack of knowledge and associated frustration from user representatives. [19][20][21][22] 'Ideas being ignored/ treated with disregard', considered an aspect of power and control, was included in light of descriptions of user representatives' ideas not being listened to or marginalized within discussions. 18,21 The intention is for the evaluation instruments to be applied to various PPI contexts and, thus, emphasis is placed on group dynamic characteristics rather than environmental or structural characteristics. It is also envisioned that the instruments are used within a multimethod research design, which includes insights from interviews and/ or focus groups. This triangulation of data enables the group dynamics assessment to be placed in the context of immediate or output measures of partnership effectiveness in line with the conceptual framework developed by Schulz et al 15 (Figure 1). The case study provided in this paper utilizes the Active Involvement of Users in Research Observation Schedule and Questionnaire together with a focus group method to explore both PPI group dynamic characteristics and the immediate impact of PPI from the user representatives' perspective.

| Objectives
The objectives of the present paper are to: 1. Describe (a) group dynamic characteristics and (b) immediate impact of PPI from the user representatives' perspective in a case study of refugee involvement in the development of mental health intervention research.

Pilot and methodologically appraise the Active Involvement of
Users in Research Observation Schedule and Questionnaire.

| Setting
The

Group dynamic component
Interpersonal relatIons

Positive interactions
How the advisors and the research team interact. Take note of positive interactions (e.g. humorous or appreciative remarks).

Interpersonal attraction and cohesion
2. Reference to advisors' expertise How the researchers refer to the advisors and/or advisor input. Take note of comments that infer skill or knowledge (e.g. use of terms such as 'expert', 'important', 'valuable', 'helpful' 'interested to know what you think').

Linguistic barriers to advisor participation
The accessibility of the conversation to the advisors. Take note of any linguistic barriers (e.g. scientific language that is difficult to understand; interpretation issues, such as insufficient time to translate and miscommunications).
Communication processes and interaction patterns 4. Advisors showing a lack of interest/ being disengaged Advisor body language. Take note of gestures or actions that infer a lack of interest (e.g. yawning, looking away from the point of focus, looking at mobile phone, doodling, checking the time).
Interpersonal attraction and cohesion nature of advIsor contrIbutIons

Invitations to speak
Researchers directly asking advisors to comment. This can be a specific question, or asking for any further thoughts on a point of discussion.
Communication processes and interaction patterns 6. Taking the initiative to speak Advisors providing comments without being directly asked. This can include an advisor spontaneously adding to a response of another advisor (even if the first advisor was directly asked a question).

Passively agreeing with researchers
Advisors' responses to researcher questions. Take note of occasions when advisors appear to agree with researchers without active consideration. It is the level of engagement that is important. Active agreement with researchers should not be scored negatively.
Power and control

Offering insights appearing irrelevant to discussions
Advisors making comments that do not appear to be connected to the current conversation, or providing an unnecessary level of detail.

Communication processes and interaction patterns
How advIsors guIded researcH development 9. Challenging and suggesting alternatives to researchers Advisors questioning the logic or approach of the researchers and/or providing different option(s) to consider. The relevance of challenges should be considered. Only constructive challenges should be scored positively.
Social integration and influence

Incorporation of advisors' ideas in research planning
Advisor comments influencing the research plan. This could be in relation to any aspect of the research (e.g. questionnaire selection or modifications, study age range, recruitment strategies, interpretation of findings etc). It can include intentions to act on advisors' comments (e.g. 'We should try to pilot that questionnaire with more people').

Social integration and influence
11. Ideas being ignored/treated with disregard Active consideration of advisors' ideas. Take note of occasions when advisors' input appears to be overlooked. Actively challenging the advisors' input should not be scored negatively. It is a lack of consideration that is the focus.
Power and control

Decisions made without the input of advisors
Research project decisions or intentions. Take note of when decisions appear to be made and whether advisors were involved in the process. Be mindful of decisions made in break-out group formats or during breaks. The decision does not have to necessarily reflect the advisors' choice, but their input should have been sought/offered and considered.

Power and control
English and an English-to-Arabic interpreter attended the meeting.
The Refugee Advisors were compensated for their time at an hourly rate. Nine researchers who attended the meeting, including international project advisors from UK, Norway and United States, provided feedback on the PPI group dynamics as part of the present study.

| Data analysis
The observations were coded according to the pro-forma by the second author (ZB), who carried out the live observation. The first author (GW) reviewed the observations and the appropriateness of the codes. The scores from the questionnaires were summed for all the researchers and divided by the number of researchers to provide a mean score for each item ( Table 2). The focus group discussion was transcribed verbatim, translated into English and analysed using a thematic approach. 24 The analysis categories were deduced from the question guide topics, but the conception of themes was data-  Incorporation of advisors' ideas in research planning (+) Low

1.1
Decisions made without the input of advisors (−) Low

1.9
TA B L E 2 Overview of observation protocol and questionnaire components, including observation levels and questionnaire mean scores

| Ethics
Ethical approval was granted for the study, as part of the associated RCT ethics submission, by the Regional Ethical Review Board up to the meeting. The decision was taken to record consent orally, rather than in writing. A key reason for this was that some advisors were illiterate. We were also conscious of the power dynamic relating to the consent procedure. A prepared statement was read aloud to the advisors, with assistance from an interpreter, and the advisors were asked to state whether or not they were happy to proceed. The statement included the following: the research intentions, including the nature of the questions they would be asked in the focus group; that we would write about the research findings; that no names would be used in reports or papers; and that the conversation would be audio-recorded and safely stored so we could listen to it again.

| RE SULTS
The results are presented with insights from the Active Involvement of Users in Research Observation Schedule and Questionnaire first, covering group dynamic characteristics (Table 2), followed by insights from the focus group, covering the immediate impact of PPI from the user representatives' perspective ( Figure 2).

| Interpersonal relations
Positive interactions between the Refugee Advisors and the research group were frequently observed throughout the meeting. There were many humorous remarks made, resulting in group laughter. These were related to personal attributes of the attendees (names), the topic of the meeting (supporting refugee children) and the process (translations between languages). There were also a number of appreciative remarks (e.g. 'I am deeply touched and grateful to be here. I feel wonderful, thank you'). The frequency of 'positive interactions' was also scored highly by researchers (mean score of 4.2 out of 5). Reference to the Refugee Advisors' expertise (e.g. researchers stressing the importance of having equal voices in the discussion) was observed, but to a lesser extent. However, researchers scored 'reference to advisors' expertise' relatively highly (mean score of 4.1 out of 5). 'Linguistic barriers to participation' were commonly observed, which included issues arising from interpretation. There were practical interpretation issues, such as the interpreter not having sufficient time to translate a video presentation.
Miscommunications were also evident, including the difference between accompanied and unaccompanied refugee minors and accessing voluntary support opposed to volunteering. Broader interpretation issues included the interpreter neglecting to translate some speech that was not directed towards the advisors and the interpreter offering personal insights instead of those of the advisors.
The frequency of 'linguistic barriers to participation' was also scored relatively highly by researchers (mean score of 3.4 out of 5). There were several observations of 'advisors showing a lack of interest or F I G U R E 2 Refugee Advisor focus group themes being disengaged'; however, the researchers rated the frequency relatively low (mean score of 1.3 out of 5).

| Nature of advisor contributions
Throughout the meeting, Refugee Advisors were invited to speak about various aspects of the study, including mental health helpseeking behaviours in refugee youth; cultural sensitivities; participant recruitment strategies; and questionnaire item relevance and acceptability. 'Invitations to speak' frequency was rated very highly by researchers (mean score of 4.8 out of 5). Refugee Advisors taking the initiative to speak was also commonly observed, but was scored as relatively infrequent by researchers (mean score of 2.3 out of 5).
'Passively agreeing with researchers' and 'offering insights appearing irrelevant to discussions' were infrequently observed. However, 'offering insights appearing irrelevant to discussions' received a moderate frequency score from researchers (mean score of 2.8 out of 5).

| How advisors guided research development
There was a moderate level of Refugee Advisors challenging and suggesting alternative ideas to researchers observed (e.g. recruitment strategy, study age range). This was reflected in the researchers' scoring (mean score of 2.9 out of 5). There were few (but

| Immediate impact of PPI from the user representatives' perspective
In the focus group, the Refugee Advisors conveyed a perceived impact on the research development process. They repeatedly spoke of how their input could be 'useful' to the research team. They described being helpful and expressed how their presence and their contributions, which they often termed as 'information', were advantageous to the research meeting. Their purpose within the meeting seemed to be clear; they were present to give their personal perspective on the different aspects of the research design.
I gave them information that could be useful. There was a very welcoming atmosphere, welcoming towards everyone.
(RA3, refugee father) Second, they spoke as though they had acquired knowledge from the process. They spoke directly about 'knowledge' and 'information' and benefitting from the research discussion.
I benefited myself from their questions.
(RA1, refugee mother) There was a strong underlying sense of appreciation, which was inferred as related to both the perceived equality and acceptance and the acquired knowledge. They gave anecdotal accounts of how the research team had been kind throughout the day and their gratitude for this. They also expressed a sense of comfort in the setting, which was taken to represent enjoyment in being part of the research meeting and overall process. (RA2, refugee youth)

| Nature of advisor contributions
There were high levels of 'invitations to speak' and 'taking the initiative to speak' and low levels of 'passively agreeing with researchers'.
However, the questionnaire, and to some extent the observation, Effective involvement, and a consensus on the relevance of input, is underpinned by the patient/ member of the public being skilled in identifying what the researcher doesn't know or has assumed.

| APPR AISAL OF THE AC TIVE INVOLVEMENT OF US ER S IN RE S E ARCH OBS ERVATI ON SCHEDULE AND QUE S TIONNAIRE
The observation protocol and questionnaire are promising tools for evaluating group dynamic characteristics of PPI. The present study was a pilot and will inform further development. The observer in the present study was unable to code all observations live and thus reviewed the protocol after the meeting to code remaining observations. Although this is often reported in observation studies, refinement of the coding guidance and pro-forma layout could improve usability. A scoring system, as commonly utilized in observational protocols, is also required. The inter-rater reliability of the observation protocol should be evaluated. The validity of corresponding questionnaire items measuring perception of PPI group dynamic characteristics by meeting participants should be tested, and how far these correlate with observations should be further explored. Comparison of the observational and questionnaire data from the present study indicates reasonable concurrent validity. However, focusing solely on the frequency of the characteristics and assuming equal weighting across them may not be appropriate. The significance of how advisors guide research development is conceivably greater than interpersonal relations or the nature of advisor contributions, given that the purpose of PPI is to inform research design and conduct. Therefore, the weighting of the observation protocol scoring system requires careful consideration. The practicality and validity of using the questionnaire with all individuals involved in the PPI process should be explored. This further triangulation of data could help to understand differences in perceptions and consensual variation across items, which will inform score weighting. Wider scale application (with more advisors in further meetings) will inform the development process and improve generalizability of the tools. Moreover, the application with other communities in other contexts will be important.

| CON CLUS IONS AND IMPLIC ATIONS OF THE RE S E ARCH
Case study findings indicate a need for thorough PPI preparation, perhaps skill-based training, 27 for everyone involved in the PPI process. The case study focused on PPI in the research development phase. To gain a full understanding of the impact on the research process, a longer-term evaluation is required. Prospective longitudinal studies could capture how impact changes over time. This relates to impact on the patients or members of the public and on the research project (e.g. recruitment rates, participant-reported acceptability).
The advisors in the present study indicated high levels of appreciation and empowerment; however, this impact could diminish over time and the 'feedback loop' in understanding how their involvement has actually affected the research could be an important part of the process in maintaining both appreciation and empowerment.
The methodological approach piloted in this study offers a promising way to not only evaluate the involvement of refugees in the development of mental health research, but PPI more broadly. As described above, the research tools require further refinement and validation. This would be greatly informed by a renewed review of the literature. The reviews currently available on evaluative PPI literature include research published up to 2012. 8,9 Given the increased interest in PPI over the last few years, with many research funders making PPI mandatory and the inception of PPI special issues and dedicated journals, it is reasonable to assume the knowledge base has grown.