Understanding multi‐stakeholder needs, preferences and expectations to define effective practices and processes of patient engagement in medicine development: A mixed‐methods study

Abstract Background The holistic evolution of patient engagement in medicines development requires a more detailed understanding of the needs of all involved stakeholders, and one that better accounts for the specific needs of some potentially vulnerable patient populations and key stages in medicines development. Objective The purpose of this convergent mixed‐methods study was to better understand the needs of different stakeholders concerning patient engagement at three key stages in medicines development: research priority setting, clinical trial design and early dialogues with Health Technology Assessment bodies and regulators. Design This study brought together findings from three sources: i) an online questionnaire, ii) face‐to‐face consultations with two potentially vulnerable patient populations, a workshop with Health Technology Assessment bodies, and iii) three‐step modified Delphi methodology. Results Overall stakeholders still need additional varied support mechanisms to undertake, sustain or measure value of patient engagement. Health Technology Assessment bodies need better rationale for patient engagement in early dialogue and tools to support its implementation. Improved awareness and understanding of the need and value that involving patients, who are often considered as potentially vulnerable, can bring is needed, as is better accommodation of their specific needs. Similarly, weighted Delphi categories were as follows: aims and objectives, and sustainability. Several additional themes were common across the three key stages in medicines development. Conclusion This broad‐reaching study provides the blocks needed to build a framework for patient engagement in medicines development. Patient or Public Contribution Patients were involved in review and interpretation of data.


| INTRODUC TI ON
There is increasing consensus among stakeholders that patient engagement (PE) in medicines development is critical to fostering patient access to better designed innovative therapeutic solutions and delivering more effective health outcomes for patients. 1 7 While there are many initiatives to involve patients across that continuum, inconsistency and fragmentation remain the norm. 1,[8][9][10] This is especially so for the meaningful engagement of potentially vulnerable patient populations who often have additional or specific needs (such as people with dementia and young people). 11,12 Patient engagement also needs to be understood better at key upstream stages of medicines development that are comparatively poorly serviced by current efforts such as research priority setting, clinical trial design and early dialogues with health technology assessment (HTA) bodies and regulators. Achieving systematic and meaningful PE are challenging and must meet the needs, expectations and context 13,14 of the different stakeholders involved. 3,8,15,16 A better understanding of multi-stakeholder needs, preferences and expectations for patient engagement would provide a more solid foundation for empowering actions such as the creation of tools and frameworks that can holistically enhance effective, meaningful and sustainable PE.

| Aims and objectives
We aimed to generate criteria for effective PE in medicines development, focused on three key stages where systematic PE is generally less mature compared to other stages and taking into consideration the specific needs of, and support that should be in place for, potentially vulnerable patient populations, and which are not always appropriately addressed in current PE approaches. These key stages and patient populations were as follows: (i) research priority setting (RPS) -providing opinion, providing evidence and/or being part of a group that decides what is important to research, (ii) clinical trial design (CTD) -designing protocols, discussing patient burden, discussing patient-related outcomes, iii) early dialogues with regulators and HTA bodies (ED) -early discussions between industry, HTA bodies and regulators (and in some contexts with payers) regarding developmental plans for a medicinal product and to ensure they meet the requirements and iv) potentially vulnerable patients, in the context of this project, include (but are not limited to) people with dementia and their carers, and young people.

| DE S I G N AND ME THODS
This work was conducted within the context of the PARADIGM project, funded by the Innovative Medicines Initiative (IMI), that developed ways to ensure that patients are always meaningfully involved in the development of medicines. 7 This study took a convergent mixed-methods approach combining quantitative methods and a qualitative approach, using consultations set within the framework of patient and public involvement (PPI). The learning and key emerging themes from each stage were incorporated into each following stage. This brought together three components: i) an online questionnaire to capture overall identification of needs, expectations and preferences of effective PE from all involved stakeholders, ii) separate face-to-face consultations with two specific groups of potentially vulnerable patient populations, and a separate workshop with representatives from HTA bodies -to gather greater insight on the needs and preferences of these groups beyond the results of a survey, and (iii) modified Delphi exercise to identify and prioritize the minimum agreed criteria for effective and meaningful PE at the three key stages of medicines development ( Figure 1). PARADIGM consortium partners AE and FSJD sourced patients involved in the two consultations from pre-existing working groups within each respective umbrella organization. Each was led by their own staff and followed its own standard institutional processes for organizing PPI activities. 17,18 Delphi panel experts were identified from the PARADIGM consortium networks, and standard institutional processes were followed for undertaking the Delphi exercise.

| Stage 1: Survey
This survey aimed to identify current needs and expectations for PE across medicines development. The survey was constructed in two phases. Firstly, issues identified from existing literature on PE in medicines development 8,15,16,19,20 were prioritized during a face-to-face workshop involving a multi-stakeholder working group. This informed the structure of a survey which was piloted over two weeks using respondents from each respective stakeholder group.
The final survey was made up of 15 general questions that all stakeholder groups completed. Within this general section, two questions were constructed in matrices allowing more than one choice per row, six questions were structured as visual analogue scales (assigning 0-100 points based on respondents' impressions on PE) and four questions were multiple-choice, allowing for more than one option to be chosen. Stakeholder groups: the patient community, industry, regulators, policymakers, HTA bodies and payers, research funders and HCPs (clinical academics) also had additional separate sections comprising matrices and multiple-choice questions. Within the survey, we initially sought to gain a broad benchmark of PE across the medicines lifecycle, including dialogues with regulators and HTA bodies that involve the licensing of medicines, HTA assessment, and pricing and reimbursement. Hereafter, the workshop with HTA bodies and Delphi panels only explored the definition of early dialogues with regulators and HTA.
The survey was administered in English in an online tool, Survey Gizmo, over a four-week period in 2018. A snowball technique was utilized to cascade the survey within consortium members' internal and external networks to reach an estimated sample population of 10,000 (with 95% CI), with a minimum calculated sample size needed of 370.
All key stakeholders in medicines development were targeted: regulators, HTA bodies, industry (pharmaceutical, biotechnology and medical technology companies) health-care professionals (clinical academics), patients and patient representatives (from diseasespecific or agnostic organizations, and non-affiliated individual patients), policymakers, research funders and research and academia (research institutes and universities). Findings are described as total responses or percentage of respondents to a given question or theme.

| Stage 2: Face-to-face consultations and workshops
The specific needs, expectations and preferences of young people, and people with dementia and their carers were explored through separate face-to-face consultations, involving respective experienced staff, in order to better understand the specific PE needs of these groups of patients.
In the case of people with dementia, members of Alzheimer Europe's (AE) European Working Group of People with Dementia (EWGPWD) 21 and their carers participated in a one-day consultation F I G U R E 1 Infographic of three-stage convergent methodology. The method contained three components -each informing the next: i) an online questionnaire to capture needs, expectations and preferences, ii) face-to-face consultations with two groups of potentially vulnerable patient populations, and a separate workshop with representatives from HTA bodies, and iii) three-

| Stage 3: Three-step RAND modified Delphi methodology
The modified Delphi methodology is a recognized method for prioritization of diverse variables through consensus. 25,26 Delphi questionnaires were developed based on the outputs from stages one and two, combined with a review of existing frameworks for PE. 10 Experts for each of the Delphi panels were convened using a snowball method. Experts had to hold recognized expertise and experience within their topic panel in relation on PE to guarantee that the group reflected the view of a majority (initial expert group size; RPS = 24, ED = 26, CTD = 31) ( Table 1).
Each panel was balanced as far as possible for geographical coverage and sex (see Table 1

| RE SULTS
Here we present the findings of each of the three stages and the overall conclusions. Additional results are available athttps://imiparad igm.eu/our-work/. 28 Note: The stakeholder group patient community is differentiated into: Patients (including carers) †, and patient advocates and patient organizations ‡. Health-care professionals (HCP) -clinical academics, Health Technology Assessment bodies (HTA), industry (pharmaceutical companies (Pharma), small and medium enterprises (SME) and Biotechnology companies (Biotech)), other § (include, but not limited to individuals that identify as their primary affiliation of; charity, consultant, independent expert, think tank, industry association, social association, funder, NGO's, or identified as having multiple relevant affiliations.

Total number of respondents
TA B L E 2 Survey respondents by stakeholder from the total of 372 English respondents

| Survey general characteristics
A total of 372 respondents completed the survey in English ( Table 2).
The largest respondent (stakeholder) group was the patient community (patients (including carers), patient advocates and/or patient organizations) (35.8%), followed by industry (34.9%). Respondents completed the survey from 48 countries -a majority from the UK and United States (28.2% and 16.9%, respectively) ( Figure 2). The group 'Other' comprised 36 countries, both within and outside the European Union (18%, n ≤ 5 respondents from each).

| What is the status quo of PE today?
Current perception of PE is low but ideal expectations are high.

| What are the desired outcomes of PE?
Greater patient-centric input, patient-relevant outcomes and communication are desired the most.
Respondents were asked to indicate up to three most desired outcomes of PE from a separate list of predefined outcomes for each of the three stages. The top three are as follows (see also Supplementary Table 1

Consultations: Increasing and supporting involvement of potentially vulnerable populations in PE.
Incorporating the lived experience and accommodating reasonable adjustments help to recognize and acknowledge the value patients bring.
Several common barriers to increasing and sustaining patient involvement emerged from the consultations with young people, and people with dementia and their carers, along with recommendations to overcome these ( Table 3).
Understanding and incorporating the 'lived experience' of the patient (and not just the parent or carer) was considered to be paramount in adding genuine value to a given PE activity and in reflecting the expected outcomes. This can also serve to readdress many misconceptions of people with dementia and young people -namely

F I G U R E 7
What is required to do more effective patient engagement (total respondents). All stakeholder responded. Respondents could select more than one option. In the 'Other' category, additional resources highlighted included 'a clear framework and guidelines on how to engage with patients' and 'funding/ financial support from other stakeholders, particularly in providing funding/ reimbursement/tokens of gratitude for patients' [Colour figure can be viewed at wileyonlinelibrary.com] that they are not able to or willing to be involved, an incorrect understanding of the condition(s) that they are living with, the added value that they can contribute, and the specific considerations (physical, mental, socioeconomic) of these populations.
The diversity of the patients involved in any PE activity was considered important to allow the broad range of experiences patients could contribute and provide equal opportunities for those who can be involved to do so. For example, different age groups, TA B L E 3 Major themes identified from two separate consultations (people with dementia (and their carers) and young people) and how to improve patient engagement (PE) with these populations. Other themes and full results are available at 28

Meaningful methods to enhance meaningful PE in patient populations according to the input provided
Voice of the person with the condition

People with dementia
The people with the condition are the 'experts by experience' and their input is unique.

Young people
The people with the condition are the 'experts by experience' and their input is unique.

Redress misconceptions
People with dementia Myths/misconceptions surrounding the type of disease, ability/willingness to participate, need for support. Young people Misconceptions about unable, or unwilling to contribute properly to PE. The early engagement in a research priority setting is feasible; the experience and/ or support young patients can provide in the design of clinical trials protocols, participation in regulatory activities is feasible if a suitable framework is established.

Diversity of patients involved and inclusion
People with dementia Account for differences such as age group, country, type and stage of dementia -mild, moderate and advanced dementia etc Young people Account for age group, country, different or complementary expertise to that of parents.

Equal opportunities for patients to participate by ensuring accessibility needs are met People with dementia
The structure and format of the PE activity should consider the needs of the person with dementia: the accessibility of the materials received, the use of plain language and avoiding the use of jargon, acronyms and highly technical terms. Young people Include age appropriate formats and language.

Raising awareness of PE opportunities
People with dementia Provision of relevant information, support and training to patients, carers and other stakeholders interacting with them Young people Promoting autonomy, respect and equality. Ethical principles and the children's rights need to be considered in the design of involvement activities for children and young people.

Reasonable adjustments for travel/ accommodation, accessible information, training sessions/personal support, financial support/ reimbursement People with dementia
Travel and accommodation costs incurred should be covered for both the person with dementia and his/her carer. Organizer of the PE activity should designate a 'named person or a single point of contact' with whom the person with dementia could speak to freely Accessible and understandable information about the PE activity Young people Travel and accommodation costs incurred should be covered for both the young person and their parents. Timing so that it doesn't interfere with school lessons Educational material and information (written and verbal) should be education level with appropriate language.
A facilitator from a YPAG should be available to provide the right personal support to a young person in both the preparation and during the PE activity.  For the other two key stages, Sustainability was ranked equally low.
Conversely, the category, Aims and objectives (and related category, Key elements of practice design, which included aims and objectives) were ranked similarly high across all three key stages. Criteria within this category centred on that the aims and objectives of PE practices should meet the expectations of patients and/or focus on patients' needs and interests. They should also be agreed upon up front by all and be understandable to all involved participants (see Supplementary Tables 2-4).
At the criteria level overall, there were differences in the wording and definitions, across the three key stages (Table 5), the main difference, however, was in their respective weightings. The following general criteria were thus fairly consistent in their definition, across categories and the three key stages:

| Involvement and participation
Patients' participation should be properly planned, taking into account timing requirements, accessibility and vulnerability. More specifically, it should consider specific patient's circumstances and characteristics linked to but not limited to possible physical or mental impairments, cultural background, age and other relevant features (eg recordings, virtual communication and use of language, format of meetings, the venue and information provided). Additionally, an up-to-date single point of contact or a named person with whom patients can communicate when needed for information and/or support, is made available throughout their involvement in the activity (eg during ED).

| Legal and ethical considerations that govern PE -including a code of conduct
It is necessary to have any relevant policy directives, legal, ethics, governance requirements and/or regulatory framework about how to engage patients included as part of PE practices. More specifically, ensuring that codes of conduct are adhered to and conflicts of interest are addressed and managed through accountability and transparency. Notably, participants raised as yet unresolved considerations of how to effectively balance conflicts of interests with suitable patient participation.

| Building capacity to support the PE process
The competencies that are required to perform the PE activity itself  Note: The final Delphi round aimed to i) reach consensus among the disagreement items not obtained after the first and second round; ii) Merging and rephrasing criteria and categories; iii) weighting categories; iv) weighting individual criteria. Starting from a list of approx. 50 questions for each decision point, after round 3 this was reduced to an agreed 20-25 criteria separately for each key stage of medicines development, positioned under 8 or 9 major categories. All weightings across categories and criteria within each category, equate to 100. The full breakdown of round 1 and 2 results can be seen at. 28 backgrounds and the circumstances of potentially vulnerable patients involved in setting research priorities.

| Evaluation of the PE process
Generally, methods, tools and monitoring systems should be in place to evaluate the PE practice, including a framework. An evaluation of the outcomes should be linked to the aims and objectives of the PE practice in each setting. Additionally, the outcomes should be shared with all the participants involved and procedures should be in place so that the conclusions of the evaluation are used to support a continuous improvement process. The rate of evolution here will likely be at a different rate from industry and regulators. From the reported perspective of HTA bodies, the proposed 'building block' approach allows work to progress in parallel and at a different rate to other stakeholders and geographies, but remain linked and sympathetic to an evolving ecosystem approach to PE.

| Limitations
The online survey was completed only by those who are familiar with and access to the technology, and a good understanding of Finally, there were dropouts of Delphi panellists during the process for a variety of reasons. In the CDT Delphi group, many respondents were from North America; hence, some bias in interpretation of responses and weightings was inevitable. Despite this, the overall number and expertise balance within and between each group was monitored, and stakeholders' representativeness were not at risk.

| CON CLUS ION
This multi-stage study adds significant value by building on some of the previously identified gaps between stakeholder specific understanding and importance of PE, and who should be responsible for leading efforts to improve it. 8,15,31 These findings provide detailed and tangible building blocks to all involved stakeholders as to where stakeholder needs and expectations are, and where greater alignment could occur. The work of the PARADIGM consortium and its partners continue to address many of these signals holistically through multi-stakeholder mechanisms 35,37,57 in the continued evolution of meaningful, ethical and sustainable PE.

ACK N OWLED G EM ENTS
The authors gratefully acknowledge all those PARADIGM consortium members who helped design, test and validate the survey and Delphi materials. We thank all those survey participants and Delphi panellists who dedicated their time to help make these a success and to those participants who participated in the face-to-face consultations facilitated by Alzheimer Europe

AUTH O R CO NTR I B UTI O N S'
All authors contributed to the design of this study and the writing of this paper.

DATA AVA I L A B I L I T Y S TAT E M E N T
The data that support the findings of this study are available from the corresponding author upon reasonable request.