Evaluation of patient engagement in medicine development: A multi‐stakeholder framework with metrics

Abstract Background Patient engagement is becoming more customary in medicine development. However, embedding it in organizational decision‐making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. The objective of this project was to develop a monitoring and evaluation framework, with metrics, to demonstrate impact and enhance learning. Methods A consortium of five patient groups, 15 biopharmaceutical companies and two academic groups iteratively created a framework in a multi‐phase participatory process, including analysis of its application in 24 cases. Results The framework includes six components, with 87 metrics and 15 context factors distributed among (sub)components: (a) Input: expectations, preparations, resources, representativeness of stakeholders; (b) Activities/process: structure, management, interactions, satisfaction; (c) Learnings and changes; (d) Impacts: research relevance, study ethics and inclusiveness, study quality and efficiency, quality of evidence and uptake of products, empowerment, reputation and trust, embedding of patient engagement; (e) Context: policy, institutional, community, decision‐making contextual factors. Case study findings show a wide variation in use of metrics. There is no ‘one size fits all’ set of metrics appropriate for every initiative or organization. Presented sample sets of metrics can be tailored to individual situations. Conclusion Introducing change into any process is best done when the value of that change is clear. This framework allows participants to select what metrics they value and assess to what extent patient engagement has contributed. Patient contribution Five patient groups were involved in all phases of the study (design, conduct, interpretation of data) and in writing the manuscript.


| INTRODUC TI ON
In the past decade, patients, industry, regulators, researchers and health professionals have identified opportunities to improve the medicine development process. Firstly, studies have shown that there is sometimes a mismatch between the research priorities of patients and the research that is conducted by academia and the pharmaceutical industry. [1][2][3] As a result, some new medicines and therapeutic interventions that enter the market are perceived to have little or no added value for patients. 2 Secondly, the design of studies is not always optimized for the experience of study participants. 4,5 This can result in unnecessary burden for patients, avoidable protocol amendments and a delay in access to new medicines and technologies. 4,6 Thirdly, the transparency of studies can be improved by making positive as well as negative findings accessible to those who need to make decisions about their own health. 5 Finally, clinical trials focussing solely on the evidence required for regulatory approval often lack patient-relevant outcome measures. Medicines and technologies may therefore enter the market without a full understanding of the benefits for patients. 7 Evidence suggests that engaging patients in medicine development results in studies that align better with patients' needs and benefit from enhanced performance in terms of efficiency and quality. [8][9][10][11] Pharmaceutical companies and researchers across the world continue to expand their efforts to engage patients in research and development (R&D). Patient engagement is slowly becoming more common. However, embedding and systematizing it in organizational decision-making remains challenging, partly due to lack of agreement on its value and the means to evaluate it. 12,13 There is a tension between its intrinsic value, reflecting a democratic approach of fairness, transparency and accountability ('nothing about us without us' 14 ) and an instrumental approach referring to patient engagement as a means to improving the quality of research. 15 Indeed, some argue that patient engagement cannot be seen as an intervention to be evaluated, but that it is a prerequisite for a people-centred health-care system. 16 Patient engagement may best be described as a process of knowledge exchange needed to better integrate patient perspectives, needs and priorities 17 rather than a typical intervention, which requires a different evaluation approach.
A recent literature review shows that some metrics and methods are available, but that these are not sufficient to understand (a) the mechanisms to impact, nor (b) whether the interaction between researchers and patients leads to a culture change. 18 Previous studies have identified a number of challenges to assessing the impact of patient engagement, such as the lack of well-defined endpoints, the delayed nature of impact, the absence of reliable measurement tools and accepted criteria for judging the success of engagement. 18,19 The value of patient engagement can vary by stakeholder's perspectives and therefore the measures of interest will differ accordingly. 20 Numerous frameworks provide guidance for undertaking patient engagement. Far fewer support evaluation of engagement; those that do outline concepts rather than provide detailed operational guidance. 21,22 Research suggests that co-designed evaluation frameworks are most likely to be locally relevant and used in practice but these tend to be context-specific and may be difficult to apply to different initiatives. 22 The PARADIGM consortium, a public-private multi-stakeholder partnership co-led by the European Patients' Forum and The European Federation of Pharmaceutical Industries and Associations (EFPIA), aimed to develop a framework to support collaborative evaluation of patient engagement in the field of medicine development. Our preparatory literature review indicated that an evaluation framework required to show the return on patient engagement from all stakeholders' perspectives, had not previously been developed. In common with Boivin et al, 23 we required such a tool to be evidence based, to encompass all stakeholders' perspectives and to be comprehensive and user-friendly.
Accordingly, the aim of this research project was to co-design a framework for monitoring and evaluating patient engagement initiatives, in order to support meaningful engagement in medicine development. Our research combined the perspectives of patient organizations, industry, academics, regulators and Health Technology Assessment (HTA) bodies. We focused on three decision points at which the patient perspective is likely to be valuable: research priority-setting, design of clinical trials and early dialogues with regulators and HTA bodies. This paper presents the conclusions of this work: a co-designed evaluation framework which can be used to assess the quality and impact of patient engagement in medicine development for all stakeholder groups.
the rigour needed to effect and evaluate change in a way that the learning and experience it provides is transferable and can be embedded in future initiatives. 25 Therefore, because we sought to develop a scientifically robust evaluation framework for the change to sustainable patient engagement, we chose to adopt a combination of theoretical approaches to inform this project.
We considered various evaluation approaches on which to base our framework including those shown in Table 1 and briefly discuss the applicability of these approaches for evaluating patient engagement. Some studies of patient engagement use impact evaluation; for most of these, it remains unclear how impact was (or was not) reached. Impact evaluation cannot easily be used to evaluate patient engagement initiatives 26 due to the varied approaches to patient engagement and the varied contexts in which it takes place. Furthermore, the impact of patient engagement is influenced by multiple factors. We argue that the impact of patient engagement can best be determined by applying a set of linked measures. 18 Theory-based evaluation approaches (eg programme evaluation, realist evaluation) 27 have been recommended for evaluating patient engagement, 28,29 as these approaches also focus on the conditions necessary for effectiveness. 25,27 Therefore, we drew on theory-based evaluation approaches for the development of the monitoring and evaluation (M&E) framework. Although the primary objective of the project was to develop a framework to demonstrate the 'return on the engagement' for all players, we decided to also include metrics that assess whether or not the conditions for reaching the intended impacts are in place to stimulate reflection and continuous improvement of engagement practices.
For example, we adopted the importance of context in evaluating outcomes from a realist evaluation approach. Furthermore, we used a logic model approach, often used in programme evaluations, to identify metrics that relate to each other (set of metrics).
A logic model explains how activities are understood to contribute to a chain of results that produce impacts. In addition, we involved all stakeholders in the framework development process to ensure identification of a wide variety of impact metrics (eg impacts for research, people and organizations). • Clarification of terminology and language to be used.

| ME THODS
The working group was involved in all phases of the project (as described below). They provided feedback on documents and versions of the framework, co-analysed case study data, were involved in writing publications and in other dissemination activities. All working group members had a say in the framework development process by participation in monthly working group calls, workshops, polling/voting tools and by providing written feedback. The researchers facilitated the decision-making process by providing options, draft documents, questions, theoretical guidance and overall coordination of the research.

| Terminology
A variety of terminology is used in the literature. In Table 2, we provide an overview of terms and definitions used by the authors throughout this paper. This approach is suitable for evaluating singular interventions where there is a direct causal influence of the intervention, in a setting that can be controlled (to reduce confounding) and in which one group can be compared to another.

Programme evaluation
Is the programme or initiative designed in a way it can achieve its intended outcomes?
• Descriptive study design This approach is suitable for evaluating complex interventions, in a setting where a series of outcomes lead to the final impacts.
Realist evaluation 'What works in which circumstances and for whom?' • Single or multiple case study design This approach is appropriate for evaluating initiatives or programmes that seem to work but where 'why', 'when', 'how' and/or 'for whom' is not yet understood, in a setting where the context influences how an intervention is implemented and how actors respond to it (or not).

| Study design and approach
We used participatory action research (see Figure 1) to develop and refine the M&E framework, as described by Van Mierlo et al 2010. 30

Patient engagement
The effective and active collaboration of patients, patient advocates, patient representatives and/or carers in the processes and decisions within the medicines lifecycle, along with all other relevant stakeholders when appropriate. 51 This may include activities at specific decision points and/or on-going collaborations throughout the R&D cycle.

Monitoring
The formative evaluation of patient engagement practices in order to strengthen these practices.

Evaluation
The 'systematic acquisition and assessment of information to provide useful feedback about …' patient engagement practices. 52 Summative evaluation examines the effects of patient engagement practices on various measures including outcomes, impact and cost-benefit.

Value
The benefits of patient engagement (in relation to the direct and indirect costs) for individuals and organizations (thereby acknowledging that value can have different meanings to different people, for example value for money, value for time, value for health).
Research priority-setting Any process aimed at constructing priorities or agendas for health research and medicine development, that raises awareness and change the way research funding is allocated.
Design of clinical trials Any process aimed at the development or design of clinical trials for medicine development at any stage.
Early dialogues with regulators and Health Technology Assessment (HTA) bodies Any process in which medical technology developers communicate with regulatory bodies and/or HTA bodies prior to health technology assessment. Early dialogue can happen only with regulators (eg scientific advice), jointly with regulators and HTA bodies (to discuss data requirements to support decision-making on marketing authorization and reimbursement simultaneously) or only with HTA bodies (eg EUnetHTA multi-HTA dialogues) Components By components, we mean dimensions or criteria used for monitoring and evaluation, which need to be translated into measures called 'indicators' or 'metrics', which may be quantitative and/or qualitative. By subcomponents, we mean categories with metrics related to a specific component.

Set of metrics
An agreed group of metrics that relate to each other and align to a certain context, with methods/tools to collect the information.
'Forward' and 'backward' metrics By 'forward' and 'backward', we mean metrics that link to the given metric and would be measured before or after the given metric is measured.
Linked metrics By 'linked' or 'related', we mean additional metrics that complement the given metric. Metrics which are relevant to measure in combination with other metrics (eg to improve understanding of how impact can be reached).

Short and long-term results
With short-term results, we mean outcomes that can be measured directly after engagement activities. With long-term results, we mean impacts that become evident in the years after engagement activities. This could be during or after an active collaboration or research study.

Reflexivity
The capacity to reflect upon (social) practices, assumptions, beliefs and values and to challenge and change those that are undesirable through enquiry, dialogue and learning.
multi-stakeholder working group was then sought and resulted in version three, which entered the test phase.

| Phase 2: Framework applicability test
The aim of the test phase was to apply the draft framework to realworld patient engagement initiatives in the process of medicine development. We used the M&E framework to validate the identified (sub)components and to select and test suggested metrics in practice. Our approach to this was to use a case study design (Yin 2018) 31 in which researchers each worked with separate cases on Phase 2 of the study (see Figure 1).

Participant case studies
Partners of the IMI-PARADIGM consortium agreed to contribute and participate with case studies of patient engagement initiatives in medicine development. In total, 24 patient engagement initiatives were included as cases (see Table 3).

Data collection
Applicability test data were collected between June 2019 and May 2020. Case study contributors were asked to describe their patient engagement initiative per component of the M&E framework (see Figure 2). Furthermore, they were asked to select appropriate metrics per component of the framework for M&E of their initiative(s). They could select metrics identified in the design phase, metrics they were already measuring or suggest new metrics. Their input was analysed by their assigned member of the research team (LV, TF, NG, CG, SF, LD or TS). Reflection meetings were held between the researchers and the case contributors to discuss the framework, metrics and their applicability. A tailored 'set of metrics' for each case was developed using an iterative approach. Cases were coded to ensure anonymity and data were stored by the research team on the VU University's encrypted platform.

Data analysis
A descriptive, qualitative approach was used to analyse the data per component of the framework in a cross-case analysis. 31 A case report was sent to each case for discussion and validation. The research team co-analysed the metrics from all cases; discrepancies were discussed and similar metrics were merged. Next, the working group members involved in writing this article co-analysed anonymized case data. Lessons learned from the cases for framework and metrics development were discussed. Furthermore, the applicability of the framework was reviewed and any changes that derived from the case studies were discussed during the consensus and alignment phase.

| Phase 3: Framework consensus and alignment
The aim of the consensus and alignment phase was to develop agreed 'sets of metrics'. An online consensus-building workshop was held with all stakeholder groups. All working group members were invited (n = 32 participated). It was agreed that 'sets of metrics' would be created per patient engagement objective as identified in the case studies. Participants prepared in advance by reading a document with

| RE SULTS
The framework (

| Objectives of patient engagement
Every initiative is unique and has its own objectives describing the aims the initiative intends to achieve. Impacts are related to objectives in terms of the intended long-term results achieved. However, impacts could also include unintended consequences which are important to elicit. Seven overarching objectives of patient engagement were derived from the case studies: •

| Preparations
This subcomponent includes metrics that assess the accessibility and preparedness for patient engagement. It is believed that those who are well prepared can participate more effectively, resulting in better outcomes. Therefore, some cases (n = 4) decided to monitor the accessibility of preparatory materials and/or feeling of preparedness of those involved.

Nr of cases (n = 24)
Initiated

| Activities/Process
The process gives an indication of how the implementation of patient engagement is progressing and can elucidate areas for improvement.

| Management
By management, we refer to how engagement activities are facilitated, for example, metrics that assess 'satisfaction with the moderation' or 'satisfaction with support from activity organisers'. These metrics help organizers improve the facilitation of patient engagement activities, and thereby indirectly the impacts.

| Interactions
With interactions, we are referring to the quality of interactions during the process, for example metrics that assess 'feelings of trust', 'honesty', 'transparency', 'respect', 'give and/or take relationship' and 'opportunity to contribute', factors that indirectly influence learnings, changes and impacts.

| Satisfaction
By satisfaction, we refer to the overall experience of those involved with the engagement process. Measuring satisfaction with patient engagement activities is seen as important by most cases (n = 17) as this is considered a predictor for willingness to continue collaboration and the overall value of patient engagement.  Learnings and changes metrics could also be tailored to a patient engagement initiative. For example, the objective of Case 3 was to get access to medicines for people co-infected with two viruses; they tailored metrics to specific learnings, for example 'awareness of co-infected people's situation' and specific changes, for example 'number of clinical trials changing inclusion and exclusion criteria'.

| Impacts
If the acquired learnings are put into practice, then long-term im-

| Study ethics and inclusiveness
This subcomponent includes metrics that assess the diversity and accessibility of studies. For example, some patients suggested assessing the impact on inequalities in research, such as the 'number of trials including under-represented groups'.

| Study quality and efficiency
This subcomponent includes metrics that assess the speed of studies and influencing factors such as study participants' experiences of trials. A number of cases selected metrics related to study quality and efficiency as their overall objective is to improve the efficiency of medicines R&D. Some cases tried to make a comparison between enrolment rates of trials that engaged patients in the planning phase against expected rates for the specific trial, or tried to compare trial sites that incorporated patient recommendations and those that did not implement this recommendation within the same trial. Various contextual factors influence trial efficiency (including ethics, regulations, trial sites); therefore, it is argued that efficiency cannot unequivocally be linked to patient engagement. However, study participants' experience is seen as a predictor for the efficiency of trials by cases. The experience of study participants, in turn, may depend on the level of influence patients had on the protocol. Some cases also selected metrics which can be seen as predictors for recruitment rates, such as patients' understanding/accessibility of information materials and consent form or patients' willingness to participate in clinical trials.

| Quality of evidence and uptake of products
This subcomponent includes metrics that assess the quality and availability information used for decision-making by regulatory bodies and patient communities as well as the accessibility and usability of products. For example, industry-led cases which involved patients in early dialogues with regulators selected metrics related to the quality of evidence such as 'percentages of trials with patient-reported outcomes measures' and 'degree to which patient engagement will help to demonstrate the value of a product to regulatory bodies'.
Regulatory bodies seemed interested in measuring 'the quality of the feedback provided by patients' and 'the quality of scientific advice provided by regulators/HTA in terms of relevance to patients' needs'.

| Empowerment
This subcomponent includes metrics that capture the process of be-

| Developing a tailored framework
The wide variation in selected metrics per component of the framework shows how difficult it is to uniformly define or standardize metrics for evaluation of patient engagement.

Policy context factor (Case study 3)
There were no regulatory requirements or recommendations for studying therapies in people living with two different viral infections affecting their health at the same time (co-infection); co-infection was only viewed as a risk factor for those people to participate into clinical trials.
Therefore, there was no incentive for industry to provide access to clinical trials for co-infected people, decreasing the chance of implementing patient recommendations related to access issues. (barrier).

Institutional context factor (Case study 17)
The company has a long history of patient engagement for [disease] and is therefore familiar with the common issues faced by patients in this therapeutic area. Consequently, the learnings and changes, and potentially the impact of the patient engagement initiative may be modest. (barrier).

Decision-making context factor (Case study 14)
The trial protocol was already approved when discussing potential recruitment challenges and retention with patients; therefore, patients had little influence on the study focus, study population, design of the trial etc Thus, the impact on study participant experience and ultimately the efficiency of the trial, may be limited. (barrier).

Community context factor (Case study 6)
There is a well-established European patient organization for involvement of patients with [disease] in the drug development process. This facilitates (early) access to a diverse group of patients, influencing the potential learning, changes and impacts. (enabler).

| D ISCUSS I ON
To address the need for metrics for M&E of patient engagement in medicine development, the PARADIGM project created a framework with metrics. In this section, we reflect on the framework and its value, looking at four criteria (a) evidence-base, (b) multi-stakeholder perceptions, (c) comprehensiveness, and (d) utility. In addition, we reflect on the methodological strengths and limitations of this participatory research project.

| Reflections on the framework and its value
The principal strength of the framework is that it is evidenceinformed and iteratively developed with all relevant stakeholders.
We integrated literature and existing frameworks [33][34][35][36] as well as quality guidance documents such as PFMD Quality Guidance, 37 the TransCelerate Toolkit 38 and EUPATI guidance 39 with case studies.
We have incorporated dimensions of realist evaluation in order to

Box 2 A case example Background
This case was initiated by a pharmaceutical company, bringing together industry staff and patients to discuss issues regarding patient experience in clinical trials.
Step 1: setting objectives The objective of the case was to improve experiences of patients participating in clinical trials to ultimately speed up medicines' development. The purpose of M&E was to 1) prioritize patient engagement activities in order to spend the available resources wisely, 2) learn about and educate colleagues on when and how patient engagement adds value to clinical trials, 3) provide feedback about learnings and the resulting outcomes and impact on those involved and 4) improve collaborations and other patient engagement activities.

Steps 2 and 3: theory of change, tailored set of metrics with methods/tools
There is a burden for patients who participate in clinical trials (context).
[Company] has chosen to engage patients as it is believed it can make the trials more patient-friendly (input). Through patient advisory board meetings, [company] gains insight into patient needs (activities/process). Patients provide input for trial design decisions both before and after protocol approval for example on visit structure, frequency, materials, outcome measures. The activities lead to new insights (eg about patient experiences) and changes in trial designs (learnings and changes), which lead to satisfactory patient experiences in trials, higher recruitment and retention in comparison against expected for the clinical trial (impacts), and therefore faster approval (overall objective).
The initial M&E plan focussed heavily on measuring impact. While data from the Study Participant Feedback Questionnaire (SPFQ) can evaluate whether the initiative reaches its objectives, monitoring proxy metrics give insight into whether or not the initiative is working the way it is intended and whether the acquired insights are put into practice to achieve impacts. By using the framework, the initiative created a tailored set of metrics (see Table 4). A pre-and post-meeting survey was developed by the company to collect information from industry staff and patients involved. Other methods are also used including interviews and logbooks.
Step 4: reflexive monitoring A feedback loop has been created. The outcomes of each patient advisory board meeting are discussed with the research team (eg learnings and expected actions), then followed up every six months to continuously track the implementation of these actions. The patients who participated receive an outcome report summary directly after the meeting and an update about the actions implemented every six months. The company plans to implement the SPFQ survey at the start, during and at the end of trials. The survey may need to be adjusted to link the study participant experiences to specific changes made based on patients' input. The M&E results have been used to educate colleagues, inform meeting agendas and enhance the patient engagement initiative.

Reflection on the application of the framework and its value
The industry partners selected metrics during an interactive in-company workshop. perceived added value of patient input (impact); and the number or percentages of projects with patient engagement (impact). The framework could be used at an organizational level; however, a generic set of metrics that can be used at an organizational level has not been created within this project and should be further explored.
The selected metrics could be transferable to engagement initiatives in other settings (eg health research or care), but specific (impact) metrics probably require adaptations. However, as Greenhalgh and colleagues found, the relational work involved in planning monitoring and evaluation may be more important than any subsequent framework itself. 22 Our findings align with this conclusion; the interaction and trust built between actors involved in co-creating a framework that can be tailored to suit the needs of different groups is likely to be more locally relevant and used than a one-size-fits-all framework. Generating a tailored framework together (eg through a multi-stakeholder workshop) will influence participants (eg motivation), knowledge (eg incorporation of research, experiential and contextual knowledge) and the process of implementation (eg ownership, testimony of end-users). 22 We noticed that some case study participants struggled to understand the linkages between components of the framework, in particular how impacts were triggered, blocked or modified by contextual factors. This suggests that those working in the context may not 'see' these factors until an external actor points them out, also known as the cultural phenomenon 'fish don't know they are in the water' described by Derek Sivers. 41 The application of the framework helps to shine a light on the 'water'. The interpretation of findings requires training and guidance to ensure that contextual factors and metrics relevant to the theory of change are not overlooked.
This suggests that applying the framework is a capability that stake-

| Methodological strengths and limitations
A strength is that we used an iterative multi-stakeholder approach to create the framework. The framework is in line with other recently proposed metrics 21,44 though more inclusive in its scope.
Ideally, patient engagement occurs throughout the full medicines' development cycle, as an on-going activity. A limitation of our research is the limited number of cases that applied the framework to research priority-setting or early dialogue decision-making points as there is less patient engagement in these areas. Therefore, the framework may be more focused on metrics for the decision-making point related to design of clinical trial. We used literature to complement the case studies and had wide discussions with partners about metrics for other decision-making points.
The framework, being developed through a multi-stakeholder approach, is prone to bias in relation to those stakeholders represented in the process. Since patients and regulatory bodies were less well represented than industry, the framework may include more metrics relevant to industry. Stakeholders were mostly Western-Northern European-based, and there was limited participation from Central-Eastern Europe and young people. To correct for this, an additional multi-stakeholder workshop has been held for these groups to gather their perspectives and any new metrics that derived from this workshop were included in the framework.
The sets of metrics have hardly been tested in practice because very few cases had the opportunity to measure metrics during the project. Therefore, limited insights have been gathered about optimal methods or tools for measurement. However, we created an overview of sample questions, methods and tools drawn from literature and evaluation documents collected which is a starting point for measurement. The framework could also be used to improve the comprehensiveness and rigour of existing measurement tools. The co-creation of appropriate measurement tools requires further investigation and flexible approaches. Mixed methods and multiple tools are needed as one survey will not be able to capture the complexity and impact of patient engagement. 45 Longitudinal research is needed in this area as it takes time before impacts become evident.
Further application of the framework is needed to co-create sets of metrics for different collaborations and contexts. This may result in additional or new proposed metrics and insights. The application of locally relevant frameworks is necessary to better understand in which contexts certain practices lead to valuable patient engagement and why initiatives fail or succeed. To that end, we aim to develop an online interactive tool which enables users to tailor the framework to their situation. This could result in co-analysis of data gathered by different initiatives and ultimately stimulate continuous improvement of engagement practices and the creation of an evidence library that reinforces the need for a culture shift towards a patient-centred R&D system.

| CON CLUS IONS
Monitoring and evaluation of patient engagement can enhance meaningful and sustainable patient engagement. The created M&E framework helps to monitor progress and demonstrate impact.
There is a large variety in the purposes of M&E and the objectives of patient engagement; accordingly, metrics vary per initiative and stakeholder group. Evaluation studies can help to understand in which contexts certain practices lead to valuable patient engagement and why initiatives fail or succeed. We argue that the value of patient engagement can best be understood by measuring metrics related to all components of the M&E framework using a multistakeholder, reflexive approach.

ACK N OWLED G EM ENTS
We want to thank all PARADIGM work package 3 partners for their contributions to the project. Furthermore, we would like to thank Nicole Goedhart, Callum Gunn, Léa Darvey and Laiba Husain, for their contribution to the design of this study and collection and analysis of data, and Nick Fahy for workshop facilitation. We would also like to thank all workshop and case study participants/contributors.

CO N FLI C T O F I NTE R E S T
None declared.