Evaluating the “return on patient engagement initiatives” in medicines research and development: A literature review

Abstract Background Showing how engagement adds value for all stakeholders can be an effective motivator for broader implementation of patient engagement. However, it is unclear what methods can best be used to evaluate patient engagement. This paper is focused on ways to evaluate patient engagement at three decision‐making points in the medicines research and development process: research priority setting, clinical trial design and early dialogues with regulators and health technology assessment bodies. Objective Our aim was to review the literature on monitoring and evaluation of patient engagement, with a focus on indicators and methods. Search strategy and inclusion criteria We undertook a scoping literature review using a systematic search, including academic and grey literature with a focus on evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut‐off of publications after July 2018. Data extraction and synthesis Data were extracted from 91 publications, coded and thematically analysed. Main results A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for the evaluation of benefits and costs of patient engagement. Discussion and conclusions Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures.


| INTRODUC TI ON
There is increasing consensus among stakeholders that patient engagement in research and development (R&D) of medicines provides benefits for patients, researchers, industry, regulatory bodies, payers and policy makers. [1][2][3] The case for patient engagement is often presented in ethical and political terms referring to fairness, transparency and accountability. 4,5 Methodological arguments consider the experiential knowledge of patients-acquired by their personal experience of a condition-as valuable to improving the quality and relevance of the research. [6][7][8] The inclusion of patients in decision making about the development of new innovative medicines is a substantial change, requiring time and (financial) commitments from researchers, industry and patients. 2,4 Despite efforts to promote and support patient engagement in research, the prevalence of patient engagement in medicines research and development remains low. 9,10 Patient engagement has not been fully embedded in the health research system, partly because it is not yet clear to all involved what the added value is. 11 To address this need, an increasing number of studies aim to evaluate the impact of patient engagement, underscoring the growing interest in the "return on engagement," or why it makes sense for patients, society and industry. 2,12 The perceived value of patient engagement practices can vary for different stakeholder groups, and the metrics of interest will therefore differ accordingly. 13 For example, for researchers and industry partners it might be about evidence that patient engagement improves the quality and efficiency of research and the uptake of findings, whilst for patients it might be more about influencing the R&D agenda to develop medicines for unmet needs.
Some argue that evidence is needed to justify the 'business case' for engagement. This could also help to establish a financial model to support engagement. 2,14,15 Evaluation could also define the genuine value of patients' contributions, contributing to valued rather than tokenistic inclusion for appearances' sake. 16 There is also some resistance; people are concerned about assessing impact too simplistically. Some question whether it is fair to evaluate the value of patient input in isolation, and not that of others such as key scientific leaders, 12 not least because it may be the synergy of working in partnership that produces benefit. 17 As mentioned by Staniszewska, it is important to recognize that "any form of measurement sits within a political or ideological context that cannot be ignored." 13 Nonetheless, there is a desire to assess the impact of patient engagement, to demonstrate better decision making, avoidance of previous errors and a contribution to continuous efficiency and quality improvement. 15,16,18 Despite this perceived importance of assessing the return on patient engagement, little is known about "what" to evaluate, and even less about "how." [19][20][21] A number of researchers have tried to assess how patient engagement makes a difference. 3,5,8,12,[22][23][24][25][26][27] Although there is no standardized way to assess the impact of patient engagement, very similar benefits, costs and challenges are reported in literature reviews. 4,17,19,20,[28][29][30][31][32] The current assessment of patient engagement is considered weak, partly because much of the evidence is mainly anecdotal 17 and because methods used have not captured the complexity, context or mechanisms of change. 17,33 Previous studies have identified a number of gaps in the literature and identified challenges such as the delayed nature of impact, inconsistent terminology, absence of accepted criteria for judging the success or quality of research, no agreed evaluation methods or framework and few reliable measurement tools. The absence of a control group-identical research carried out without patient engagement-is problematic too, particularly in an area of science where direct comparison to evaluation approaches or outcomes associated with patient engagement. No date limits were applied other than a cut-off of publications after July 2018.
Data extraction and synthesis: Data were extracted from 91 publications, coded and thematically analysed.
Main results: A total of 18 benefits and 5 costs of patient engagement were identified, mapped with 28 possible indicators for their evaluation. Several quantitative and qualitative methods were found for the evaluation of benefits and costs of patient engagement.

Discussion and conclusions:
Currently available indicators and methods are of some use in measuring impact but are not sufficient to understand the pathway to impact, nor whether interaction between researchers and patients leads to change. We suggest that the impacts of patient engagement can best be determined not by applying single indicators, but a coherent set of measures.

K E Y W O R D S
evaluation, framework, impact, literature review, medicines development, metrics, patient and public involvement, patient engagement, patient participation, research an existing standard is routinely demanded. 8,16,34,35 It is argued that to build an evidence base, some level of consensus on measurable impacts is needed, whilst others state that the outcomes of engagement cannot easily be quantified. 13,30,36 In sum, it remains unclear what methods can be best used to evaluate patient engagement.
To address the need for means of determining the "return on engagement," the aim of this paper was to scope, review and summarize the literature on monitoring and evaluation of patient engagement.
Many publications present useful guidance for conducting patient engagement and assessing the quality. [37][38][39] Evaluation studies focus mainly on qualitative methods and only occasionally link to specific outcomes. 12,33,40,41 [42][43][44] Scoping reviews are similar to systematic reviews in that they follow a structured search process; however, they are performed for different reasons. 45 Our aim was not to answer a precise question addressing the effectiveness of a certain practice, as in a meta-analysis, but to provide an overview of the breadth of the available literature about evaluating patient engagement.
Whilst the review is concerned with patient engagement at the three key decision-making points, we used broader search limits to ensure capture of related publications in other areas of health research. One of the challenges was the variety of terminology. For example, the words "measure," "metric" and "indicator" are often used interchangeably and their definitions may vary, if they are stated at all. Furthermore, the terms used for "patient engagement" differ globally. In this paper, we use the term patient engagement; in our search, we included terms such as public involvement, patient participation, community engagement and user involvement. In Table 1, we provide definitions of terms developed by the authors and as we used them in this review.

| Search methods
Prior to the database search, we did a search to identify a tentative Accordingly, with the assistance of a specialist librarian, we searched CINAHL, Embase, Medline, PsychINFO and PubMed databases for peer-reviewed published literature. The following key words were used "patient engagement" combined with « AND» "research" « AND» "outcomes," including a variation of terms combined with « OR». An overview of all search terms can be found in Table 2.

| Study selection and data extraction
Two researchers (TF, LV) independently screened all items' title and abstract. To ensure inter-rater reliability, items were marked for inclusion or exclusion with each researcher's initials, discrepancies were discussed and consensus reached. Both researchers read all the selected items in full and followed up references for final inclusion. At this stage, further exclusions were made of items that did not include methods for evaluating outcomes and/or impact of patient engagement practices in health research or health technology assessment-discrepancies were discussed, and consensus agreed for final inclusion in the data extraction and analysis. Figure 1 VAT eT Al.
When available, context and process criteria were also included.
Both researchers extracted data independently from 30% of the scientific articles and then compared their findings to agree the approach to data extraction. Thereafter, all the peer-reviewed published papers' data were extracted by LV and the grey literature by TF and LV.

| Analysis
Data were thematically analysed following Braun and Clarke's approach. 49 To achieve the summary, we coded data using the review question, aim and objective as included on the data extraction sheet.

| Consultation and validation
The preliminary results of the review were presented and discussed  that patients were involved in the study as partners. Table 3 provides an overview of the characteristics of included documents.

| RE SULTS
In this section, we present the findings of our review, first considering the three decision-making points, which were relevant to our search. Not all reported benefits and costs could easily be linked to one decision-making point. These referred to overall benefits and costs for stakeholders and general costs or challenges for research and development. Therefore, we report them separately.
Additionally, reported benefits and costs were omitted where they related to other phases of the research process (such as interpretation of research findings or dissemination of results).

| Benefits, costs and challenges for research and development
A total of 18 benefits and five costs of patient engagement at three R&D decision-making points were identified. These were grouped into 11 domains and mapped with 28 possible indicators for their evaluation. Tables 4 and 5 provides an overview of indicators per domain. Please refer to Appendices S3-S5 for more detailed indicators, evaluation methods and tools.

| Benefits of patient engagement in research priority setting
Literature suggests that patient engagement in research priority setting has several benefits. We identified four unique benefits and nine possible indicators. We clustered the benefits into three domains: usability benefits, societal benefits and funding benefits.
Usability benefits refer to impact on the topic generation and prioritization process, for example more relevant topics and priorities based on patients' needs 3,4,15,17,20,22,23,29,30,[50][51][52][53][54][55]  In the literature, quantitative methods are used to collect information about the perceived importance of studies by patients, the perceived influence of stakeholders in research priority setting, 23,59 or to compare academic and lay scores assigned to research proposal evaluation. 60 For example, studies suggest rating the importance or influence of partners in developing the research topics. 23,59 Qualitative methods are used to explore the relevance of research topics and how patients' experiential knowledge helped shape the research question. 30 The Patient-Centered Outcome Institute (PCORI) uses mixed methods (survey, focus groups, database review) to explore the perceptions incorporated into the topic selection process and the kinds of research gaps documented as important to patients and other stakeholders that were not previously identified. 61 Quantitative methods could also be used for comparison of academic and lay scores assigned to research proposals. 60 Qualitative methods are suggested for exploring similarities and differences in research priorities. 15

Usability benefits (1) Examples of indicators related to usability benefits (total: 6)
More relevant research topics and priorities, based on patients' needs 3,4,15,17,20,22,23,29,30,[50][51][52][53][54][55] Rating of influence of patients and other stakeholders 61 Rating of relevance or importance of studies 23,59 Perceptions or degree of contentment/satisfaction with the topic generation and prioritization process 96 Similarities and differences in research priorities between stakeholder groups 15 Types of research gaps reported that were not previously identified 61 Perceptions on how patients' experiential knowledge helped shaped the research question 30 Research questions, hypothesis, interventions and medical technologies become more relevant and usable for patients 24,30

Societal benefits (2) Examples of indicators related to societal benefits (total: 3)
More appropriate resource allocation, based on patients' needs 30 Comparison of academic and lay scores assigned to research proposals 60 Perceptions of public influence on funding decisions 60 Indicators of dynamics in the panel discussion 61

Ethical benefits (4) Examples of indicators related to ethical benefits (total: 1)
More appropriate, inclusive and sensitive research design 8,17,29,30,52,55,58 Number of studies that had success in gaining ethics approval 12

Methodological benefits (5) Examples of indicators related to methodological benefits (total: 4)
More appropriate wording and timing of research instruments and interventions 17

Study quality benefits (6) Examples of indicators related to study quality benefits (total: 7)
Improved recruitment and retention 23,24,29,40,69 Recruitment rates 40,69,70 Number of study participants who dropout for reasons other than adverse reactions 59 Increased diversity of study participants 66 Recruitment and retention rates among hard-to-reach population, level of diversity 61 Improved trial experience/satisfaction by study participants 2,80 Rating or explore feelings of satisfaction among study participants 15,70 Rating convenience of study visits and procedures by study participants 59 More adherence to the research protocol 93 Number of protocol amendments 59 Faster study completion 2,23 Number of studies completed within a particular timeframe 3,61

Instrumental benefits (7) Examples of indicators related to instrumental benefits (total: 1)
Higher accuracy in measuring needs and preferences of patients 71,72 Perceptions on how patient input was used and added value for assessment 75,76 Better quality of assessment (in terms of relevance and reliability to local context) 71,72

Study uptake benefits (8) Examples of indicators related to study uptake benefits (total: 2)
Uptake of evidence/approval by regulators and HTA bodies 2,73 Time to approval/response of regulators 52 Changes in the proportion of drugs recommended for reimbursement 36

| Benefits of patient engagement in regulatory processes and health technology assessment (HTA)
We identified five unique benefits of patient engagement in regula-

Non-financial costs (10) Examples of indicators related to non-financial costs (total: 2)
Biases in recruitment or findings 24,67 Perceived negative impacts of patient engagement for research and development 24 Total hours spent on engagement 24 Scientific and ethical conflict in protocol design 20 Power struggles 20 Increased time 20

Financial costs (11) Examples of indicators related to financial costs (total: 1)
Increased costs 20 Total monetary costs of engagement for research and development 24 and can lead to power struggles between researchers and patient partners. 20 Furthermore, stakeholders have raised concerns that engaged patients may want to see their clinical trials succeed, and as a result, these patients may bias the study findings. 59 It was also reported that a number of clinical research professionals fear that patient centricity is pushing them to discard traditional practices, including the use of blinded, randomized controlled clinical trials. 59 Methods to assess costs include qualitative methods to gather insights into the perceived effort of engagement as well as a quantitative method to gather insights into financial costs. For example, the costs and consequences framework developed by Blackburn et any costs related to arrangement and planning (for instance changed shifts at work or arranged care for a relative). 24 Log sheets are also used to gather insights into time and costs. 27 Open questions are used to gather insights into (non-financial) negative impacts. 24  Table 6. Please refer to Appendices S1

| Benefits, costs and challenges for stakeholders
and S2 for more detailed information on benefits, costs and challenges for patients and other stakeholders.
Multiple tools have been developed to assess the benefits and costs for stakeholders. The Evaluation Toolkit is a resource designed for practitioners of the health sector, produced after the completion of a rigorous systematic review of patient and public engagement evaluation tools. 78 Boivin et al reviewed the tools and concluded that most tools were designed to collect information from patients and the public; very few instruments measure the perspectives of other stakeholder groups. The authors of the review reported that the outcomes of patient engagement were least often evaluated (55.6% of the tools), in contrast to the engagement process and context. The most common focus of tools that measure outcomes was on perceived, self-reported impacts. Methods are qualitative (eg interviews, focus groups) and quantitative for perceived self-reported benefits (eg surveys using Likert scales). Self-administered questionnaires and surveys were the most common types of tools identified. 21

| D ISCUSS I ON
To address the need for means of determining the "return on engagement," the aim of this paper was to review the literature on monitoring and evaluation of patient engagement. This review identified a range of benefits, costs and challenges that patient engagement can have on R&D and describes several indicators associated with their monitoring and evaluation. In addition, we summarized the overall reported benefits, costs and challenges for stakeholders involved in patient engagement initiatives. In this section, we reflect on the indicators and methods found in this review and consider the review's methodological strengths and limitations.

| Strengths and limitations of this review
To our knowledge, this is the first literature review that attempts to capture the existing publications about the evaluation of patient engagement practice as it relates to medicines development. It both maps outcomes and impacts of patient engagement with suggested measures for each decision-making point in R&D.
Very few publications refer to costs or negative impact of engagement, compared with positive findings. This may be because people tend not to report negative outcomes and impacts despite their being just as important. There were very few studies that considered patient engagement in the HTA process, and only, three publications were authored by (and for) the pharmaceutical industry. Furthermore, of the papers included in our review, very few reported that they had involved patients; therefore, the conclusions derived from the studies may be based on the perspectives of researchers. For this review, a meeting was held to discuss preliminary findings with a broad range of stakeholders in our project and the co-authors of this paper work for patient representative groups and industry. We therefore feel that our findings may be considered relevant to a broader audience than a predominantly academic one.
Our focus on the measurement of impact of patient engagement in the development of medicines has resulted in several limitations to our review. Because this is a scoping review rather than a systematic review, we may have missed relevant articles. Our search focused on titles and abstracts of publications and three decisionmaking points, which means that some articles (eg related to other time points) have been excluded. We specifically searched for outcomes and impact of patient engagement in the R&D of medicines; therefore, our paper does not include context or process indicators, or the indicators per stakeholder group. Furthermore, we cannot draw hard conclusions about the relationship between input, outcomes and impact with respect to the benefits and costs for the people and organizations involved in patient engagement. Finally, we had to exclude articles not published in English. Whilst we are aware that most publications on this topic are written in English originating from the UK and North America, we acknowledge that we may have missed relevant publications in other languages.

| CON CLUS IONS
For patient engagement in the development of medicines to become standard practice at the key decision-making points of priority setting, clinical trial design and regulatory and HTA processes, benefits need to be demonstrable to all stakeholders. This literature review has mapped benefits, costs and challenges with indicators in current literature. Discrete tools and methods for evaluation are less apparent, as is evidence of their application. The approaches to evaluation we found are largely qualitative, and our review suggests that there are few quantitative tools and no standardized approaches to assessing the outcomes and impact. The reported costs, challenges and benefits are largely congruent, with agreement that there is a need for consensus-based monitoring and evaluation frameworks that include metrics.
We suggest that the development of a coherent set of measures warrants further investigation and that the benefits, costs and challenges of patient engagement for all stakeholders should be given more consideration (rather than the current focus on benefits for research). To this end, we will co-develop and test an evaluation framework with stakeholders using a reflexive monitoring approach in real-life cases of patient engagement in medicines research and development.

ACK N OWLED G EM ENTS
We want to thank Nicole Goedhart from the Athena Institute for assisting with the analysis. We also would like to thank Carina Pittens from the Athena Institute for reviewing the first drafts.
We are grateful for the support of Nia Roberts, librarian at the University of Oxford. Furthermore, we very much appreciate the feedback from Joanna Crocker from the University of Oxford and María José Vicente Edo from Instituto Aragonés de Ciencias de la Salud.

CO N FLI C T O F I NTE R E S T
None declared.

DATA AVA I L A B I L I T Y S TAT E M E N T
The data that support the findings of this study are available from the corresponding author upon reasonable request.