Meaningful patient and public involvement in digital health innovation, implementation and evaluation: A systematic review

Abstract Introduction The importance of meaningfully involving patients and the public in digital health innovation is widely acknowledged, but often poorly understood. This review, therefore, sought to explore how patients and the public are involved in digital health innovation and to identify factors that support and inhibit meaningful patient and public involvement (PPI) in digital health innovation, implementation and evaluation. Methods Searches were undertaken from 2010 to July 2020 in the electronic databases MEDLINE, EMBASE, PsycINFO, CINAHL, Scopus and ACM Digital Library. Grey literature searches were also undertaken using the Patient Experience Library database and Google Scholar. Results Of the 10,540 articles identified, 433 were included. The majority of included articles were published in the United States, United Kingdom, Canada and Australia, with representation from 42 countries highlighting the international relevance of PPI in digital health. 112 topic areas where PPI had reportedly taken place were identified. Areas most often described included cancer (n = 50), mental health (n = 43), diabetes (n = 26) and long‐term conditions (n = 19). Interestingly, over 133 terms were used to describe PPI; few were explicitly defined. Patients were often most involved in the final, passive stages of an innovation journey, for example, usability testing, where the ability to proactively influence change was severely limited. Common barriers to achieving meaningful PPI included data privacy and security concerns, not involving patients early enough and lack of trust. Suggested enablers were often designed to counteract such challenges. Conclusions PPI is largely viewed as valuable and essential in digital health innovation, but rarely practised. Several barriers exist for both innovators and patients, which currently limits the quality, frequency and duration of PPI in digital health innovation, although improvements have been made in the past decade. Some reported barriers and enablers such as the importance of data privacy and security appear to be unique to PPI in digital innovation. Greater efforts should be made to support innovators and patients to become meaningfully involved in digital health innovations from the outset, given its reported benefits and impacts. Stakeholder consensus on the principles that underpin meaningful PPI in digital health innovation would be helpful in providing evidence‐based guidance on how to achieve this. Patient or Public Contribution This review has received extensive patient and public contributions with a representative from the Patient Experience Library involved throughout the review's conception, from design (including suggested revisions to the search strategy) through to article production and dissemination. Other areas of patient and public contributor involvement include contributing to the inductive thematic analysis process, refining the thematic framework and finalizing theme wording, helping to ensure relevance, value and meaning from a patient perspective. Findings from this review have also been presented to a variety of stakeholders including patients, patient advocates and clinicians through a series of focus groups and webinars. Given their extensive involvement, the representative from the Patient Experience Library is rightly included as an author of this review.


| INTRODUCTION
Patient and public involvement (PPI) is frequently cited as a moral obligation, 1-3 with increasing regularity in mandatory policies across a variety of domains including healthcare design and delivery, 4 research, 5-7 regulation, 8 education 9 and perhaps more recentlydigital health innovation. While consensus on what to call PPI is not yet available, 10 its potential benefits are widely acknowledged, including enhanced relevance, quality and authenticity [11][12][13][14][15] ; generation of alternative and innovative ideas 13,14,16,17 ; stakeholder empowerment 14,[18][19][20] ; emancipation 21,22 ; democratization 23-25 ; and enhanced sustainability. 11,16 However, while the COVID-19 pandemic arguably led to an unprecedented increase in both the innovation and the implementation of digital health technologies, [26][27][28] this was often at the expense of meaningful involvement, 29,30 with PPI largely still 'seen as "nice to have" but not essential'. 30,p.30 For example, as stated by Richards and Scowcroft,30 'The COVID-19 pandemic saw statutory policy commitments to patient and public involvement and shared decision making in health systems abandoned, the "nothing about us without us" mantra left hanging in the breeze. 30 As a result, despite a strong policy rhetoric supported by national agendas such as the UK Long Term Plan and the Digital First Strategy, the extent to which patients and the public are involved in digital health innovations, implementation and evaluation remains largely unknown. 26 Such findings may have important implications for policy makers, innovators and regulators as evidence suggests that patient involvement in digital health innovation can reduce patient-related anxieties and reticence to use digital health services 30 and related technologies. Furthermore, as identified in existing literature, 31 evidence-based guidance on how to carry out meaningful PPI in the rapidly evolving field of digital health is lacking, highlighting a further gap in existing knowledge and understanding.
While acknowledging the number of systematic reviews already conducted on PPI in specific fields such as medical regulation, 8 healthcare services 4 and research, 6,32 justification for this review stems from the increasing interest and 'critical' need attributed to PPI in digital health innovation 31,[33][34][35] ; purported lack of attention paid to patient perspectives during digital health innovation design 31 ; increasing use of digital health innovations worldwide 36 ; and acknowledged importance of working with patients to ensure innovation relevance, value and acceptability. 31 while it is widely accepted that digital health technologies should be codesigned, 33 to the researchers' knowledge, this is the first systematic review of its kind to explore PPI in digital health innovation, implementation and evaluation, highlighting its novel contribution.
The aim of this study was to therefore conduct a systematic review to explore: (i) how, if at all, patients and the public are involved in digital health innovation, implementation and evaluation and (ii) identify factors that affect meaningful PPI in digital health innovation, implementation and evaluation as there may be unique considerations in digital health such as digital skills, patient connectivity and confidence that may be less applicable in other areas of PPI.
The review questions we sought to address were: 1. How, if at all, are patients and the public involved in digital health innovation, implementation and evaluation? 2. What are the barriers and enablers for supporting meaningful PPI in digital health innovation, implementation and evaluation?
In the absence of a single agreed term, 38 the term 'patient' is used to be inclusive of end-users, clients, service-users, survivors, citizens, consumers, customers, carers and caregivers. While recognizing the important distinctions between these terms, 39 47 ; ability to uncover innovative information often in an earlier form following a recognized time lag between research and peer-reviewed publication; and its ability to potentially minimize bias in a comprehensive search. 47 Given the rapidly evolving field of digital health innovation, the incorporation of grey literature can be well justified for the purposes of this review.
All review searches were conducted over a 2-day period (30 June 2020-01 July 2020).

| Study selection
Studies were selected through a two-stage process. First, due to the large number of abstracts returned, five reviewers (RB, HB, SS, KE,

| Data extraction
Six reviewers (RB, HB, SS, KE, JTR, SP) independently undertook data extraction using a piloted data extraction form to facilitate data extraction consistency. Information extracted included author name, publication date, study location, population and methodology, digital health technology type, stage of involvement, that is, innovation/design, implementation and/or evaluation and reported barriers and enablers.

| Data analysis and synthesis
Review findings were analysed using inductive thematic analysis as proposed by Braun and Clarke. 48 Identified themes were synthesized using a narrative approach, defined as 'an approach to the systematic review and synthesis of findings from multiple studies that relies primarily on the use of words and text to summarize and explain findings of the synthesis' according to Popay et al.'s 43,p.5 guidelines.

| Registration
The review protocol is published on the PROSPERO website (registration number CRD42020201432).

| Patient and public involvement
This review was designed with significant involvement of a patient representative from the Patient Experience Library. Review findings have also been presented in a multistakeholder focus group involving patients, patient advocate organizations, clinicians and digital health innovators as a sense-checking exercise. The patient representative is rightly included as a coauthor of this review, given their involvement and contributions.  were explicitly defined in included articles, with Norman's user-centred system design most commonly used. 49 3.2 | How, if at all, are patients and the public involved in digital innovation, implementation and/or evaluation?

| Descriptive summary
In response to the first review question, patients were often most involved in the final stages of an innovation journey, for example, usability testing, where the ability to proactively influence change was severely limited. Methods often used to involve patients and the public included think-aloud interviews, focus groups and surveys.

| Perceived importance
Many articles conceded that meaningful PPI is both necessary and fundamental, capable of achieving 'unexpected consequences' that are both 'rewarding and fulfilling'. 50 For example: Similarly, 'the stepwise iterative process revealed elements critical to an effective intervention, which otherwise could have been easily missed'. 52 An 'increased sense of awareness, ownership, and identifica-tion… by the fact that content better reflects the context, needs and wants of the target community' was also described as a beneficial outcome of PPI. This increased sense of ownership was also felt to help 'decrease resistance to health messages (also called counterarguing) as the messages are perceived as coming from sources that have been internally validated, rather than being foreign and external to the community', 59 highlighting a further benefit attributed to PPI in digital health innovation.

| Reported impacts
Leading on from the benefits outlined above, several authors also reiterated the importance of PPI in digital health innovation and implementation due to its resulting impact. For example, as suggested by Camerin et al., 57 'this study empirically confirmed that the adoption of a participatory approach to the design of eHealth interventions and the use of personalized contents enhance the overall effectiveness of systems. Therefore, more time and effort should be invested in involving patients in the preliminary phases of systems' development, maximizing the likelihood to observe the desired effects'. Similarly, 'overall, the findings from this study confirm the importance of including PPI at the early design stage of medical devices' 60 ; 'the perspective of person-centred care helped us apply a broader scope involving the patient as a person in both the process and the final product'. 61 3.4 | What are the barriers and enablers to meaningful PPI in digital health innovation, implementation and evaluation?

| Barriers
However, despite a range of reported benefits, the reviewed articles also described a multitude of barriers ( This can prevent equitable and authentic involvement, with some patients withholding information that may be integral to innovation design, implementation and/or evaluation. The presence of carers and/or parents was also felt to prevent some people from being honest in their responses Data privacy, security and trust Failure to establish trust and provided assurances of data privacy and security can affect people's willingness to be involved and inhibit product acceptance Bias A number of biases can be introduced into PPI initiatives including the setting in which PPI activities are undertaken, recruitment methods, for example, primarily all online, preventing involvement from those considered to be digitally excluded, volunteer bias (people who opt in or take part in research/innovation may have greater interests/ motivations and may therefore present the perspective of early adopters only), sharing existing digital innovations/ideas before data collection inducing possible response bias, previous involvement in digital innovation creation when evaluating the product leading to possible social desirability bias, exclusion from data analysis and shortened evaluation times that may present an overinflated experience Practical difficulties including reimbursement, identifying times when everyone is free, not recording involvement sessions, not articulating why it may be beneficial to get involved, not considering the cost of downloading required content, for example, data charges, and sharing incorrect contact information This can significantly hinder PPI activities and experience and collection of meaningful insights Abbreviation: PPI, patient and public involvement.
ideas for the design and keep in mind the needs of each user and the realities of their work settings while also allowing for the variety of users, their information needs, and their assumptions about how a such a system should-and does-work'. Similarly, it could be argued that the purpose of involvement is to respond to patient input as it emerges throughout the innovation journey. This is often at odds with gaining ethical approval, with committees often needing to know what will exactly be done, how, by who and when. Difficulties related to ethical approval processes are often universal, but appear particularly problematic in relation to codesign and implementation research. 70

| Disconnect between developers and patients
Other areas of contention included a disconnect between developers/researchers and patients, meaning that decisions made did not always reflect patient requirements, often defaulting to developer or researcher assumptions and preferences. PPI also often required developers to work in a way they were not 'used to dealing with'. For example, 'the other stage of the design process was related to the dialogue between the researchers who collect empirical data and the technical partners who develop the system who are not in contact with the end-users. In particular, we can report that personas and scenarios were not enough to ensure an efficient dialogue between stakeholders. It was difficult to convince technical partners that they should focus on personas when developing the application. Also scenarios looked too narrative for them compared to functional specifications that they were used to dealing with'.

| Enablers
Although less frequently discussed, some articles also described several facilitators (Table 3)  T A B L E 3 Suggested enablers to support meaningful PPI in digital health innovation, implementation and evaluation

Suggested enablers Supporting quotes
Commit to sharing power, working in equal partnerships that treat insights equally, irrespective of their source Specifically 'better balancing the power relations that exist' 71 ; 'a democratic partnership with appropriate distribution of power', 76 or 'in bidirectional equitable partnerships'. 77 Similarly, 'the methods for the design expressly included patients and staff with all voices treated equally and regarded as key contributors to design'. 49 Involve patients early on 'Findings from this study confirm the importance of including PPI at the early design stage of medical devices'. 59 Work in an interactive, open-minded and adaptive manner 'The whole process required flexibility, an open mind, and a willingness to revise material iteratively'. 78 Similarly, 'it was necessary to take a highly iterative approach'. 75 Work to establish trust 'It was critical to ensure timely and consistent follow-up in response to any technical or personal issues that are reported by the participants. This is an important part of building participants' trust in the intervention and the staff. Trust of the programme and trust of outreach workers was a priority issue, which needed to be addressed during all aspects of the programme roll-out. It was helpful to brand the Mobile Link programme and have Cambodian government buy-in so that women know that the programme they are signing up for is medically accurate and trustworthy'. 74 Be sensitive to people's spiritual, religious and cultural beliefs/values Considering peoples 'spiritual, religious and families values when designing digital health innovations' 63 is imperative, particularly when working with indigenous and Hispanic communities.
Create engaging activities 'It is important to ensure that the methods and user activities fully engage the participants' 71 Communicate clearly, regularly and inclusively in an ageappropriate and developmentally appropriate way, including the perceived benefits of taking part 'A developmental or age-appropriate approach is needed regarding the content and design of a programme, and accounting for the range of interests and tastes'. 76 Suggested use of 'glossaries, use of visual aids/picture topic clues, and videos to facilitate understanding of information'. 79 'Full commitment requires motivating the participants and convincing them about the usefulness of the project, which proved to be more complicated than we thought. The primary motivating factor was their feeling of participating in the creation of services for the future'. 69 Offer people involved a choice of communication methods For example, 'WhatsApp was a significant production asset, useful in soliciting feedback from community members who did not regularly use email and did not feel comfortable editing scripts using Google Drive'. 58 Clarify people's roles, decision-making processes and manage expectations Creating a 'memorandum of understanding' 77 and generating 'ground rules' 80 or 'rules of conduct' 81 were identified as helpful ways to clarify roles and manage expectations. Other suggestions included 'knowing each others role in the relationship' 82 and clarifying to partners involved 'that their individual wishes will not always be met' (83).

Provide clear instructions, tech support and relevant device access
'The elderly can be insecure because they are afraid of doing something wrong, so giving clear directions and affirmation is important. They also often need repeated explanations and daily training or courses in learning how to use a tablet for instance'. 69 'Members of the research team set up patients' phones and supported them throughout the study. Phones with the app pre-loaded were available on loan for people without an Android phone' (84).
'Provide a hotline in case of technical difficulties' (85).
Allow time for people to become familiar with the tech 'Given access for a minimum of 2 months to allow sufficient time to work through the programme'. 76 Work with local organizations to facilitate recruitment 'One of the core principles of patient participation relates to ensuring that engagement is made as easy, feasible and as flexible as possible… With these goals in mind, it was deemed that participant recruitment through a familiar agency… would be optimal'. 59 Other ways of facilitating recruitment included recruiting through existing patient lists or 'established patient groups' (86). (Continues) Identification of these reportedly unique aspects of supporting meaningful PPI in digital innovation further accentuates the novel contribution of this review.

| Strengths and limitations
The strengths of this review include its development with an Information Specialist, application of an internationally recognized systematic review process 78 and incorporation of grey literature, considered to be a vital adjunct to traditional database searches, 47 given its ability to uncover innovative information in an earlier form and codesign with patient representation from the outset. The integration of a previously disparate literature that remains a growing area of international interest (i.e., PPI in digital health innovation, implementation and innovation) 36 into one singular source of information is also considered to be a particular strength of this review, given the increasing interest and critical need attributed to PPI in digital innovation. 31,[33][34][35] Furthermore, the extensive results uncovered may be indicative of the extensive search strategy undertaken and adoption of inductive thematic analysis, avoiding the use of predefined and potentially restrictive frameworks.
However, the limitations of this review must also be acknowledged. Encourage developers and patients to work together in the same room 'We also recommend organizing meetings between developers and users, like test sessions during which the developers would be present to see for themselves the ways in which end-users actually use the technology. Another option would be to use video in order to show developers the reactions of the end-users when interacting with the devices and application'. 69 Create a safe space for people to share their thoughts and ideas 'We put the primary focus on ensuring all stakeholders felt a part of the process and opened up about their experiences without feeling judged. During all phases, we highlighted the importance of anonymity for this purpose and thus did not collect the demographic information of the participants' (87); 'Each session took place at a convenient venue (e.g., community clinic) on weeknights, ranged from 90 to 120 min, and was audio recorded. Before each session, participants shared a meal and informally discussed community health and events. All meetings began with an opening prayer by church leadership to set an atmosphere of creativeness, inspiration, and togetherness among the attendees' (88).
Hold activities in suitable locations 'Interviews were held either at Cardiff University or a location convenient for the participant (e.g., home and school). During the interviews, young people stated they were able to discuss the programme openly and appreciated that they could choose the location, and whether they were seen with their parents or carers'. 76 Provide people with a choice of how and who they would like to be involved 'Young people were asked whether they would like to be interviewed alone or with a parent or carer. The parent or carer was also asked whether they would like to be interviewed separately'. 76 researchers and regulators, recognizing that the participatory or codesign process may help to address existing power imbalances, particularly when tailored to individual needs and cultural sensitivities of individual community groups. Similar to patient involvement and coproduction in research, efforts are often unsuccessful due to 'structural inequalities… that impede coproduction'. 81,p.2 As such, there is a need to question the extent to which meaningful PPI can truly operate on an equal footing in digital health innovation if it is to achieve its egalitarian and utilitarian potential. 82 Other implications of this review include the importance of identifying ways to establish trust and transparency amongst patients and the public in digital health innovations, particularly with regard to data privacy and sharing. Establishing trust around these areas is