Supporting the evaluation of public and patient engagement in health system organizations: Results from an implementation research study

Abstract Background As citizens, patients and family members are participating in numerous and expanding roles in health system organizations, attention has turned to evaluating these efforts. The context‐specific nature of engagement requires evaluation tools to be carefully designed for optimal use. We sought to address this need by assessing the appropriateness and feasibility of a generic tool across a range of health system organizations, engagement activities and patient groups. Methods We used a mixed‐methods implementation research design to study the implementation of an engagement evaluation tool in seven health system organizations in Ontario, Canada focusing on two key implementation outcome variables: appropriateness and feasibility. Data were collected through respondent feedback questions (binary and open‐ended) at the end of the tool's three questionnaires as well as interviews and debriefing discussions with engagement professionals and patient partners from collaborating organizations. Results The three questionnaires comprising the evaluation tool were collectively administered 29 times to 405 respondents yielding a 52% response rate (90% and 53% of respondents respectively assessed the survey's appropriateness and feasibility [quantitatively or qualitatively]). The questionnaires' basic properties were rated highly by all respondents. Concrete suggestions were provided for improving the appropriateness and feasibility of the questionnaires (or components within) for different engagement activity and organization types, and for enhancing the timing of implementation. Discussion and Conclusions Our study findings offer guidance for health system organizations and evaluators to support the optimal use of engagement evaluation tools across a variety of health system settings, engagement activities and respondent groups.


| INTRODUC TI ON
As citizens, patients and family members take on numerous and expanding advisory and leadership roles 1,2 in health system organizations, attention has turned to evaluating these efforts. Though the underlying goals for evaluation may differ, 3 there is broad consensus about the need for stronger evidence to understand which engagement approaches work best, in which contexts and with what effects. 3,4 The number of tools and frameworks available to support the evaluation of public and patient engagement has increased steadily over the last number of years. Two recent systematic reviews identified 27 tools and over 100 metrics for evaluating patient engagement in research and health system decision making; however, most were developed without direct involvement of citizens or patients, or robust testing in different organizational contexts. 5,6 The Public and Patient Engagement Evaluation Tool (PPEET) was an early contribution to the engagement evaluation field. Developed by researchers and public and patient engagement practitioners, the goal of the PPEET was to provide a generic evaluation tool that could be used in a variety of health system organizations to evaluate and continuously improve the quality of public and patient engagement (PPE) activities. During its early development, the PPEET underwent usability testing with a limited group of respondents and end-users (including patients and members of the public) who provided feedback on the structure, layout, comprehensibility, ease of use and overall utility of the survey. 7 Given the widespread use of the tool since its launch in 2015, we set out to systematically assess its suitability for implementation across a broader range of organizational settings, engagement activities and user populations. The contextspecific nature of engagement practice requires evaluation tools to be carefully designed or chosen for optimal use. Results from this implementation research study have informed modifications to the PPEET and can offer guidance for health system organizations and evaluators to support the optimal use of engagement evaluation tools in a variety of health system settings.

| Description of the Public and Patient Engagement Evaluation Tool (PPEET)
The PPEET is comprised of three surveys, each aimed at different target groups for evaluation. The participant survey aims to assess the processes, outputs and perceived impacts of engagement activities from the perspectives of citizens, patients and family members who may have various roles in the engagement process (e.g. consultant, advisor, partner). The project survey aims to evaluate the planning and execution of the engagement activity from the perspective of engagement implementers (e.g. engagement project leads, staff members). Finally, the organization survey aims to assess the culture and practices supporting public and patient engagement from the perspective of senior management and leadership in organizations (e.g. executive team members, board members). Each survey is structured around four core principles of "quality engagement" informed by an evidence synthesis and expert input: (a) integrity of design and purpose; (b) influence and impact; (c) participatory culture; and (d) collaboration and common purpose. 7 The surveys include Likertscale and dichotomous questions, as well as open-ended questions embedded throughout the survey.

| RE S E ARCH DE S I G N AND ME THODS
We used a mixed-methods implementation research design [8][9][10] to study the PPEET's implementation in seven health system organizations in Ontario. Guided by Proctor et al's (2011) taxonomy of eight implementation outcomes (acceptability, adoption, appropriateness, feasibility, fidelity, cost, penetration and sustainability), we selected the two outcomes that most closely aligned with the stage of the tool's implementation (i.e. early) and our focus on further testing its usability, suitability and feasibility in different organizational contexts: (a) appropriateness (perception among stakeholders that the evaluation tool was relevant, useful and suitable for the target audience and organization); and (b) feasibility (the extent to which the tool could be implemented in a particular setting). 8,11 Each implementation outcome variable was assessed through a combination of quantitative and qualitative data collected from multiple sources using purposeful sampling. 12 Quantitative (survey) data provided insights into the implementation of the PPEET and its appropriateness and feasibility from the perspective of the end-user (see Appendix S1 for feedback survey). Qualitative data (open-ended feedback provided through surveys, interviews and group discussions) supported efforts to understand, interpret and explain the quantitative data and to provide detail about the tool implementation process (appropriateness and feasibility). 13 The study was reviewed and approved by the Hamilton Integrated Research Ethics Board (HIREB).

| Recruitment of collaborating organizations
Study collaborators included early adopters of the PPEET (i.e. organizations using or interested in the tool immediately following launch), selected using the following inclusion criteria: organizations with different health system mandates (e.g. health-care delivery vs provincial/local planning and quality improvement organizations), and organizations serving different types of communities or populations (e.g. large urban vs small northern community; disease-specific vs population-specific). The final group of seven collaborating organizations included large, urban, academic health science centres/networks (N = 3), provincial and/or regional health quality and integration organizations (N = 3), and one community-based health services organization (N = 1; Table 1). Collaborating organizations committed to implementing each of the PPEET surveys (project and organization surveys once each, participant survey two to three times each). Each organization identified at least one staff member and at least one citizen/patient involved with PPE activities within the organization to join a Study Advisory Committee (SAC) to involve patients and organization staff at key stages of the research process. The committee included 16 members (5 patient partners and 11 staff members across the seven participating organizations) who were also invited to participate in different aspects of the study as survey respondents, key informant interviewees and/or debriefing call participants.

| Recruitment of survey respondents
In consultation with the research team, staff members within each of the collaborating organizations, identified the engagement activities that would be evaluated and oversaw the recruitment of citizens, patients, family members and staff to complete the surveys. The research team had no direct involvement in the recruitment process or contact with survey participants. All individuals involved with each of the selected engagement activities were recruited to participate in the survey. Recruitment methods varied by organization; each organization documented their recruitment process including the number of individuals approached to complete a survey and the recruitment strategies used. Organizations could opt to collect the survey data themselves or have the data collected directly by the research team through an online survey platform (LimeSurvey). Organizations collecting data shared it securely with the research team. When the online survey platform was used, the research team programmed the survey and shared a survey link with the organization, which was distributed to potential respondents. Completed surveys were stored directly on the secure online platform. No identifying information was collected through the survey, ensuring the anonymity of respondents.

| Post-implementation activities
Following each implementation, a report summarizing the results of the survey was shared with the partner organization. Organizations were encouraged to share evaluation results with the engagement participants. A member of the research team completed a 15-to 30-minute debriefing call with the PPE project lead and/or the public/patient representative in each partner organization to discuss the PPE activity evaluated and the process used to implement the survey (eg recruitment strategies, dissemination of survey results).
During these calls, feedback on the appropriateness and feasibility of the tool was collected from the implementer's perspective (e.g. survey implementation time, concerns with the survey, feedback from users). Additional feedback on the tool implementation process was collected through three advisory committee meetings held over the 1.5-year project.

| Data analysis
PPEET feedback survey data were analysed using descriptive statistics for the quantitative components and qualitative thematic analysis for the open-ended questions. 14 Analysis was completed initially at the aggregate level and subsequently by type of organization and by type of engagement activity (where applicable).

| RE SULTS
Each of the PPEET instruments was implemented in at least four of the seven partner organizations and between 1 and 7 instances within each (   (Table 3).
Qualitative themes are presented, by survey type, in Table S1.

| Participant survey
Each partner organization implemented the participant survey at least once and as many as seven times over the study period. In total, 271 participant surveys were administered across the 7 organizations and 17 engagement activities and 152 surveys were completed.
Of these, 131 respondents completed at least two of the feedback questions, which are the focus of our analysis ( Table 2). Demographic characteristics of feedback survey respondents are presented in

| Quantitative results
Overall, the quantitative responses yielded favourable views towards the survey (Table 5). Nearly all respondents indicated that the survey (96.9%) and its layout (97.7%) were easy to use and that the instructions were clear and helpful (94.7%). Most found the questions easy to understand (87.0%). Just over two-thirds of respondents did not think any important questions were missing (68.7%).
The survey was also viewed positively across all types of engagement activities with some minor differences. Respondents who participated in short-term activities gave slightly higher ratings than those in the other two groups. Respondents who participated in knowledge translation and exchange (KTE) activities stated most frequently that important questions were missing (25.0%) compared to the other two groups (Table 6), although this difference was not statistically significant. Results were quite uniform across the three organization types. Individuals from provincial and regional health system organizations were significantly more likely to report that important questions were missing (32.2%; P = .02) and those from community-based health system organizations stated most frequently, although not statistically significant, that the questions were hard to understand (25.0%) ( Table 7).

| Qualitative results
Over half of respondents (Table 2) provided written comments about the questionnaire as a complement to the scaled questions.
Much of the written feedback supported the quantitative results and reinforced views that the questions were appropriate, clear and relevant and that the survey was thorough and straightforward (see Appendix S1). The participant survey appeared to work equally well in both health-care delivery and provincial/regional health organization settings. The only activity evaluated within a community-based health-care delivery organization was a KTE activity; therefore, additional work is needed to confirm the appropriateness and feasibility of the survey in this specific context.
Major areas of feedback focused on the appropriateness and feasibility of the survey and specific sections or items within it, and the need for the tailoring to specific contexts. Minor areas of feedback included requests for additional questions to allow for more detailed exploration of the engagement process, minor wording changes and a request for "not applicable" response options.

Appropriateness of the questionnaire for different types of engagement activities
Although those who participated in short-term engagement activities felt that questions in this survey "were valid and in-scope of the activity" (AHSC-3, Activity 13), many others questioned its appropriateness for longer-term and KTE activities.
For respondents who were involved in longer-term engagement activities such as councils and committees, the survey presented challenges in determining the unit of engagement to be evaluated (i.e. a specific activity or meeting or the engagement process as a whole) and how to assess the features of multiple meetings when the quality of engagement might vary from one meeting to the next. This led a number of respondents to state that the survey was better suited to a onetime activity, a view that was shared by the Study Advisory Committee.
This survey is evaluating approximately 10 meetings and doesn't lend itself well to that assessment…some meetings diversity of opinions was sought, sometimes not.
(AHSC-1, Activity 3) The questions seem better directed to a one-time, one-topic focus group rather than an ongoing council.
(HSO-3, Activity 12) For KTE activities, respondents cited the challenge of questions that were "very broad, hard to relate to the [activity] format/context" (HSO-1, Activity 2) and, correspondingly, the need for them to be "more detailed and specific" (CHSO-1, Activity 11). questions about income and age. This also likely explains why over a third of respondents did not share their income information in this section of the survey. There were concerns that responses to these questions might be used to prioritize the perspectives of some over others. A particular concern raised by a respondent involved in a long-term activity was how the survey findings would be anonymized as "the questions asked compared to the demographics of [those participating] make it very easy [for the respondent] to be identified"

Relevance of specific sections or questions and the need for tailoring
(HSO-2, Activity 6). Another noted that these types of demographics did not seem appropriate for the particular setting they were involved in, and that questions related to individual's health status and experience with the health-care system would be more relevant.

P-value
Academic health science centres/networks (n = 60) Provincial and regional health system organizations (n = 59)

Community-based health system organization (n = 12)
Overall, survey was easy to use

| Project survey
The project survey was implemented by four separate organizations

| Quantitative results
Generally, respondents felt the survey was easy to use (87.5%) and the instructions were clear and helpful (87.5%), and few felt important questions were missing (6.3%). Across all surveys, the project survey respondents were least likely to feel the layout was easy to work with (87.5%, P = .04). Although three-quarters of respondents (75%) reported that the results of this survey would be useful for their organization, over two-thirds indicated that questions were not easy to understand (68.8%) which was significantly higher than the PPEET participant and organization questionnaires (P < .001), suggesting the need for significant revisions (Table 5).

| Qualitative results
Almost all respondents (Table 2)  The length of the survey was of concern to some respondents who felt it was "onerous to complete" (HSO-2, Activity 9). Respondents felt that the number of questions overall could be reduced as well as the number of open-ended questions more specifically. A desire for questions to "focus a bit more on more of a micro level" (HSO-2, Activity 10) was also communicated to make them easier for staff to respond to. A member of the Study Advisory Committee noted that the length of the survey presented challenges for staff to find the time to complete it and recommended using the survey as a guide for conversations with staff about the engagement activity. The survey's utility in encouraging best practices was also noted: it's kind of human nature…if we put [an item] on the checklist and we know that we're being evaluated by it, we're sort of directly and indirectly pushing people to do best practice if you know you're going to be evaluated on it at the end of the day, right?

| Organization survey
The organization survey was implemented by 5 of the 7 organizations but achieved the lowest response rate of the three (37%); all respondents completed feedback questions, and 35% provided open-ended comments). Four of the organizations used it to evaluate their public and patient engagement work across the organization. One organization used it to evaluate engagement within a specific part of the organization.

| Quantitative results
As with the other two surveys, most respondents felt this survey was easy to use (95.3%), the instructions were clear and helpful (95.3%), and the layout was easy to use (95.3%). Some felt important questions were missing (14.0%) and most found the questions easy to understand (90.7%). Almost all (90.7%) thought the results would be useful for their organization (Table 5).
Overall, respondents from health-care delivery organizations were positive about the survey with few individuals indicating any concerns with the survey within the quantitative questions. Albeit not statistically significant, respondents from provincial and regional health system organizations indicated more often that there were important questions missing from the survey (27.3%) and that the questions were not easy to understand (18.2%) ( Table 8).

| Qualitative results
Just over one-third (35%) of those who completed the feedback questions on the organization questionnaire provided open-ended comments, compared to over 50% and 90% for the other surveys (Table 2) On the theme of the appropriate organization type, the organization survey was critiqued for not adequately capturing the approach to engagement in an organization with a strong community orientation.
Engagement is part of the fabric of a true community organization. This survey puts the concept of public and patient engagement as a separate entity, it is not and therefore the questions are not in some instances relevant. (CHSO-1) The relevance of the survey to smaller-sized organizations was also questioned-specifically organizations where there are fewer resources to dedicate to engagement, which could influence the responses to the survey questions. A related question about the framing of the survey was raised by one of the hospital sector respondents who noted that the survey seemed to view engagement as needing "additional resources or infrastructure rather than as a thread through existing structures" (AHSC-3).
Identifying the "right" individuals to participate in the survey and ensuring that the questions were relevant to range and type of respondents who might complete it within organizations was also identified as important. For example, one individual felt that "some questions were also hard to answer because they're out of the scope of what a manager would know" (HSO-1).
As with the project survey, a number of questions were cited for improved clarity, framing or contextualizing. Notably, greater precision was requested about the level within the organization at which TA B L E 8 Organization survey feedback results, by organization type

Feedback question Response
Organization type

P-value
Academic health science centres/networks (n = 26) Provincial and regional health system organizations (n = 11)

Community-based health system organization (n = 6)
Overall, survey was easy to use

| D ISCUSS I ON
Our results provide insights into the appropriateness and feasibility

| Study contributions, strengths and limitations
The number of engagement evaluation tools and frameworks is growing. Associated with this trend are encouraging examples of more rigorous tool development and testing, with the direct involvement of patients. 6,15 However, studies focused on assessing the robustness of engagement evaluation tools in a real-world setting or in the specific context of health-care delivery and system planning organizations are still few in number. 16 We believe this study is one of the first efforts to systematically assess the appropriateness and feasibility of an engagement evaluation tool across a range of organizational settings, engagement activities and user populations, using a recognized framework and selected outcome variables. 8

| CON CLUS ION
This study provides insight into how the PPEET (and other evaluation tools) can be more optimally implemented to evaluate a range of public and patient engagement activities within a variety of health system organizations. Working closely with different health organizations to understand the appropriateness and feasibility of the PPEET surveys provided valuable information about how to improve the PPEET in simple yet effective ways. We look forward to continued efforts to develop and rigorously assess the PPEET and other engagement evaluation tools to support and continuously improve the quality of engagement work carried out in health system organizations.

CO N FLI C T O F I NTE R E S T
The authors have no conflicts of interest to report.

DATA AVA I L A B I L I T Y S TAT E M E N T
The data that support the findings of this study are available from the corresponding author upon reasonable request.