Development of the Engage with Impact Toolkit: A comprehensive resource to support the evaluation of patient, family and caregiver engagement in health systems

Abstract Introduction Recent shifts in the patient, family and caregiver engagement field have focused greater attention on measurement and evaluation, including the impacts of engagement efforts. Current evaluation tools offer limited support to organizations seeking to reorient their efforts in this way. We addressed this gap through the development of an impact measurement framework and accompanying evaluation toolkit—the Engage with Impact Toolkit. Methods The measurement framework and toolkit were co‐designed with the Evaluating Patient Engagement Working Group, a multidisciplinary group of patient, family and caregiver partners, engagement specialists, researchers and government personnel. Project activities occurred over four phases: (1) project scoping and literature review; (2) modified concept mapping; (3) working group deliberations and (4) toolkit web design. Results The project scope was to develop a measurement framework and an evaluation toolkit for patient engagement in health systems that were practical, accessible, menu‐driven and aligned with current system priorities. Concept mapping yielded 237 impact statements that were sorted, discussed and combined into 81 unique items. A shorter list of 50 items (rated 8.0 or higher out of 10) was further consolidated to generate a final list of 35 items mapped across 8 conceptual domains of impact: (1) knowledge and skills; (2) confidence and trust; (3) equity and inclusivity; (4) priorities and decisions; (5) effectiveness and efficiency; (6) patient‐centredness; (7) culture change and (8) patient outcomes and experience. Working Group members rated the final list for importance (1–5) and identified a core set of 33 items (one for each of the 8 domains and 25 supplementary items). Two domains (priorities and decisions; and culture change) yielded the highest overall importance ratings (4.8). A web‐based toolkit (www.evaluateengagement.ca) hosts the measurement framework and related evaluation supports. Conclusion The Engage with Impact Toolkit builds on existing engagement evaluation tools but brings a more explicit focus to supporting organizations to assess the impacts of their engagement work. Patient Contribution Patient, family and caregiver partners led the early conceptualization of this work and were involved at all stages and in all aspects of the work. As end‐users of the toolkit, their perspectives, knowledge and opinions were critical.


| INTRODUCTION
Recent advances in the field of patient, family and caregiver engagement in research and health systems have focused greater attention on its measurement and evaluation. 1,2 As Boivin et al. state in their BMJ editorial, ʻif we are serious about involvement, we need to be equally serious about evaluation'. 1 Just as the goals for engaging patients vary, so too do the goals for its evaluation. Early interest in evaluation has primarily focused on improving the quality of engagement practices. As engagement practices have become more firmly embedded in organizations, the goals for evaluation have shifted to assessing the outcomes and impacts of these activities and processes. 3,4 A growing number of tools for evaluating patient engagement have been developed over the last decade, followed by several systematic reviews that have attempted to summarize them and assess various aspects of their quality. Reviewing this literature is challenging because of the multiplicity of review objectives and the diversity of evaluation resources that range from conceptual frameworks and measures to practical supports that include data collection instruments. Boivin et al. systematically reviewed and appraised 27 tools used to evaluate the patient and public engagement in research and health system decision-making. 5 Most tools were designed to support the quality improvement of engagement activities within organizations and health research teams. Deficiencies were noted across the tools in terms of lack of scientific rigour (only 11% were explicitly informed by a literature review and just 7% were tested for reliability) and the absence of public and patient engagement in their design and reporting (56% of the time but mainly in the piloting stage, and only 18.5% of tools were designed to report evaluation results to patients and the public). 5 If patient and public engagement is founded on the principles of reciprocity and equal partnership, its evaluation should uphold the same principles.
Dukhanin et al. 6 focused at a more microlevel by developing a taxonomy of 116 process and outcome metrics for evaluating engagement in patient, public, consumer and community engagement in organization-, community-and system-level healthcare decision-making.
Their taxonomy was then used to assess the coverage of evaluation metrics in 23 evaluation tools published between 1973 and 2015. Tools varied in their coverage of the taxonomy; nearly twice as many process metrics were identified (72) compared to outcome metrics (44). Among the outcome metrics, more addressed internal outcomes (the effect of engagement on the organization itself, its services or engagement participants) as compared to external outcomes (i.e., the effect of engagement beyond the organization where it occurs). Of particular interest is the authors' proposal for a maturity model approach to evaluation that includes three levels that organizations can progress through over time. 6 Dukhanin et al. 6 note important deficiencies in the tools and identified priorities for future research, namely, (i) the lack of rigorous testing of the tools supporting the need to better understand the validity and reliability of the tools and metrics in different healthcare contexts and (ii) the diversity of metrics identified, suggesting the need for consensus-building to identify and disseminate core metrics.
Greenhalgh et al. 7 review of 65 frameworks for supporting, evaluating and reporting patient and public engagement in health research revealed the diverse and theoretically heterogeneous nature of engagement literature and concluded that the diversity in purposes and scientific underpinnings of existing frameworks limited transferability across contexts. They concluded that ʻa single, onesize-fits-all framework may be less useful than a range of resources that can be adapted and combined in a locally generated co-design activity' developed with patient collaborators. 7 Current engagement evaluation tools offer limited support to organizations seeking to build incrementally towards impact measurement. Most tools with an exclusive focus on impact measurement have been developed for application to health research. The lack of engagement evaluation tools designed specifically for patient engagement in health system planning, design and decision-making is a notable gap. We address these gaps through the development of the Engage with Impact Toolkit, an online evaluation toolkit including an impact measurement framework and evaluation supports designed by and for a broad range of health system partners including engagement leads; patient, family and caregiver partners and evaluation specialists working at different levels of health system planning.

| Setting and context
This research was conducted in Ontario, Canada. In 2019, the provincial government introduced a major health system transformation designed to foster more integrated care through the establishment of Ontario Health Teams (OHTs). Comparable to the accountable care organizations in the United States, OHTs are ʻgroups of providers and organizations that are clinically and fiscally accountable for delivering a full and coordinated continuum of care to a defined geographic population'. 8 OHTs are expected to ʻuphold the principles of patient partnership, community engagement and system co-design… (and) meaningfully engage and partner withand be driven by the needs ofpatients, families, caregivers and the communities they serve'. 8 Early experience with OHTs suggests that patient, family and caregiver partners are engaging and partnering within these new systems in different ways and with varied success. The need for frameworks and metrics to link these activities to relevant outcomes and impacts has also been documented, 9,10 reinforcing similar messages from the broader engagement evaluation literature.

The Public and Patient Engagement Collaborative at McMaster
University is a supporting partner for the implementation of OHTs and provides engagement evaluation support across two of the eight building blocks of the OHT model: (i) patient partnership and community engagement and (ii) performance measurement, quality improvement and continuous learning. The Public and Patient Engagement Collaborative's work with OHTs across these three building blocks laid the foundation for the development of the Engage with Impact Toolkit (henceforth, the Toolkit). In this context, patient engagement refers to patients, families and caregivers collaborating with members of the health system (e.g., healthcare professionals, health system administrators, government employees and others) to improve healthcare quality. 11 2 | METHODS

| Phase 2: Modified concept mapping process
The literature review was augmented by a modified concept mapping process. Concept mapping is a participatory research method used to engage key partners in brainstorming, idea sorting and interpretation of results 15 and has been used successfully in the Ontario context to integrate patient perspectives to inform health system decisionmaking. 16 We adapted this method to the Working Group's structure, mandate and timeframe (the details of each stage are described below).

| Brainstorming items
Working Group members, in addition to a larger sample of patient engagement community members reached through snowball sampling, were invited to participate in the brainstorming stage. Working Group members identified the need to expand the size and diversity of the brainstorming group to ensure that a larger and more diverse set of perspectives and experiences would be captured. Each Working Group member was invited to identify up to five patient partners, engagement leads, health system leads or others working in the field in Ontario to be invited to participate in the brainstorming stage. Once the list of brainstorming participants was finalized, links to the online brainstorming survey were emailed to participants by a member of the research team. A focus prompt is used to structure the brainstorming stage. The focus prompt was a survey question that asked respondents to complete the following sentence: ʻWhen patient, family and caregiver partners are engaged in health system organizations, the impact that patient engagement and partnering has on … is…'. The sentence was asked for each of the following levels of impact: people, programmes, organizations and health systems.
Respondents were asked to identify as many impacts as they could for each statement.

| Sorting and combining items
Research team members sorted and combined the impact statements (henceforth, items) (e.g., duplicates removed, similar items consolidated). These items, along with those identified through the literature in phase 1, were categorized by four levels of impact: people (individuals involved in an engagement activity-patient partners, staff, etc.), programme (programme design and delivery, governance decisions), organization (the health system organization-e.g., the OHT, hospital, etc.) and health system (e.g., the provincial health system)-an organizing framework commonly used in the programme evaluation literature that was adapted to the health systems context. 17

| Rating items
A second survey was administered to the same group of respondents to rate the statements in answer to the following question: How important is it to measure this impact? Respondents rated the importance of measuring each item from 1 to 10 (allowing for decimal values), where 1 was least important and 10 was most important. Respondents were reminded that the focus was on the importance of measuring the impact, not the importance of the impact itself. Statements were presented within their level groups (i.e., people, programmes, organizations and health systems). Additionally, respondents were asked to identify any additional impacts that were not presented but should have been, in each category.
Average ratings for each item were calculated and used to identify the impact domains and items to be considered for the measurement framework.

| Interpretation of results and framework development
Following concept mapping methodology, where research team members and concept mapping contributors interpret ideas together in a facilitated group meeting, 15 results of the brainstorming, sorting and rating processes were shared, discussed and interpreted through a virtual working group meeting and several one-on-one discussions with Working Group members virtually and by e-mail. These discussions were used to identify key conceptual categories and further consolidate items as part of the measurement framework development process described in further detail below.

| Phase 3: Working group deliberations and measurement framework development
Concept mapping results were presented to the Working Group for discussion and deliberation. The research team presented a preliminary organizing framework of impact domains and items that were further refined. Items that were rated 8.0 or higher (out of 10) in terms of their importance to be measured were chosen for inclusion in a final survey to refine the list of items in each domain.
A final survey of Working Group members invited them to rate each item (within each domain) on a scale of 1-5, where 1 indicated that the item was not important at all (and could be removed from the framework) and 5 indicated that the item is extremely important and should be included as a core item within the framework.

| Phase 4: Toolkit web design
The concept mapping results were integrated with the feedback from the working group deliberations to inform the development of the web-based toolkit (the Engage with Impact Toolkit). The structure and overall design concept for the toolkit were determined by the Working Group.  Twenty-four individuals responded to this survey (response rate, 41.4%).
The mean importance ratings ranged from 6.1 to 9.7.

| Phase 3: Working group deliberations and the measurement framework
Through the Working Group's iterative process of reviewing the literature, seeking feedback from the broader engagement community, and deliberating internally, a final organizing framework was developed that clustered the 81 items around the following conceptual categories ('domains') of impact: (1) knowledge and skills; (2) confidence and trust; (3) equity and inclusivity; (4) priorities and decisions; (5) effectiveness and efficiency; (6) patient-centredness; (7) culture change and (8) patient outcomes and experience. In a separate step, each item was also mapped onto the different impact levels (people, programme, organization and health system) to follow recognized programme evaluation frameworks.
Within each of the domains, the item with the highest importance rating was labelled a 'core' item-an item that would be strongly recommended in all evaluations (items in bold in Table 2). Where importance ratings were low and written feedback was negative, items were dropped. For example, the item ʻpatient partners are frustrated by their patient engagement and partnering experiences' was excluded as the mean importance rating was 3.5 and many respondents took issue with the negative tone of the item, noting comments such as ʻI think we want a set of desirable impacts'.

| Phase 4: Toolkit web design
The Engage with Impact Toolkit is a module-based tool that provides users with guidance and support to evaluate patient, family and Staff feel more confident in participating in patient, family and caregiver engagement and partnering activities 3.7 Patient, family and caregiver engagement and partnering lead to increased trust in the health system 3.5 Equity and inclusivity Patient, family and caregiver engagement and partnering contribute to more equitable programmes because they are shaped by a diversity of perspectives

4.7
Patient, family and caregiver engagement and partnering contribute to more equitable decision-making within health systems

4.5
Priorities and decisions Patient, family and caregiver engagement and partnering shape the organization's plans/priorities/ decisions/policies

4.8
Patient, family and caregiver partners influence the final decisions/outcomes 4.8 Patient, family and caregiver partners are involved in decision-making 4.8

Effectiveness and efficiency
Patient engagement and partnering contribute to the design of effective programmes because they reflect the priorities of those who will be using the programmes

4.5
Patient, family and caregiver engagement and partnering contribute to cost-effective programming across the organization 3.3 Patient, family and caregiver engagement and partnering support the use of healthcare dollars in a fiscally responsible way

3.3
The organization's patient, family and caregiver engagement and partnering programme is cost-effective 3.1 Patient-centredness Patient, family and caregiver engagement and partnering make the health system more aware of the patient journey and challenges in the system

4.4
Patient, family and caregiver engagement and partnering facilitate the ability for organizations to provide client-centred services 4.3 Patient, family and caregiver engagement and partnering contribute to programming that is more patient-centred 4.3 Engaging patient, family and caregiver partners increases the relevance of the programme as it reflects the needs of patients/families/caregivers 4.0 Organizations are better able to offer the right services, in the right way at the right time 3.9

Culture change
The organization adopts a culture of co-designing and co-producing healthcare Required revisions were implemented.
The full toolkit was launched in January 2022 and can be found at www.evaluateengagement.ca.

| DISCUSSION
As the public and patient engagement field matures, the need for robust but accessible evaluation tools and metrics is a high priority to Patient, family and caregiver engagement and partnering are valued and recognized by the health system 4.3

Patient outcomes and experiences
Patient, family and caregiver engagement and partnering contribute to improved patient outcomes 4.5 Patient, family and caregiver engagement and partnering contribute to improved patient care 4.3 Patient, family and caregiver engagement and partnering contribute to improved patient satisfaction/ experience 4.1 metrics 6 and the utility of resources that can be adapted and combined in a locally generated co-design activity. 7 We reflect on several substantive, methodological and practicerelated aspects of the measurement framework results and the development of the evaluation toolkit as a whole. First, the conceptual mapping of the domains and items in Table 2 offers several insights into the emphasis given to some domains over others. For example, the larger number of items in the organizational culture and trust and confidence domains reinforces the view of engagement as a journey focused on culture change and a reorientation to patient-and family-centred care, and as relational work that at its core is about relationship-building and trust. 19 Second, our final mapping of domains and items reflects the strong desire of our working group members to take a broad lens to measure impacts that include all partners (i.e., staff, leadership team members), not just patient, family and caregiver partners. Current approaches to evaluating patient, family and caregiver engagement often focus on evaluating the impacts of patient and caregiver partners specifically, in isolation from other team members. This often places the onus on patient partners to drive change, to be effective partners and to champion engagement, which was viewed as highly burdensome to patient partners and antithetical to an organization-wide focus on supporting patient engagement. This concern had a strong shaping effect on the final set of domains and impact items, ensuring that there was a balance of items focused on the impacts of patient partners themselves as well as the impacts of staff/organization leads, and on the toolkit design process as a whole.
Methodologically, our adapted concept mapping approach may offer helpful guidance for future co-design work in the engagement field. In framework. First, the low response rate (40%) obtained for the rating activities is of some concern. Completion rates of between 65% and 70% are more typical for the rating stages of the concept mapping process. 20 We did not collect information about respondents versus nonrespondents in this activity but offer two possible explanations for the lower response rate. First, rating activities often followed Working Group meetings where preliminary discussions about ratings were held; thus, members may have felt that they had already provided their feedback through meetings and did not want to take the extra step of completing the formal rating survey. A second explanation is that individuals recruited to participate in the rating survey through snowball sampling may have had conflicting priorities, given their organizational roles focused on various pandemic-related priorities.
A second potential weakness of our study is that participants were asked to rate the items based on their importance and not their feasibility. Including feasibility as one of the rating items is common in concept mapping to keep a focus on actionable items (i.e., highly relevant, easy-to-implement items). However, working group members felt that it would be difficult for them to assess feasibility, so it was excluded. The omission of feasibility considerations likely resulted in the identification of items that were challenging to measure. For example, only a very general set of health system-level impacts was generated, which suggests that this is a priority area of focus requiring further development. Our hope is that by gaining experience using the toolkit's logic model approach, organizations will be better supported to trace the impacts of their engagement work more systematically to be able to assess broader health system impacts. Further, while equity and inclusivity were identified as a priority area for many on the Working Group, a limited number of items were identified in this domain. This may be reflective of the state of the patient and caregiver engagement field, where the considerable focus has been placed on the importance of engaging in an inclusive manner and in pursuit of equity goals, but significant gaps remain in turning principles into action. 21,22 We anticipate that this section of our toolkit will expand over time as more attention is paid to this area.

| CONCLUSION
As the public and patient engagement field matures, the focus on evaluation is a growing priority to support continuous learning and improvement and to demonstrate the critical links between the involvement of patients, family members and caregivers in system planning and design, and improved patient experience and quality of care. The Engage with Impact Toolkit is an early contribution to the measurement of these impacts. Rigorously codeveloped by patient, family and caregiver partners, engagement specialists and researchers, it is a living resource that will be continuously refined to reflect user experiences and new developments in the field.