Patient and public engagement in research and health system decision making: A systematic review of evaluation tools

Abstract Background Patient and public engagement is growing, but evaluative efforts remain limited. Reviews looking at evaluation tools for patient engagement in individual decision making do exist, but no similar articles in research and health systems have been published. Objective Systematically review and appraise evaluation tools for patient and public engagement in research and health system decision making. Methods We searched literature published between January 1980 and February 2016. Electronic databases (Ovid MEDLINE, Embase, Cochrane Database of Systematic Reviews, CINAHL and PsycINFO) were consulted, as well as grey literature obtained through Google, subject‐matter experts, social media and engagement organization websites. Two independent reviewers appraised the evaluation tools based on 4 assessment criteria: scientific rigour, patient and public perspective, comprehensiveness and usability. Results In total, 10 663 unique references were identified, 27 were included. Most of these tools were developed in the last decade and were designed to support improvement of engagement activities. Only 11% of tools were explicitly based on a literature review, and just 7% were tested for reliability. Patients and members of the public were involved in designing 56% of the tools, mainly in the piloting stage, and 18.5% of tools were designed to report evaluation results to patients and the public. Conclusion A growing number of evaluation tools are available to support patient and public engagement in research and health system decision making. However, the scientific rigour with which such evaluation tools are developed could be improved, as well as the level of patient and public engagement in their design and reporting.


| INTRODUC TI ON
In the past decade, there have been rapid developments in patient and public engagement 1,2 as illustrated by a tenfold increase in the number of articles published annually on the subject. 3 As the commitment to patient and public engagement has grown, so too has the call for robust evaluations. 4,5 Without adequate evaluation tools in place, it is difficult to ensure the integrity of engagement principles and practices, assess the outcomes of engagement, learn from current practices and demonstrate accountability for public investments.
Evaluative efforts on patient and public engagement have expanded recently, with the development of evaluation principles and frameworks 4,6-10 and an increasing number of published evaluations. 6,[11][12][13][14][15][16][17][18][19][20] The development of structured evaluation tools has been slower, and mostly performed through unpublished, projectspecific instruments, thus limiting the potential for comparison and mutual learning across engagement projects. Systematic reviews of evaluation instruments have been conducted for patient engagement in individual health-care decision making, 21-23 but need to be expanded to other domains of patient engagement, including research and health system decision making.

| OBJEC TIVE
The aim of this study was to systematically review and appraise existing evaluation instruments for patient and public engagement in research and health system decision making.
This work was conducted as part of the Canadian Strategy for Patient-Oriented Research (SPOR), which promotes patient engagement in research and health system transformation, hence our focus on these 2 engagement domains. 24 As part of the SPOR strategy, methodological SUPPORT Units (Support for People and Patient-Oriented Research and Trials) in each province are mandated with strengthening patient and public engagement. We also conducted a systematic assessment of identified evaluation tool to characterize their main strengths and weaknesses, guide engagement practitioners' choice of specific instruments and orient the future development of such tools.

| Definitions and scope
This systematic review was based on CIHR's definition of "patient engagement" as a meaningful and active collaboration in the governance, priority setting, and conduct of research, as well as in knowledge translation. 24 The "public" includes any "people who bring the collective voice of specific, affected communities." 24 "Evaluation" refers to "the systematic acquisition and assessment of information to provide useful feedback about some object." 25 Within the context of this systematic review, evaluations could focus on the context, process or impacts of patient and public engagement. 6 "Evaluation tool" refers to any instrument that can help to systematically acquire and assess information about patient and public engagement activities. This may include questionnaires, scales, interview guides or observation grids for use by engagement participants, organizational sponsors or external evaluators.

| Review method
We chose a critical interpretive synthesis method to guide our review. Critical interpretive synthesis allows for the conceptual translation of quantitative and qualitative studies, as well as non-empirical papers. 26 This approach is particularly well suited to the synthesis of diverse types of literature, such as quantitative and qualitative, published and grey, and health and social sciences, for which the phenomena of interest, populations, interventions and outcomes vary and may not be well-defined.

| Inclusion and exclusion criteria
We included original and review articles on evaluations of patient and public engagement in research and health system decision making that included evaluation tools, and background papers offering critical discussions of key evaluation tools that pertain to patient and public engagement in research and health system decision making.
Documents were excluded if they focused strictly on patient and public communication; patient engagement in individual health-care decisions; mechanisms to engage patients and the public as research subjects; theoretical and/or conceptual frameworks that were not linked to an evaluation tool; and evaluation tools not used in the context of research and health system decision making.

| Data extraction
All search results were transferred to a reference database and duplicates were removed. The titles and abstracts were screened by 2 members of the research team before retrieving the full-text versions of the references included. Discrepancies between reviewers were resolved by consensus. In total, 4 reviewers were involved in the inclusion process, and 1 reviewed all citations, abstracts and articles.
Two research team members independently reviewed and summarized the peer-reviewed articles and grey literature using a standardized extraction sheet, compiling the following information: authors; year of publication; main stated purpose; and the name, dimensions and domains, development procedure, theoretical/conceptual foundation and psychometric properties of the evaluation tool. Tool developers who did not publish the tools but only the process or use of their tool were contacted. Two members of the research team independently appraised the evaluation tools using the assessment grid and its 5-point rating system. (See Data S1 for the complete assessment grid.) The evaluation tools' readability level was assessed using the Flesch Reading Ease test (score of 70 or more, or 7th-grade level).

| Integrated knowledge translation
Evaluation tool users, including patients and members of the public with engagement experience, were involved in all stages of the project, using an integrated knowledge translation approach. 27

| RE SULTS
After removing duplicates, 10 663 unique references were identified and 648 full-text articles were reviewed ( Figure 1). Forty potential evaluation tools were identified. Thirteen of those were excluded because they were not specific to patient and public engagement (n = 9), did not relate to research and health system decision making (n = 1) or did not actually include an evaluation tool (n = 3). Few evaluation tools' developers responded to requests for additional information and unpublished tools.

| Description of evaluation tools
Twenty-seven patient and public engagement evaluation tools met the inclusion criteria (full references for those included are found in Table S1). Of the tools included, 14 are specific to research, 11 focus Self-administered questionnaires and surveys were the most common type of tool identified (n = 15). T he stated purpose behind most of the tools was the improvement of patient and public engagement activities. Table 2 describes the included evaluation tools. Table 3  Two of the tools covered all assessment criteria for usability.

| Assessment of the tools
All tools described the purpose of the instrument, and the majority (74.1%) were available free of charge. The most important usability issue identified related to readability, with only 11.1% of tools requiring a 7th-grade or lower reading level. Description of included evaluation tools (name, authors, country, year, type and objectives). "Questionnaire" defined as a set of written questions used for collecting information; "Survey" is a set of questions used to aggregate data for statistical analysis; "Scale" is used to measure or order entities with respect to quantitative attributes of traits; "Index" is a compound measure that aggregated multiple indicators in order to summarize and rank specific observations. TA B L E 2 (Continued) evaluation tools for engagement in research and health system decision making, most of which were developed in the last decade. These findings suggest that engagement evaluation activity is increasing around the globe and provide an important basis for future evaluation work.

| D ISCUSS I ON
One of this review major contributions is that it not only identifies existing evaluation tools for patient and public engagement, but it also systematically assesses their main strengths, weaknesses and characteristics using predetermined criteria codeveloped with key stakeholders, thus complementing related work in the area. 28 The goal of our assessment grid was not to provide an overall quality score, but aimed to guide user's selection of tools to fit their own evaluation needs. As such, our assessment grid cannot be used to "rank" evaluation tools or to identify the "best" tool, but helps identify the strengths and weaknesses of each.
A number of potential weaknesses were identified regarding evaluation tools' development process. First, scientifically rigorous methods must be used to develop evaluation tools, including more frequent psychometric testing and validation studies. 29 The fact that only a small number (11%) of instruments are informed by a literature review in at least 2 databases is disconcerting, pointing to a potential duplication of effort (new tools being created because existing ones are unknown) and misalignment with key dimensions of engagement documented in the scientific literature. Secondly, efforts must be made to address the lack of an explicit conceptual framework in most tools, which is significant given the importance of linking empirical evaluation with an explicit theoretical foundation. 6,30,31 Lastly, the high level of literacy required to understand most instruments should be addressed, particularly because patients and members of the public are the target users of most evaluation tools and because engagement with vulnerable populations is a frequent concern. 32 In line with the ethos of participation, key stakeholders are often engaged in the development of evaluation tools for engagement.
However, the involvement of patients and the public has largely been limited to the data collection stage and rarely extended to the design of evaluation instruments or the reporting of evaluation results.
The predominance of context and process evaluation instruments is surprising, given the frequent calls for greater evaluation of the impact of engagement outcomes. 33 Evaluating the context and process of engagement is consistent with the objective of developmental and formative evaluation as a means of improving engagement practices. 7,12,34,35 The number of evaluation tools measuring "perceived self-reported impacts" as the main measure for outcomes suggests a need for new evaluation tools based on observable impacts. 36

| Strengths and limitations of the review
The main strengths of this review are its comprehensive search strat- reporting guidelines for patient and public engagement research. 14,37 Evaluators and journal editors should pay special attention to publishing evaluation instruments, and properly index them to facilitate the development and use of common instruments across studies. Furthermore, a number of assessment criteria could not be fully assessed due to a lack of published information. It is possible that certain assessment criteria were applied but not reported by tools' developers (eg, literature review was performed but not reported), thus reducing the assessment score of the tool. The readability level of about a third of the tools could not be assessed because the complete tool was unavailable. rapid growth of the engagement evaluation field, it is possible that relevant evaluation tools are under development or that existing tools are being refined: an update of this review is recommended in a few years.

| CON CLUS ION
A growing set of tools is available for the evaluation of patient and public engagement in research and health system decision making.
Knowledge of the tools' specific strengths and weaknesses can guide