Developing quality criteria for patient‐directed knowledge tools related to clinical practice guidelines. A development and consensus study

Abstract Background Patient‐directed knowledge tools such as patient versions of guidelines and patient decision aids are increasingly developed to facilitate shared decision making. In this paper, we report how consensus was reached within the Netherlands on quality criteria for development, content and governance of these tools. Method A 12‐month development and consensus study. The consortium worked on four work packages: (a) reviewing existing criteria; (b) drafting the quality criteria; (c) safe‐guarding the acceptability and feasibility of the draft criteria by participatory research in on‐going tool development projects; and (d) gaining formal support from national stakeholders on the quality criteria. Results We reached consensus on a 8‐step guidance; describing minimal quality criteria for (a) the team composition; (b) setting the scope; (c) identifying needs; (d) the content and format; (e) testing the draft; (f) finalizing and approval; (g) dissemination and application, and (h) ownership and revision. The participants of the on‐going tool development projects were positive about the quality criteria in general, but divided as to the degree of detail. Whereas some expressed a clear desire for procedural standards, others felt that it would be sufficient to provide only general directions. Despite the different views as to the degree of detail, consensus was reached in three stakeholder meetings. Discussion We successfully collaborated with all stakeholders and achieved formal support from national stakeholders on a set of minimum criteria for the development process, content and governance of patient‐directed knowledge tools.


| INTRODUC TI ON
The knowledge in health care is expanding daily-so that keeping up with knowledge is a challenge. 1 The development of knowledge tools is intended to support clinicians to keep pace and to improve their decision making. Many knowledge tools such as clinical practice guidelines, protocols or clinical pathways have been developed over the years. 2 With the increasing call for a patient revolution, 3 further tool types have been added to the mix, including patient decision aids. A key source of information for patient-directed knowledge tools is clinical practice guidelines. Clinical practice guidelines summarize research evidence systematically and provide recommendations on a specific clinical topic. 4 Nowadays, the GRADE approach is used as a framework to rate the quality of the evidence and to assess the strength of the recommendations taking into account the balance between benefits and harms, resource use and feasibility considerations. The GRADE method also recognizes the collective patient perspectives; 5 the strength of the recommendations is also affected by the patients' appreciation of advantages and level of acceptability of disadvantages of the intervention, such as side-effects and treatment burden. Worldwide, patients and patient representatives increasingly take part in the development of guideline recommendations. 6 Next to patient participation on a collective level, efforts are made to adapt or enrich guidelines so as to facilitate patient participation on an individual level, in clinical decision making. 7 Patient participation is especially important in case of preference-sensitive decisions where multiple options exist or where the benefits and harms of the intervention may be assigned a different weight by different patients. 8 Illustrative examples to facilitate patient participation in clinical decision making are patient versions of a guideline such as a lay summary, or patient decision aids for specific preference-sensitive decisions attached to the guideline document.
Some guidance for the content of lay summaries of guidelines is provided by the Guidelines International Network. 9 In 2006, standards were formulated for the content of patient decision aids by the International Patient Decision Aids Standards (IPDAS) group, a multistakeholder process that led to a self-assessment checklist. 10 Further work led to a measure IPDASi 11 and a set of criteria that should be met to achieve a "minimum" acceptable standard. 12 Guideline developers are experimenting to derive the information for patient decision aids-evidence on benefits and harms of interventions and on patient considerations and patient preferences-directly from the guideline. 13 Much is also happening with respect to development of patientcentred knowledge tools such as patient versions of a guideline or patient decision aids in the Netherlands, on various sides of the care equation. 14 Patient organizations are gaining a more accurate picture of the information needs that these tools must satisfy. Professional and scientific associations feel a responsibility to ensure the accuracy and effectiveness of medical information supplied via such tools. Web and tool designers continue to introduce ever more userinterface friendly tools.
For this study, we used the following definition: a patientdirected knowledge tool synthesizes and distils the highest quality knowledge and research, is aimed directly at the patient (and next of kin), with the goal to engage patients in dialogue or deliberation during a clinical encounter or to support and/or improve patient decision making which may or may not take place during a clinical encounter. 15 Yet these patient-centred knowledge tools are subject to a multitude of varying definitions and criteria, especially regarding the patient versions of guidelines, and the development process. [16][17][18][19][20][21][22] As a result, it is difficult for parties to distinguish what is truly important from what is not or what type of patient-directed knowledge tool is in fact the correct means to achieve the stated purpose. In addition to quality criteria, the need for national governance is also felt strongly, as many initiatives by patient organizations and professional bodies to develop patient-directed knowledge tools exist side by side. This situation has resulted in an uncoordinated, partly overlapping mixture of publicly and privately/commercially available patient decision aids. 23 Moreover, some of the patient decision aids do not seem to follow the rigid and multistakeholder methods to review the evidence base, as is common in clinical practice guidelines.
A guidance for the development of reliable patient versions of guidelines and patient decision aids can serve to integrate all existing knowledge and previously developed expertise, allowing the stakeholders to work together more effectively and more efficiently. The purpose of such a guidance is to promote the development of highquality, reliable and publicly available patient-directed knowledge tools, which will contribute to achieving properly informed patients and shared decision making.
Initiated by the National Health Care Institute of the Netherlands, a consortium of healthcare stakeholders started to develop such guidance, supported by academic researchers. Apart from validity, feasibility was important given the conflict between the aim of high-quality knowledge tools and the limited resources to develop such tools. The purpose of this article is twofold. First, we describe the methods used for arriving at the guidance as an illustrative example of how formal support from national stakeholders can be reached. Second, we present the list (guidance) of minimum quality criteria for the development, content and governance of patient information on guidelines and patient decision aids, as well as the way in which these tools can be connected to the clinical practice guidelines.

| ME THODS
The National Health Care Institute of the Netherlands initiated and granted this 12-month development study that was composed of a literature review, a feasibility check and a consensus procedure. We took the position that we needed various types of input and processes to ensure a successful consensus process. To this end, we designed a consortium (see first four authors and last two authors) that worked together continuously in an iterative process using crossfertilization, without being hindered by hierarchy. A representative of the Dutch Federation of Patients' Organisations (last author) was co-leading the project with the first author. (WP1) We searched for formal criteria and methodologies in the scientific literature, in policy reports, and on websites by developers of guidelines and patient decision aids. The search strings that we used to explore PubMed are described in Table 1, as are the websites to search for the grey literature. This inventory supplied the basis for the first draft of the guidance. One of the researchers of WP1 made a first selection of the search based on title and abstract, and excluded references clearly not fulfilling the inclusion criteria. All fulltext versions that resulted from this first selection were downloaded and assessed along the pre-set inclusion and exclusion criteria. In case of doubt, a second researcher was consulted to reach consensus on inclusion or exclusion.
Inclusion criteria for literature on patient versions of guidelines: • The paper describes the development process of a patient version or lay summary of a clinical practice guideline.
• Explicit description of the methods used (be it short or extensive) for development.
• English or Dutch language.
Exclusion criteria: • Papers describing only the process of patient participation in the development of a clinical practice guideline.
Inclusion criteria for literature on patient decision aids: • The title reports the term "development" or "design." • The abstract reports the development of a patient decision aid as the aim of the paper.
• Description of the development process of disease-specific or generic decision aids.
• Explicit description of the methods used (be it short or extensive) for development.
• English or Dutch language.
Exclusion criteria: • Papers describing the development of tools that stretch further than patient decision aids (eg, social support and self-management).
• Papers describing the development of tools on other decisions than medical decisions.
(WP2) The project was coordinated via monthly meetings of all WP leaders, complemented by numerous one-to-one contacts. We used definitions of the patient-directed knowledge tools that were recently formulated in another Dutch national consensus procedure; see

| RE SULTS
3.1 | How did we arrive at the guidance?

| (WP1) Inventory of existing methods and criteria in scientific and grey literature
We found 51 hits in PubMed, of which four studies were included that describe criteria for developing patient versions of guidelines (see Appendix S1). The grey literature revealed many websites pub- • Language and jargon used in the guidance were often found to be too academic.
• The amount and complexity of the work to map the patients' perspective in the scoping and needs assessment phase, for example by organising a focus group or a questionnaire survey, were often underestimated. Due to limitations in resources and the high workload, work should not be done twice, in the guideline working group and in the patient tool development group. Moreover, the required minimum number of two patients in the team-as was prescribed in the earlier drafts-was a concern, as well as the • Formal authorization of the tool was not regarded necessary by all stakeholders, with the argument that the guideline was already approved.
• All project leaders plead for a central portal to host the patient-directed knowledge tools, supported by a national party taking care of the governance of the tools.

| (WP4) The consensus meetings
For the first meeting, 28 out of 43 invited experts were present.
When asked to mark the most important sections of the guidance, In retrospect, it can be observed that the quest for clear and outspoken procedural standards that was verbalized by some project leaders in WP3 was strongly echoed in the first meeting but that it faded away in the second meeting, while only crude instructions for each step were regarded sufficient in the third and final meeting.

| The guidance
WP1 provided rich data for formulating eight distinct development steps in the guidance ( Table 2) Box 2 The guidance. A brief description of each step for the development of patient information on a guideline or a patient decision aid (PDA) connected to specific guideline recommendation(s) 1. TEAM The team composition is discussed with the relevant patients' and professional associations. The team has an independent chair and a process support member/secretary, along with at least one patient(-representative) with first-hand experience (acquired by the patients' organization). Membership of the team is approved based on written Declarations of Interests. An editor with experience in writing copy for a non-expert audience will be involved in the team. If the development of the tool is part of a guideline project, the team will (ideally) be commissioned by the guideline working group itself, which has budgeted the developmental work.

SCOPE
The team checks the availability of existing tools and establishes the objective, the target group and the rough form of the tool.
Patient information: Determine where the guideline is failing to meet patients' information needs. After all, guidelines for practice are typically drawn up from the perspective of the care provider. Whenever possible, address the major underlying questions patients have about the guideline, as well as the key recommendations.
PDA: Select one or more recommendations from the guideline that have to do with the decision at hand and that are preference-sensitive in nature.
3. NEEDS There are multiple ways to identify the needs of patients: a review of the literature, and/or additional qualitative or quantitative methodologies for collecting data, such as focus groups or questionnaires.
Patient information: Concerns any additional needs that have not yet been elaborated during the guideline development, for example with regard to multimorbidity, ethnic minorities, alternative interventions and self-management.
PDA: Involves questions the patients and their proxies may have when faced with taking a specific decision. Which needs, preferences and attributes influence a given patient's decision making? This might involve information needs and psychosocial needs, along with important strategies for self-management in connection with the illness or condition, and should also include the variations between patients.

CONTENT For both type of tools describe:
• The target group and medical condition/symptom/healthcare topic.
• The guideline(s) serving as the basis (in part) for the creation of the information on evidence, etc.
• The source of funding, who has ownership, year of publication and expiry date (if applicable).

• The interests of each member of the development group (conflict of interest).
Patient information: Describe the guideline recommendations on which patients would want to be informed in terms that a layperson can understand. Mention frequently used examples of professional jargon so that patients can become familiar with them. The patient information will additionally indicate the following aspects: • Point out where aspects have consciously been omitted and/or emphasis has intentionally shifted (if applicable), as compared to the guideline.
PDA: Describe the situation/decision at hand and the relevant recommendation(s) from the guideline, in terms that a layperson would understand. The PDA will describe the following aspects (at minimum): • An explanation that the patient has a choice; that he/she is facing a preference-sensitive decision.
• A description of the medical/care options, including the option to wait and see (if applicable) and an explanation of the procedure for each medical/care intervention.
• The desired and undesired outcomes (side-effects) of the medical or care options, and the burden of treatment.
• The likelihood and risks of the outcomes, expressed as numeric data with equal denominator of population in natural frequencies and an identical length of time; preferably displayed in population diagrams; framed both positively and negatively (chances of both survival and fatality, for example); and in the case of risk reduction presented, at minimum, in terms of absolute (and potentially relative) risk reduction.
• An evidence table in which the medical/care options are summarized and compared in terms of a few key aspects.
• Ensure explicit mention of the attributes found in step 3 that are important for patients to keep in mind as they consider their options and elicit their values. These attributes must contribute to the key aspects described in the evidence table.

TEST
The development team will present the draft to the relevant professional, scientific and patients' associations for the purpose of obtaining feedback. The parties will assess whether the patient perspective is sufficiently reflected, ensure understanding of people with low literacy, and if the medical content is accurate. If the guideline working group is still active, the draft will be presented to that group for feedback as well.
6. FINALIZING The development team establishes the final knowledge tool and presents it to the relevant professional, scientific and patients' associations for approval. This regards approval at the process level, that is concerning the creation of the knowledge tool.
Ownership is determined and formally established.

DISSEMINATION
The tool will ideally be submitted to the National Health Care Institute as a section of the relevant guideline(s). The knowledge tool becomes accessible to the public and is preferably made available at a central location, including points for attention to facilitate the actual application/implementation in healthcare practice.

OWNERSHIP
The owner(s) of the knowledge tool will manage the tool and determine when the information is due to be revised: in any case when the expiry date has been reached. Ideally, the need for revision of the tool will be considered when the guideline as a whole is revised.
(such as a lay summary or decision aid) best fits the objectives of the development group; (b) minimum criteria for the eight development steps, content and governance of each tool (Box 2); and (c) supplemental, detailed and concrete suggestions for each step in a second layer of information (14 pages in total, not presented, available on request).
Developers deviating from these minimum criteria would have to provide a rationale for why a criterion does not apply ("comply or explain").
The steps need not always be followed in linear fashion, as the guidance establishes the criteria for an effective development process rather than laying out a strictly prescribed series of ordered steps. In the event that the development of the knowledge tool (patient information on a guideline or a patient decision aid based on a specific guideline recommendation) is part of a guideline project, the development team will ideally be commissioned by the guideline working group itself.
For patient versions of guidelines, the patients' information needs together with the subject of the guideline will form the basis for the content of the patient version; the guideline itself should be the most important source of information. Rather than assuming just one guideline as a starting point, this situation might mean that multiple guidelines will need to be integrated and translated into a single patient information document, as this process will more effectively address the desires and perceptions of the target group. Or, alternatively, it might mean that only a limited number of guideline modules will be applied in creating patient information documents. A need to develop one or more patient decision aids is especially indicated when one or more of the guideline's key recommendations are preference-sensitive in nature.

| D ISCUSS I ON
We successfully collaborated as a consortium of researchers and end-user representatives, with patient participation achieved at the highest level of involvement, to achieve formal support from national stakeholders on a set of minimum criteria for the development process, content and governance of patient-directed knowledge tools related to clinical practice guidelines. What we provide is not a detailed "recipe" for development but rather a series of recommendations based on the "state of the art" and feasibility considerations.
A number of potential limitations should be mentioned. Our project was explicitly embedded in the guideline context, we did, for example not include patient versions of systematic reviews. The guideline context may be a limiting context for developing patientdirected knowledge tools. As the starting point of a clinical practice guideline is predominantly the clinicians' perspective, important issues for patients may not be covered in the guideline. The assignment from the National Health Care Institute was aimed specifically at guidelines in the context of curative health care. While the literature is unclear in this regard, it is possible that the content of the guidance might have been different had representatives from public health, long-term and palliative health care been included. One strength of this project is the systematic approach and involvement of all national stakeholders, from patients to policymakers, with patient representatives in a co-leading role. We believe that the involvement of all stakeholders from the writing phase of the project proposal contributed to the successful collaboration. Another strength of the project is that the guidance was developed with prospective feasibility checks parallel to the nine on-going development projects.
The relevance of patient-directed knowledge tools being publicly available has also been acknowledged in the UK, with one of the main institutions developing guidelines committed to develop patient decision aids based on clinical guidelines. 28 The relevance of this process was recently underpinned by empirical evidence in the Netherlands.
The Dutch College of General Practitioners launched a non-commercial public website in March 2012 that provides easy access to patient versions of guidelines. Since its launch, the website has grown to become one of the most visited Dutch healthcare websites. Healthcare usage in primary care seems to have decreased by 12% after the launch of the website. 29 We expect the criteria to evolve over time as they are further tested through developers using patient versions of guidelines and patient decision aids, as well as by adding new tools to the guidance.
The next challenge will be the effective implementation of the guidance as a further step towards ensuring the development of highquality, reliable and publicly available patient-directed knowledge tools with the support and acceptance of professional associations Formal steps towards accreditation have not yet been taken, the question being whether this procedure is needed and is warranted given the current level of evidence. In December 2016, the USA National Quality Forum released national standards for the certification of patient decision aids. 30 The certification criteria are meant to be used for "complete" patient decision aids, which are standalone, independent tools for patients facing a clinical decision. Our guidance, although not formulated along the lines of certification criteria, is highly comparable with the USA criteria in terms of content. The only USA criterion that we do not cover is that the patient decision aid should report readability levels.
We believe that this study can be seen as an inspirational example for other countries that are facing the same challenges with regard to the development and governance of cliniciandirected and patient-directed knowledge tools such as guidelines, guideline summaries, patient versions of guidelines and patient decision aids.

CO N FLI C T O F I NTE R E S T
All authors are involved in the development of clinical practice guidelines, none of them has any commercial interest, and they all stem from non-profit organizations.