An evaluation of WHO emergency guidelines for Zika virus disease

Abstract Background In the face of an unclear causal association between Zika virus in utero exposure and congenital abnormalities and urgent demand for guidance, the World Health Organization (WHO) had to produce timely and trustworthy guidelines during the 2016 Public Health Emergency of International Concern (PHEIC). Methods This is a cross‐sectional evaluation of WHO emergency guidelines produced during the Zika virus disease PHEIC from 1 February to 18 November 2016. We assessed adherence to WHO publication requirements and the reporting of guideline development processes associated with trustworthiness. In the absence of quality appraisal tools for guidelines developed under compressed timeframes, we applied the Appraisal of Guidelines for Research and Evaluation (AGREE II) tool. Results We included 21 guidelines (13 de novo and 8 updates). Six guidelines used a formal evidence review process. Most guidelines involved external experts in the development process and collected declarations of interest. Peer review was reported in six documents. Most emergency guidelines included updating plans. The highest scoring AGREE II domain was clarity of presentation (median score 78%); the lowest scoring domain was applicability (median score 18%). Conclusion WHO developed moderate‐ to high‐quality emergency guidelines in the challenging context of a PHEIC. We found improvement opportunities for WHO guideline development teams in the use of evidence to formulate recommendations, the collection of declarations of interest, reporting of conflicts of interest, and the use of existing WHO organizational quality assurance processes.


INTRODUCTION
Zika virus (ZIKV) disease and its multiple long-term complications (eg, congenital malformations and developmental abnormalities) continue to impose a considerable burden on affected communities, con- There is no treatment available for ZIKV infection, 3 thus the current state-of-the science for ZIKV disease management is centered on public health interventions to limit disease transmission to vulnerable populations and to provide supportive care for newborns and infants with congenital malformations and developmental abnormalities [5][6][7][8] requiring prolonged allocation of finite health care resources. 9,10 Furthermore, the implications for blood product safety and maternal, child and reproductive health highlight the cross-cutting nature of ZIKV disease and yet again reveal deficiencies in international preparedness and capacity to respond to biologic threats.
In the early phases of the 2015-2016 ZIKV disease outbreak in the Americas, the causal association between in utero exposure to ZIKV and congenital abnormalities was unclear 7 leaving public health authorities without a clear path to produce timely, high-quality and trustworthy guidelines. Urgent demand, scarcity of structured scientific data, political instability in affected areas leading to uncertainty about implementation capability, and the need to produce multiple guidelines concurrently were additional challenges besetting WHO ZIKV disease guidelines apart from traditional clinical or public health guidelines. Finally, there is no international consensus on the optimal processes and methods for developing such guidelines. In response to these challenges, the WHO Guidelines Review Committee Secretariat and the Department of Pandemic and Epidemic Diseases launched emergency guideline templates and prototypical guideline development processes which built on lessons learned with other recent public health emergencies. 11 The objective of this study was to describe the characteristics related to trustworthiness of WHO ZIKV guidelines published in response to the ZIKV Public Health Emergency of International Concern (PHEIC) in 2016, and to assess their quality in terms of stakeholder involvement, rigor of development, clarity of presentation, inclusion of implementation considerations, and management of conflicts of interests. We extracted guideline characteristics and information on publication format. In addition, we tracked updates throughout the study period, identified translations to regionally important languages, and described guideline quality assurance and control efforts such as external peer review, Guidelines Review Committee (GRC) approval (WHO's internal quality control process for guidelines) or presence in e-Pub.

METHODS
We also identified and extracted data on the characteristics, format and related text for discrete recommendations within each included guideline. WHO defines recommendations as statements that tell endusers what to do in specific situations to achieve the best health outcomes. 12 We further defined discrete recommendations as easily identifiable statements that were distinct or separated from the body of the guideline text or clearly labeled as recommendations.
All data were extracted by one coauthor and independently checked by a second one. This study did not undergo ethics review.

Guideline quality assessment
Although validated quality assessment tools specific for emergency guidelines are not available in the medical literature, 13 the same principles for quality apply to all types of guidelines. Therefore, for this evaluation, we assumed that the guideline trustworthiness principles of transparency; minimization of risk of bias in the assessment of primary studies, synthesis of evidence and in the formulation of recommendations; and the presentation of implementable recommendations apply to all types of guidelines and recommendations (Table 1). 14 TA B L E 1 Essential steps for developing trustworthy guidelines 1.
Guideline development processes and funding sources need to be detailed and accessible.

2.
Contributors need to disclose all relevant interests, and conflicts need to be appropriately managed.

3.
The guideline development group should be multidisciplinary and balanced, including relevant stakeholders and persons affected by the recommendations.

4.
High-quality evidence reviews should underpin recommendations.

5.
Each recommendation should be accompanied by a rationale statement, an assessment of the certainty of the evidence, the strength of the recommendation, and any differences in opinion among the guideline development group members.

6.
Recommendations should be clearly articulated and precise.

7.
External review of the draft guideline should encompass a full spectrum of relevant stakeholders.

8.
Plans for updating should be included and emerging data should be monitored.
In addition, four independent appraisals of each guideline were performed using the Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument. 15,16 This widely accepted tool contains 23 items across six domains: scope and purpose; stakeholder involvement; rigor of development; clarity of presentation; applicability; and editorial independence. The appraisers were trained to use AGREE II in accordance with the latest version of the instrument manual. 17 We report the prevalence of key characteristics and present AGREE II scores for each of the six domains for each guideline (scaled to a percentage of the maximum score) and then report scores across guidelines using the median value and the interquartile range for each domain. All statistical analyses were conducted using Stata 12.1 (Stata-Corp LP, College Station, Texas, USA).

RESULTS
We identified 21 WHO ZIKV guidelines published during the study period, of which 13 were developed for the first time (de novo) and 8 were updates (  Figure 2).
For the recommendation-level assessment, we identified 58 discrete recommendations contained in 6 of the 13 guidelines that were current at the time of data extraction (November 2016) (

DISCUSSION
Our study suggests that WHO can efficiently and effectively develop and publish timely, moderate-to high-quality guidelines in the very Standards for the assessment of guideline quality such as the components for trustworthy guidelines 14 and AGREE II, [15][16][17] were developed for clinical guidelines in the nonemergency context where there is a longer time-period available for development, knowledge gaps and end-user needs may be much clearer, there may be more (high-quality) structured scientific evidence, and the implementation settings and health systems will likely be less fragile and chaotic. Thus, the applicability of these tools to emergency guidelines may be limited and our evaluations must be interpreted with caution. Certainly, these tools should not be used to determine if a specific emergency guideline is trustworthy, high-quality and impactful in areas affected by ZIKV disease outbreaks.
The high scores in the AGREE II quality domains of "scope and purpose" and "clarity of presentation" are similar to those reported in a previous evaluation of WHO emergency guidelines. 11 These high scores may be attributable to the operational focus of both groups of guidelines, requiring narrow and specific health questions, clear target populations and a concise format. In addition, the items that constitute these two AGREE II domains were clearly described in the template used for most of the guidelines in our study. Our findings are also consistent with a previous evaluation of standard WHO guidelines produced outside of emergency situations, 18 thus the high scores in these domains may reflect other aspects of WHO guideline processes that were not captured in our study.

F I G U R E 2 Zika virus disease guidelines AGREE II scores
Note: AGREE II quality domain scores (vertical axis) are based on standardized scores of the four independent reviewers for each domain (horizontal axis) (11)(12)(13). The boxes represent the interquartile ranges, and the middle line the median score. The whiskers represent the ranges.

Characteristic n (%)
Recommendation extracted from a guideline that used some type of evidence review We did not evaluate many important questions such as how to prioritize topics for guideline development during an emergency, how to assess existing guidelines for relevance and applicability, optimal dissemination methods and tools, the uptake by end-users, and most importantly, the impact on health outcomes of the affected populations. We also did not address the validity of the recommendations issued in the emergency context. As new data and updated guidelines become available, the validity of the previous recommendations and the robustness of the processes used to formulate them should be assessed and the results used to improve processes and methods in future.
WHO has an intentionally broad definition for guidelines, which may have resulted in misclassification of some information products as guidelines when the purpose of the document was not to provide normative guidance. However, we had a clear operational definition and eligibility criteria for guidelines and 90% of the included documents used the emergency interim guidance template, suggesting that misclassification was likely rare. Another potential limitation was our reliance on information that was reported in guidelines; we did not query guideline authors for additional information. We chose to focus primarily on rigor of quality of development as well as WHO's reporting requirements; we did not apply the CheckUp checklist for reporting of updated guidelines although many of its items were encompassed by our evaluation. 19

CONCLUSION
WHO developed moderate-to high-quality emergency guidelines in the challenging context of a PHEIC. We found improvement opportu-

AUTHORS' CONTRIBUTIONS
SLN conceived the research question, designed the study, contributed to data analysis and interpretation, and finalized the paper. MF conceived the research question, designed the study, contributed to data collection, analysis, and interpretation, and drafted the first version of the paper. HL contributed to data collection, analysis, and interpretation. VIS contributed to data collection, analysis, and interpretation and prepared the study figures. TVP, YHAL, QW, and AL contributed to data collection. All authors had access to all data and reviewed the drafts, made critical comments, and approved the final submission version.

DISCLAIMER
The authors alone are responsible for the views expressed in this article and they do not necessarily represent the views, decisions, or policies of the institutions with which they are affiliated.