Recommendations to increase the impact of maternal and childbirth health systematic reviews in the Americas
Dr Frédérique A. Jacquérioz, Center for Evidence-based Global Health, Tulane University School of Public Health and Tropical Medicine, 1440 Canal Street, Ste 1820, New Orleans, LA 70112, USA.
This paper summarises the discussions and the recommendations formulated during a meeting in March 2007 on the challenges and strategies to increase the impact of maternal and childbirth health systematic reviews in the Americas. The discussions addressed three specific themes: (1) performing systematic reviews (2) updating existing reviews, and (3) diffusing and implementing evidence into practice. Practical recommendations were devised for each theme in small group discussions.
This paper outlines the major recommendations and thoughts devised by the meeting participants on the strategies to increase the impact of systematic reviews on maternal and childbirth health issues in the Americas. Specific topics discussed during the meeting include: (1) the performance of systematic reviews, (2) the updating of existing reviews, and (3) the diffusion and implementation of evidence into practice. Challenges, opportunities and innovative strategies were addressed for each topic through a series of expert presentations and panel discussions. In addition to experts from the USA and Canada, we specifically invited researchers from Latin America and Europe to broaden the perspectives. Presentations were followed by small group discussions to formulate practical recommendations. Although the meeting covered the maternal and childbirth health field, recommendations concern the method, format and delivery of systematic reviews and could be applied to the improvement of reviews in any other health fields.
During the last few decades, major progress has been made in evaluating the effects of common and high-priority health interventions and in producing high-quality systematic reviews of evidence. Worldwide, the Cochrane Collaboration (CC) has been the pioneer in the evidence-based movement and is specifically dedicated to producing up-to-date, accurate systematic reviews on the potential benefits of healthcare interventions.1 In the United States, the Agency for Healthcare Research and Quality (AHRQ) has been leading this effort since the early 1990s through its Evidence-based Practice Centres (EPC) programme.2 Under this programme, the AHRQ seeks to synthesise scientific evidence on topics relevant to clinical, behavioural, economic and delivery issues, especially for Medicare and Medicaid programmes, and to promote their use locally in making evidence-based health care decisions. The EPCs generate reviews called evidence reports or technical assessments. At present, many other organisations and institutions around the world prepare and disseminate systematic reviews and evidence-based knowledge.3
In the field of maternal and child health, the Cochrane Pregnancy and Childbirth Group and the Cochrane Neonatal Group have been the most productive, with currently 534 reviews and 128 protocols published on the online Cochrane Library. In Latin America, the Latin American Centre for Perinatology has trained professionals on how to perform a systematic review in all of the 23 Latin American countries and has promoted in all of them the implementation of evidence-based maternal and child health interventions.4 In regards to the EPCs, nine evidence reports on obstetric and gynaecological conditions have been published.
Systematic reviews are considered the least biased approach to summarise and present evidence from studies on the effects of health care interventions. The distinctive features and strengths of systematic reviews consist of their explicit and rigorous methods, including the systematic search of all independent studies. The use of statistical methods to pool and summarise the quantitative data of independent studies, called meta-analysis, increases the power to detect treatment effects. The quality of evidence, however, closely depends on the quality of the initial studies from which data are drawn. Systematic reviews differ from the traditional narrative reviews that are by definition non-systematic and often led by expert opinion, therefore more subject to biases.
Cochrane systematic reviews (CSR) require additional criteria, such as a narrow and well-defined question, a replicable method, a systematic search of studies regardless of language, geographical location, and publication status, a timely updating and interventions evaluated through randomised controlled trials when available. As such, CSRs are considered the model of systematic reviews. Common criticisms of the CSRs are that they cover mostly clinical areas of medicine and address primarily the benefits of an intervention, with less emphasis on risks, potential harms or cost-benefits. These latter issues are of particular importance for health policy makers, insurers and health care managers to inform their decision making. In addition, CSRs are driven by authors’ interests and expertise instead of a prioritised health topic agenda.
In recent years, the CC has been making efforts to expand its activities to the public health field and priority global health issues with the creation of the Cochrane Health Promotion and Public Health Field, as well as by addressing the needs of the different stakeholders with the establishment of the Cochrane Effective Practice and Organisation of Care Group.5,6 The former aims to produce reviews on interventions relevant for public health professionals, to disseminate the reviews to targeted consumers and to develop adequate methodologies to prepare reviews on complex public health interventions, including methods to use data from non-randomised studies.5 The latter produces systematic reviews of interventions designed to ‘improve professional practice and the delivery of effective health services. This includes various forms of continuing education, quality assurance, informatics, financial, organisational and regulatory interventions that can affect the ability of health care professionals to deliver services more effectively and efficiently.’6 With the creation of the Cochrane Developing Countries Network, similar efforts have been made to involve more researchers from developing countries, to offer training in the preparation of reviews and to produce reviews on priority health issues relevant for these countries, including in the field of maternal and child health.7
Recommendations for the preparation of systematic reviews
At the meeting, of which this supplement issue of the journal is a summary, most speakers were associated with the CC and, as a result, their presentations focused on the challenges and strategies to improve CSRs. However, in the discussions, the presence of many participants from various other institutions and organisations opened the discussions beyond the CC strategy. The recommendations reported in this paper reflect these various perspectives in the preparation of systematic reviews. The debate was especially dynamic and relevant in the field of public health, where complex and large-scale interventions are challenging to summarise and often involve the inclusion of findings from non-randomised and qualitative studies.
Organisations involved in systematic reviews research differ in terms of the appropriate strategies and methods to review complex public health interventions, with the CC being the most conservative. However, most agreed that evidence from observational and qualitative studies should be used and summarised to highlight the discussion of standard systematic reviews drawn from randomised controlled trials and to provide the best existing evidence when no randomised controlled trials are available. Hence, observational and quasi-randomised studies provide informative evidence on the effectiveness of an intervention, even though these types of studies have to be interpreted carefully, given their higher probability of biases.8 Likewise, qualitative studies are instructive by giving information regarding the way in which an intervention is perceived by all of those involved in implementing and receiving interventions. Qualitative research studies can help to ‘answer questions about the impact, appropriateness and acceptability of interventions and thus enhance the scope, relevance and utility of effectiveness reviews’.9
If the relevance of using evidence from non-randomised and qualitative studies was not an issue, the meeting participants agreed that appropriate and rigorous methods to select, appraise and summarise this evidence are lacking and research in this field is urgently needed. Regardless of the level of evidence of the independent studies, reviews with a systematic approach and rigorous methods produce a higher level of evidence than a narrative review or expert opinion alone. Hence statistical methods allowing the pooling of the results from observational studies will be especially useful in practice. Within the CC, methods groups, such as the Non-Randomised Studies Methods Group and the Qualitative Research Methods Group, are addressing these issues.10 Similarly, the EPC programme developed technical evidence reports, such as the ‘Rating Systems for Strength of Scientific Evidence’ report.11 Participants suggested that standard guidelines on how to develop and conduct observational studies and to report results should be created in the line of the Consolidated Standards of Reporting Trials (CONSORT) statement for randomised controlled trials.12
Systematic reviews of evidence are not only valuable to health care providers and public health professionals but also to a wide range of other stakeholders, including health policy makers, insurers and health care managers. The evidence they most need, apart from the effect of an intervention, is data on the cost–benefit, risks and harms of a given intervention. However, these questions are often not addressed in CSRs, mainly because such information is missing in primary studies and/or authors are not qualified to analyse such data. Participants suggested that efforts be made to include cost-benefit analysis when data are available with the support of health economists and to exert a pressure on the research community to better address the costs of interventions as part of primary studies.
As mentioned earlier, CSRs are driven by authors’ research interests. Evidence from priority health interventions may not be available if researchers in the field have no time or interest in undertaking a review. To improve the relevance of CSRs, participants suggested that each Cochrane Review Group develop a list of high-priority topics for review, in agreement with an advisory board, and make it available to interested reviewers. Some review groups, including the Pregnancy and Childbirth Group and the Neonatal Group, have already adopted this strategy. In addition, senior researchers who are willing to undertake a priority review but who have no time to commit could be matched with a volunteer reviewer disposed to perform the day-to-day work.
In the line of improving the relevance of systematic reviews, participants strongly suggested that stakeholder representatives (e.g. clinicians, patients, insurers, policy makers, managers, public health professionals and researchers from developing countries) be included in the selection of topic/title, as well as in every step of the protocol and review preparation. This could be ensured by creating an international advisory board for each topic. Likewise, through its Cochrane Developing Countries Network, the CC is making major efforts to increase the participation of researchers from less-developed countries to aid in developing an agenda on their priority health issues. In existing reviews, adding subgroup analysis on data specific for developing countries was recommended when enough data are available. This can also be done as part of an update. All participants were in agreement regarding the importance of developing a body of knowledge on global health issues, with specific answers for poor and technologically less-developed settings.
The AHRQ favours the approach of topic relevance by contracting EPCs to perform evidence synthesis on topics of high priority to its public and private healthcare partners and the health care community in general.2 Topics are nominated by non-federal partners, such as professional societies, health insurers, health care providers and consumers. In addition to topic relevance, these partners should provide information on how the evidence will be diffused and translated into clinical practice and how the impact of these new practices will be measured. Then, topics are selected based on specific factors, including ‘burden of disease, health equity concerns, controversy of available data, costs, potential impact of the new intervention or procedure, or relevance to the needs of Federal healthcare programmes’.2 However, EPC evidence reports and technical assessments review only domestic issues and studies conducted in the industrialised world and therefore offer a somewhat limited perspective.
Recommendation for updating reviews
Updating is a defining feature of the CSRs, unlike other systematic reviews. The recommended time frame for updating is every 2 years. The author of the original review is expected to stay informed in the review field and to take the lead in the updating process. However, many reviews are not updated in a timely manner because of a lack of motivation and time. Additionally, updating may require as much work as was originally required to prepare the original review. Thus, there is little reward to undertake this considerable task every 2 years, when new data are unlikely to change the review conclusions or new data are not available. Meeting participants considered it more appropriate for the frequency of updates to depend upon the topic, as well as the number and quality of new data, rather than upon a fixed time. Health disciplines that advance rapidly may require yearly or more frequent updates, while for others that evolve slowly, an update every 5 years would be appropriate.
To facilitate the work of reviewers, a system of centralised and permanent searches to keep track of new studies should be established at Cochrane Review Groups for each health field. When a certain quantity of data is available, authors are contacted for update. When no data are available, results and dates of the search are stated in the review, so consumers know that the review is still up-to-date. In the presence of reviews with non-significant meta-analysis results, it is useful to know when new data are likely to change the conclusions, and thus when an update is worthwhile. For this purpose, Barrowman and colleagues developed a method that predicts the number of participants required in the new independent studies to change the results of the original review.13 Although their method needs further evaluation and improvement before being recommended for general use, it represents a promising tool.
Participants suggested that innovative strategies be developed to motivate and assist reviewers, such as a registry of volunteers interested in a certain topic, a registry of authors to look to for help, the promotion of updates as ‘training’ for preparing a review, the advertisement of reviews that need new authors for update, and providing authorship for the person that performs the update (separate MEDLINE entries). Several of these suggestions are currently being debated within the CC to improve updating.
The CC provides guidelines to determine when an update is so substantial that it warrants a separate review.14 Usually, any change in the protocol means a new review, unless this concerns the inclusion of a subgroup or cost-benefit analysis. Regardless of what is done, transparency in reporting is crucial. The date of last search and a report of any changes in methods and analysis should be reported and justified.
Although these suggestions are targeted more towards the updating of the CSRs, they can certainly be applied to the updating of systematic reviews from other organisations. As for the EPC evidence reports and technical assessment, there are no specific guidelines or systematic searches in place. An update is undertaken on an individual basis when substantial new data are published, especially when results are controversial and may have an impact on practice.
Recommendations for diffusing and implementing evidence
In many areas of medicine and public health, we know what works for common and priority healthcare interventions. However, transfer of this knowledge into practice is ineffective.15,16 Organisations such as the CC provide up-to-date, accurate systematic reviews of evidence and make them available to consumers but play a limited role in ensuring that consumers acknowledge evidence and change their practices. Major efforts should be made to better diffuse and implement evidence.
Online access to evidence through bibliographic databases, specialised web sites, peer-reviewed journals and other publications is not an issue for most consumers in the industrialised world. However, in developing countries, online access is often lacking and represents a barrier to diffusion. Participants suggested that information be available for developing countries on CD-ROM and/or other appropriate media and distributed widely such as through international development agencies, research teams, ministers of health, universities. In the field of maternal and child health, an excellent example of this strategy is the free subscription to the WHO Reproductive Health Library CD-ROM provided to clinicians and health workers in developing countries,17 in which the most reliable, evidence-based healthcare information on reproductive health drawn from CSRs is made available. The Library also proposes commentaries by experts who describe the involvement of the CSRs for resource-limited settings, as well as summaries of the practical applications of Cochrane review findings. Another example is the Cochrane Developing Countries Network, which aims to improve the diffusion of adequate summaries of evidence to front-line health workers in developing countries and to increase their awareness and access to appropriate sources. Certainly more health agencies, organisations and societies could play an active role in advocating for the better diffusion and use of systematic reviews of evidence, both locally and globally.
Meeting participants suggested that consumer and professional organisations, such as medical societies, peer-reviewed journals, public health schools, World Health Organisation, federal agencies and health insurance agencies, play the role of ‘bridges’ to aid in diffusing systematic reviews of evidence to their consumers. These ‘bridges’ would designate representatives within the organisation that would have the task of transferring evidence into context, according to their targeted audience. This would include, for instance, translating the lengthy CSRs into an easy-to-read format and presenting summaries of evidence. Peer-reviewed journals could publish these summaries on priority healthcare interventions, with an emphasis on practical recommendations for clinicians. Obstetrics and Gynecology is a pioneer of this approach.18 The CC and other organisations could offer training workshops to help these representatives translate the relevant information and maintain the integrity of the evidence. In clinical academic and public health settings, providing teaching material on how a review was performed would be appropriate to enhance students’ interest and awareness of the evidence-based approach. Each ‘bridge’ would develop a specific format and delivery services adapted to its audience. Further research is needed to evaluate the effectiveness of this ‘bridges’ strategy in different settings. In the same line, the EPC programme provides technical assistance to its stakeholders to help them translate evidence into practical and adapted tools.2 In addition, health care partners who nominate topics for evidence reports are required to provide information on their strategies to diffuse and implement evidence into practice.
In today's world, the amount of scientific literature published every year is overwhelming, and it is almost impossible for clinical and public health professionals to stay up to date in a specific field and to appraise the quality of every independent study. There is an undisputed need for data from primary studies to be evaluated, pooled and summarised into a digestible product for consumers. Systematic reviews appear to be the right approach to provide up-to-date and accurate evidence. In the past decades, major efforts have been made to prepare systematic reviews on health care interventions, and evidence now exists to improve patient care and health outcomes. However, summarising evidence is only the first step in the transfer of knowledge into practice. Next steps include effective dissemination and implementation of evidence-based interventions in health care settings and the flexible and pertinent updating of evidence.
In this Supplement issue, we have reported recommendations from meeting participants to increase the impact of systematic reviews regarding these three steps, with major suggestions highlighted below:
- • Preparation – More systematic reviews on public health interventions are needed. However, these reviews pose methodological challenges when interventions are evaluated through non-randomised designs. New methods to summarise and pool data from observational studies are required to achieve this need. More active participation from stakeholders in topic selection for reviews, as well as in protocol and review preparation, is highly recommended. Efforts should also be made to increase the number of reviews on global health issues with specific answers for less developed countries.
- • Updating – A flexible updating, according to health field advances and the number and quality of new data, along with a Cochrane centralised literature search, is recommended to motivate authors and to aid in facilitating their task.
- • Dissemination and implementation – Efforts should be made to translate findings from systematic reviews into easy-to-read formats with applicable recommendations and summaries adapted to each group of consumers and different economic settings. Private and public partners aiming at improving health care services should play the role of bridges to translate evidence into practice for their own targeted group of consumers.
The views expressed in this paper are those of the authors and do not necessarily reflect the views or policies of the meeting participants’ organisations or institutions.
The authors thank Andrea Meyer for assistance with this manuscript. The meeting on maternal and childbirth health systematic reviews was held on March 1–3, 2007, in New Orleans, Louisiana and was generously sponsored by the Agency for Healthcare Research and Quality (AHRQ) through Grant #R13 HS016273 and the Tulane University School of Public Health and Tropical Medicine.
Conflict of interest
The authors declare no conflict of interest in this work.