Delivery arrangements for health systems in low-income countries: an overview of systematic reviews

  • Protocol
  • Overview

Authors

  • Agustín Ciapponi,

    Corresponding author
    1. Southern American Branch of the Iberoamerican Cochrane Centre, Argentine Cochrane Centre IECS, Institute for Clinical Effectiveness and Health Policy, Buenos Aires, Capital Federal, Argentina
    • Agustín Ciapponi, Argentine Cochrane Centre IECS, Institute for Clinical Effectiveness and Health Policy, Southern American Branch of the Iberoamerican Cochrane Centre, Dr. Emilio Ravignani 2024, Buenos Aires, Capital Federal, C1414CPV - C1181ACH, Argentina. agustin.ciapponi@hospitalitaliano.org.ar. aciapponi@gmail.com.

    Search for more papers by this author
  • Simon Lewin,

    1. Norwegian Knowledge Centre for the Health Services, Global Health Unit, Oslo, Norway
    2. Medical Research Council of South Africa, Health Systems Research Unit, Tygerberg, South Africa
    Search for more papers by this author
  • Gabriel Bastías,

    1. Pontificia Universidad Católica de Chile, Department of Public Health, School of Medicine, Santiago, Chile
    Search for more papers by this author
  • Lilian Dudley,

    1. Stellenbosch University, Division of Community Health, Faculty of Medicine and Health Sciences, Cape Town, South Africa
    Search for more papers by this author
  • Signe Flottorp,

    1. Norwegian Knowledge Centre for the Health Services, Oslo, Norway
    Search for more papers by this author
  • Marie-Pierre Gagnon,

    1. Centre de Recherche du CHU de Québec (CRCHUQ) - Hôpital St-François d'Assise, Québec, QC, Canada
    Search for more papers by this author
  • Sebastian Garcia Marti,

    1. Institute for Clinical Effectiveness and Health Policy, Buenos Aires, Capital Federal, Argentina
    Search for more papers by this author
  • Claire Glenton,

    1. Norwegian Knowledge Centre for the Health Services, Global Health Unit, Oslo, Norway
    Search for more papers by this author
  • Cristian A Herrera,

    1. Pontificia Universidad Católica de Chile, Department of Public Health, School of Medicine, Santiago, Chile
    2. Pontificia Universidad Católica de Chile, Evidence Based Health Care Program, Santiago, Chile
    Search for more papers by this author
  • Charles I Okwundu,

    1. Stellenbosch University, Centre for Evidence-based Health Care,Faculty of Medicine and Health Sciences, Tygerberg, Cape Town, South Africa
    Search for more papers by this author
  • Newton Opiyo,

    1. Kenya Medical Research Institute/Wellcome Trust Research Programme, Child and Newborn Health Group, Nairobi, Kenya
    Search for more papers by this author
  • Andrew D Oxman,

    1. Norwegian Knowledge Centre for the Health Services, Global Health Unit, Oslo, Norway
    Search for more papers by this author
  • Tomas Pantoja,

    1. Pontificia Universidad Católica de Chile, Evidence Based Health Care Program, Santiago, Chile
    2. Pontificia Universidad Católica de Chile, Department of Family Medicine, Faculty of Medicine, Santiago, Chile
    Search for more papers by this author
  • Elizabeth Paulsen,

    1. Norwegian Knowledge Centre for the Health Services, Global Health Unit, Oslo, Norway
    Search for more papers by this author
  • Blanca Peñaloza,

    1. Pontificia Universidad Católica de Chile, Evidence Based Health Care Program, Santiago, Chile
    2. Pontificia Universidad Católica de Chile, Department of Family Medicine, Faculty of Medicine, Santiago, Chile
    Search for more papers by this author
  • Gabriel Rada,

    1. Pontificia Universidad Católica de Chile, Evidence Based Health Care Program, Santiago, Chile
    2. Pontificia Universidad Católica de Chile, Department of Internal Medicine, Faculty of Medicine, Santiago, Chile
    Search for more papers by this author
  • Fatima Suleman,

    1. School of Pharmacy and Pharmacology, University of KwaZulu-Natal, Division of Pharmacy Practice, Durban, KZN, South Africa
    Search for more papers by this author
  • Charles Shey Wiysonge

    1. Stellenbosch University, Division of Community Health, Faculty of Medicine and Health Sciences, Cape Town, South Africa
    2. Stellenbosch University, Centre for Evidence-based Health Care, Faculty of Medicine and Health Sciences, Cape Town, South Africa
    Search for more papers by this author

Abstract

This is the protocol for a review and there is no abstract. The objectives are as follows:

The objectives of the overview are to:

  • provide a broad overview of what is known about the effects of delivery arrangements for health systems in low-income countries based on the findings of up-to-date systematic reviews;

  • identify needs and priorities for evaluations of alternative delivery arrangements based on the findings of included systematic reviews;

  • identify needs and priorities for systematic reviews of the effects of delivery arrangements for which we are unable to find a reliable, up-to-date systematic review;

  • inform decisions about refinements of the framework for types of delivery arrangements outlined in Table 1.

Background

This is one of four overviews of systematic reviews of strategies for improving health systems in low-income countries (Herrera 2014; Pantoja 2014; Wiysonge 2014). The aim is to provide broad overviews of what is known about the effects of health system arrangements, including delivery, financial and governance arrangements and implementation strategies. This overview addresses delivery arrangements.

The scope of each of the four overviews is summarized below.

  1. Delivery arrangements include changes in who receives care and when, who provides care, the working conditions of those who provide care, coordination of care amongst different providers, where care is provided, the use of information and communication technology to deliver care, and quality and safety systems.

  2. Financial arrangements include changes in how funds are collected, insurance schemes, how services are purchased, and the use of targeted financial incentives or disincentives (Wiysonge 2014).

  3. Governance arrangements include changes in rules or processes that determine authority and accountability for health policies, organisations, commercial products and health professionals, and the involvement of stakeholders in decision making (Herrera 2014).

  4. Implementation strategies include interventions designed to bring about changes in healthcare organizations, the behaviour of healthcare professionals or the use of health services by healthcare recipients (Pantoja 2014).

Description of the condition

How services are delivered can have impacts on the effectiveness, efficiency and equity of health systems. Outcomes that can potentially be affected by changes in delivery arrangements include patient outcomes (health and health behaviours), the quality or utilisation of healthcare services, resource use, health care provider outcomes (such as sick leave), and social outcomes (such as poverty or employment) (EPOC 2013). Impacts on these outcomes can be intended and desirable or unintended and undesirable. In addition, the effects of delivery arrangements on these outcomes can either reduce or increase inequities.

Health systems in low-income countries differ from those in high-income countries in terms of the availability of resources and access to services. Thus, some problems in high-income countries are not relevant to low-income countries, such as how best to deliver expensive technologies that are not available in low-income countries. Similarly, some problems in low-income countries are not relevant to high-income countries, such as how to delivery services that are already widely available or not needed in high-income countries. Our focus in this overview is specifically on delivery arrangements in low-income countries. By low-income countries, we mean countries that are classified as low or lower-middle-income by The World Bank Group 2014. Because upper-middle-income countries often have a mixture of health systems with problems similar to both those in low-income countries and high-income countries, our focus is relevant to middle-income countries, but excludes consideration of conditions that are not relevant in low-income countries and are relevant in middle-income countries.

Description of the interventions

The types of interventions that will be included in this overview are listed in Table 1 using a structure derived from the taxonomy for health system arrangements used in Health Systems Evidence (http://www.mcmasterhealthforum.org/hse/).

Table 1. Types of delivery arrangements
  1. * Systems include structures or organizational arrangements.

Delivery arrangement Definition
Who receives care and when
Queuing strategiesDifferent ways of managing waiting lists
Group versus individual careProviding care to groups versus individual patients
Who provides care
Pre-licensure educationHow health professionals are educated
Recruitment and retention strategiesStrategies for recruiting to and retaining health workers in specific areas or types of work
Movement of health workers between public and private careStrategies for managing the movement of health workers between public and private organizations
Role expansion or task shiftingExpanding tasks undertaken by a cadre of health workers or shifting tasks from one cadre to another
Self-managementShifting the provision of care to patients or their families
Co-ordination of care
IntegrationIntegration of the delivery of different type of services
Packages of careIntegrated packages of care such as the Integrated Management of Childhood Illness (IMCI)
Case management

Use of individuals, often specially trained nurses, to coordinate

care for patients with multiple or complex needs

Disease management

Programs designed to manage or prevent a

chronic condition using a systematic approach to care and potentially employing multiple ways of influencing patients, providers or the process of care

Care pathwaysStrategies to link evidence to practice for specific health conditions. These strategies detail the local structure, systems and time-frames to address recommendations
TeamsCare provided by teams or interdisciplinary collaboration
Communication between providersSystems* or strategies for communication between health care providers
Referral systemsSystems* for managing referrals of patients between health care providers
Discharge planningSystems* for planning the discharge of patients from facilities
Where care is provided
Site of service deliveryChanges in where care is provided including home versus facility, inpatient versus outpatient, specialized versus non-specialized facility
Intermediate careServices designed to facilitate the transition from hospital to home
Specialist outreachRegular visits by specialist providers to primary care or rural hospital settings
Generalist outreachRegular visits by generalist doctors to primary care or rural hospital settings
Transportation servicesArrangements for transporting patients from one site to another
Mobile unitsMobile facilities that visit and deliver services on a regular basis
Facilities and equipmentChanges in health care facilities or equipment
Size of organisationsConsequences of differences in the size of health service provider organisations
Procurement and distribution of suppliesSystems* for procuring and distributing drugs or other supplies
Information and communication technology
Health information systemsHealth record and health management systems
Patient reminder and recall systemsSystems* for recalling patients for follow-up or prevention
E-HealthThe combined use of electronic communication and information technology in the health sector. This includes the use of digital data – transmitted, stored and retrieved electronically – for clinical, educational and administrative purposes
Quality and safety systems
Quality monitoring and improvement systemsSystems* for monitoring and improving the quality of health care
Safety monitoring and improvement systemsSystems* for monitoring and improving the safety of health care
Working conditions of health workers
WorkloadChanges in the workload of health workers
Work environmentChanges in the working environment of health workers
Staff supportProvision of staff support to health workers
Health and safety systemsSystems* for protecting or promoting the health and safety of health workers

How the intervention might work

Changes in delivery arrangements can affect health and related goals in multiple ways and can have both desirable and undesirable effects. Examples of how changes in different types of delivery arrangements might lead to improvements in health systems and thereby better health outcomes are listed in Table 2.

Table 2. Examples of how changes in delivery arrangements might work
Delivery arrangement How this might work
Who receives care and when
Queuing strategiesStrategies such as increasing capacity or productivity might reduce waiting times by increasing the supply of services. Strategies such as co-payments, explicit referral criteria or clinical priority scores might decrease waiting times by reducing or managing demand
Group versus individual careGroup care might expand coverage by increasing the numbers of patients health workers can see and might improve effectiveness through peer support
Who provides care
Pre-licensure educationStrategies that help to ensure that students complete their education might improve access to care by increasing the supply of health professionals
Recruitment and retention strategiesStrategies that help to recruit health professionals to underserved areas or keep them there might improve access to care and equity
Movement of health workers between public and private careStrategies that attract or keep health workers in the public sector might improve access to care, equity and sustainability
Role expansion or task shiftingRole expansion or task shifting form more to less specialized health workers might improve access, coverage and equity
Self-managementShifting responsibility for care from health workers to patients might improve access for other patients, empower patients and reduce resource use
Coordination of care
IntegrationBringing together several service functions might increase service coherence and reduce fragmentation, thereby improving access, utilization and efficiency. On the other hand, vertical (non-integrated programmes) might improve the delivery of effective interventions, thereby improving health outcomes.
Packages of carePackages of care, such as the Integrated Management of Childhood Illnesses, might improve coverage, delivery quality and utilisation of effective interventions and thereby improve health outcomes
Case managementCase management might improve quality of care and patient compliance and efficiency by ensuring that patients are followed up and reducing fragmentation
Disease managementDisease management might improve the quality of care and efficiency by reducing fragmentation
Care pathwaysAn evidence-based plan of care that aims to promote organized and efficient multidisciplinary patient care might improve the quality of care and efficiency
TeamsMultidisciplinary teams of health professionals might improve the quality of care, reduce delays and fragmentation and thereby health outcomes
Communication between providersImproved communication between providers might improve the quality of care and efficiency
Referral systemsEffective referral systems might improve the quality of care by helping ensure that patients who need specialized care receive it and improve efficiency by reducing inappropriate referrals
Discharge planningStrategies that help to ensure that patients are discharged as soon as they are ready might improve efficiency by reducing unnecessary hospital utilisation. Strategies that help to ensure that patients are managed appropriately following discharge might improve the quality of care and efficiency by reducing re-hospitalisation
Where care is provided
Site of service deliveryProviding services closer to patients (e.g. in rural areas) might improve access and utilisation
Intermediate careFacilities that offer a transition between hospital care and home care might improve efficiency by reducing the length of hospital stays and might improve the quality of care following discharge from the hospital
Specialist outreachProviding specialist services closer to patients (e.g. in rural areas) might improve access.
Generalist outreachProviding generalist services closer to patients (e.g. in rural areas) might improve access.
Transportation servicesStrategies that make it easier for patients to travel to and from health facilities might improve access and utilisation
Mobile unitsMobile units might improve utilisation by making it easier for patients to access services.
Facilities and equipmentStrategies that improve the availability of facilities and equipment might improve access and utilisation
Size of organisationsLarger organisations might improve efficiency because of economies of scale. They might also improve the quality of care for procedures where there are better outcomes with a high volume. On the other hand changing the size of organisations (e.g. mergers) might reduce efficiency and quality of care during a transition period because of the need to integrate different systems. Also, very large organisations may be difficult to manage, increase administrative costs and have communication problems that might reduce efficiency and quality of care
Procurement and distribution of suppliesStrategies that improve the procurement and distribution of supplies might reduce resource use and improve the quality of care by ensuring that necessary supplies are available
Information and communication technology
Health information systemsHealth information systems might improve the quality of care and efficiency by improving communication, coordination and decision making
Patient reminder and recall systemsPatient reminder and recall systems might increase utilisation and the quality of care by helping to ensure that patients receive effective interventions
E-HealthElectronic communication of health information might improve access to care by making it easier for patients and generalists to consult with specialists and for information to be shared between patients, providers and the health system
Quality and safety systems
Quality monitoring and improvement systemsMonitoring systems might help to ensure that problems with the quality of care are identified and addressed. Routine, structured processes to address problems might help to improve the quality of care
Safety monitoring and improvement systemsMonitoring systems might help to ensure that problems with safety are identified and addressed. Routine, structured processes to address problems might help to improve safety
Working conditions of health workers
WorkloadStrategies to manage workloads might improve efficiency by helping to ensure health workers have an optimal amount of work. They might improve access to care by reducing burn-out, absenteeism and loss of health workers
Work environmentImprovements to the work environment might improve the quality of care and efficiency by improving working conditions. They might improve access to care by helping to attract and retain health workers
Staff supportStaff support might reduce burn-out, absenteeism and loss of health workers and thereby improve access to care
Health and safety systemsHealth and safety systems might reduce injuries and illness among health workers and thereby improve access to care and reduce resource use needed to care for injured or ill health workers

Why it is important to do this overview

Our aim is to provide a broad overview of what is known from existing systematic reviews on the effects of delivery arrangements for health systems in low-income countries. Such a broad overview can help policy makers, their support staff and stakeholders to identify strategies for addressing problems and improving their health systems. This overview will also help to identify needs and priorities for evaluations of delivery arrangements, as well as priorities for systematic reviews of the effects of delivery arrangements. The overview will also help to refine the framework outlined above for considering alternative delivery arrangements.

Changes in health systems are complex. They may be difficult to evaluate, the applicability of the findings of evaluations from one setting to another may be uncertain, and synthesizing the findings of evaluations may be difficult. However, the alternative to well-designed evaluations is poorly designed evaluations, the alternative to systematic reviews is non-systematic reviews, and the alternative to using the findings of systematic reviews to inform decisions is the use of non-systematic reviews to informed decisions.

Other types of information, including context specific information and judgements, including judgements about the applicability of the findings of systematic reviews in a specific context, are still needed. Nonetheless, this overview can help people making decisions about delivery arrangements by summarizing the findings of available systematic reviews, including estimates of the effects of changes in delivery arrangements and the certainty of those estimates, by identifying important uncertainties identified by those systematic reviews and by identifying where new or updated systematic reviews are needed. The overview can also help to inform judgements about the relevance of the available evidence in a specific context (Rosenbaum 2011).

Objectives

The objectives of the overview are to:

  • provide a broad overview of what is known about the effects of delivery arrangements for health systems in low-income countries based on the findings of up-to-date systematic reviews;

  • identify needs and priorities for evaluations of alternative delivery arrangements based on the findings of included systematic reviews;

  • identify needs and priorities for systematic reviews of the effects of delivery arrangements for which we are unable to find a reliable, up-to-date systematic review;

  • inform decisions about refinements of the framework for types of delivery arrangements outlined in Table 1.

Methods

We will use the methods described below in all four overviews of health system arrangements and implementation strategies in low-income countries (Herrera 2014; Pantoja 2014; Wiysonge 2014).

Criteria for considering reviews for inclusion

We will include systematic reviews that:

  • have a "Methods" section with explicit selection criteria;

  • assess the effects of delivery arrangements (as defined in Background);

  • report at least one of the following types of outcomes: patient outcomes (health and health behaviours), the quality or utilisation of healthcare services, resource use, health care provider outcomes (such as sick leave), or social outcomes (such as poverty or employment);

  • are relevant to low-income countries (countries that are classified as low or lower-middle-income by The World Bank Group 2014);

  • were published within the past 10 years.

Judgments about relevance to low-income countries are sometimes difficult to make. We will base these judgements on an assessment of the likelihood that the health systems arrangements that are considered in a review address a problem that is important in low-income countries, would be feasible, and would be of interest to decision makers in low-income countries, regardless of where the included studies were conducted. So, for example, arrangements that require technology that is not widely available in low-income countries will be excluded. At least two of the overview authors will make judgements about the relevance to low-income countries and discuss with the other review authors whenever there is uncertainty. We will tabulate information from excluded reviews for which there was uncertainty about the relevance to low-income countries in the 'Characteristics of excluded reviews' table.

We will exclude reviews that only search for and include studies from a specific high-income country (excluding studies from other countries). We will exclude reviews published 10 or more years ago as these are highly unlikely to be up-to-date. We will also exclude reviews that have limitations that are important enough that the findings of the review are not reliable (Appendix 1).

Search methods for identification of reviews

We searched Health Systems Evidence (http://www.mcmasterhealthforum.org/hse/) in November 2010 using the following filters:

  • health system topics = delivery arrangements;

  • type of synthesis = systematic review or Cochrane review;

  • type of question = effectiveness;

  • publication date range = 2000–2010.

In March 2013 we searched PDQ ("pretty darn quick")-Evidence (http://www.pdq-evidence.org/) using the filter "Systematic Reviews" with no other restrictions. We will update that search periodically, excluding records that were entered into PDQ-Evidence prior to the date of the last previous search.

PDQ-Evidence is a database of evidence for decisions about health systems. It includes systematic reviews, overviews of reviews (including evidence-based policy briefs) and studies included in systematic reviews. The following databases are searched for PDQ-Evidence with no language or publication status restrictions:

  • Cochrane Database of Systematic Reviews (CDSR);

  • Database of Abstracts of Reviews of Effectiveness (DARE);

  • Health Technology Assessment Database;

  • PubMed;

  • LILACS;

  • Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) Evidence Library;

  • 3ie Systematic Reviews and Policy Briefs;

  • World Health Organization (WHO) database;

  • Campbell Library;

  • Supporting the Use of Research Evidence (SURE) Guides for Preparing and Using Evidence-Based Policy Briefs;

  • European Observatory on Health Systems and Policies;

  • UK Department for International Development (DFID);

  • National Institute for Health and Care Excellence (NICE) public health guidelines and systematic reviews;

  • Guide to Community Preventive Services;

  • Canadian Agency for Drugs and Technologies in Health (CADTH) Rx for Change;

  • McMaster Plus KT+;

  • McMaster Health Forum Evidence Briefs.

The detailed search strategies for PubMed and LILACS can be found in Appendix 2. All records in the other databases are screened.

In addition, we will screen all of the Cochrane Effective Practice and Organisation of Care (EPOC) Group systematic reviews in Archie (the Cochrane Collaboration's central server for managing documents) (http://archie.cochrane.org/) and the reference lists of relevant policy briefs and overviews of reviews.

Data collection and analysis

Selection of reviews

Two of the overview authors will independently screen the titles and abstracts found in PDQ-Evidence to identify reviews that appear to meet the inclusion criteria. Two other authors (AO and SL) will screen all of the titles and abstracts that could not be confidently included or excluded after the first screening to identify any additional eligible reviews. One of the overview authors will conduct and screen the focused searches described above.

One of the overview authors will independently apply the selection criteria to the full text of potentially eligible reviews and assess the reliability of reviews that meet all of the other selection criteria (Appendix 1). These judgements will be checked independently by two other authors (AO or SL).

Data extraction and management

We will summarize each included review using the approach developed by the SUPPORT Collaboration (Rosenbaum 2011; http://www.supportsummaries.org/). We will use standardized data extraction forms to extract data on the background of the review; the interventions, participants, settings and outcomes; the key findings; and considerations of applicability, equity, economic considerations, and monitoring and evaluation. We will assess the quality of the evidence for the main comparisons using the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach (EPOC 2013; Guyatt 2008; Schünemann 2011a; Schünemann 2011b). Each completed SUPPORT Summary will be peer-reviewed and published on an open access website (http://www.supportsummaries.org/).

Assessment of methodological quality of included reviews

We will assess the reliability of systematic reviews that meet the inclusion criteria using criteria developed by the SUPPORT and SURE collaborations (Appendix 1). Based on these criteria, we will categorize each review as having:

  • only minor limitations;

  • limitations that are important enough that it would be worthwhile to search for another systematic review and to interpret the results of this review cautiously, if a better review cannot be found;

  • limitations that are important enough to make the findings of the review unreliable and the review should not be included in the overview.

Data synthesis

The methods used to prepare a SUPPORT Summary of each review are described in detail elsewhere (http://www.supportsummaries.org/support-summaries/how-support-summaries-are-prepared/). Briefly, for each included systematic review we prepare a table summarizing what the review authors searched for and what they found, we prepare summary of findings tables for each main comparison and we assess the relevance of the findings for low-income countries. The SUPPORT Summaries include key messages, important background information, a summary of the findings of the review and structured assessments of the relevance of the review for low-income countries. The SUPPORT Summaries are reviewed by the lead author of each review, at least one content area expert, people with practical experience in low-income settings and an Cochrane EPOC Group editor (SMB, SL, AO, NO or GR). The authors of the SUPPORT Summaries respond to each comment and make appropriate revisions and the summaries are copy edited. The Cochrane EPOC Group editor determines whether the comments have been adequately addressed and the summary is ready for publication on the SUPPORT Summary website (http://www.supportsummaries.org/).

We will organize the review using a modification of a taxonomy as indicated in Health Systems Evidence (http://www.mcmasterhealthforum.org/hse/). This framework will be adjusted iteratively to ensure that all of the included reviews are appropriately categorized and that all relevant health system arrangements and implementation strategies are included and organized logically. We will prepare a table listing the included reviews and types of delivery arrangements for which we have not been able to identify a reliable, up-to-date review. We will also prepare a table of excluded reviews. This will include reviews that address a question for which another (more up-to-date or reliable) review was included, reviews that are more than 10 years old (for which a SUPPORT Summary was prepared), reviews with results that are considered not to be transferable to low-income countries, and reviews with limitations that are important enough that the findings of the review are not reliable.

We will describe the characteristics of the included reviews in a table that will include the date of the last search, any important limitations, what the review authors searched for and what they found. Our detailed assessments of the reliability of the included reviews will be summarized in a separate table showing whether each criterion in Appendix 1 was met for each review.

Our structured synthesis of the findings of our overview will be based on two tables. The main findings of each review will be summarized in a table that will include the key messages from each SUPPORT Summary. In a second table we will report the direction of the results and the certainty of the evidence for each of the following type of outcomes: health and other patient outcomes; access, coverage or utilisation; quality of care; resource use; social outcomes; impacts on equity; health care provider outcomes; adverse effects (not captured by undesirable effects on any of the preceding types of outcomes), and any other important outcomes (that do not fit into any of the preceding types of outcomes) (EPOC 2013). The direction of results will be categorized as: a desirable effect, little or no effect, an uncertain effect (very low certainty evidence), no included studies, an undesirable effect, not reported (i.e. not specified as a type of outcome that was considered by the review authors), or not relevant (i.e. no plausible mechanism by which the type of health system arrangement could affect the type of outcome).

We will take into account other relevant considerations besides the findings of the included reviews when drawing conclusions about implications for practice (EPOC 2013). This includes considerations related to the applicability of the findings and likely impacts on equity. Our conclusions about implications for systematic reviews will be based on types of delivery arrangements for which we were unable to find a reliable, up-to-date review and limitations identified in the included reviews. Our conclusions about implications for future evaluations will be based on the findings of the included reviews (EPOC 2013).

Acknowledgements

The following colleagues helped to produce the SUPPORT Summaries upon which this overview will be based: Mickey Chopra, Susan Munabi-Babigumira and Peter Steinman. Mike English and Sasha Shepperd provided comments on an earlier version of this protocol. We would also like to acknowledge the review authors and others who have provided feedback on the SUPPORT Summaries upon which this overview is based.

Appendices

Appendix 1. SUPPORT Summaries checklist for making judgements about how much confidence to place in a systematic review

Review:
Assessed by:
Date:
Section A: Methods used to identify, include and critically appraise studies

A.1 Were the criteria used for deciding which studies to include in the review reported?

Did the authors specify:

_ Types of studies

_ Participants

_ Intervention(s)

_ Outcome(s)

Coding guide - check the answers above

YES: All four should be yes

_ Yes

_ Can't tell/partially

_ No

Comments (note important limitations or uncertainty)

A.2 Was the search for evidence reasonably comprehensive?

Were the following done:

_ Language bias avoided (no restriction of inclusion based on language)

_ No restriction of inclusion based on publication status

_ Relevant databases searched (including Medline + Cochrane Library)

_ Reference lists in included articles checked

_ Authors/experts contacted

Coding guide - check the answers above:

YES: All five should be yes

PARTIALLY: Relevant databases and reference lists are both ticked off

_ Yes

_ Can't tell/partially

_ No

Comments (note important limitations or uncertainty)

A.3 Is the review reasonably up-to-date?

Were the searches done recently enough that more recent research is unlikely to be found or to change the results of the review?

Coding guide – consider how many years since the last search (e.g. if more than 10 years the review is unlikely to be up-to-date) and whether there is ongoing research

_ Yes

_ Can't tell/not sure

_ No

Comments (note important limitations or uncertainty)

A.4 Was bias in the selection of articles avoided?

Did the authors specify:

_ Explicit selection criteria

_ Independent screening of full text by at least 2 reviewers

_ List of included studies provided

_ List of excluded studies provided

Coding guide - check the above

YES: All four should be yes

_ Yes

_ Can't tell/partially

_ No

Comments (note important limitations or uncertainty)

A.5 Did the authors use appropriate criteria to assess the risk for bias in analysing the studies that are included? ( See Appendix for an example of criteria - Assessing Risk of Bias Criteria for EPOC Reviews)

_ The criteria used for assessing the risk of bias were reported

_ A table or summary of the assessment of each included study for each criterion was reported

_ Sensible criteria were used that focus on the risk of bias (and not other qualities of the studies, such as precision or applicability)

Coding guide - check the above

YES: All four should be yes

_ Yes

_ Can't tell/partially

_ No

Comments (note important limitations or uncertainty)

A.6 Overall – how would you rate the methods used to identify, include and critically appraise studies?

Summary assessment score A relates to the 5 questions above.

If the “No” or “Partial” option is used for any of the questions above, the review is likely to have important limitations.

Examples of major limitations might include not reporting explicit selection criteria, not providing a list of included studies or not assessing the risk of bias in included studies.

_ Major limitations (limitations that are important enough that the results of the review are not reliable and they should not be used in the policy brief)

_ Important limitations (limitations that are important enough that it would be worthwhile to search for another systematic review and to interpret the results of this review cautiously, if a better review cannot be found)

_ Reliable (only minor limitations)

Comments (note any major limitations or important limitations).
Section B: Methods used to analyse the findings

B.1 Were the characteristics and results of the included studies reliably reported?

Was there:

_ Independent data extraction by at least 2 reviewers

_ A table or summary of the characteristics of the participants, interventions and outcomes for the included studies

_ A table or summary of the results of the included studies.

Coding guide - check the answers above

YES: All three should be yes

_ Yes

_ Partially

_ No

_ Not applicable (e.g. no included studies)

Comments (note important limitations or uncertainty)
B.2 Were the methods used by the review authors to analyse the findings of the included studies reported?

_ Yes

_ Partially

_ No

_ Not applicable (e.g. no studies or no data)

Comments (note important limitations or uncertainty)

B.3 Did the review describe the extent of heterogeneity?

_ Did the review ensure that included studies were similar enough that it made sense to combine them, sensibly divide the included studies into homogeneous groups, or sensibly conclude that it did not make sense to combine or group the included studies?

_ Did the review discuss the extent to which there were important differences in the results of the included studies?

_ If a meta-analysis was done, was the I2, chi square test for heterogeneity or other appropriate statistic reported?

_ Yes

_ Can't tell/partially

_ No

_ Not applicable (e.g. no studies or no data)

Comments (note important limitations or uncertainty)

B.4 Were the findings of the relevant studies combined (or not combined) appropriately relative to the primary question the review addresses and the available data?

How was the data analysis done?

_ Descriptive only

_ Vote counting based on direction of effect

_ Vote counting based on statistical significance

_ Description of range of effect sizes

_ Meta-analysis

_ Meta-regression

_ Other: specify

_ Not applicable (e.g. no studies or no data)

How were the studies weighted in the analysis?

_ Equal weights (this is what is done when vote counting is used)

_ By quality or study design (this is rarely done)

_ Inverse variance (this is what is typically done in a meta-analysis)

_ Number of participants

_ Other, specify:

_ Not clear

_ Not applicable (e.g. no studies or no data)

Did the review address unit of analysis errors?

_ Yes - took clustering into account in the analysis (e.g. used intra-cluster correlation coefficient)

_ No, but acknowledged problem of unit of analysis errors

_ No mention of issue

_ Not applicable - no clustered trials or studies included

Coding guide - check the answers above

If narrative OR vote counting (where quantitative analyses would have been possible) OR inappropriate table, graph or meta-analyses OR unit of analyses errors not addressed (and should have been) the answer is likely NO.

If appropriate table, graph or meta-analysis AND appropriate weights AND the extent of heterogeneity was taken into account, the answer is likely YES.

If no studies/no data: NOT APPLICABLE

If unsure: CAN’T TELL/PARTIALLY

_ Yes

_ Can't tell/partially

_ No

_ Not applicable (e.g. no studies or no data)

Comments (note important limitations or uncertainty)

B.5 Did the review examine the extent to which specific factors might explain differences in the results of the included studies?

_ Were factors that the review authors considered as likely explanatory factors clearly described?

_ Was a sensible method used to explore the extent to which key factors explained heterogeneity?

_ Descriptive/textual

_ Graphical

_ Meta-regression

_ Other

_ Yes

_ Can't tell/partially

_ No

_ Not applicable (e.g. too few studies, no important differences in the results of the included studies, or the included studies were so dissimilar that it would not make sense to explore heterogeneity of the results)

Comments (note important limitations or uncertainty)

B.6 Overall - how would you rate the methods used to analyse the findings relative to the primary question addressed in the review?

Summary assessment score B relates to the 5 questions in this section, regarding the analysis.

If the “No” or “Partial” option is used for any of the 5 preceding questions, the review is likely to have important limitations.

Examples of major limitations might include not reporting critical characteristics of the included studies or not reporting the results of the included studies.

_ Major limitations (limitations that are important enough that the results of the review are not reliable and they should not be used in the policy brief)

_ Important limitations (limitations that are important enough that it would be worthwhile to search for another systematic review and to interpret the results of this review cautiously, if a better review cannot be found)

_ Reliable (only minor limitations)

Use comments to specify if relevant, to flag uncertainty or need for discussion
Section C: Overall assessment of the reliability of the review
C.1 Are there any other aspects of the review not mentioned before which lead you to question the results?

_ Additional methodological concerns

_ Robustness

_ Interpretation

_ Conflicts of interest (of the review authors or for included studies)

_ Other

_ No other quality issues identified

C.2 Based on the above assessments of the methods how would you rate the reliability of the review?

_ Major limitations (exclude); briefly (and politely) state the reasons for excluding the review by completing the following sentence: This review was not included in this policy brief for the following reasons:

Comments (briefly summarise any key messages or useful information that can be drawn from the review for policy makers or managers):

_ Important limitations ; briefly (and politely) state the most important limitations by editing the following sentence, if needed, and specifying what the important limitations are: This review has important limitations.

_ Reliable ; briefly note any comments that should be noted regarding the reliability of this review by editing the following sentence, if needed: This is a good quality systematic review with only minor limitations.

Appendix 2. PubMed and LILACS search strategies

PubMed

From 2000 to present. Update: weekly

#1. MEDLINE[Title/Abstract]

#2. (systematic[Title/Abstract] AND review[Title/Abstract])

#3. meta analysis[Publication Type]

#4. #1 OR #2 OR #3(Methods filter for systematic reviews –Clinical Queries–Max Specificity)

#5. overview[Title] AND (reviews[Title] OR systematic[Title]

#6. meta-review[Title]

#7. review of reviews[Title]

#8. review[Title] AND systematic reviews[Title]

#9. umbrella[Title] AND (review[Title] OR reviews[Title] OR systematic[Title])

#10. policy[Title] AND (brief[Title] OR evidence[Title])

#11. #5 OR #6 OR #7 OR #8 OR #9 OR #10 (Methods filter for overviews)

#12. #4 OR #11 (Methods filter for systematic reviews and for overviews)

LILACS

From 2000 to present. Update: weekly

(TW:"revision sistematica" OR TW:"revisao sistematica" OR TW:"systematic review" OR MH:"review literature as topic" OR MH:"meta-analysis as topic" OR PT:"meta-analysis")

OR

(PT:revision AND (TW:metaanal$ OR TW:"meta-analysis" OR TW:"metaanalise" OR TW:"meta-analisis" OR TI:overview$ OR TW:"estudio sistematico" OR TW:"systematic study" OR TW:"estudo sistematico" OR TI:review OR TI:revisao OR TI:revision OR TI:systematic OR TI:sistematico))

OR

((TW:overview OR TW:"estudio sistematico" OR TW:"systematic study" OR TW:"estudo sistematico") AND (TI:review OR TI:revisao OR TI:revision OR TI:systematic OR TI:sistematico))

Contributions of authors

All of the authors contributed to drafting and revising the protocol.

Declarations of interest

Some of the overview authors are EPOC editors or authors of potentially relevant reviews.

Sources of support

Internal sources

  • Department of Family Medicine, School of Medicine, Pontificia Universidad Católica de Chile, Santiago, Chile.

  • Institute for Clinical Effectiveness and Health Policy, Buenos Aires, Argentina.

  • Norwegian Knowledge Centre for the Health Services, Oslo, Norway.

  • University of Cape Town, Cape Town, South Africa.

External sources

  • Norwegian Agency for Development Cooperation (Norad), Oslo, Norway.

Ancillary