Contribution of Systematic Reviews to Management Decisions



Systematic reviews comprehensively summarize evidence about the effectiveness of conservation interventions. We investigated the contribution to management decisions made by this growing body of literature. We identified 43 systematic reviews of conservation evidence, 23 of which drew some concrete conclusions relevant to management. Most reviews addressed conservation interventions relevant to policy decisions; only 35% considered practical on-the-ground management interventions. The majority of reviews covered only a small fraction of the geographic and taxonomic breadth they aimed to address (median = 13% of relevant countries and 16% of relevant taxa). The likelihood that reviews contained at least some implications for management tended to increase as geographic coverage increased and to decline as taxonomic breadth increased. These results suggest the breadth of a systematic review requires careful consideration. Reviews identified a mean of 312 relevant primary studies but excluded 88% of these because of deficiencies in design or a failure to meet other inclusion criteria. Reviews summarized on average 284 data sets and 112 years of research activity, yet the likelihood that their results had at least some implications for management did not increase as the amount of primary research summarized increased. In some cases, conclusions were elusive despite the inclusion of hundreds of data sets and years of cumulative research activity. Systematic reviews are an important part of the conservation decision making tool kit, although we believe the benefits of systematic reviews could be significantly enhanced by increasing the number of reviews focused on questions of direct relevance to on-the-ground managers; defining a more focused geographic and taxonomic breadth that better reflects available data; including a broader range of evidence types; and appraising the cost-effectiveness of interventions.

Contribuciones de las Revisiones Sistemáticas a las Decisiones de Manejo


Las revisiones sistemáticas resumen integralmente la evidencia sobre la efectividad de las intervenciones de conservación. Investigamos la contribución de las decisiones de manejo hechas por este creciente cuerpo de literatura. Identificamos 43 revisiones sistemáticas de evidencia de conservación, 23 de las cuales hicieron algunas conclusiones concretas relevantes al manejo. La mayoría de las revisiones se dirigían a intervenciones de conservación relevantes a las decisiones políticas; sólo el 35% consideraba intervenciones de manejo sobre-la-causa prácticas. La mayoría de las revisiones cubrieron solo una pequeña fracción de la amplitud geográfica y taxonómica a la que buscaban dirigirse (mediana = 13% de los países relevantes y 16% de los taxones relevantes). La probabilidad de que las revisiones tuvieran por lo menos algunas implicaciones para el manejo tendió a incrementar conforme la cobertura geográfica incrementaba y a declinar conforme aumentaba la amplitud taxonómica. Estos resultados sugieren que la amplitud de una revisión taxonómica requiere de una consideración cuidadosa. Las revisiones identificaron una media de 312 estudios primarios relevantes pero excluyeron 88% de estos por deficiencias en el diseño o fallas para coincidir con otros criterios de inclusión. Las revisiones resumieron en promedio 248 juegos de datos y 112 años de actividad de investigación, pero la probabilidad de que sus resultados tuvieran por lo menos algunas implicaciones para el manejo no incrementaron mientras la cantidad de investigación primaria resumida aumentaba. En algunos casos, las conclusiones fueron elusivas a pesar de la inclusión de cientos de conjuntos de datos y años de actividad de investigación acumulada. Las revisiones sistemáticas son una parte importante del juego de herramientas en la toma de decisiones de conservación, aunque consideramos que los beneficios de las revisiones sistemáticas podrían ser mejorados significativamente al incrementar el número de revisiones centradas en preguntas con relevancia directa a administradores sobre-la-causa; definiendo una amplitud geográfica y taxonómica más enfocada que reflejo los datos disponibles; incluyendo un rango más amplio de tipos de evidencia; y evaluando la efectividad de costo de las intervenciones.


Despite the aim of conservation science being to inform and guide management (Meffe et al. 2006), decision makers report that it can be difficult and time-consuming to access available science (Fazey et al. 2004; Zavaleta et al. 2008) and that research findings can be challenging to interpret (Pullin & Knight 2005). Debate in the literature and conflicting research findings can also cause managers to mistrust scientific information (Young & Van Aarde 2011). Compounding these challenges, the peer-reviewed literature often does not address questions of direct relevance to conservation managers (Whitten et al. 2001; Fazey et al. 2005) or deliver information when needed (Kareiva et al. 2002; Linklater 2003). These impediments to using science to guide practice have contributed to the poor use of empirical evidence to inform management decisions (Sutherland et al. 2004; Cook et al. 2010).

The challenge of translating science into practice is not unique to conservation but is common to many applied disciplines (Pfeffer & Sutton 1999; Pullin & Knight 2001). Concern about the lack of evidence in medical practice (Forsyth 1963; Cochrane 1972; Maynard & Chalmers 1997) stimulated an evidence revolution (Pullin & Knight 2001) that promoted randomized, controlled trials as the standard for credible evidence (Stevens & Milne 1997). To help practitioners manage the rapid increase in available evidence (Chalmers 1993), systematic reviews were developed as a tool to collate (systematically search the available literature), filter (identify credible sources of evidence), synthesize (analyze the body of evidence to determine the overall effect of an intervention), and disseminate the evidence for the effectiveness of potential and currently used treatment options on a topic for practitioners (Higgins & Green 2011). The rigorous methodological and statistical protocol associated with systematic reviews minimizes bias and improves their transparency and repeatability (Pullin & Stewart 2006; Newton et al. 2007). These factors make systematic reviews more comprehensive and less open to potential bias than other review formats that summarize the literature in an unstructured way (e.g., narrative reviews) (Roberts et al. 2006). A distinct focus on making recommendations for management and a systematic search of both the peer-reviewed and gray literatures generally distinguish systematic reviews from traditional meta-analytical studies. The practice of systematic reviews has been widely adopted in medicine, health sciences, education, criminology, and several other disciplines (Hansen & Rieper 2009).

The benefits of an evidence-based approach to medical practice have led several authors to promote systematic reviews as a tool for integrating science into conservation practice (e.g., Pullin & Knight 2001; Fazey et al. 2004; Sutherland et al. 2004). Systematic reviews facilitate evidence-based conservation practice by providing managers with an overview of relevant, trustworthy empirical evidence pertinent to a decision (Pullin & Stewart 2006). The evidence movement has been instrumental in highlighting the importance of evaluating the effectiveness of management interventions (Ferraro & Pattanayak 2006) so that decision makers do not waste time and money on ineffective or potentially harmful management interventions (Pullin & Knight 2001). By combining the replicates from multiple studies, systematic review can also maximize the value of primary research, generating greater explanatory power, which may reveal effects not detected by the original individual studies (Mulrow 1994).

To facilitate systematic reviews and make them freely available to managers, the Collaboration for Environmental Evidence (CEE) ( was developed (Pullin & Knight 2009). The collaboration was modeled on the Cochrane Collaboration for medical practice and the Campbell Collaboration for social programs. The CEE has developed detailed guidelines (Pullin & Stewart 2006; CEBC 2010) to assist authors wishing to conduct systematic reviews of conservation interventions.

With evidence-based conservation being embraced by the conservation community as a desirable approach to decision making, it is timely to review the benefits to conservation practice arising from systematic reviews. We examined how the method of systematic review has been applied to evidence about the effectiveness of conservation interventions and the benefits they have provided to conservation practice. We measured the level of guidance systematic reviews offer conservation managers and quantified the types of conservation questions being addressed; geographic and taxonomic breadth of reviewed topics; and the quantity of primary research available in a format suitable for systematic review. We considered the benefits to environmental management arising from systematic reviews and how systematic reviews might be improved.


Criteria and Method for Selection of Systematic Reviews

We based selection of systematic reviews on the criteria outlined in the guidelines for systematic reviews of conservation evidence published by Pullin and Roberts (2006). Accordingly, we ensured that reviewers had clearly defined a question focused on evaluating the effectiveness of a conservation intervention with implications for management or policy; used a systematic and objective approach to search the literature without limiting the publication period; established clear and transparent criteria for the inclusion of studies in the review; synthesized available data through a meta-analysis or scoring system; and discussed the findings in relation to the implications for management or policy.

Pullin and Roberts (2006) recommend that authors formulate the review question in consultation with managers and stakeholders, assess the methodological quality of relevant studies, evaluate the sources of heterogeneity in the data sets and document the data extracted from studies included in the review. Published reviews often do not document such steps, although this did not disqualify them from our sample because lack of documentation does not mean these steps were not conducted. Formal quantitative analyses within systematic reviews ideally involve a meta-analysis that includes summary effect sizes for each data set weighted according to some measure of its importance (Pullin & Stewart 2006). However, we included systematic reviews that used any quantitative approach to meta-analysis or used a scoring approach, where the results of each study were tallied according to whether the intervention yielded positive, negative, or equivocal results.

To identify relevant studies, we used electronic searches of ISI Web of Knowledge (i.e., ISI Web of Science: Science Citation Index Expanded 1945–present and ISI Proceedings: Science and Technology Proceedings 1990–present) and Google Scholar and searched for the terms “systematic review” and “conservation.” We refined each of these searches by excluding results from unrelated disciplines, such as medicine, chemistry, or physics, to limit the number of irrelevant articles returned. We also searched the online CEE library for systematic reviews listed as completed. We extracted several pieces of information from each review that met our selection criteria (summarized later and detailed in Supporting Information).

Types and Breadth of Reviews

We determined whether the key audience for the review findings was policy makers or on-the-ground managers (Supporting Information).

We were interested in whether there is a trade-off between the geographic and taxonomic breadth of a review and whether it had direct implications for management. We used the selection criteria described in the review to establish the intended breadth of the review (the pool of research from which data could be drawn—e.g., all birds). We then used the data captured within the meta-analysis to determine the realized coverage of the review (e.g., number of bird species with data included in the meta-analysis). We measured geographic scope by the number of countries relevant to the review topic and realized geographic coverage by the proportion of the geographic scope for which data were represented in the meta-analysis. Taxonomic scope was the number of species relevant to the review topic, and taxonomic coverage was the proportion of the taxonomic scope with data included in the meta-analysis. We estimated taxonomic scope when the definition was ecological rather than strictly taxonomic (e.g., ground- and cliff-nesting birds). If reliable estimates could not be found, we omitted the review from the analysis of scope. In many cases, such as when estimates could be obtained for only some taxonomic groups, only the minimum number of taxa could be estimated. For the purpose of analyses, we normalized the geographic and taxonomic coverage variables by log10 transformation.

Evidence Captured by Reviews

To estimate the level of evidence available for each systematic review, we recorded the number of studies the review authors reported as relevant to the review topic after evaluating the title and abstract. Although we assumed abstracts accurately reflected the contents of the paper, if this was not always the case, our results will overestimate the number of relevant studies. Although the majority of authors reported the number of relevant studies, in some cases it was necessary to contact the corresponding author to request these data. We also recorded the number of studies that met the criteria for inclusion in each review. This figure captures the studies included in the meta-analysis and, if relevant, those studies included only in sections of the review that provide a qualitative synthesis of the relevant literature.

To capture the research effort underpinning a meta-analysis, we recorded the number of independent statistical units included in each meta-analysis. We also calculated the total number of years of data collection embodied in the primary studies being reviewed. Although both these measures of research effort are imperfect, they are indicative of the effort involved in generating the primary literature underpinning a topic. Where the number of years of research effort was not reported in a systematic review, we examined the original primary studies to extract the number of years of data collection reported therein. When this figure could not be obtained, we generated an estimate from the available data. Five systematic reviews had 65–129 primary studies in the meta-analysis, and owing to the time-consuming nature of examining each primary study, we used a random sample of one-third of studies to generate the number of years of data collection. For analyses the number of independent data sets and number of years of data collection were normalized by log10 transformation.

Findings from Reviews

The goal of systematic review is to provide a dispassionate synthesis of the best available evidence, not to be prescriptive about how a practitioner should act. Practitioners must still interpret this information according to the relevant socioeconomic and ecological circumstances (Chalmers 1993). Therefore, we evaluated the findings arising from the data synthesis, given the limitations of each review, and classified them according to their implications for management practice (Table 1) and conservation science. Where possible, we recorded the overall effect size emerging from meta-analyses. We then investigated whether attributes of the reviews contributed to the degree to which they could unambiguously inform management decisions.

Table 1. Description of the categories used to identify whether systematic reviews of conservation evidence have implications for management
Implications for managementDescriptionExample
YesThe review presents unequivocal evidence across the full scope of the review with direct implications for the conduct of management.Culling is not an effective method of reducing the abundance of yellow-legged gulls (Larus michahellis) to protect threatened waterbirds (Oro & Martínez-Abraín 2007).
Some direct implicationsFindings partially address the scope of the review, providing guidance relevant to some management contexts. This includes reviews that lack data on some of the relevant management alternatives and those where findings vary according to the environments and taxonomic groups being considered.Two of 4 herbicides significantly reduce Rhododendron ponticum, but the best concentration and method of application cannot be determined. Results for the other herbicides were inconclusive (Tyler et al. 2006).Some mammal and bird populations are displaced by road infrastructure, but the effect varies with the habitat type and species considered (Benítez-López et al. 2009).
NoNo conclusions can be drawn from the review due to insufficient data, equivocal evidence for the effectiveness of an intervention, or confounding variables in the original studies.All studies had poor design and no conclusions could be drawn (Isasi-Catalá 2010).Herbicides used close to conservation areas appear to decrease invertebrates but most studies did not control for the use of fertilizers or were greenhouse trials (Frampton & Dorne 2007).


Number of Systematic Reviews

We conducted literature searches in March 2012 and located 3375 articles (ISI Web of Knowledge, n = 42; Google Scholar, n = 3290; CEE, n = 43). The majority of articles was irrelevant because they were duplicate records; discussed rather than conducted a systematic review; did not address a conservation intervention; were systematic reviews of taxonomy; or were related to medical interventions. After reviewing these articles, 43 met our criteria for systematic reviews of conservation evidence (Supporting Information). Supporting Information provides a list of studies that met some of our criteria and the reasons they were ultimately excluded.

Findings from Reviews

Two of the 43 reviews provided guidance across the full geographic and taxonomic breadth of the review. However, 21 reviews had direct implications for only some interventions of interest or some management contexts (Fig. 1). Conclusions about the effect of the intervention could not be drawn from the remaining 20 reviews due to small sample sizes, confounding variables, or conflicting results across the primary studies (Fig. 1).

Figure 1.

Number of systematic reviews (n = 43) within each recommendation category (Table 1) sorted by review topic.

Although most reviews provided only some guidance for management practice, almost all (93%) identified existing knowledge gaps and made recommendations for the direction of future research effort. A large proportion (79%) also highlighted common flaws in existing research methods and recommended changes to experimental design to improve the value of future primary research.

Types of Review Questions

Reviews tended to address questions of conservation policy (65%) (Supporting Information) relevant to the efficacy of high-level conservation tools, such as habitat corridors (Davies & Pullin 2007; Doerr et al. 2010; Eycott et al. 2010) and marine protected areas (Stewart et al. 2009a), or emerging conservation issues, such as the effect of wind farms on birds (Stewart et al. 2007a). A minority of reviews (35%) addressed issues directly relevant to on-the-ground managers (Supporting Information), such as the effectiveness of options to control individual weed species (e.g., Kabat et al. 2006; Roberts & Pullin 2006; Roberts & Pullin 2007a; Stewart et al. 2007b).

Although the topic of reviews did not affect the type of findings arising from the reviews (Fig. 1), larger overall effect sizes were detected in reviews that addressed topics relevant to on-the-ground managers (mean effect size = 2.95) than reviews that addressed questions of conservation policy (mean effect size = 1.60; t = 2.62; df = 26; p = 0.03).

Breadth of Reviews

In many cases, the geographic and taxonomic scope of reviews was not clearly defined or no obvious restrictions on scope were mentioned. Thirty percent of reviews restricted the geographic scope to a small number of countries (≤10), but most reviews had a much broader geographic scope (median = 141 countries [SE 14], range = 1–196 countries) (Supporting Information). However, the realized coverage of reviews from data actually included in the meta-analysis tended to be much lower (median = 13% of countries [SE 6]) (Table 2 & Fig. 2). We also found a distinct geographic bias toward western countries, particularly those in western Europe and North America. Reviews that provided some management guidance showed a non significant tendency to include data from a greater proportion of the countries relevant to the scope of the review than those that were unable to provide guidance (t = –2.02; df = 37; p = 0.33) (Fig 2). Likewise, many of the reviews had a very wide taxonomic scope (range = 1–936, 244 species) but realized a much narrower taxonomic coverage (median = 16% of species [SE 8]) (Table 2). Reviews overrepresented primary studies featuring vertebrates, particularly birds, relative to global species richness and underrepresented more diverse taxonomic groups, such as invertebrates and plants. There was no clear difference between the proportion of species represented between reviews that could or could not provide guidance to managers (t = 0.49; df = 23; p = 0.62). However, the only reviews to fully address the stated scope were restricted to a single species or small functional groups of species (Fig. 3).

Table 2. The geographic and taxonomic coverage of systematic reviews of conservation evidence
   GeographicTaxa ofTaxonomic
ReferenceReview topicPlaces of interestcoveragea (%)interestcoverageb (%)
  1. a

    Percentage of relevant countries represented in the meta-analysis.

  2. b

    Percentage of relevant species represented in the meta-analysis.

  3. c

    Could not be calculated.

  4. d

    Not applicable.

Abella 2009effects of fire on vegetation in the Mohave and Sonoran DesertsU.S.A.1plantsCBCc
Bayard & Elphick 2010effects of fragmentation on birdsglobalCBCcbirds10
Benitez-Lopez et al. 2009effects of roads and other infrastructure on mammal and bird populationsglobal9mammals and birds1
Bowler et al. 2012effects of community forest management on global environmental benefits and local welfareglobal5CBCcCBCc
Brooks et al. 2006effects of development on ecological, economic, attitudinal, and behavioral outcomesglobal7NAdNAd
Bussell et al. 2010effects of draining and re-wetting peatland soils on carbon stores and greenhouse gas fluxesglobal6bogs, fens and mires47
Dalrymple et al. 2011effects of reintroductions on plant extinctionsglobal7plants<1
Davies & Pullin 2007effects of hedgerows on movement between fragments of woodland habitatNorth America, Europe19woodland speciesCBCc
Davies et al. 2008effects of current management recommendations on saproxylic invertebratesglobal4saproxylic invertebratesCBCc
Doerr et al. 2010effects of structural connectivity on the dispersal of native species in Australia's fragmented landscapesAustralia100Australian flora and fauna1
Eycott et al. 2012effects of landscape matric features on species movementglobal10fauna<1
Frampton & Dorne 2007effects on invertebrates of pesticide usage at crop edgesglobal<1terrestrial invertebrates<1
Gusset et al. 2010effects of wild dog (Lycaon pictus) reintroductions in South AfricaSouth Africa100wild dogs (L. pictus)100
Holt et al. 2008effects of predation on prey abundance in the United KingdomU.K.50vertebrates3
Isasi-Catalá 2010effects of translocating problematic jaguars (Panthera onca) on human-predator conflictsSouth and Central America18jaguar (P. onca)100
Kabat et al. 2006effects of control and eradication interventions on Japanese knotweed (Fallopia japonica)Australia, Europe, North America, New Zealand15Japanese knotweed (F. japonica)100
Kalies et al. 2010effects of thinning and burning in southwestern conifer forests on wildlife density and populations in the United StatesU.S.A.100birds, mammals, reptiles, amphibians in southwestern conifer forests, U.S.A.16
Kettenring & Adams 2011effects of control on invasive plant speciesglobal13Plants4
Mant et al. 2011effects of liming streams and rivers on fish and invertebratesEurope, North America9fish and invertebratesCBCc
Martinez-Abrain et al. 2010effects of recreational activities on nesting birds of preyglobal6birds of prey4
McLeod et al. 2008effects of controlling European red foxes (Vulpes vulpes) biological diversity and agricultural production in AustraliaAustralia100Australian birds, mammals, reptiles, amphibians, livestock, invertebrates, poultry18
Newton et al. 2009effects of grazing on lowland heathland in northwest EuropeEurope12lowland heathlandCBCc
Oro & Martínez-Abraín 2007effects of large gulls on sympatric threatened waterbirdsEurope, Africa11threatened waterbirds100
Peppin et al. 2010effects of postwildfire seeding in forests of the western United StatesU.S.A.100forest speciesCBCc
Radwan et al. 2010effects of reduced major histocompatibility complex diversity on vertebrate populationsglobal10vertebrates<1
Roberts & Pullin 2006effects of different management on Spartina speciesglobal3Spartina alternifolia, S. anglica, S. densiflora, S. patens, S. x townsendii40
Roberts & Pullin 2007aeffects of management interventions on ragwort speciesU.K., Ireland, North America, Australia, New Zealand, South Africa, Argentina83ragwort (Senecio jacobaea and S. aquaticus)100
Roberts & Pullin 2007beffects of land-based schemes (incl. agri-environment) on farmland bird densities in the United KingdomU.K.100farmland birds in the United Kingdom31
Showler et al. 2010effects of public access on ground-nesting and cliff-nesting birdsglobal6ground-nesting and cliff-nesting birdsCBCc
Smith et al. 2010effects of predator removal on bird populationsglobal6birds1
Smith et al. 2011effects of nest predator exclusion on bird populationsglobal2birds<1
Stacey et al. 2011effects of arid land springs restoration on hydrology, geomorphology, and invertebrate and plant species compositionEurope, North and South America, Australasia6invertebrate and plant species associated with land springsCBCc
Stewart & Pullin 2008effects of grazing stock type and intensity on the conservation of mesotrophic old- meadow pastureU.K.20mesotrophic “old meadow” pasture community100
Stewart et al. 2005effects of moorland burning in heath and bogs on biological diversity and effects of burning on blanket bogU.K., Ireland60bogs, heaths, and montane communities37
Stewart et al. 2007aeffects of wind farms on birdsglobal4Birds23
Stewart et al. 2007beffectiveness of Asulam for bracken (Pteridium aquilinum) control in the United KingdomU.K.75bracken (P. aquilinum)100
Stewart et al. 2007ceffects of salmonid stocking in lakes on native fish populations and other fauna and floraglobal2freshwater fishes, reptiles, amphibians, invertebrates, and flora<1
Stewart et al. 2009aglobal ecological effects of temperate marine reservesglobal4marine fishes, algae, and invertebrates2
Stewart et al. 2009beffects of engineered in-stream structure on salmonid abundanceglobal4salmon species, trout species and Cottus gobio12
Timonen et al. 2011woodland key habitats as biological diversity hotspots in boreal forestsSweden, Finland, Norway, Denmark, Latvia, Lithuania, Estonia, Russia38boreal forest speciesCBCc
Tyler et al. 2005effects of management interventions on American mink (Mustela vison) populationsEurope, North America21American mink (M. vison)100
Tyler et al. 2006effects of management interventions on Rhododendron ponticumU. K., Ireland, Austria, Belgium, France, Netherlands, New Zealand40R. ponticum100
Waylen et al. 2010effects of local cultural context on community-based conservation interventionsglobal16NAdNAd
Figure 2.

Proportion of relevant countries with data included in the meta-analysis (geographic coverage) of the systematic review on the basis of type of implications for management practice (n = 41) (error bars indicate SE).

Figure 3.

Taxonomic scope of the systematic review according to the type of implications for management practice (n =40).

Evidence Captured by Reviews

The reviews we evaluated commonly reported large numbers of relevant studies (μ = 315 [SE 85]) (Fig. 4); however, the strict inclusion criteria for reviews and the limited quality of much of the available primary literature led to a median of only 12% of relevant studies being included in the meta-analysis. The reasons for this high attrition of relevant studies included: a different measure of outcome (e.g., measuring the effect of an intervention on fecundity instead of mortality); no measure of variance reported for the data set; research methods that failed to meet quality standards (e.g., no data collected prior to the intervention and lack of experimental controls); and use of multiple management interventions that masked the effect of the intervention of interest.

Figure 4.

Number of relevant studies found and those included in systematic review grouped by the type of implications for conservation practice (n =40).

Despite 88% of the available research being excluded from systematic reviews, the meta-analyses still captured a large research effort. On average, the primary studies included in each meta-analysis represented 284 independent data sets (SE 109) and a combined total of 112 years (SE 24) of data collection. However, we found no statistically significant difference between reviews that could or could not provide guidance to managers for either the number of data sets (t = –1.43; df = 37; p = 0.82) or the number of years of data collection (t = 1.60; df = 39; p = 0.11) embodied in the meta-analysis.


We located 43 systematic reviews on the effectiveness of conservation interventions published from January 2001 through March 2012. Just over half had direct implications for conservation practice, but almost as many could not directly inform management (Fig. 1). Although direct comparisons among disciplines can be problematic and the small sample size of available reviews undoubtedly limits inference, this result is consistent with reviews within the fields of medicine, social welfare, education, and criminology, where 54% of systematic reviews could not draw conclusions about the effectiveness of interventions (Hansen & Rieper 2009). Considerable time and money are required to produce a review (up to $US300,000) (CEBC 2010), so the frequency with which reviews fail to provide implications for management may discourage authors from conducting systematic reviews.

It has been suggested that systematic reviews of conservation interventions are not well suited to addressing broad policy issues (Fazey et al. 2004). Ecological phenomena vary enormously among species and environments (Hawkins et al. 2003; Magurran et al. 2010), and reviews with a policy focus tended to encompass a broad geographic and taxonomic scope, reflecting this variation. Reviews with a broader geographic scope highlighted geographic variation in the effectiveness of interventions (Fig. 2), and reviews addressing more than one functional group could provide guidance only for a narrow set of management contexts within their overall breadth (Fig. 3). Reviews focusing on big-picture conservation interventions tend to provide little guidance for individual conservation managers because there is often substantial heterogeneity in results across different spatiotemporal scales and for different focal species (e.g., Stewart et al. 2009a). We echo cautions raised in the medical literature that broad review questions increase the heterogeneity in the circumstances under which interventions are made, making findings harder to interpret (Higgins & Green 2011).

Taking an evidence-based approach to developing conservation policy is clearly desirable (Pullin et al. 2009). However, we found that the available data rarely justify conducting reviews with a broad geographic and taxonomic scope because only a fraction of this scope (13–16%) is actually realized. Despite attempting to address diverse taxonomic and geographic regions, the individual reviews were generally highly taxonomically and geographically biased due to limitations in the available data. Research from North America and Europe was overrepresented as was research on particular taxonomic groups, such as birds and mammals. Despite the rigor of the systematic review method, it could thus be argued that many reviews provide little information to managers that cannot be provided through traditional narrative reviews or research synopses (, which summarize the primary literature on a topic in a more informal manner. Moreover, the poor quality of much of the available literature and the restrictive criteria imposed by quantitative meta-analysis methods mean that much potentially relevant science might be excluded from a systematic review. Given the expense, time, and expertise required to conduct a systematic review, careful consideration is needed to determine which conservation questions warrant a systematic review versus a narrative review or other form of information synthesis.

The breadth of many of the reviews may explain why there were often hundreds of relevant studies reported during the literature search phase of the reviews (Fig. 4) and a significant research effort (μ = 284 data sets; μ = 112 years) captured by the analyses. Research effort was unrelated to the strength of the meta-analysis, such that a review with little data was as likely to yield clear findings as one representing hundreds of data sets or years of data collection. Therefore, contrary to other disciplines that attribute high failure rates of reviews to a lack of available primary data (Hansen & Rieper 2009), it is possible that this is not a primary concern in conservation science.

One aspect of systematic reviews that we could not address is whether reviews change management practices. The types of evidence valued by managers is an understudied area (but see Pullin & Knight 2005; Cook et al. 2012), and managers’ perspectives on the value of systematic reviews and other types of information syntheses could provide valuable insight into the types of questions best suited to each approach and which research questions should be given priority (Braunisch et al. 2012). We observed a tendency for systematic reviews to address big-picture conservation issues, sometimes at the expense of relevance for on-the-ground managers. Bilateral communication between scientists and managers throughout the review process is likely to improve the relevance of systematic reviews (Cook et al. 2013).

Benefits of Systematic Reviews

We found that systematic reviews are effective at exposing important knowledge gaps. For example, studies of the effectiveness of invasive species control rarely quantify the benefits for biological diversity (Kettenring & Adams 2011). Practitioners are an important resource for identifying key knowledge gaps in conservation science (Braunisch et al. 2012). However, the well-documented mismatch between conservation science and information priorities for on-the-ground managers (e.g., Whitten et al. 2001; Fazey et al. 2005) highlights the important role systematic reviews can play in articulating key research priorities to scientists.

The research quality standards imposed by systematic reviews frequently highlight methodological problems with published research, and most reviews make recommendations for how methods should be strengthened to provide greater value for management decisions. For example, research on the effectiveness of invasive species control methods often use greenhouse studies that do not reflect management conditions (Roberts & Pullin 2007a). Likewise, when research findings are obstructed by confounding variables, reviews can highlight necessary improvements to experimental protocols (e.g., Frampton & Dorne 2007). Therefore, even when reviews fail to provide guidance for on-the-ground managers, they can yield many useful recommendations about how to improve the quality of management-relevant conservation science.

Opportunities to Improve Systematic Reviews of Conservation Evidence

We suggest there are several opportunities to improve the application of the systematic methodology to yield greater benefits for conservation management: encourage more reviews, select the appropriate topic and breadth, make better use of available data, and evaluate cost-effectiveness.

The small number of systematic reviews conducted leaves many critical conservation management issues without a comprehensive synthesis of existing evidence. The imperative for academics to publish in high-impact journals (Gibbons et al. 2008) and the considerable time and money required to conduct systematic reviews (CEBC 2010) mean they are currently relatively unattractive. Additional avenues for funding and publishing reviews are required and more open-access scientific platforms for distilling relevant information, such as Environmental Evidence, would benefit conservation managers. To streamline the review process, software now exists to capture, store, and synthesize primary data for reviews (e.g., Eco Evidence—Webb et al. 2011). Making evidence summaries available to other users through a communal database allows existing reviews to be easily updated or modified so that information can be used in new reviews. This practice reduces duplication of effort and could provide a valuable resource for decision makers searching for reliable evidence pertaining to a specific topic.

Incentives for academics to publish may explain why we observed a mismatch between the narrow review questions likely to be more valuable to decision makers and the prevalence of high-level conservation questions likely to be interesting to a broad scientific audience. Knowledge maps highlight where gaps in the available literature warrant the review topic being refined to provide the greatest value for decision makers (CEBC 2010). Encouraging authors to conduct reviews with a narrower scope focused on informing on-the-ground management decisions would help streamline the review process and simplify the interpretation of findings (Higgins & Green 2011). Narrow review topics may not elucidate the sources of heterogeneity in the effectiveness of an intervention that are helpful for conservation policy; overview reviews developed for evidence-based medicine (Whitlock et al. 2008) can achieve this by drawing together the results of several narrowly focused reviews. However, the cost-effectiveness of overview reviews has not been examined.

Not all of the 88% of relevant studies excluded from reviews provide meaningful evidence. However, the stringent requirements for data inclusion in a meta-analysis undoubtedly result in potentially useful information being excluded (Pullin & Stewart 2006). To reflect a broader evidence base without compromising on the quality of science, formalized scoring approaches, such as causal criteria analyses (Norris et al. 2012) and Bayesian approaches that capture expert knowledge (Newton et al. 2007) can be used. These less-stringent methods capture up to 30% more data than meta-analysis and can test a broader range of hypotheses (Greet et al. 2011). Other disciplines have also broadened their definitions of credible evidence to suit the questions being addressed (Hansen & Rieper 2009) and are using evidence typologies to define the appropriate types of evidence according to the nature of the research question (Petticrew & Roberts 2003). Similarly, methods are being developed to identify rigorous qualitative research (Higgins & Green 2011) that can complement, enhance, extend, and supplement the quantitative analysis in systematic reviews (Petticrew & Roberts 2003; Higgins & Green 2011).

The cost of an action can materially alter conclusions about the best interventions for a given context (Baxter et al. 2006). We believe that to be of greatest valuable to managers, systematic reviews should include an explicit cost-effectiveness analysis (Segan et al. 2011). Some review authors have recognized the importance of including costs (e.g., Doerr et al. 2010; Kettenring & Adams 2011) but have been prevented from analyzing the cost-effectiveness of different interventions by poor reporting of management costs within primary studies. A further impediment is the lack of guidelines for capturing cost-effectiveness in systematic reviews of conservation interventions (Pullin & Stewart 2006).

Value of Systematic Reviews

The need for effective management highlights the value of tools to synthesize and distribute credible evidence to decision makers. Systematic reviews can be an important part of the conservation decision making tool kit; however, the current application of this method generally fails to harness their full potential, and at present systematic review is most useful for summarizing the current state of knowledge, identifying knowledge gaps, and highlighting where the quality of existing research needs to be improved. We believe that the benefits of systematic reviews could be enhanced by increasing the number of reviews focused on questions of direct relevance to on-the-ground managers, a more focused geographic and taxonomic scope that better reflects the available data, greater use of existing knowledge that includes a broader range of evidence types, and inclusion of an appraisal of the cost-effectiveness of interventions.


This research was conducted with the support of funding from the Australian Research Council Centre of Excellence in Environmental Decisions and the Environmental Decisions Hub of the National Environmental Research Program. We thank A. Pullin and W. Sutherland for valuable discussions and feedback during the development of this manuscript.