Systematic Review Checklist

A Standardized Technique for Assessing and Reporting Reviews of Life Cycle Assessment Data

Authors

  • Jennifer M. Zumsteg,

  • Joyce S. Cooper,

  • Michael S. Noon


Jennifer M. Zumsteg, M.D., Harborview Medical Center, Department of Rehabilitation Medicine, Box 359740, Seattle, WA 98104, USA. Email:zumsteg@uw.edu, https://catalyst.uw.edu/workspace/rehab/18144/

Summary

Systematic review, including meta-analysis, is increasingly utilized in life cycle assessment (LCA). There are currently no widely recognized guidelines for designing, conducting, or reporting systematic reviews in LCA. Other disciplines such as medicine, ecology, and software engineering have both recognized the utility of systematic reviews and created standardized protocols for conducting and reporting systematic reviews. Based largely on the 2009 Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, which updated the preferred format for reporting of such reviews in biomedical research, we provide an introduction to the topic and a checklist to guide the reporting of future LCA reviews in a standardized format. The standardized technique for assessing and reporting reviews of LCA (STARR-LCA) is a starting point for improving the utility of systematic reviews in LCA.

Introduction

Approaches that set out to understand systems and processes in a comprehensive way and can account for a range of impacts, such as life cycle assessment (LCA), are of value in better informing the factual grounds of decisions in areas like business and policy. Individuals engaged in related decision making require reliable information on which to base decisions. Some of this information may be about natural resources or manufacturing systems, such as understanding the contribution to climate change from the production of a kilogram of corn grain. These inquiries are not necessarily easy to answer and often the most correct answer will be, “it depends.”

Opportunities in Life Cycle Assessment

Advancements in LCA methodology, such as approaches in consequential LCA (Earles and Halog 2011), that better reconcile overlapping, incomplete, or conflicting data are important in ongoing process improvement and related policy recommendations (Hunkeler and Rebitzer 2005). Real-life applications for deriving a so-called best answer from LCA data have arrived. For example, the U.S. Energy Independence and Security Act of 2007 calls for the ability to directly compare life cycle greenhouse gas (GHG) emissions for different fuel sources (Energy Independence and Security Act 2007). Efforts to reach conclusions about whether fuel source X or Y has fewer GHG emissions can be approached in many ways, and such summary information or opinions may carry the title of “review.”

Characteristics of Reviews

The scope and structure of a review in any field will vary based on its purpose. The goal of a review may be to provide a survey of previous publications, critique current work, or make summary projections about future discoveries. A review may discuss data, opinions, or practices from a focused geographical region, it may explore one specific technology or system, or it may aim to provide comprehensive coverage about a topic.

The structure of the review may be different if the focus is on summary information, providing an expert opinion, critiquing available literature, or reevaluating existing data. When a comprehensive literature search is conducted in a transparent and unbiased way, it can be considered a “systematic review” (Neely et al. 2009). Such systematic reviews may also be meta-analyses when one of the aims is combining or reanalyzing data from previous studies to answer new questions, improve accuracy, or identify sources of variation.

The purpose of this article is to describe key factors for conducting and reporting systematic reviews, including meta-analyses, in LCA, based on established recommendations from other fields of study. We propose a checklist for a standardized technique for assessing and reporting reviews of LCA data (STARR-LCA) to assist in structuring the process of conducting and reporting systematic reviews. While we anticipate that the STARR-LCA checklist may be practically useful in a range of review applications, our current focus is systematic reviews, and so the terminology in the checklist has been shortened to just “review.”

To ease the burden on the reader, we simplify the terminology in the remainder of the current report to allow the concept of “previous studies” to include not only full LCA reports, but also data obtained from other sources such as life cycle inventory (LCI) data from public or private databases. Similarly, in the current discussion we consider meta-analysis a subset of systematic reviews, as data for meta-analyses are often obtained through a prospectively defined literature search. We acknowledge that meta-analyses can occur outside the scope of a systematic review of the literature. Information about key terms for this article is provided here:

  • • Systematic review: A structured evaluation of the literature with the goal of answering a specific research or application question with a synthesis of the best available evidence. Generally published to share these results with a wide audience for consideration and implementation.
  • • Meta-analysis: A melding of data from multiple studies, usually involving additional mathematical analyses, with the goal of utilizing this synergy of information and data size to answer questions that cannot be answered by existing individual studies or to improve the certainty or impact of known findings by increasing the sample size. Meta-analyses are often performed as part of a systematic review.
  • • Life cycle assessment (LCA): Also known as life cycle analysis, LCA is a standardized framework (ISO 2006a, 2006b) that can improve our understanding of the impacts of a system or product through the stages of its manufacturing, utilization, and disposal. LCA studies can take many forms, but typically include a clear goal and definition of the scope; quantification of what natural resources are utilized during the life cycle of a product, and what is released to the environment in that same cycle; and an assessment of the impacts (usually focused on harm) of the releases to the environment. The process is expected to be iterative and has many steps of interpretation along the way. The assessment is often limited to environmental impacts, such as global warming potential; however, the LCA framework allows for consideration of financial costs and impacts related to social metrics such as job creation.
  • • Inclusion and exclusion criteria: Parameters set intentionally at the start of a review process to inform which articles, data, and opinions will be included and/or excluded in the summary analysis of the review. Example criteria include a range of years, geographical region, technology type, or functional unit definition. Criteria may be described to either include previous studies (such as including studies published between 1960 and 1980) or to exclude previous studies that are known to be out of the scope of the review (such as in conducting a review of data related to technology A and therefore planning to exclude data related to technology B that is discovered in the review process).

A schematic for our view of how systematic reviews and meta-analyses fit into the overall organization of knowledge in the scientific community is provided in figure 1.

Figure 1.

A schematic of systematic reviews and meta-analyses within the wider context of knowledge within the scientific community.

The Evolving State of Reviews in Life Cycle Assessment

Variety of Review Types

Reviews have been published in many areas related to LCA; table 1 provides selected examples that illustrate a range of topics and review purposes.

Table 1.  Examples of reviews in life cycle assessment (LCA)
Topic area References
Bioenergy Cherubini et al. 2009; Farrell et al. 2006; Larson 2006; Rowe et al. 2008; Whitaker et al. 2010; von Blottnitz and Curran 2007
Building materials Werner and Richter 2007
Buildings Sartori and Hestnes 2007
Construction industry Ortiz et al. 2009
Ecosystem services in LCA Zhang et al. 2010
Electricity generation Bhat and Prakash 2008
Farming Mondelaers et al. 2009
Livestock products De Vries and De Boer 2010
Management options for paper waste Villanueva and Wenzel 2007
Nuclear power Sovacool 2008
Packaging options von Falkenstein et al. 2010
Pavements Santero et al. 2011
Remediation technologies Lemming et al. 2010
Wind power Lenzen and Munksgaard 2002; Kubiszewski et al. 2010
Woody biomass-to-liquid production Sunde et al. 2011

Interest in systematic reviews of the literature, with or without meta-analysis, is increasing in many disciplines, including LCA. In the medical literature the frequent publication of systematic reviews is reflected in the manuscript title “Seventy-five trials and eleven systematic reviews a day: How will we keep up?”(Bastian et al. 2010), and there are recent systematic reviews on other topics such as education (Salleh et al. 2011), employee retention (Ghapanchi and Aurum 2011), ecology (Kettenring and Adams 2011), and engineering (Khaddaj-Mallat et al. 2011).

Systematic Reviews

Systematic reviews vary in scope, quality, and relevance and may seek to expand on existing data by pooling results in a way that provides a higher degree of certainty, less error, or answers an entirely new question using existing data. The strategy of meta-analysis evolved out of educational research in the 1970s; the term was coined by Gene Glass to describe the process of utilizing statistics to merge the results of multiple studies (Hedges 1986). In LCA, conducting meta-analyses can be challenging as the individual studies of interest often have understandably different parameters. One approach to resolving this conflict is where “harmonization” of data from relevant studies is performed to allow for further insights from previously heterogeneous data (Arvizu et al. 2011). Locating previous data appropriate for harmonization may require a systematic review of the literature (Moomaw et al. 2011), yet there is currently no consensus regarding how such reviews should be approached.

In addition to the systematic review approach utilized for data harmonization, other LCA reviews provide a starting point for considering appropriate protocols. In a review of existing LCAs, Villanueva and Wenzel (2007) followed a four-phase protocol that included (1) an inventory of existing LCA studies, (2) definition of criteria for the selection of LCA studies, (3) selection of LCA studies, and (4) identification of key issues. Whitaker and colleagues (2010) conducted a systematic review where the protocol was not only defined ahead of time, but was published as a reviewed manuscript (Rowe et al. 2008).

Standards provide guidance about best practices for conducting and reporting LCA studies while providing flexibility in how reviews of any type should be conducted (ISO 2006a, 2006b). A lack of consensus regarding best practices in systematic reviews, specifically, has many disadvantages, such as less efficient peer review, suboptimal electronic search capabilities for locating systematic reviews once they are published, and inconsistencies in terminology. Fortunately, knowledge from other disciplines can be leveraged to improve systematic reviews in LCA.

Systematic Review Formats Utilized in Other Disciplines

Guidelines for performing and reporting the results of systematic reviews are utilized in areas of research including software engineering (Staples and Niazi 2007), ecology (Pullin and Stewart 2006), and medicine (e.g., Centre for Reviews and Dissemination 2009; Van Tulder et al. 2003). Health care has strongly advocated for utilizing the best available evidence for decision making (e.g., prescribing patient treatment on the basis of drug trials) and the framework of evidence-based medicine (Sackett et al. 1996) is designed to complement the expertise of individual practitioners.

The systematic review framework from evidence-based medicine has been successfully utilized in other disciplines, such as ecology (Gates 2002) and software engineering (Baldassarre et al. 2008). Similarly, we propose a checklist of essential topics to assist in designing, conducting, and reporting systematic reviews in LCA. The checklist content is intended to facilitate planning successful reviews, improve the localization of review publications in literature searches, ease the ability to update content in future reviews, and allow more transparency of methods to increase the ease of peer review and appropriate generalization of findings. The topics and commentary in our STARR-LCA checklist are guided largely by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement (Moher et al. 2009) and elaboration document (Liberati et al. 2009), which are widely used general guidelines for systematic reviews in medicine.

A Standardized Technique for Assessing and Reporting Reviews of Life Cycle Assessment Data (STARR-LCA)

A summary of STARR-LCA checklist items is provided here:

  • 1Review title, keywords, and abstract
  • 2Rationale for the review
  • 3Review question and objectives
  • 4Description of review protocol
  • 5Findings and features of the individual studies in the review
  • 6Assessment of bias
  • 7Synthesis methods (qualitative and quantitative)
  • 8Limitations of the review
  • 9Summary of findings and conclusions

An explanation of each category and discussion of related issues is expanded below. A sample worksheet that can be used to support study planning, manuscript preparation, and peer review is included in the supporting information available on the Journal's Web site.

Review Title, Keywords, and Abstract

An informative title and appropriate keywords can greatly assist in the application of scientific knowledge by improving the ability of others to identify appropriate reports (Dickersin et al. 1990; Montori et al. 2005). The title should identify the manuscript as a systematic review, meta-analysis, or both. We recommend that journal keyword databases include the terms “meta-analysis” and “systematic review” in order to support appropriate classification and maximize search functions.

In an age of information overload (Anex and Focht 2008; Fraser and Dunstan 2010), a structured summary of an article, such as a standardized abstract format, is essential in outlining for the reader the central question of the report, the general methodology, a summary of the results, and an estimate of the strength of that evidence, as well as highlights of potential relevance to the reader (Lebrun 2007). We suggest that abstracts for systematic reviews related to LCA include at least the following nine components: (1) background; (2) objectives; (3) data sources; (4) study eligibility criteria; (5) scope, system boundaries and functional unit; (6) study appraisal and synthesis methods (including any original data adjustments, such as harmonization); (7) results; (8) limitations; and (9) conclusions with implications of key findings.

Rationale for the Review

In the context of what is already known, describe why the systematic review, with or without meta-analysis, was a worthwhile effort. In the context of current knowledge, explain how the review may add to the knowledge base of readers from a variety of backgrounds or disciplines.

Review Question and Objectives

A valuable systematic review will be structured around a focused, answerable question that is worth answering and can be clearly summarized (Counsell 1997). There are established strategies for defining such answerable search questions. In health care, the acronym PICO describes one common model for question structure where the key components are the population of study, the intervention, a comparison group, and the outcome of interest (Booth and Fry-Smith 2003; Liberati et al. 2009). A systematic review question in the health sciences using the PICO structure could be phrased as: “For men and women over the age of 50, does the management of a heart attack with treatment X or treatment Y result in a shorter average hospital stay?” If the question had been defined simply as, “what is the best treatment for a heart attack?” we would be troubled with still defining what “best” really means and which treatment options we were considering.

Other question design methods that outline the key components of interest or that perform a limited literature review to better inform a planned systematic review can also be valuable, especially if the amount of literature available on a topic or the appropriate scope for a review question is not initially clear (Booth and Fry-Smith 2003; Torgerson 2003). A structured question format, such as PICO, has a range of advantages, including helping to facilitate optimal applied research questions, streamlining peer review of research, and even assisting students in defining appropriate questions to support evidence-based practice (Holloway et al. 2004).

Question Format: Product, Impact, Flow, and Type

Based on PICO, we introduce a recommended question format, PIFT, related to LCA that includes definition of the product or process category being assessed, the impacts of interest, the flows or economic sectors contributing to the impact, and the types of LCA of interest. We recommend this structured question format for systematic reviews; however, it can also guide other review types or studies.

LCA studies are typically categorized as one of three types: process based, input-output based, or a combination of these methods (hybrid). Process-based LCA, in which data are obtained based on parameters of known, physically utilized components, can be further divided into attributional and consequential types (Finnveden et al. 2009). In contrast, input-output models are based on economic inputs to a system, such as the EIOLCA model created by the Green Design Institute at Carnegie Mellon University (2008).

The sample review question, “Utilizing process-based LCA methods, what is the contribution to climate change from the production of a kilogram of corn grain?” would fit the PIFT approach. This question format can be further defined by adding relevant parameters such as geographic region, technology type, or years of interest. Additional examples of structured questions from LCA studies and details regarding each question element using the PIFT mnemonic are provided in table 2.

Table 2.  Examples of life cycle assessment (LCA) questions structured using product, impact, flow, and type (PIFT)
P: Product or process I: Impact(s) of interest F: Flows or economic sectors included T: Type(s) of life cycle assessment Sample question
U.S. health care sector in 2007Climate changeNorth American Industrial Classification System (NAICS) sectorsEconomic input-outputWhat does EIOLCA using NAICS sectors tell us about the climate change impact of the U.S. health care sector in 2007? (Chung and Meltzer 2009)
The four major classes of U.S. mail in 2006Total energy consumption, climate change, solid waste disposalPaper, paper board, plastic, fuelsProcess-based attributionalWhat does attributional LCA tell us about the total energy consumption, climate change, and solid waste disposal related to service for the four major classes of U.S. mail? (SLS Consulting 2008)

Description of Systematic Review Protocol

Generally a team of individuals participates in a systematic review, and a prospectively designed protocol is helpful for consistency, accuracy, and overall organization. Some disciplines have published detailed guidelines governing the methods of review studies (e.g., Higgins and Green 2011). In health care research, some investigators advocate for the additional rigor of prospectively registered review protocols available for peer review and a database of review protocols (Booth et al. 2011). The review's protocol and methods provide a cohesive understanding about how the review study will be planned, conducted, and reported. Details of the protocol may also describe the tasks for different investigators, redundancy in tasks to reduce bias and increase accuracy, and planned methods for resolving disagreements about interpretation of the study data (Neely et al. 2009).

A clear outline of the steps undertaken in a review can provide the transparency needed for replication, as well as appropriate updates when additional data are generated in future studies. We recommend that, at a minimum, a systematic review protocol include five key components: (1) a general description of how studies or data will be discovered for consideration of further review (e.g., literature or database search, manual review of the bibliographies of key publications, previous studies performed by the author's laboratory, solicitation of unpublished data); (2) enough information about any electronic search strategies that the search can be replicated; (3) clear parameters describing how the decision will be made to include or exclude individual studies and data for further analysis in the systematic review and/or meta-analysis; (4) a clear plan for recording and summarizing data from the individual studies, including what parameters or data points are of interest; and (5) what, if any, meta-analyses are planned.

Many discussions of systematic reviews acknowledge the iterative nature of such undertakings (e.g., Liberati et al. 2009; Pawson et al. 2005; Pino et al. 2008; Tranfield et al. 2003), and we advise that protocols for systematic reviews in LCA and any associated meta-analysis methods be trialed for usability by the study group, reviewed, discussed, and revised as indicated during the course of the review. Any revisions should be outlined and commented upon in subsequent publications and the protocol described should be transparent and complete enough to allow for replication.

Findings and Features of the Individual Studies in the Review

In order to allow for critical interpretation and an understanding of the basis of the review study, key information about the individual studies or sources of data included in the review should be provided. The quality of any review or meta-analysis is dependent on the individual studies upon which it is built. Assisting the reader in understanding the important data, flaws, and strengths of the individual studies is key to a robust and useful review and provides context for the reader that facilitates appropriate application of the review study results. Assembling key elements of the individual studies can be complex, and presenting the characteristics or results in a visual format (e.g., table format or plots) can be helpful (Liberati et al. 2009). Summarizing reviewer insights regarding strengths and weakness of individual studies, including elements such as confidence intervals, distributions, and individual study limitations, also assists the reader in evaluating and understanding the steps from individual studies to systematic review conclusions.

Assessment of Bias

Merriam Webster's Collegiate Dictionary (2012) defines the concept of bias in several ways, including “bent, tendency,”“an inclination of temperament or outlook, especially a personal and sometimes unreasoned judgment,” and “systematic error introduced into sampling or testing by selecting or encouraging one outcome or answer over others.” As human beings we naturally have affiliations and views that unconsciously shape our interpretation of data and the conclusions we draw. Understanding the presence and extent of bias in research is a difficult but important task. When performing a systematic review there are at least two levels of bias that need to be assessed: bias within each original study included in the review, and bias occurring across the summarized studies.

Bias Within Studies

Bias at the level of individual studies often relates to selection of study methods and data interpretation. In LCA, sources of bias include data sources (Mattila et al. 2010), categorization, normalization, or weighting (Reap et al. 2008), as well as variability in the accuracy of models used in the assessment (Finnveden et al. 2009). This form of bias is important to recognize, as it can impact the interpretation of a study, though the presence of bias does not necessarily invalidate a study.

Bias Across Studies

Publication bias is often the main type of bias across studies; it can be a concern when published studies do not accurately reflect the actual range of studies that have been performed (Ioannidis 2008). For example, studies with results that are deemed statistically significant may be more likely to be submitted and accepted for publication over findings that are not statistically significant, and there may be political obstacles to submitting articles that conflict with current practices (Johnson and Dickersin 2007; Taubes 1998). However, assessing variation in LCA data sources, such as LCA databases, can carry additional challenges given the frequent lack of uncertainty estimates. Published LCAs that are included in a systematic review may sometimes be different at face value, but not necessarily significantly different statistically.

Additional sources of bias across studies, such as failing to locate appropriate reports to include and the effects of financial conflicts of interest, should be evaluated as part of the review process; this category of bias can often be minimized by utilizing a well-designed review protocol (Miller 2000). The funding source for the review should be disclosed in publication to allow the reader to evaluate potential bias from conflicts of interest. Our understanding of sources and effects of bias continues to evolve in LCA. Additional work to better conceptualize bias specific to LCA would be useful.

Synthesis Methods (Qualitative and Quantitative)

Reviews may utilize a wide range of methods to synthesize the best available evidence in answering a focused question. Approaches range from qualitative work, such as grouping and summarizing of expert opinions, to quantitative synthesis, in which the published data are adjusted to a common value or other statistical methods are utilized as part of a meta-analysis. There is also value in understanding what adjustments are utilized in a meta-analysis to “even the playing field” and compare a set of otherwise different studies using a new common set of parameters. The authors may comment on the generalization of the parameters utilized in harmonization or a meta-analysis, as well as how the adjusted parameters themselves influence or provide insight into study variability. Whatever synthesis methods are undertaken in a review, they should be clearly described and justified based on the study protocol and the available data included in the review.

Limitations of the Review

In order to assist the reader in appropriate interpretation of the review results it is important to highlight and discuss limitations in any review publication. Reviews in LCA are likely to have parameters defined by factors including time frame, geography, and technology type, and so it is helpful for the authors to discuss where their findings likely can and cannot be applied based on the scope of the review. Limitations are also generally an excellent opportunity to highlight important questions for further research.

Summary of Findings and Conclusions

The crux of a review is the summary of evidence that was found to provide the best possible answer to the PIFT-formatted question posed at the start of the systematic review. The ultimate goal is to provide a general interpretation of the results in the context of other evidence. Specific points relative to practical problem solving or setting policy may be especially useful to highlight.

Summary of Recommendations for the Structure of Reporting Systematic Reviews in Life Cycle Assessment

Guidelines and Checklists Are Needed

The amount of data generated from life cycle assessment methodology is steadily increasing; reconciling conflicts between data sets and leveraging existing data to answer new questions are important tasks for advancing decision-making capacity related to LCA. Systematic reviews of the literature, with or without meta-analysis, can help summarize the existing data in meaningful ways to better answer existing questions or answer altogether new questions.

While the International Organization for Standardization (ISO) provides a framework for conducting LCA studies in general, there is currently no direct guidance about best practices in conducting systematic reviews in LCA. While there are a variety of reviews that have been published on LCA, there is great variability in methodology and reporting, which makes use of the information difficult. Other disciplines have successfully applied the systematic review structure common in biomedical research; we introduce the STARR-LCA checklist as a starting point for standardizing systematic reviews in LCA.

STARR-LCA Checklist

The nine items on the STARR-LCA checklist provide many tangible recommendations for improving the state of systematic reviews in LCA. Article titles should clearly identify the manuscript as a review with or without meta-analysis, and keyword listings for both individual reports and database structure should include the terms meta-analysis and systematic review. A structured summary or abstract should be provided at the beginning of a review; discussions regarding an editorial standard for structured summaries in LCA reviews would be worthwhile. The reason for the systematic review should be clear in the context of what is already known.

An answerable question should be the foundation for a review—and while question articulation can be an iterative process, a focused question should be drafted at the start of the systematic review process. We recommend using our PIFT question format. Additional question parameters such as the technology type, geographic region, or time may be relevant depending on the research question. A systematic review protocol should be prospectively written to guide researchers and, ideally, made available for peer review with key areas as noted above.

We also recommend an assessment of bias both for each study included in the review and across the entire review. Potential sources of bias include data, modeling parameters, valuation decisions, publication bias, and incomplete review of the relevant literature, as well as direct conflicts of interest. Bias based on methodology does not necessarily invalidate a study, but it is important to report and understand in order to optimize appropriate interpretation of the study. Detection and minimization of bias in LCA is a new concept worthy of ongoing discussion.

We acknowledge the wide range of both qualitative and quantitative synthesis methods that may be used. We recommend that, in the interest of transparency, methods be clearly described and any new data presented as part of the publication. We also recommend that authors specifically highlight the limitations of the systematic review as a way to assist the reader in result interpretation and highlight important questions for future research. Finally, the conclusions of the review should be clearly stated.

Our STARR-LCA checklist provides a framework to consider the elements of a well-designed review. The intent of the STARR-LCA checklist is to facilitate advancements in designing, conducting, and reporting systematic reviews, including meta-analyses, in LCA. We look forward to further discussion of the STARR-LCA checklist items through input from the LCA community, including advancements in LCA methodology.

Acknowledgements

Support for this work was provided by the Rehabilitation Medicine Scientist Training Program through the National Center for Medical Rehabilitation Research, grant 7K12HD001097-13.

Supporting Information S1: This supporting information provides a sample STARR-LCA worksheet for use in study planning, manuscript preparation, and peer review.

About the Authors

Jennifer M. Zumsteg is an acting assistant professor at the University of Washington, Department of Rehabilitation Medicine, Seattle, WA, USA. Joyce S. Cooper is an associate professor at the University of Washington, Department of Mechanical Engineering, Seattle, WA, USA. Michael S. Noon is a graduate fellow at the University of Washington, Department of Civil and Environmental Engineering, Seattle, WA, USA.

Ancillary