Using logic models to capture complexity in systematic reviews

Authors


Laurie M. Anderson, Washington State Institute for Public Policy, 110 Fifth Avenue SE, Suite 214, Olympia, WA 98504-0999. Tel: (360) 586-2792, Fax: (360) 586-2793.

E-mail: LAnderson@wsipp.wa.gov

Abstract

Logic models have long been used to understand complex programs to improve social and health outcomes. They illustrate how a program is designed to achieve its intended outcomes. They also can be used to describe connections between determinants of outcomes, for example, low high-school graduation rates or spiraling obesity rates, thus aiding the development of interventions that target causal factors. However, these models have not often been used in systematic reviews. This paper argues that logic models can be valuable in the systematic review process. First, they can aid in the conceptualization of the review focus and illustrate hypothesized causal links, identify effect mediators or moderators, specify intermediate outcomes and potential harms, and justify a priori subgroup analyses when differential effects are anticipated. Second, logic models can be used to direct the review process more specifically. They can help justify narrowing the scope of a review, identify the most relevant inclusion criteria, guide the literature search, and clarify interpretation of results when drawing policy-relevant conclusions about review findings. We present examples that explain how logic models have been used and how they can be applied at different stages in a systematic review. Copyright © 2011 John Wiley & Sons, Ltd.

Introduction

At a time when public resources are strained by costly social and health programs, policy makers and program administrators are looking for programs supported by scientific evidence of effectiveness when choosing among alternatives. Systematic reviews are a powerful decision-making tool as they summarize a body of scientific research and identify implications for policy and practice-level decisions (Petticrew & Roberts, 2006; Cooper & Hedges, 2009). More specifically, they can reduce bias associated with single studies and non-systematic reviews by operationalizing procedures to search the literature to ensure that relevant studies are captured and by specifying objective criteria to measure and gage program effects. However, because of the complex nature of policy questions in diverse fields (e.g., education, health, social welfare, and criminal justice), it is challenging to communicate the dynamics of interventions that operate at individual, group, and social system levels, and to highlight the factors that might influence program effectiveness. Systems thinking, according to a recent World Health Organization report on the topic, provides “a deliberate and comprehensive suite of tools and approaches to map, measure and understand these dynamics…” (WHO, 2009). A logic model is one such tool. Although logic models have been used in the field of program planning and evaluation research to illustrate program inputs, processes, and desired intermediate and long-term outcomes, they have not often been used to guide systematic review methods. Here, we describe the utility of logic models in conducting systematic reviews. The paper begins by reviewing the more familiar application of logic models in program planning and evaluation research. We then describe the phases of conducting a systematic review and explain how logic models can be used at various points in this process. Here, models can help conceptualize a complex review question and specify the analytic links to test the plausibility that a program works as intended. We conclude by discussing the difficulties in developing these models and offer recommendations.

Logic models in program planning and evaluation

A logic model is a graphic description of a system and is designed to identify important elements and relationships within that system. Program planners and evaluators have used logic models to illustrate how a program works to solve identified problems (Bickman, 1987). In doing so, the model describes a theory of change used to put in place the intervention components necessary to accomplish that change (i.e., program inputs, processes, and outcomes) (Frechtling, 2007). The result is a picture of hypothesized causal relationships that might include information situating the program within an economic, social, and political context. Logic models try to make explicit underlying assumptions, whether formal theory or other presumptions, for achieving the desired results from an intervention (Millar et al., 2001). For program planners, mapping a proposed intervention helps identify the human and financial resources needed to operate a program, the program activities targeted by these resources, and the intended products of program activities, for example, the types, levels, and reach of services delivered by the program (Kellog Foundation, 2004). For program evaluators, a logic model forms the basis for a process evaluation to assess the implementation of program components as planned (Helitzer et al., 2010). It also provides evaluators with a means to document whether desired changes have been achieved based on specified outcome measures. Information on the economic, social, or political contexts in which the program operates might explain influences that boost or attenuate effects. Finally, beyond utility for program planning and evaluation, logic models provide a tool to facilitate communication amongst program planners, evaluators, and a range of stakeholders by making assumptions upon which programs are predicated more transparent and causal mechanisms more explicit.

Utility of logic models in systematic reviews

In the field of research synthesis, conceptual models and causal diagrams are under-utilized. Yet, they have the potential to make systematic reviews more transparent and ultimately more cogent to decision-makers, by making explicit the underlying assumptions about causal relationships and program theory. Logic models promote systems thinking by illustrating the relationship of the parts to the whole and highlighting congruencies and inconsistencies. In doing so, they can draw attention to aspects of complex problems that might otherwise be overlooked. For example, urban renewal policies and programs in cities of the United States in the 1960s intended to clear large areas of slum housing to make way for modern developments, including high-rise public housing developments for families displaced in the process. Instead of improving the lot of these families by providing new apartments, the housing projects disproportionately concentrated African–American families into segregated islands of highly concentrated poverty and destroyed what were once more integrated communities (Fullilove, 2001). The importance of considering competing definitions of the determinants of a problem cannot be overemphasized, and logic models contribute to this process. This is particularly pertinent to systematic reviews of complex social and population health programs or policies that incorporate system-level changes (Craig et al., 2008; Shiell et al., 2008; Shepperd et al., 2009). In these reviews, a description and analysis of complex causal pathways are more advantageous than an oversimplified “black box” approach that provides little insight into factors that influence program success or failure. In particular, when the research synthesis goes beyond a simple bivariate relationship to elaborate the chain of events that lead to outcomes based on a set of precursor variables (Becker, 2009), a graphic interpretation helps in understanding the potential mechanisms of action. These causal pathways can involve non-linear phase changes (e.g., tipping points), feedback loops and interactions between components, and unanticipated positive or negative effects (Shiell et al., 2008). Thus, in research synthesis, logic models can facilitate the process of gathering and integrating studies of complex interventions and better inform interpretations of cumulative results (Joffe & Mindell, 2006). The practical contribution of logic models, both conceptual and analytical, is evident at most stages of the systematic review process, as illustrated in Box 1. These stages range from conceptualizing the review focus to drawing conclusions about the cumulative evidence. The following examples provide illustrations of the contributions of logic models in systematic reviews in the fields of public health, education, child welfare, and health-services research.

Box 1. Added value of using logic models in systematic reviews

Scoping the review

  • Refining review question
  • Deciding on lumping or splitting a review topic
  • Identifying intervention components

Defining and conducting the review

  • Identifying relevant study inclusion/exclusion criteria
  • Guiding the literature search strategy
  • Explaining the rationale behind surrogate outcomes used in the review
  • Justifying need for subgroup analyses (e.g., age, sex/gender, socioeconomic status)

Making the review relevant to policy and practice

  • Structuring reporting of results
  • Illustrating how harms and feasibility are connected with interventions
  • Interpreting results based on intervention theory and systems thinking

Scoping the review

The first stage in research synthesis involves defining the review question and determining what research evidence is germane to the problem or hypothesis of interest (Cooper, 2007). Understanding the “big picture” can stimulate debate on priority areas where evidence can play a role in shaping policy and practice and where it is less likely to do so. This overview encourages the reviewer to consider potential contingencies or competing phenomena within a social system that might affect the success or failure of a program or policy to achieve their objectives. Often, the social and health concerns that confront policymakers require action across what might be considered separate policy goals, some with competing aims, to reach a solution. Scoping the review, therefore, is essential in defining and refining the review question, in deciding on whether this review question needs to be addressed through several related systematic reviews, and in identifying intervention components at different levels (Box 1).

We use the example of rising obesity rates in a society. Obesity rates have risen sharply internationally, putting populations at higher risk for several chronic diseases and health conditions. Local, state, and national governments are seeking effective solutions to this public health problem. But complex problems must be structured before they can be solved (Dunn, 2008). This problem might require interventions that seek changes in multiple systems, such as transportation, agriculture, commerce, education, and health care. In such a situation, a logic model can help untangle the influence of different determinants of obesity and identify strategies for improving outcomes, sometimes from different points of departure. Delineating a large and complex topic into a series of review questions might be necessary. Logic model conceptualizations help reviewers make decisions on organizing intervention topics into a set of interrelated review questions. Having a coherent starting point provides a common understanding of the multiple causes of the problem and helps define the conceptual boundaries for a set of reviews. Seeing the broader picture also might point to those policies or contextual factors that might attenuate or boost program effects (Hawe et al., 2009). Figure 1 illustrates how multilevel relationships can be used in a logic model depicting food supply and population health. The overarching question is “Where do intervention opportunities exist to improve the nutritional quality of the food supply and change consumer demand patterns to ultimately improve population health outcomes, particularly with respect to obesity?” (Anderson et al., 2004). Here, systematic reviewers mapped factors that they considered modifiable determinants to identify targets for program and policy action. The depiction of a population's food supply environment as the constellation of macroeconomic and community-level factors that influence choice (physical, social, legal, and policy) helped to identity the leverage points (e.g., school settings) where programs might affect desired change (e.g., reduced soda consumption, increased physical activity). It also suggested the need to intervene collaboratively across sectors where concomitant factors (e.g., food and beverage advertising to children) might attenuate the effects of interventions (e.g., school nutrition education programs). As a result, reviewers identified intervention programs and policies at each level of the logic model. These were prioritized based on the likelihood of improving population health, acceptability to stakeholders, and feasibility with available resources. The model generated a series of systematic reviews of program and policy strategies to reduce obesity: (i) school nutrition education programs; (ii) school physical activity and education requirements; (iii) worksite programs to increase employee physical activity and improve dietary behaviors; (iv) direct marketing of fruits and vegetables in communities through farmers' markets; (v) subsidized food vouchers for low-income families for vegetables and fruits; (vi) menu labeling that includes caloric and other nutritional information; and (vii) taxation policies for soda and energy-dense, low-nutrient snacks. The logic model pointed to the grouping of actions (lumping or splitting decisions) based on setting and population such as schools, worksites, and communities; on behavioral change interventions via education and skill building; and on policy strategies such as low-income food subsidies, menu labeling, and soda and snack taxes. The results of the series of systematic reviews documented limited (worksite interventions) or no effect (school interventions) on physiologic indicators of obesity such as weight, body mass index, and percent body fat. Limited evaluative literature was found on community farmers' markets, menu labeling, and taxes on energy-dense products, which suggests the need for more primary studies. In addition to environmental settings and consumer behaviors, the findings suggested that tackling the problem of obesity might require changes at the food supply level, where agricultural policies and food production promote inexpensive, energy-dense processed foods and beverages whereas healthier foods (e.g., produce and whole grains) remain costly and less accessible (Drewnowski, 2010; Drewnowski & Eichelsdoerfer, 2010; Popkin, 2011).

Figure 1.

Food supply and population health.

Defining and conducting the review

Collecting the research evidence

The reasoning underpinning data collection and evidence synthesis in a systematic review should be transparent. A logic model provides the framework for identifying study inclusion/exclusion criteria, for guiding the search strategy (such as search terms and databases and strategies or filters included), for identifying relevant outcomes, and for examining differences among studies and along dimensions of interest (Box 1). Logic models clarify the reviewers' reasoning by enhancing understanding of the theory of change underpinning programs or policies. When sufficient evidence is available, it can help reviewers identify distinguishing characteristics of those more or less successful programs and evaluate whether the programs are more beneficial for some populations or subgroups. This approach is valuable to decision-makers who must interpret the cumulative evidence in the systematic review to draw conclusions about the relevance of results to specific problems and populations. Systematic reviews can yield different conclusions based on how the research question is operationalized; literature is searched; studies are included or excluded from the review; data are analyzed; and cumulative research is interpreted and presented (Cooper & Hedges, 2009). Logic models represent an accessible and transparent way of justifying such decisions, and of examining differences among related systematic reviews. Elaborating differences and similarities among reviews can advance knowledge when areas of disagreement among conclusions are evident.

In a recent systematic review of after-school programs for low-income elementary school youth, Zief et al. included the logic model shown in Figure 2 (Zief et al., 2006). Unstructured and unsupervised after-school time among school-age youth has been linked with increased risk-taking behaviors, victimization, and poor academic outcomes. In this review, the authors examined evidence from experimental evaluation studies that illustrated how after-school programming affected changes in student supervision, student academic support, and participation in enriching activities, factors identified in the model as mechanisms that mediate change in youth outcomes. Their logic model specifies intervention components hypothesized to change conditions necessary to improve youth outcomes. In doing so, program characteristics thought to contribute to program success are identified for coding and analysis (i.e., program goals and activities, implementation features, resources and staffing). Several types of supervised after-school programs include enriching activities such as sports, arts, mentoring, tutoring, or youth development. Although the logic model's focus is on low-income elementary youth, a priori justification for subgroup analysis is made explicit by illustrating the possible influence of student demographic characteristics, prior academic achievement, family background, and school and community characteristics on all stages of the model. In this way, the logic model elucidates hypotheses for how the intervention is expected to work, and also considers how disadvantage might interact with the hypothesized mechanisms of action (Tugwell et al., 2010).

Figure 2.

Logic model for understanding theory of change for low-income elementary youth in after-school program.

In collecting research evidence on after-school programs, the Zief et al. model specified the population, intervention, and outcomes of interest. They identified parameters for literature searches that, coupled with study methodology (e.g., experimental), structured the search strategy (Lefebvre et al., 2008). Rather than lumping all supervised after-school activity programs together, as previous reviews had done, student academic support was identified as a critical inclusion criterion because the objective was to examine effects on academic outcome. To support the hypothesized causal path to longer term academic outcomes, data also were collected on intermediate behavioral and social-emotional outcomes. Parental outcomes, although not required by the inclusion criteria, were viewed as potentially moderating program effects. Six previous reviews of after-school programs were found to have dissimilar results. The authors pointed to the differences in program definition and underlying theory of change illustrated in their logic model and methodological differences in their review (i.e., exhaustive literature searches, restriction to experimental study designs). This logic model communicates the authors' reasoning regarding the nature of evidence needed to support inferences relative to program effectiveness (Rog, 1995) and highlights critical factors to consider when generalizing review results to decision-makers' circumstances.

Explicating theory

Although systematic reviews more commonly attempt to determine if a program is effective, a systematic review can be used as a means of theory-building. Weiss describes program theory as “the mechanisms that mediate between the delivery (and receipt) of the program and the emergence of the outcomes of interest” (Weiss, 1998). Testing hypotheses regarding mediators of program effects require that a sufficient body of theory-relevant research has been conducted. If so, the set of studies can be used to examine common causal mechanisms and to clarify empirical relations between the mediator and the main effects (Wood & Eagly, 2009). Here, a logic model can guide inquiry into the theory that underpins an intervention. A recent test of the theory supporting Multidimensional Treatment Foster Care (MTFC) used the logic model illustrated in Figure 3 to document how “resiliency” acts as a mediating mechanism whereby youth in the child welfare system achieve positive social and behavioral outcomes (Leve et al., 2009). In the United States, the number of children who experience neglect and maltreatment has increased, and these children subsequently become involved with the child welfare system. These children and their parents exhibit high rates of behavioral and emotional problems. Although much is known about negative outcomes in this population, less is understood about resiliency processes that allow some children to prevail when exposed to adverse life experiences. MTFC integrates early work on family coercion theory (Patterson, 1982) with more recent theory on resilience processes in child development (Masten, 2001) and neurobiological functioning consequent to early adversity experienced by children (Fisher et al., 2006). In this logic model, Leve et al. present the MTFC intervention components and illustrate how they might act directly or as mediating mechanisms on resilience processes, specifically, supportive interpersonal relations and adaptive neurobiological functioning. Their goal is to explicate the theory underlying the intervention and test whether it is empirically supported. Using evidence from four trials that used MTFC intervention components, they collected data on parenting and attachment relations, peer group process, mentoring adults, social support, and stability of the home context. Evidence from multiple studies supported the links among parenting, attachment, peer group processes, and resilience outcomes. But not all hypothesized mechanisms (mentoring adults, parental social support, and stable home context) were sufficiently supported, as only single studies attested to those links. Hence, replication in future studies is needed. Thus, the logic model advanced program theory underpinning MTFC interventions by providing a lucid argument in support of the resiliency model based on evidence from intervention trials and indicated where connections between resilience mechanisms and resilience outcomes remain to be validated or refuted in new primary studies.

Figure 3.

Multidimensional Treatment Foster Care as a strength-based intervention promoting child and adolescent resilience in youth exposed to early adversity.

Logic model as an analytic map

Thus far, we have described the utility of logic models in conceptualizing systematic review questions and collecting research evidence. However, another factor to consider is the use of logic models as analytic tools. These are more concise than conceptual logic models that attempt to capture the “big picture”. Analytic logic models seek to demonstrate a chain of logic between inputs and outcomes and to capture any possible alternative explanations. Indeed, specifying all relevant causal relationships a priori, uninfluenced by the findings of the review, should help reduce bias in researcher judgment. In the case of systematic reviews conducted for the purpose of developing practice guidelines, the research must pass the scrutiny of practitioners, consumers, insurers, policy makers, and others. The logic underlying this process should result in a well-articulated rationale for a practice recommendation with clear evidence in support of the conclusion (Harris et al., 2001). For example, the United States Preventive Services Task Force uses analytic logic models to define the questions that must be answered before issuing practice guidelines. The Task Force develops a model of causal pathways indicating linkages in evidence that are essential to determine the effectiveness of the preventive service (Woolf et al., 1996). This model sets the focus of admissible evidence. For example, Figure 4 illustrates the chain of logic that evidence must support, linking screening and interventions for childhood obesity to improved health outcomes (Whitlock et al., 2005). Arrow 1 represents evidence that directly links screening to changes in health outcomes. If the effectiveness of the overarching link can be established, it might not be necessary to gather information on sub-component links. However, where direct evidence is lacking, examining the sequence of effects, including potential adverse effects, might demonstrate whether benefits will result from screening and intervention. Framing each link in the model as a research question facilitates the identification of key search terms. It also organizes the presentation and interpretation of review findings. When making practice recommendations, the Task Force considers: the quality of evidence for each link; the degree to which a complete chain of linkages is supported by adequate evidence; the similarity or fit across populations represented in the evidence base to support generalizable conclusions; and the degree to which the evidence connecting the preventive service and health outcomes is direct (Harris et al., 2001). In reaching their conclusion, they assess both benefits and harms to determine the magnitude of the net benefit. The Task Force, using the logic model in Figure 4, found evidence to recommend intense behavioral interventions (Arrow 4) for obese children aged 6 years and older for improvements in weight status (Whitlock et al., 2010). They also advise that although combining behavioral counseling with drugs (sibutramine or orlistat) among children 12 years and older improve short-term weight status, no long-term data exist on weight regain if the drug is discontinued and harms associated with these drugs in children is uncertain (Whitlock et al., 2010).

Figure 4.

Screening and intervention for childhood obesity.

Making the review relevant to policy and practice

Public policymakers are faced with finding remedies for complex problems like failing schools, unaffordable housing, high recidivism among criminal offenders, to name just a few. Bringing science to bear on deep-rooted and difficult problems is the aim of evidence-based policy, so programs that demonstrate effectiveness can be implemented and those that fail to produce desired outcomes can be redesigned or eliminated. But how reliably does a systematic synthesis of research provide a gage of effects when that program or policy is applied in real-life settings? At the heart of policy-relevant synthesis is an attempt to provide evidence that corresponds to the conditions under which programs or policies are applied in practice. The efficacy and expediency of interventions supported by a body of research are often found to be minimal or even unsuccessful in practical applications. Arriving at a valid statistical inference requires an unambiguous statement of the quantity to be estimated. A coherent review question, supported by methods suitable for answering that question, is fundamental to produce a synthesis of research that is applicable to drawing inferences for policy. Hedges suggests that the first step in a policy-relevant synthesis is to classify the relevant variables that have a systematic effect on study results: treatment type, study context, and study design (Hedges, 1997). Treatment characteristics include ways in which the nominal treatment varies systematically across studies, such as duration, intensity, mode of treatment administration, for example delivery by the researcher versus a classroom teacher. Study context involves characteristics of the target populations, including demographics and prior exposures or experiences, and the settings in which the research was conducted (e.g., individualized programs, community-based). Study design involves technical features to ensure interval validity and characteristics of the outcome measures (e.g., psychometric properties, proximal or distal outcomes.). Review inclusion criteria should classify the conceptual and technical requirements of included studies. But logic models, such as those shown in Figures 1-4, can further reduce uncertainties about the applicability of programs or policies to different populations and settings by presenting unambiguous links among the types of interventions, contexts, and outcomes. A systematic examination of these relationships might indicate how some studies included in the synthesis have, in fact, measured different things; here, a combined estimate of program effects would be misleading (Hedges, 1997). For example, more intense variants of a treatment, when studied in highly favorable contexts and among subjects thought to be most susceptible, might yield different estimates of effect than under less favorable conditions. A logic model provides a structure for reporting such characteristics in the amassed body of research and interpreting the results accordingly.

Challenges in using logic models in systematic reviews

In this paper, we argue that logic models are useful in systematic reviews of “complex” program or policy interventions to enhance evaluative reasoning and methodological transparency. This is somewhat ironic, as logic models attempt to reduce and simplify intervention components into essential inputs, processes, outcomes, and contexts. If reviewers tried to capture all hypothesized links between multi-faceted and interacting components with feedback loops and tipping points, the logic model could easily become uninterpretable. Complex programs can require large amounts of data for tracking each step of each causal chain (Weiss, 1997). Thus, a balance must be struck between an all-inclusive model and a useful heuristic tool that illustrates the important components and that therefore guides the research towards the most fruitful questions. A further consideration is the investment of review authors' time in developing the logic model. It is not simple to identify the right set of constructs and variables, achieve a proper level of abstraction, and to order the links among variables. Often existing frameworks can be adapted if necessary. This process is aided when review authors have both methodological and content-level expertise. However, different groups of researchers might construct different logic models for the same problem. This is not surprising, because researchers also might arrive at different conclusions in systematic reviews of the same topic if, for example, they specify dissimilar search strategies and inclusion criteria or if they choose to evaluate different outcomes as evidence of program success. In this case, a logic model makes more conspicuous any differences in the researchers' conceptual interpretation of the relationships between a program's activities and its intended outcomes.

Concluding thoughts and recommendations

We contend that systematic reviews of the effects of interventions could benefit from the use of logic models. This is consistent with recent calls for greater use of systems thinking in evaluating programs and policies, moving evaluation beyond the “black-box” approach to an approach that considers inputs, outputs, initial, intermediate, and eventual outcomes, along with program processes, contexts, and constraints (Pawson et al., 2005; Hawe et al., 2009). This paper does not propose to provide a “how to” manual on building logic models, as this information is available elsewhere (Kellog Foundation, 2004). Instead, we emphasize the use of logic models as tools to adopt elements of systems thinking in systematic reviews, which may be helpful to systematic reviewers and users alike. Reviewers might find the detailed “thinking through” and description of processes and contexts it involves valuable in understanding and describing an intervention, eventually moving beyond statements of whether something “works” towards a richer explanation of why and when it does or does not work. Users might find that the richer description facilitated by the inclusion of logic models meets their needs more closely than simple statements about effectiveness or ineffectiveness, or about the need for “more evidence”. We believe that logic models inform the conceptualization and development of systematic reviews. As an analytic framework, logic models make a unique contribution to produce and improve this form of research, thereby encouraging the translation of evidence into policy. We recommend the use of logic models in the identification of the complex links between determinants, outcomes, and intervention components and to guide the technical aspects of the systematic review process.

Acknowledgements

The authors acknowledge the excellent suggestions provided by the anonymous reviewers of the manuscript, which improved content and clarity, and the expert guidance of the Associate Editor. We are most grateful.

Eva Rehfuess gratefully acknowledges financial support from the Munich Centre of Health Sciences.

Ancillary