Guidance on the use of complex systems models for economic evaluations of public health interventions

Abstract To help health economic modelers respond to demands for greater use of complex systems models in public health. To propose identifiable features of such models and support researchers to plan public health modeling projects using these models. A working group of experts in complex systems modeling and economic evaluation was brought together to develop and jointly write guidance for the use of complex systems models for health economic analysis. The content of workshops was informed by a scoping review. A public health complex systems model for economic evaluation is defined as a quantitative, dynamic, non‐linear model that incorporates feedback and interactions among model elements, in order to capture emergent outcomes and estimate health, economic and potentially other consequences to inform public policies. The guidance covers: when complex systems modeling is needed; principles for designing a complex systems model; and how to choose an appropriate modeling technique. This paper provides a definition to identify and characterize complex systems models for economic evaluations and proposes guidance on key aspects of the process for health economics analysis. This document will support the development of complex systems models, with impact on public health systems policy and decision making.


| INTRODUCTION
There have been calls for greater use of complex systems methods to inform public health decision making (Diez Roux, 2011;Lich et al., 2013;Rutter et al., 2017).Computational models can be useful to evaluate public health interventions operating within complex systems, however, there are few examples of economic evaluations employing complex systems models (Shiell et al., 2008;Squires & Boyd, 2019).

| What is a computational model?
A computational model describes a simplified representation of reality in which a system is described using mathematical relationships (Caro et al., 2012).Such models combine evidence to quantify the future performance of parts of the system and address questions that are difficult to answer using primary empirical research approaches (Brennan et al., 2006).Within public health this includes: planning services, supporting infectious disease surveillance, policy impact analysis, economic evaluation, testing determinants of disease patterns, investigating disease trajectories, and testing intervention scenarios (Briggs et al., 2016).Models can characterize the population at the individual-level, in which people are distinct units, or at the aggregate level, using population averages.

| Features of a complex system
A system refers to problem situations characterized by having interconnected elements (Meadows, 2008), with multiple causes and consequences (Chapman, 2004).System complexity increases with the intricacy of the relationships between elements (Rickles et al., 2007).Therefore, it is not the number of interactions that is the defining characteristic of dynamic complexity, but rather the nature of interactions and their generation of emergent outcomes (Holland, 2014).Emergent outcomes are properties, perhaps observed at an aggregate-level of a complex system, that cannot be predicted by considering the elements within it in isolation, and are more than just the sum of its parts.Thus, in complex system problems, the effects of any single intervention cannot be determined in isolation.Each decision that is made depends on others, multiplying the counterfactuals that need to be considered (Ornstein et al., 2020).
Public health challenges can be conceptualized as complex systems problems because they involve: (i) mutual interdependencies between elements of the system, where effect directions, sizes, accumulation and timings are not well-understood or captured by research methods grounded in linear models of cause and effect; (ii) actors who have diverse sets of priorities, values and understanding of the problem; (iii) costs, benefits and harms spread across many parts of the system; and (iv) deep uncertainties due to rapidly shifting geo-political and economic contexts potentially changing population, demographic and/or behavioral dynamics and the interplay between social determinants and service system factors that influence the impacts of interventions (Meier et al., 2019).
Obesity provides a public health example of a systems problem in which interplay between numerous biological, environmental, social, political and economic factors influences obesity, which in turn can have implications for policy evaluation.A model describing a simple pathway between disease trajectory and health outcomes will be sufficient if the intervention produces consistent effects regardless of context.However, if the intervention interacts with other factors affecting the evolution, and consequences of the disease, this modeling approach will overlook emergent outcomes.

| Economic modeling in public health
Economic evaluation is a core component of all phases of intervention research in public health (Skivington et al., 2021).Economic evaluation using modeling techniques can estimate the value of public health investments, exploring incremental and population effects of changes in policies (Squires & Boyd, 2019).Modeling approaches used in economic evaluation often develop from the methods of health technology assessment, that is, decision-trees and Markov models, in which the implementation context is not explicitly modeled (Lawson et al., 2022).Discrete Event Simulation has been recommended to extend model complexity in economic evaluations (Karnon & Haji Ali Afzali, 2014), whereas in the public health literature complex systems models are often identified by other model traditions, such as system dynamics or agent-based models (ABM)s (Atkinson et al., 2015;Bicket et al., 2020;Carey et al., 2015;McGill et al., 2021;Xue et al., 2018).Guidance is needed to overcome barriers in translating cross-disciplinary knowledge given the breadth of the systems science literature (Trochim et al., 2006).

| Complex systems modeling in public health
Complex system methods for public health have been discussed extensively in the literature (Leischow & Milstein, 2006;Lich et al., 2013;Luke & Stamatakis, 2012;Rutter et al., 2017;Tracy et al., 2018;Trochim et al., 2006).However, several reviews highlight the competing definitions of complex systems (Leischow & Milstein, 2006;Lich et al., 2013;Trochim et al., 2006).As such, it is useful to specify the distinguishing features of a Complex System Model (CSM) that can be identified from the model structure and mathematical relationships, rather than focusing on modeling traditions for example, Systems Dynamics and agent-based models.A definition of a CSM for public health economic modelers will enable consistent labeling of models, limit conceptual stretching, a more efficient description of methods, and, where appropriate, encourage the adoption of CSM.

| Aims of this guidance
The aims of this manuscript are to (i) propose a definition of computational CSM that highlights the critical modeling features for economic modelers to use in public health models (Section A), (ii) support economic modelers to identify situations when a CSM is needed (Section B), (iii) identify appropriate modeling types for economic modelers to choose from (Section C), (iv) highlight useful approaches/methods in CSM (Section D).

| METHOD
The guidance was written in consultation with an international consortium of academic experts in CSM and related fields.In September 2020, 44 academic experts from the project leader's network and identified using a snowball approach to recruitment were invited to attend two workshops (October 2020 and March 2021).Participants included established academics and researchers along with PhD students, to provide an understanding of the challenges involved in modeling in public health.Of these, 42 attended the workshops and 36 contributed to the manuscript.
The aims and scope of the first workshop were informed by an organizing committee (PB, AB, HS, KE) and a scoping review of the literature on CSMs in public health (Table S1 in the supplementary material).The scoping review aimed to identify examples of CSM and catalog the various modeling methods used (Table S2 in the supplementary material).The review used and adapted a previous systematic review of complex systems methods (Carey et al., 2015).
Discussions from the two workshops provided direction on the scope, structure, and content of this guidance document.Details of the review and workshops are described in Appendix A and Appendix B of the supplementary material.
We tested the application of the definition of complex systems models against a set of CSM case studies.The case studies were identified by either the literature review or workshop participants to represent different model types, including but not limited to economic studies, across a range of public health topics.This process identified eight public health CSMs.We systematically cataloged methods used in the development of these models with input from workshop participants.

| SECTION A: DEFINITION OF COMPLEX SYSTEMS MODELS
A public health oriented CSM is a quantitative, dynamic, non-linear model that incorporates feedback, and interactions among model elements, in order to capture emergent outcomes and estimate health, economic and potentially other consequences to inform public policies.
To aid with the interpretation of the definition and guidance, we provide a glossary of terms in Table S3 of the supplementary material.The definition recognizes four overlapping critical features: dynamic, non-linear, feedback and interaction that can be programmed into an economic model that in combination give rise to the properties of a complex system.A figure illustrating the critical features is provided in Figure S1 of the supplementary material.A model incorporating the critical features will add complexity and predispose the model to complex properties.In contrast, the quantity of elements, or the intricacy of the intervention (Shiell et al., 2008), may make the model complicated, but not necessarily complex.Complexity is not the same thing as complication: non-complex models can be complicated, and complex models can be (relatively) simple.There are numerous examples of non-complex, but complicated, system models informing public health decisions linking public health policies to a broad range of outcomes (Springmann et al., 2016;Holmes, J et al., 2014;Thomas et al., 2022).These models do not include the critical features, that is, not dynamic (Springmann et al., 2016), do not include interactions (Holmes et al., 2014;Springmann et al., 2016), and do not include feedback loops (Holmes et al., 2014;Springmann et al., 2016;Thomas et al., 2022).
High system complexity is characterized by several properties: feedback, adaptation, emergent outcomes and non-linearity.These properties can be programmed into a CSM if it has the critical features so that the changes that ripple through a system (i.e., an intervention) are non-linear (not proportional to the size of the initial stimulus), adaptive and lead to emergent outcomes.In a CSM model changes to one element cause dynamic changes in other parts of the model, which continue to feedback around the model amplifying or dampening the initial change and resulting in further changes to the initial element.The ramifications of these relationships will be greater with more critical features connecting model elements in the system.

| SECTION B: WHEN SHOULD, AND SHOULDN'T, COMPLEX SYSTEM MODELS BE USED?
Figure S2 in the supplementary material illustrates a framework within which the model structure is decided upon and Figure 1 provides a decision tool to help modelers identify whether to develop a CSM.The following discussion expands on the questions in Figure 1, relating them to observations from case studies, and practical considerations.Prior to using Figure 1, it would be necessary to have a detailed understanding of the system (See Section D).
Public health problems often operate in complex systems.Economic models may be developed to enhance understanding of the patterns of behavior, mechanisms and processes of a complex system, or adapted from models with this purpose.Therefore, simplification would impact the utility of the model.Probst et al. developed a CSM which aimed to implement a theoretical model of social norms in an individual-level model to explain population-level trends in drinking behavior (Probst et al., 2020).The interactions and feedback between individual-level drinking behavior and social structures are inherent to the theory.They tested the theoretical framework with three hypotheses to provide insights into drinking behavior and identify interventions.In other case studies, it was possible to report whether feedback loops enhanced or mitigated intervention effects as they reverberated around the system (Keogh-Brown et al., 2019;Occhipinti et al., 2021;Stankov et al., 2019;Tobias et al., 2010).
It is essential to understand how the intervention interacts with the system, and specifically whether the relationships linking the intervention to the consequences include the critical features of complexity.Complex interventions do not need a CSM if the effects are not expected to change the properties of a complex system (Shiell et al., 2008), or if the effects can be approximated in other ways.CSMs are particularly useful in economic evaluations of multiple interventions, applied to different elements or levels (individual or aggregate) within the model (Keogh-Brown et al., 2019;Occhipinti et al., 2021;Probst et al., 2020;Stankov et al., 2019;Tobias et al., 2010;Viana et al., 2014).Comparisons of multiple interventions across different parts of the system are at greater risk of bias in a non-CSM model.The effectiveness of the intervention may depend on the context in which they are applied (Dodd et al., 2010;Occhipinti et al., 2021;Viana et al., 2014), and this context may be modified by other interventions.Infectious disease models illustrate the benefits of incorporating spatial structures to characterize the context and impact interactions between individuals in the model, and transmission rates (Ferguson et al., 2006;Riley & Ferguson, 2006).
It is also important to consider what consequences, perspectives, and time horizons are important to stakeholders and/or policymakers.For example, Tobias et al. simulate the interaction of smoking with the initiation rate of smoking for future generations (Tobias et al., 2010).This feedback loop is less likely to impact the findings if future costs and QALYs are discounted.
Once the modeler has identified whether a CSM should be considered according to the issues described above it is important to consider the practical constraints of model development and these are detailed in the second half of Figure 1.It would be advisable to identify what model types might be compatible with the problem (see Section C) and what data is available to answer these questions.Model planning needs to be sufficient to consider resources, and consultation with stakeholders, and may be iterative as unexpected problems arise.
The model must be fit for purpose and developed with the decision making context in mind.Computationally expensive CSMs may limit the capacity to generate timely and comprehensive analysis for fast moving policy decisions.The modeler needs to consider where to invest time and resources to respond to the decision problem.Non-complex models may be advantageous if their simplicity allows the modeler to accommodate other model features.The value of the investment in modeling must be balanced against opportunity costs.It is wise to keep things as simple as possible, because it can require a lot of time, effort and resources to make complexity tractable.The effort should be justified.
CSMs can require more assumptions that are not closely tied to strong experimental evidence.When developing CSMs researchers should consider how gaps in evidence will be filled and whether the tools for this are obtainable, that T A B L E 1 Eight selected Public Health Complex System Model (CSM) case studies that demonstrate the key features and insights of the approach.

Authors
Aims Study description

Why it counts as complex systems model
What useful insights did the CSM provide?

Dodd et al., 2010
To explore the effect of human immunodeficiency virus (HIV) transmission epidemiology on the impact of universal test-and-treat interventions.

Partial differential equations.
The model includes infectiousness that changes over time since HIV infection; under antiretroviral therapy infectiousness is reduced and lifeexpectancy extended.

Non-linear:
The model is a non-linear dynamical system.
The impact of a universal test-and-treat intervention was shown, for matched prevalence, to depend on heterogeneity and mixing of contacts.In some situations, less aggressive interventions achieved the same results, whereas in others, reductions were lower; annual strategies were not necessarily the most cost-efficient.The potential for incomplete implementation or coverage to increase long-term antiretroviral therapy (ART) costs was demonstrated.

Dynamic:
The model accounts for the population dynamics of HIV infection over time.

Interactions:
The PDEs model assortative mixing between highand low-risk segments of a heterosexual population.
Feedback: There is positive feedback -higher infection prevalence drives higher incidence of infection.

Brailsford et al., 2012
To evaluate the costs and benefits of alternative breast cancer screening policies in a screening model that incorporates human behavior.

Discrete event simulation.
A three-phase discrete event simulation was built to model breast cancer and screening policies and extended to include patients' behavioral characteristics.is, theory, data, or calibration.Combining multiple theories and adding parameters to be calibrated will make it harder to validate using external data, takes time and risks becoming less transparent and useful to stakeholders (Whitty, 2015).

Non
Increasing the number of model parameters to estimate with calibration may lead to overfitting to the data (Basu & Andrews, 2013).
It is worth considering how data gaps and the features of complexity might amplify uncertainty in the model.Calibrating multiple inputs can identify combinations of inputs that generate very different outcomes.Similarly, increasing the number of complex relationships can make the model highly sensitive to initial conditions and increase uncertainty in model outcomes.As the stimulus works its way through each set of relationships it can result in large differences in consequences (Calder et al., 2018).Additional resources may be needed to understand which relationships impact outcomes and communicate uncertainty to decision makers.Nevertheless, decisions are uncertain and simplification of the relationships risks generating incorrect and artificially precise results.Decision uncertainty is not removed by adopting a simple model, and simplification can be damaging to user confidence.Complexity will affect how the model is used and valued, so it is important to be sensitive to the needs of the decision maker in order to avoid over-simplification or overwhelming the audience with complexity.Non-CSMs impose assumptions that exposure variables are independent, and relationships between exposures and outcomes are unidirectional, linear, and constant through time (Page et al., 2018).If a non-CSM modeling approach is adopted, it is important to consider how the system violates these assumptions and be explicit about the direction of bias in the description of the model.If the features of complexity are believed to be important to the decision problem, but cannot be incorporated it will be necessary to discuss the potential limitations.Occhipinti et al. modeled suicidal behavior due to psychological distress where social determinants of psychological distress are also consequences of it (Occhipinti et al., 2021).It would be possible to have a unidirectional relationship between the social determinants and psychological distress, without feedback and under-estimate F I G U R E 1 A decision tool to identify whether a complex systems model is recommended to address a decision problem. the benefits of interventions to reduce psychological distress.By recognizing this limitation, a discussion of the direction and implications of bias in the model is possible.However, some complex relationships will be difficult to assess without incorporating the complexity in the model.For example, a similar CSM illustrated how unintended consequences for service demand negated intervention effects to produce much lower health benefits than predicted in a linear model, allowing policymakers to have strategic discussion about the whole system (Atkinson et al., 2020).

| How to select model types?
The selection of the type of model should be integrated into the decision on whether a CSM is needed with similar considerations for the decision problem, system features, and data (Figure 1).Each model type will imply different abstractions and assumptions about the system being modeled.Whilst there is a range of modeling types available, they typically differ across a few dimensions namely deterministic/non-deterministic, static/dynamic, discrete/continuous, individual/population, mathematical logic/algebra (Calder et al., 2018).Therefore, modelers can consider the problem across these dimensions when selecting a model type.Some will accommodate certain features more easily than others.
A toolkit for model selection can be used to guide modelers to what methodology is preferred for a given problem (Jin et al., 2021).The revised Brennan toolkit includes three tools: (a) the taxonomy (Table 2), (b) the checklist (Jin et al., 2021) and (c) a decision flowchart (Squires et al., 2016).Other useful resources have been developed to map the purpose and object of the problem to system dynamics, ABM and discrete-event simulation models simulations (Marshall et al., 2015).

| Suggested model types for Complex System Model
System dynamics models and ABMs are commonly presented as the methods available in complex systems problems (Morshed et al., 2019).Typically, a system dynamics approach adopts an aggregate perspective, whereas ABMs allow for individual-level simulation of behaviors, heterogeneity, and interactions between agents.However, the options for CSMs extend beyond these two dominant methods.
In other disciplines different labels may be used for model types and these are also relevant to the public health context.For example, Computable General Equilibrium (CGE) models have been used to model choices that have cross sector impacts that influence the indicators of the national economy (Keogh-Brown et al., 2019).Partial differential equation models can be used to describe the dynamics of infectious disease transmission (Dodd et al., 2010).Both CGE and Partial differential equation models have key similarities to system dynamics and describe aggregated populations.In infectious disease modeling, the term individual-based model is often used for ABMs.IBMs have long been used to include spatial and social population structure relevant to transmission (Riley & Ferguson, 2006), and have been commonly used in modeling COVID-19 and impacting policy decisions (Ferguson et al., 2020).IBMs allow easy inclusion of behavioral change to project its effects on transmission dynamics (Verelst et al., 2016).Moreover, multiple approaches can be combined in what some have called Hybrid models (Brailsford et al., 2019), integrating ABM, system dynamics and discrete-event simulation approaches into a unified model.A framework for hybrid simulation methods has been proposed (Mykoniatis & Angelopoulou, 2020), although to date very few applications have been identified within public health (Brailsford et al., 2019;Freebairn et al., 2020).We provide summary descriptions of modeling approaches in Appendix C as an introduction to the broad range of modeling options available, also providing links to relevant examples, good practice guides and web resources for further information.
Given the broad overlap among the methods described above, it may not be advisable to prescribe model types to specific public health problems, as it should depend on the nature of the question to be answered.Clusters in the adoption of model types can be driven by the research disciplines, expertise, and traditions from which they develop.Systems dynamics and CGE models are often used where the problem has broad boundaries, such as policies impacting more than public service (Keogh-Brown et al., 2019;Occhipinti et al., 2021); ABMs tend to be used in public health where human behavior and social networks are considered important, such as understanding health behaviors around alcohol, smoking, diet and physical activity (Probst et al., 2020;Stankov et al., 2019); and discrete-event simulations where healthcare resource constraints need to be modeled, such as cancer screening and mental health services (Brailsford et al., 2012;Viana et al., 2014).Individual level models are better at estimating the effects of heterogeneity, and exploring equity impacts.

| SECTION D: WHAT PROCESSES ARE IMPORTANT WHEN DEVELOPING CSMS?
CSMs introduce additional challenges for the modeler, which can be addressed with processes, approaches, and methods described in this section and highlighted in case study examples (Table S4: supplementary material).

| Stakeholder engagement
Strong communication with all relevant stakeholders throughout model development is essential to ensure that the model is fit for purpose (Squires et al., 2016).Within a CSM there may be broad and diverse perspectives and the choice of stakeholders may evolve during the project as the understanding of the complex system develops.Engaging stakeholders in co-production of a CSMs has been shown to improve model transparency, understanding of the modeling process, and may build trust and acceptability of the model and its outputs (Freebairn et al., 2018).Stakeholder engagement should also include engagement with public representatives in line with recommendations for public health research (Staniszewska et al., 2021).

| Understand and identify the problem
A documented understanding of the problem is imperative to develop and justify the model structure and type (Squires et al., 2016).The understanding of the problem will evolve as evidence becomes available, new stakeholder perspectives identified, or changes to the system occur within the timeframe of the project.
Diagrammatic representation of the system can be very useful to develop consensus and agreement between modelers and stakeholders, particularly where stakeholders have diverse perspectives.There is a vast array of approaches and methods that can be employed to develop a systems map.Commonly used methods in public health modeling include: group concept mapping (Koh et al., 2019;Lich et al., 2017), causal loop diagrams (Urwannachotima et al., 2019), and soft systems methodology (Checkland, 2000).

| Setting the model boundary
The specification of the model boundary is somewhat subjective.The decision on the model boundary should be made transparently, justifying and documenting reasons for inclusion and exclusion of each component in the understanding of the problem (Squires et al., 2016).It is important that CSM boundaries are well described, including the level of detail for each element (Robinson, 2008).This will facilitate appropriate interpretation of the model results, considering the broader elements of the Note: Based on the original Brennan taxonomy, and revisions (Brennan et al., 2006;A. D. Briggs et al., 2016).
system which have not been quantified.Understanding the key variables and concepts will aid the parameterization of the model and help to prioritize where investment of resources is justified.

| Incorporating data and evidence
Health economic models are often based on epidemiological models derived from empirical data.However, CSMs can be based on empirical observations or abstract constructions.For health economic policy evaluation CSM may use both techniques, but in most cases will require some empirical estimates and statistical techniques and causal inference.Table S4 in the supplementary material shows example statistical techniques used in our eight exemplar complex systems models.Model building and data processing require expert methodological knowledge to implement, and this can increase the expertise required for a modeling project.

| Creating synthetic individual data from observed data
Synthetic data creation techniques can be used if individual level data are not available for individual-level models (Probst et al., 2020) and data are not available for all parts of the system.Drawing on spatial microsimulation techniques, microdata can be constructed where there is detailed attribute information available from a survey or sample dataset and sample representation from census or administrative data.Techniques such as Iterative Proportional Fitting (Lomax & Norman, 2016) can be used to reweight sample data.Alternatively, Combinatorial Optimization (Smith et al., 2021), can be used to synthesize and replicate individuals.In both cases the resulting dataset is a combination of the attribute rich (micro) data and the sample representation of the administrative (macro) data.

| Making causal inferences from observational data
Feedback loops are a critical feature of CSM, but the model parameters within feedback loops may be difficult to measure due to time-varying confounding by other variables.A confounder is a variable that influences both the exposure and outcome variable.Time-varying confounding occurs when confounders have values that change over time because they are also affected by the (changing) exposure (Daniel et al., 2013).Therefore, in feedback loops between exposures and confounders, time-varying confounding is expected.In the presence of such exposure-confounder loops, simple regression does not identify causal effects because it provides a biased estimate of the true effect of the exposure on outcome (Kuehne et al., 2022).It may be necessary to develop bespoke statistical analyses if resources and data availability allow it.Analyses should be designed alongside the conceptual mapping.A taxonomy of methods to control for time-fixed confounding in observational studies has been developed to support statistical model selection (Ali et al., 2019).The causal analysis of empirical data affected by time-varying confounding requires more sophisticated causal inference methods (e.g., g-methods) (Johnson et al., 2009;Robins et al., 2004).

| Calibration of model inputs to model outputs
Calibration is the process of estimating the model parameters to obtain a match between observed and simulated patterns.A Bayesian calibration framework seeks to generate a posterior distribution of calibration parameters and model outputs, conditional on the calibration target (Chrysanthopoulou et al., 2021;Menzies et al., 2017).Non-Bayesian methods aim to identify sets of calibration parameters for which the model best reproduces the calibration target (Chrysanthopoulou et al., 2021).Bayesian calibration has the advantage of capturing uncertainty using probability distributions, which is compatible with probabilistic sensitivity analyses (Chrysanthopoulou et al., 2021;Vanni et al., 2011), but it can be computationally demanding (Chrysanthopoulou et al., 2021).In contrast frequentist calibration methods can overfit the model by implying no uncertainty exists in the model parameters (Vanni et al., 2011).Calibration will be a necessary step in the parameterization of many CSMs, and a strength of a modeling approach, as certain parts of the model will not have data that can be used to parameterize them a priori (Occhipinti et al., 2021;Probst et al., 2020).Regardless of what calibration process is used, calibration can encounter identifiability problems where there is insufficient data for the number of parameters to calibrate (Basu & Andrews, 2013).
There are a broad range of calibration methods, although their use is often poorly documented, and non-reproducible (Hazelbag et al., 2020).Calibration reporting guidelines have been developed (Stout et al., 2009), and should be adopted when reporting calibration.

| Computational efficiency
Complex System Model models may encounter problems with computational capacity, particularly with micro-level models.A computational expensive model that has many parameters to calibrate may exceed the project resource limits.Consideration of computational efficiency is important and will depend on how and when the outputs will be used.Computational limitations may require the model boundary to be revised to reduce complexity and still maintain tractability.Approaches to deal with computational efficiency, such as model emulation (Rothery et al., 2020) and distributed simulation (Taylor, 2019), could be used.

| Model uncertainty
Sensitivity analysis and parameter uncertainty analysis can be used to communicate uncertainty to stakeholders.This can be challenging in CSMs because analyses of uncertainty may be limited by computational capacity, and other time constraints (Rothery et al., 2020), the model developers may not understand the source of uncertainty, and uncertainty can affect how answers from the model are used.Model structures, as well as parameters, may be uncertain and recent approaches have used machine learning to search across different structural configurations of a CSM (Vu et al., 2020).Proponents of a Weight of Evidence approach to evaluate and weight evidence (Dion et al., 2021) advocate holding space for multiple ways of understanding the same issue.The modeler should consider what uncertainties exist beyond what has been programmed into the model and report where complex features have not been included in the model, and how this might impact findings.

| Model validation
Model verification and validation ensures that results from models are accurate and can be confidently used by decision makers.Model developers should consult validation typologies (Eddy et al., 2012), recommendations (Vermeer et al., 2022) and validation assessment tools, which can help assess whether sufficient validation of a CSM has been undertaken (Vemer et al., 2016).Verifying and validating a CSM is a continuous process to be performed throughout the life-cycle of a study (Balci, 1994) and needs to be designed based on the resources and data available.CSM's may introduce additional challenges during validation, and we propose potential solutions (Table 3).It is impossible to prove that a model is valid, so verification and validation is a process of increasing confidence in a model to the point that it can be used for decision-making (Robinson. 1999).
Co-production and model transparency can be used to ensure models are subject to face validity and verification checks, which will help build trust from decision-makers.Examples include testing face validity and model verification through discussion of model structures with experts (Brailsford et al., 2012;Occhipinti et al., 2021;Tobias et al., 2010), Verification through discussions with stakeholders should be an iterative and multifaceted process throughout all stages of conceptual modeling, model formulation, coding and use (Williams, 2018).Model transparency should at a minimum require detailed documentation of the model to allow for it to be reproduced (Eddy et al., 2012;Vermeer et al., 2022).However, calls for "Open-Source" models provide an opportunity to build trust in models, and improve validity (Dunlop et al., 2017).
External validation compares CSM output with retrospective data (Brailsford et al., 2012), and cross-validation compares CSM output with other models (Viana et al., 2014).The processes of external validation and calibration may be inherently intertwined, and the stages of model calibration and validation may be indistinguishable if there are not enough data available to separate these tasks (Stankov et al., 2019).Comparing model outcomes against data not used in to inform model parameters can be challenging for public health CSM models.The data may not be routinely measured, or only measured with sampling or reporting bias.Furthermore, the feedback loops may change over time changing the relationships between exposure, confounders and outcomes.CSMs should be validated at the individual-level, or in constituent parts, to test whether each part represents the real world with sufficient accuracy, and system-level, or overall model, to confirm if the emergent dynamics of the system outcomes are reproduced, adding to the validation tasks required (Vermeer et al., 2022).

Validation type Description
Challenges for complex systems models Recommendations for CSM

Face validity
A review of the model structure, evidence used, and results to ensure they all make clinical and logical sense.

Cross-validation
The process of comparing model results with those of other models produced for the same problem.
Challenge 3: There are less likely to be other models developed to address similar public health problems to compare against.
1) Adoption of our definition of complex systems models will help to identify similar modeling approaches.2) Greater investment is needed to fund cross-validation of model programmes.

External validity
Compares model results with real world data.
Challenge 4: If calibration is needed the process of calibration and validation becomes connected.
A detailed description of calibration and validation processes is needed to ensure transparency in model development methods.It should be clear that the process of validation uses different data sources from the calibration process.
Challenge 5: Real world data may not be available for all variables and consequences.
Detailed documentation of what variables and consequences have been, and have not, been validated.

(Continues)
T A B L E 3 (Continued)

Validation type Description
Challenges for complex systems models Recommendations for CSM

Predictive validity
Compares the predicted model results to prospective real world observed events when they become available.
Challenge 6: The predictive validity of complex systems models may be difficult to establish.Real-world shocks to the system, such as the Covid-19 pandemic, may distort observations and change the mechanisms of the system.The feedback loops between exposure confounder and outcomes might change with time.
CSMs should be revisited and assessed for predictive validity as new data become available.The timelines and funding structures of public health projects may not facilitate this.However, public health CSMs require substantial investment and can be adapted and updated to address new public health questions.Adapting and updating models provides an opportunity for the model to develop and evolve as evidence and understanding of the system evolve.

| DISCUSSION
This paper provides a definition for CSMs and guidance on developing CSMs for public health economic modelers.It is intended as a helpful tool for economic modelers, but may also help stakeholders, those commissioning models, and those critically appraising models to identify when a CSM is justified.The definition specifies key features that distinguish complex models from non-complex models to provide clarity to health economic modelers in the context of growing demands for complex systems approaches to public health evaluations.The absence of a clear definition of a CSM may have led to inconsistent labeling of models, less effective evidence reviews, less efficient description of methods, barriers to interdisciplinary research, and may have hampered the adoption of complex systems models.We recommend that CSMs are required when there are processes involving dynamics, feedback loops, non-linearity and interactions which produce emergent outcomes that matter to the decision problem.Such modeling provides a deeper understanding and analysis of the likely impact of changing factors which affect the system.We discussed useful resources to select and implement macro-level complex system models and individual-level models as well as hybrids of these.Finally, we identify challenges that modelers will face when modeling public health decisions and propose approaches and techniques that researchers may need to consider when designing CSMs.
Complex System Model is developing apace to support and inform public health and health policy decisions.Our hope in producing this guidance is that it can provide an accelerated learning curve, both for those new to this field and for those already involved in developing such models.We hope this document will be a lever for improved understanding and engagement with CSM and hence have an impact on public health systems policy and decision making.
This guidance document provides a step toward classifying computational models in public health according to the inclusion of complexity.The aims of this guidance document do not extend to setting research priorities for methodological development.However, workshop participants identified a lack of technical guidance specific to the task of developing CSMs for public health, meaning researchers need to access a diffuse and diverse literature from across multiple disciplines.Further research should consolidate best-practice guidance to support skills development and training in this field.Understanding and overcoming the barriers to CSMs, for example, the resources and technical expertise needed, data requirements, limitations of model validation, will be critical to support the adoption of these methods by modelers and policymakers.
Summary of challenges when undertaking validation of Public Health Complex Systems Models-categorized by ISPOR task force types of validation.

AFFILIATIONS1
School of Health and Related Research, University of Sheffield, Sheffield, UK 2 British Medical Journal Technology Appraisal Group, London, UK 3 MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, Scotland, UK 4 School of Geography, University of Leeds, Leeds, UK 5 Department of Public Health, LaTrobe University, Melbourne, Australia 6 Centre for Public Health, Queen's University Belfast, Belfast, UK

Authors Aims Study description Why it counts as complex systems model What useful insights did the CSM provide?
(Continues)T A B L E 1 (Continued) An increase in the desire to drink led to the most meaningful changes in the population's drinking behavior indicating the high levels of autonomy in decisions to drink.A higher degree of "receptiveness" toward normative influence can be considered a prerequisite to behavioral changes.T A B L E 1 (Continued)

Authors Aims Study description Why it counts as complex systems model What useful insights did the CSM provide?
T A B L E 1(Continued)

Authors Aims Study description Why it counts as complex systems model What useful insights did the CSM provide?
T A B L E 1(Continued)

Authors Aims Study description Why it counts as complex systems model What useful insights did the CSM provide?
T A B L E 1(Continued)

Authors Aims Study description Why it counts as complex systems model What useful insights did the CSM provide?
T A B L E 1 (Continued)Abbreviation: DES, Discrete Event Simulation.

Cohort/aggregate-level/counts Individual-level Expected value, continuous state, deterministic Markovian, discreet state, stochastic Markovian, discrete date Non-markovian, discrete state
Brennan taxonomy of model structures.
T A B L E 2