Strengthening a weak link: transparency of causal loop diagrams — current state and recommendations

Transparency is a critical aspect of systems science. While transparency of quantitative models has been assessed, transparency of their qualitative structures has been less scrutinized. We assess the transparency of causal loop diagrams (CLDs), a key qualitative visualization tool in system dynamics. We evaluate System Dynamics Review ( SDR ) publications and a sample of most-cited comparable articles in other journals. We assess the inclusion of a plain-language methods statement, overall discernibility of the methods, and identi ﬁ cation of causal link sources. Reviewing 72 articles ( SDR : 36; other journals: 36), only 44%, 38%, and 25% fully satisfy each criterion, respectively. SDR articles are characterized by higher transparency in the clarity of CLD development method and communication of causal link sources, yet the potential for enhancement is evident. We provide speci ﬁ c recommendations to increase the transparency of CLDs. Transparent reporting bene ﬁ ts original research authors, future expansion of CLDs, and the systems science community. Copyright © 2023 The Authors. System Dynamics Review published by John Wiley & Sons Ltd on behalf of System Dynamics Society.


Introduction
As a cornerstone of scientific methodology, transparency is openly expressing all research details and procedures.In systems science and simulation modeling, transparency has been a frequent subject of concern over the last decade (Jalali et al., 2021;Martinez-Moyano, 2012;Rahmandad and Sterman, 2012).The recent and rapid proliferation of COVID-19 models has prompted calls for more transparency in reporting quantitative simulation models (Barton et al., 2020;Jalali et al., 2020).In system dynamics (SD), while substantial attention has been paid to the transparency of quantitative models, where methods and reporting standards have been established (e.g.Rahmandad and Sterman, 2012), there has been less discussion about the transparency of qualitative visualization tools.Causal loop diagrams (CLDs), with or without a stock and flow structure, are a common and powerful tool for a qualitative and visual explanation of the structure and dynamics of a system, thought experimentation within systems, expansion of mental models, and stakeholder engagement (Baugh Littlejohns et al., 2018;Sterman, 2000).While there has been considerable progress in reporting SD models, a major question remains: we build CLDs to make a problem transparent; but do we make our CLDs transparent?
Transparency in SD matters greatly-regardless of whether a qualitative CLD is transformed into a quantitative simulation model-to build confidence in perceptions of SD research by the greater scientific community.The literature has established best practices for an array of topics related to transparency in qualitative SD research methods (Black, 2013;Black and Andersen, 2012;Kim and Andersen, 2012;Luna-Reyes and Andersen, 2003;Tomoaia-Cotisel et al., 2022).However, a review of the degree that transparency is achieved in CLDs is currently missing.
We aim to address these gaps.Particularly, the twofold objective of the current analysis includes an investigation into the status of transparency of CLDs, followed by the provision of a set of recommendations regarding the transparency of CLD reporting.Note that in our defined scope, we solely focus on basic aspects of reporting transparency.We do not assess other critical quality attributes of CLDs, such as clear link and loop polarities and variable names with discernible sense of direction, among others.

Methods
Our review process followed the scoping review methodology (Arksey and O'Malley, 2005) to characterize the degree of transparency of CLD reporting present in a specific set of SD literature.We defined as the set of relevant literature a collection of research articles published between 2018 and 2021 and containing the presentation of a CLD.Given that System Dynamics Review (SDR) is the flagship journal within the field of SD, we sought to investigate transparency characteristics in SDR articles.Additionally, we reviewed a parallel set of publications in other journals to capture a broader set of literature not limited to one journal.We selected publications separately by journal type to investigate whether a problem (if any) is pervasive in the literature.If papers published in the SDR-where high standards are expected-show deficiencies in transparency, it suggests a broader need for improvement in the wider literature.A detailed description of the search, screening, and inclusion determination steps is provided in the online supporting information.
We evaluated the included studies to determine the degree to which they fulfill three basic transparency criteria.We chose these criteria based on two previous transparency assessments in modeling (Jalali et al., 2020;Jalali et al., 2021), in which over 20 transparency criteria were reviewed.For simplicity, we selected three to apply in the current context because these three criteria are universal across CLDs.While other criteria focus on technical aspects like detailed model equations, such specifics fall outside the scope of CLDs.In other words, we chose them based on our reasoning of essential, minimum elements required for the reader to understand the methods and sources that generated the CLD.The first two referred to the CLD development process, and the third referred to the CLD visualization.
The three assessment criteria were: (i) the article includes a direct statement in plain language that describes the method of how the CLD was generated (i.e. via literature review, interviews, group model building (GMB), authors' intuition, or combinations of these approaches); (ii) even if such a direct statement is given, the degree that the method for the development of model structure is discernable by the reader (i.e.given the full breadth of the information contained in the article text and figures); and (iii) the degree of identification of the information source for causal links depicted in the CLD.The first criterion was summarized with a binary yes/no designation.A lack of a clear statement on the CLD development method does not preclude authors from describing their CLD and its sources.Thus, papers could still achieve scores for the second and third criteria despite a "no" for the first.Within the second and third criteria, a score out of 3 possible points was assigned.The scoring rules were defined as: 1 = unclear/not described; 2 = described with some detail missing; 3 = fully identified.Criterion 3 could be satisfied if source identification takes place directly in the visualization of the CLD, in a tabular form, or otherwise discernible based on the text provided.For CLD generation approaches based less on quantitative data or literature, such as those based exclusively on individual inputs, for example, GMB, citations on individual causal links in the visualization would not apply equivalently.While we note that practical approaches to identify the sources of specific links as attributable to individual inputs are possible, such as via detail in methods text or within an illustration, such cases were considered not applicable for the third criterion for the purpose of our review.Table 1 presents the summary of the three transparency assessment criteria.
After collecting data from the completed extraction sheet, we identified practical recommendations for improving CLD transparency.We based these on two factors: (i) literature sources and (ii) our experience, along with observations from our review.The literature offered foundational recommendations to enhance CLD reporting transparency across multiple approaches.Building upon prior work, we categorized core approaches to create CLDs and presented recommendations for each approach.Observed gaps in reporting and other insights gleaned from our assessment of recent CLDs affirmed the need for specification of such transparency guidance and supplemented our recommendation framework.We summarized these recommendations in the Discussion section.

Results
The search strategy and inclusion criteria resulted in 72 articles: 36 from SDR and 36 from other journals (see Figure S1 in the online supporting information).For the first transparency assessment criterion, only 32 articles (44%) directly communicated the methods underlying the CLD structure.Twenty-seven (38%) fully satisfied the second criterion regarding the degree of clarity in communicating the approach driving the CLD structure, receiving a score of 3. Twenty-four (33%) and 21 (29%) received scores of 2 and 1, respectively, for this second criterion.Finally, only 18 (25%) fully satisfied the third criterion regarding the extent to which individual causal link sources in the CLD structural representation were discernable to the reader.Thirty-nine (54%) and 9 (13%) had respective scores of 2 and 1 for this third criterion.Table S1 reports the details of the analysis.
Results separated by articles appearing in SDR versus other journals are displayed in Figures 1-3. Figure 1 shows that both SDR and non-SDR studies had a similar proportion (44%) that omitted a direct method statement explaining the CLD development approach.The CLD development method was fully expressed. 3 The CLD development method was described only in part and had some detail missing.

2
The CLD development method was either unclear or not described.As shown in Figure 2, CLDs presented in SDR articles contained higher clarity versus other journal articles in their communication of the process of CLD development.Specifically, 89% of SDR articles received a score of at least 2, compared to 53% in other journals.However, with less than half of the articles receiving the top score of 3, the results demonstrate apparent room for improvement across all literature sets.Figure 3 illustrates mixed outcomes for the third criterion under evaluation.While CLDs from SDR generally fared slightly better than those from other journals in denoting sources for individual causal links (86% vs. 73% achieving a score of at least 2), the difference was more pronounced for those achieving the top score of 3%-44% in SDR articles compared to 6% in other journals.
An article from the sample we reviewed that is characterized by high scores in our transparency assessment may prove to be an illuminating example when contrasted with methods from lower-scoring articles.Matchar et al. (2018) is one of the reviewed articles that received the highest scores possible across all three criteria.For the first criterion, the CLD development approach is directly identified by Matchar et al. (2018) as coming from a combination of literature and input from experts and stakeholders.The Matchar et al. (2018) methods description employs clear language about the development process in a concise identification of steps to first develop a conceptual model supported by literature, followed by presentation of the conceptual model to experts for verification of structure and assumptions, and finally by implementation of an iterative refinement process supported further by expert input.The article thus satisfies our second transparency criterion concerning the overall clarity of the CLD development method.Meanwhile, an article that scored 1 does not describe the CLD method in any way.An article that scored 2 gives only a partial indication to state that a model has been developed on a particular topic, and while authors cite some literature for a subset of the key relationships, it remains unclear to the reader whether the literature served as a foundation for CLD development or if that literature was used rather to validate relationships represented that may have been constructed using an alternate method.Finally, to contrast articles in terms of the third and final transparency criterion via specific examples, Matchar et al. (2018) present a CLD visualization where each causal link either directly maps to a specific information source, or they are attributed to expert or stakeholder input as described in the methods text.Note that while the format of producing citations directly in the diagram is one way to satisfy this criterion, other ways such as an adequate description in the text about causal link sources would equally be sufficient for a score of 3.
Additional examples of articles that scored highly across the transparency criteria may prove useful.Rahmandad et al. (2021) present a model to estimate disease transmission using the established Susceptible-Exposed-Infected-Recovered (SEIR) model framework as a foundation.This article scores "yes," "3," and "3" in the three transparency assessment criteria.The authors extend the SEIR model and clearly delineate which aspects are the novel model features and that the new features are built around author assumption and literature.Another article that scores "yes," "3," and "3" in the three transparency assessment criteria concerns modeling water resource management and also utilized an existing framework (in this case, the Driver-Pressure-State-Impact-Response (DPSIR) model) as a starting place to build an extension (Zare et al., 2019).Authors base the extension on two sources: literature and interviews.They generate the CLD extension in the context of a case study, where links are based on literature sources, and proceed to confirm and refine the CLD based on informal interviews with experts and stakeholders familiar with the problem.Individual links are then able to be traced to cited literature provided in the text and are otherwise attributed to qualitative information gathered in interviews.The limited aspects of these models, which extend beyond the base SEIR and DPSIR models, simplify the explanation of sources.This makes it easier for readers to trace the new parts of the CLDs to the methods description and sources.

Discussion
The results of our transparency assessment showed that despite our evaluation of highly basic transparency criteria, there remains substantial room for improvement in the current SD modeling literature.Other transparency criteria related to modeling research beyond this baseline level have been the subject of investigation within detailed studies (Jalali et al., 2020;Jalali et al., 2021).Our results show that transparency in publications found in SDR compared to publications in other journals was similar in the provision of clear CLD development approach statement (criterion 1).SDR publications performed better than other journals in their overall clarity of communicating the CLD development approach and causal link sources (criteria 2 and 3), yet they still have deficiencies.
Below we discuss the importance of transparency in reporting CLDs and provide additional examples.We then provide our recommendations for increasing transparency in reporting CLDs.

Why transparency
Causal loop diagrams that are not fully transparent in their reporting have inherent weaknesses.The general approach to creating a CLD as well as the source underlying specific causal links are ambiguous to the reader when articles presenting a CLD are not fully transparent.As observed in this review, deficiencies in CLD transparency are not uncommon in recent literature.
Not only does transparency (in three dimensions: data, analytics, and production (Moravcsik, 2014)) offer advantages to individual publications and subsequent model-building activities by increasing trust and confidence in the validity of models, but it also provides powerful benefits to the field of SD.Advances in transparency for SD models can benefit the field by inspiring trust in the underlying methods in the view of the greater scientific community and general public who are not necessarily familiar with SD approaches.Richardson identified in 1996 that "confidence and validation" were among the problems threatening the future of SD research (Richardson, 1996).
Improvements in transparency in CLDs can benefit various parties.The author group that creates the CLD benefits when it adopts a more transparent representation since such methods are more easily identifiable as rigorous scientific research (Prager et al., 2019).Additionally, to create quantitative models based on such CLDs, modelers have a better idea of purely assumed or hypothesized versus expert, group, or literature-based causal links when the base CLD is presented with transparency.Quantified model builders, possibly external to the group that created the original CLD, gain clearer direction.They can better prioritize data sources when quantifying the model.
Overall, poor transparency negatively affects perceptions about validity in the scientific approach (Yarborough et al., 2019) and therefore also degrades trust in models.As Sterman noted on the state of the SD at its sixtieth anniversary, "replacing a poor mental model with a diagram, archetype, or simulation that is not grounded in evidence and is poorly tested may create more harm by providing false confidence and more deeply embedding flawed mental models" (2018, pp.39).From a broader perspective, Moravcsik noted that transparent foundations are the required starting place before technologically advanced methods serve a purpose to build informative and appropriately interpretable models (Moravcsik, 2014).While flexibility in mental models behind CLD building has many advantages, neglecting to clearly communicate the sources of information gives the impression that this powerful tool lacks scientific merit.Reproducibility is upheld as a characteristic of rigorous scientific research that goes hand in hand with transparency (Nosek et al., 2015).Transparency thus serves another important end in enabling the reproducibility of the research product.A lack of full transparency in reporting CLDs inherently limits reproducibility.

Existing best practices
A possible source of deficiencies in transparency is researchers' only partial adherence to existing best practices for methods reporting.The literature offers exemplary guidelines for transparent research methods.These contributions include the areas of qualitative data generation (Kim and Andersen, 2012;Luna-Reyes and Andersen, 2003;Turner et al., 2013); implementation of qualitative inputs to SD models (Tomoaia-Cotisel et al., 2022); and visual representations (Black and Andersen, 2012) and reporting (Black, 2013) in participatory (i.e.group) model building.If such guidelines are followed closely, the resulting work will include at least a minimum level of transparency, clarify how the causal links were interpreted from their sources, and help identify potential sources of bias in the development of the CLD.Thus, not only the establishment of best practices for transparent research but also the improvement of adherence to those best practices are areas worthy of prioritization.
A challenge to the appropriate provision of details sufficient to warrant good transparency is expected to be the word count limits in journals.However, authors must leverage the opportunity to provide a higher level of detail in documentation via online appendices or repositories.It should also be noted that the responsibility to report transparent CLDs falls primarily with the respective publications' authors and secondarily with the journals' peer reviewers.
Articles published earlier than the four-year window used in this review also offer useful examples that provide insight as to good practice in transparent presentation of CLDs.One such example is by Wittenborn et al. (2016) on modeling major depressive disorder.The study authors thoroughly cite the sources for information within their CLD.An excerpt is provided below in Figure 4 that displays one of the feedback loops and the associated causal links with citations.
Not only the identification of sources but also the strength of the evidence behind individual causal links can be indicated visually by the width of arrows as done by Hu et al. (2011) in a qualitative system map.In another example, Repenning and Sterman (2001) rigorously explain the details and sources of information that went into the development of their CLD with a stock-and-flow structure for a model based on various ethnographic analyses and iterative model building.They present each feedback loop in a stepwise fashion and, while presenting each loop, further integrate the sources in the text by often referring back to these sources within the explanations.Similar stepwise-fashion examples include Jalali and Kaiser (2018) and Jalali et al. (2017).
In another study, Kim and Andersen (2012) demonstrate a systematic approach to transforming the qualitative information source (i.e.text) via coding to words-and-arrow diagrams which can then be implemented in a CLD. Figure 5 illustrates part of this process.Here, the numbers indicate a previous table's words-and-arrow causal argument numbers derived from the text rather than from citations.
We mention these earlier articles to highlight noteworthy examples and motivate aspiring to a high standard of transparency in SD reporting.Additionally, we provide these examples not to imply that every article must employ the same presentation style but rather to document effective approaches authors have used to achieve transparency of methods and information sources.Transparency recommendations for reporting CLD Ensuring transparency in the process of developing CLDs is fundamental to validating the credibility and reliability of research findings.Table 2 presents six approaches to develop a CLD.First, a formalized systematic literature review process is one approach to create a qualitative representation of a system via CLD.Standard processes and guidelines for conducting systematic literature reviews have been widely published (Higgins et al., 2019;Moher et al., 2015;Page et al., 2021;Tricco et al., 2018).Recommendations from the literature guiding systematic literature review methods and best practices carry forward in their application to CLD building.For example, the search strategy, screening, and data extraction processes should be described in adequate detail, and inclusion of a PRISMA flow diagram is recommended to clearly communicate the core elements applied in the review.
Second, publications presenting guidance on methods also inform recommendations to guide alternative forms of literature-based approaches, such as a scoping literature review or other review-based approaches, when these review types are utilized rather than a formal systematic literature review (Colquhoun et al., 2014;Peters et al., 2015).For these review-based processes, the basic elements of the search strategy-such as approximate timing of the initial search, databases used, and key words applied in the search-should be clearly described.Literature-based approaches, whether less formal reviews or systematic literature reviews, present the opportunity to label literature sources of individual causal links shown in a CLD visualization.Labeling or otherwise communicating the sources of individual causal links can help to reduce ambiguities regarding which source(s) informed which piece(s) of the model.
Third, information guiding the development of a CLD may also be sourced directly using human input, such as via interviews with experts, community members, or other stakeholders.Under this approach, CLD builders should report out interview subject characteristics and describe the process to elicit input, such as a structured or semistructured interview and interview questions (Ford and Sterman, 1998).They should also report how information learned from the interviews was recorded and interpreted (Kim and Andersen, 2012) and eventually how such information was synthesized and used to identify causal processes in CLDs.
Fourth, GMB is another approach for CLD building where the information is sourced directly from group participants.The SD literature contains various guidelines for GMB (Hovmand, 2014).Modelers should identify details of the process including participant and modeling facilitator selection and characteristics, the set up of workshop, and other relevant attributes such as scripts used.A description of the process taken to synthesize community input is also recommended.Methods that rely on an aggregation of inputs wherein individual causal links are not necessarily traceable to a single person or group may not align directly with the above-illustrated manner of specifying sources.GMB-based methods are no less valid than other methods, although they do present distinct needs for transparent reporting of details such as description of interview subjects' background, sufficient detail on stakeholders' perspectives, and potential sources of bias, among other reporting essentials.Documentation generated during GMB activities serves as an important tool for later translation of the information gathered to construct a CLD.Tools such as scripts formalize methods to record information and improve the transparency of processes implemented (Hovmand et al., 2012).Tailored methods structures may also be employed for GMB regarding sensitive topics (Deutsch et al., 2022).
Fifth, the authors' own understanding of the system, or authors' assumptions, is another source to build a CLD.Model builders should be clear in their identification of authors' assumptions as the basis of the CLD structure.We also recommend disclosing potential limitations introduced by this approach.Importantly, while implicit biases are inherently present across all modeling methods, CLDs often include authors' understanding of the system and possess a high degree of subjectivity to their respective backgrounds and perspectives.Hence, authors should further enhance transparency by distinguishing causal links based on their understanding versus other sources (e.g. through different line patterns within the visualization or in a table).Overall, we recommend combining this approach with other approaches noted above to increase scientific rigor.
Finally, many models do not conform to exactly one type of the approaches introduced above but instead use a combination of such approaches.For example, a CLD may be built primarily based on information from a targeted literature search process and be supplemented with information from human subject inputs via interviews, for example (Beaulieu et al., 2022).We refer to a combination of two or more approaches as ensemble methods.In these cases, it is important to state the various methods adopted and to also align methods with best practices for each respective element used in the CLD building process.

Study limitations
The publication date range of studies included in the review was limited to 2018-21 (the initial version of this report was submitted for peer review in 2022).Additionally, the sorting performed by most cited favors the inclusion of articles published in earlier years, although we do not expect the transparency of 2018 articles to be systematically different from 2021 articles in a way that would impact our findings.A more sophisticated approach would consider the age of each article, especially when longer study durations are considered.We did not extract certain details like the article's purpose or if it was part of a series of related articles.These omitted details might have significant implications for assessing transparency.For instance, if an earlier article in a series provided an explanation of structural model sources, our review would not account for this information.Also, this report was limited by the extent of the three transparency criteria assessed.A more thorough review could inspect a wider variety of transparency aspects as well as adherence to best practices for CLD development.

Conclusion
Our review study documents a deficiency present in the transparency of CLD reporting.Relative to other types of simulation modeling methods, which are sometimes limited to expressions of mathematical equations to communicate the underlying model components, SD models have CLDs available as an invaluable tool to communicate the model structure visually.CLDs are also valuable in their own right, independent of performing a simulation exercise.
Overall, transparent CLDs empower the reader to identify the approach used to generate the CLD-whether that be prior research, individuals' opinion, group model building, or some combination of these.Improvements in the transparency of CLDs allow a variety of researchers to take full advantage of this tool.Transparency in CLDs as outlined in the recommendations presented in this article can thus help improve the clarity of evidence underlying models and uphold the perception of the field as one built on solid scientific rigor.

a
Harvard Medical School, Boston, Massachusetts, United States b MIT Sloan School of Management, Cambridge, Massachusetts, United States Fig. 4. Excerpt of feedback loops within a causal loop diagram that cites the source of information for each causal link, adapted fromWittenborn et al. (2016)

Table 2 .
Recommendations for transparency of causal loop diagrams (CLD)General recommendationsTransparency in three dimensions: data source and analysis and CLD development and production • Ensure that readers can discern the source of information for all causal links • Distinguish between causal links that are based on authors' intuition versus links resulting from other sources • Include a detailed description of the approach used to create the CLD • Use journals' supplements or other repositories to present detail to complete the recommendations above a Recommendations are based on literature sources where cited or authors' experience and observations otherwise.bPRISMA:Preferred Reporting Items for Systematic Reviews and Meta-Analyses.cWe recommend combining this approach with other approaches in the table to increase scientific rigor.