Best practices in system dynamics modeling

Authors

  • Ignacio J. Martinez-Moyano,

    Corresponding author
    1. University of Chicago, Computation Institute, Chicago, IL, U.S.A.
    • Argonne National Laboratory, Decision and Information Sciences Division, Argonne, IL, U.S.A.
    Search for more papers by this author
  • George P. Richardson

    1. Rockefeller College of Public Affairs and Policy, University at Albany, NY, U.S.A.
    Search for more papers by this author

  • The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. This work was funded in part by the U.S. Department of Homeland Security.

Correspondence to: Ignacio J. Martinez-Moyano. E-mail: imartinez@anl.gov

Abstract

This research explores opinions about best practices in system dynamics modeling elicited from a distinguished group of experts in the field. We address three questions: What do practitioners believe is the best way to undertake system dynamics modeling? What specific core activities are essential for exemplary action during the different stages of the modeling process? What do experts believe are the most important practices during the different stages of the modeling process? The researchers used a multi-method approach involving interviews, virtual meetings using the Internet, statistical analysis of the generated data and, finally, a facilitated face-to-face meeting in which experts discussed the results of the study and their implications. The results of this research include 72 best practices grouped into six categories that reflect the stages of the system dynamics modeling process. Copyright © 2013 System Dynamics Society.

There is no monolithic canon that everyone agrees on. David F. Andersen1

Are we saying that we cannot come up with simple context-independent advice? Yaman Barlas

Overview

This paper begins with a brief review of the extensive literature available on best-practice implementation and rule formation. The authors then review the system dynamics literature on best practices in the field, concluding that most of the development of best practices is described, reasonably enough, in our important teaching texts. The paper then presents the multi-stage method employed in this research to compile the opinions of a selected set of experts in the field, including past presidents of the System Dynamics Society and winners of its awards.

The study produced a large number of statements about what these experts see as best practices in the field during six stages of the modeling process. Several filtering procedures, described below, were used to narrow the statements down to a manageable number of assertions about best practices during the six stages of system dynamics modeling—tracking both the perceived importance of each assertion and the extent of agreement among the experts. Table 1 presents an overview of the results. The paper concludes with identification of the most important of the almost 200 statements elicited. Developing this list was also a multi-stage process that eventually involved rewording the statements to capture majority opinion. Of particular interest is the identification of the highest-importance statements with which there is low agreement (the second row in Table 1), as well as those having high agreement.

Table 1. Summary of the number of statements of best practice elicited, broken down by modeling stage, perceived importance, and extent of agreement among the participantsThumbnail image of

Background

The search for improvement is not new—neither is the one for best practices. According to Nattermann (2000), best-practices research can be one of the most effective tools for improvement in organizations. However, “the process of identifying and transferring [best] practices is trickier and more time consuming than most people imagine” (O'dell and Grayson, 1998, p. 155). More than a century ago, Frederick Taylor (1911) sought empirical foundations for best practices in organizations. Thus began the legitimate search for best practices as mechanisms for organizational improvement.

In this investigation, following the approach used by Martinez-Moyano (2005), best practices are identified as likely precursors to rules and principles of practice in the quest for improvement. Practices identified as “best” and considered relevant for action have rule-like characteristics capable of influencing action. Rules, by being interpreted and implemented, motivate and delimit action. Action, in turn, influences the creation of results that, through adaptation, shape goal setting. Formalization and operationalization of goals effectively determine new rules and changes in existing rules—resulting in an evolutionary cycle of rule development and change (see Figure 1).2

Figure 1.

Basic feedback mechanisms of rule change

Bardach (1994) defines the goal of best-practices research as akin to widening the range of solutions to organizational problems—implying that implementing best practices can become the solution to problems in organizations. Overman and Boyd (1994) categorize best-practices research as a version of the method of inductive practice-to-principles research and define it as “the selective observation of a set of exemplars across different contexts in order to derive more generalizable principles and theories” (p. 69). They hold that best-practices research should be pragmatic, practice driven, innovative and entrepreneurial, positive and prescriptive, and commercial and user friendly.

Following extensive examples of best-practices research in organizations (Bogan and English, 1994; Boxwell, 1994; Cook, 1995; Andersen and Pettersen, 1996; Codling, 1998; Reider, 2000), researchers have linked the concepts of best practices and benchmarking to how outstanding companies evolve over time (Collins and Porras, 1994; American Productivity and Quality Center, 1997; Fitz-enz, 1997). In the government context, best-practices research is exemplified by Keehley et al. (1997), Holzer and Callahan (1998) and the United States General Accounting Office (1995), among several. Furthermore, academicians have worked to clarify the best way to identify best practice and benchmarking procedures (see Zairi, 1996; Geva-May and Wildavsky, 1997).

Sources for identifying best practices include literature reviews, industry trends, dedicated best-practice centers, Internet resources, networking, organized benchmarking activities and site visits, cooperative agreements, and the advice of experts (United States General Accounting Office, 1995; Hughes, 1996; Jarrar and Zairi, 2000; Dattakumar and Jagadeesh, 2003; Kyro, 2003; Menzies et al., 2006; Demortain, 2008).

Identifying and clarifying best practices can be challenging, and the implementation of those practices can be overwhelming (Pressman and Wildavsky, 1984). Szulanski (1996), in his investigation on how companies can make use of best practices, documents the impediments that individuals and organizations face in identifying and implementing best practices. Pfeffer and Sutton (2000) identified several barriers to implementing best practices in productivity improvement, including the powerful knowing–doing gap.

An example of an attempt to explore best practices in a field of knowledge is presented by Bourque et al. (2002) in their exploration of fundamental principles of software engineering. Bourque et al. characterize their search for core concepts in the software engineering field as a journey geared toward improving the practice in that field. The research project presented in this paper has a similar aim: to help improve the practice of system dynamics modeling by identifying premises that can be used as powerful guidelines for improved practice.

The practices presented in this paper result from the thought processes of the group of experts who participated in this project. The results reflect what these system dynamics experts consider best practice; the best practices identified here can be followed as rules of practice to improve the level of proficiency in the development of system dynamics models. Conceivably, following the advice contained in the best-practice statements presented here—that is, following these rules related to model building—will improve the practice of modeling. Such improvements will, over time, translate into more accurate models and model-based insights and improved interventions.

Best practices in system dynamics modeling

In the system dynamics literature, there are no explicit texts about best practices. However, this literature brings together a number of implicit examples related to the concept of best practices from different threads, starting with the earliest work done in Industrial Dynamics (Forrester, 1963) and World Dynamics (Forrester, 1973) and later in collections of papers such as Elements of the System Dynamics Method (Randers, 1980), Modeling for Management (Richardson, 1997), and Modeling for Learning Organizations (Morecroft and Sterman, 1994). Specific studies provide insight on what practices are currently used, such as in Benchmarking the System Dynamics Community (Scholl, 1995), or indicate where in the literature we can find them, such as in Desert Island Dynamics: An Annotated Survey of the Essential System Dynamics Literature (Sastry and Sterman, 1993). Moreover, textbooks such as Introduction to System Dynamics Modeling (Richardson and Pugh, 1981) and Business Dynamics (Sterman, 2000) have selected, and implicitly reflect, the practices their authors consider “best.” These textbooks capture many, if not all, of the best practices in the field. The work cited here is just a small sample of the work developed by experienced system dynamics practitioners over the past 40 years.

The aim of this paper is to recognize that, although much has already been published and revealed about the system dynamics modeling process, sometimes making sense of all of the material is difficult, especially for novices in the field. This research recognizes that “there is little consensus about good modeling practice, no commonly held set of rules and standards to guide the modeling community” (Scholl, 1992, p. 263) and that something can be done about it.

This paper addresses the following questions: What is the best way to undertake system dynamics modeling? What specific core activities are essential for exemplary action during the different stages of the modeling process? What are the most important practices during the different stages of the modeling process?

The set of practices identified in this paper is intended to be independent of the type of system modeled, the tool used to develop the model, the purpose of the model, and the type of practitioner. We define practices that meet these criteria as core practices. We believe that, despite the accomplishments of many talented individuals in the field, we have yet to identify the core practices of system dynamics modeling (Barlas, 1995; Scholl, 1992). This research attempts to contribute to the ongoing definition of that core.

This paper presents the results of our research in two parts. First, a descriptive section details the data obtained. These data represent what experts in the field consider best practice. Then, a prescriptive section follows with a synthesized—and somewhat modified—wording of the experts' statements. This step was taken to make the result of the project more concise and usable. In this section, we consolidate the findings and present them in a way that we think can help system dynamics modelers, from novices to experts, get more consistent results and rule out variability in the process as a source of problems in the results they find. Our purpose is to present a non-comprehensive set of rules that can partially define the core of what system dynamicists do in a typical system dynamics modeling process. Table 2 lists examples (on the left) of the kinds of statements about best practice initially made by the group of experts who participated in the study, together with examples (on the right) of the modified summary rule-like statements we constructed to capture the sense of the different original statements. Our goal was to arrive at clear, concise, well-grounded statements like those on the right. The results of this research and the comments expressed by the panel of experts suggest that, as proposed by work by Bourdieu (1977), the development of a theory of practice in system dynamics modeling seems to be warranted.

Table 2. Sample of descriptive and prescriptive parts of the paper
Stage of system dynamics modeling processOriginal statements (raw data)Rule-like statements (prescriptive)
In exemplary problem identification and definition, you should …Listen carefully to client storiesSpeak with and listen carefully and reflectively to problem owners (clients) to identify and understand the problem
 Let the most senior client say “what brought us together” 
 Talk and listen reflectively to problem owners (clients) 
 Make sure you understand the client's problem 
 Ask client sufficient questions—avoid giving premature answers 
 Check whether (dis)agreement on the problem exists (when you are working with more than one person) 
 Clarify purpose (e.g., strategy/policy, theory building, education, training)Explicitly clarify and state the purpose of the modeling effort (e.g., strategy development, policy analysis, theory building, education, training)
In exemplary system conceptualization, you should …Avoid rigid separation of identification/conceptualization/formalization stagesApproach the conceptualization process creatively and from different angles, avoiding a rigid separation of the identification and conceptualization stages
 Approach conceptualization from different angles like a new creation 
 Recognize that conceptualization is creative—there are no recipes 
 Discuss the dynamic hypothesis with a study teamGenerate a dialogue with the problem's owners that focuses on their mental models and on dynamic hypotheses
 Engage in conversations around conceptual building blocks
 Elicit client's mental models 

Approach

Method and participants

We used a multi-method approach that included interviews; a Web-based participation meeting (Rohrbaugh, 2000); statistical analysis of the collected data; and a facilitated, face-to-face discussion meeting of the results and future research opportunities.

We decided to convene a group of experts for this study because experts embody cultural legitimacy and technical rationality (Demortain, 2008). In addition, like Mumpower and Stewart (1996, p. 193), we defined experts as those who are regarded as such by others within a certain field of knowledge or activity. The group of experts that participated in this investigation was composed of presidents of the System Dynamics Society and winners of awards from the Society3 (i.e., the Jay W. Forrester Award, the Lifetime Achievement Award, and the Lifetime Service Recognition Award). The group was chosen to include individuals with the highest, uncontroversial levels of recognition in the system dynamics field.

The number of experts invited was 27; of these, two declined to participate because of time constraints. The participation level in the study was 80 % (20 out of the 25 experts). The levels of participation fluctuated throughout the different stages of the process: participation was the highest during elicitation, followed by the prioritization segment, and then by clustering. The main professional activity of the participants in the study is distributed as follows: 16 experts (64%) were identified as academics only, five experts (20%) as practitioners only, and four experts (16%) as both.

Design

Because “there are many ways in which system dynamicists represent the widely practiced, although informally implemented, modeling heuristics they use” (Saeed, 1992, p. 251), the framework used in this study was determined on the basis of the existing literature and on suggestions from a group of experts. Table 3 presents three illustrations of how the system dynamics modeling process is described in the literature and how these descriptions compare with the framework used in this study—a modification of Richardson and Pugh's (1981, pp. 16–17) seven-stage framework with one additional stage.4

Table 3. Approaches to the system dynamics modeling process—Stages of system dynamics modeling processThumbnail image of

The overview of the system dynamics modeling approach presented in Figure 2 contains two outstanding characteristics: (1) it is depicted as a cycling, iterative process; and (2) it explicitly presents the key products of the process as integral parts of it (italicized and underlined in the figure): understanding of the model and understanding of the problem and the system. In a typical system dynamics study, “the model is a means to an end, and that end is understanding”5 (Richardson and Pugh, 1981, p. 16). Any system dynamics modeling effort should have as its goal increased understanding of the problem under study and of the system in which it is taking place.

Figure 2.

Overview of the system dynamics modeling approach (adapted from Richardson and Pugh, 1981, Figure 1.11, p. 17)

The second step in this research project, a Web-wide participation meeting (see Figure 3), had three parts: (1) idea elicitation, (2) idea clustering, and (3) idea prioritization. The three parts were consecutive and designed to generate the highest participation possible. We conducted two meetings each lasting six weeks. The first meeting covered the initial three stages of the modeling process framework that we used; then, after a recess period, a second meeting was conducted to address the final three stages.

Figure 3.

Description of the study method

Following studies by Rohrbaugh (2000), the meetings were facilitated by 404 Tech Support (LLC), which administered the Web-wide participation pages that were used for this study. In the first part—idea elicitation—the participants were invited to list ideas related to questions posted on the Web site for the meeting.

We used six elicitation questions:

  1. If you were offering advice on the best way to undertake system dynamics modeling, what specific core activities would you say are essential for exemplary PROBLEM IDENTIFICATION AND DEFINITION? In this area, what are the most important practices of modeling work?
  2. If you were offering advice on the best way to undertake system dynamics modeling, what specific core activities would you say are essential for exemplary SYSTEM CONCEPTUALIZATION? In this area, what are the most important practices of modeling work?
  3. If you were offering advice on the best way to undertake system dynamics modeling, what specific core activities would you say are essential for exemplary MODEL FORMULATION? In this area, what are the most important practices of modeling work?
  4. If you were offering advice on the best way to undertake system dynamics modeling, what specific core activities would you say are essential for exemplary MODEL TESTING AND EVALUATION? In this area, what are the most important practices of modeling work?
  5. If you were offering advice on the best way to undertake system dynamics modeling, what specific core activities would you say are essential for exemplary MODEL USE, IMPLEMENTATION, AND DISSEMINATION? In this area, what are the most important practices of modeling work?
  6. If you were offering advice on the best way to undertake system dynamics modeling, what specific core activities would you say are essential for exemplary DESIGN OF LEARNING STRATEGY/INFRASTRUCTURE? In this area, what are the most important practices of modeling work?

After a 2-week period of idea generation, participants classified the ideas generated into categories on the basis of similarity. We then compared the individually generated categorized clusters of ideas to identify the ideas that were considered by the participants, as a group, to belong together.6

In the third part, idea prioritization, participants assigned priority scores to the clusters of ideas according to the relative importance of each one as essential for the particular area covered. To complete this task, participants received the next set of instructions.

Instructions for participants: After clicking on the button “Prioritize Categories,” you will see X categories with one to ten ideas listed in distinct clusters. At first, all X categories are shown with 100 points as equally important, but you may believe that some categories of best practices may be more or less important in this specific area. You may raise or lower the 100 points for each category as you prefer: a category with 1000 points would be interpreted as a more important best practice by ten times than a category with 100 points. Any time you click on a “Sort” button, your screen will be refreshed with all the categories reordered by your changes. The full set of X ideas is displayed below.

Each participant assigned points to the different clusters; the sum of these points was used to normalize the ratings. Then, the normalized ratings were combined and weighted to obtain the total assessment for each cluster. Four thresholds were chosen for the selection process: highest importance, high importance, average importance, and low importance.7

Results

We present the results of the study in two categories, following the naming conventions used by Meadows et al. (1982, p. 270) in their lessons about the modeling process: This is the way modeling should be: high agreement best-practice statements and Let all the flowers flourish: low agreement best-practice statements.

This is the way modeling should be: high agreement best practices

Tables 4-9 present the results of the study, sorted by relative importance and phrased in language we devised to capture the sense of the different statements contributed by the panel of experts. (Tables A–F in the online supporting information contain the original wording of the statements.) These tables are part of what constitutes best practice in system dynamics modeling, as expressed by the group of experts from the System Dynamics Society who participated in the study.

Table 4. Best-practice statements in problem identification and definition
ImportanceSummarized statements
HighestElicit and thoroughly understand the client's identified problem
Clarify the purpose of the work
Elicit reference modes—graphs over time—of all variables clients see as central to the problem
Ask what is generating the current problem behavior of key variables
Formulate a dynamic hypothesis—an initial concise overview of the feedback structures believed to be responsible for the problem behavior
HighClearly identify the clients for the work
Identify and engage key stakeholders
Clarify the symptoms that initiated the modeling proposal
Agree on the time horizon for the model and settle on the appropriate time unit
Identify desirable and undesirable future behaviors, over time, of key variables
AverageVerify whether problem identified by the client is suitable for system dynamics study
Form an appropriate study team consisting of technical experts and system participants
Generate a concise and specific problem statement, identifying a clear dynamic feedback phenomenon
Identify and set clear expectations for the duration and budget of the study
Identify all available data sources and base the study on these data
Table 5. Best-practice statements in system conceptualization
ImportanceSummarized statements
HighestApproach system conceptualization creatively, from different perspectives
Elicit clients' mental models to help develop the building blocks of the dynamic hypothesis
Identify important accumulations (stocks) early in conceptualization
HighStrive for an endogenous dynamic hypothesis
Make sure the boundary of the dynamic hypothesis is large enough to enable the endogenous point of view
Identify key variables representing problematic behavior
AverageMake sure that each variable is measurable, at least in principle
Make appropriate use of all relevant data
Document the process of conceptualization
Table 6. Best-practice statements in model formulation
ImportanceSummarized statements
HighestStart small and simple; build toward complexity and completeness
Assure dimensional consistency from the beginning and use units to facilitate formulation
Ensure that all equations make common sense and all parameters have real-life meaning
HighManage model complexity to fit the problem and the audience; strive for clarity and simplicity
Simulate early and often in the formulation stage; test extensively along the way
Involve the clients in discussions of model behavior and structure
AverageDevelop a small, aggregate prototype model to focus the work and test the dynamic hypothesis
Avoid cascading (i.e., chained, nested) graphical functions and other complex equation formulations
Strive to capture the actual bounded rationality of actors in the system
Build a model structure that reflects the actual limited rationality and available information in the real system
Acquire experience with many types of models from the literature, and seek out expert guidance
Start with a small model (a one-page model)
Table 7. Best-practice statements in model testing and evaluation
ImportanceSummarized statements
HighestEnsure that model behavior is consistent with historical or reference behavior modes
HighEnsure that each equation is robust under extreme conditions, and verify that the entire model responds appropriately to extreme, but plausible, shocks and values
Analyze unexpected results to identify their causes
Test each equation for logical plausibility
AverageEnsure that all variables and parameters have real-world meaning
Engage the client in evaluating model structure and behavior
Use partial-model tests to refine structure and parameter values
Discover high-leverage parameters and structure that strongly influence model behavior
Table 8. Best-practice statements in model use, implementation, and dissemination
ImportanceSummarized statements
HighestMake sure the entire modeling exercise revolves around the problems of concern to the client and audience
HighTell clear, coherent stories of model behavior using simple language and pictures of system structure
AverageCapture policy insights in simple phrases or lessons (“chunks”) that clients can grasp intuitively
Assist clients with plans for implementation of model-based policy recommendations
Involve clients throughout the entire modeling process
Focus on implementation from the start, and target implementable policy options
Phrase reports and documentation in the language of the clients and audience
Involve clients in model-based policy testing
Table 9. Best-practice statements in design of learning strategy/infrastructure
ImportanceSummarized statements
HighestInvolve clients in telling model-based system stories and illustrate the stories appropriately using simplified causal diagrams
HighProvide clear guidance for the use of flight simulators and model-based games, carefully debriefing participants after all learning exercises to ensure that they understand model-based insights
Explain counterintuitive model behavior to contribute real system insights
AverageDevelop simplified, small models focused on selected issues and interesting patterns of behavior
Design the learning engagement so that those who must make the decisions will learn from the modeling process and model-based insights
Secure top management support and involvement
Carefully consider who needs to learn from the model-based study and how they will learn

One advantage in the way the results of this investigation are presented is that they are accessible in a concise form; the reviewer need not go through thousands of pages in books and papers to identify them. In addition, because these results originate from a group of experts, rather than a single expert, they are more representative of best practices than those found in any single text. Further, the group of experts who participated in the study shares the perception of these practices as being the best in the field.

Of course, a compendium of best practices cannot entirely substitute for textbook reviews. However, it can become a natural complement to textbook-type materials for the practitioner: a quick reference for system dynamics practice. In fact, this guide would be most useful to those who have read the textbooks already and are familiar with the way in which system dynamics literature discusses the necessary steps in the modeling process. Even for novice practitioners, however, the compendium offers a succinct guide that can direct them to textbooks and other resources to increase their knowledge. The compendium of best practices might be of the least value if used as a stand-alone resource. Table 10 presents a summary of the best-practice statements that have the highest importance and high agreement.

Table 10. Summary of best-practice statements with highest importance and high agreement
StageRule-like statements (prescriptive)
In exemplary problem identification and definition, you should …Speak with and listen carefully and reflectively to problem owners (clients) to identify and understand the problem
Explicitly clarify and state the purpose of the modeling effort (e.g., strategy development, policy analysis, theory building, education, training)
Identify the reference modes of central processes to be studied for the purpose of clarifying expectations of future behavior
Ask why the current behavior of key variables generated is the way it presently is, and what is causing it
Formulate a dynamic hypothesis
In exemplary system conceptualization, you should …Approach the conceptualization process creatively and from different angles, avoiding a rigid separation of the identification and conceptualization stages
Generate a dialogue with the problem's owners that focuses on their mental models and on dynamic hypotheses
Identify the critical accumulations (stocks) that describe the system, making certain that their names are nouns — not verbs or action phrases
In exemplary model formulation, you should …Develop the structure through a series of simple to more comprehensive models, adding detail as needed to improve realism and show policy impacts
Formulate equations that make sense, carefully supporting variables and parameters with data or experience and making certain that all have real-life meanings
Make sure that the model always exhibits dimensional consistency
In exemplary model testing and evaluation, you should …Compare simulated behavior patterns with real behavior (data) using statistical measures of pattern fit, not point-by-point fit
In exemplary model use, implementation, and dissemination, you should …Make sure that the entire modeling process revolves around the problems of concern of the audience (problem owner, client)
In exemplary design of learning strategy/infrastructure, you should …Use simplified causal-loop diagrams to tell system stories in a variety of ways, rather than relying on the model to tell its own story

Let all the flowers flourish: low agreement best practices

Table 1 (at the beginning of the paper) provides a summary of the results of the study for each stage of the model-building process. The table shows that two out of every three practices (126 of 198) originally contributed by members of the panel of experts are judged by their colleagues (the rest of the panel) as having low importance.

The second row in Table 1 shows the number of statements in each modeling stage that were judged by the panel to be of highest importance, but about which panel members disagreed. These highest-importance/low-agreement statements may provide important insights.8 Meadows et al. (1982), in their analysis of the contributions of global models to the advancement of the modeling practice, use disagreement among modelers as a source of knowledge and understanding. They conclude, while studying the 11 general disagreements they found, that “methodological problems of global modelers are common to all social-system modelers” (Meadows et al., 1982, p. 269). These highest-importance/low-agreement best-practice statements are an interesting finding of this study because they represent a great opportunity to expand our understanding of the different theories, methods, and procedures used in the field by highlighting possible conceptual bifurcations (see Table 11; see also Table G in the online supporting information for the original wording of the statements). They can become fertile soil to generate distinct threads of thinking within the field and can enhance the field if, as Meadows et al. (1982) suggest, we let all flowers flourish.

Table 11. Best-practice statements with highest importance and low agreement
StageRule-like statements (prescriptive)
Problem identification and definitionIn exemplary problem identification and definition, two competing approaches emerged:
model the system (class) vs. model the problem (case)
Because in this stage you should …Model the class to which the case belongs, not the case at hand
Model the particular case being studied and identify the class of system to which it belongs
System conceptualizationIn exemplary system conceptualization, two competing approaches emerged:
Initiate conceptualization with causal-loop diagrams vs.
Initiate conceptualization with stock-and-flow diagrams
Because in this stage you should …Use causal-loop diagrams to describe the key feedback structures in the dynamic hypothesis (or set of dynamic hypotheses)
Identify the critical accumulations (stocks) that describe the system and the system boundary iteratively by sketching causal-loop diagrams
Identify the critical accumulations (stocks) that describe the system, the flows that change them and the influences on the flows
Model formulationIn exemplary model formulation, two competing approaches emerged:
(1) Start small and add complexity as necessary, always having a running model vs.
Formulate big chunks of complexity at a time. And
(2) Use realistic operational thinking to add/modify structure vs.
Use extreme conditions thinking to add/modify structure
Because in this stage you should …Begin with a small core feedback structure, simulate it, and analyze it. Refine it, as appropriate. Then add structure operationally, simulate and analyze, and so on iteratively, never straying too far from a simulatable model
Keep in mind extreme condition tests while writing model equations. Simulate the model under different extreme conditions and modify it if it fails to respond appropriately
Model testing and evaluationIn this stage you should…Test and validate as an iterative process
Assure dimensional consistency in all equations
Ask: Do I understand the behavior?
Model use, implementation and disseminationIn this stage you should…Study important, dynamic problems as opposed to open, static, short-term problems.
 
Design of learning strategy/infrastructureIn design of learning strategy/infrastructure, two competing approaches emerged:
Emphasize and use the model built as source of learning and insights vs.
Emphasize and use the modeling process as source of learning and insights
Because in this stage you should …Use flight simulators and well-designed interactive learning environments to motivate and educate clients
Emphasize the learning process and outcomes more than the model itself

Discussion

In this research project, the panel of experts identified 27 practices as having the highest importance when compared with the other practices identified. Tables 12 and 13 provide a summary of these 27 practices, organized by stage of the model-building process and by level of agreement. We will use Tables 12 and 13 as a framework to discuss the practices identified in this investigation.

Table 12. Summary of best-practice statements with highest importance (I)
StageRule-like statements (prescriptive)Agreement level
In exemplary problem identification and definition, you should …Speak with and listen carefully and reflectively to problem owners (clients) to identify and understand the problem.High
Explicitly clarify and state the purpose of the modeling effort (e.g., strategy development, policy analysis, theory building, education, training)
Identify the reference modes of central processes to be studied for the purpose of clarifying expectations of future behavior
Ask why the current behavior of key variables generated is the way it presently is, and what is causing it
Formulate a dynamic hypothesis
Model the class to which the case belongs, not the case at handLow
Model the particular case being studied and identify the class of system to which it belongs
In exemplary system conceptualization, you should …Approach the conceptualization process creatively and from different angles, avoiding a rigid separation of the identification and conceptualization stagesHigh
Generate a dialogue with the problem's owners that focuses on their mental models and on dynamic hypotheses
Identify the critical accumulations (stocks) that describe the system, making certain that their names are nouns—not verbs or action phrases
Use causal-loop diagrams to describe the key feedback structures in the dynamic hypothesis (or set of dynamic hypotheses)Low
Identify the critical accumulations (stocks) that describe the system and the system boundary iteratively by sketching causal-loop diagrams
Identify the critical accumulations (stocks) that describe the system, the flows that change them, and the influences on the flows
In exemplary model formulation, you should …Develop the structure through a series of simple to more comprehensive models, adding detail as needed to improve realism and show policy impactsHigh
Formulate equations that make sense, carefully supporting variables and parameters with data or experience and making certain that all have real-life meanings
Make sure that the model always exhibits dimensional consistency
Begin with a small core feedback structure, simulate it, and analyze it. Refine it, as appropriate. Then add structure operationally, simulate and analyze, and so on iteratively, never straying too far from a simulatable model.Low
Keep in mind extreme condition tests while writing model equations. Simulate the model under different extreme conditions and modify it if it fails to respond appropriately.
Table 13. Summary of best-practice statements with highest importance (II)
StageRule-like statements (prescriptive)Agreement level
In exemplary model testing and evaluation, you should …Compare simulated behavior patterns with real behavior (data) using statistical measures of pattern fit, not point-by-point fitHigh
Test and validate as an iterative processLow
Assure dimensional consistency in all equations
Ask: Do I understand the behavior?
In exemplary model use, implementation, and dissemination, you should …Make sure that the entire modeling process revolves around the problems of concern of the audience (problem owner, client)High
Study important, dynamic problems as opposed to open, static, short-term problemsLow
In exemplary design of learning strategy/ infrastructure, you should …Use simplified causal-loop diagrams to tell system stories in a variety of ways, rather than relying on the model to tell its own storyHigh
Use flight simulators and well-designed interactive learning environments to motivate and educate clientsLow
Emphasize the learning process and outcomes more than the model itself

In problem identification and definition, there is high agreement regarding the importance of (1) involving problem owners in the modeling process, (2) clearly identifying the purpose of the modeling, (3) formulating a dynamic hypothesis, and (4) clearly articulating the dynamics of the problem using current and expected patterns of behavior. A high level of agreement in these practices reflects a well-established set of processes in the field, leading to a shared understanding of how to identify and define a dynamic problem. However, there seems to be a lower level of agreement regarding where the focus of the modeling effort should be; i.e., whether to model the system (class of case being studied) or the problem (case). This lower agreement was also captured by Meadows et al. (1982, p. 270) as a tension among global modelers about whether to answer well-defined questions (case) versus representing many aspects of a system (class). In the case of the group of experts in this study, the lower level of agreement regarding model focus may be related to the experts' different backgrounds and professional orientations. Academics tend to favor drawing generic lessons from their modeling efforts (Forrester, 1963, 1973), while practitioners are more interested in the specifics of the cases they study (e.g. Lyneis et al., 2001; Graham and Ariza, 2003).

In system conceptualization, there is a high level of agreement about using different approaches creatively to gain a clear understanding of the mental models of the problem's owners. Such an approach leverages the power of thinking in terms of dynamic hypotheses and identifying the critical stocks that describe the system. The high level of agreement about conceptualizing, both by focusing on how the agents in the system think about the problem and by using dynamic hypotheses as a guiding framework, reflects a shared understanding of the importance of a multi-level grounding of model development using empirical evidence (mental models of system participants) and theoretical thinking (dynamic hypothesis). In this stage, even though there is good agreement about starting with major stock variables, there seems to be a lower level of agreement about whether to iteratively use a casual-loop diagram approach or a stock-and-flow approach to conceptualize. This finding may reflect a methodological difference in approaches; there are solid rationales for both. Conceptualizing by using causal-loop thinking favors an endogenous perspective (the unit of focus is the feedback loop), while conceptualizing using a stock-and-flow approach favors the identification of critical system elements that provide the system with inertia and determine its dynamics. Both approaches, when combined and used iteratively, may yield the best results in system conceptualization.

In model formulation, there is a high level of agreement about using evidence (i.e., data and subject matter experts' experience), ensuring that all equations have real-life meaning, and ensuring dimensional consistency. In addition, both creating simple-to-more-comprehensive models and only adding detail to improve realism and articulate policy options are considered the best ways to formulate the model. These practices describe model formulation in broad terms, which probably contributes to the identified level of importance. Other, more-specific practices were also identified by the experts at levels of high or average importance (see Table 6).

Examples of recent efforts to provide tools to help modelers implement agreed-upon practices in an automated way (e.g., identifying cascading graphical functions, identifying excessively complex formulations) and to specify reporting guidelines include work by Rahmandad and Sterman (2012) and Martinez-Moyano (2012). However, two sets of competing approaches regarding how to formulate models have emerged. In the first approach, starting small and adding complexity as necessary and continuously having a running model is contrasted with formulating big chunks of complexity at a time. This first disagreement seems incompatible with the best practice of creating simple-to-more-comprehensive models. However, the difference between the two seems to be the emphasis on never straying too far from a simulatable model. In the practice with high agreement, the criterion for adding complexity is to improve realism and to be able to show policy implications without constraining the amount of complexity added at one time. In the practice with low agreement, there is a clear constraint on how much complexity to add. The lower agreement suggests to us that there are groups of experts who formulate piece by piece, always trying to have a running model at hand, and other groups who prefer to formulate in big chunks and who are not as concerned about having continuously running prototypes.

The second disagreement centers on using realistic, operational thinking versus extreme-conditions thinking to add/modify the model structure. Some experts believe that using extreme-condition tests is crucial in model formulation to ensure that all equations make sense and are constrained by real-life behavior; others believe that using extreme-conditions thinking might actually push the formulation into a regime that will never arise in reality. An alternative explanation might be that, for some experts, this practice belongs in the model testing and evaluation stage of the process (see Table 21–4, Sterman, 2000, pp. 859–861).

Although “behavior reproduction tests cannot prove a model is correct or reliable” (Sterman, 2000, p. 879), in model testing and evaluation, comparing simulated behavior patterns with real behavior (data) using statistical measures of pattern fit, not point-by-point fit, was the only high-agreement practice identified by the panel of experts. This practice is a powerful mechanism to uncover flaws in the model and to add/decrease complexity or modify the structure of the model. In addition, a low level of agreement was found regarding the iterative approach to testing and confidence building. This finding may be attributable to differences in experts' approaches to increasing confidence in models. Some practitioners seem to prefer incremental actions; others seem to prefer tests that are more radical.

Although achieving dimensional consistency is considered a high-agreement best practice in model formulation (previous stage of the modeling process), in model testing and evaluation, it generated a lower level of agreement. This may be attributable to experts' belief that this practice actually belongs in the previous modeling stage. This finding adds a new temporal dimension to the study of best practices; it seems that although what model developers do is important, when they do it also seems to be critical. Alternatively, low agreement regarding whether dimensional consistency is an important practice in model testing and evaluation might signal that this practice is considered more a formulation principle (Forrester, 1968, p. 6–2; Richardson and Pugh, 1981, p. 264) than a test to be performed after the formulation process is completed (Forrester and Senge, 1980; Sterman, 2000). An additional potential source of disagreement is the relatively artificial discretization of the modeling process used in this study (as depicted in Table 3).

In model use, implementation, and dissemination, although making sure that the modeling process is centered on the concerns of the problem owners was the only practice found to be of highest importance—generating a high level of agreement among the members of the panel—having the clients' point of view and interest as the focus of the model was identified as being of high and average importance. This clear agreement in the role of the client in the process is indicative of the general orientation toward relevance and actionable results in system dynamics modeling. The lower level of agreement about making sure that the problem under study is dynamic (as opposed to static) likely reflects a timing disagreement; experts may believe that, at this point in the model-building process, it is too late to make sure that the right type of problem is being studied.

In design of learning strategy/infrastructure, the panel of experts determined, with a high level of agreement, that using simplified causal-loop diagrams to tell system stories in a variety of ways, rather than relying on the model to tell its own story, was of the highest importance. This single practice captures the idea that communicating the findings of the modeling process should be deliberative and purposeful; if it is allowed to be emergent, critical insights may be overlooked. Although there is agreement in terms of purposefully designing the learning process, there is disagreement with respect to what topics are most important to emphasize in this process. Some experts believe that emphasizing and using the model itself as source of learning and insights is of the outmost importance, and others favor emphasizing and using the modeling process to provide insights. These competing approaches could open a dialogue about how to communicate to large audiences the insights generated by our modeling studies.

Reflections of the panel of experts

The final activity was a facilitated meeting in which the experts discussed the findings of the study and the possibilities for future research in the system dynamics community. This facilitated dialogue took place in July 2002 during the 20th International Conference of the System Dynamics Society in Palermo, Italy. Fourteen members of the group participated in the facilitated meeting.9

There was consensus among the experts regarding the importance of modeling as a way to understand reality and develop useful interventions in complex systems. The experts also identified as desirable two types of best practices: quality control practices (used to avoid problems) and innovative practices (used to improve modeling). Finally, the experts suggested additional ways to study and identify the state of the art in system dynamics modeling. Suggested alternative methods include in-practice observation, context-dependent exploration, and laboratory-based problem solving by experts using models.

Conclusions

Some may consider the identification of best practices in system dynamics by a group of highly recognized experts in the field as a search for an ultimate truth that will inevitably cut off all debate, shutting down further development of the field. We disagree. Our stance is aligned with that of Hines (2002), who believes that “a huge amount of exploring remains to be done, filled with importance, drama and fun” (p. 1). The purpose of this research is to help illuminate a path that can lead to many different truths that a community, through its experts, recognizes as valuable and important. Empanelling experts and giving them the opportunity to express their preferences for practice, both in a closed and in an open format, allows the many truths, embodied in current best practice, to be exposed, examined, and ultimately improved.

All of these considerations lead to the conclusion that researching best practices can be productive, and yet it remains challenging. Stimulating a dialogue about how to improve the practice in a knowledge community can be instrumental in driving improvement in the field. Furthermore, because the system dynamics field has been expanding in terms of the types of systems modeled and the number and type of practitioners, the practices used and the views about those practices have been evolving. Therefore, continued exploration and integration of best practices in the field seems warranted, creating momentum in the evolutionary process of rules change.

The highest-importance/low-agreement best-practice statements we encountered can be considered a vehicle to expand the frontiers of the field and drive improvements in the development of system dynamics models. Identifying a comprehensive list of practices and order-ranking them is not explicitly addressed in the literature, perhaps because what makes an identified practice a best practice is the way in which the personal judgment of the practitioner and the social judgment of the community come together. The interaction of the individuals and the community generates the social construction of best practices.

It is our hope that this empirical study of opinions about best practice will trigger learning processes in the system dynamics community that are based in best-practices research and action. It seems that “this is the first full turn of the scientific method exploring these data” (Graham, 2002) and that there are a variety of methods to clarify what the experts do and why. Involving more expert practitioners in future studies has the potential of yielding a more comprehensive view of the status of practices considered best practice and their use in the field. Furthermore, the precise definition of expert practitioner should be of interest to the community as it may help provide incentives for improvement through example and mentoring.

Studying in greater depth the low-agreement best-practice statements—their nature, implications, and possible avenues for expansion and improvement of the field—offers a promising area for development. Finally, closely linking the results of this work with published literature in the field may also help practitioners improve and provide additional avenues for them to identify the wealth of actionable knowledge in the field.

Acknowledgements

We would like to acknowledge the invaluable contribution of Dr. John Rohrbaugh to this paper. Also, we thank all the individuals who participated as subject matter experts in this investigation. Finally, we would like to thank the anonymous reviewers and the editors who helped strengthen this paper with their insightful comments and suggestions.

  1. 1

    Comments to the Panel of Experts during the Special Plenary Session on Best Practices in System Dynamics Modeling in Palermo, Italy (2002).

  2. 2

    Fernandez et al. (2001) and Morgan (2001) include the notion of evolution in best practices research by analyzing how the adoption of a particular process or best practice reshapes the practice itself over time.

  3. 3

    Up to 2002.

  4. 4

    The experts who participated in the initial interviews are: Dr. David F. Andersen, Dr. George P. Richardson, and Dr. Barry Richmond. The added stage—design of learning strategy/infrastructure—was an original idea and contribution of Dr Barry Richmond.

  5. 5

    That is, understanding the system and understanding the model as a means of being able to intervene in the system and change the way behavior is progressing.

  6. 6

    We used a 75% agreement threshold.

  7. 7

    For details on the computation used, and for specifics about the thresholds, see Martinez-Moyano (2005).

  8. 8

    A highest-importance/low-agreement statement was identified when a cluster of practices was ranked highest in priority and highest in dispersion of priorities.

  9. 9

    This number constituted 100% of the study participants who were present at the conference.

Biographies

  • Ignacio J. Martinez-Moyano is Computational Social Scientist at the Decision and Information Sciences Division, Argonne National Laboratory, and Research Fellow at the Computation Institute, The University of Chicago. He received his Ph.D. from the University at Albany, State University of New York. He studies how rules change in organizations; in particular, how human judgment and action shape, and are shaped by, such change. He has published in academic journals such as Organization Science, the Journal of Public Administration Research and Theory, and the System Dynamics Review.

  • George P. Richardson is O'Leary Professor of Public Administration and Policy in the Rockefeller College of Public Affairs and Policy at the University at Albany, and affiliated Professor of Informatics in the College of Computing and Information. He is the author of Introduction to System Dynamics Modeling with DYNAMO (1981) and Feedback Thought in Social Science and Systems Theory (1991, 1999), both of which were honored with the System Dynamics Society's Forrester Award, and the edited two-volume collection Modeling for Management: Simulation in Support of Systems Thinking (1996).

Ancillary