Embedding communication concepts in forecasting training increases students’ understanding of ecological uncertainty

: Communicating and interpreting uncertainty in ecological model predictions is notoriously challenging, motivating the need for new educational tools which introduce ecology students to core concepts in uncertainty communication. Ecological forecasting, an emerging approach to estimate future states of ecological systems with uncertainty, provides a relevant and engaging framework for introducing uncertainty communication to undergraduate students, as forecasts can be used as decision support tools for addressing real-world ecological problems and are inherently uncertain. To provide critical training on uncertainty communication and introduce undergraduate students to the use of ecological forecasts for guiding decision-making, we developed a hands-on teaching module within the Macrosystems EDDIE (Environmental Data-Driven Inquiry and Exploration; MacrosystemsEDDIE.org) educational program. Our module used an active learning approach by embedding forecasting activities in an R Shiny application to engage introductory students in data science, ecological modeling, and forecasting without needing advanced computational or programming skills. Pre-and post-module assessment data from > 250 undergraduate ecology students indicate that the module signiﬁcantly increased students’ ability to interpret forecast visualizations with uncertainty, identify diﬀerent ways to communicate forecast uncertainty for diverse users, and correctly deﬁne ecological forecasting terms. Speciﬁcally, students were more likely to describe visual, numeric, and probabilistic methods of uncertainty communication following module completion. Students were also able


Introduction
Communicating uncertainty in ecological models is a pressing challenge across ecology, motivating the need for new educational tools to train students in understanding and interpreting uncertainty in model predictions.Uncertainty in ecological model predictions is inherent across ecological disciplines, ranging across population and community ecology models (e.g., Halpern et al. 2006, Bird et al. 2021), disease ecology models (e.g., Briggs et al. 2009, McClintock et al. 2010), landscape ecology models (e.g., Wu et al. 2006, Lechner et al. 2012), and ecosystem models (e.g., Link et al. 2012, Melbourne-Thomas et al. 2012).Sources of uncertainty in ecological models include uncertainty in model parameter estimates, initial conditions, and the underlying processes being modeled (Dietze 2017).Combined together, these sources of uncertainty can have important implications for interpreting model results, as well as their utility in decision-making (e.g., Berthet et al. 2016, Cheong et al. 2016).However, uncertainty is rarely communicated or is communicated poorly (Boukhelifa andDuke 2009, Hullman 2020), hindering the use of model output for both advancing ecological understanding and decision-making (Joslyn andSavelli 2010, Milner-Gulland andShea 2017).This is likely because uncertainty is a difficult concept for most individuals to understand (Belia et al. 2005), as well as to mathematically quantify and represent graphically with visualizations (Spiegelhalter et al. 2011, Potter et al. 2012, Bonneau et al. 2015).Given low levels of visualization literacy in both the general and scientific population (Maltese et al. 2015), educational tools to improve communication of ecological model uncertainty are critically needed.
Ecological forecasting provides a powerful framework for teaching students uncertainty communication and data science skills, which are increasingly needed for 21 st century careers (Rieley, 2018, Vought andDroegemeier, 2020).Ecological forecasts, which are future, out-of-sample model predictions of ecological variables with quantified uncertainty (Table 1), can serve as useful decision support tools for a variety of users (Tulloch et al. 2020, Bodner et al. 2021).Because of the utility of forecasts in both informing decision-making and the testing of ecological theory (Dietze et al. 2018, Lewis et al. 2022a, Carey et al. 2022), ecological forecasting is a rapidly growing sub-field of ecology (Lewis et al. 2022b).
Many near-term (day to decade ahead) ecological forecasts are developed using the iterative forecasting cycle (Lewis et al. 2022b), which has the potential to teach students foundational ecological forecasting concepts (Moore et al. 2022a).The iterative, near-term forecasting cycle consists of multiple steps, which parallel the scientific method: 1) make a prediction about ecological phenomena, 2) develop a model which represents that hypothesis, 3) quantify uncertainty around predictions, 4) generate a forecast with uncertainty, 5) communicate the forecast to users, 6) assess the forecast with observations, and 7) update the forecast with new data (Dietze et al. 2018, Moore et al. 2022a).Altogether, teaching this iterative framework in ecology courses could improve student understanding of complex ecological concepts (Selutin and Lebedeva 2017), as well as uncertainty visualization skills.
Communicating and interpreting ecological forecast visualizations presents several unique challenges.First, forecasts are inherently uncertain, yet they are needed to guide environmental management decisions, making it critical to properly communicate the uncertainty associated with forecast predictions (Berthet et al. 2016).Second, while there are numerous studies on visualizing data uncertainty (Olston and Mackinlay 2002, Potter et al. 2012, Smith Mason et al. 2017, Wiggins et al. 2018), little consensus has emerged as to the best approach for visualizing forecast uncertainty for both end user comprehension and decision support.Third, it has been well-documented that different approaches to visualizing uncertainty result in varying levels of comprehension by users (Ramos et al. 2013, Cheong et al. 2016, McKenzie et al. 2016, Kinkeldey et al. 2017).Altogether, these challenges emphasize the need for thoughtful representation of uncertainty in forecasts, as well as the need for educational materials that teach students how to interpret and develop forecast visualizations for decision support applications.
Several pedagogical methods may be useful for incorporating uncertainty visualization skills into introductory ecological forecasting education.First, having students create their own visualizations has been shown to improve data visualization literacy (Huron et al. 2014, Börner et al. 2016, 2019, Alper et al. 2017).Second, teaching students how to produce a range of visualizations for the same forecast using a toolbox of different visualization styles may enable them to communicate their forecast to a broader range of users, as well as adapt their visualizations for different user needs.For example, teaching students how to communicate uncertainty in a single forecast using multiple methods (e.g., representing uncertainty with numbers, words, icons, and graphs such as maps or time series; sensu Spiegelhalter et al. 2011) can help illustrate the multitude of ways uncertainty can be visualized and build students' ability to interpret diverse forecast visualizations.Third, teaching students to communicate forecast uncertainty using thresholds which are directly meaningful for decision-making has proven utility in uncertainty communication (Kox et al. 2018).For example, communicating a forecast of the abundance of an endangered species as a forecast index (e.g., the likelihood of encountering that endangered species at a site) may be a more effective communication style for some forecast users by placing forecast output in a decision-making context (see Table 1 for definitions).Fourth, emphasizing the importance of identifying forecast users and specifically the decisions which could be made with forecasts could increase the relevance of ecological forecasting for students.Presenting ecological concepts in culturally and societally relevant contexts is known to stimulate student engagement (Cid and Pouyat 2013, Vance-Chalcraft and Jelks 2022, Henri et al. 2022), and can lead to more collaborative and effective research and management broadly within the scientific community (Armitage et al. 2009, Cvitanovic et al. 2013).
In addition to the pedagogical approaches above, integrating the concepts of decision science (e.g., through structured decision-making or decision use cases, see Table 1 for definitions; Clemen andReilly 2004, Gregory et al. 2012) may help students better understand the needs of different forecast users, and correspondingly lead to improved forecast visualizations.Current ecological forecasting teaching materials have largely been methodology-focused, omitting application and communication components (Willson et al. 2022).This focus on methods skill-building, while very valuable, may fail to engage introductory students who have yet to master the computational and quantitative skills needed for forecast development.
To introduce students to key concepts in uncertainty visualization and communication in the context of using near-term ecological forecasts for real-world decision-making, we developed a 3-hour teaching module, "Using Ecological Forecasts to Guide Decision-Making," as part of the Macrosystems EDDIE (Environmental Data-Driven Inquiry and Exploration; MacrosystemsEDDIE.org) program.The module entailed a short introductory lecture, three scaffolded, hands-on forecasting activities embedded within an online interactive tool built using an R Shiny application (Chang et al. 2022) and discussion questions.Instructors were also provided with a pre-module student handout which included suggested readings and discussion questions to provide students with background information before beginning hands-on module activities.To test the effectiveness of our interactive teaching module on students' ability to learn uncertainty communication and foundational ecological forecasting concepts within a decision support framework, we conducted pre-and post-module assessment surveys.We analyzed the student assessment data to determine how completion of the module affected: 1) students' ability to interpret and communicate uncertainty in forecast visualizations, and 2) students' understanding of foundational ecological forecasting concepts.

Methods Module Overview
We designed Macrosystems EDDIE (Environmental Data-Driven Inquiry and Exploration; Macrosystem-sEDDIE.org)Module 8 "Using Ecological Forecasts to Guide Decision-Making" to teach students uncertainty communication and foundational ecological forecasting concepts within a decision support framework.This is the 8 th module in the Macrosystems EDDIE teaching module series (Carey et al. 2020, Hounshell et al. 2021, Moore et al. 2022a).Specifically, the module activities encompassed a range of decision support concepts and applications, such as structured decision-making through role-playing and identification of forecast user needs.The version of the module used for this study is archived and available for download from Woelmer et al. (2022aWoelmer et al. ( , 2022b)).All module materials are publicly available for use and are iteratively updated following user feedback; the most recent version of the module can be accessed at: https://serc.carleton.edu/eddie/teachingmaterials/modules/module8.html.Our assessment focused on measuring student understanding of uncertainty communication and foundational ecological forecasting as two important yet currently overlooked concepts within undergraduate ecology curricula (Willson et al. 2022).This module, following the Macrosystems EDDIE pedagogical framework (Carey et al. 2020), consisted of a suite of three self-contained, scaffolded activities (Activities A, B, and C) which can be adapted to meet the needs of individual lecture or laboratory classes.The three activities taught students different ways to visualize forecasts (Activity A); how uncertainty in forecast visualizations can influence decision-making (Activity B); and how to create visualizations of probabilistic ecological forecasts tailored to a specific user (Activity C).All Macrosystems EDDIE modules follow the 5E Instructional Model (Bybee et al. 2006), which uses activities to enable engagement, exploration, explanation, elaboration, and evaluation.This module, as well as other Macrosystems EDDIE modules, are primarily geared towards the undergraduate level but can also be applied in graduate-level courses (e.g., Moore et al. 2022a).
Because uncertainty interpretation and communication are not commonly integrated into undergraduate ecology education (Willson et al. 2022), we introduced students to a broad suite of methods currently applied in visualization and decision science within the module.These methods include: 1) creating one's own visualizations (Huron et al. 2014, Alper et al. 2017, Börner et al. 2016, Börner et al. 2019), 2) visualizing uncertainty in multiple ways (sensu Spiegelhalter et al. 2011), 3) using meaningful thresholds for decisionmaking (Kox et al. 2018), 4) identifying forecast users to increase engagement and relevance (Cid and Pouyat 2013, Henri et al. 2022, Vance-Chalcraft and Osborne Jelks, 2022), and 5) considering forecast user decision needs to guide visualization development (Raftery 2016).
Our module assessment (described below) focused on two learning objectives (LOs) taught throughout the module activities.The two LOs were: LO1) describe what ecological forecasts are and how they are used (Activity A, B, C); and LO2) identify different ways to represent uncertainty in a visualization (Activity A, B, C).In addition to LO1 and LO2, this module included four additional LOs for instructors: LO3) identify the components of a structured decision (Activity B); LO4) discuss how forecast uncertainty relates to decision-making (Activity A, B, C); LO5) match forecast user needs with different levels of forecasting decision support (Activity A, C); and LO6) create visualizations tailored to specific forecast users (Activity C).The activities within the module were designed to meet all six LOs, with several activities targeting multiple LOs (Appendix S1: Table S1).Our focus on LO1 and LO2 for the assessment was motivated by the importance of increasing representation of foundational ecological forecasting and uncertainty communication concepts, respectively, in undergraduate ecology curricula (Appendix S1: Table S1).

Detailed module description
The module included an introductory PowerPoint lecture, a suite of three activities embedded within an R Shiny application accessed in a web browser, and discussion questions.First, the PowerPoint presentation (˜20 minutes) introduced students to the key concepts taught in the module, including a general introduction to ecological forecasting and a case study of an ecological forecasting application with visualization examples.Instructor notes for each slide were provided, as well as an 'Introduction to R Shiny' guide for students and instructors who were not previously familiar with using R Shiny applications.
For the case study within the introductory PowerPoint lecture, students were given an example of a forecast of the future distribution of the invasive spongy moth (Lymantria dispar ) and introduced to different types of forecast users and corresponding decisions that different forecast users could make, as well as different ways of visualizing the same forecast for individual forecast users' decision use cases (Table 1).For example, a homeowner deciding whether to treat the oak trees on their property to prevent spongy moth invasion might benefit from a forecast index visualizing the percent likelihood of spongy moth colonization in a particular location.In contrast, a natural resource manager deciding where to prioritize conservation efforts of a native competitor of spongy moth might prefer a map of spongy moth densities and associated uncertainty across the region.Through the case study, students were shown a range of visualization types that can be altered to suit different decision use cases.Students were taught about how uncertainty can be represented and communicated using several methods, including numbers, words, icons, and graphs.For example, using the same forecast, uncertainty could be communicated with numbers ('22% chance of a spongy moth outbreak'), words ('low risk of spongy moth outbreak'), an icon (showing a 'traffic light' symbol indicating 'green' for low risk), or a graph (a map of the likelihood of an outbreak across a region) (Appendix S1: Figure S2).Within these four categories, students were taught how to communicate forecast output (e.g., the density of spongy moths in a given area, see Table 1 for an example), which uses output directly from a forecast model.In addition, they were taught to communicate using a forecast index, which is forecast output that is translated into an index based on some threshold which is meaningful to decision-making (e.g., the likelihood of a spongy moth outbreak; Table 1, Appendix S1: Figure S2).
Second, following the presentation, students were instructed to access the module via the R Shiny application and work through the module activities A, B, and C with a partner.R Shiny is an interactive tool built within the R coding environment that allows users to interact with complex data through a simple web browser interface (Chang et al. 2021, Kasprzak et al. 2021), increasing the ease of use.Applications developed using R Shiny have been proven effective at teaching students challenging topics in a variety of educational settings (e.g., Fawcett 2018, Moore et al. 2022a).All module activities were designed to meet one or more of the module LOs (Appendix S1: Table S1).
Within the Shiny app, students first completed Activity A, "Explore ecological forecast visualizations and decision use," in which they individually selected an ecological forecast from a curated list of current forecasting systems (Appendix S1: Table S2), answered several embedded questions about how their selected forecast is visualized and how it can be used, and then compared their answers with their partner.Through these activities, students directly addressed LO1 ('Describe what ecological forecasts are and how they are used') by analyzing forecasts and identifying forecast users and LO2 ('Identify different ways to represent uncertainty in a visualization') by analyzing how or whether their forecast visualizes uncertainty.
In Activity B, "Make decisions using an ecological forecast," students completed an in-depth case study in which they role-played as resource managers and made decisions about optimizing multiple objectives using two different forecast visualizations (Figure 2A).The use of role-playing as an active form of learning has documented success in education, especially in science education (Howes and Cruz 2009), but has not been tested in ecological forecasting education specifically.Students were given a case study in which they were asked to role-play as water managers and make decisions about whether or not they should allow a swimming race in a drinking water reservoir given different forecasts of potentially toxic algal blooms occurring at the time of the race (see Appendix S1: Text S1 for a full description of the case study scenario).
As part of Activity B, students were taught to use structured decision-making techniques to apply their management objectives for the drinking water case study.Specifically, students were taught the PrOACT structured decision-making tool (see Table 1 for definition, e.g., Hammond et al. 2002, Hemming et al. 2022).With a goal of optimizing four different management objectives identified using the PrOACT tool (Figure 2A.3), students created hypotheses about how to manage the drinking water reservoir each day as the forecasts were iteratively updated over time (Figure 2A.1; Appendix S1: Figure S3).They completed this objective twice, using forecast visualizations which represented uncertainty using two different methods (Figure 2A.1, 2A.2).Students were encouraged to work through this activity independently and consult with their partner as needed.Finally, students answered questions about how different forecast visualizations influenced their ability to make decisions about managing the reservoir.The culminating discussion of Activity B asked students to discuss how they might improve or alter the visualizations for their decision needs as a water resource manager.Students addressed LO1 in Activity B by using ecological forecasts to make decisions, and LO2 by making decisions using different types of uncertainty visualizations.
In Activity C, "Create a customized visualization of an ecological forecast for a forecast user," students worked individually to choose a different forecast user that was not a drinking water manager (e.g., a swimmer) of the same drinking water forecast they used in Activity B (Figure 2B-C).Students identified a decision to be made by their forecast user (e.g., whether or not to go swimming in a lake based on an algae threshold).Based on the decision that they identified, students created a customized forecast visualization for their user.Additionally, students explored the underlying forecast distribution by examining the mean, median, and upper ranges of the forecast to better understand the uncertainty underlying the forecast.Lastly, students compared their visualizations with their partner, who chose a different forecast user.This Activity C advanced student understanding of LO1 by connecting the forecast to a variety of potential users.By comparing across forecast users, students were also encouraged to think about how different users might benefit from different types of visualizations (Figure 2B and 2C), contributing to their understanding of LO2.
At the end of Activity C (as well as between completion of each activity, time permitting), instructors were guided to bring the student pairs back together for a full group discussion and answer any remaining questions.A list of discussion questions for the instructor to use as prompts was provided for each Activity in the Instructor Manual.For example, to recap Activity A, instructors could ask students to discuss how they were able to tell whether visualizations included uncertainty and if there were some types of visualizations that made it more or less difficult to recognize and interpret forecast uncertainty.For Activity B, instructors could ask students to present their decisions in the case study and explain how the trade-offs among their management objectives influenced their decision-making.Lastly, for Activity C, instructors could ask students to discuss the visualization that they chose for their forecast user and how it related to their forecast user's decision needs, as well as what they would do if they had to create a visualization which served multiple forecast user needs.

Instructional Information and Accessibility
As noted above, all materials for teaching this module are publicly available.The teaching materials (introductory lecture, introduction to R Shiny guide, pre-module student handout, and instructor's manual) are archived in the Environmental Data Initiative repository (Woelmer et al. 2022a).The R Shiny application code is archived in the Zenodo repository (Woelmer et al. 2022b) and can be accessed at http://module8.MacrosystemsEDDIE.org.To access the R Shiny application, students only need an internet connection and a web browser.For students without consistent access to an internet connection, the R Shiny code can be downloaded from Zenodo (Woelmer et al. 2022b) and run locally on a computer using R and RStudio software.
Macrosystems EDDIE Module 8 is designed to be taught either in-person, virtually, or in a hybrid modality.The Instructor's Manual includes instructions on best practices for facilitating the module in each modality.For example, we recommend using Zoom breakout rooms with four students per room (two pairs of students) if teaching virtually, or having students sit together in pairs if teaching in-person.
To increase accessibility for users, the module includes alternative (alt) text descriptions of all images.In addition, text throughout the module was adjusted to have sufficient levels of contrast for improved readability, following Moore et al. (2022a).

Module Assessment
To assess the effects of our module on student learning and answer our research questions, we administered pre-and post-module assessment surveys to undergraduate students before and after module completion, respectively.In total, we tested the module in four undergraduate courses at four different universities with N = 314 consenting students and 7 unique instructors (Table 2).Not all students completed every question so the number of responses per assessment question varied.All students who completed the assessment were undergraduates in their second year or later and were enrolled in General Ecology, Zoology, or Freshwater Ecology courses.Because the module was taught across a variety of institutions, course types, classroom formats, and student experience levels, we were not able to control for these variables in our design, and thus focused our analysis on the total pool of consenting students who completed the module.Instructors were recruited via personal communication, participation in conference workshops, or through an email listserv.The module was taught both virtually and in-person (Table 2), though the majority of students (92%) completed the module with in-person instruction.
As described above, the goal of the assessment was to measure the effects of the module on students' ability to understand foundational ecological forecasting concepts (LO1) and uncertainty communication (LO2; Figure 1, Table 3).We grouped the questions by LOs, resulting in three questions which measured foundational ecological forecasting concepts (LO1) and five questions which measured uncertainty communication concepts (LO2, Figure 1).
The assessment included multiple-choice and qualitative, open-ended questions (Table 3).Pre-and postsurveys were identical and administered via an online, secure portal run by the Science Education Research Center at Carleton College.All students and faculty consented to participate in the study per our Institutional Review Board (IRB) protocols .Analysis of assessment surveys We analyzed multiple-choice and qualitative assessment questions from the pre-and post-module surveys.Multiple-choice questions (Q1-2, 5-9) were scored by whether students selected the correct answer.Qualitative questions (Q3-4) were scored using a rubric developed by two Macrosystems EDDIE coordinators, following a standardized two-step process (see Appendix S1: Text S2 for methodology), based on the rubric methodology of Moore et al. (2022a) and Miles et al (2020).A detailed description of the coding criteria for both Q3 and Q4 is included in Appendix S1: Tables S5 and S6, respectively.We also screened answers to Q4 (Table 3) for the presence of three keywords related to uncertainty communication ('icon', 'color', and 'forecast output/index').We recorded whether the keywords were present or absent in student responses but did not consider responses correct unless students also explained how the keywords were used to communicate uncertainty.
To determine the overall performance within and across LO1 (foundational ecological forecasting) and LO2 (uncertainty communication), we calculated the percent correct within each LO (i.e., resulting in a score for LO1 and LO2) for each student.For the two qualitative questions, which included multiple open-ended responses, student responses were considered 'correct' if they identified at least one benefit of ecological forecasting (Q3) and at least one way of communicating uncertainty (Q4).
We used paired Wilcoxon signed-rank tests to analyze the differences between pre-and post-survey responses on both multiple-choice and qualitative questions as well as the grouped categories.Due to varying class sizes, instruction, student experience levels, and teaching modalities across the four institutions, all data were pooled and analyzed together.Statistical significance was defined as p < 0.05.All analyses were conducted in R version 4.2.1 (R Core Team, 2022).

Results
Our assessment data indicate that student understanding of foundational ecological forecasting and uncertainty communication concepts increased after module completion (Figures 3, 4).Specifically, students identified significantly more ways to communicate uncertainty in a forecast, and were significantly more likely to identify 'decision-making' and 'prediction' as important benefits of ecological forecasts (Figure 4).Across the two LOs, students scored higher in foundational ecological forecasting concepts after completing the module, but showed strong growth in understanding both ecological forecasting and uncertainty communication concepts from pre-to post-module surveys (Figure 5).Student understanding of uncertainty communication Students were more likely to correctly identify and describe multiple ways to communicate uncertainty in forecast visualizations after completion of the module (Figures 3, 4b, 4d; Table 4).The percent of students able to correctly distinguish among different ways to visualize forecast uncertainty increased from 33% pre-module to 60% after module completion (p < 0.001; Figure 3e, Table 4).In addition, students were significantly more likely to identify and interpret differences between two visualizations that had varying representations of uncertainty after module completion (33% of students pre-module vs. 49% post-module; p <0.001; Figure 3d, Table 4).We also observed post-module increases in the percent of students who correctly interpreted a forecast visualization (49% pre-module; 52% post-module) and matched a forecast visualization with a forecast user decision need (42% pre-module; 49% post-module), but these increases were not statistically significant (p = 0.34 and 0.28, respectively; Figure 3c, 3f; Table 4).
Students also showed increased comprehension of uncertainty communication after module completion in our qualitative assessment.When asked to describe two different ways to communicate uncertainty, the number of correct answers students provided increased from an average of 0.2 ± 0.5 on the pre-survey (S.D.) to 1.1 ± 0.8 on the post-survey (p < 0.001; Figure 4d, Table 4).Specifically, the number of students who identified numeric, visual, or probabilistic methods to visualize uncertainty increased significantly after module completion (all p < 0.001; Figure 4b), while student responses which included text or multiple predictions increased, but not significantly (both p = 0.18; Figure 4b, Table 4).The number of students who identified 'numeric' methods to visualize uncertainty (e.g., 'standard deviation', 'statistical confidence intervals', 'intervals', or 'ranges') increased from 8% on the pre-survey to 29% on the post-survey (p < 0.001; Figure 4b, Table 4).Additionally, student responses which included 'visual' descriptions of uncertainty (e.g., 'boxplots', 'shaded area around a line', 'error bars') increased from 8% to 41% (p < 0.001; Figure 4b, Table 4), while responses including 'probabilistic' methods to visualize uncertainty (e.g., 'percentage likelihood', 'probability of exceeding a certain threshold') increased from 2% to 15% from the pre-survey to the postsurvey (p < 0.001; Figure 4b, Table 4).
Several keywords were significantly more prevalent in post-module than pre-module responses to the question about identifying uncertainty communication methods (Q4, Table 3).None of our keywords were identified in the pre-survey responses, but 4% (n = 9) of students included the word 'icon' and 13% (n = 31) of students included 'color' in their answers when asked to identify ways of communicating uncertainty in the post-survey.In addition, 11% (n = 27) of students named 'forecast output' or 'forecast index' in their postsurvey responses.However, we note that of the 27 students who listed 'forecast index/output' as a way to visualize uncertainty in their post-module response, only three students correctly described these terms in the context of uncertainty communication.For example, one student explicitly described what they meant by a 'forecast index/output' ("Forecasts can be visualized through figures that show forecast output, which have direct information on it, and forecast index, which contains a meaningful threshold that is based off what decision is being made"), demonstrating a deeper understanding than students who mentioned forecast index/output without a definition (e.g., "You can visualize uncertainty with a forecast index and a forecast output").

Student understanding of foundational ecological forecasting
We found that student understanding of foundational ecological forecasting concepts increased after module completion (Figures 3, 4; Table 4).Students were significantly more likely to correctly define an ecological forecast after completing the module, with an increase from 27% pre-module to 78% of students answering correctly post-module (p < 0.001; Figure 3a).Student understanding of how forecast uncertainty changes over time also increased from 52% to 58% after module completion, although this increase was not statistically significant (p = 0.16; Figure 3b, Table 4).
We saw a significant increase in the total number of benefits of ecological forecasts identified by students after module completion (p < 0.001; Figure 4c, Table 4).Certain benefits were more likely to be mentioned than others in the student responses.Specifically, students were more likely to identify how forecasting can be used for facilitating decision-making (which was included in 13% of pre-module responses and 35% of post-module responses), and predicting a future event (included in 47% of pre-module and 59% of post-module responses; Figure 4a, Table 4).Correspondingly, the number of "I don't know" responses to the question about forecast benefits significantly decreased from 38% to 8% after module completion (Figure 4a, Table 4).Students also identified other benefits of ecological forecasting in both the pre-and post-survey (e.g., benefits related to management or policy, as well as increased understanding of ecological systems or models), although these themes were not significantly more prevalent in student responses after module completion (Figure 4a).
Overall, student responses to the open-ended, qualitative question about forecast benefits showed an expansion in their understanding of how ecological forecasts can be used.Several students provided fairly simplistic answers about the benefits of forecasts in their pre-module responses and then more nuanced and complex answers in their post-module responses.For example, one student answered "Limiting the effects of climate change" in their pre-module response as a forecast benefit, while their response after module completion showed a more in-depth understanding of forecast applications: "Ecological forecasting allows people to better understand how environmental conditions will change and how that will impact them."Another student similarly wrote in their pre-survey response to the forecast benefit question, "Maybe it'll show the effects of climate change in response to what we do now?" and then expanded to "They [forecasts] can assist in planning future events as well as management or conservation for at risk areas" in their post-survey response.

Performance across ecological forecasting and uncertainty communication learning objectives
Student performance improved from pre-to post-module in correctly answering questions on both foundational ecological forecasting concepts (LO1) and uncertainty communication (LO2; Figure 5).The increase in performance was stronger for students who scored lower on the pre-survey (Figure 5b).Students were more likely to correctly answer questions on ecological forecasting concepts than questions on uncertainty communication concepts on the pre-module assessment (Figure 5a).Both LOs showed strong growth after module completion, with many students who answered zero questions correctly on the pre-survey answering all questions correctly on the post-survey in both categories (Figure 5b).More students scored 100% (all LO-specific questions answered correctly) on foundational ecological forecasting concepts than on uncertainty communication both before and after module completion, though we note that the number of questions in the two categories differed (Figure 1).

Discussion
Our results indicate that completion of a 3-hour module can significantly improve undergraduate ecology students' understanding of uncertainty communication and ecological forecasting.While the percentage of correct answers increased for all assessment questions after module completion, students were more likely to perform higher on foundational ecological forecasting concepts than uncertainty communication concepts prior to module completion.Higher initial performance on ecological forecasting questions may be because students were more familiar with ecological forecasting concepts relative to uncertainty communication concepts before completing the module.However, students showed more growth in describing multiple ways to communicate uncertainty than in identifying benefits of ecological forecasting (Figure 4c, 4d).Below, we explore the implications of our results for undergraduate education in uncertainty communication, ecological forecasting, and ecology broadly.Improved uncertainty communication skills and implications for visualization literacy Students identified significantly more ways to communicate uncertainty following module completion, indicating that the module introduced students to a toolbox of approaches for developing and understanding uncertainty in ecological visualizations.Before completing our module, the majority (85%) of students were unable to describe any ways to communicate uncertainty, while after module completion 72% were able to describe one or more ways to communicate uncertainty (Figure 4d).Being able to identify and describe multiple methods for uncertainty communication is an important skill, as the method used to visualize uncertainty can have a substantial effect on user comprehension and decision-making (Nadav-Greenberg et al. 2008, Ramos et al. 2013, Cheong et al. 2016, McKenzie et al. 2016, Kinkeldey et al. 2017).For example, using summary visualizations (e.g., boxplots) can decrease users' cognitive load and increase the speed of decision-making, but are more likely to lead to misinterpretation (Ruginski et al. 2016, Correll et al. 2018).In contrast, ensemble-based visualizations (i.e., forecast visualizations that show all possible model outputs) may provide users with more information about the whole spread of uncertainty, but viewers may overweight certain ensemble members, leading to inconsistent decision-making (Padilla et al. 2017a).Given that there is no single "best" visualization method for uncertainty communication due to differences in decisionmaking needs (Spiegelhalter et al. 2011), the ability to create a variety of visualization options and adapt visualizations based on forecast user feedback is critical for developing effective uncertainty visualizations.
All of the methods for uncertainty communication included in the students' post-module responses are aligned with current state-of-the-art practices for uncertainty communication in visualization science."Visual" and "numeric," the two uncertainty communication methods most commonly reported by students in postmodule responses (Figure 4b), mirror the two key uncertainty representation techniques ("visualization" and "quantification") identified in a recent review (Kamal et al. 2021).Probabilistic methods, which were also significantly more common in student responses following module completion, can decrease cognitive load and increase use and understanding of visualizations for decision support (Kox et al. 2018).Some students (n = 8) also identified "text" as a useful method for uncertainty communication (Figure 4b), but "text" was almost always (7/8 students) reported in addition to another form of communication (e.g., "visual," "numeric," or "probability"), following literature which shows that text is most useful for explaining and providing context for visualizations (Carr et al. 2018).Additionally, common keywords throughout student responses included "color" and "icon."Thoughtful use of color palettes (e.g., by using discrete rather than continuous color palettes) has been shown to be a powerful tool in representing ranges of uncertainty (Padilla et al. 2017b, Correll et al. 2018).Similarly, the use of icons or symbols has been shown to improve user understanding and usability of decision support tools in diverse settings (Galesic et al. 2009, Garcia-Retamero et al. 2010, Zikmund-Fisher et al. 2014, Kamal et al. 2021), potentially by decreasing the cognitive load required to interpret the communication.While neither text, icons, or numbers alone are typically most effective in scientific communication of complex ideas (Larkin andSimon 1987, Tait et al. 2010), the student post-module responses are reflective of a common theme in the visualization literature that using multiple communication forms increases user comprehension and confidence in decision-making (Fagerlin et al. 2005, Spiegelhalter et al. 2011).Ultimately, the module increased students' ability to communicate uncertainty using multiple approaches, a key skill for developing decision support tools for forecast users.
Overall, the module shows promise for increasing visualization literacy and introducing much needed skills in uncertainty communication to undergraduate ecology students.Most students who completed this module had little to no prior experience with uncertainty communication, but showed substantial improvement in performance after module completion, indicating that a 3-hour module can help build these critical skills (Figure 5).Students' lack of previous exposure to uncertainty communication is likely because undergraduate ecology classes do not currently include uncertainty communication and visualization literacy topics as often as, e.g., ecological modeling and prediction (Willson et al. 2022).Because the communication of uncertainty is just as important as the quantification of uncertainty in forecasts for ensuring that forecast visualizations guide end users' decision-making, it is critical that science communication and visualization science, including incorporation of end user decision needs in visualization development, are included in ecological forecasting training (Robinson et al. 2012, Schwartz et al. 2017, Eisenhauer et al. 2021).
Overall, given the importance of uncertainty communication not only in ecology, but across scientific disciplines broadly (e.g., medicine, meteorology, economics; Tait et al. 2010, Ferstl et al. 2017, Wesslen et al. 2022), improving students' ability to interpret and produce uncertainty visualizations may help enable student participation in a variety of scientific disciplines.Moreover, providing students with improved visualization literacy and uncertainty communication skills will yield a more data-literate population, regardless of students' future careers.This module provides an important first step for incorporating visualization literacy coursework across undergraduate curricula broadly and initiating training in critical visualization interpretation and communication skills.Increase in student understanding of foundational ecological forecasting concepts In addition to expanding students' uncertainty communication skills, completion of the 3-hour module improved students' understanding of foundational ecological forecasting concepts.Following module completion, students were significantly more likely to correctly define an ecological forecast as a future prediction of environmental conditions with uncertainty (Figure 3a; Appendix S1: Table S1).Overall, developing a common definition of "forecast" is important for furthering the field of ecological forecasting, as having common definitions enables meaningful discourse on topics within and across disciplines, providing a scaffold for interdisciplinary work to address complex socioecological problems (Lélé andNorgaard 2005, Robinson et al. 2012).Given that ecological forecasting is an emerging field (Woelmer et al. 2021), codifying definitions in training materials enables undergraduate ecology students to more effectively discuss and learn forecasting topics.
Student understanding of the benefits of ecological forecasting also significantly increased after module completion.Specifically, we found a significant increase in the number of students who identified 'decision-making' and 'prediction' as benefits, but saw only a minimal increase for 'policy,' and decreases for 'management' and 'understanding.' Since 'decision-making' and 'prediction' were emphasized throughout the module, it is unsurprising that these two benefits of ecological forecasting were most commonly provided in student responses.However, the small decrease in responses related to management and understanding is surprising, and may indicate that the module did not sufficiently focus on the benefits of forecasts for management or ecological understanding.For example, while Activity B used a management-centered role-playing example, the activity was primarily focused on the effect of visualization type on decision-making, rather than how forecasts could be integrated into management workflows.Similarly, the module did not emphasize how ecological forecasts can advance understanding of ecosystems and testing of ecological theory or be integrated into policy-making decisions, leaving an opportunity to bolster these forecast applications in future iterations of the module.Alternatively, it is possible that students who identified "decision-making" as a benefit of forecasts may have had policy or management decisions in mind, but not specifically stated this.
We note that students' ability to correctly identify that uncertainty should increase the further into the future a forecast is made (Q2) showed only marginal growth (Figure 3b), leaving room for improvement in teaching this foundational concept of ecological forecasting (sensu Dietze et al. 2018) Lofton et al. 2022).
Integration of decision support concepts into ecology curricula Integrating applied decision-making concepts into ecological forecasting and uncertainty communication lessons heeds a widespread call to make ecological research more societally relevant (e.g., Belovsky et al. 2004, Ruhl et al. 2022).Training that incorporates components of translational ecology (e.g., science communication, end user engagement, structured decisionmaking, multidisciplinary training) has long been recommended for ecologists at all career stages (Robinson et al. 2012, Schwartz et al. 2017, Eisenhauer et al. 2021), but resources targeted at the undergraduate level have been lacking (Bakermans and Pfeifer 2018).Our module aims to close this gap by incorporating complex water management decision-making scenarios with multiple end users to engage students in solving real-world problems, while also developing visualization literacy skills.While we did not quantitatively assess the engagement of students in the module, we received open-ended positive feedback from many students.Students reported that the module was enjoyable and important ("I really enjoyed looking at decision analysis in an ecology class;" "I think it's very important to talk about this in science classes").Additionally, student responses suggest that they found that the module was interesting ("This was informative and a really interesting way for me to realize the actual impacts of ecological forecasting") and novel ("It was helpful because I came in with no information").
Ultimately, introducing students to applications of ecological forecasting for real-world decision-making may help recruit students to work in this subdiscipline, as well as highlight the importance of using ecology to produce actionable tools to address societal problems (e.g., Enquist et al. 2017).Many ecological forecasters have already begun to integrate decision support and uncertainty communication components into their forecasts by making forecasts which are actionable, useful, and targeted towards forecast users (e.g., Gerst et al. 2019, Turner et al. 2020, Jackson-Blake et al. 2022).Our experience with this module indicates that even a short (3-hour) exposure to decision support and uncertainty communication concepts can increase students' understanding of potential applications and benefits of ecological forecasting.

Module caveats and opportunities for future use
The impact of this module on student learning of decision science and uncertainty communication skills could be improved in several aspects.First, during module development, we intentionally introduced students to a single method of structured decision-making (PROACT) and a limited number of uncertainty communication methods (i.e., visual, numeric, probabilistic) to provide a simplified introduction to the decision and visualization sciences.While students showed successful understanding of the PrOACT tool (Appendix S1: Figure S4), the addition of other decision support components, including solutions-oriented decision-making theory (Deitrick and Wentz 2015) or additional methods of structured decision-making (Gregory et al. 2012), would increase students' breadth of understanding of decision science.Inclusion of a broader variety of decision support concepts could also lead to improved performance on the decision-related questions (e.g., Figure 3c, 3f).Second, allowing students more control over visualizations within the Shiny app (e.g., additional visualization or personalization options, inclusion of R-based coding activities) would likely increase students' visualization literacy (Huron et al. 2014, Alper et al. 2017, Börner et al. 2016, Börner et al. 2019).Third, due to time constraints, our module asked students to imagine what type of forecast visualization would best meet different end users' needs, rather than asking them to actively engage in co-development of visualizations with different forecast users, which would likely increase the utility of the visualization (Raftery 2016, Padilla et al. 2017b, Gerst et al. 2019).Fourth, shifting the focus of the case study in Activity B to be customizable for specific, nearby ecosystems which are directly relevant to students' everyday lives could potentially increase engagement and student learning (Cid and Pouyat 2013, Henri et al. 2022, Vance-Chalcraft and Osborne Jelks, 2022).For example, students living in areas where wildfires are common may be more engaged analyzing a case study presenting a decision-making scenario on wildfire forecasts.While we recognize the value in including additional content on uncertainty communication, decision science, and ecological forecasting, we note that expanding the module may make it less feasible for instructors to add into their ecology curricula.
Several caveats should be considered when interpreting the assessment results from our module.First, we used a pre-and post-module methodology because instructors were unable to divide their classes into treatment and control groups for instruction.Second, there were many factors which were not held constant across the classrooms that tested our module, including student experience level, instructor experience level, classroom size, institutional familiarity with forecasting and others that could influence the effect of the module on individual student learning.Third, due to the length of the module and limitations of our assessment survey, our analysis provides only a limited understanding of students' knowledge gain.Future longer-term assessments are needed to assess student growth over a longer duration of time.

Conclusions
Communication of model uncertainty is of paramount importance for advancing the utility of ecological research findings for decision-making.Our teaching module provides an introduction to concepts and skills needed for ecology students to increase their visualization literacy, engage in data science applications, and develop decision support tools.Introduction to ecological forecasting concepts at an early educational stage, including an improved understanding of the importance of ecological forecasting for societal benefit, is increasingly necessary for training the next generation of predictive ecologists to meet both European (Nativi et al. 2021) and U.S. government agency directives (Vought and Droegemeier, 2020;Arsenault et al. 2020;NOAA, 2022;CDC 2022).Moreover, by teaching ecological forecasting and uncertainty communication skills via a real-world decision-making scenario, this module helps to emphasize the relevance and lower the barrier of entry to ecology.Through an R Shiny interface that is easy to implement for educators in a range of classroom experience levels, this 3-hour, adaptable module fills a critical gap in undergraduate ecology curricula.By introducing students to uncertainty communication and ecological forecasting early in their careers, this module can help train the next generation of ecologists to conduct societally relevant research and tackle pressing ecological challenges.

ACKNOWLEDGEMENTS:
We are grateful to the students who participated in this module and their instructors, especially Kait Farrell, Hilary Dugan, Kait Reinl, Sheila Lyons-Sobaski, Matt Aiello-Lammens, and graduate teaching assistants, who provided constructive feedback which substantially improved the quality of the module.We thank Melissa Kenney for her invaluable feedback on module activities, as well as the Carey and Schreiber Labs for their feedback throughout module development, especially Caroline Bryant, Arpita Das, and Jacob Wynne, who completed pilot versions of this module.We thank Kait Farrell and Alex Hounshell for their contributions to the Macrosystems EDDIE program, as well as Kristin O'Connell, Monica Bruckner, Ashley Carlson, Cailin Huyck Orr, Ellen Iverson, and the Science Education Resource Center at Carleton College for coordinating website development and survey administration.Lastly, we thank members of the Ecological Forecasting Initiative and the Global Lake Ecological Observatory Network for providing support and feedback on module content and fostering community around these topics.Our study was financially supported by NSF DEB-1926050, DBI-1933016, and DEB-1926388.AUTHOR CONTRIBUTIONS: CCC developed the Macrosystems EDDIE program.CCC and RQT expanded the EDDIE program for forecasting and acquired funding for the project.WMW and CCC conceived of the study design.WMW led the development of module materials, with substantial guidance from RQT, CCC, and TNM.TNM assisted with R Shiny app development, as well as administration of module assessment.WMW, TNM, and MEL co-analyzed qualitative assessment questions.WMW led the analysis of module assessment results and manuscript writing, with significant input from CCC and MEL.All authors edited and approved of the final manuscript.

CONFLICT OF INTEREST STATEMENT:
The authors declare no conflict of interest.3).A description of questions can be found in Table 3 and Appendix S1: Table S3.

3.
Percentage of students who answered multiple-choice questions correctly in the pre-and post-module surveys.Asterisks (***) indicate a statistically significant difference between the pre-and post-module survey according to a Wilcoxon signed-rank test (p < 0.001).Colors of the bars correspond to LO1: Foundational ecological forecasting (EF) concepts (green) and LO2: Uncertainty Communication (orange; Figure 1, Table

Figure 1 Figure 2
FIGURES . To complement this module and provide additional training in foundational ecological forecasting concepts, we suggest pairing this module with other Macrosystems EDDIE modules (Module 5: Introduction to Ecological Forecasting, Moore et al. 2022b; Module 6: Understanding Uncertainty in Ecological Forecasts, Moore et al. 2021; or Module 7: Using Data to Improve Ecological Forecasts,

Table 1 .
Glossary of uncertainty communication and ecological forecasting terms taught in Macrosystems EDDIE Module 8: "Using Ecological Forecasts to Guide Decision-Making" as well as examples of how each term is applied to real-world, near-term forecasting.