From theory to practice in pattern-oriented modelling: identifying and using empirical patterns in predictive models

To robustly predict the effects of disturbance and ecosystem changes on species, it is necessary to produce structurally realistic models with high predictive power and ﬂ exibility. To ensure that these models re ﬂ ect the natural conditions necessary for reliable prediction, models must be informed and tested using relevant empirical observations. Pattern-oriented modelling (POM) offers a systematic framework for employing empirical patterns throughout the modelling process and has been coupled with complex systems modelling, such as in agent-based models (ABMs). However, while the production of ABMs has been rising rapidly, the explicit use of POM has not increased. Challenges with identifying patterns and an absence of speci ﬁ c guidelines on how to implement empirical observations may limit the accessibility of POM and lead to the production of models which lack a systematic consideration of reality. This review serves to provide guidance on how to identify and apply patterns following a POM approach in ABMs (POM-ABMs), speci ﬁ cally addressing: where in the ecological hierarchy can we ﬁ nd patterns; what kinds of patterns are useful; how should simulations and observations be compared; and when in the modelling cycle are patterns used? The guidance and examples provided herein are intended to encourage the application of POM and inspire ef ﬁ cient identi ﬁ cation and implementation of patterns for both new and experienced modellers alike. Additionally, by generalising patterns found especially useful for POM-ABM development, these guidelines provide practical help for the identi ﬁ cation of data gaps and guide the collection of observations useful for the development and veri ﬁ cation of predictive models. Improving the accessibility and explicitness of POM could facilitate the production of robust and structurally realistic models in the ecological community, contributing to the advancement of predictive ecology at large.


I. INTRODUCTION
Environmental change threatens the loss of biodiversity and associated ecosystem functions, services, and stability. Rapidly changing environments pose additional challenges for environmental scientists and managers who aim to strengthen resilience of ecological systems and to maintain a safe operating space (Johnstone et al., 2016). To achieve this, we must understand the dynamics of these complex systems, which are controlled by decisions, interactions, and adaptations of agents (e.g. humans, organisations, animals, plants, bacteria, etc.), and the various feedback mechanisms between them and the environments they inhabit. Individual or agent-based modelling (IBM/ABM) is increasingly used to synthesise knowledge gained through empirical studies, assess whether hypotheses developed about system functions are plausible, and make predictions about system dynamics for potential scenarios (Vincenot, 2018). While correlation-based statistical models are effective in elucidating relationships between factors (mostly metric variables), these methods are unsuitable to analyse complex interactions among multiple (including non-metric) factors and are often unreliable for making predictions beyond the scope of the data used in their development (Urban et al., 2016). For forecasting under altered conditions, it is necessary not only to describe observed dynamics phenomenologically, but according to underlying mechanistic processes. Models are suitable for mapping responses of ecological systems to environmental changes and disturbances only if emergent behaviour of virtual agents sufficiently resembles key behaviours of the real system (e.g. if simulated animals switch their migration routes because the modelled availability of food resources changes; simulated farmers learn the practices of agrofarming because their neighbours have benefitted from it; or plants develop more root biomass to uptake more water under conditions of drought stress).
To successfully develop mechanistically correct, and thus structurally realistic, representations of key processes in ABMs, experienced modellers propose the implementation of a powerful strategy called pattern-oriented modelling (POM, see e.g. Wiegand et al., 2003;Grimm & Berger, 2016;Grimm, Ayll on & Railsback, 2017;Railsback & Grimm, 2019;Railsback & Harvey, 2020). POM is a strategy for confronting models with empirical patterns, which are observed at different hierarchical levels of organisation, that can be used to balance the level of model complexity and increase the chance of capturing the processes in the system necessary for addressing a model's purpose (Grimm & Railsback, 2012). This strategy assumes that patterns that can be observed in an ecological system are fingerprints of underlying processes and consequently can be used in process inference.
According to Grimm et al. (1996), patterns are characteristic and clearly identifiable structures in nature or in data extracted from nature. They represent information hidden in system structures which emerges as an outcome of interactions between internal processes and constraining factors (e.g. environmental conditions, human alterations, etc.) (Wiegand et al., 2003). Patterns can be classified as being strong or weak. Strong patterns provide a strong indication of the underlying processes in a system and are typically described quantitatively (Grimm & Railsback, 2012). Weak patterns are often qualitative and can be produced by many model structures and processes. Therefore, weak patterns do not provide much information on system function on their own, though when multiple weak patterns are used together they can provide just as much or even more information than a single strong pattern (Wiegand, Revilla & Knauer, 2004).
Biological Reviews (2021)  However, when selecting patterns, it is important to consider whether each pattern is relevant to the model's purpose. Using the model's purpose as a filter allows for narrowing of potential parameter values, testing of model performance, and for defining model structure as each pattern requires certain components to be present in the model structure. For example, if the purpose of a model is to test various behaviours to investigate which result in movement patterns matching observations, reproductive patterns may not be useful for addressing this specific purpose and the inclusion of the processes needed to collect these patterns from simulations would likely overcomplicate the model. Information contained in patterns can be described qualitatively (e.g. trends) or quantitatively, including by summary statistics (e.g. into a single number, such as an average), as functions of variables (e.g. distance with time), or spatial descriptions (e.g. by computing values on a spatial grid). Modellers must define specific criteria for establishing model fit, for example by using distance measures (e.g. error calculations) that can estimate differences between summary statistics generated for observations and those of model outputs.
It is not enough that model outputs may reproduce patterns observed in the real world, they must also reproduce them for the right reasons, meaning that processes which lead to pattern formation in nature should also drive emergence of patterns in the model. For example, plants compete for locally limited resources (light, water, nutrients). Therefore, during the development of a plant population, the number of plants decreases (a portion dies due to density-dependent competition) while average biomass of surviving plants increases. This phenomenon, which is called self-thinning, manifests in two patterns: (i) the spatial distribution of plants, which is usually random after seed establishment and germination, becomes regular over time (space and associated resources are distributed more and more evenly), and (ii) the increase in average biomass with the reduction in plant density follows a characteristic line, the so-called self-thinning line, which has a typical slope (Zeide, 2010;Peters, Olagoke & Berger, 2018).
Agent-based models which describe plant populations should reproduce these patterns (among others). However, their suitability for understanding system dynamics remains limited if the reproduction of patterns is achieved using imposed phenomenological functions linking overall plant density with averaged plant biomass. To describe essential processes mechanistically, plants must have a spatially explicit position which clearly defines their local neighbourhood, and it may be important to include competition between neighbouring plants for resources, such as light (an asymmetric process, as taller plants receive more light than shadowed ones). In addition, empirical patterns also define model output that must be recorded: here plant biomass or metrics that allow a subsequent calculation of biomass.
Generally speaking, POM is a strategy to achieve structural realism of models. It encourages use of multiple patterns, observed at different scales and hierarchical levels of ecological systems, as filters for (i) selecting variables including input and output, (ii) designing submodels for processes including individual behaviour and feedback mechanisms with the environment, and (iii) parameterising and optimising models (Grimm et al., 2017). Additionally patterns should be used in testing and evaluating model outputs and reimplementations (Railsback & Grimm, 2019). In this sense, using empirically observed patterns becomes an integral part of the entire modelling cycle , which consists of six steps: formulating the research questions (the model's purpose), assembling hypotheses about system functioning, choosing model structure, and model implementation, analysis, and communication.
Like the modelling cycle, in which steps must be replicated several times, POM also forms an iterative cycle. Patterns are initially selected to define the preliminary model structure. After a first round of tests as to whether the chosen submodels correctly reproduce observed patterns, additional observations are often identified, which further refines the knowledge of the system. Pattern reproduction drives stepwise refinement of model structure as an inability to reproduce patterns by an initial model structure can be used to pinpoint processes which must be added or refined in the next modelling cycle.
Therefore, POM is an effective method for linking empirical science to theory development: empirically observed patterns are used to develop a theory for mechanisms on the agent-level that explain patterns observed at population, community, and ecosystem levels . Processes at higher hierarchical levels are vice versa identified that shape patterns on individual agent levels. In other words, POM provides a means to bridge bottom-up processes with top-down processes, which together control the dynamics of ecological systems but are often separated by modelling approaches.
Simultaneous replication of multiple natural patterns can provide evidence that a model is fit for its intended purpose and that its predictions can be used to support real-world decision making (Grimm et al., 2020). The more relevant patterns that a model can reproduce, particularly including vague or weak patterns, the more likely that the model has captured necessary mechanisms for answering a particular question. Alternatively, demonstrably incorrect assumptions can be tested. If results of these tests indicate a reduced ability to reproduce important patterns, this increases confidence in the reliability of model predictions for decision makers (Grimm et al., 2020). Patterns themselves can be subjective and contain sampling and analytical biases as our limited knowledge of systems drives data collection techniques, analytical methods, and selection of patterns to be used in POM, and these decisions are often rooted in familiarity (Grimm et al., 2020). Considering that each pattern contains its own suite of biases, it is crucial that multiple patterns which span different hierarchical levels are considered when developing and testing ABMs using POM.
Although use of POM would greatly benefit the field of predictive ecology by providing a systematic framework for Biological Reviews (2021)  Guidance for pattern-oriented modelling the consideration of reality within the system of interest, explicit uptake of POM has not risen meaningfully in the last decade despite rapid increases in the production of ABM/IBMs (Scopus search 06 June 2020; Fig. 1). While modellers may be using this approach, or elements of it, without explicitly using the term POM, the absence of increases in explicit adoption of POM may additionally be related to a lack of accessibility of the practice. According to our experiences, modellers, particularly newcomers in ABM, often find it challenging to identify relevant empirical patterns in their system. Additionally, once patterns are identified, modellers are often unsure about how and when they should apply their specific patterns following the POM approach. Wiegand et al. (2003) called for a systematic classification of patterns which could be used in POM to advance conservation practice by providing guidelines on which types of patterns are particularly useful for predictive modelling. These guidelines would not only help modellers identify which patterns would most benefit the determination of model parameters and processes, but also enhance empirical practice by highlighting the value of specific biological information for predictive modelling.
Here we present such a classification of empirical patterns and provide specific guidelines throughout on how properly to apply empirical patterns in ABMs. In this review we aim to make POM development more accessible by providing detailed answers to the questions (see Fig. 2A): (i) where in the ecological hierarchy can we find patterns; (ii) what kinds of patterns are useful; (iii) how should simulations and observations be compared; and (iv) when in the modelling cycle are patterns used? Each section builds on the previous text by using introduced terms and concepts to describe characteristics of useful patterns and guide modellers through effective implementation strategies. We additionally provide a conceptual framework for the POM process (Fig. 2B), which modellers can follow to consider the key concepts from each section, allowing them to cast their net as widely as possible when identifying appropriate patterns and/or ways to use them in the modelling cycle, whilst at the same time avoiding the various pitfalls outlined in section VI. This framework encourages modellers to identify relevant patterns in their system by looking across the categories outlined in the What section and the hierarchical levels presented in Where, decide where to best employ each pattern in the modelling cycle (When), and then consider How to best approach comparing patterns between observed and modelled systems. We believe that if these steps are taken intentionally, better modelling outcomes can be reached and better decisions made from those models. As ecologists, we focus on patterns that are observed in ecological systems. Nevertheless, we are convinced that the given examples and classifications are also useful for readers from other disciplines and encourage both ecologists and other scientists to recognise patterns in their own systems and to use them effectively.

II. WHERE IN THE ECOLOGICAL HIERARCHY CAN WE FIND PATTERNS?
A primary goal of POM is to link low-level individual processes of a system to those occurring on higher levels of organisation (e.g. linking aspects of behavioural and population ecology). To establish this ecological 'micro-macro' link when modelling, we must simultaneously employ patterns observed for different levels of ecological hierarchy. As ABMs often focus on individuals of one or more species and their environments, hierarchical levels of interest are typically the individual, population, community, and ecosystem levels (Fig. 3). The level of a particular pattern depends on the entities (individuals, populations, etc.) for which that pattern is observed. It is important to note that patterns do not 'exist' for those entities, but instead reflect our point of view and how we characterise entities and their processes (e.g. using summary statistics). For example, patterns observed on the individual level relate to observations of or processes experienced by an individual agent, such as its basic characteristics, behaviour, movement, or physiology. When collections of individuals or entire populations are observed to form a pattern, those patterns are said to be on the population level. Population-level patterns aggregate characteristics of the structure of populations and document changes in time and/or space. When individuals from multiple species are considered for pattern formation, patterns fall on the community level. Communities can be represented similarly to populations by using observations of their structure, dynamics, and distributions. If both abiotic and biotic aspects of a modelled environment are observed, patterns are on the ecosystem level. Ecosystem-level patterns relate to interactions between species and non-living aspects of their environments Trends in the production of pattern-oriented and agentbased modelling publications. The number of annual publications in pattern-oriented modelling (POM; in red) using the search criteria: All = "pattern-oriented modelling"; and for agent-based modelling (ABM; in blue) using: All = "agentbased modelling" or "individual-based modelling" (source: Scopus). Biological Reviews (2021) 000-000 © 2021 The Authors. Biological Reviews published by John Wiley & Sons Ltd on behalf of Cambridge Philosophical Society. and the dynamics of key abiotic resources like water, wind, carbon, and nitrogen.
Patterns observed for all relevant levels of a model should be considered when following a POM approach (e.g. ABMs including multiple species should employ patterns observed for individuals, populations, and communities if available). Examples of POM-ABMs that used patterns at multiple levels include Warwick-Evans et al. In ecology, a multi-scale approach is natural when a mechanism can be attributed to the molecular, cellular, or organismal level (Wilensky & Reisman, 2006). Similar attention should be afforded to potential patterns emerging from multiple levels of an ABM-POM, as a model focused on patterns at only one level may not capture important patterns on other levels. It is the interactions among processes occurring at different levels that defines the ecology of a system (Grimm & Railsback, 2012).

III. WHAT KINDS OF PATTERNS ARE USEFUL?
When deciding to use the POM approach to ABM development, patterns may initially seem obscure and it may be unclear what kinds of information are useful or available for developing or testing a model. By looking at similar models that have employed POM as examples, useful patterns employed in a particular field can be identified. These examples may also help guide application of patterns or data collection to fill current knowledge gaps. It is important to note that only patterns that are important to the model's purpose should be used. Even well-documented and clear patterns should only be included if they are relevant to the model's purpose. Seeking out patterns that relate to the processes necessary for achieving the purpose can minimise distractions and limit the inclusion of unnecessary detail in the model.
Here we present a classification of patterns used in ecological ABMs to consolidate trends and identify useful information sources and general patterns employed in these models. We identified seven categories for classifying patterns based on the ecological concepts and processes to which they relate. It is important to note that these categories are not patterns in themselves but instead represent processes or approaches to Guidance for pattern-oriented modelling characterising individuals and their environments which can be used to group related patterns found therein. These categories in their presented order are: behaviour, energetics, movement, genetics and evolution, structure, dynamics, and distributions (see Fig. 3). Categories are not mutually exclusive, for example a pattern falling into the structure category may include an energetic component such as relative frequencies of energy intake rates of a population. Agentbased models using POM often include numerous patterns from different categories. By considering examples of ABMs which have used POM explicitly or implicitly we collect and present patterns in each of the pattern categories found to be especially useful for POM-ABM development and provide specific examples. In Appendix S1 (see online Supporting Information) we summarise patterns used in selected publications from each category and briefly describe Where, When, and How these patterns were employed.
(1) Behaviour Organism behaviours are likely influenced by numerous and potentially complex ecological drivers, and ABMs are useful to model such drivers due to their inherent flexibility. Some standard patterns used for characterising and quantifying animal behaviour include activity budgets, habitat selection, and diet compositions.
Activity budgets, also called time budgets, represent the proportion of time animals spend in defined behavioural states. Activity budgets are often measured empirically using visual or tag-derived individual observations and are compared directly to simulated activity budgets using the percentage of time attributed to different states (Warwick-Evans et al., 2018). Behaviours used in activity budgets can be classified using binary states such as active versus inactive (Halle & Halle, 1999;Haythorne & Skabar, 2013) or foraging versus non-foraging (Semeniuk et al., 2012), or by using more specific behaviours like resting, foraging, travelling, socialising, parenting, etc. (Frair et al., 2008;Bialozyt et al., 2014;Warwick-Evans et al., 2018). Activity budgets may be combined with measures such as energetics, which can be used to calculate costs associated with specific behavioural budget patterns (Warwick-Evans et al., 2018).
Patterns describing habitat selection are often used to study individual resource use (Liu et al., 2013;Bateman et al., 2015). These patterns may be quantified as percentage of time spent in different habitat types or at specific locations (Railsback & Harvey, 2002;Semeniuk et al., 2012), by using compositional analysis (Aebischer, Robertson & Kenward, 1993) to test whether two habitat compositions were drawn from the same distribution, or resource selection functions which compare used versus available or unused resources (Boyce et al., 2002). Use of specific areas can emerge from differences in movement characteristics which enable individuals to spend a higher proportion of their time in high-quality habitat and transit through low-quality habitat Bateman et al., 2015), or by 'active' choice of particular areas by individuals depending on structural characteristics or the state of individuals (Liu et al., 2013;Chudzi nska et al., 2016).
Diet compositions are also generally quantified as a proportion (e.g. the percentage of the diet consisting of different prey types). Total percentage of diet attributed to different prey can be represented either by mass or as a percentage of total energy intake (Travers-Trolet, Shin & Field, 2014). Prey may be classified to the species level, by functional group, or by more general types. In Kane et al. (2016), for non-obligate scavengers, prey was categorised as being scavenged or live caught and the percentage of energy needs met by scavenging for species of different masses was used as a pattern to characterise foraging.
(2) Energetics Integration of energetic processes into ABMs are commonly used to link survival and fecundity of individuals to their environment, to link foraging strategies and behaviour, or to investigate food-limited populations Massardier-Galatà et al., 2017). Energetic patterns often quantify individual energetics using a standard energy unit such as joules or watts. These patterns can be broadly grouped by whether they relate to rates of energy intake or energy use of individuals.
Patterns in energy intake generally represent foraging success of individuals. These patterns can be quantified using the number of specific prey items ingested, weight of ingested matter (e.g. in grams), or energy gained (e.g. in joules) per unit time, often a day (Roese, Risenhoover & Folse, 1991;Testa et al., 2012;Chudzi nska et al., 2016;Gallagher et al., 2021). Semeniuk et al. (2012) estimated daily net energy intake in megajoules of woodland caribou (Rangifer tarandus) using empirical ingestion rates of lichen in kilograms and then converted to energy using a literature-derived energetic conversion factor. The resulting range of daily energy intake was then used as a pattern to evaluate performance of five potential behavioural scenarios implemented in a model.
Empirical energy use patterns may be estimated directly by measuring oxygen consumption or CO 2 production using respirometry or doubly labelled water (Mueller et al., 2012;Jager & Ravagnan, 2015;Jager, Salaberria & Hansen, 2015). However, as it can be difficult to obtain these measures in wild animals, values are frequently based on laboratory or captive studies. Alternatively, indirect measures may serve as proxies for energy requirements, such as patterns in ventilation rates, heat loss, or body temperature (Jager & Ravagnan, 2016;Beltran, Testa & Burns, 2017;Malishev, Bull & Kearney, 2018;Desforges et al., 2020) and potentially movement metrics such as dynamic body acceleration (Qasem et al., 2012;Chimienti et al., 2020). Changes in mass or stored energy, particularly individual growth rates or body composition changes, can also provide insight on individual energy use or balance (Mueller et al., 2012;Dey et al., 2017;Boyd et al., 2018;Gallagher et al., 2021). Energy use patterns may change with time due to temporal differences in food availability or energetic requirements, such as Biological Reviews (2021)  Guidance for pattern-oriented modelling with seasonal shifts in mass or body composition at different life-history events [e.g. reproduction or maturity (Beltran et al., 2017, Heinänen et al., 2018, Desforges et al., 2019]. (

3) Movement
The rapidly developing field of movement ecology has turned to ABMs as a tool for modelling animal movements as they allow for space-use and movement patterns to emerge from individual responses to information in their environment. When appropriately interpreted, movement patterns may prove particularly useful for modelling adaptive individuals (Railsback & Harvey, 2020). While many kinds of movement are considered behaviour, we treat these separately due to the wealth of movement information being currently collected using rapidly developing bio-logging technologies. Empirical patterns in animal movement are often derived from animals equipped with tags (Liukkonen et al., 2018;Nabe-Nielsen et al., 2018;Merkle et al., 2019;Chudzinska et al., 2021) or by using point observations such as checkpoints or scent marks (Lewis, White & Murray, 1997;Heinänen et al., 2018). Broadly speaking, these patterns can describe either path characteristics or space use of individuals.
Path-level analyses form an important source of patterns in spatially explicit ABMs focused on animals as individual movements are often a core component of these models. Primary path metrics, such as step lengths and turning angles, can be easily extracted directly from changes in position between timesteps (Seidel et al., 2018). These metrics are routinely used to model movement strategies (e.g. correlated random walks; Nabe-Nielsen et al., 2013), or to compute secondary metrics using summary statistics or by focusing on larger scales. Useful patterns can be found in both primary and secondary metrics. Examples of secondary path metrics commonly used as patterns in ABMs are dispersal or displacement distances, residence times, travel distances, and first passage times (Bailleul et al., 2013;Nabe-Nielsen et al., 2013). Some examples of models using these metrics are Schiegg, Walters & Priddy (2005) where patterns in sexspecific natal dispersal distances were used to evaluate a red-cockaded woodpecker (Picoides borealis) population model and Liukkonen et al. (2018) who used patterns of distances from haul-out sites to calibrate Saimaa ringed seal (Phoca hispida saimensis) movements. Recently more advanced analytical methods in movement ecology, such as state space modelling, have been employed for extraction of secondary movement metrics in ABMs. In Zhang et al. (2017), movements of black petrels (Procellaria parkinsoni) were modelled using hidden Markov models to compare observed GPS tracks to modelled trajectories. Both primary and secondary metrics can be useful in finding patterns in the synchrony of collective movements [e.g. flocking or schooling (Huth & Wissel, 1994;Cambuí & Rosas, 2012;Hemelrijk & Hildenbrandt, 2015)]. In addition, rather than focusing on metrics extracted from paths, entire (or portions of) empirical animal paths can serve as patterns to be reproduced when testing how environmental conditions or movement rules drive behaviours like dispersal or migration (Aben et al., 2014;Merkle et al., 2019).
By aggregating animal movements, patterns in space use can be derived that provide insight on how organisms interact with their landscapes (Seidel et al., 2018). Instead of using consecutive points in a path, space-use analyses often use collections of positions to estimate home ranges or territories used by animals. Home ranges and territories can be characterised by metrics including their size, distribution, overlap, and shape (Liu et al., 2013;Carter et al., 2015). As these patterns emerge from the movement strategies of individuals, they are often used in the evaluation stage of ABM development (Carter et al., 2015;Malishev et al., 2018).

(4) Genetics and evolution
ABMs have been increasingly used for modelling evolutionary processes as they can directly model interactions which drive their dynamics, such as geno-and phenotype, fitness, and behaviour of individuals in changing environments (Pierson et al., 2015;Pontarp et al., 2019). Individual genetics, mediated by interactions between individuals and their environments, shape the evolution of populations and communities. Patterns formed by these processes may be gathered using a variety of methods including genomics, field-based measurements, and, for small and short-lived species, laboratory-based controlled breeding or fitness experiments (Haafke, Abou Chakra & Becks, 2016). Patterns related to genetics and evolution can be described using genetic metrics, trait distributions, and phylogenetic metrics.
Recent advances in genetic technologies have provided various metrics that can serve as patterns for parameterising and evaluating models with explicit genetic processes. For instance, distributions of allele frequencies can be used as patterns when assigning model genotypes (Auffarth et al., 2017) and pedigrees or genetic heterozygosity may be used to estimate inbreeding levels (van de Kerk et al., 2019). Inbreeding depression can be parameterised or evaluated using genomic patterns in the number of lethal equivalents or deleterious alleles, whereas approaches based on shared haplotype lengths (e.g. timing and magnitude of admixture, population size changes, and divergence times) can be used as patterns estimating genetic changes consistent with outbreeding and hybridisation (Pierson et al., 2015).
While it is possible to model organisms down to the gene level, many models instead choose to focus on phenotypic traits when modelling evolution. These traits can be related to the life history or fitness of individuals and are subject to selection, mutation, and inheritance. Trait-based patterns are generally represented as the relative frequencies of traits that are empirically measurable, such as those related to size or fecundity (Ayll on et al., 2016; van Petegem et al., 2016). Patterns in species traits can be independent of space or be measured along gradients, such as in latitude or resources (Bédécarrats & Isselin-Nondedeu, 2012; van Petegem et al., 2016).
Biological Reviews (2021)  The evolutionary histories of communities can be represented by patterns within their phylogenetics. Patterns in phylogenetic trees, divergence, species richness, and number of lineages-through-time can be useful metrics for parameterising and evaluating models with evolutionary processes. In a model of salamander evolution, Barnes & Clark (2017) used patterns in species richness, divergence, and lineagesthrough-time for model parameterisation.

(5) Structure
Structure provides an important source of patterns commonly used in POM-ABMs. These patterns characterise individual metrics and are often presented as relative frequencies. Structural patterns can be observed at all hierarchical levels, but there are differences in the common use of these patterns for each level.
On the individual level, species can be characterised using aspects of their morphometrics or life history. Commonly used structural patterns on this level often relate to length, mass, or height of individuals, or reproduction and mortality such as individual fecundity or longevity. Goedegebuure et al. (2018), in a model of southern elephant seal (Mirounga leonina) energetics, used empirical patterns of individual morphometrics and life history, including maximum length, lifetime reproductive success, pup survival probability, and maximum lifespan to evaluate the results of a sensitivity analysis.
Patterns observed in population structure often characterise a population's size or age composition or its demographic rates. The size structure of populations can relate to any measurable morphometric feature of species, such as mass, length, height, diameter, or area (Winkler & Heinken, 2007;Seidl et al., 2012;Boyd et al., 2018;Pavlowich, Kapuscinski & Webster, 2019); while age structure can be presented for specific years or grouped into age classes. An example where population age structure was used as a pattern can be found in Nabe-Nielsen et al. (2014) where emergent relative frequencies of age groups were used to evaluate a model of harbour porpoise (Phocoena phocoena) population dynamics. These metrics are often presented together, such as in patterns of average mass by age (Boyd et al., 2018). Population structure can also be characterised by species life history or demographics, using metrics such as lifetime reproductive success or sex ratio (Swanack, Grant & Forstner, 2009;Topping et al., 2010a;Rughetti et al., 2017). Rather than being displayed as relative frequencies, structure can additionally be presented using basic statistical properties such as the mean or range of population attributes [e.g. mass or age (Goedegebuure et al., 2018;Warwick-Evans et al., 2018)], and can relate to specific temporal events such as maturation or first reproduction (Schiegg et al., 2005;Benton, 2012).
Similar to patterns of population structure, community structure patterns are formed by aggregating characteristics of individuals in a community (e.g. relative frequencies of height; Rödig et al., 2017). An example using a community structure pattern can be found in Bédécarrats & Isselin-Nondedeu (2012), where relative frequencies of binned morphometrics, here specific leaf area, was used as a pattern. In forest models, treeline patterns, such as those forming from height and horizontal or vertical structure, are often used to characterise forest structure (Berger et al., 2006;Wiegand et al., 2006). Frequency distributions of spatial metrics can also be used as patterns in community-level models, such as the relative frequencies of range sizes (Bini et al., 2006;Rangel, Diniz-Filho & Colwell, 2007;Barnes & Clark, 2017). An example using this pattern is Rangel et al. (2007), where the range size-frequency distribution of approximately 3100 bird species was used for parameterisation of a community model spanning the entire continent of South America.
While some structural patterns observed for communities are similar to those used for populations, communities can additionally be characterised by community-specific measurements. These metrics generally relate to the number or abundance of species in a community. Common community-specific metrics used as patterns in multispecies ABMs are species richness, rank-abundance, beta-diversity, and species-area relationships (May, Huth & Wiegand, 2015). These patterns provide insight on community make-up and, as high-level patterns, are often used for model selection or evaluation. Additionally, functional response curves provide another useful community-specific structural pattern as they reveal information on densitydependent interactions among community members (Goss-Custard et al., 2006).
Patterns formed by the structure of ecosystems extend patterns on lower levels by considering the structure of abiotic and physical aspects of environments. Ecosystem-level structure can be presented in various ways but is often observed as some species measurement of interest under differing abiotic levels. An example of such a presentation of ecosystem structure is in Weiss et al. (2014) where modelled diversity, species richness, and biomass of grasslands under conditions of low and high abiotic resource availability were compared to empirical measurements. Trends found for responses to resource availability may also be useful, as in Jakoby, Grimm & Frank (2014), where the relationship between precipitation and green biomass production was used as a pattern. In ecotoxicological ABMs, dose-response curves are often used to represent ecosystem structure. These curves quantify and characterise effects of differing concentrations of toxicant exposure on species metrics such as body size, reproductive rates, survival, or population growth (Meli et al., 2013;Johnston et al., 2014;Desforges, Sonne & Dietz, 2017). Examples of dose-response curves used as POM patterns are Desforges et al. (2017), who used the relationship between maternal liver concentration of polychlorinated biphenyls (PCBs) and kit survival to evaluate toxicant impacts on metabolic processes in the mink (Mustela vison) and Johnston et al. (2014) who used the effects of varying concentrations of the pesticides chlorpyrifos and copper oxychloride on earthworm (Eisenia fetida) mass and reproductive outputs as a pattern for model evaluation. Guidance for pattern-oriented modelling level models interested in carbon and nitrogen dynamics, structure of ecosystems can also be presented from this perspective. Carbon stocks and balances have been used in forest models, such as Fauset et al. (2019) who evaluated an Amazonian forest model using empirical carbon cycling measurements derived from wood, leaf, and root biomass and respiration.
System structure can additionally be characterised using patterns in the interactions between individuals and their environments. A simple interaction index can be estimated as the frequency and number of unique interactions. More sophisticated interaction metrics include network statistics such as nestedness and modularity. Nestedness is a nonrandom interaction pattern of bipartite networks, in which interactions of specialist nodes are a subset of the interactions of generalist nodes forming a core of highly connected generalists (Dupont et al., 2004;Bascompte, 2009). Examples of generalist nodes can be flowers visited by many pollinators or a water source used by many animals. Modularity is the degree to which a network is organised into subgroups of nodes that interact more among themselves than with nodes from other subgroups, i.e. modules (Dupont et al., 2009). Network statistics have been used as patterns in modelling plantpollinator networks (Newton et al., 2018;Chudzinska et al., 2020).

(6) Dynamics
Dynamics of species provide another useful source of patterns that document how individuals and their environments change in time. Dynamics may be evaluated seasonally, be period-specific, or follow long-term trends. These patterns can be observed for all presented hierarchical levels.
Patterns in the dynamics of individuals are presented as temporal changes in individual characteristics, such as their morphology, behaviour, or life history. Changes in mass or body fat of individuals are commonly used dynamic individual-level patterns that are often considered in models with an energetic component (Amano et al., 2006;Johnston et al., 2014;Desforges et al., 2019). For example, Desforges et al. (2019) used seasonal changes in body mass of wild muskoxen (Ovibos moschatus) for model evaluation. Changes in animal behaviour may be presented as temporal variation in activity patterns, as seen in Beltran et al. (2017) and Halle & Halle (1999).
Population dynamics patterns often document changes in size, represented by abundance or density, or vital rates of populations. Abundances can be presented as the number of individuals or total biomass and are often generalised for the entire population or region, while densities are specific to a unit area (Liu et al., 2013;Pais & Cabral, 2017;Boult et al., 2018). Both can be presented for different cover or habitat types to visualise relative differences, as in Durell et al. (2006) and Amano et al. (2006). These patterns are often displayed monthly to capture seasonal dynamics (Mullon et al., 2003) or for longer time series to follow interannual variability (Boult et al., 2018). Characteristics of population cycles, when present, can be useful patterns in themselves, such as amplitude or frequency of cycles (Radchuk, Ims & Andreassen, 2016) and, for growing or shrinking populations, population growth rates can be used for evaluating models or for understanding interannual fluctuations (Wang & Grimm, 2007;Testa et al., 2012).
Patterns related to changes in population vital rates (e.g. survival, reproduction, and mortality) can be useful for informing or evaluating dynamics (Wang & Grimm, 2007), or for modelling processes such as recruitment (Swanack et al., 2009). These rates can be presented for entire populations or be refined by demographics, such as by presenting mortality or reproductive rates of different age classes or survival rates for each sex (Stillman et al., 2000;Rossmanith et al., 2007;Rughetti et al., 2017). Rates of production or pregnancy, reproductive success, and juvenile survival can be sensitive to environmental changes or disturbance, making these patterns useful for models focused on effects of change (Liu et al., 2013;Zurell et al., 2015;Boult et al., 2018;Boyd et al., 2018;Goedegebuure et al., 2018). Density-dependent mortality in ABMs is usually not explicitly considered, but instead emerges from competition for resources among individuals, making patterns of densitydependent mortality rates of species particularly valuable when available (Stillman et al., 2000;Seidl et al., 2012;Zurell et al., 2015). In addition, if focused on modelling metapopulation dynamics (e.g. McIntire, Rompré & Severns, 2013), rates of immigration and emigration can also be useful patterns.
Community dynamics patterns are formed by changes in community structure over time and can emerge from interactions among modelled species. Dynamics of community-specific metrics such as relative abundance, biomass, richness, or diversity of species (or functional types) are often used in community-level models as they allow for the visualisation of changes in community composition (Berger et al., 2006;Weiss et al., 2014;Barnes & Clark, 2017). Predator-prey cycles have proved a very popular pattern used in community-level ABMs that include predation (Topping, Høye & Olesen, 2010b;Topping, Dalkvist & Grimm, 2012;Chivers et al., 2014;Haafke et al., 2016). These cycles can reflect overall abundance of species or episodic presence dependent on local resources (Railsback & Johnson, 2011). The oscillatory behaviour of predator-prey cycles can be characterised by metrics including amplitude, symmetry, mutual dependency, or stability (Chivers et al., 2014). Additionally, dynamics of mortality can be useful patterns as they can identify changes in predation rates over time (Haythorne & Skabar, 2012;Dey et al., 2017) and, in food web models, mediate emerging trophic levels of species (Travers-Trolet et al., 2019). For models of parasitism, mortality and infection rates of hosts are often used as patterns that can be influenced by both biotic and abiotic factors (Railsback & Johnson, 2011;Honkaniemi et al., 2018). Changes in community dynamics may be observed through coverage or presence patterns, as in Brandt & McManus (2009) where changes in coral community cover over a Biological Reviews (2021)  five-year period were used as a pattern in a coral disease ABM.
Ecosystem-level temporal changes in the interactions between abiotic and biotic components in environments form useful patterns. In Cipriotti et al. (2014) seasonal dynamics of water content of shallow soils were used to evaluate potential models of intraspecific competition in grass-shrub steppes. When modelling carbon and nitrogen dynamics, models of microorganisms, such as bacteria and phytoplankton, generally use patterns in the changes of available and stored carbon and nitrogen over time and often consider different molecular sources of these elements (Banitz, Gras & Ginovart, 2015;Hellweger et al., 2016). Patterns in the dynamics of ecosystems can also be useful in ecotoxicological models, as in Johnston et al. (2014), where changes in body mass of individual earthworms (Aporrectodea caliginosa) were tracked after exposure to differing levels of two toxicants.

(7) Distributions
For models with a spatial component, species distributions can contain important patterns as they emerge from interactions between individuals and their environments. Distribution patterns can be gathered empirically using surveys and are often visualised using maps of presence-absence or occupancy, or are processed into surface patterns or distribution models. Here we present distribution patterns observed for the population, community, and ecosystem level.
Patterns of population distributions are commonly used to develop and test spatially explicit population models. Messager & Olden (2018) used population-level presence-absence distribution maps from surveys as a pattern when predicting spread of an invasive crayfish species (rusty crayfish; Faxonius rusticus) in a river system. Distributions are often combined with local abundances or densities to characterise where and to what extent locations are used by species Chudzi nska et al., 2016;Heinänen et al., 2018). They can also be visualised both in time and space, as in Le Fur & Simon (2009), where spawning distribution of round sardinella (Sardinella aurita) was compared to empirical knowledge both by location and by month of the year. Vegetation models often rely on point patterns to visualise distributions given that individuals do not move beyond seed dispersal (Wiegand & Moloney, 2013;Cipriotti et al., 2014). In Jeltsch, Moloney & Milton (1999), point patterns of savanna trees from aerial photographs of the southern Kalahari were compared to simulated point patterns to understand patterns of tree distribution better. Additionally, rather than using distribution patterns to ensure that species occupy the same locations in space as observed, they can be used to investigate the formation of general patterns observed in nature, such as the mosaics formed by beech trees of different developmental stages in Rademacher et al. (2004).
Fully mapped vegetation plots, such as those of ForestGEO (forestgeo.si.edu), provide a rich source of plant distribution patterns. In each forest plot (typically 25 to 50 hectares), all trees exceeding 1 cm diameter at breast height are tagged, measured, identified to species, and recorded in a data set. ForestGEO is one example of an increasing number of data sources containing spatial distribution patterns gathered for individual organisms. The wealth of knowledge contained within these data sets allows for the elucidation of a vast number of patterns at several hierarchical levels (e.g. May et al., 2015).
When modelling communities, many useful patterns can be found in distributions of species and in the relationships, or lack thereof, among species occupancy patterns. Rather than using individual species abundance or density, as are used for populations, when modelling multiple species, community metrics such as species richness can be used for visualising distribution patterns (Rangel et al., 2007). These patterns can be derived empirically using various approaches including aerial surveys, forest inventories, or remote sensing (Groeneveld et al., 2002;Wild & Winkler, 2008;Rödig et al., 2017). In plant community models, patterns of coexistence have proved useful patterns as they provide insight on interactions among species. These patterns are usually displayed as abundances, densities, or probabilities of presence which visualise species zonation and co-occurrence in specific locations or along environmental or topographic gradients (Groeneveld et al., 2002;Winkler & Peintinger, 2014). Community distributions can also be used to replicate spatial pattern formation, often visualised using presence-absence or point patterns (Wild & Winkler, 2008;Cipriotti et al., 2014). These distribution patterns can be used to study spatial associations and arrangements typical of the system, as in Wild & Winkler (2008) where the formation of a mosaic pattern in krummholz-grassland, visualised using aerial photography, was reproduced by a model investigating the coexistence of these groups. Distribution patterns on the ecosystem level have been used to relate the spatial arrangement of organisms to abiotic anthropogenic stressors. In Nabe-Nielsen et al. (2018), densities of harbour porpoises at different distances from an underwater noise source over different time intervals was used as a pattern for calibrating the animals' responses to noise. In Semeniuk et al. (2014), distances between boreal caribou (Rangifer tarandus caribou) and industrial features were used to evaluate potential model scenarios.

IV. HOW SHOULD SIMULATIONS AND OBSERVATIONS BE COMPARED?
(1) Comparing observed and simulated patterns The approach used to determine model fit to an observed pattern depends on (a) whether the emergent pattern is at the same resolution as the observed pattern, (b) the nature of the pattern (e.g. whether it is qualitative or quantitative), (c) its dimensions (e.g. temporal or spatial), and (d) which modelling phase the pattern is being used in (see Section II. Where). Points b and c are not mutually exclusive, and patterns of any dimension can be analysed qualitatively or Biological Reviews (2021)  Guidance for pattern-oriented modelling quantitatively. Extensive literature exists on the different approaches used to assess model fit to observations, particularly related to statistical analyses (e.g. Hilborn & Mangel, 1997;Burnham & Anderson, 2002;Waller et al., 2003;Hartig et al., 2011). Below we present some commonly used methods in POM-ABMs.
(a) Resolution and resampling When comparing model results and empirical observations, it is important to ensure that results are 'collected' in a similar manner (e.g. matching sampling rate and sample size). Challenges with collecting observations empirically should be considered when comparing observations to model outputs as model results often have a higher temporal or spatial resolution than observations and do not contain missing information (Fig. 4A). Moreover, individuals sampled in the observed patterns usually represent a subset of the study system and may be biased towards more easily observable individuals or associated with large errors and uncertainties. Because ABMs conversely enable constant monitoring of entire populations or communities, it is important to consider whether modelled results should be resampled when evaluating pattern fit, as exemplified by Jeltsch et al. (1999) and discussed by Wiegand et al. (2003). Matching resolution is especially important for statistical comparison of patterns. Results of classical statistical tests (such as P-value based statistics) may be misleading as they are influenced by resolution of the observations so actual differences or similarities between patterns can be skewed by sample size (White et al., 2014). The virtual ecologist (VE) approach (Zurell et al., 2010) can be used to sample model outputs in a manner that is reflective of the process, methodological tools, and biases associated with collecting information empirically (Zurell et al., 2010;Grimm & Railsback, 2012).

(b) Qualitative and quantitative comparisons
As a first step, visual, or qualitative, comparisons should be used to evaluate model fit. Qualitative comparison is helpful for assessing whether the patterns fall within the range of observations or if model results are reflective of system trends (Fig. 4B). Categorical comparisons are often used in POM, where model results are compared to a specific range, such as a confidence interval (Railsback & Grimm, 2019). If outputs fall into the range, they are classified as fitting. Categorical comparisons are often preferable to 'best-fit' as, rather than optimising results to a single value, they consider uncertainty in observations. Additionally, a categorical classification approach can be used to retain all model parameterisations that fit the pattern, allowing for consideration of parameter uncertainty. Visual comparison may be useful for comparing spatial patterns (Kułakowska et al., 2014), such as distribution core areas or kernel home ranges (Chion et al., 2011). When distributions are highly scattered or aggregated, quantitative comparisons such as percentage overlap may be misleading [see discussion in Fieberg & Kochanny (2005) and Winner et al. (2018)]. Qualitative comparison is common in theory development where observed patterns are frequently hypothetical or focused on trends, as exemplified by Smith & Huston (1989) where plant functional types were studied based on adaptations for simultaneous use of water and light. Qualitative model fits that agree with observed patterns can be described as being 'consistent' with empirical knowledge (i.e. not deviating too much from observations; Liu et al., 2013).
Taking a POM approach to quantitative model assessment can be an effective method of overcoming major points of criticism against ABMs, such as uncertainty in model parameter values, error propagation, and ad hoc methods of model selection (Martínez et al., 2011). Quantitative comparison can be used to measure differences between simulated and observed patterns and evaluate model performance. These comparisons may be particularly useful for elucidating values of unknown parameters by calibration, or 'inverse modelling', or for identifying underlying system mechanisms in model selection (see Section V. When for details on these modelling stages). Model evaluation and qualification may also benefit from an explicit quantification of how well model outputs match the observations, also called goodness-of-fit, which can be used for model testing or to compare model fit between different applications. Quantitative assessment of model fit occurs following three primary steps (as presented in Hartig et al., 2011). First, data must be reduced in dimensionality to summary statistics which act as patterns. Summary statistics should be minimally sufficient, meaning that they are reduced in complexity as much as possible without losing information that is pertinent for model calibration or selection (Hartig et al., 2011). For example, extraction of primary path metrics from movement tracks (e.g. distribution of step lengths and turning angles, Fig. 4A) may act as sufficient summary statistics which can be useful for calibrating movement behaviour. Simulated summary statistics should be extracted in the same way as the observed patterns to ensure that they can be compared directly (as in the VE approach).
Second, a distance measure should be defined to compare the difference between observed and simulated summary statistics, this serves as an approximation of the likelihood function. Calculated errors are often used as distance measures in POM-ABMs ( Fig. 4C; Kramer-Schadt et al., 2004;Chudzinska et al., 2020;Gallagher et al., 2021). Frequently used are mean absolute error (MAE) and root mean squared error (RMSE). Both measurements express average model prediction error in units of the variable of interest. Additionally, fit criteria must be established for determining what distance between a specific parameter combination or model structure and the data can be considered acceptable. Fit criteria are useful for categorising parameter values or model structures as likely or unlikely based on the goodness-of-fit of their outputs compared to observations (Hartig et al., 2011). Observed data can be used to define ranges for summary statistics, also known as rejection filters, which correspond to an acceptable level of model fit (e.g. Cipriotti et al., 2012). When permutation tests are used, these criteria can take the form of percentile-based thresholds .
The third step is effectively to sample parameter space or alternative model structures using efficient sampling and optimisation techniques. This step is particularly relevant for inverse modelling, where it can be beneficial to identify the maximum or shape of the likelihood or posterior densities as a function of parameter values (Hartig et al., 2011). This computationally intensive task can be carried out more efficiently using two general types of algorithms. Optimisation algorithms are used to identify parameter values which yield the tightest fit to observed patterns, while sampling algorithms, such as Approximate Bayesian Computation (ABC) and Markov Chain Monte Carlo (MCMC) or particular filters, can be used to explore the shape of the resulting likelihood or posterior densities (Martínez et al., 2011;Cipriotti et al., 2012;Hartig et al., 2014;Boult et al., 2018). Rejection-ABC selects parameter values randomly from large distributions of possible values and uses rejection filters for identifying likely parameter combinations (van der Vaart, Johnston & Sibly, 2016). This approach can retain multiple patterns as summary statistics for simultaneous fit assessment. For example, Boyd et al. (2018) used rejection-ABC to parameterise mortality of eggs and larvae and the strength of predator density dependence using three population-level patterns in a model of mackerel (Scomber scombrus) population dynamics. See Hartig et al. (2011) for detailed discussion on maximum likelihood and Bayesian method use in quantitative pattern analysis.
(c) Dimensions of patterns Patterns can occur in either, or possibly both, temporal and spatial dimensions and this influences how a pattern is analysed. Here we define temporal patterns as those that are measured repeatedly in time, and spatial patterns as any pattern containing location-related information. Additional pattern dimensions may be present, such as system state, however, here we focus on spatial and temporal dimensions as they may require specific analytical methods.
Processes in most ABMs occur in discrete time steps or events (Railsback & Grimm, 2019). Accordingly, many patterns used in POM-ABMs relate to time. Temporal patterns may take the form of seasonal or long-term trends and can be analysed following a variety of approaches, but time series analysis may be useful for describing and evaluating these patterns (Fig. 4D). When comparing observed and simulated time series, many modellers opt to inspect fit qualitatively (Seidl et al., 2012;Liu et al., 2013;Johnston et al., 2014; Biological Reviews (2021)  Guidance for pattern-oriented modelling Stenglein et al., 2015). While visual inspection may be sufficient in some cases to ensure that model outputs reproduce general trends present in observations, in many cases modellers must perform statistical analyses to evaluate model goodness-of-fit. Simple permutation tests may be useful when establishing pattern fit for a time series, as in Wiegand et al. (2004). Time-domain methods in time series analysis can be used to extract patterns from time series, such as autocorrelation analysis (Radchuk et al., 2013). Various error measurements may be used as they allow for error to be evaluated at each time point; examples include error measurements mentioned above in Section IV.1b (MAE and RMSE) and additional measures such as the sum of standardised squared errors (SSSE), the mean relative error (MRE), symmetric mean squared error (SMSE), and residual sum of squares (RSS) Marques et al., 2019;Gallagher et al., 2021). Other metrics that have been used when comparing time series are index of agreement, correlation coefficient, and coefficient of determination Boult et al., 2018;Heinänen et al., 2018;Desforges et al., 2019).
Patterns that vary spatially can contain information on specific positions, distances, areas, and other metrics derived from these measures. While most ABMs are restricted to two spatial dimensions, some focus on one-or three dimensions (e.g. Strand, Huse & Giske, 2002;Rademacher et al., 2004). Patterns relating to movement metrics, species aggregations, spatial autocorrelation, and total space use are often characterised spatially. These sources may require spatial statistics to understand and compare their relationships, such as percentage overlap of home ranges or territories (Chion et al., 2011;Nabe-Nielsen et al., 2013), fuzzy Kappa statistics (Pérez, Dragi cevi c & White, 2013), or quantification of changes in distributions, habitat selection, or individual movements, such as distance travelled or displacement (Fig. 4E). However, as mentioned above, qualitative comparison of these results is also recommended.
(2) Using multiple patterns Although using multiple patterns in POM-ABMs can produce structurally realistic models that capture generative processes of a system, it can be challenging to evaluate fit simultaneously for a set of patterns. For multipattern assessment, fit criteria for measuring differences between simulated and observed summary statistics must first be clearly defined for each pattern. The criteria selected for determining acceptable model fit are needed to dictate how a simultaneous evaluation of pattern fit can be carried out. For example, a time series of animal counts may contain three patterns, forming different levels of information. A weak pattern could be found in the range of count observations. The acceptance criteria for such a pattern could be to accept all simulations for which the outputs fall within the observed range, so that only extreme cases are excluded. Alternatively, if using a permutation test to evaluate pattern fit, as in Wiegand et al. (2004), extreme cases can be excluded by accepting all simulations for which the calculated distance measure is smaller than that of 95% of the randomised simulations. If a trend is present in the time series (e.g. a growth rate) this trend can form a pattern. The fit criteria can be defined using the permutation test as a threshold corresponding to simulations for which the distance measure for growth rate is less than that of 5% of randomised simulations. This criterion will select for simulations which perform better than would be expected by random chance. Finally, if evidence exists for seasonal or short-term fluctuations in the observations that are tied to environmental processes that are included in the model, a threshold for pattern match may be defined to accept all simulations for which the distance measure (e.g. sum of squares) is again smaller than 5% of randomised simulations.
Once patterns and their fit criteria have been established, an approach is needed simultaneously to evaluate fit of model outputs to each summary statistic. Model fit can be established by simply using a series of categorical comparisons or rejection filters to identify which patterns each version of the model or parameter set reproduces (Railsback & Harvey, 2002;Railsback & Johnson, 2011). A global ranking method can assess fit of multiple patterns, whereby fit metrics are combined into a single 'global ranking' criterion, such as by computing errors (Chion et al., 2011) or developing cost functions (Thiele, Kurth & Grimm, 2014) for all patterns. The ranking is the sum of ranks computed according to chosen criteria for all patterns, after which global ranking can be used to select the best model or parameter combination. Another option, which can be used for both quantitative and qualitative patterns, is to order or weigh patterns from most to least biologically important (or most related to the model's purpose) or from strongest to weakest. Preference (i.e. a higher weighting) should be given to parameter combinations or model scenarios which best reproduce the most important and strongest patterns (e.g. Stenglein et al., 2015). While weak patterns are usually less specific and therefore often easier to reproduce, multiple weak patterns, observed at different hierarchical levels, can often achieve higher structural realism with less effort than focusing on only one strong pattern Grimm & Railsback, 2012). Such an approach is exemplified by Chudzi nska et al. (2016) and Railsback & Johnson (2011).
Nevertheless, recent statistical methods should be applied with care as likelihood functions of ABMs can often not be calculated explicitly. For this reason, it can be difficult to couple ABMs with maximum likelihood and Bayesian approaches. Hartig et al. (2011) introduce a couple of methods (e.g. ABC) as alternatives.
'Training' and 'test' sets can be used when sufficient data exist to partition each pattern into two sets, one for use in model parameterisation and one for evaluation. For example, in Boult et al. (2018), rejection-ABC was used to calibrate 11 unknown parameters for elephants using patterns from one population as a 'training' set and then evaluated the resulting model against a 'test set' of the same patterns for additional elephant family groups.

V. WHEN IN THE MODELLING CYCLE ARE PATTERNS USED?
When developing an ABM, POM can be used in all modelling phases, depending on the model's purpose and information available. The modelling cycle can be broken up into two major phases: (1) model development, and (2) testing (Fig. 5). Model development involves selecting patterns related to model design (i.e. using known attributes and mechanisms of the system to develop model structure; Wiegand et al., 2003), parameter selection (i.e. determine parameter ranges and values either directly from the data or by using inverse approaches), and/or model selection (testing hypotheses following the ABM theory development cycle; Railsback & Grimm, 2019). Model testing includes using independent patterns that were not considered during model development (also called 'secondary predictions') for model evaluation (using patterns to demonstrate model accuracy, also called validation; Rykiel, 1996) and model qualification [using patterns to discern model performance beyond the spatiotemporal scale for which it was originally developed; also called corroboration (Augusiak, Van den Brink & Grimm, 2014), revalidation (Rykiel, 1996), replication (Wilensky & Rand, 2015) or reimplementation (Grimm & Railsback, 2012)].
While the approach to ABM development and the modelling cycle have been thoroughly covered previously (Grimm & Railsback, 2012;Railsback & Grimm, 2019), we discuss here some general trends that may be helpful when deciding when to employ specific patterns. Ideally a sufficient quantity of patterns would be available for the modelled system such that patterns could be used selectively, however in reality, data are often sparse and modellers must take advantage of all available knowledge. However, the point in which a pattern is used (e.g. during model development or testing) can be decided strategically based on the nature of the pattern so that each pattern is employed most effectively.
In model design, it is important to use patterns that are relatively well understood and documented in terms of the mechanisms that lead to reproducing the pattern. Individual-level patterns are often used here as they can help identify processes that should be included in the model. Patterns that are intended to be used at later stages in the modelling cycle can be considered in model design to establish the spatiotemporal resolution of the model, which can help to reduce the need for resampling later when evaluating model fit. For designing and testing submodels, we recommend using patterns qualitatively as qualitative analysis can show clearly whether the model includes the necessary mechanisms. At this point using clearly defined qualitative criteria is advantageous as they are easily reproducible and not dependent on parameterisations and calibrations done later in the modelling cycle, making qualitative analysis more convincing than quantitative analysis which could potentially be misleading at this stage.

Guidance for pattern-oriented modelling
When selecting values of unknown or uncertain parameters, well-defined patterns may be used to calibrate a model. Model calibration often necessitates using hundreds or thousands of simulations and potentially numerous patterns, so it may prove beneficial at this stage to use patterns that can be evaluated quantitatively. Patterns used for parameter selection may fall on any hierarchical level but should be directly related to the processes being calibrated, e.g. using energy intake patterns to calibrate energy budget parameters.
Model selection is functionally similar to model parameterisation, except, rather than parameters, mechanisms are unknown. Less well-understood and qualitative patterns may be used here to evaluate potential mechanisms of a pattern or model processes and to test hypotheses (Jeltsch et al., 1998). Like parameter selection, patterns used here should be related to processes involved in the alternative models being tested.
Once model development is complete, POM-ABMs should evaluate emergent secondary predictions of a model using independent patterns spanning all levels of organisation, although patterns on higher levels, i.e. population, community, or ecosystem, are most often used. Temporal and spatial patterns may be used at this stage to test the fit of simulation outputs in space and time. For qualification, ABMs based on general principles can be applied to alternative regions, species, or conditions and here reimplementation can be tested using general patterns. Patterns that are readily available or obtainable for different conditions or systems should be used so that the model can easily be reapplied and evaluated using the same set of patterns.

VI. POTENTIAL PITFALLS
While POM is an established and systematic framework, the inherent subjectivity in the application of the approach may lead to several potential pitfalls.
Patterns observed in systems yield insight on their internal processes and, as such, it is crucial that suitable patterns are identified for model development and testing using the model's purpose as a filter so that processes which are not relevant to the model's purpose can be omitted. For instance, if the model's purpose is to forecast the population abundance of a species under altered conditions, such as with future land-management scenarios, population-level patterns of the structure and dynamics of the population should be used when available. Additionally, if the model can replicate multiple versions of these patterns under various conditions, including natural and managed ecosystems, this provides support that the model is flexible enough and that it includes the necessary mechanisms for predicting future population dynamics under altered conditions. By using relevant patterns as a metric for determining the structure of a model, POM can help in deciding what level of complexity is necessary for an ABM Grimm & Railsback, 2012). The systematic classification of commonly used patterns presented herein can facilitate the identification and inclusion of relevant patterns for many model purposes.
Patterns from all relevant hierarchical levels must be used when available. By not including patterns occurring on all levels considered in the model, the link between lower and higher levels may be missed. The ability to span this ecological 'micro-macro' link is an especially relevant strength of POM and enables realistic simultaneous consideration oftop-down and bottom-up interactions occurring among different hierarchical levels. Missing this micro-macro link, which drives system processes, by not including patterns on all appropriate levels may lead to uninformed predictions.
When testing models, it is important to ensure that the patterns used are truly emergent outputs of the model rather than imposed relationships. In this stage, the ability of the model to replicate empirical patterns based on low-level processes is being evaluated, so model outputs used here must be emergent.
Another error that can be made is model overfitting. By including many patterns only in the parameterisation process and calibrating parameters to fit these patterns, models can become overfitted which diminishes their ability to be reapplied to new systems or under alternative conditions. Overfitting models to patterns can be avoided by reserving a subset of patterns to be used in model evaluation and, when relevant patterns are available, model reimplementation can be used as a method for assessing model flexibility. Overfitting can also occur when criteria used for determining model fit are too narrow or defined based on uninformative stochastic noise. Fit criterion should be flexible enough to include potential uncertainty due to observer bias and interpretation errors contained within patterns, so that otherwise wellperforming models are not excluded solely because they do not fit noise. Understanding the actual pattern within data, rather than forcing a model to fit every data point precisely, is key for avoiding such problems.
Using too few patterns can also potentially be problematic. Too few patterns may lead to reduced structural realism of a model and to the common issue of equifinality, where many model structures and parameterisations are all found to reproduce empirical patterns. However, using too few patterns may be unavoidable for data-poor systems. It is up to the individual modeller to be aware of the potential for equifinality and to decide how to use the patterns available to them; higher level patterns should be reserved for model evaluation whenever possible to allow for testing of emergence in the model.
The 'wolf-sheep predation' model (Wilensky, 1997;Wilensky & Reisman, 2006) of predator-prey population dynamics reproduces classical Lotka-Volterra (LV) dynamics (Lotka, 1926;Volterra, 1926), but uses only a single pattern, cyclical population dynamics of predators and prey. Cyclic changes in prey and predator abundance, as described by the LV model, lead to the conclusion that a system is only stable if the population of predators is controlled by the number of prey, which is then controlled by the amount of Biological Reviews (2021)  available food and number of predators. In nature many factors contribute to predator-prey dynamics including body condition, age, phenotype, social structure, and densitydependent processes in both predator and prey (Stahler et al., 2013;Smout, King & Pomeroy, 2020), along with interactions within the community, such as anti-predatory behaviour (Abrams & Matsuda, 1997). While the basic wolf-sheep model predicts cyclical predator-prey dynamics, it does so under conditions which ignore these known factors. The model has since been improved by including processes such as density dependence and predator-prey behaviour, while still maintaining cyclic conditions (De Roos, McCauley & Wilson, 1991;Cuddington & Yodzis, 2000). This demonstrates how investigating a complex system through "sets of conceptually simple mechanisms" (Grimm et al., 2005, p. 991), as opposed to using simple analytical equations, can result in a more rigorous and tangible explanation of the system. This typifies the change in our thinking of scientific theory described by  enacted by the POM approach.

VII. CONCLUSIONS
(1) By considering empirical patterns when developing models, POM can facilitate some of the traditionally most challenging modelling stages. Patterns are particularly useful as criteria for deciding what should or should not be included in a model. This ensures that the model is "as simple as possible but as complicated as necessary" to address its purpose (Sun et al., 2016, p. 56). Patterns can be used to assist with making decisions on how to represent individual behaviour. By considering empirical knowledge in this process, the behaviour of modelled individuals can be developed in a manner that allows for testing which low-level processes may underlie behaviour. Modelling individual behaviour based on low-level processes is a primary motivation for developing ABMs (Railsback & Grimm, 2019) and explicit use of POM in this process allows for a better understanding of the mechanisms involved and ensures that behaviours are reflective of empirical observations. (2) The pattern classification and guidelines provided here are not only applicable to the modelling process but can also be used to improve how we collect empirical information by identifying knowledge gaps. Such an approach can be mutually beneficial to both modelling and empirical science by iteratively informing the steps necessary to gain a better understanding of the system being investigated. Two appropriate approaches which could be employed in this manner are: (i) Pattern Oriented Sampling (POS) (Briant et al., 2018) which aims to focus on key elements of the system in question and collect field observations that can then be compared with model outputs; (ii) the 'virtual ecologist' approach (Zurell et al., 2010) (see Section IV.1a) which outlines the potential of using models to inform further collection of empirical observations and refine field efforts by testing collection methodologies.
(3) The ability of POM to standardise critical modelling practices, like reimplementation, can push forward ABM theory (Lorscheid et al., 2019) by facilitating the development of generic reusable model components.
The establishment of a suite of submodels which have been successfully confronted with empirical data can expedite collective progress by accelerating individual model development. We advocate for the explicit use of POM to simplify the process of comparing the success of different approaches and submodels in alternative applications and under variable conditions. (4) With rapid developments in the accessibility and use of both complex systems models, in particular ABMs, and technologies for the collection and analysis of observations of natural systems, the POM approach provides a powerful method for coupling these methodologies for the production of robust predictive tools. Innovative statistical methods can optimise POM-ABMs by elucidating multi-level patterns from increasingly large data sets. (5) The POM-ABM approach is widely applicable across scientific disciplines, as characteristics like level, structure, dynamics, etc. can be relevant to any collection of agents. (6) Adoption of POM may be limited due to the difficulty in recognising and applying patterns but expanding its use could lead to the production of models with better predictive power, enhanced realism, and improved credibility. In an effort to make POM more accessible and promote strategies for its application, we provided guidance for identifying and using patterns following the POM approach and highlight a broad range of ecology-based examples. We hope that with this resource, along with the examples in Appendix S1, we can make POM more accessible and help to improve complex systems modelling in general by inspiring new ideas for pattern identification and use.

VIII. ACKNOWLEDGEMENTS
We are grateful to Steven F. Railsback and Volker Grimm for their guidance, constructive comments, and valuable suggestions throughout the development of this resource. We would also like to thank Darren Miller and Erik Schilling for reading and providing useful comments on an earlier draft of the manuscript. The idea for this review was born out of the 12th international summer school in individualand agent-based modelling held in 2018 by TU Dresden. We would like to thank the instructors of the course and