### Summary

- Top of page
- Summary
- Introduction
- Materials and methods
- Results and analysis
- Discussion
- Acknowledgements
- References
- Supporting Information

**1.** Observation error is the uncertainty in population size that results from not only sampling error but also migration, population heterogeneity, observer error, population interactions with weather and habitat, analysis of observational data, and other sources of error that may introduce Gaussian or non-Gaussian noise between the observed population and its modelled dynamics.

**2.** We investigate the use of the normal inverse Gaussian (NIG) distribution as a model of observation error that flexibly captures processes such as undercounting, overcounting and outlying observations. The NIG distribution captures asymmetry and heavy-tailedness in an interpretable parametric model that includes the popular Gaussian observation model as a limiting case.

**3.** The implications of using the NIG model are explored by fitting nonlinear density-dependent population models with environmental stochasticity to animal census data. We pay particular attention to the estimated per capita growth rate, estimates of future population size and quasi-extinction risk.

**4.** We use Bayes factors to evaluate the support for hypotheses for non-Gaussian observation error model, including priors that represent alternative hypotheses of asymmetry in the observation model. Support for these flexible observation models are contrasted with the special case of a Gaussian observation model.

**5.** Flexible observation error may affect estimates of population per capita growth rates and predictions of extinction risk. The dependence of estimates, and predictions, on the choice of observation model may occur even if the data provide comparable support for both Gaussian and non-Gaussian observation errors. Thus, for some populations with census data, the relative degree of prior belief in alternative hypotheses of process and observation model structure will significantly affect ecological predictions with management implications, such as quasi-extinction risk.

### Introduction

- Top of page
- Summary
- Introduction
- Materials and methods
- Results and analysis
- Discussion
- Acknowledgements
- References
- Supporting Information

State space models have emerged as the pre-eminent approach to accommodate observation error and are now a popular and well-recognized tool for inference in both linear and nonlinear models of population dynamics (e.g. de Valpine & Hastings 2002; Calder *et al.* 2003; Clark & Bjornstad 2004; Staples *et al.* 2004; Buckland *et al.* 2007). State space models (SSMs) typically consist of a simplified process model of population dynamics for the unobserved ’true’ population sizes, which are known as latent states, coupled with an observation model that describes the discrepancy between observations and the modelled latent states. Just as a chosen process model necessarily approximates the diversity of factors that affect a population's dynamics, an observation model approximates the diversity of factors that affect the data generation mechanism.

Perhaps more abstractly, the appropriate observation model may also vary with the choice of process model because the latent states are jointly determined by the choice of process model, the choice of observation model and the observations themselves (see Methods). For example, a process model that focuses on mate limitation may suggest an observation model that relates it to the number of reproducing females, whereas a process model of intraspecific resource competition may suggest an observation model that relates it to the total number of individuals in a population that compete for various limiting resources.

The wide variety of mechanisms and processes that impinge on population estimates creates the potential for non-Gaussian observation error relative to the modelled latent states. The possibility of non-Gaussian observation error for continuous response data in state space models is fully recognized by the statistical literature (e.g. Kitigawa 1987; Carlin *et al.* 1992; Durbin & Koopman 1997) and more recently in ecology (Buonaccorsi & Staudenmayer 2009; Knape *et al.* 2011). Here, we explore how a parametric observation model that accommodates Gaussian error, bias, skewness and outliers in the observation model may affect the choice of competing process models, estimates of density dependence, estimation and prediction of latent states, and functions of these quantities. We compare this flexible observation model, which is known as the normal inverse Gaussian (NIG; Barndorff-Nielsen & Prause 2001), with the traditional Gaussian observation model that is used routinely for SSMs applied to annual census data (e.g. Clark & Bjornstad 2004; Dennis *et al.* 2006; Knape & de Valpine 2012). We use both simulated data, where the true process and observation models are known, and real data to demonstrate the implications of modelling flexible observation error using the NIG observation model and to develop guidelines for applying the NIG observation model to annual census data in ecological time series.

### Discussion

- Top of page
- Summary
- Introduction
- Materials and methods
- Results and analysis
- Discussion
- Acknowledgements
- References
- Supporting Information

We have used the normal inverse Gaussian (NIG) distribution to capture different hypotheses of observation error in ecological state space models, and have found that the choice of observation model affects estimation and prediction in studies of both synthetic and real data. In scenarios where the true observation model is incompletely known and hence possibly misspecified, we demonstrate how to develop interpretable priors for the observation model using the NIG triangle (Fig. 1) and closed form expressions of the NIG's moments (Appendix S1). In real data applications, we find statistical evidence that provides equivalent support for the NIG relative to the Gaussian observation model. The choice of model structure and relative prior belief in competing observation models are therefore important factors that affect estimates of per capita growth rate curves and predictions of population size.

The NIG is one flexible family of models that includes as limiting cases the Gaussian and Cauchy distributions, but other choices exist (e.g. Peters *et al.* in press). The NIG is a useful candidate because it retains interpretability while capturing a broad spectrum of heavy-tailed and asymmetric families of observation models. For instance, most previous studies of annual census data have used the Gaussian observation model (e.g. Clark & Bjornstad 2004; Dennis *et al.* 2006; Knape & de Valpine 2012), and so here we have used a prior that places greater support for low values of steepness in the NIG observation model. The closed form expression of the NIG's standard deviation also allows us to assess comparability with the prior specification for the Gaussian observational model (Appendix S1). The prior for the standard deviation of the observation noise is an important consideration even in linear Gaussian models, where estimating unknown observation and process noise can sometimes lead to peaks in the likelihood surface where either process or observation noise is zero (e.g. Dennis *et al.* 2006; Knape 2008), as was seen in the condor and parrot populations using a linear SSM (Staples *et al.* 2004). Here, in both NIG and Gaussian observation models, we have used a prior constraint on the observation variance that ensures non-negligible observation error (Appendix S1) consistent with a state space model.

Alternative priors on the NIG parameters can propose interpretable observation models that capture different hypotheses of the measurement process. This flexibility comes with the cost of two more parameters relative to the Gaussian special case. This is particularly troublesome for the case of annual census data, which are usually sparse, and for which the use of uninformative priors under the NIG model may lead to overly broad credible intervals of the latent states. Informative priors, perhaps chosen using simulation studies to help visualise and test assumptions, can be adopted to limit the flexibility of the observation model. Importantly, the NIG models allow a much wider range of observational hypotheses than we have space for here and, in general, allow practitioners to develop priors that can be tailored to the population under analysis.

Both synthetic and real analyses are used to establish some guidelines for the application of the NIG observation model in ecological state space models. We use synthetic data to show that Bayes factors provide one possible model selection mechanism to find the ’true’ observation model (Table S2-1, Appendix S2). However, in a second synthetic study that models varying detectability of a closed population, we demonstrate that if the true observation model is unknown, then the fitted and misspecified observation models may both receive similar support from the data. This can occur even if one of the fitted state space models poorly captures the true latent process, as seen in Fig. 3. It is therefore important to consider the relative prior support for the hypothesized NIG and Gaussian observation models.

Bayes factors provide a unified mechanism to incorporate these prior beliefs in model choice with information from the observed data into the model comparison. This approach to model comparison also integrates over the parametric uncertainty rather than depending on an optimized parameter set, such as the maximum likelihood estimate, as in AIC or BIC. For instance, in model *M*_{1}, a trade-off exists between the parameters *r* and θ that controls the strength of negative density dependence near the carrying capacity *K* (Thomas *et al.* 1980). Polansky *et al.* (2009) note that multimodalilty and flat likelihood surfaces can occur under *M*_{1}, even without observation error, and model selection procedures that depend on an optimised parameter set may therefore be misleading. Moreover, Bayes factors naturally account for model complexity and small sample sizes in the model comparison (Kass & Raftery 1995).

The real data analyses provide the case where both the process and observation models are unknown. For our case studies, given a choice of process model, model comparison using Bayes factors does not find overwhelming support for either hypothesis of Gaussian or non-Gaussian observation error (Table S2-4, Appendix S2). Therefore, the choice of observation model and its accompanying prediction of extinction risk will be driven by one's prior belief in model structure, that is, the relative plausibility of asymmetry or steepness in the observation model. In some cases, such as the parrot population, different prior beliefs in alternative observation models will make little difference in estimates of interest, such at the PGR curve or quasi-extinction risk (Fig. 5). In other cases, such as the condor population, different prior beliefs may lead to different predictions, such as quasi-extinction risk, with consequences for informing management decisions. In the case study of the California condor population, for example, weak evidence for an Allee effect is present regardless of the observation model (i.e. *M*_{2} has greater support than *M*_{1}). Nevertheless, given a choice of *M*_{2}, the observation model choice affects predictions of extinction risk (Fig. 4) and its prior probability is therefore relevant.

### Supporting Information

- Top of page
- Summary
- Introduction
- Materials and methods
- Results and analysis
- Discussion
- Acknowledgements
- References
- Supporting Information

Appendix S1. Prior choices for process and observation models.

Appendix S2. Estimation: algorithm details, synthetic analyses and real data results.

As a service to our authors and readers, this journal provides supporting information supplied by the authors. Such materials may be re-organized for online delivery, but are not copy-edited or typeset. Technical support issues arising from supporting information (other than missing files) should be addressed to the authors.

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.