## Introduction

Density dependence in animal populations affects important ecological considerations such as per capita growth rates and extinction risk (Courchamp *et al.* 1999; Sabo *et al.* 2004; Kramer & Drake 2010). Annual census data are often used to estimate the strength of density dependence (e.g. Sibly *et al.* 2005; Gregory *et al.* 2010), but this approach introduces the possibility of observation error. Ignoring or misspecifying observation error in ecological time series creates bias in parameter estimates and may lead to spurious conclusions of density dependence when fitting models of population dynamics (Dennis *et al.* 2006; Freckleton *et al.* 2006; Knape 2008).

State space models have emerged as the pre-eminent approach to accommodate observation error and are now a popular and well-recognized tool for inference in both linear and nonlinear models of population dynamics (e.g. de Valpine & Hastings 2002; Calder *et al.* 2003; Clark & Bjornstad 2004; Staples *et al.* 2004; Buckland *et al.* 2007). State space models (SSMs) typically consist of a simplified process model of population dynamics for the unobserved ’true’ population sizes, which are known as latent states, coupled with an observation model that describes the discrepancy between observations and the modelled latent states. Just as a chosen process model necessarily approximates the diversity of factors that affect a population's dynamics, an observation model approximates the diversity of factors that affect the data generation mechanism.

Freckleton *et al.* (2006) review many potential sources of observation error in animal census data. The disparate sources of observation error include *sampling error*, which is generated in the process of subsampling populations (Morris & Doak 2002; Dennis *et al.* 2006; Freckleton *et al.* 2006); *observer error*, where observers sample differently when counting individuals (Spearpoint *et al.* 1988; Cunningham *et al.* 1999; Moore *et al.* 2004) or misidentify species (Miller *et al.* 2011); *metapopulations*, where the observed population is open to immigration and emigration (Spearpoint *et al.* 1988; Werham *et al.* 2002; Freckleton *et al.* 2006); *heterogeneity*, where populations and sampling effort may vary in space and time (Recher 1989; Morris & Doak 2002; Dennis *et al.* 2006); *proxies*, where indices of population size such as nesting sites or number of breeding adults may be used (Werham *et al.* 2002; Freckleton *et al.* 2006); *detectability*, where population detectability varies with rates of breeding failure (Green & Hirons 1988), working conditions (Morris & Doak 2002; Dennis *et al.* 2006), weather (Spearpoint *et al.* 1988; Morris & Doak 2002; Freckleton *et al.* 2006) and interactions between methodology and habitat type (Wolfe & Kimball 1989; Cunningham *et al.* 1999); and *analysis*, where disparate observational data are synthesized to develop a population estimate over time for a given location (Wilbur 1980; Recher 1989).

Perhaps more abstractly, the appropriate observation model may also vary with the choice of process model because the latent states are jointly determined by the choice of process model, the choice of observation model and the observations themselves (see Methods). For example, a process model that focuses on mate limitation may suggest an observation model that relates it to the number of reproducing females, whereas a process model of intraspecific resource competition may suggest an observation model that relates it to the total number of individuals in a population that compete for various limiting resources.

The wide variety of mechanisms and processes that impinge on population estimates creates the potential for non-Gaussian observation error relative to the modelled latent states. The possibility of non-Gaussian observation error for continuous response data in state space models is fully recognized by the statistical literature (e.g. Kitigawa 1987; Carlin *et al.* 1992; Durbin & Koopman 1997) and more recently in ecology (Buonaccorsi & Staudenmayer 2009; Knape *et al.* 2011). Here, we explore how a parametric observation model that accommodates Gaussian error, bias, skewness and outliers in the observation model may affect the choice of competing process models, estimates of density dependence, estimation and prediction of latent states, and functions of these quantities. We compare this flexible observation model, which is known as the normal inverse Gaussian (NIG; Barndorff-Nielsen & Prause 2001), with the traditional Gaussian observation model that is used routinely for SSMs applied to annual census data (e.g. Clark & Bjornstad 2004; Dennis *et al.* 2006; Knape & de Valpine 2012). We use both simulated data, where the true process and observation models are known, and real data to demonstrate the implications of modelling flexible observation error using the NIG observation model and to develop guidelines for applying the NIG observation model to annual census data in ecological time series.