Estimating density dependence and latent population trajectories with unknown observation error


  • Geoffrey R. Hosack,

    Corresponding author
    1. CSIRO Mathematics, Informatics and Statistics, Marine Laboratories, Castray Esplanade, Hobart, TAS 7000, Australia;
      Correspondence author. E-mail:
    Search for more papers by this author
  • Gareth W. Peters,

    1. School of Mathematics and Statistics, University of New South Wales, Sydney, NSW 2052, Australia
    Search for more papers by this author
  • Keith R. Hayes

    1. CSIRO Mathematics, Informatics and Statistics, Marine Laboratories, Castray Esplanade, Hobart, TAS 7000, Australia;
    Search for more papers by this author

Correspondence author. E-mail:


1. Observation error is the uncertainty in population size that results from not only sampling error but also migration, population heterogeneity, observer error, population interactions with weather and habitat, analysis of observational data, and other sources of error that may introduce Gaussian or non-Gaussian noise between the observed population and its modelled dynamics.

2. We investigate the use of the normal inverse Gaussian (NIG) distribution as a model of observation error that flexibly captures processes such as undercounting, overcounting and outlying observations. The NIG distribution captures asymmetry and heavy-tailedness in an interpretable parametric model that includes the popular Gaussian observation model as a limiting case.

3. The implications of using the NIG model are explored by fitting nonlinear density-dependent population models with environmental stochasticity to animal census data. We pay particular attention to the estimated per capita growth rate, estimates of future population size and quasi-extinction risk.

4. We use Bayes factors to evaluate the support for hypotheses for non-Gaussian observation error model, including priors that represent alternative hypotheses of asymmetry in the observation model. Support for these flexible observation models are contrasted with the special case of a Gaussian observation model.

5. Flexible observation error may affect estimates of population per capita growth rates and predictions of extinction risk. The dependence of estimates, and predictions, on the choice of observation model may occur even if the data provide comparable support for both Gaussian and non-Gaussian observation errors. Thus, for some populations with census data, the relative degree of prior belief in alternative hypotheses of process and observation model structure will significantly affect ecological predictions with management implications, such as quasi-extinction risk.


Density dependence in animal populations affects important ecological considerations such as per capita growth rates and extinction risk (Courchamp et al. 1999; Sabo et al. 2004; Kramer & Drake 2010). Annual census data are often used to estimate the strength of density dependence (e.g. Sibly et al. 2005; Gregory et al. 2010), but this approach introduces the possibility of observation error. Ignoring or misspecifying observation error in ecological time series creates bias in parameter estimates and may lead to spurious conclusions of density dependence when fitting models of population dynamics (Dennis et al. 2006; Freckleton et al. 2006; Knape 2008).

State space models have emerged as the pre-eminent approach to accommodate observation error and are now a popular and well-recognized tool for inference in both linear and nonlinear models of population dynamics (e.g. de Valpine & Hastings 2002; Calder et al. 2003; Clark & Bjornstad 2004; Staples et al. 2004; Buckland et al. 2007). State space models (SSMs) typically consist of a simplified process model of population dynamics for the unobserved ’true’ population sizes, which are known as latent states, coupled with an observation model that describes the discrepancy between observations and the modelled latent states. Just as a chosen process model necessarily approximates the diversity of factors that affect a population's dynamics, an observation model approximates the diversity of factors that affect the data generation mechanism.

Freckleton et al. (2006) review many potential sources of observation error in animal census data. The disparate sources of observation error include sampling error, which is generated in the process of subsampling populations (Morris & Doak 2002; Dennis et al. 2006; Freckleton et al. 2006); observer error, where observers sample differently when counting individuals (Spearpoint et al. 1988; Cunningham et al. 1999; Moore et al. 2004) or misidentify species (Miller et al. 2011); metapopulations, where the observed population is open to immigration and emigration (Spearpoint et al. 1988; Werham et al. 2002; Freckleton et al. 2006); heterogeneity, where populations and sampling effort may vary in space and time (Recher 1989; Morris & Doak 2002; Dennis et al. 2006); proxies, where indices of population size such as nesting sites or number of breeding adults may be used (Werham et al. 2002; Freckleton et al. 2006); detectability, where population detectability varies with rates of breeding failure (Green & Hirons 1988), working conditions (Morris & Doak 2002; Dennis et al. 2006), weather (Spearpoint et al. 1988; Morris & Doak 2002; Freckleton et al. 2006) and interactions between methodology and habitat type (Wolfe & Kimball 1989; Cunningham et al. 1999); and analysis, where disparate observational data are synthesized to develop a population estimate over time for a given location (Wilbur 1980; Recher 1989).

Perhaps more abstractly, the appropriate observation model may also vary with the choice of process model because the latent states are jointly determined by the choice of process model, the choice of observation model and the observations themselves (see Methods). For example, a process model that focuses on mate limitation may suggest an observation model that relates it to the number of reproducing females, whereas a process model of intraspecific resource competition may suggest an observation model that relates it to the total number of individuals in a population that compete for various limiting resources.

The wide variety of mechanisms and processes that impinge on population estimates creates the potential for non-Gaussian observation error relative to the modelled latent states. The possibility of non-Gaussian observation error for continuous response data in state space models is fully recognized by the statistical literature (e.g. Kitigawa 1987; Carlin et al. 1992; Durbin & Koopman 1997) and more recently in ecology (Buonaccorsi & Staudenmayer 2009; Knape et al. 2011). Here, we explore how a parametric observation model that accommodates Gaussian error, bias, skewness and outliers in the observation model may affect the choice of competing process models, estimates of density dependence, estimation and prediction of latent states, and functions of these quantities. We compare this flexible observation model, which is known as the normal inverse Gaussian (NIG; Barndorff-Nielsen & Prause 2001), with the traditional Gaussian observation model that is used routinely for SSMs applied to annual census data (e.g. Clark & Bjornstad 2004; Dennis et al. 2006; Knape & de Valpine 2012). We use both simulated data, where the true process and observation models are known, and real data to demonstrate the implications of modelling flexible observation error using the NIG observation model and to develop guidelines for applying the NIG observation model to annual census data in ecological time series.

Materials and methods

Process models

We compare two population dynamic models that capture nonlinear density dependence. Both models, M1 and M2, are stochastic and include a Gaussian distributed process error term that reflects variability in the underlying population growth process, such as environmental stochasticity, that is not captured by the deterministic component of the model. Process error in all of the models is assumed to behave multiplicatively on the natural scale (Clark 2007).

The theta–Ricker model (M1) flexibly captures negative density dependence (Thomas et al. 1980; Turchin 2003). On the log scale this model is

image(eqn 1)

where r > 0 is the intrinsic rate of growth, K > 0 is the carrying capacity and θ > 0 determines the form of density dependence. For instance, model M1 captures the popular Ricker model when θ = 1, but can also allow for negative density dependence to affect the per capita growth rate at population sizes well below the carrying capacity when θ < 1 (Clark et al. 2010). Equation 1 is sometimes called the theta–logistic (e.g. Sibly et al. 2005; Clark et al. 2010), although others reserve that term for an alternative formulation (e.g. Thomas et al. 1980).

A flexible–Allee model (M2) permits both strong and weak Allee effects. On the log scale, this equation is

image(eqn 2)

If A > 0, then the per capita growth rate (PGR: log(Nt/Nt−1)) can be negative at low population sizes, which is a ‘strong’ Allee effect (Courchamp et al. 2008). This effect can occur, for instance, if individuals cannot find mates at a low population density. If −K < A < 0, then the PGR decreases as population size decreases (see Appendix S1), but remains positive, which is a ‘weak’ Allee effect. Model M2 originated in the ecological literature with a study by Lewis & Kareiva (1993) and is discussed in context with other phenomenological Allee models by Boukal & Berec (2002). M2 has been used to survey population time series for demographic Allee effects (Gregory et al. 2010).

Normal inverse Gaussian (NIG) observation errors

The NIG model takes its name from the fact that it represents a normal variance-mean mixture that occurs as the marginal distribution for a random variable Y when considering a pair of random variables (Y,Z) where Z is distributed as an inverse Gaussian inline image and Y conditional on Z is (Y|Z = z)∼N(μ+βz,z) (Barndorff-Nielsen 1997). The resulting density function for the NIG model is given by


where K1[·] is a modified Bessel function of the second kind with index 1 (Olver 1972), inline image and h(y) = ((y)22)1∖2.

The NIG parameters have the constraints inline image. The parameter α is inversely related to the heaviness of the tails, where a small α corresponds to heavy tails that can accommodate outlying observations. The skewness is directly controlled by the parameter β, where negative (positive) values of β result in a left (right) skew, which can accommodate under- or overcounting, and β = 0 is the symmetric model. The location (or translation) of the distribution is given by the parameter μ and the scale of the distribution is given by the parameter δ.

The NIG distribution is a useful model for observation error because it captures many plausible observation models. Notably, when β = 0 and μ is arbitrary, the NIG model asymptotically approaches the Gaussian observation error model inline image as α → ∞. When α = β = 0 with μ and δ arbitrary, the NIG model approaches the Cauchy distribution. It can also approximate the skewness and kurtosis of the log normal, Student's t and gamma distributions, among others (Oigard et al. 2005).

The shape of the NIG distribution can be conveniently summarized with a graphical representation called the NIG shape triangle (Barndorff-Nielsen & Prause 2001). This plot uses indices of steepness and asymmetry, which are analogous to kurtosis and skewness, given by


with 0 < Steepness < 1 and −1 < Asymmetry < 1. Distributions with Asymmetry = 0 are symmetric, and the Gaussian and Cauchy distributions occur as limiting cases for (Asymmetry, Steepness) near (0, 0) and (0, 1), respectively. Figure 1 provides a graphical representation of example NIG probability density functions and their relation to the NIG triangle.

Figure 1.

 The NIG triangle plot shows the diversity of probability density functions and their relation to steepness and asymmetry. For symmetric distributions (Asymmetry = 0), the NIG approaches the Gaussian as Steepness→0 (triangle apex) and approaches the Cauchy as Steepness→1 (top middle of triangle). Each density is standardized to have mean 0 and variance 1; the scales of the y-axes vary to aid visualization. The values of steepness used in the plotted examples are (across rows starting from top left) 0·99, 0·99, 0·99, 0·99, 0·99, 0·99, 0·99, 0·67, 0·67, 0·67, 0·67, 0·67, 0·33, 0·33, 0·33, 0·01 and the corresponding values of asymmetry are −0·99, −0·66, −0·33, 0·00, 0·33, 0·66, 0·99, −0·66, −0·33, 0·00, 0·33, 0·66, −0·33, 0·00, 0·33, 0·00.

NIG and Gaussian likelihoods

Standard SSMs assume that the observations (y1 : T = y1, y2,…, yT) at each time t are independent and identically distributed given the latent state (nt), which is on the log scale in Equations 1 and 2. Hence, the NIG observation model produces a likelihood model denoted by


where the location parameter at each time t is set equal to the latent state (μt = lognt) within the functions g(·) and h(·) as previously defined. The likelihood for Gaussian observation errors is given by


Hierarchical Bayesian modelling for state space models

We construct a hierarchical Bayesian model that allows us to include prior ecological knowledge when performing estimation and inference. The model treats the population size as an unobserved (latent) first-order Markov stochastic process to be estimated at discrete time points t over the time interval [1,T]. Under this assumption, the posterior of interest is

image(eqn 3)

where the parameters associated with the process and observation models are generically denoted as a random vector Ψ; a realization of this vector is denoted in the lower case ψ. The vector of unobserved latent states N1 : T (with realizations n1 : T) will be jointly estimated with the model parameters.


The Gaussian observation model requires a prior for the standard deviation of the observation noise, σω. Following suggestions by Gelman (2006), this is given a uniform prior, inline image. This prior choice ensures that there is at least a 95% chance that the observed population count is within an order of magnitude of the true population size and that there is at least a 5% chance that the observed value is more than c. ±10% of the true population count (see Appendix S1).

The NIG observation model requires priors for α, δ and β. We choose priors for α and δ that allow for all possible values of steepness, but give greater weight towards values that approach the Gaussian distribution (Table S1-1, Appendix S1). Using the beta distribution, which extends the standard beta distribution with support (0,1) to arbitrary support (Patil et al. 1984), we use different parameterizations of the static parameter β (see above and Fig. 1 for the meaning of β relative to the NIG triangle) to model several hypotheses of asymmetry: overcounting (βpos), undercounting (βneg), symmetry (βcen) and uninformative (βflat); see Fig. S1-3 and Table S1-1 (Appendix S1).

For the process model parameters, priors are chosen that allow for a wide range of initial population sizes, carrying capacities, and forms of PGR curves. These priors are listed in Table S1-1, and details on the rationale behind these choices are given in Appendix S1.

Model comparison

Bayes factors compare two competing models to assess the relative odds that the observed data are explained by two hypothesized combinations of a process model and an observation model, denoted by H1 and H2, such that BF12 = p(y1 : T|H1)∖p(y1 : T|H2). Here, p(y1 : T|Hi) is the probability of the observed population counts given the hypothesis Hi, which we call the model evidence. If both model choices have equivalent prior probabilities, then BF12 is the posterior odds in favour of H1; otherwise, the posterior odds are equal to BF12 multiplied by the prior odds, p(H1)∖p(H2) (Kass & Raftery 1995). For equal prior odds, if log10BF12 > 0.5, then this is considered substantial evidence in favour of model H1 (Jeffreys 1961). See Kass & Raftery (1995) for additional details regarding Bayes factors and their comparison to other model selection criteria such as AIC and BIC. A key difference from these latter two criteria is that Bayes factors automatically account for the choice of priors and model complexity even with small sample sizes.

Estimation and prediction

We evaluate differences among choices of process and observation models by comparing (1) NIG triangle plots, (2) predictions of the per capita growth rate (PGR) curves, (3) estimates of the unobserved latent states N1 : T and (4) predictions of future latent states NT+1,…,T+τ (τ ≥ 1) and associated extinction risk. We calculate quantiles of the posterior PGR curve estimates for each model based on the posterior distributions of the static parameters that form the deterministic components of the process model (e.g. r, K, see eqns 1 and 2). Importantly, the PGR estimates vary nonlinearly with population size such that small differences in their parameters may have large impacts on the predictions of the latent states.

Estimates of the unobserved latent states within sample are available from the posterior distribution (eqn 3), and predicted latent states are available from the predictive posterior distribution (Appendix S2, eqn S2-5). Morris & Doak (2002) define quasi-extinction risk, in the cumulative sense, as the probability that the population hits the quasi-extinction threshold q between a time (say, T) and a future time (T + τ, τ > 0). When a population falls below this boundary (q), then this might trigger a management response such as a reduction in harvest, or possibly lead to developing different models of population dynamics such as those that include demographic stochasticity (Morris & Doak 2002). Following this definition and using the posterior predictive distribution for the latent states NT+1 : T+τ, the definition for (quasi-)extinction risk as a function of τ is given by


Estimation procedure

The Bayesian hierarchical model developed in eqn 3 allows us to extract posterior credible intervals and other estimates with ease, but requires that we sample from high dimensional posterior distributions, where the dimension is given by the dimension of both the static parameters and the population states, dim(Ψ) + dim(N1 : T), with T possibly large. For Bayesian estimation of ecological SSMs with Gaussian observation error, Gibbs sampling is an approach (Millar & Meyer 2000; Calder et al. 2003; Clark 2007) that is easily implemented for conventional models using WinBUGS (Lunn et al. 2000). It can, however, be difficult to tune and slow for the Markov chain to converge to the target distribution if there is dependence between the state variables and parameters (e.g. Newman et al. 2009; Andrieu et al. 2010).

Here, we use a hybrid approach called particle Markov chain Monte Carlo (PMCMC) that is specifically designed for the joint estimation of latent states and static parameters in SSMs (Andrieu et al. 2010). Ecological applications of PMCMC can be found in Jones et al. (2010); Peters et al. (2010) and Knape & de Valpine (2012). We combine this approach with an adaptive proposal (see Roberts & Rosenthal 2009) for the static parameters that preserves ergodicity and eliminates much of the tuning needed for other methods. For the real data analyses, trace plots show that the estimates for all parameters mix suitably (Appendix S2). For the results presented in the next section, we use thinned samples (thin = 100) drawn from the last half of the adaptive phase. R code (R Development Core Team 2011) used for these analyses is available from the authors.

For Bayes factors, we calculate the model evidence using an importance sampling approach. As proposed by Sinharay & Stern (2005), the important sampling distribution is a multivariate Student's t-distribution, which we centre at the posterior mean, with covariance estimated from the posterior samples drawn from each model, and 1 d.f. (Appendix S2).

Simulated data

NIG and Gaussian observation models

Synthetic data are generated from all permutations of process model and observation model combinations with T = 25. The ‘true’ values of β are drawn from the respective prior (Table S1-1, Appendix S1), and all other parameters are fixed to the values r = 0.2, K = 100, logN0 = log50, σε = 0.3, θ = 1 (for process model M1), A = 0 (for process model M2), α = 5.774 and δ = 0.519. The latter two values ensure moderate levels of steepness in the NIG observation models to illustrate the sometimes subtle effect of non-Gaussian observation error.

Limited detectability observation model

Kery & Schaub (2012) consider the case of limited detectability where observations are binomially distributed, inline image, and d is the probability of detectability. Following the discussion in the Introduction, we examine a slightly complicated case where annual detection probabilities vary through time. We assume that detection probabilities are uniformly distributed, inline image, and the observations are generated using a normal approximation to the binomial distribution, inline image. Synthetic latent states are generated from M1 with parameters as above.

Real data

California condor

Dennis et al. (1991) predicted the quasi-extinction risk of the endangered California condor (Gymnogyps californianus) based on the highest estimates available from annual October surveys 1965–1980 (sensu Snyder & Johnson 1985). All individuals were removed from the wild by 1987. Buonaccorsi & Staudenmayer (2009) generalized the analysis of Dennis et al. (1991) to include non-Gaussian observation error for a density-independent model. This time series has a number of sources of observation error, such as the inclusion of nonreproducing immature individuals in the population counts (Snyder & Johnson 1985), that suggest possible overcounting of individuals (Wilbur 1980).

Puerto Rican parrot

Dennis et al. (1991) also estimated the quasi-extinction risk of the endangered Puerto Rican parrot (Amazona vittata), and Buonaccorsi & Staudenmayer (2009) considered possible non-Gaussian observation error. Unlike the condor population, the parrot population was closely monitored during the sampling period 1969–1989. Population estimates consist of the largest adult wild bird count or estimates adjusted to account for possible undercounting (Dennis et al. 1991). The counts for this population and that of condor are provided by Staples et al. (2004: Supplement 1).

Eurasian sparrowhawk

The population time series of Eurasian sparrowhawk (Accipiter nisus), unlike the two endangered populations above, is from a non–endangered population species. The counts are of the number of nests (Wyllie & Newton 1991) and may therefore be an undercount because they do not include non-nesting individuals (Newton & Rothery 1997). The data are publicly available (NERC 1999, Data ID 6575).

Results and analysis

Synthetic analyses

NIG and Gaussian observation models

The accuracy of the estimation technique is demonstrated using a controlled synthetic study. The true latent states and synthetic observations for each process model and observation model combination are plotted in Fig. S2-1 and S2-3 (Appendix S2), which also show the NIG triangle plots and latent states as estimated from the correctly specified process and observation models. These figures, and those for the static parameters (Fig. S2-2 and S2-4, Appendix S2), show that across all model combinations, the estimation procedure captures every static parameter and almost all of the true latent states within the 95% credible intervals when the true underlying model is known.

A reasonable question to ask is whether or not the correct model can be selected from the above competing models. We focus on a single data set (plotted in Fig. 2), with latent states simulated from process model M2, overlaid with NIG observation error generated using an informative prior for overcounting (βpos). The Bayes factor results suggest substantial support for this correctly specified model over most others (Table S2-1, Appendix S2, Fig. 2top). An exception is the combination of the correct observation model with process model M1, which suggests that the Allee effect in M2 is not evident in this particular random data set.

Figure 2.

 Simulation from a process model with an Allee effect (M2) and an NIG observation model that hypothesizes overcounting (prior parameterization βpos in Table S1-1, Appendix S1). Top figure shows model fits for the correctly specified M2 and NIG (βpos) combination. Bottom figure shows fits for the fitted M2 and (misspecified) Gaussian observation model combination. Far left column: The NIG triangle. The orange point shows the true values of asymmetry and steepness in the NIG observation model. For the NIG observation models, 50 000 points drawn from the posterior are plotted, which are restricted to the right half of the NIG triangle because of the prior belief of overcounting. The Gaussian observation model is represented by a single black point at the apex of the triangle (Steepness = 0,Asymmetry = 0). Centre left column: The orange line shows the true PGR curve. The estimated 95% CI is given by the shaded area and the median PGR by the white line. Centre right column: The orange line shows the true latent population states, and the points show the observations simulated from the NIG observation model. The 95% credible interval (CI) of the latent states is given by the shaded area and the posterior means by the white line. Note that the 95% CI is for the true latent states and not the observations. Far right column: Predicted quasi-extinction risk with the quasi-extinction boundary set to q = K∖2 (solid line) and q = K∖5 (dashed line).

We also compare the estimates from the correctly specified model with those of the Gaussian model (Fig. 2bottom) that appears to overestimate the carrying capacity. Several of the marginal posterior means appear to differ somewhat from the NIG model (Table S2-2, Appendix S2). The resulting PGR curve falls outside of the 95% CI for the Gaussian model, whereas the curve falls within the 95% CI for the correctly specified NIG observation model (Fig. 2). Similarly, the true latent states often fall outside the 95% CI of the Gaussian model (for example, between years 5 and 9), but are always close to the posterior means in the NIG model. The predicted latent states reflect these differences and translate into a higher predicted extinction risk under the NIG model. In this study, Bayes factors were able to identify the correct observation model, and the choice of observation model significantly affects both estimates and predictions.

Limited detectability observation model

The next question concerns the more realistic case where the observation model approximates an unknown observation process and is misspecified. For this example, we use the synthetic data generated from the limited, varying detectability observation model (see Methods), which is plotted in Fig. 3. It must be said that many ways exist to measure and model detectability (e.g. Royle & Dorazio 2008; Buckland et al. 2011; Kery & Schaub 2012), however, in some cases all that may be available is a vector of counts and limited expert knowledge of the population and the counting process. In the absence of a clear mechanistic observation process to model, the hypothesis of undercounting owing to limited detectability may be represented with an NIG observation model having an informative prior on asymmetry (HNIG(βneg)). This model, however, receives weak support against it when compared with the simpler Gaussian observation model (MG), with a log10BFNIG(βneg), G = −0·2. Nevertheless, the posterior means of the estimated latent states closely approximate the true latent states under the NIG observation model, whereas the true latent states consistently fall above the 95% CIs in the slightly favoured Gaussian case. As a result, the estimated extinction risk is higher in the Gaussian case.

Figure 3.

 Simulation from a process model (M1) and an observation model with varying detectability. Top figure shows fits from an NIG model with informative prior for undercounting, NIG (βneg). Bottom figure shows fits from a Gaussian observation model. Both observation model fits use the correctly specified process model M1. Note that the true observation model does not belong to the NIG family and so it is not represented in the NIG triangle; otherwise, notation is as in Fig. 2.

Clearly, if the true latent states are unknown then it is difficult to say which of these misspecified observation models is the best (recall that the process model is correctly specified in this simulated example). Relying on the Bayes factors would suggest that if the two models are given equal prior probability, then the posterior odds that HNIG(βneg) is the correct model are c. 1 : 1·6. To have substantial posterior odds in favour of HNIG(βneg) requires that p(HNIG(βneg))≈6·3×p(HG). Therefore, if one had substantial prior evidence that overcounting occurred then HNIG(βneg) would have substantial support over HG in terms of the posterior odds.

Real data analyses

For consistency, we retain the same general models and priors, given in Table S1-1 (Appendix S1), used in the synthetic studies to demonstrate how different data sets affect estimates and predictions. Before presenting results, we note that population specific information may suggest alternative priors for Ψ. For instance, it may be appropriate to penalize NIG observation models deemed overly asymmetric by modifying the prior p(β) (Appendix S1). It may also be appropriate to consider alternative observation models, such as a limited detectability model (e.g. Kery & Schaub 2012), or alternative process models. For example, density-independent models have previously been considered for the condor and parrot populations (Dennis et al. 1991; Staples et al. 2004). However, for populations in general, carrying capacity may be limited even for small populations (e.g. such as in a wildlife refuge or marine park), and so we retain negative density dependence in this section. Moreover, model M1 allows for negative density dependence to have strong effects at population sizes well below carrying capacity (Sibly et al. 2005; Clark et al. 2010), and model M2 includes an Allee effect relevant for small populations.

Each process model (M1 and M2) is fit using both a Gaussian observation model and an NIG model with an informative prior that specifies overcounting (βpos), symmetry (βcen) or undercounting (βneg). In addition, for completeness, we also fit an uninformative NIG observation model for asymmetry (βflat), which provides a wide range of potential asymmetric distributions. The Bayes factors, however, suggest no compelling evidence for the NIG observation models with uninformative asymmetry (Table S2-4, Appendix S2).

In the Results and Discussion below, we therefore focus on the NIG observation models with informative priors for asymmetry and the Gaussian observation model. As for the synthetic studies, we report the estimated NIG triangle, PGR curves, estimated and predicted latent states, and extinction risk. We use the minimum observation ymin as a common known benchmark to set two quasi-extinction thresholds, q = ymin and q = ymin∖2. Using ymin avoids confounding with model estimates of the unknown K and makes comparisons of extinction risk comparable across the three real data sets.


Here, an informative NIG prior, βpos, represents the hypothesis that the number of condor is overcounted (e.g. Wilbur 1980). Model comparison shows weak support for M2 over M1 (Table S2-4, Appendix S2); although this evidence is not overwhelming, it is consistent with the scenario of an Allee effect affecting this small population. Given a choice of M2, the observation models all have similar support based on Bayes factors despite the greater number of parameters in the NIG model versus the Gaussian, which is, however, slightly favoured. Although the marginal posterior estimates p(Ψ|y1 : T) are not overly dissimilar (Table S2-5, Appendix S2), the informative NIG model of overcounting suggests that the PGR curve may be more negative than the Gaussian model suggests and also estimates that the posterior means of the latent states are lower than the observations (Fig. 4). The NIG model of overcounting therefore predicts heightened risk of extinction relative to the Gaussian model. The condor's very low population size means that even the small absolute difference in estimated latent states between these two observation models translates into large differences in predicted extinction risk.

Figure 4.

 Estimates and predictions for California condor. A rug along the x-axis of the PGR plot shows observed population sizes (some marks overlaid). The quasi-extinction thresholds in the last column correspond to 12 individuals (black line) or six individuals (grey line). The true parameter estimates and latent states are unknown and hence unplotted. Otherwise, notation is as in Figs 2 and 3.


As the parrot population is closely monitored, we use an informative NIG prior, denoted βcen, that gives greater support to the centre of the NIG triangle (i.e. places greater support for symmetric distributions). Both this observation model and the Gaussian model produce similar estimates of Bayes factors, marginal posterior estimates and PGR curves. The 95% CI estimates for the latent states is broader in the NIG models than in the Gaussian models (Fig. 5). This does not translate into differences in extinction risk because the predictive posterior of the latent states (eqn S2-5, Appendix S2) is determined by integrating over all static parameters and latent states N1 : T given all observations. Unlike the above results for condor at time T, the parrot population has a high probability of increase over the next 10 years, such that under all model combinations the probability of hitting the minimum observed population size is negligible. Under either choice of observation model, M2 provides more favourable predictions for the latent states than M1, as would be expected if the population were to surpass an Allee threshold in the early years of the time series.

Figure 5.

 Estimates and predictions for Puerto Rican parrot. The quasi-extinction thresholds in the last column correspond to 14 individuals (black line) or seven individuals (grey line). Notation is as in Fig. 4.


Here, an informative NIG prior (βneg) includes the hypothesis of undercounting, and the Bayes factors give only weak evidence in favour of the Gaussian versus the NIG observation models (Table S2-4, Appendix S2). Although the estimated 95% CI for the latent states is much broader for the NIG observation models (Fig. 6), the marginal posterior estimate of K appears slightly narrower for the NIG model than the Gaussian model (Table S2-5, Appendix S2).

Figure 6.

 Estimates and predictions for Eurasian sparrowhawk. The quasi-extinction thresholds in the last column correspond to 29 nests (black line) or half this value (grey line). Notation is as in Fig. 4.

The PGR plot, however, is useful to consider because it shows the degree of density dependence over a range of populations sizes. Here, the graphical depiction of the PGR curve clearly shows that the 0·975 quantile of the predicted PGR curve is highest under the NIG models for population sizes near the observed values (shown in the rug in Fig. 6). This translates into within-sample estimates of the latent states being skewed towards higher population sizes in the NIG models, which extends the upper bound on the 95% CI for the latent states.

In contrast to the PGR curve, the linearized model at the asymptotic equilibrium point K (∂Nt∖∂Nt−1|Nt−1 = K) is the traditional measure of the strength of density dependence. The condition for (local) stability is that the absolute value of the strength of density dependence is less than one. For instance, this measure shows that only the sparrowhawk population suggests non-negligible probability of overcompensatory negative density dependence; in Table S2-5 (Appendix S2), the 95% CI admits values −1 < ∂Nt∖∂Nt−1|Nt−1 = K < 0 that provide damped oscillatory dynamics (see May & Oster 1976). For all choices of process and observation models, the posterior means of the strength of density dependence are well within the range required for stable dynamics.


We have used the normal inverse Gaussian (NIG) distribution to capture different hypotheses of observation error in ecological state space models, and have found that the choice of observation model affects estimation and prediction in studies of both synthetic and real data. In scenarios where the true observation model is incompletely known and hence possibly misspecified, we demonstrate how to develop interpretable priors for the observation model using the NIG triangle (Fig. 1) and closed form expressions of the NIG's moments (Appendix S1). In real data applications, we find statistical evidence that provides equivalent support for the NIG relative to the Gaussian observation model. The choice of model structure and relative prior belief in competing observation models are therefore important factors that affect estimates of per capita growth rate curves and predictions of population size.

The NIG is one flexible family of models that includes as limiting cases the Gaussian and Cauchy distributions, but other choices exist (e.g. Peters et al. in press). The NIG is a useful candidate because it retains interpretability while capturing a broad spectrum of heavy-tailed and asymmetric families of observation models. For instance, most previous studies of annual census data have used the Gaussian observation model (e.g. Clark & Bjornstad 2004; Dennis et al. 2006; Knape & de Valpine 2012), and so here we have used a prior that places greater support for low values of steepness in the NIG observation model. The closed form expression of the NIG's standard deviation also allows us to assess comparability with the prior specification for the Gaussian observational model (Appendix S1). The prior for the standard deviation of the observation noise is an important consideration even in linear Gaussian models, where estimating unknown observation and process noise can sometimes lead to peaks in the likelihood surface where either process or observation noise is zero (e.g. Dennis et al. 2006; Knape 2008), as was seen in the condor and parrot populations using a linear SSM (Staples et al. 2004). Here, in both NIG and Gaussian observation models, we have used a prior constraint on the observation variance that ensures non-negligible observation error (Appendix S1) consistent with a state space model.

Alternative priors on the NIG parameters can propose interpretable observation models that capture different hypotheses of the measurement process. This flexibility comes with the cost of two more parameters relative to the Gaussian special case. This is particularly troublesome for the case of annual census data, which are usually sparse, and for which the use of uninformative priors under the NIG model may lead to overly broad credible intervals of the latent states. Informative priors, perhaps chosen using simulation studies to help visualise and test assumptions, can be adopted to limit the flexibility of the observation model. Importantly, the NIG models allow a much wider range of observational hypotheses than we have space for here and, in general, allow practitioners to develop priors that can be tailored to the population under analysis.

Both synthetic and real analyses are used to establish some guidelines for the application of the NIG observation model in ecological state space models. We use synthetic data to show that Bayes factors provide one possible model selection mechanism to find the ’true’ observation model (Table S2-1, Appendix S2). However, in a second synthetic study that models varying detectability of a closed population, we demonstrate that if the true observation model is unknown, then the fitted and misspecified observation models may both receive similar support from the data. This can occur even if one of the fitted state space models poorly captures the true latent process, as seen in Fig. 3. It is therefore important to consider the relative prior support for the hypothesized NIG and Gaussian observation models.

Bayes factors provide a unified mechanism to incorporate these prior beliefs in model choice with information from the observed data into the model comparison. This approach to model comparison also integrates over the parametric uncertainty rather than depending on an optimized parameter set, such as the maximum likelihood estimate, as in AIC or BIC. For instance, in model M1, a trade-off exists between the parameters r and θ that controls the strength of negative density dependence near the carrying capacity K (Thomas et al. 1980). Polansky et al. (2009) note that multimodalilty and flat likelihood surfaces can occur under M1, even without observation error, and model selection procedures that depend on an optimised parameter set may therefore be misleading. Moreover, Bayes factors naturally account for model complexity and small sample sizes in the model comparison (Kass & Raftery 1995).

The real data analyses provide the case where both the process and observation models are unknown. For our case studies, given a choice of process model, model comparison using Bayes factors does not find overwhelming support for either hypothesis of Gaussian or non-Gaussian observation error (Table S2-4, Appendix S2). Therefore, the choice of observation model and its accompanying prediction of extinction risk will be driven by one's prior belief in model structure, that is, the relative plausibility of asymmetry or steepness in the observation model. In some cases, such as the parrot population, different prior beliefs in alternative observation models will make little difference in estimates of interest, such at the PGR curve or quasi-extinction risk (Fig. 5). In other cases, such as the condor population, different prior beliefs may lead to different predictions, such as quasi-extinction risk, with consequences for informing management decisions. In the case study of the California condor population, for example, weak evidence for an Allee effect is present regardless of the observation model (i.e. M2 has greater support than M1). Nevertheless, given a choice of M2, the observation model choice affects predictions of extinction risk (Fig. 4) and its prior probability is therefore relevant.


We thank Arnaud Doucet, Jonas Knape, Scott Foster and Toby Patterson for comments, suggestions and discussion. We also thank an anonymous reviewer’s helpful comments and suggestions that improved the manuscript. GWP gratefully acknowledges the support of CSIRO during this research collaboration.