Consider a population consisting of N animals in a capture–recapture experiment over m capture occasions, j = 1,2,…,m. Let Yij be a binary outcome, equaling 1 if the ith animal is being caught on the jth capture occasion and 0 otherwise. Let Yi = (Yi1,Yi2,…,Yim)′ be a random vector with the capture history of individual i. Let be the number of times the ith animal has been caught in the course of the trapping closed population study. Let ti be the time the ith individual is first captured. Heterogeneity in captured probabilities is often explained by observed individual covariate xi, such as age, sex, weight. For simplicity, we consider xi a single covariate, but the model can be easily generalized for xi to be considered a vector of covariates. Let the probability that the ith animal is captured on any trapping occasion j, be
is the design matrix, β = (β0,β1)′ is the vector of parameters associated with the covariates, and h(u) = (1+exp(−u))−1 is the logistic function. This is an Mh model where variation in capture probabilities among individuals is explained by the covariate xi. The probability of not capturing the ith individual on the jth occasion is (1−pi(β)), and the variance of Yij is pi(β)(1−pi(β)) (Liang and Zeger 1986). Then, Ti∼Bin(m,pi(β)) and πi(β) = 1−(1−pi(β))m is the probability of individual i being captured at least once, given the covariate xi. Let the set of distinct individuals captured at least in one occasion be indexed by i = 1,2,…,n and uncaptured individuals would be indexed by i = n + 1,…,N without loss of generality. To estimate the population size, once an estimate of β is obtained (), the Horvitz–Thompson estimator may be used as in Huggins (1989).
Generalized estimating equations approach
Let be the covariance matrix of Yi, where, Ai = diag[Var(Yi1),Var(Yi2),…,Var(Yim)] is a m×m diagonal matrix and Ri(α) is known as the working correlation structure among Yi1,Yi2,…,Yim to describe the average dependency of individuals being captured from occasion to occasion. A GEE approach permits several types of working correlation structure Ri(α) (for details, see Diggle et al. 1994). For the description that follows, and for simplicity, we consider an independence working correlation structure, Ri(α) = I where I is an identity matrix. The covariate xi is never known for the individuals that have not been captured. Therefore, Yij is conditional on the captured individuals (n) (i.e., Ti ≥ 1) with the corresponding observed individual covariates similar to Huggins (1989) and Zhang (2012). The probability that the ith individual is captured on the jth occasion (pij) given that the ith individual is observed at least once is, . Let , and Di be the matrix of derivatives ∂μi/∂β′, where μi = (μi1,μi2,…,μim)′, hence Di = AiXi. The variance vij of Yij given Ti ≥ 1 is . Considering, Vi = diag(vij), an estimator of β can be obtained by solving the following generalized estimating equations:
If covariate xi (i = 1,2,…,n) is available for captured individuals, then the model becomes pi(β) = h(Xiβ). This model is not equivalent to any of those discussed in Otis et al. (1978), rather this model is a restricted version of their model Mh (Huggins 1991). If pi(β) = h(Xiβ), then following Zhang (2012), estimating equations (2) can be simplified to
Methods based on a partial likelihood
The full likelihood of all model parameters is proportional to
As the number of total individuals, N, is unknown and the covariates are not known for individuals that are never captured, this likelihood cannot be directly evaluated. The conditional likelihood (Huggins 1989) is the first product component, and it can be formulated as a GLM (Huggins and Hwang 2011) for the positive Binomial distribution (Patil 1962). It may be rewritten as
When the full likelihood is partitioned into a product of conditional densities, then a partial likelihood (Cox 1975) may arise considering some of the product terms, but it involves only the parameters of interest, isolating the nuisance parameters. Therefore, the partial likelihood, PL(β), is the first product of the equation (6), which is the likelihood of the number of recaptures after the first capture (Stoklosa et al. 2011). For a given ti, (Ti − 1)|ti∼Bin(m−ti,pi(β)), which is used to estimate the parameters β.
To utilize a simple GLMM with a random effect, we suppose that pi(β) = h(Xiβ + σbzi) where zi is a realization of the standard normal random variable , with σb>0. The use of random effects reflects the belief that there is heterogeneity that cannot be explained by covariates. The partial likelihood can be considered as the joint distribution of the response and the random effects. To estimate β and σb, the marginal likelihood of the response is obtained by integrating out the random effects. The integration can be approximated by penalized quasi-likelihood (Breslow and Clayton 1993), which enables parameter estimation via an iterative procedure.