## Introduction

Population geneticists have been traditionally interested in homozygosity excess, reflecting either inbreeding or hidden population structure (e.g. see Chakraborty & Li 1992). In this vein, null alleles have long been an issue for widely deployed genetic markers, and that remains true for microsatellites, which may be characterized by an increased frequency of null alleles (Dakin & Avise 2004). Moreover, many data originating from next-generation sequencing and chip-based approaches also suffer from severe null allele problems (Franke *et al*. 2008; Hohenlohe *et al*. 2011; Pfender *et al*. 2011).

The utility of codominant markers in revealing heterozygote deficiency and population structure (notably *F*_{IS} and *F*_{ST}) is complicated by the presence of hidden null alleles (Chakraborty *et al*. 1992; Brookfield 1996; Chapuis & Estoup 2007), because they contribute to apparent homozygote excess, over and above the effects of inbreeding and reduced gene flow. Undetected null alleles may also introduce bias into the estimation of allelic frequencies, affecting any subsequent population genetic analyses (Dakin & Avise 2004). Conversely, failure to account for inbreeding (*F*_{IS} > 0) will lead to overestimates of null allele frequencies (Van Oosterhout *et al*. 2006).

Joint estimation of both the inbreeding coefficient (*F*) and the null allele frequency (*r*) is a formidable challenge (see Chybicki & Burczyk 2009). Different estimation methods have been developed to estimate or establish both parameters. One might estimate the null allele frequency (*r*), using an externally supplied estimate of *F*. Alternatively, one might jointly estimate *F* and a set of *r*-coefficients, employing a different *r*-parameter for each locus.

There are currently two methods of dealing with the estimation of null allele frequencies in nonequilibrium populations. First, Van Oosterhout *et al*. (2006) have proposed an algorithm using external estimates of *F*. The method has been implemented within an Excel macro (Null Allele Estimator, http://www.microchecker.hull.ac.uk/), henceforth referred to as the VO method. Chybicki & Burczyk (2009) have designed an alternative likelihood-based method for a simultaneous estimation of *F* and *r*, henceforth referred to as the CB method, implemented in INEst software (www.genetyka.ukw.edu.pl/index_pliki/software.htm).

The VO method has been used in several empirical studies (e.g. Basic & Besnard 2006; Perrin *et al*. 2007; Potter *et al*. 2008; Billard *et al*. 2010; Elias *et al*. 2010; Shirk *et al*. 2010) and has also been proposed to adjust allele frequencies by including provision for a null allele (e.g. Dewoody *et al*. 2006; Carlon & Lippé 2007). Notwithstanding its popularity, theoretical validation for this method remains unavailable, and its reliability remains at issue. While Van Oosterhout *et al*. (2006) state that the system ‘returns a single real solution’, the possible existence of two solutions is acknowledged in the implementation of the method (null allele estimator). The matter needs further attention.

The CB method was published with a detailed evaluation of its performance (Chybicki & Burczyk 2009), but only over a restricted range of conditions: sample sizes (*N > *50); moderately inbred populations (*F *<* *0.2); and *L *=* *5 or 10 loci. It would be useful to evaluate the performance in this method over the wider range of parameters encountered in practice (e.g. *N *<* *50, *F *>* *0.2, *L *≥* *10).

The aims of this comparison were (i) to provide some clarifications on the VO and CB methods, using theoretical analysis and simulation; (ii) to assess their actual performance when dealing with a wide range of inbreeding, by comparing a modification (VO_{m}) and the CB method; and (iii) to improve our understanding of the arrays of situations that favour one method over the other.