How to separate genetic and environmental causes of similarity between relatives


Loeske E. B. Kruuk, Institute of Evolutionary Biology, School of Biological Sciences, University of Edinburgh, Edinburgh EH9 3JT, UK.
Tel.: +44 131 650 5515; fax: +44 131 650 6564;


Related individuals often have similar phenotypes, but this similarity may be due to the effects of shared environments as much as to the effects of shared genes. We consider here alternative approaches to separating the relative contributions of these two sources to phenotypic covariances, comparing experimental approaches such as cross-fostering, traditional statistical techniques and more complex statistical models, specifically the ‘animal model’. Using both simulation studies and empirical data from wild populations, we demonstrate the ability of the animal model to reduce bias due to shared environment effects such as maternal or brood effects, especially where pedigrees contain multiple generations and immigration rates are low. However, where common environment effects are strong, a combination of both cross-fostering and an animal model provides the best way to avoid bias. We illustrate ways of partitioning phenotypic variance into components of additive genetic, maternal genetic, maternal environment, common environment, permanent environment and temporal effects, but also show how substantial confounding between these different effects may occur. Whilst the flexibility of the mixed model approach is extremely useful for incorporating the spatial, temporal and social heterogeneity typical of natural populations, the advantages will inevitably be restricted by the quality of pedigree information and care needs to be taken in specifying models that are appropriate to the data.


The biological diversity observed within a population may be due to genetic variation, environmental variation or a combination of both. Evolutionary biologists frequently want to quantify the former as it is a prerequisite for an evolutionary response to selection. For a quantitative phenotypic trait, this requires an estimate of the additive genetic component of the phenotypic variance (its heritability), which is typically calculated from the phenotypic similarity between relatives (Falconer & Mackay, 1996; Lynch & Walsh, 1998). However, similarities between relatives may be due to shared environmental conditions as much as to heritable genetic effects. Confounding environmental effects are likely to be particularly prevalent when considering wild populations in unmanaged environments, in contrast to the controlled breeding schemes typical of domestic or laboratory populations. They are therefore of particular concern for studies of the evolutionary genetics of wild populations. Furthermore, nongenetic sources of similarity between relatives are interesting in their own right: for example, an understanding of the role of parental care or of shared home territories may be a critical component of studies of life-history evolution or behavioural ecology.

Many studies of wild animal populations have used experiments that involve cross-fostering offspring to different parents to separate genetic from environmental sources of covariance. These cross-fostering experiments have been widely and successfully employed in numerous studies of bird populations, especially of passerines (for example, see review in Merilä & Sheldon, 2001), and occasionally in other animal systems (e.g. McAdam et al., 2002; Rauter & Moore, 2002). However, they can only separate off environmental effects that occur after the cross-fostering has taken place. Very early environmental effects, for example those due to differential maternal investment in progeny size, may still be confounded with genetic effects. Furthermore, they are only feasible when offspring can be readily captured, marked, successfully placed in the care of different parents and then measured at a later stage.

Alternatively, genetic and environmental sources of resemblance between relatives can be separated statistically by comparing the covariance between strategically chosen groupings of individuals (Falconer & Mackay, 1996). We focus here on the use of statistical methodology developed in animal breeding, specifically the ‘animal model’, which simultaneously exploits similarities between groups of relatives of varying degrees (Lynch & Walsh, 1998). Animal models have a long and well-established history in animal and plant breeding, and their more recent application to data from wild populations (starting with Réale et al., 1999) has opened up a wealth of new areas of investigation (for a review see Kruuk, 2004). Several studies of wild species have used animal models to explicitly estimate environmental causes of similarities between relatives in life-history traits (e.g. Kruuk et al., 2000; Pettay et al., 2005; Charmantier et al., 2006), morphological traits (e.g. Kruuk et al., 2002; Charmantier et al., 2004; Wilson et al., 2005b; Hadfield et al., 2006), measures of parasite resistance (Coltman et al., 2001), physiological traits (Nespolo et al., 2003) and behavioural traits (MacColl & Hatchwell, 2003), and even to distinguish maternal genetic from maternal environment effects (e.g. Wilson et al., 2005a). These analyses often reveal substantial contributions of shared environment effects to the total phenotypic variance: for example, common environment effects due to shared nests (brood effects) account for 43% of variance in helping behaviour in long-tailed tits (MacColl & Hatchwell, 2003) and 49% of variance in body condition in collared flycatcher fledglings (Meriläet al., 2001b).

However, despite its increasing popularity in evolutionary biology, there has been little discussion to date of the animal model's suitability for the analysis of data from wild populations, rather than the agricultural or laboratory studies with controlled breeding designs for which it was originally designed (but see Postma, 2006; Quinn et al., 2006). Where comparisons are possible, it appears that the animal model generally returns lower estimates of the heritability of a given trait in a given population than traditional techniques such as parent–offspring regression (Kruuk, 2004; Quinn et al., 2006). However, the assumption that this is due to the traditional techniques being more susceptible to bias from shared environment effects has not been tested (van der Jeugd & McCleery, 2002). There has also been little exploration of the quality of pedigree and phenotypic data necessary for animal models to be able to achieve the separation of common environment from genetic effects achieved with cross-fostering studies. Finally, because of its unparalleled flexibility in allowing numerous different terms to be included in a model, it is also possible to construct animal models containing heavily confounded sources of variance, and the resulting partitioning of this variance may not always be predictable (Clement et al., 2001; Satoh et al., 2002). The aims of this paper were therefore twofold: first, to compare the efficacy of animal models, experimental approaches and more traditional analytical techniques for separating environmental and genetic sources of covariance between relatives, and secondly to illustrate the limitations of the animal model approach when dealing with potentially confounded sources of variance. We do so using a combination of simulation studies and analyses of empirical data.


Quantifying common environment effects using the animal model

The animal model estimates the heritability of a trait by assessing the phenotypic covariance between all pairs of relatives in a population (Lynch & Walsh, 1998), in contrast to more traditional methods of estimating heritability from covariances between pairs of a single type of relative, such as parents and offspring, or full siblings (Falconer & Mackay, 1996). The approach is particularly useful for extracting maximal information from the multigenerational but sometimes patchy pedigrees typical of many long-term studies of wild populations (Kruuk, 2004). The model partitions each individual animal (or plant)’s phenotype for a given trait into a linear sum of different fixed and random effects, with specific variance structures specified for the different random effects. Of these, the most important is a random effect (ai) of an individual's additive genetic merit (or breeding value) which is specified to have variance VA, the additive genetic variance: the analysis then uses the pedigree structure to compare the covariance in phenotype between different pairs of relatives. Under the standard assumption of a Gaussian distribution for each of the random effects in the model, the statistical likelihood for the unknown fixed effect parameters and the respective variance components (for example, VA and the residual variance VR) can then be derived, to give maximum-likelihood estimates of the parameters of interest. In practice, restricted maximum likelihood (REML) is usually used to remove bias due to fixed effects (alternatively, a Bayesian approach may be used: Sorensen & Gianola, 2002).

Various types of common environment effects (see definitions in Table 1) can increase phenotypic covariance between relatives, and if not accounted for will therefore cause upward bias in estimates of additive genetic variance VA. For example, in a system with monogamous mating, multiple offspring from a single breeding event are expected to be full siblings, and the variance between broods or clutches is defined as 0.5VA + VC, where VC is the variance due to common environment effects. Any analysis that ignores common environment effects (i.e. assumes VC = 0) will therefore overestimate VA by 2VC. Under a polygamous mating system in which half siblings share a common environment, the problem is even more severe as VA will be overestimated by 4VC.

Table 1.   Definitions of terminology used to denote different components of phenotypic variance generated by shared environmental effects.
Common environment effectsEnvironmental effects shared by groups of individuals, for example effects shared by groups of relatives that are not due to genetic effects
Maternal effectsEffects on offspring phenotype shared by offspring of the same mother, independent of additive genetic effects; treated as a special case of common environment effects, although may be due  to maternal genetic effects (below)
Maternal genetic effectsEffects on offspring phenotype determined by maternal genotype but independent of additive genetic effects
Brood effectsExample of common environment effects shared by siblings in the same brood or clutch
Permanent environment effectsEnvironmental effects on individual's phenotype that are constant across (or common to) repeated measures on that individual
Nested effectsScenario in which the levels of one variable are nested within the levels of another variable
Confounded effectsScenario in which effects of two variables cannot be distinguished from each other

Environmental effects shared among groups of individuals (Table 1) are modelled within an animal model framework by the inclusion of additional random effects. For example, to quantify the covariance amongst individuals sharing common environments (e.g. nest or home range), a random term identifying the respective common environment is added to the animal model. Distinguishing common environment effects from residual effects requires multiple individuals in each group experiencing a particular common environment, such as several chicks in a nest. Furthermore, as we discuss in further detail below, to separate common environment from genetic effects, it is also necessary that related individuals exist in several environments and/or that there is variation in the degree of relatedness between individuals sharing a particular common environment.

Maternal effects, defined as any effect of a mother's phenotype on her offspring's phenotype extra to the additive effects of the genes it inherits from her (Mousseau & Fox, 1998), are an important special case of common environment effects (Table 1; see Räsänen & Kruuk, 2007 for examples). Maternal effects are most efficiently dealt with by testing for increased covariance amongst groups of maternal siblings. In an animal model, this involves fitting a random effect of the identity of each individual's mother, representing maternal effects that are assumed to be independent of each other and to have variance VM. Again, the analysis requires multiple offspring per mother, and, to adequately separate maternal effects from additive genetic effects, data on individuals who are related but have different mothers. The assumption that the maternal effects generated by different mothers are independent of each other can be relaxed to generate more complex models in which maternal effects are split into maternal genetic (variance VMg) and environmental (variance VMe) effects.

In addition to the effects of environmental conditions shared amongst relatives, we also consider environmental effects that simply generate permanent differences between individuals (Table 1): here the shared environment is that which an individual shares with itself at different time points. This permanent environment effect variance may be due to constant differences in conditions experienced by different individuals throughout their lives (for example, differences in individual territory quality), or to long-term effects of conditions experienced at a critical stage of life (for example, during early development). Multiple measurements on each individual are required, so that the residual variance left after correcting for additive genetic variance can be partitioned into between- and within-individual differences (the latter being due to factors such as phenotypic plasticity, stochastic developmental errors, measurement errors and random stochasticity). Within the animal model framework, an additional term specifies the permanent environment effect of each individual, usually defined simply by fitting individual identity as an additional random effect.

A general form of an animal model of a trait y is therefore given by:


where y is a vector of all phenotypic observations across individuals, X is a design matrix relating the values of y to one or more fixed effect parameters given in the vector β, a is a vector describing the additive genetic effects, Za is an incidence matrix relating each of the additive genetic effects to an individual's phenotype, the other random effects are given by uk each with corresponding incidence matrix Zk, and e is a vector of residual effects. More elaborate models can be constructed in which different random effects vary as functions of particular variables such as environmental conditions or age; these are random regression animal models (Meyer, 1998; Wilson et al., 2005b).

Note that although we refer to the effects in Table 1 collectively as ‘environmental’, the terminology is potentially misleading given that they may each potentially incorporate other sources of (nonadditive) genetic variance. Specifically, ‘brood effects’ indicated by increased covariance amongst full siblings may contain dominance genetic effects; ‘maternal effects’ may be genetic in origin, and ‘permanent environment effects’ will incorporate any dominance or epistatic genetic effects. As we illustrate below, substantial overlap or nesting of the effects described above is possible.

Comparison of analytical techniques using simulations

We used simulated data sets to test two expectations: first, that estimates of additive genetic variance from an animal model should be less inflated by the presence of common environment effects than analytical approaches that rely on only a single type of close relative, even in the situation in which common environment effects are not identified explicitly; and, secondly, that the magnitude of any such advantage will depend on the amount of information available on different relatives, i.e. the pedigree structure. Because most published applications of the animal model in wild populations have focused either on socially monogamous hole nesting birds or polygynous ungulates (Kruuk, 2004), we modelled: (i) a system typical of a bird population in which groups of full siblings share common environment effects due to nest environments prior to fledging (brood effects), using life-history and demographic data taken from studies of great tit and blue tit populations (Parus major, Perrins, 1979 and Parus caeruleus, Hadfield et al., 2006); and (ii) a system typical of a uniparous polygynous ungulate, where maternal half siblings born in different years share maternal effects, using data taken from a red deer population (Cervus elaphus; Clutton-Brock et al., 1982). Due to the different life-history strategies of these two groups, namely different patterns of survival, recruitment and maturation, the number and type of relatives for which information is available is expected to differ, thereby generating different pedigree structures. We varied levels of immigration, which will play a critical role in determining the average number of relatives of different degrees in a population, to assess the impact of changing pedigree structure. Estimates of quantitative genetic parameters were compared across data sets with varying numbers of generations, using either animal models or traditional statistical techniques.

Individual-based models were set up with each individual having an annual probability of breeding and surviving to the following year (see Appendix for details). All starting populations consisted of 150 breeding individuals, with first-time breeders being recruited into the population annually, drawn either from offspring born in the population the previous year (recruits) or from an out-bred base population (immigrants). The bird simulations modelled a monogamous mating system in which pairs formed at random at the start of each year and raised broods in the absence of extra pair paternity, whereas the ungulate simulations modelled a polygamous mating system in which each offspring was the product of a random mating. In the bird simulations we created 1500 pedigrees of each length between 1 and 10 years, and for the ungulate simulations 500 pedigrees of each length between 1 and 30 years. All pedigrees, both within and across years, were independent. This was replicated for four recruitment ratios: 0.25, 0.5, 0.75 and 1, resulting in 60 000 independent pedigrees for each pedigree type (see Appendix for further details of simulations).

All individuals in the starting population and all immigrants were assumed to be unrelated and to form a base population. Individuals of the base population were assigned breeding values sampled from a distribution ai ∼ N(0, 0.3). The breeding values of subsequent generations were calculated as a linear function of parental breeding values and a Mendelian sampling effect (Clement et al., 2001):


where ai is the breeding value of the ith individual, ap1i and ap2i are the breeding values of individual i’s parents, and Mi is the Mendelian sampling effect for individual i in the absence of inbreeding, with Mi ∼ N(0, 0.5*0.3). This gave the populations an expected additive genetic variance of 0.3.

We modelled two sources of common environment effects: for the bird simulation, a ‘brood effect’c experienced by all offspring of a single breeding attempt, and for the ungulate simulation, a ‘maternal effect’m experienced by all offspring of a given mother across all her breeding attempts in different years. In both cases, the common environment effects were sampled from a distribution c (or m) ∼N(0, 0.2). All individuals had a residual error term sampled from e ∼ N(0, 0.5), and the phenotype of an individual was defined as the sum of these genetic and environmental effects. The phenotypic trait of interest therefore had zero mean and unit variance, with an additive genetic variance VA of 0.3 (and hence a heritability of 0.3) and a common environment variance (VC or VM) of 0.2.

For both systems we analysed the data using an animal model, with and without the common environment effects modelled explicitly. These models were then compared with parent–offspring regressions in both systems, and to full-sib analysis of variance in the bird system and half-sib analysis of variance in the ungulate system (Falconer & Mackay, 1996). R was used for pedigree simulation and regression analyses (R Development Team, 2005), and ASReml was used to fit animal models and mixed model sib analyses (Gilmour et al., 2002).

Empirical data from cross-fostering experiments

Our second assessment of alternative techniques for separating shared environment effects used empirical data from cross-fostering experiments on collared flycatchers (Ficedula albicollis). We used measurements on collared flycatcher chicks on tarsus length and body condition (residual of body weight corrected for tarsus length), made on the nest-box population in the south of the Swedish island of Gotland (for further information see Kruuk et al., 2001; Meriläet al., 2001b). Two data sets were analysed. The first contained all individuals who were raised in their natal nest, and thus were not cross-fostered (n = 17 766 for tarsus, 17 714 for condition). The second contained all individuals from nests that were used in a cross-fostering experiment, in which some (but not all) chicks in a given nest were moved to a different nest at approximately 2 days post-hatching, typically in a swap with chicks from a second nest (n = 2896 for tarsus, 2894 for condition; this is a more restricted data set than that used in Meriläet al. (2001b) and Kruuk et al. (2001). Note that there was some overlap in the two data sets, consisting of chicks in nests involved in cross-fostering experiments that were not themselves moved. For further details of the cross-fostering experiments, see Merilä (1996, 1997). The data used were from 1980 to 1999, from a study area subdivided into 18 sub-areas.

Two forms of REML analyses were used: a full-sibling mixed model and an animal model. For analysis of the nonfostered chicks, the full-sib analysis contained a random term for nest of origin, and the variance due to this term then estimated half the additive genetic variance VA. For the cross-fostered chicks, the model contained random effects of nest of origin and nest of rearing; again, the former was taken as an estimate of half VA. We also tested an alternative model containing three random effects of duplicate (a factor identifying the pair of nests involved in the translocation, which will account for covariance within pairs in factors such as time of season; Merilä, 1996, 1997), nest of origin and nest of rearing; however, this model was not significantly better than the simpler model of two random effects, and did not significantly affect the estimates of VA, and so we present results from the latter for ease of comparison with the animal model analyses.

For the animal model analyses, in additional to the additive genetic effect, we included a random effect for nest of origin (for the nonfostered data set) and for both nest of origin and nest of rearing (for the cross-fostered data set). All models (both full sibling and animal) also included additional random effects of year and area, and for the cross-fostered data set a fixed effect of clutch size manipulation (i.e. the change in the number of chicks in a brood, fitted as a multilevel factor). All analyses were run in ASReml version 2 (Gilmour et al., 2002).

Nesting and confounding of components of variance in the animal model

Our third set of analyses aimed to illustrate the use of the animal model for dissecting different components of variance, and also to show how substantial confounding or overlap between these different effects is possible. The linear nature of the animal model (or any such mixed model) means that as many or as few additional random effects can be incorporated, and these random effects can be both nested and cross-classified. Random effects are often nested within each other for biological reasons. For instance, brood effects may be nested within maternal effects, and by fitting brood effects alone the maternal effects can often be adequately modelled. However, it should be noted that when there is little or no replication at the lowest level of the hierarchy, for example if each brood only contained a single individual, then the variance associated with this level is not identifiable from the residual variance and modelling of higher levels needs to be considered. The effects of including or omitting levels of a nested hierarchy may often be intuitive and determined by the biology of the species and the nature of the questions being addressed. A more difficult set of models to evaluate arises when the random effects are partially cross-classified, as is often the case with additive genetic and maternal genetic effects. For these types of models the degree of confounding between different terms can be complicated and it may be hard to assess which parameters can be reliably disentangled without the danger of overfitting.

Choice of an appropriate model will therefore require a proper appreciation of the limitations of the available data, as well as an understanding of the biology of the system. Below, we compare models of causal components of variance in three phenotypic traits in two wild ungulate populations: parturition date in Soay sheep, and antler size and birth weight in red deer. Data were taken from the long-term studies of the feral population of Soay sheep Ovis aries living in the Village Bay area of the island of Hirta, St Kilda, Scotland (see Clutton-Brock & Pemberton, 2004 for further details) and the wild population of red deer C. elaphus in the North Block of the Scottish island of Rum (see Clutton-Brock et al., 1982; Kruuk et al., 2002 for further details). For each trait, we started with fitting the most complex model and assessed the impact on other variance components (via likelihood ratio tests) of sequentially dropping different random effects.

Parturition date in Soay sheep

The date in calendar days at which a female gave birth to a lamb or lambs in a given year was fitted in an animal model, with fixed effects of whether or not she gave birth to twins, and whether or not she was 1 year old or greater. In addition to an additive genetic effect, the random effects fitted were a maternal effect (the identity of the mother of the female), a permanent environment effect (from repeated measurements of each female across multiple breeding attempts) and a year of breeding effect (to model temporal variation). Data used were from 1985 to 2000, incorporating a total of 2063 breeding events by 573 mothers. See Wilson et al. (2005a) for further details of the system, although note the different treatment of parturition date; here, we model it as a trait of the mother, rather than a trait of the offspring (i.e. birth date) as in this previous study.

Antler size in red deer

Male red deer grow a new set of antlers every year, generating multiple measurements on adult males across their lifespan of approximately 10 years. In addition to heritable effects, antler size is also determined by environmental conditions at the time of growth (Kruuk et al., 2002). In an animal model of antler weight for males aged 5 years or more, age (as a multilevel factor) was fitted as a fixed effect, and additive genetic, permanent environment, year of birth and year of trait expression (antler growth) were fitted as random effects. The data used were from 1974 to 1998, containing 437 measures on 165 individuals.

Birth weight in red deer

Birth weight of red deer calves (taken as the residual of weight in kg against age in hours at capture, for calves caught at up to 2 weeks of age) was fitted in an animal model incorporating both an additive genetic and a maternal genetic effect. The remaining variance between different mothers was then also modelled with a maternal environment effect, describing permanent differences between mothers that are apparently environmental in origin. For the Rum red deer, each individual in the study population was assigned to a matriline based on its matrilineal descent from one of the females present in the base population at the start of the study (Clutton-Brock et al., 1982). Because female deer tend to associate with matrilineal relatives, matriline may thus represent a higher stratum shared environment effect akin to a territory effect, grouping together mothers of the same matriline. We therefore fitted matriline as a fourth random effect (after additive genetic, maternal genetic and maternal environment), so that individual maternal effects were nested within matrilines. Finally, because inter-annual variation in environmental conditions affects birth weight, year was also fitted as a random effect. Data were available on 1673 calves, born between 1971 and 2002 to 452 mothers. The results are shown first for models excluding the maternal genetic effect (models 1–4 in Fig. 5) and then including it (models 5–8).

Figure 5.

 Proportion of phenotypic variance in red deer birth weight accounted for by genetic and common environment effects. Models 1–4 were fitted without a maternal genetic effect, 5–8 with one. Heritability reflects direct additive genetic variance (as distinct from total heritability incorporating maternal genetic effects); maternal E is maternal environment effect; maternal G is maternal genetic effect; and * denotes the statistically best model based on likelihood ratio tests. See methods for full details of models.


Comparison of analytical techniques using simulations

The simulation results illustrated the impact on estimates of additive genetic variance of failing to account for shared environment effects. For both pedigree types, parent–offspring regressions showed a very small downward bias (Table 2), and were not biased upwards by shared environment effects, as is expected in the absence of transgenerational environmental effects (Rossiter, 1996; Lynch & Walsh, 1998). This small downward bias was also seen in animal models with very small sample sizes (data not shown). In both cases, the biases were small with reasonable pedigree depths, and are therefore unlikely to compromise conclusions taken from such studies. In the bird pedigree, full-sib analyses overestimated VA by twice the brood effect variance VC, exactly as expected (Table 2). For the ungulate pedigree, paternal half-sib analyses of variance were unbiased, as paternal half sibs tended to be born to different mothers through random mating, whereas maternal half-sib analyses of variance overestimated VA by four times the maternal effect variance VM (Table 2).

Table 2.   Summary statistics for analyses of simulated pedigrees, comparing parent–offspring regressions, analysis of variance and animal models with and without shared environment effects modelled explicitly.
ComponentTrue valuesPedigree typeAnalysis typeMean parameter estimate (±SE)Sampling varianceType I error rate
  1. Significance values refer to the probability that the parameter estimates are different from the true values, or whether the type I error rate is significantly different from 0.05. *P < 0.05; **P < 0.01; ***P < 0.001. A binomial test was used to test the significance of the type I error rate.

Parent–offspring regression
 VA0.3BirdMid-parent–offspring0.295 ± 0.001***0.0080.070***
 VA0.3UngulateFather–offspring0.298 ± 0.0020.0280.051
 VA0.3UngulateMother–offspring0.296 ± 0.002*0.0230.052
Analysis of variance
 VA0.3BirdFull sib0.698 ± 0.000***0.0031.000***
 VA0.3UngulatePaternal half sib0.298 ± 0.0010.0080.062***
 VA0.3UngulateMaternal half sib1.097 ± 0.002***0.0230.885***
Animal model without shared environment effects
 VA0.3BirdAnimal model0.593 ± 0.001***0.0040.999***
 VA0.3UngulateAnimal model0.523 ± 0.002***0.0050.920***
Animal model with shared environment effects
 VA0.3BirdAnimal model0.300 ± 0.0010.0060.053
 VC0.2BirdAnimal model0.200 ± 0.0000.0010.051
 VA0.3UngulateAnimal model0.301 ± 0.0000.0050.054
 VM0.2UngulateAnimal model0.200 ± 0.0000.0010.048

Animal models without common environment effects modelled also returned upwardly biased estimates (Table 2), but the bias decreased with the number of generations studied and with the proportion of recruits (Fig. 1a and 1b), being least when immigration was least. Nevertheless, the reduction in bias reached a plateau after several generations and remained considerable even in a closed population (no immigration). The bias was of greater magnitude for the bird system (Fig. 1b), where proportionately more closely related individuals tended to share a common nest environment. In the ungulate pedigree, a polygynous mating system allowed related individuals to be distributed across a wider number of mothers, reducing the potential bias (note that this reduction in bias must have been particularly strong to counteract the fact that maternal effects are shared between half siblings and so can potentially inflate VA by four times VM).

Figure 1.

 Simulation results. (a) and (b) Changes in mean VA estimates from an animal model without shared environment effects modelled under varying levels of recruitment for (a) ungulates and (b) birds. Different lines represent different proportions of first-time breeders: (—) 0.25 of first-time breeders are recruits; (- - -) 0.5; (· · ·) 0.75; and (–·–·–) 1 (i.e. no immigration). True value of VA = 0.3. (c) Sampling variances of VA for ungulate pedigrees estimated using a paternal half-sib analysis and an animal model with parental effects modelled. (d) Sampling variances of VA for a bird pedigree estimated using a parent–offspring regression (- - -) and an animal model with parental effects modelled (—). The four lines for each type of analysis are in the order of increasing recruitment.

By contrast, there was no evidence of bias in the animal models in which the underlying common environment or maternal effects were explicitly modelled (Table 2). The sampling variance of parameter estimates from the animal model was markedly smaller than all of the traditional analyses, and had a roughly log–log linear relationship with pedigree depth (Fig. 1c and d). The level of recruitment had a small effect on the sampling variance of VA estimates from the ungulate pedigree (Fig. 1c), but bird pedigrees showed a strong reduction in sampling variance with high recruitment (Fig. 1d). This was presumably because genetic and environmental effects are often confounded in a monogamous system unless birds return to breed the following year, or produce descendants that return to breed. This was supported by the fact that immigration rates had much smaller effects on sampling variance when an animal model was run without brood effects modelled (results not shown).

Using the approximate standard errors derived from the inverse of the information matrix, and a critical t-value of 2, type I error rates were not significantly different from 0.05 in those animal models with parental effects modelled (Table 2), indicating that the frequency with which variance components are estimated to be significantly different from the true value was no greater than expected by chance. By contrast, the paternal half-sib analysis and the mid-parent–offspring regression both had inflated type I error rates (Table 2). This was probably due, in part, to the small bias in the VA estimate, but will also be because related individuals were often treated as independent data points, resulting in an overestimation of the degrees of freedom, and an underestimation of the variance in the base population. However note that, despite this apparent reliability of t-tests based on the standard errors, likelihood ratio tests will still provide a more accurate tool for testing hypothesis with real data (Pinheiro & Bates, 2000).

Cross-fostering data results

Analysis of the collared flycatcher data showed substantial common environment effects for both tarsus and body condition (Table 3). As expected, full-sib analyses of nonfostered data returned unfeasibly high estimates of heritability for both traits (Table 3; Fig. 2). Similar to the simulation results, for the nonfostered data, the estimates of heritability returned by the full-sib model exceeded those returned by the animal model by an amount (0.58 and 0.96) very close to twice the animal model's estimate of the common environment, or nest of origin, variance (0.29 and 0.49).

Table 3.   Comparison of estimates returned for variance components and heritability for (a) tarsus length and (b) body condition in collared flycatcher fledglings, for chicks that were not fostered (NF), and for chicks in broods that were split in cross-fostering experiments (CF), using either full-sib (Fullsib) or animal (AM) models.
ModelData V A (SE) V origin (SE) V rearing (SE) V year (SE) V area (SE) V res V P h 2 (SE)Origin (SE)Rearing (SE)
  1. Variance components are given by: VA = additive genetic variance; Vorigin = nest of origin; Vrearing = nest of rearing in the cross-fostered data set; Vyear = year of breeding; Varea = sub-area of the study area; Vres = residual variance; VP = total phenotypic variance; and h2 = origin and rearing are the proportions of VP corresponding to VA, Vorigin and Vrearing respectively. See methods for model details. In the full-sib model, VA is estimated from 2*Vorigin.

(a) Tarsus
FullsibNF0.192 (0.005)0.057 (0.020)0.007 (0.004)0.1550.4110.935 (0.086)
AMNF0.146 (0.011)0.122 (0.006)0.058 (0.020)0.009 (0.004)0.0790.4140.353 (0.023)0.289 (0.019)
FullsibCF0.093 (0.009)0.055 (0.007)0.022 (0.013)0.012 (0.009)0.1490.3300.562 (0.051) 
AMCF0.072 (0.032)0.058 (0.016)0.054 (0.007)0.022 (0.013)0.012 (0.008)0.1130.3300.218 (0.092)0.176 (0.049)0.164 (0.022)
(b) Condition
FullsibNF1.145 (0.03)0.139 (0.050)0.018 (0.012)0.5431.6941.242 (0.052)
AMNF0.522 (0.054)0.919 (0.033)0.137 (0.049)0.010 (0.013)0.2691.8610.281 (0.019)0.487 (0.019)
FullsibCF0.222 (0.033)0.787 (0.061)0.191 (0.100)0.024 (0.026)0.5501.7740.250 (0.038)
AMCF0.269 (0.12)0.096 (0.058)0.793 (0.062)0.195 (0.101)0.019 (0.024)0.4091.7820.151 (0.065)0.054 (0.033)0.445 (0.036)
Figure 2.

 Comparison of estimates of heritability of collared flycatcher chick (a) tarsus length and (b) condition, with or without cross-fostering (Xfoster), and using a full-sibling model (‘full-sib’) or an animal model (‘AM’). Error bars represent one SE.

For the cross-fostering data, the full-sib model again returned higher estimates of VA and heritability than the animal model (Table 3; Fig. 2). There was evidence from the animal model of very early common environment effects acting prior to fostering, apparent as a nest of origin effect; this was particularly marked for tarsus length (Table 3). Secondly, more surprisingly, the animal model estimates of additive genetic variance and heritability were substantially higher for the nonfostered data than for the data from the cross-fostering experiments (Fig. 2; Table 3).

Nesting and confounding of components of variance in the animal model

The analyses of data on parturition date in Soay sheep (Fig. 3) and antler size (Fig. 4) and birth weight (Fig. 5) in red deer revealed several patterns of nesting and confounding between different components of variance.

Figure 3.

 Proportion of phenotypic variance in parturition date in Soay sheep on St Kilda accounted for by genetic and common environment effects. Different models (1–5) contain different random effects, and * denotes the statistically best model based on likelihood ratio tests. Heritability reflects additive genetic variance; maternal E is maternal effect; and permanent E is permanent environment effect. See methods for full details of models.

Figure 4.

 Proportion of phenotypic variance in antler weight in red deer on the Isle of Rum accounted for by genetic and common environment effects. Different models (1–5) contain different random effects, and * denotes the statistically best model based on likelihood ratio tests. Heritability reflects additive genetic variance; permanent E is permanent environment effect; and antler year is year of growth of antlers. See methods for full details of models.

Nesting of different environmental effects

Analyses of all three traits showed evidence of nesting of different components of variance, apparent as concurrent increases in the estimates of particular components when other components were dropped from the model. For Soay sheep parturition date, a small and nonsignificant component of variance due to maternal environmental effects was largely subsumed into the permanent environment effects when maternal identity (i.e. the mother of the female giving birth) was not fitted explicitly (model 1 vs. model 2, Fig. 3); this is as expected if maternal effects did not diminish with age and so had a permanent effect on an individual's phenotype. There was a similar pattern for year of birth in the antler weight analysis (model 1 vs. model 2, Fig. 4): when dropped from the model, the permanent environment effect variance increased by a similar amount (0.057 vs. 0.048). For deer birth weight, in the models without maternal genetic effects, variance due to matriline was largely incorporated into the maternal environment effect when matriline was not included in the model (model 1 vs. model 2, Fig. 5). However, the heritability also increased from 0.075 to 0.135, indicating some failure of the model to successfully distinguish between additive variance and covariance within groups of matrilineal relatives (see below).

Confounding between temporal and shared environment effects

There was a significant component of variance between different years of trait expression for all three traits, presumably due mainly to the strong effects of prevailing environmental conditions. However, this temporal variation appeared to be confounded with the other sources of variation in different ways for the different traits. Dropping year of birth from models had least effect in the analyses of red deer birth weight (Fig. 5, model 2 vs. 4 and model 6 vs. 8). For Soay sheep parturition date, dropping year of measurement from the model caused a reduction in the permanent environment effect (Fig. 3, model 2 vs. 4). By contrast, for antler size, dropping year of measurement had the opposite effect of increasing the permanent environment effect (Fig. 4, model 2 vs. 4).

Confounding between additive genetic and environmental effects

For both sheep parturition date and deer antler size, dropping the permanent environment effect from a model resulted in a corresponding increase in the heritability of the trait, regardless of the other effects fitted (sheep parturition date: Fig. 3, model 2 vs. 3 and 4 vs. 5; antler size: Fig. 4, model 2 vs. 3 and 4 vs. 5). Similarly, omitting maternal environmental effects in a model that included maternal genetic effects resulted in a corresponding increase in the maternal genetic variance component (deer birth weight: Fig. 5, model 6 vs. 7). Finally, in situations where year and permanent environment effects were confounded (such as antler size, Fig. 4), dropping both from the model then resulted in a predictable upward bias in the estimate of heritability.

Confounding between maternal genetic and other effects

For deer birth weight, models without a maternal genetic effect fitted consistently returned a higher heritability, implying that, if not fitted explicitly, some of the variance due to maternal genetic effects will be mistaken for additive genetic variance (Fig. 5, models 1–4 vs. 5–8; see also Clement et al., 2001). However, the majority of the maternal genetic variance was identified as general maternal environment effects variance in models without the maternal genetic effect. A further layer of confounding with maternal genetic effects became apparent in that, on dropping matriline (groups of female relatives, presumably sharing similar territories) from the model, the maternal genetic variance increased correspondingly (Fig. 5, model 5 vs. 6).


We have shown how common environment effects may generate phenotypic similarities between relatives that are equivalent or even greater in magnitude to those due to genetic effects, and that a failure to acknowledge their potential impact can seriously derail quantitative genetics analyses. The animal model is a powerful tool with which quantitative geneticists working with wild populations can accommodate these additional components of variance. There are probably several reasons why the estimates of trait heritabilities it provides appear to be less biased by confounding environmental variation than those of traditional analyses. Most importantly, as we have shown here, the animal model provides an efficient way of explicitly modelling environmental sources of phenotypic variation. Corrections for common environment effects are of course equally feasible using more traditional analytical techniques: for example, the heritability of lay date in female great tits estimated from parent–offspring regression is substantially reduced when explicitly corrected for spatial auto-correlation between mothers and daughters (van der Jeugd & McCleery, 2002), generating a heritability identical to that estimated from an animal model analysis of the same data (McCleery et al., 2004). However, the mixed model framework probably provides a substantially more efficient means of modelling such effects.

Secondly, the animal model's use of covariances across all possible pairs of relatives means that the impact of shared environment effects on heritability estimates will be relatively less even when such effects have not been correctly identified and fitted in a model. Our simulation results showed that, in such a situation, the ability of the animal model to remove the bias generated by close relatives’ phenotypic similarity increased with increasing levels of recruitment from within the population and with pedigree depth. In a similar vein, several studies, largely in the animal breeding literature, have illustrated improvements in parameter accuracy (rather than bias due to common environment effects) generated by increasing either the depth and connectedness of the pedigree (e.g. Hanocq et al., 1996; Clement et al., 2001) or the number of generations for which phenotypic data are available (Quinn et al., 2006).

The analysis of collared flycatcher chick morphology indicated, first, the existence of very early common environment effects acting prior to cross-fostering: having accounted for the variance between nests that would be expected on the basis of this estimate of VA, there was still further variance between groups of full siblings sharing a nest of origin. This component of variance could arise through several mechanisms: it could be due to dominance effects, or to differences in maternal investment in egg size or parental care during the first 2 days post-hatching (Kruuk et al., 2001; Krist & Remes, 2004), or it could conceivably be an artefact of cross-fostering where the mean ages of reciprocally cross-fostered broods may differ, even if only by a few hours. The nest of origin variance was particularly marked for tarsus length, representing more than a sixth of the total variance, whereas measures of condition at fledgling were somewhat less affected by such early common environments. Such very early common environment effects may have biased estimates of VA from the full-sibling analyses of cross-fostering data, which required the assumption that all variance between groups of full sibs sharing a nest of origin is due solely to additive genetic effects.

We found striking differences in results from an analysis of the data set of nonfostered collared flycatcher chicks vs. the data set of chicks in nests used in cross-fostering experiments, with estimates of heritability and VA in unmanipulated nests being substantially higher than those derived using cross-fostering experiments for both analytical techniques (see also Meriläet al., 2006). We cannot rule out the possibility that this difference in genetic architecture is due to some extent to consistent, large-scale difference between the two data sets, although this seems unlikely given the spread of experiments across different years and different sub-areas (and the correction for year and area effects). The difference between the data sets in the total phenotypic variance (Table 3) could possibly be explained by the fact that the phenotypic variance in a trait will contain a contribution from the covariance between direct genetic effects expressed in the offspring and the maternal genetic effects expressed by the mother raising them (Riska et al., 1985; Lynch & Walsh, 1998). Because cross-fostered chicks are not related to the mother raising them, this covariance will be zero, such that the phenotypic variance of cross-fostered chicks will be less than that of nonfostered chicks by an amount equal to the direct-maternal genetic covariance (Riska et al., 1985). However, this scenario offers no explanation for the reduction in VA between the two data sets, which could suggest that the estimates from the animal model of nonfostered data were also still biased upwards by shared environment effects. If so, it appears that even a fully specified animal model may still suffer from confounding between genetic and environmental effects; the animal model's exploitation of more extensive pedigree data may still be insufficient to prevent estimates of VA being artificially inflated by the brood effects experienced by full siblings sharing a nest. This is presumably due to the fact that more distant relatives, for whom shared environments will not inflate covariance, contribute proportionately less to the estimates of VA, particularly for pedigrees such as the bird system where low recruitment rates mean that full siblings will make up a substantial proportion of known relatives of each individual (see also Postma, 2006). Although these effects may be less for pedigrees containing proportionately more distant relatives for each individual (such as the ungulate pedigrees considered here), our results suggest that, if feasible, both statistical and experimental means should be used to reliably separate shared environment effects.

The estimates of trait heritability for the red deer and Soay sheep empirical data varied by an order of magnitude depending on the other random effects included in the model, with the statistically optimal models consistently returning the lowest, or near lowest, estimates of additive genetic variance and heritability (Figs 3–5). An upward bias in estimates of genetic variance when shared environment effects were not considered occurred for maternal genetic variance as much as for additive genetic variance: in the analysis of red deer birth weight, failure to account for the shared environments experienced by related mothers (defined by their matriline) inflated the estimates of maternal genetic variance (Fig. 5, model 5 vs. 6). Because estimates of the ‘total’ heritability in a population will incorporate both direct and maternal genetic variance (Wilham, 1972), these biases indicate that as much care needs to be taken in accounting for shared environment effects that might inflate maternal genetic variance as for those that might inflate the additive genetic variance (Alfonso et al., 1997; Satoh et al., 2002). Where the data contained repeated measures, failure to model a permanent environment effect also increased estimates of VA. This was because in a repeated measures data structure where a permanent environment effect has not been specified, the model interprets the covariance amongst all observations on a single individual (i.e. that due to permanent environment effects) as covariance amongst a series of clones with a coefficient of coancestry of 0.5, thus inflating the estimates of VA. Omitting maternal environmental effects in a model that includes maternal genetic effects is exactly equivalent: multiple offspring from the same mother are interpreted as having different mothers with a coefficient of coancestry of 0.5, leading to an inflation of estimates of the maternal genetic variance.

We assessed maternal effects by testing for increased covariance amongst groups of maternal siblings. Maternal effects may alternatively be quantified from the covariance between mother and offspring phenotype after correcting for genetic effects (Kirkpatrick & Lande, 1989), but note that in many systems and for many traits, the action of maternal effects may not increase mother–offspring covariance in a particular trait: for example, improved maternal provisioning of offspring because of a mother's access to favourable resources could mean that she produces consistently large offspring, but it does not require her to be larger herself. The approach taken here of fitting maternal identity as an additional random effect will indicate the occurrence of maternal effects if there is greater covariance amongst relatives joined by maternal links than by paternal links in the pedigree (Clement et al., 2001). However, in the presence of paternity errors, for example where extra-pair paternity is unrecognized or if paternities are assigned imperfectly using molecular data, additive genetic variance may therefore be mistaken for maternal effects. Typical rates of extra-pair paternities, or even of error in genetic assignment (e.g. 20% of offspring, Marshall et al., 1998), are thought to have relatively minor effects on heritability estimates: for example, simulation of pedigrees with 10% paternity errors shows a reduction in heritability of only 6% (J. Hadfield, unpublished data; see also Charmantier & Réale, 2005).

Estimates of the nongenetic components of variance were similarly sensitive to the inclusion of other random effects in the models of sheep and deer data, but sometimes in unpredictable ways. For example, in the analysis of sheep parturition date, dropping the component of variance due to year reduced the estimate of permanent environment effect variance; this was possibly because consistent differences between individuals were less apparent when not corrected for the system's substantial interannual variation due to intermittent, large-scale crashes in population density (Clutton-Brock & Pemberton, 2004). By contrast, dropping year from the model of antler size increased the estimate of the permanent environment effect variance, possibly because environmental conditions in the red deer study have undergone a single major transition over the study period due to a dramatic increase in population density, with a corresponding decline in antler size (Kruuk et al., 2002).

Our analysis therefore revealed the potential for considerable nesting or overlap between the alternative random effects that can be fitted in any one animal model. If, for example, relatives are distributed nonrandomly with respect to time or space, confounding between genetic and environmental effects may cause problems. In such scenarios, when sources of variation are confounded, the precision with which each variance component can be estimated declines. This may result in nonsignificant variance components for environmental effects that in reality are an important source of variation. If their nonsignificance is used as a criterion for dropping such terms from the model, the remaining estimates of genetic variance may be severely biased. Because of this, we recommend careful evaluation of alternative models, with due attention to biological expectation and consideration of the extent to which removing a particular source of environmental variance from a model affects other estimates. Diagnostics to detect overparameterized models have been developed for other branches of statistics and in principle should be applicable to mixed models. In particular, collinearity can be diagnosed from the design matrices, for example through calculation of a ‘condition number’, which can be calculated as the square root of the ratio of the largest to the smallest eigenvalues of the model information matrix (Belsley, 1991). The condition number can be used as an indication of how sensitive certain parameter estimates will be to small changes in the structure of the data set. However, its use is not without criticism (Stewart, 1987), and to our knowledge its application within mixed model frameworks is yet to be fully explored. Confounding will also cause problems for the prediction of breeding values using the animal model (Postma, 2006).

Many of the points in this paper are aimed at the analysis of data from natural populations, where the researcher has no control over the type of environment experienced by an individual. In a controlled laboratory setting, proper experimental design can go a long way to minimizing the magnitude of shared environment effects. However, in many cases – for example maternal effects due to innate differences between mothers – shared environment effects cannot be entirely eliminated by experimental design. Similarly, marker-based methods of estimating heritability will also be affected by the existence of shared environment effects (Garant & Kruuk, 2005). Finally, it is worth emphasizing that, rather than merely being a statistical nuisance requiring correction, all of the above environmental effects are of considerable ecological and evolutionary interest in their own right: for example, the potential impact of variation in parental care has numerous evolutionary implications, such as for the evolution of different breeding systems and life histories. For an evolutionary ecologist, assessing the importance of such effects relative to other determinants of phenotypic diversity is often as critical as estimating the genetic variances and covariances.


Thanks to Juha Merilä for first drawing attention to the effect of cross-fostering on heritability estimates; to Alastair Wilson, Erik Postma, Juha Merilä and two anonymous referees for very useful comments on the manuscript; to Sonya Clegg, Tim Coulson, David Elston, Dany Garant, Kate Lessells, Ian Owens and Ally Phillimore for helpful discussions; to Lars Gustafsson, Juha Merilä and Ben Sheldon for use of the Gotland collared flycatcher data; to Tim Clutton-Brock and Josephine Pemberton for use of the Rum red deer and St Kilda Soay sheep data; and to the Royal Society and the Leverhulme Trust for funding.


Appendix: Simulation population structure details

The number of offspring produced annually by each individual was derived by sampling an empirically derived probability density distribution of annual fecundity (see Figure S1 in Supplementary Material). For the ungulate system, the initial population also contained an age-structured juvenile population. The number of individuals in each age class was determined by the first eigenvector of the Leslie matrix characterizing the population. In both red deer (Clutton-Brock et al., 1982) and great tits (Perrins, 1979), annual survival probabilities are age dependent. To simplify the models we distinguished juvenile (pre-breeding) and adult (breeding) age classes, with annual survival probabilities that give the same stable age distribution of juvenile and adults seen with age-dependent mortality. For the bird population, we assumed that the juvenile annual transition probability of 0.24 covers the transition from year 0 to year 1. Thereafter, adult annual transition probabilities of 0.43 apply (data taken from Perrins, 1979). In red deer, where age at first breeding and patterns of mortality are sex specific, we took values averaged over the sexes. A juvenile annual transition probability of 0.83 applied to ages 0–5 years, and individuals aged 5 years or over had an adult survival probability of 0.84 (data taken from Catchpole et al., 2004). Because males breed later and suffer earlier senescence, there is a sex bias in the sexually viable population (Clutton-Brock et al., 1982). To accommodate the increased reproductive skew this creates, we introduced a female sex bias of 1.74 : 1 at birth; this bias is entirely artificial, but generated the necessary adult sex ratio without requiring sex-specific survival probabilities to be specified. To avoid problems with fluctuations in population size, density dependence was incorporated by first-time breeders replacing individuals that had died. It is usual in pedigrees from wild populations for only a subset of the individuals to have been measured. Analyses were therefore based on juvenile traits, where base population individuals were assumed to have missing phenotypic records.