Indirect genetics effects and evolutionary constraint: an analysis of social dominance in red deer, Cervus elaphus


Alastair J. Wilson, Institute of Evolutionary Biology, School of Biological Sciences, University of Edinburgh, Edinburgh EH9 3JT, UK. Tel.: +44 0131 650 5499; fax: +44 0131 650 5455; e-mail:


By determining access to limited resources, social dominance is often an important determinant of fitness. Thus, if heritable, standard theory predicts mean dominance should evolve. However, dominance is usually inferred from the tendency to win contests, and given one winner and one loser in any dyadic contest, the mean proportion won will always equal 0.5. Here, we argue that the apparent conflict between quantitative genetic theory and common sense is resolved by recognition of indirect genetic effects (IGEs). We estimate selection on, and genetic (co)variance structures for, social dominance, in a wild population of red deer Cervus elaphus, on the Scottish island of Rum. While dominance is heritable and positively correlated with lifetime fitness, contest outcomes depend as much on the genes carried by an opponent as on the genotype of a focal individual. We show how this dependency imposes an absolute evolutionary constraint on the phenotypic mean, thus reconciling theoretical predictions with common sense. More generally, we argue that IGEs likely provide a widespread but poorly recognized source of evolutionary constraint for traits influenced by competition.


In animal societies, it is widely expected that social dominance is positively associated with fitness. This is because dominant individuals, by definition, should have greater access to limiting resources such as food (Herberholz et al., 2007), mates (Willisch & Neuhaus, 2010) or territories (Franck & Ribowski, 1993). In organisms that might loosely be classified as displaying higher levels of ‘sociality’ (Costa & Fitzgerald, 2005), it is also common for dominant individuals to monopolize reproduction and receive benefits from subordinate helpers (Clutton-Brock et al., 2001). Thus, social dominance is generally viewed as being under positive selection, although empirical studies have highlighted that exceptions can and do occur (Ellis, 1995; Qvarnström & Forsgren, 1998; Verhulst & Salomons, 2004).

A critical requirement for adaptive evolution is that trait variation within populations is, at least in part, caused by heritable differences among individuals. If selection acts on such a heritable trait, then quantitative genetic theory states that the population mean of that trait should evolve over time (Falconer & Mackay, 1996). However, common sense tells us that mean dominance cannot evolve in this way, because the trait is always observed through contests with one or more other individuals in a group (Barrette, 1987; Drews, 1993). For the case of dyadic interactions, there must be one ‘winner’ and one ‘loser’ in every contest, such that the mean rate of winning must always be equal to one half.

Here, we conduct a quantitative genetic analysis to estimate the genetic basis of variation in dominance among wild red deer, Cervus elaphus, on the Isle of Rum, Scotland. We show that social dominance is both heritable and under positive selection, but also that phenotypic expression is subject to indirect genetic effects (IGEs) that arise from interactions among individuals (Moore et al., 1997). It has previously been argued that incorporating these IGEs into quantitative genetic models is essential for understanding the evolution of social dominance (Moore et al., 2002). Here, we estimate these effects, show that their presence resolves the apparent conflict between theory and common sense and provide an empirical demonstration of a potentially widespread source of evolutionary constraint.

Social dominance is generally inferred from consistent differences between individuals in their ability to win contests (Drews, 1993) and has been studied in wild populations across a wide range of animal taxa. In some cases, there is evidence that dominance rank can be passed from parent to offspring (e.g. ‘maternal rank inheritance’ in primates and spotted hyenas, Holekamp & Smale, 1991). However, it has not generally been possible to demonstrate a genetic basis to such patterns in the wild, and there is clear evidence that they can arise from social rather than genetic mechanisms (e.g. East et al., 2009). Thus, to our knowledge, no study has demonstrated a genetic basis of variation to dominance rank per se in the wild, although two strong lines of evidence suggest that genetic effects are likely. First, work under laboratory conditions has demonstrated genetic variance for dominance, for example through selection experiments (Craig et al., 1965; Moore et al., 2002) or through directly estimating heritabilities (Nol et al., 1996). Secondly, dominance is often correlated with (and believed to be causally related to) aspects of morphology (e.g. body size, weapon size) and behaviour (e.g. aggression) that are generally found to be heritable in natural populations as well as in the laboratory (Horne & Ylönen, 1998; Kruuk et al., 2008).

Numerous indices have been proposed to rank individuals within a hierarchy, and while debate continues as to their relative merits (Bayly et al., 2006; Bang et al., 2010), a feature common to most is a recognition that the probability of success in any contest depends on the relative strengths of the focal individual and the opponent. Dyadic contest outcomes can therefore be viewed as arising from the interaction of two phenotypes and by extension (if there is genetic variance for dominance) of two genotypes (Moore et al., 1997; Wilson et al., 2009a). An important consequence of this is that if social dominance is a heritable trait, then contest outcomes are likely subject to both direct and IGEs.

Indirect genetic effects, or ‘associative effects’ (following, Griffing, 1967), occur when the genotype of one individual has a causal influence on the phenotype of another (Moore et al., 1997). As has long been recognized, such effects can have major consequences for the evolution of phenotypic traits under selection – whether artificial or natural (Griffing, 1981a,b; Wolf, 2003; Bijma & Wade, 2008). In the current context IGEs, genetic variation for competitive ability (or contest winning) is expected to result in IGEs on the observed phenotype (i.e. an individual’s success in contests, Moore et al., 2002). Moreover, there is an expectation of a negative covariance between direct and IGEs associated with competition (Wolf, 2003; Bijma et al., 2007b). This is because a gene that predisposes to winning a contest when carried by a focal individual will predispose to losing if that gene is carried by the focal individual’s opponent instead. Theoretical models show that this source of negative genetic covariance could provide an important source of evolutionary constraint, reducing (or even reversing) the expected phenotypic change for a trait under selection (Moore et al., 1997; Wolf et al., 1998; Bijma et al., 2007b). Recent empirical studies on laboratory and livestock systems have provided convincing evidence that IGEs can influence a range of traits (Bijma et al., 2007a; Mutic & Wolf, 2007; Bleakley & Brodie, 2009; Wilson et al., 2009a). However, apart from the specific case of maternal genetic effects (e.g. Wilson et al., 2005), the role of IGEs in natural populations has received limited empirical scrutiny to date (but see e.g. Brommer & Rattiste, 2008).

In what follows we perform quantitative genetic analyses to test the genetic basis of variance in social dominance, defined as success in pairwise contests, in red deer on the Scottish island of Rum (Clutton-Brock et al., 1982). We first test the a priori expectation that social dominance is positively correlated with fitness in this population before testing for genetic variation in dominance and IGEs on contest outcomes. Finally, we parameterize a recently proposed model of IGEs (Bijma et al., 2007b) to test the hypothesis that IGEs on social dominance must exist if the trait is heritable, and show that these provide a source of absolute evolutionary constraint that resolves the apparent conflict between common sense and quantitative genetic prediction.

Materials and methods

Phenotypic data

The red deer population in the North Block of the Isle of Rum, Scotland, has been the subject of an individual-based study since 1971, with detailed life-history and behavioural data collected on individually identifiable animals from birth to death (Clutton-Brock et al., 1982). Here, we used data from observations of dyadic dominance interactions recorded between 1974 and 1995. The available data relate to low-level interactions among females and (predominantly) young males outside of the breeding season. Adult stags are very poorly represented in the data as they are largely nonresident in the study area and spend little time in association with female groups outside of the annual rut period. Thus, the available data are limited in that they include neither observations of contests between stags during the rut nor interactions between adult males out with this period. Across all interactions recorded, the mean age of individuals observed was 5.57 years in females but just 1.75 years in males.

At the time of observation, one member of each dyad was scored as the winner based on agonistic behaviours including nose and ear threats, displacements, kicking, boxing, bites and chases (see, Clutton-Brock et al., 1982 for a full description of these behaviours). Note that in what follows we analyse the observed data on contest outcomes directly, rather than by using these observations to first estimate a measure of dominance rank for each individual. Although the latter may seem more intuitive, our approach is statistically advantageous because dominance ranks should themselves be recognized as individual-level statistics that are estimated with error (Poisbleau et al., 2006). Consequently, investigations into the causes or consequences of dominance rank variation must find a way to incorporate this error if statistical analyses are to be considered robust. Here, we avoid the pitfall of doing ‘statistics on statistics’ by using a mixed-model framework that allows our hypotheses to be tested by modelling the observed data in a single step (see e.g. Hadfield et al., 2010, for further discussion). In total, the data comprise 10 517 observations of dyadic interactions (6215 female–female, 3771 female–male, 516 male–male and 15 involving one individual of unknown sex). About 1278 distinct individuals were observed in at least one contest, with a mean of 16.5, and a maximum of 179, observations per individual.

Pedigree structure

The red deer pedigree has been constructed from a combination of behavioural observation and molecular paternity analysis using microsatellite data (full details are presented in Walling et al., 2010). On average, individual paternity assignments are made with 97.6% confidence. Within the entire red deer pedigree, 1336 individuals are informative with respect to the genetics of dominance, either by virtue of being observed in one or more phenotypic records from contests (N = 1278) and/or because their pedigree position provides information on the relatedness among phenotyped individuals. Among this informative subset, 1184 maternities and 648 paternities are known. With respect to the genetics of social dominance, the informative portion of the pedigree is defined by many pairwise pedigree relationships, including 2459 and 3132 maternal and paternal pairwise sibling relationships, respectively. The pedigree is highly convoluted, with many known relationships both within and among cohorts (Fig. 1). Pedigree statistics and the graphical representation of the pedigree were generated using the R package pedantics (Morrissey & Wilson, 2010).

Figure 1.

 The structure of the portion of the red deer pedigree that is informative with respect to the genetics of variation in dominance (N = 1336). Numbers on the left-hand side indicate cohort affinities, some of which are estimated based on age at first known reproduction and typical sex- and age-specific patterns of reproductive success. Red lines denote maternal links, and blue lines denote paternal links. Individuals with one or more records from dominance interactions are denoted with black dots.

Selection on social dominance

To confirm our a priori expectation that dominance is under positive selection, we formally tested its relationship with fitness, estimated as the number of offspring produced by an individual over its lifetime (lifetime breeding success, LBS). Because the distribution of LBS differs greatly between the sexes (i.e. the male distributions are much more highly skewed) analyses were performed separately for males (N = 456) and females (N = 579). Note that for males, fitness values were estimated as the number of offspring assigned in the pedigree structure on the basis of microsatellite genotype data. Each individual was assigned an estimate of fitness, and individuals observed as juveniles were included regardless of whether they survived to breeding age. Note, however, that individuals could only be included in the analysis if they were phenotypically informative (i.e. had been observed in dominance interactions), and our data are therefore not informative with respect to the potential fitness consequences of avoiding interactions. Any animal known to have died an unnatural death (i.e. to have been shot) was excluded prior to analysis.

For each sex, selection was estimated in two ways. First, for each individual i, we simply determined mean contest success as the proportion of interactions observed across i’s lifetime in which i was scored as the winner. We then estimated selection as the ordinary least-squares regression coefficient of fitness (LBS) on mean contest success. To facilitate comparison with selection estimates in the literature, we also regressed relative fitness (i.e. inline image) on mean contest outcome scaled to standard deviation units to obtain the standardized linear selection differential (Endler, 1986). Whereas the simple linear regression is appropriate for estimating the selection differentials, statistical inference is not valid in the absence of residual normality. Consequently, we also fitted negative binomial regressions in each sex to test the hypothesized positive association between mean dominance. This approach is appropriate given that LBS is a count and has positively skewed distributions in both sexes.

Secondly, we estimated (for each sex separately) the within-individual covariance between dominance and fitness using a bivariate linear mixed effects model with individual identity fitted as a random effect. For each observed interaction, one (randomly chosen) individual within the pair of animals was assigned a phenotypic value of 0 or 1 according to the contest outcome. Henceforth, we refer to this individual as the focal animal and the other individual as the opponent. This approach was taken such that each dyadic observation contributes a single phenotypic record to the analysis, which is appropriate as the phenotypic status of the focal completely determines that of the opponent and vice versa. We modelled fitness (LBS) and contest outcome as response variables, with identity of the focal individual as a random effect on both traits. In addition to controlling for pseudoreplication, the random effect of focal individual allows us to model dominance as a latent individual-level variable that influences observed contest outcomes and is assumed to vary among animals. It also allows us to estimate the covariance between this latent variable (dominance) and fitness.

Our second approach has the advantage of allowing the fitness-phenotype covariance estimate to be conditioned on known sources of trait variation. Here, we included age difference (focal individual’s age in years minus opponent’s age in years) and sex code as fixed effects on observed phenotype. We included a linear effect of age difference as older individuals are (on average) dominant to younger ones (Clutton-Brock et al., 1982; Thouless & Guinness, 1986; Fig. 2). This represents a potential source of bias in the first simple regression estimate of selection because individuals that live longer will be the older individual in a higher proportion of their dyadic interactions and will also have (on average) higher fitness (even if dominance is not causally related to longevity or fecundity). Thus, the age effect could in itself be sufficient to cause apparent positive selection on dominance. Preliminary data analysis also revealed that while females tended to win more often in cross-sex contests, this is likely due to the different age distributions between the sexes. In fact, considering only interactions between animals of the same age, males were significantly more likely to win in contests against females than vice versa (Fig. 2). We therefore included an additional fixed effect of sex code as a three-level factor corresponding to the possible dyadic interaction types (1 = same sex dyad, 2 = female focal with male opponent, 3 = male focal with female opponent).

Figure 2.

 Proportion of observed dominance interactions won in encounters between animals of differing age or sex. Older individuals tend to be dominant to younger ones, and while females are more likely to dominate males overall, this is because of differences in average age between the sexes (see text); males tend to dominate females of the same age. Deviations from null expectations (i.e. 50 : 50 proportions) are highly significant with binomial probabilities < 0.0001 for all three comparisons.

A limitation of this analysis is that we are currently unable to fit the multivariate generalized linear mixed effect model that would be most appropriate to these two response variables (as contest outcomes are binary observations and LBS is best modelled using a negative binomial or quasi-Poisson distribution). We therefore fitted the model assuming errors follow a multivariate normal distribution and note that statistical inferences drawn from the model should be treated as provisional.

Repeatability and genetic variance for social dominance

Genetic and environmental components of variance for social dominance were estimated from phenotypic and pedigree data. Models were formulated to test for repeatability of social dominance, the presence of additive genetic variance and finally the presence of IGEs. Testing repeatability requires that the total variance in a trait can be partitioned into within- and among-individual components. Because this is only possible given repeated measurements on individuals, all quantitative genetic analyses were carried out on the observed data (i.e. contests) rather than on summary lifetime statistics for each individual. Dominance was therefore modelled with generalized linear mixed effect models (GLMM) using a logit link function to relate binary observations to an underlying (normal) latent scale.

Models fitted

Each dyadic interaction contributes a single record to the data analysis. For observation x of focal individual i with opponent j, the observed contest outcome (0/1) is given as:


where δ is a draw from a Bernoulli distribution with probability logit1(lijx), lijx is thus the liability on the logit scale of i winning the contest. We fitted a series of models, identical in their fixed effects but differing in their random effects, to predict lijx:

image(model 1)
image(model 2)
image(model 3)

Fixed effects included the linear function of age difference (years) and sex code (as described earlier).

In model 1, we included a random effect of focal identity focali to estimate the between focal individual variance (VF). If scaled by the total phenotypic variance, VF can be interpreted as the repeatability of focal dominance. Model 1 also contained a random effect of opponent identity opponentj to estimate the variance in opponent effects (VO), the extent to which different opponents have repeatable effects on contest outcomes. Under model 2, the influence of the focal individual i is partitioned into an additive genetic effect (aFi) and a nongenetic ‘permanent environment’ effect peFi, allowing estimation of the corresponding variance components VF.A and VF.PE Under this model, only direct genetic effects are included and VF.A is simply the conventional additive genetic variance. Model 2 is therefore a standard animal model (Wilson et al., 2009b) for a trait with repeated measures, in which the opponent effect is treated as a purely environmental source of variance. In model 3, we apply a similar decomposition of opponentj to test for IGEs. This allows partitioning of VO into additive genetic (VO.A) and nongenetic (VO.PE) components, where VO.A is the variance attributable to IGEs.

Models 1–3 allow focal and opponent influences to be estimated and partitioned into genetic and environmental components and they also contain an implicit but likely erroneous assumption that no correlations exist between the direct and indirect effects. As the designation of focal and opponent individuals is in fact completely arbitrary for each observation x, it follows that focal and opponent effects should be perfectly correlated within individuals. Thus, for any individual i, it can be argued that focali = −opponenti, and therefore, aFi = −aOi, and peFi = −peOi. From this it follows logically that we expect equalities to exist among the variances such that VF.A = VF.O and VPE.A = VPE.O whereas the correlations between direct and indirect effects (denoted rG.FO and rPE.FO for genetic and permanent environment correlations, respectively) should equal −1. To test this expectation, we fitted two further models. Model 4 was specified as model 3 but with a potentially nonzero correlation between direct and IGEs directly estimated to test our expectation that they would equal −1. Finally, we compared the effect size estimates under model 4 to those in a model constrained to be consistent with our expectations described elsewhere (Model 5). Thus, we estimated the magnitude of genetic and permanent environment effects under the forced conditions that VF.A = VO.A, rG.FO = −1, VF.PE = VO.PE, rPE.FO = −1.

Random effects were assumed to be drawn from normal distributions with means of zero and variance/covariance matrices among individuals of inline image and inline image. Variance components are as described earlier (note that here we use σ2 to denote a true variance as opposed to V to denote its estimate above) and are assumed to be homogeneous across sexes. I is an identity matrix with order equal to the number of individuals; A is the additive numerator relationship matrix containing the individual elements Aik = 2Θik; and Θik is the coefficient of coancestry between any pair of individuals i and k in the pedigree.

Validation of quantitative genetic analyses

All quantitative genetic models were solved by penalized quasi-likelihood (PQL) implemented in the program ASReml v2 (Gilmour et al., 2006). This method is appropriate for the binary observations of contest outcomes but is not without difficulties. First, estimates of random effects can be biased when GLMMs are solved by PQL approximations (Goldstein & Rasbash, 1996; Rodriquez & Goldman, 2001; Bolker et al., 2009). We therefore undertook power and sensitivity analyses, using simulation-based approaches advocated by Morrissey et al., (2007) to assess the performance of the GLMMs, as fitted in our dataset. Specifically we sought to determine whether biases could exist in our analyses, given the particular conditions of our analyses (i.e. size and structure of the pedigree, number of observations and distribution of observations among individuals). A second difficulty is that appropriate statistics for hypothesis tests of random effects in GLMMs solved by PQL are unknown. We therefore used the simulations to determine, again under the particular conditions of our analyses, whether evaluation of the ratio of the estimate to its estimated standard error, as reported by ASReml, provided practical means of evaluating statistical significance.

Full details of these simulation-based analyses are presented in Appendix 1. In brief, the results indicated that our estimates of genetic variance components are likely to be downwardly biased (a known problem with PQL approximations, Bolker et al., 2009). However, under simulation conditions designed to mimic the actual analyses of dominance in red deer, these biases are not large enough to hinder the biological interpretation of our model-based estimates (see Appendix 1). Simulations also demonstrated that the common ‘rule of thumb’ of assuming statistical significance when the ratio variance/standard error ≥ 2 is justifiable here (and is expected to be conservative with respect to a nominal significance level of α = 0.05). Lacking a method to generate exact P-values, we have therefore used this criterion as a de facto test of statistical significance for (co)variance components in what follows. Note that we do not advocate the application of this criterion to other studies unless simulations specifically tailored to the appropriate data structures can be shown to provide similar support.


Selection on social dominance

Consistent with our a priori expectation of positive selection, regression analyses showed positive relationships between fitness and social dominance. Ordinary least-squares regression analysis of LBS on mean dominance suggested a significant relationship in females [β = 5.94 (SE 0.466), P < 0.001] though not in males [β = 1.07 (SE 0.693), P = 0.125]. These estimates correspond to standardized linear selection differential estimates (SE) of 0.400 (0.032) for females and 0.207 (0.135) for males, respectively. The P-values from simple linear regression given earlier are not strictly valid given the deviation from residual normality, but negative binomial regressions of LBS on mean dominance yielded qualitatively similar results (females: coefficient = 1.398 (SE = 0.117), P < 0.001; males: coefficient = 0.647 (SE = 0.511), P = 0.206).

Under the bivariate mixed model we estimated the within-individual covariance (SE) between focal dominance (conditioned on age difference and sex code) and LBS as 0.057 (0.032) in females. Although positive as expected, this effect was marginally nonsignificant under a two-tailed test (likelihood ratio test comparison to a reduced model with covariance fixed at zero: inline image, P = 0.084). Dividing by the among-individual variance estimate for focal dominance yielded a regression coefficient of β = 2.88 (1.61) and a corresponding standardized selection differential of 0.333 (0.187). A positive covariance term was also estimated in males and was found to be statistically significant using a likelihood ratio test [within-individual covariance (SE) = 0.139 (0.060); inline image, P = 0.019], with a regression coefficient of β = 5.65 (2.45) and a standardized selection differential of 1.639 (0.712). As a caveat to these results, we reiterate that the standard assumption of multivariate residual normality was violated, and P-values associated with these bivariate models should therefore be treated with appropriate caution.

Thus, estimates of the strength of selection do differ under the two approaches used. This is primarily because of accounting for age effects in the bivariate mixed model, which were significant predictors of contest outcome for both focal females [age difference coefficient (SE) = 0.058 (0.001), F1,7468.1 = 3615, P < 0.001] and focal males [age difference coefficient (SE) = 0.045 (0.002), F1,2389.4 = 324.7, P < 0.001]. In contrast, sex code was not a significant predictor of contest outcome for either sex. For focal females, the predicted mean outcome (SE) was 0.461 (0.009) against a female opponent and 0.447 (0.012) against a male (F2,7915 = 1.67, P = 0.189). For focal males, the corresponding predicted means were 0.497 (0.020) against a male opponent and 0.453 (0.016) against a female (F2,2279.6 = 2.43, P = 0.089).

Genetic influences on social dominance

Repeatability and genetic (co)variances for social dominance

Under model 1, we found evidence for significant effects on focal dominance of both the focal and opponent identities (Table 1). Thus, both individuals in the interaction have a repeatable effect on the observed phenotype consistent with among-individual variance in social dominance. Furthermore, we found evidence for a heritable component of variance in focal dominance (Table 1). Under model 2, the estimated direct additive genetic variance was estimated as 0.314 (0.106), yielding an estimated direct heritability of 0.095 (0.032) if we make the standard assumption for the logit link that residual variance = Π2/3.

Table 1.   Components of variance (on the liability scale) for social dominance estimated from models 1–4.
ModelVFVF.AVF.PEVOVO.AVO.PErG,FOrPE.FOParameter constraints
  1. Variance components for focal individual (VF) and opponent (VO) estimated under model 1 are partitioned into additive genetic (VF.A, VO.A) and permanent environment (VF.PE, VO.PE) in models 2–5. Estimated standard errors are indicated in parentheses and indicate statistical significance at the nominal level (α = 0.05) for all variance component estimates (and the direct–indirect genetic correlation (rG,FO) under Model 4). rPE.FO denotes the within-individual correlation between direct and indirect permanent environment effects. Absence of a standard error indicates a parameter is fixed because of a model constraint and is therefore not freely estimated (see text for details).

11.02 (0.094)  0.836 (0.082)  00No direct–indirect effect covariance
2 0.314 (0.106)0.706 (0.113)0.849 (0.083)  00No direct–indirect effect covariance
3 0.366 (0.113)0.677 (0.113) 0.269 (0.093)0.575 (0.099)00No direct–indirect effect covariance
4 0.364 (0.119)0.956 (0.129) 0.363 (0.112)0.819 (0.118)−0.913 (0.065)−1.080 (0.030)No constraints
5 0.346 (0.105)0.910 (0.110) 0.3460.910−1−1rG.FO, rPE.FO = −1, VF.A = VO.A, VF.PE = VF.PE

Because it is widely known that social inheritance of dominance rank can occur (Holekamp & Smale, 1991), we tested for nongenetic maternal effects that, if present, are expected to cause upward bias in VA.F by adding a random effect of maternal identity to Model 2. Maternal variance was low and nonsignificant [VM = 0.077 (0.061)], and the estimate of VA.F showed virtually no reduction on the inclusion of VM [VA.F = 0.312 (0.113)]. Based on these results, we concluded that nongenetic maternal effects are unlikely to represent an important bias on additive effects here and we did not include them in subsequent models (but see later for important discussion relating to this issue). In comparison with the GLMM results, we also estimated heritability under model 2 using data on the observed (0/1) scale and a normal animal model and then transformed this estimate to the liability scale following Dempster & Lerner, (1950). This approach gave slightly lower estimates of h2 = 0.041 (0.013) on the observed scale and 0.064 (0.020) for the liability scale.

Models 3 and 4 indicate that the opponent variance contains a significantly heritable component consistent with the presence of IGEs on the expected contest success of focal individuals (Table 1). Furthermore, under Model 4, we estimated a statistically significant genetic correlation between direct and IGEs of −0.913 (0.065), in close accordance with our expectation of a negative genetic correlation of −1. Under Model 4, direct and indirect additive components were of nearly identical magnitude; the permanent environment variance components associated with focal and opponent individuals were also very similar (Table 1). These results are therefore quantitatively consistent with our logical argument that focal and opponent effects must be drawn from the same distribution but be perfectly negatively correlated. Under this scenario (model 5), additive and permanent environment variance estimates were again significant with VF.A = VO.A = 0.346 (0.105) and VF.PE = VO.PE = 0.910 (0.110). Although parameter estimates are very similar under models 4 and 5, it is difficult to formally compare models 4 and 5 as likelihood-based methods (e.g. likelihood ratio test, AIC) are not valid using the PQL solutions obtained here. As a way to informally compare model fits, we calculated the variance in working residuals (on the observed data scale, i.e. observed contest outcome – predicted contest outcome). For model 4 we estimated Vworking residuals as 0.103 whereas the corresponding estimate for model 5 was 0.101. Thus, while the variances in working residuals were almost identical, a lower estimate was actually obtained under model 5 despite the fact that fewer parameters were estimated.


Our analyses support the a priori expectation that dominance is positively associated with fitness, estimated here as LBS, in red deer. This conclusion is qualitatively consistent across the various complementary selection analyses that we conducted, although statistical support was equivocal in some cases. For example, the simple regression approach yielded an estimate of very strong directional selection in females, but this may be upwardly biased by age effects. Specifically, females that live longer will have higher lifetime fitness (on average), but may also have higher mean dominance by simple virtue of being the older individual in more of their dyadic interactions. The correlation between age and dominance rank was previously known in this population (Clutton-Brock et al., 1982; Thouless & Guinness, 1986) and has also been reported in other ungulate populations (Festa-Bianchet, 1991). Accordingly, with age controlled for in the bivariate model of female LBS and focal dominance, the estimated strength of selection was reduced by approximately half (though the standardized selection differential estimate of 0.333 remains high relative to the distribution of estimates in the literature, Kingsolver et al., 2001). Conditioning the selection estimates on age effects in this way reduces the potential bias from age effects but will be overly conservative if higher dominance does causally influence lifetime fitness by effects on survival and hence longevity.

In males, the relationship between dominance and fitness was also positive, although much weaker (and nonsignificant) when selection on mean dominance was estimated from simple regression. However, in males, age effects may induce downward (rather than upward) bias in the selection estimate given that only young focal males were included in the data. Accordingly, with age difference effects accounted for, the estimated strength of selection on dominance in males was actually stronger in males (standardized selection differential of 1.639) than in females, a finding that has been reported elsewhere (Ellis, 1995). Although direct comparison between the sexes is complicated by the differing age distributions of females (all ages) and males (primarily juveniles), a similar result was obtained if the dominance data for females were restricted to observations of juveniles (< 3 years; results not shown). Although adult male–male interactions during the rut were not included in our study, one possibility is that strong selection on male focal dominance results, at least in part, from a high within-individual correlation between dominance during feeding interactions as a juvenile and success in adult male–male contests later in life. The ability to hold a harem and defend it from other males during the rut is a key determinant fitness in adult males (Clutton-Brock et al., 1982).

Thus, our results are largely consistent with positive selection on focal dominance in both sexes although our data certainly do not allow us to prove a causal relationship between the trait and fitness (particularly as we have considered only a single phenotypic trait, Lande & Arnold, 1983). Nonetheless, plausible mechanistic hypotheses linking dominance to fitness do exist. For example, previous analyses of female–female interactions in the Rum red deer population have shown that feeding rate is decreased in subordinates by the close proximity of dominant neighbours, most likely due to movements away by the subordinates that require a break from feeding (Thouless, 1989). Correlations between feeding time and social rank have also been shown in female red deer elsewhere (Veiberg et al., 2004), whereas female–female aggression has been linked to resource competition in other ungulates (Festa-Bianchet, 1991; Robinson & Kruuk, 2007).

Our models of focal dominance also provided evidence that the outcome of dyadic interactions is repeatable within individuals. In other words, there is evidence that, after conditioning on age difference and sex, there is among-individual variance in the likelihood of winning interactions. This finding is broadly consistent with earlier conclusions relating to this population (Thouless & Guinness, 1986). Consistent differences among individuals may arise from a variety of nonmutually exclusive processes including ‘winner’ and ‘loser’ effects (when current contest outcome depends on, and is positively correlated with, prior contest outcomes, Dugatkin, 1997), differences in morphological traits (e.g. size) associated with assessment of resource holding potential (Parker, 1974) or among-individual variation in agonistic behaviours involved in mediating contest outcomes. Whereas our data do not allow us to disentangle these mechanisms, previous analyses have shown a correlation between female dominance rank and body size in this population, suggesting that this is likely to be an important trait (Clutton-Brock et al., 1984).

Our analyses also indicate that a substantial proportion (approximately 27% under model 4) of the among-focal individual variance in dominance is attributable to direct additive genetic effects. Furthermore, our results also supported the final hypothesis that contest outcomes are subject to IGEs. Thus, the dominance status of a focal individual will depend to some extent upon its genotype, but whether a focal individual wins any given contest will also depend upon the genotype of the opponent with which it interacts.

It is important to acknowledge that there is abundant evidence for social inheritance of maternal dominance rank in some taxa (most notably primates and hyenas, Holekamp & Smale, 1991). In the absence of experimental manipulation, it would be foolish to completely rule out the possibility of unmodelled mechanisms of social inheritance upwardly biasing our estimated genetic parameters. However, we note that the animal model framework is far more robust in this regard than classical approaches such as mother–daughter regression. This is because estimates of the genetic parameters are informed by distant relatives (including many that share ancestry through paternal relationships only). Furthermore, the model can be extended to explicitly model maternal effects (Kruuk & Hadfield, 2007), allowing a direct test of the hypothesis that maternal identity explains variation in contest success. Here, there was no statistical support for the presence of (nongenetic) maternal effects and inclusion of maternal identity as a random effect in the model had little impact on genetic parameter estimates. We therefore believe our conclusion that dominance is heritable to be statistically robust.

Accepting that dominance is indeed under direct selection (i.e. there is a causal relationship between focal dominance and fitness) and heritable, then the presence of IGEs provides a resolution to the apparent paradox that a heritable trait is unable to respond to selection (i.e. with one winner and one loser in each contest, the mean contest outcome must always equal 1/2). Recent theoretical work has shown that the expected response to selection depends not just on the conventional additive genetic variance, but also on the indirect genetic variance, the genetic covariance between direct and indirect effects and the relatedness structure among interacting individuals (Wolf et al., 1998; Bijma et al., 2007b; Bijma & Wade, 2008). We have argued that in a contest between two animals there is only one independent observation. If the focal individual is observed to win the contest, then it follows that the opponent must have lost. As our designation of focal and opponent individuals is arbitrary, it is necessarily true that if the focal genotype influences contest outcome (i.e. there are direct genetic effects), then so must the opponent genotype (i.e. there must be IGEs). This argument is supported by the results of our empirical analyses.

Furthermore, if direct and IGEs on dominance must have equal variances and a perfect negative correlation, the sum of VA.F and COVA.FO must equal zero. For the case of two interacting individuals, the sum of VA.F and COVA.FO determines the amount of heritable variation on which selection among individuals can act (from equation 15 of Bijma & Wade, 2008, for n = 2). Under the constraints imposed in model 5, this sum must necessarily equal zero, whereas under model 4 we estimated this sum (with SE) as 0.032 (0.052). Note also that under model 5 the variance in total breeding values (sensu, Bijma et al., 2007b) is also zero, and therefore, a response in the mean phenotype is not expected, regardless of whether multilevel selection is present and irrespective of the relatedness structure (Bijma & Wade, 2008). Thus, model 5 posits a zero-sum game in which the perfect negative correlation between direct and IGEs creates an absolute constraint on mean contest winning rate, a trait that is both heritable and under selection. We note that although formulated and parameterized somewhat differently, model 5 is essentially equivalent to a theoretical model proposed by (Wolf et al., 2008) as a resolution for the lek paradox. In both contexts, competition among individuals is predicted to cause maintenance of additive genetic variance and an absence of observable phenotypic evolution.

While there is an absolute constraint on the evolution of mean observed phenotype (i.e. contest outcome), this should not be taken to mean that a constraint on genetic change is present. Neither does it follow that the phenotypic means of traits contributing to dominance are evolutionarily constrained (discussed further later). In fact breeding values for, and gene frequencies at loci influencing, dominance are free to evolve. Verbally, the argument here is that a gene predisposing a focal individual to dominance should be selected and so increase in frequency. However, while a focal individual in a subsequent generation will be more likely to carry that gene, it will also be more likely to encounter an opponent carrying it (thus negating any advantage with respect to contest outcome). This represents a particular case of ‘environmental deterioration’ (Fisher, 1958; Frank & Slatkin, 1992) or the ‘treadmill of competition’ (Wolf, 2003), which occurs when the phenotypic gains naively expected from selection on (and spread of) a more competitive genotype are not realized because of increased competition in the social environment (Bijma, 2010; Hadfield, J. D., Wilson, A. J. and Kruuk, L. E. B., submitted). Although it is not feasible to conduct such an experiment in red deer, the phenotypic consequences of an underlying genetic response can be made apparent if individuals are transplanted to a different social environment. This was demonstrated by Moore et al. (2002) who found that despite little change in agonistic behaviour within selected lines of cockroaches, after seven generations individuals from high-selected lines were consistently dominant in trials against individuals from low-selected and control lines.

In providing an explanation for what we already know must be true (i.e. that the mean proportion of contests won cannot evolve), it could be argued that our analyses provide relatively little biological insight into the evolutionary dynamics of social dominance in red deer on Rum. We also note that quantitative genetic models of the sort developed here are intended to predict the per generation selection response for a phenotypic mean, not changes in higher-order moments (e.g. variance or skew in dominance), or evolutionary stable strategies (Maynard Smith & Price, 1973).

However, aside from the point that we are now able to empirically estimate the genetic components of (co)variance and so recover this ‘truth’, we hope that the present result also highlights the need for wider recognition of IGEs and their consequences. For instance, although indirect effects (genetic or otherwise) must exist for dominance interactions, the same logic can be extended to any resource-limited trait where phenotype is determined, at least in part, by competitive interactions among individuals. Thus, if competitive ability (i.e. ability to acquire and/or monopolize a limited resource) is heritable, then we expect IGEs to act as evolutionary constraints on resource-limited traits (e.g. growth). Under the IGE modelling framework employed here (i.e. Muir & Craig, 1998; Bijma et al., 2007b), genetic variance in competitive ability should be manifest as a negative covariance between direct and IGEs on resource-limited traits.

We also note that the advantages of directly modelling the observed data using mixed effects models are certainly not limited to genetic studies. For example, in contrast to derived dominance metrics, the observed data come from a defined and known distribution, whereas effects such as age or sex are readily incorporated as explanatory covariates. Furthermore, by testing the causes and consequences of dominance interactions directly, we can avoid the widespread, but statistically unfortunate practice of analysing dominance metrics under the assumption they are known without error (Poisbleau et al., 2006).

A further point of interest is that if dominance (or in a more general context simply competitive ability) is under selection, then the evolutionary consequence of IGE-based constraints will vary among correlated traits according to their causal relationships. For instance, consider a (purely hypothetical) situation in which a morphological trait (e.g. weapon size) was a strong determinant of success in contesting for access to a limited food resource and therefore a strong determinant of growth. If weapon size were heritable, then we would expect positive (direct) genetic correlations between it and both dominance and growth rate. Consequently, we might naively predict that direct selection on dominance would induce positively correlated responses in the other two traits. However, with an IGE-based constraint on dominance recognized then this expectation changes. Whereas the trait causal for dominance (i.e. weapon size) would still evolve as a correlated response to the selection, the downstream trait dependent on contest outcomes (i.e. growth) would be constrained.

In conclusion, our results highlight that social dominance can be a heritable trait. However, it is also a latent trait that, rather than being directly measured, is inferred through observations of contest outcomes. Winning or losing a contest is not a phenotypic observation that can be treated as belonging to a single individual. Rather contest outcomes result from interacting phenotypes and – so by extension – from interacting genotypes (Moore et al., 1997). If the outcomes of these interactions are influenced by additive genetic effects, then they must also be influenced by IGEs. It has been argued that IGEs arising from competitive interactions could represent a widespread, but poorly recognized source of evolutionary constraint for resource-limited traits under natural selection (Cooke et al., 1990; Hadfield, J. D., Wilson, A. J. and Kruuk, L. E. B., submitted). For the particular case of dominance as defined here, we know a priori that no amount of selection can result in an increase in the mean observed phenotype, but it is only by explicit recognition and estimation of IGEs that we can demonstrate the genetic basis of this constraint.


We are grateful to Scottish Natural Heritage for allowing us to work on Rum and to the Rum community for their support. Our particular thanks go to the field assistants on Rum and to the Natural Environment Research Council, the Biotechnology and Biological Sciences Research Council (BBSRC), and the Royal Society for supporting the long-term data collection on Rum. AJW is funded by a BBSRC David Phillips fellowship, MM by the Natural Sciences and Engineering Research Council of Canada and LK by the Royal Society.


Appendix 1: validation of GLMM methodology

Simulation approach

We simulated varying rates of additive genetic variance, repeatable among-individual variation and within-individual variation (essentially overdispersion in the context of GLMM analyses) for social dominance. The currently available pedigree for the red deer system has an estimated overall confidence level of 97.6%, and because most remaining null assignments are likely to unsampled individuals from outside of the study area (Walling et al., 2010), we simulated breeding values over the available pedigree, assuming that it is complete and correct. In doing so, we effectively treat all null parentage links in the pedigree as links to unrelated founders, and we deemed this a safe approximation because the population outside the study area is relatively large compared to the number of immigrants to the study area. We composed simulated outcomes of each contest in the real data according to the probabilistic formula


where zijx is the observed phenotype on the 0/1 scale, δ is a draw from a binomial distribution, and lijx is the latent scale probability of that individual i wins contest x against opponent j and is calculated as


where a represents breeding values, b represents nongenetic repeatable individual effects, and e represents a residual error on the latent scale. We simulated data for 100 replicates of each of 16 parameter sets, comprising all possible combinations of Va, Vb and Ve with the following values: Va of 0, 0.125, 0.25 or 0.5; Vb of 0 or 0.5; and Ve of 0 or 0.5. We analysed each simulated data set using model 4 (as described above). For every replicate analysis, we recorded the estimate of Va and the ratio of the estimate to its estimated standard error, so that we could subsequently evaluate both bias in Va and the utility of comparison of the magnitudes of the estimate to its estimated standard error for hypothesis testing.

Simulation results

We observed an overall trend for estimates of Va to be downwardly biased by about 10% across all nonzero simulated levels of additive genetic variance for dominance (Fig. 3a). The presence of simulated environmental variation had a tendency to very slightly increase this bias (Fig. 3a). These two patterns in bias are expected for PQL solutions to GLMMs of a binary trait (Bolker et al. 2009, Rodriquez & Goldman, 2001; Goldstein & Rasbash, 1996), but under the conditions of our analyses of variation in dominance in red deer, these biases are not large enough to hinder the biological interpretation of our model-based estimate.

Figure 3.

 Evaluation of (a) bias and (b) power in penalized quasi-likelihood solutions to generalized linear mixed model analyses of dominance in red deer, Cervus elaphus. Each cluster of four bars with decreasing shading represents results from simulations with values of nongenetic repeatable variance (Vb) and within-individual variance (Ve) set to (i) Vb = 0, Ve = 0, (ii) Vb = 0.5, Ve = 0, (iii) Vb = 0, Ve = 0.5, (iv) Vb = 0.5, Ve = 0.5. The 1 : 1 line on panel (a) indicates where unbiased estimates should lie.

Under the null hypothesis with respect to additive genetic variance, the ratio of estimated Va to the estimated standard error of Va proved to be a conservative test statistic. Application of the common ‘rule of thumb’ that ratios over 2 are statistically justifiable, no false positives occurred in the simulated data for the four combinations of magnitudes of nongenetic sources of variance that we simulated. Given that hypothesis testing of Va is a one-tailed test, i.e. we typically do not attach biological meaning to negative Va, threshold Z-value of 1.64 corresponds to α = 0.05 may be justifiable. At this threshold significance level, false-positive rates ranged between 0% and 3%. When we simulated the presence of additive genetic variation for dominance, our study design proved to provide substantial power to detect even modest levels of Va (Fig. 3b). Whereas estimates of Va were largely unbiased by Vb (Fig. 3a), the presence of nonzero Vb did reduce power to detect Va (Fig. 1b). At the lowest level of Va that we simulated (i.e. Va = 0.125), average power (i.e. the rate of significant tests) was estimated as 1 in the absence of repeatable among-individual variation and 0.755 when Vb = 0.5, at the threshold values of the ratio of the estimate to estimated standard error of 1.64. The corresponding estimates of power were 1 and 0.655 at the ‘rule of thumb’ threshold of 2 for this ratio.