The genealogical view of speciation


Kerry L. Shaw, Department of Biology, University of Maryland, College Park, MD 20742, USA.

Species concepts are paradigms that inspire questions about how new species arise, and thereby play an important role in directing the speciation biologist toward worthwhile topics for investigation. For over 50 years, Mayr’s (1942, 1963) biological species concept (BSC) has been the prevailing paradigm. According to Mayr’s BSC, a group of organisms is a species if and only if it has evolved isolating mechanisms that can effectively protect its uniquely co-adapted genome from the influx of genes from close relatives. Because Mayr (1942, 1963) also argued that most speciation occurs in allopatry, isolating mechanisms were described as devices that are not the primary targets of selection but rather are the incidental by-products of allopatric divergence. Despite its ‘incidental status’, decades of research have been devoted to understanding the genetics and evolution of reproductive isolation (RI), as this has been seen as the central issue in the study of speciation (Coyne, 1992; Orr, 2001).

Some 20 years ago in an influential series of papers, Paterson (see Paterson, 1993) dubbed Mayr’s BSC the ‘isolation concept’ and effectively called for a paradigm shift by arguing that RI is not a generally tenable criterion of species because it is merely an effect resulting from other, more primary, divergence processes. Although Wu (2001) (hereafter ‘Wu’) does not focus exclusively on mate recognition as Paterson did, his argument for the ‘genic’ view of species and speciation is largely the same. Differential adaptations should be recognized as the features that make species more than merely divergent populations of the same species, so long as those adaptive differences can be maintained in sympatry. As Wu points out, turning emphasis in speciation studies to forces of differentiation steers us toward processes directly involved in speciation, rather than incidental by-products (RI) of differentiation. Paterson (1985) spent a good deal of time on this point, and although one may object to placing critical importance on selective divergence (particularly of mate recognition systems) as solely responsible for the evolution of new species, the emphasis on divergence rather than RI is sound.

Wu also points out that an emphasis on forces of differentiation can accommodate interspecific gene flow without jeopardizing a valid diagnosis of species status. Such a change in emphasis is badly needed. In the last 15–20 years, molecular research has shown that a significant number of ‘species’ experience some level of interbreeding (Grant & Grant, 1998; Mallet et al., 1998; Schluter, 1998) or share some regions of their genome with close relatives (Ting et al., 2000) because of some level of ongoing gene flow. The conclusion is that some genomic regions can travel between species despite the fact that the species in question continue to differentiate. This represents a major adjustment in how we view species in the natural world and is justification enough for a species paradigm shift away from RI, the defining feature of Mayr’s BSC.

The middle ground

From the above discussion, the choice seems clear: as speciation biologists, we either study RI under the paradigm of the BSC or we study the primary forces of differentiation (that may or may not lead to RI incidentally). Nonetheless, Wu continues to tread a middle ground both methodologically and conceptually. Methodologically, one can easily defend interspecific hybridization studies as valuable, without promoting RI, because such studies allow us to genetically analyse the basis of species differences that are implicated in the process of speciation (Shaw & Parsons, 2002). Although the study of hybrid male sterility, one of the traditional hallmarks of RI, seems an odd choice at first because it is not a phenotype that can be selected for, Wu et al. have generated provocative results (Ting et al., 1998) that implicate a gene involved in rapid divergence of spermatogenesis function. Such a result may bear on the forces of speciation in the Drosophila melanogaster clade.

Although methodologically on firm ground, the conceptual character of Wu’s argument still smacks of RI. Consider the following summary:

‘Species are groups that are differentially adapted and, upon contact, are not able to share genes controlling these adaptive characters, by direct exchange or through intermediate hybrid populations’ (Wu, 2001).

First, the idea that species are the result of adaptive processes is imbedded in Wu’s genic view of species, a view also very much at the core of Mayr’s BSC. Secondly, the genic view requires geographical ‘contact’ to determine whether adaptive differentiation has proceeded beyond the stage of functional compatibility between two putative species. Thus, species are fully evolved if and only if the adaptive differences between them can prevent their fusion when in geographical contact. If the criterion of RI were truly removed from the genic view of species, geographical contact would play no role in our understanding of the nature of species. By Wu’s argument, it is only through the relative comparison of two putative species, as opposed to the inherent properties of each, that we can understand the nature of species. Mayr’s view was similar, and suggests probable motive for RI as the ranking criterion of the BSC:

‘The word species indicates a relationship, like the word brother. Being a brother is not an inherent property of an individual, as hardness is the property of a stone. An individual is a brother only with respect to someone else. A population is a species only with respect to other populations. To be a different species is not a matter of difference but of distinctness’ (Mayr, 1963, p. 19).

Wu’s genic view of species is ambiguous in allopatry because the fundamental integrity of incipient species can be realized only through geographical contact – a view differing little from that of Mayr (1942, 1963). To understand the process of speciation in any geographical mode, species must be understood through inherent rather than relational features. To see how important this is, consider the extremely effective argument that Mayr (1963, chapter 16) used to show that evolution in allopatry is the dominant mode of speciation. Despite recent advances suggesting that nonallopatric speciation is possible (reviewed in Via, 2001) and perhaps even common, few would doubt that allopatric speciation remains a significant, if not dominant, mode. We need a species paradigm that can lead us to understand the process of speciation in sympatry or allopatry.

The genealogical view of speciation

The genealogical view of species (Baum & Shaw, 1995; Shaw, 1998) is based on shared historical relationship, an inherent feature of a group of organisms. A genealogical species is a basal group of organisms, whose members share exclusive genealogical relationship (Baum & Shaw, 1995). Genealogical relationship in this context refers to how far back in time members of a group must trace their ancestry (e.g. assessed through composite gene-genealogies) before they are connected by progenitors in common. Like Wu’s genic view of species, there is no fixed expectation about degrees of gene exchange between species, except to say that the level of gene exchange between species cannot be so high that it destroys their defining feature of exclusive relationship. Unlike Wu’s genic view of species, genealogical species can be recognized in sympatry or allopatry because genealogical relationship can be determined independently of geographical situation.

How do genealogical species arise? Speciation resulting in genealogical species will occur by all the familiar evolutionary processes that cause differentiation: genetic drift and selection, whether in sympatry or allopatry, with influences from population size and genetic architectures of selected phenotypes. All of these components can affect the rate of differential gene lineage extinction that results in the boundaries around genealogical species.

Consider a simple case. A single lineage, characterized by sexual, biparental reproduction and panmixia, splits into two as a result of the establishment of a geographical barrier across which there is no migration. Initially, the descent-relations for any given gene within either of the two daughter populations will be polyphyletic (Avise & Ball, 1990). Over time, as extinction of some of the lines of descent of these gene copies occurs as a result of genetic drift within either daughter lineage, the genes that persist will increase in the degree to which they are genealogically related to one another within each daughter lineage. The boundary state, where the last qualitatively distinct stage has been reached, is reciprocal monophyly (Avise & Ball, 1990; Baum & Shaw, 1995).

Over time, we expect the same sort of behaviour for all genes in the genome; the transition of genes that experience directional selection, or that are linked to directionally selected regions, will occur rapidly (Kaplan et al., 1989), whereas the transition of neutral genes will occur more slowly. In the simplest case where two populations are diverging simply due to drift, genealogical species will arise at a rate that is a function of the effective population size. Exclusivity, where members of the daughter population are more closely related to each other than to organisms outside the population, applies once the majority of gene copies each find their closest genealogical relative within that population.

Now consider a more complex case. Suppose some aspect of the environment differs in each daughter population, such that natural selection affects the rate at which genes underlying life history traits or behaviour diverges between them. For example, suppose one daughter population differs from another in its dominant predator, introducing a strong selective pressure for nocturnal vs. diurnal habits. Could such selective differences cause speciation, in terms of the evolution of genealogical exclusivity? If so, we might reasonably label the targets of selection ‘speciation phenotypes’.

To test a hypothesis of the causes of speciation, one would need to know the degree to which the genome would respond to the selective pressures in each population and how much of the genome would experience an accelerated polyphyly–monophyly transition. The genealogical response will be dependent on several factors such as (1) the intensity of selection on the diverging traits, (2) the genetic architecture underlying the traits under selection (the number, interactions, chromosomal distributions and magnitude of effects of the genes involved in the trait), and (3) the degree of recombination between selected vs. nonselected regions of the genome. Mode of inheritance will also play a role in how rapidly exclusive relationship arises for any particular gene in question (Templeton, 1987). For example, in an XX/XY sex determination system, we expect that because the X chromosome has 3/4 the effective population size of diploid autosomes, the genealogical response of X linked regions will occur more rapidly than autosomal regions. Finally and perhaps most importantly, the extent to which selection favours suites of traits that collectively enhance fitness will also affect the rate at which large parts of the genome are accelerated through the transition from polyphyly to monophyly, relative to neutral expectations.

Concluding remarks

Species concepts should be thought of as models about the kinds of groups that the evolutionary process produces (Shaw, 1998). With this in mind, Wu has taken a progressive step forward by identifying assumptions underlying Mayr’s BSC and evaluating those assumptions with empirical data. Wu makes the valuable point that current knowledge on interspecific levels of gene flow undermines RI as a tenable criterion of species. Unfortunately, the genic view Wu proposes does not completely extricate RI as a criterion of species because it relies on adaptive divergence and its manifestations for the future fusion or continued divergence of incipient species in geographical contact.

What makes the genealogical view different from the genic view of Wu? Unlike the genic view where species arise fundamentally by adaptive divergence, the genealogical view is not dependent on any particular evolutionary force. Genetic drift, for example, can give rise to genealogical species. However, selection can also lead to the kind of differentiation that produces genealogical species, and may act on particular traits we might productively label ‘speciation phenotypes’. Notable among such phenotypes are those targeted by sexual selection [e.g. courtship signal/receiver phenotypes (Gray & Cade, 2000) or sexual conflict (Arnqvist et al., 2000)] and natural selection [e.g. beak morphology (Podos, 2001) or larval development time (Feder, 1998)]. In addition, the genealogical view does not predict any particular class of genes that would be appropriately labelled ‘speciation genes’, because all genes in the genome can potentially participate in the speciation process. Finally, no matter what the process that produces species under the genealogical view, they can be identified in any geographical mode.


Thanks to M. Hare, D. Baum, R. Butlin, D. Gill and T. Mendelson for valuable discussions relating to this manuscript. This work is supported by the US National Science Foundation and the University of Maryland GRB.