The evolution of a quantitative trait along an environmental gradient is a topic of interest to population genetics and evolutionary ecology (Slatkin 1978; Kirkpatrick and Barton 1997; Barton 1999; Case and Taper 2000), and is connected to the classical study of gene frequency clines (Haldane 1948; Fisher 1950; Bazykin 1969; Endler 1977). When there is stabilizing selection on a quantitative trait toward a spatially varying optimum, together with gene flow between nearby locations, the traditional expectation has been that gradual variation in the optimum ought to be mirrored by a gradual and, owing to gene flow, equally smooth or even smoother variation in the average value of the trait.

A contrasting perspective was put forward by Doebeli and Dieckmann (2003), who argued that frequency-dependent competition in trait space and local competition in geographical space, acting together with stabilizing selection along an environmental gradient, can create discrete clusters in the distribution of a trait, despite the counteracting effects of gene flow. Doebeli and Dieckmann (2003) claimed that this cluster formation could occur even when frequency-dependent competition would not give rise to disruptive selection in a spatially unstructured, well-mixed population. They emphasized the possible significance of such a process for parapatric speciation, extending earlier work (Dieckmann and Doebeli 1999) that dealt with the possible significance of evolutionary branching (Metz et al. 1992, 1996; Geritz et al. 1998) for sympatric speciation. Our aim here is to shed further light on the evolution of phenotypic clusters along environmental gradients by investigating the causes of phenotypic pattern formation.

The conclusions of Doebeli and Dieckmann (2003) were challenged by Polechová and Barton (2005), who approximated the original individual-based stochastic model by a deterministic reaction–diffusion model. Using Gaussian competition kernels in their approximate model, Polechová and Barton (2005) argued that a gradual environmental cline will lead to gradual variation in a quantitative trait. They suggested that the clustering observed in the analysis of asexual evolution by Doebeli and Dieckmann (2003) was caused by the boundary conditions of the model. In line with earlier findings (e.g., Sasaki 1997), our analysis here confirms that Gaussian competition kernels in themselves are not sufficient to cause clustering, but our overall conclusions are quite different from those reached by Polechová and Barton (2005). In particular, we identify parameter regions of our model in which clustering is a robust outcome, and we show that this outcome does not depend on particular boundary conditions. To investigate cluster formation without an impact of boundary effects, we study a hypothetical, infinitely extended environmental gradient and analyze the conditions for periodic clustering along this gradient.

Using an asexual, deterministic model, we investigate the importance of two different factors for the formation of phenotypic clusters, namely the shape of competition kernels and the strength of Allee effects. There has been a long and influential tradition in ecological modeling to focus on special types of competition kernels, mainly Gaussian and biexponential competition functions (e.g., Roughgarden 1972, 1979; MacArthur 1972; May 1973), which was motivated more by mathematical convenience than by biological realism. A basic aim of our analysis is to provide general insights into the consequences of going beyond this simplifying assumption.

An important aspect of our treatment is that we identify a general property characterizing the shape of competition kernels that promote pattern formation. The crucial property is the sign structure of the Fourier transform of the kernel, in agreement with the analysis of a situation without spatial variation by Pigolotti et al. (2007). If the Fourier transform is nonnegative, as is the case for a Gaussian, the kernel does not by itself cause clustering in models of the type studied here. In contrast, if the transform changes sign to negative values, the kernel shape can induce pattern formation. We provide an intuitive interpretation of this condition, in terms of how the intensity of competition depends on distance in trait space or geographical space.

We also included Allee effects in our analysis. These are among the most-studied phenomena in population biology and are well known for their potential to create spatial variation in abundance (Gyllenberg et al. 1999; Keitt et al. 2001) and to promote species coexistence (Hopf and Hopf 1985). Allee effects are often considered as consequences of the discreteness of individuals, such as when suitable mates or conspecific cooperators become locally rare. It is therefore natural to investigate the influence of Allee effects on phenotypic pattern formation along an environmental gradient. In our analysis, we focus on so-called weak Allee effects, through which a population's per capita growth rate is reduced, but still remains positive at low population density.

With these ingredients, we present results from a numerical analysis of the model and from an analytical approximation in the form of a reaction–diffusion equation. Our analytical approximation is accurate in the low-mobility limit. The importance of the approximation lies in providing a qualitative understanding of the causes of phenotypic pattern formation, which show interesting similarities to the processes giving rise to Turing patterns (Turing 1952). To illustrate the interplay of environmental gradients of different slopes with competition acting locally in trait space or geographical space, we present results for parameter ranges in which stabilizing selection on the trait is stronger than the diversifying influence of competition in trait space. In a spatially uniform and well-mixed situation, such conditions would give rise to a unimodal distribution of trait values (Geritz et al. 1998; Dieckmann and Doebeli 1999), which means that the phenotypic clustering observed in our analysis is gradient induced. We discuss our results in relation to previous work, including the results of Doebeli and Dieckmann (2003), and summarize how the conditions for clustering along environmental gradients obtained in our analysis may help understand patterns in real populations.