Evidence for assembly rules: limiting similarity within a saltmarsh


Correspondence author. E-mail: bastow@bastow.ac.nz


1. We sought evidence for limiting similarity, a basic aspect of community structure, in three zones of a saltmarsh. Sampling was conducted at three spatial scales (grains), from a single point up to the scale of several square metres. Twenty-three functional traits, related to the structure of the shoot and root systems and to nutrient status, were measured on each species present, separately in each community.

2. Patterns of association between plant species were compared with those expected under a null model, to assess whether plants with similar functional traits tended to coexist or to separate, i.e. whether there was environmental filtering or limiting similarity. A patch null model was used, a type that tends to be conservative but that avoids spurious evidence of limiting similarity caused by environmental pseudoreplication. One overall and four univariate test statistics were calculated, to capture possible patterns in trait space whilst minimising the problem of multiple testing.

3. In the Shrub community, overall evidence for even spacing of co-occurring species in functional-trait space, the pattern expected from the theory of limiting similarity, was seen at the area scale. In univariate tests in that community, there was evidence for even spacing in leaf lobation and leaf succulence, especially at small scales.

4. In the Rush community, there was significant evidence for limiting similarity in several traits, especially those related to canopy interactions, but also in some root characteristics. However, clustering in other traits, presumably owing to microenvironmental filtering, reduced overall tests for limiting similarity to ‘marginal significance’ (0.1 > > 0.05).

5. In the species-poor and salt-stressed Salt turf, significant departures from the null model were sporadic and not consistent, although chlorophyll characters and leaf nitrogen concentration tended to be clustered.

6.Synthesis: There was evidence for niche limitation in two of the communities – Rush and Shrub – apparently based on canopy interactions in both cases and perhaps also root interactions in the latter community. Limiting similarity can be an important force in community assembly. However, in situations when it cannot be demonstrated, we do not know whether trait-based competition is absent or whether its signal is overwhelmed by other processes.


The existence of limiting similarity is one of the basic questions in community ecology, derived from the fundamental concepts of competition and niche differentiation. Theoretical values of limiting similarity were calculated by MacArthur & Levins (1967); the concept is behind a whole body of ecological theory (Abrams 1996; Leibold 1998), and it is widely stated in textbooks (e.g. Begon, Harper & Townsend 2006). A few studies of plant communities have reported sound evidence for niche limitation (Cavender-Bares et al. 2004; Stubbs & Wilson 2004; Swenson & Enquist 2009; Kraft & Ackerly 2010), almost always intermixed with the opposite effect of trait clustering. In other studies, no evidence of limiting similarity has been found at all (Schamp, Chau & Aarssen 2008; Thompson et al. 2010). We attempt to find such evidence from a New Zealand saltmarsh, a well-studied ecosystem (Partridge & Wilson 1987a,b, 1988a,b, 1989). Our approach is similar to that of Stubbs & Wilson (2004) on a sand dune, but we here attempt to increase the generality by extending it to another vegetation type; three communities compared to one in the sand dune study, with fewer test statistics (TS) to reduce, although not eliminate, the problem of multiple significance tests.

The demonstration of limiting similarity consists of showing an even distribution of species in functional-trait space, ideally weighted by the abundances of the species [‘overdispersed’ has been used to mean evenly-spaced whereas it actually, for statistical reasons, means clustered (Greig-Smith 1983)]. There are several problems in doing this: (i) deciding what traits characterize the niche, (ii) constructing a null model, (iii) knowing what pattern limiting similarity will produce in functional-trait space and devising the appropriate test statistic for it, and (iv) avoiding confounding effects of environmental variation. These are elaborated upon below.

Traits for characterizing the niche

In explaining local coexistence, we must consider the alpha niche, i.e. the ways that plants differ in the resources that they use. Such resources are light (e.g. adaptation to use light from high or low angles or of particular spectra), water (e.g. from different depths in the soil) and nutrients (e.g. some species being more limited by N and others by P). Direct measurement of the plant resource acquisition via ‘hard’ traits (Díaz, Cabido & Casanoves 1998; McIntyre et al. 1999) such as photosynthetic response to water stress would be very difficult on a community-wide scale, but we need to select the best surrogates.

Null model

All measures of community structure must be compared to a null model, but framing this model is fraught with dangers of receiving a trivial answer to the wrong question (Wilson 1995a,b). Discussion in the literature is often of a variety of null models and of choosing the one that works ‘best’. In fact, the simple principle is to include in the null model all the features of the observed community except the one it is intended to test.

Patterns in functional-trait space and test statistics

Theoretically, limiting similarity could produce a range of patterns (Kraft & Ackerly 2010), such as large minimum distances in functional-trait space, large mean distances between nearest neighbours and low variance in the distance between species adjacent in value. The relatively few studies conducted so far give us little indication of which patterns are usual in nature.

There are many statistics available for comparing the distribution of observed species traits with those expected under the null model matching this range of patterns. Some are multi-trait (multivariate), others analyse one trait at a time (univariate). Some are based on differences between species means, others are based on niche overlap using within-species variation. We cannot use all the statistics in the literature, and all the models that could potentially be constructed due to the dangers of multiple testing. However, we cannot use just one test statistic because of our ignorance as to real-world limiting similarity patterns. A compromise is required. Care must also be taken because some measures used in the literature can give spurious results; for example, reporting a clustered distribution as an even one (Stubbs & Wilson 2004).

Environmental variation

The occurrence of a species with their particular traits is controlled not only by interspecific interactions but also by environmental filtering, i.e. according to their beta niche. This is a process well worth investigating, but it is not the issue of limiting similarity. If environmental filtering is occurring differentially across samples, it will obscure the signal of limiting similarity. More worryingly, environmental differences within the area being considered can give spurious signals of limiting similarity through environmental pseudoreplication. Suppose four environments, which have rather different species pools, are sampled, with 25 quadrats in each environment. If the 100 (4 × 25) quadrats are analysed together, any indication of assembly rules between the four environmental pools will dominate the analysis. An assembly rule may well be found, but based on four species pools with 25 near-replicate quadrats of each. This can be called pseudoreplication, because although real patterns may exist among a few species pools, using many quadrats from each inflates the effective degrees of freedom (Wilson 1999). Some previous work is in danger of falling into this trap, even deliberately sampling over an environmental gradient (Weiher, Clarke & Keddy 1998). There are several ways of avoiding such spurious results, or at least minimizing them, although at the expense of making the tests more conservative. Lake & Ostling (2009) suggested forming a null model for a quadrat based on a subset of the species restricted by trait. Armbruster (1986) and Kraft, Valencia & Ackerly (2008) used species subsets based on a priori geographical/environmental distributions and Cornwell & Ackerly (2009) subsets based distributions along an environmental gradient. Watkins & Wilson (1992) and Wilson & Gitay (1999) avoided a priori selection of environmental factors in data that were strongly spatially structured by using a patch model, i.e. basing the null model for a quadrat on a patch of physically adjacent quadrats. We advocate reducing the problem of environmental pseudoreplication by sampling in as uniform a community and environment as possible and using a patch null model.


In spite of recent efforts, there is still a considerable gap in community ecology between mathematical theories and demonstration of them in the real world. The present work aims to bridge this gap for limiting similarity. We took three adjacent saltmarsh communities, extending the work of Stubbs & Wilson (2004) on sand dunes and investigations such as those of Thompson et al. (2010) on roadside vegetation. We measured 23 traits, carefully chosen to represent the alpha niche as best possible, with a few TS that are supported by theory and by previous work. To minimize the problem of environmental heterogeneity with its attendant environmental filtering, we (i) chose an area of undisturbed coastal vegetation in New Zealand, (ii) sampled and analysed separately in three communities from salt meadow to upper marsh and (iii) in each case used a patch model. Significance of departure was determined by a randomisation test, assigning attributes to species at random by the method of Stubbs & Wilson (2004), as validated by Hardy (2008).

The hypothesis we test is that the species within these three communities are less likely to coexist if they share similar characteristics, i.e. that limiting similarity exists. We had no prior expectation of in which traits effects would be found, because all were chosen to represent the alpha niche as best we could. We would expect competition to be intense in all communities, so they represent tests of how widespread any effects are.

Materials and methods

Vegetation survey

Aramoana Saltmarsh lies near the mouth of Otago Harbour, New Zealand (45°47′S, 170°42′E; site AE1 on the map (Fig. 1) in Partridge & Wilson 1988a). At the harbour edge of the marsh, just above the bare sand, the substrate is predominately inorganic sand with low water content (mean 26.2% wet mass) and high but variable salinity (3.3 ± 1.3 S m−1). The vegetation here comprises a species-poor saltmarsh with only five species, with an average of 1.1 species per 10 × 10 cm quadrat. Assembly rules would be almost impossible to detect in this zone, and it was not analysed. Towards the top of the sampled area, the soil is acidic and organic with constantly high water content (mean 79.4% of wet weight) and a lower salinity (mean 2.4 S m−1). To the rear of the saltmarsh, above the sampled communities, are low, relict sand dunes formed c. 200 bc and now dominated by Phormium tenax (Agavaceae) and Lepidosperma australe (Cyperaceae) with exotic species such as Schedonorus arundinaceus (tall fescue) and Anthoxanthum odoratum (sweet vernal grass). (Nomenclature follows the ‘New Zealand Plants’ data base at http://www.nzflora.landcareresearch.co.nz, accessed 10 June 2011.) The vegetation everywhere is dense, indeed the Salt turf has been termed ‘salt meadow’ (Partridge & Wilson 1988b). For all the zones examined, species from lower zones are physiologically capable of growing in the salinities experienced there (Partridge & Wilson 1987a,b) and are capable of growing in the next higher zone if they are transplanted into it and competition from the surrounding vegetation is reduced (Partridge & Wilson 1988a). This demonstrates the existence of strong competition in the community. The three communities sampled (Table 1) were as follows:

Figure 1.

 The five test statistics (TS) used.

Table 1.   Species richness at four spatial scales (i.e. grains) in the three communities studied
 Per communityPer areaPer quadratPer point
Salt turf116.84.10.48

Salt turf: The community was dominated by Selliera radicans (Goodeniaceae), Sarcocornia quinqueflora (glasswort, Chenopodiaceae) and Schoenus concinnus (Cyperaceae) with scattered rushes of Apodasmia similis (jointed rush, Restionaceae) – 11 species in total (all native to the region). The cover was dense, 3–6 cm high, but with small gaps between leaves containing litter, a few small salt pans (c. 0.2 m across) and some crab burrows (2–3 cm across). The substrate is 20% organic, with the mineral fraction 90% sand (Partridge & Wilson 1989) and soil pH c. 5.0. The zone is reached by about 60% of high tides.

Rush community: There was continuous sub-canopy cover reaching to c. 10 cm, with species from the Salt turf such as Sarcocornia quinqueflora, some from the Shrub community such as Apium prostrata (‘native celery’, Apiaceae) plus other species characteristic of the Rush community such as Lachnagrostis littoralis (Poaceae). Emergent Apodasmia similis occured above these species, to 30–40 cm. All 14 species are native. At an elevation of 0.5 m above the lowest vegetation on the marsh, the community is just reached by an average spring tide. The soil comprises c. 60% organic matter, with sand at 80% of the mineral fraction, silt and clay comprising 10% each and pH c. 4.5.

Shrub community: Dominated by the saltmarsh specialist shrub Plagianthus divaricatus (Malvaceae), with Apodasmia similis and the Juncus-like Ficinia nodosa (Cyperaceae) also common – 14 species in total (11 native). The cover was continuous, with an upper stratum of P. divaricatus shrubs (c. 1.3 m tall) and A. similis clumps, the latter slightly taller and more dense. The soil is sandier and less organic than in the Rush community, with only the highest spring tides reaching the zone.

The communities had some overlap in species composition, as is inevitable in trying to maximize within-community homogeneity, but their data sets were independent.

Sampling for species co-occurrences

To allow analysis at a range of scales (spatial grains, hereafter ‘scale’) and to allow patch models to be used, the sampling was hierarchical: area, quadrat and point (cf. Fig. 2 of Stubbs & Wilson 2004). All samples were positioned within larger samples by stratified randomisation (Greig-Smith 1983). In the Salt turf community, there were as follows:

  •  six areas of 1 × 1 m,
  •  within each area, 20 0.1 × 0.1 m quadrats,
  •  within each quadrat, four points obtained using a fine sewing needle mounted on a randomly positioned pin.

Owing to the narrow width of the Rush community, the sampled areas were 10 × 0.1 m in order to obtain the same total area as the other community types. Individual plants were 1–3 m in diameter in the Shrub community, and point quadrats were not practicable, so the sampling areas were 10 × 4 m, with quadrats 0.5 × 0.5 m and ‘points’ 0.1 × 0.1 m. Because the shrub cover was patchy, 0.5 × 0.5 m quadrats containing < 50% cover of P. divaricatus were discarded and a new random quadrat was selected. Hierarchical sampling gives fewer but well-characterized samples at a large scale, and more but less well-characterized samples at a small scale, which will approximately balance. Points that contained too few species for the calculation of a test statistic were omitted from the analysis.

In all cases, all species with shoots within the quadrat or at the sample point were recorded. At the area and quadrat scales, the frequency in the next smaller unit gave a quantitative measure of abundance for each species. Seedlings (defined as individuals of shrub species that were shorter than 0.2 m or herbs that still possessed their cotyledons) were excluded from analyses. The niche occupied by a juvenile plant is often very different from that of the adult (Field & Dawson 1998).

Functional-traits survey

We selected traits that were suggested in the literature as representing the functional differences between species in resource acquisition: water, nutrients, light etc., but also to be practicable to collect. As not all the species possess photosynthetic leaf laminae, ‘leaf’ traits were measured on photosynthetic units (PSUs): for a species with leaves, PSU = the lamina of the leaflet or leaf; for species with only green stems, PSU = the stem segment that acts as a lamina. For ease of reading, ‘leaf’ is used below.

Traits (Table 2, see Appendix S1 in Supporting Information) were measured on ten randomly chosen plants of each species (not necessarily from the sampled quadrats), separately for each community in which a species was found as a species’ niche may differ between habitats (cf. Jung et al. 2010). Traits whose distributions departed significantly from normal were first transformed to improve normality (to avoid the undue influence of one tail, rather than because the significance calculated from the randomisation tests depended on it).

Table 2.   The characters measured for each species and their functional importance as seen in the scientific literature. For further detail, see Appendix S1
Plant characterFunction(s)
Shoot architecture
 Leaf number on the terminal shootPlant architecture; light capture
 Plant heightGrowth strategy; light capture
 Plant widthGrowth strategy; a key trait in C-S-R strategy; light capture
 Support fractionGrowth strategy; leaf longevity
Light capture
 Leaf inclination from the horizontalLight capture; transpiration; temperature
 Leaf thicknessLight capture; gas exchange; water retention
 Specific leaf area (SLA)Photosynthetic capacity; leaf longevity; stress tolerance
Air movement
 Leaf areaHeat load; water retention; gas exchange; light capture
 Leaf lobationHeat load; gas exchange; water retention
 Leaf shapeGas exchange; heat load
 Chlorophyll total content of the leavesLight capture; growth strategy; salt tolerance
 Chlorophyll a:b ratioGrowth strategy
 Leaf area ratio (LAR)Growth strategy
Nutrient and water storage
 Leaf succulenceSalt tolerance, support
 Nitrogen content of leavesPhotosynthetic capacity; leaf longevity
 Phosphorus content of leavesLeaf longevity; energy acquisition and utilisation
 Shoot/root mass ratioResource uptake
Nutrient and water acquisition
 Horizontal root system extentNutrient and water acquisition
 Number of primary root axesResource uptake; anchorage
 Rhizome diameterNutrient storage
 Root diameter of the thickest rootNutrient storage
 Root profileNutrient, salt and water uptake, avoidance of anaerosis
 Vertical root system extentNutrient and water acquisition

Null models

When forming a null model with which to test a hypothesis, it is essential to keep every feature of the randomized data fixed as it is in the observed data, except the feature that one aims to test. Otherwise, a departure from the null model may reflect an unrealistic aspect of the null model (Wilson 1995a,b 1999). Here, the frequencies of the species and their occurrence patterns were not part of the question, and so which species was in which quadrat and with what abundance was held at that observed (this also removes some of the danger that spatial autocorrelation might bias the results). The traits of a plant are part of an integrated individual (Díaz, Cabido & Casanoves 1999), and trait correlations were not part of the hypothesis tested here, so the group of trait values associated with a species was kept together, but assigned at random to species in the matrix. This breaks the link between species associations and traits link, which was the intention of the null model.

Assigning the traits to species with no regard to the frequency of the species would sometimes result in the values for an uncommon species being assigned to an extremely common species. Were these extreme trait values, they would be given too much weight in the randomized communities. The trait value might even be ill-adapted for the habitat or community, and this might be the reason that the species was uncommon, perhaps an accidental, present by the spatial mass effect. For example, if an uncommon species had particularly large leaves, this would affect the observed test statistic in only the few quadrats in which it was present. However, in the randomisations, these large leaves would often be assigned to a species that was common, affecting the test statistic in most of the quadrats. This would be a test of whether some character values were more common than others, which was not our question. Therefore, the traits were randomized within frequency classes as in Stubbs & Wilson (2004). Hardy (2008) also saw this problem and arrived at the same solution. The selection of frequency classes is difficult; too many classes, with few species in each, will give low power owing to the restriction upon the possible randomisations and using too few classes may be ineffective. Due to the low number of species in these communities (11–14), only two classes were used: the 50% of species least frequent across the site vs. the remaining (i.e. more frequent) species.

A patch model was used; randomisations at the quadrat scale occurred only within areas and at the point scale only within quadrats (Wilson 1999). Test statistics (see below) were calculated by comparing observed and randomized data, separately for each of the areas, quadrats or points, and these values were then averaged across the community. At the area and quadrat scales, both presence/absence and quantitative data were analysed. At the point scale, no quantitative information was available: either a species occurs at a point or it does not.

Test statistics

To reduce the problem of multiple significance tests, yet seek for a variety of patterns that limiting similarity might cause in niche space, we used five test statistics (TSs, Table 3), chosen partly on a priori grounds and partly on their ability to detect niche structure in an independent data set that of Stubbs & Wilson (2004). Experience in the latter study allowed us to reduce from 12 TSs there to the five below. The first was multivariate, the others univariate with a separate test for each trait. The first three are based on distances between the species means for each trait, the last two on niche overlap. Some can be weighted by the abundance of the species, others cannot. None are influenced by species richness, which in any case was the same in the observed and randomized data.

Table 3.   The test statistics used
StatisticBased on order along niche gradientAbundance-weighted or not?Value expected from limiting similarity*
  1. PCA, principal components analysis; TS, Test statistics.

  2. *Compared to the value expected under the null model.

Multivariate niche distance statistic
 TS1Min/max MST distance in Euclidean PCA spaceNoEitherHigh
Univariate niche distance statistics
 TS2Mean nearest neighbour distance/rangeYesNotHigh
 TS3Variance in adjacent neighbour distances/rangeYesEitherLow
Niche overlap statistics
 TS4Mean niche overlapNoEitherLow
 TS5Variance in adjacent neighbour niche overlapsYesNotLow

TS1 Min/max nearest neighbour MST distance in Euclidean PCA space [multi-trait]

In spite of the problem that some traits may be clustered and others evenly spaced, cancelling each other out (Cavender-Bares et al. 2009), we felt we should include one multi-trait test statistic to give an overall test. Using the original traits, the analysis is liable to be dominated by groups of highly correlated traits. Less correlated traits that may represent other, but important, aspects of niche differentiation might not be given due weight. A principal components analysis (PCA) was therefore performed on the correlation matrix of the traits (the traits first standardized to zero mean and unit variance) for each community. As few axes were retained as could account for 90% of the total variation, i.e. six for the Salt turf and seven for the other two communities. The distances between all pairs of species in an area or quadrat or point were then calculated in this PCA space, a minimum spanning tree (MST, cf. Nipperess & Beattie 2004) was formed and the minimum and maximum distances between adjacent species on that tree used as (Fig. 1):


Because the purpose of this analysis was to give one overall test, only quantitative data were used.

An even distribution in functional space, as would be expected from the operation of limiting similarity, would give a high value of TS1 (Table 3).

TS2 Mean nearest neighbour distance/range [uni-trait]

This statistic, as with TS3–TS5, was calculated separately for each trait (Fig. 1). Here, for each species, the shortest distance (i.e. absolute difference in trait value) to another species (its ‘nearest neighbour’) is found. Thus, two species could be each other’s nearest neighbours. Division by range standardises the trait value (cf. Kraft & Ackerly 2010). That is


A tendency for two similar species not to co-occur, the essence of limiting similarity, would give a high value. We refer to this as ‘even spacing’ for conformity with other TSs.

TS3 Variance in adjacent neighbour distances/range [uni-trait]

Using adjacent neighbours along the axis of trait values, each distance (number of species – 1) can be used only once (Fig. 1).


Even spacing would give a low value (Stubbs & Wilson 2004; cf. Kraft & Ackerly 2010).

TS4 Mean niche overlap [uni-trait]

Niche overlap was calculated using the formula of Pianka (1973) for categorical root data and that of Cody (1975) for other traits. The within-species variation in a trait was used as an estimate of niche width (as in Jung et al. 2010); if this is not realistic, the two distance statistics, TS2 and TS3, provide a fall-back. Even spacing, with the species’ niches well separated and with minimal overlap, would give a low value (Fig. 1). Random and clustered spacing would give a higher value, but would not be easily distinguished by TS4.

TS5 Variance in adjacent neighbour niche overlaps [uni-trait]

Even spacing would give a low variance in overlaps (Fig. 1; Stubbs & Wilson 2004). Clustered spacing would give a higher value, although possibly still lower than a random one.

Significance test

The value of a test statistic expected under the null model was calculated as the average value from 10 000 randomizations, and significance (i.e. the probability of the observed result under the null model) as the proportion of randomizations giving an equal or more extreme test statistic than that calculated from the observed data, multiplied by 2.0 to effect a 2-tailed test. The C++ program used was validated with random data (Stubbs & Wilson 2004). As multiple traits were tested and 0.05 of them would be expected to be Type I spurious significances, we were cautious when fewer than four traits were significant for a combination of TS and scale (four out of 23 represents a significant fraction by the binomial test). This should be taken as approximate, because the binomial test assumes the traits were independent and they are correlated.


Salt turf

The overall test here, with TS1, was not significant (Table 4). This is an unsurprising result for significant results in both directions was obtained in tests for individual traits in TS2–5: at the area scale 8 clustered, 6 evenly spaced, at the quadrat scale 7 clustered, 4 evenly spaced, at the point scale 4 clustered and none evenly spaced. For only three out of 12 TS/scale combinations were four or more traits significant (the number required by the binomial test to exclude the possibility of Type I errors owing to the multiple tests): the area scale with TS2 and TS3 and the quadrat scale with TS4. The only consistency among these three was that the species were significantly clustered for N content whenever significant. However, we note that overall in this community, the species were significantly clustered in total chlorophyll or chlorophyll a:b in six tests, although significantly spaced in one. Overall, there was only very weak evidence for the operation of limiting similarity in the Salt turf and slightly more evidence for clustering.

Table 4.   Significant departures from the null model (H0) for Euclidean PCA space and for individual traits. ‘Clustered’: deviating from H0 in the direction expected from environmental filtering; ‘Evenly spaced’: deviating from H0 in the direction expected from limiting similarity
Test statistic↓Spatial scale↓Salt turfRush communityShrub community
ClusteredEvenly spacedClusteredEvenly spacedClusteredEvenly spaced
  1. p, significant only with presence/absence data; q, significant only with quantitative data; Chlor, chlorophyll; diam, diameter; Horiz, horizontal; LAR, leaf area ratio; Root depth, Root depth profile; Support frac, Support fraction; Vert, vertical; PCA, principal components analysis; SLA, specific leaf area; TS, Test statistics.

  2. *Significant also with the original characters on which the PCA was based.

Multivariate niche distance statistic:
Min/Max link in MST Euclidean PCA space
Areas     P = 0.0010*
Quadrats   [P = 0.0867]  
Points   [P = 0.0674]  
Univariate niche distance statistics:
Mean nearest neighbour distance/range
AreasLeaf lobation
N content
Root depth
Horiz root extent
Root diam
 Chlor a:bLeaf area Primary rootsLeaf lobation
Vert root extent
QuadratsChlor total
Chlor a:b
Vert root extent
Leaf thickness
P content
Horiz root extent
Rhizome diam
Root depth
 Leaf lobation
PointsRoot diam Support frac
Vert root extent
Leaf thickness
Chlor a:b
P content
Horiz root extent
Rhizome diam
Root depth
 Leaf lobation
Variance in adjacent neighbour distance/range
AreasChlor a:b
N content p
Leaf number pSucculence qLeaf inclination p
Chlor a:b p
P content p
Root depth q
Plant width qLeaf lobation q
Root depth p
QuadratsChlor total
Root depth
  Leaf shape
P content
Support fracLeaf lobation
PointsP content  Leaf shapeSupport frac
Root depth
Leaf lobation
Niche overlap statistics:
Mean niche overlap
AreasRoot depth pLAR pShoot/Root pLeaf inclination p Leaf thickness p
Succulence q
Chlor total q
Chlor a:b
N content p
Leaf lobation p
Primary roots p
QuadratsN contentChlor total
Succulence p
Vert root extent
P content
PointsLeaf number LARSucculence  
Variance in adjacent neighbour niche overlap
AreasChlor totalLeaf number
Root depth
  Leaf area pChlor a:b
QuadratsSupport frac
Chlor total
S/RLeaf area
Root diam
SucculenceLeaf numberLeaf lobation
PointsRoot diam Root diamShoot/root
Rhizome diam
 Leaf lobation

Rush community

In this community, the overall test with TS1 gave ‘marginal significant’ spacing at the quadrat and point scales. This was in spite of the clustering of species seen in 13 of the TS/scale/trait combinations. We can therefore suspect, but not prove, that limiting similarity was operating.

Individual tests showed limiting similarity (i.e. even spacing) to be occurring in traits associated with light capture – leaf thickness (3 tests) and leaf inclination (2 tests) – and in chlorophyll a:b ratio (3 tests), although not in specific leaf area (SLA) or leaf area ratio that we envisaged would be related to photosynthesis. The air movement trait leaf shape was significantly evenly spaced twice, although in cells without a significant number of traits showing an effect. The nutrient storage trait P content showed significantly even distance spacing (TS2 and TS3) in four cases, although not in terms of overlap (TS4 and TS5). The water- and salt-related trait of succulence showed reduced niche overlap (TS4 and TS5) compared to the null model, i.e. limiting similarity, in four cases (but only two of them in cells with four significances), although not in the distance statistics.

In below-ground traits, vertical root extent was significantly clustered at smaller scales using TS2 and root diameter with TS5. However, horizontal root extent, rhizome diameter and root depth profile (i.e. distribution) were evenly spaced in some analyses.

Shrub community

Within this community, there was overall support for the theory of limiting similarity of coexisting species at the area scale (TS1, Table 4). There was a little evidence for a below-ground limiting similarity at that scale: only vertical root extent with TS2 and the number of primary roots with TS4. There were relatively consistent limiting similarity effects in leaf lobation (significant in 9 of 12 tests) and succulence (6 of the 8 tests at quadrat and point scales).


Traits and the niche

The alpha niche, representing resource use, and the beta niche, representing the external environment, are conceptually quite distinct. For example, utilisation of different sources of nitrogen and different positions in the canopy are clearly alpha-niche traits, potentially enabling species to evade Gaussian competitive exclusion and coexist (Harrison, Bol & Bardgett 2007; Lorentzen et al. 2008). Rooting depth represents differences in resource use that permit coexistence (O’Brien, Moorby & Whittington 1967; Whittington & O’Brien 1968; Dornbush & Wilsey 2010). Some traits such as frost and drought tolerance are clearly beta niche, found in species occurring in different areas and enabling them to occur there. However, some traits may have both roles. For example, deep rooting can be a beta-niche adaptation to arid environments, but also alpha-niche differentiation enabling water and nutrient uptake from a different soil horizon than other species. For our analysis, we needed traits that represented the alpha niche and we made our best efforts to obtain them.

The traits of a species may differ between environments, genetically (e.g. ecotypically), plastically or ontogenetically (be these differences adaptive or not). Such differences between our three communities (i.e. zones) do not enter into our results, because for species occurring in more than one community separate trait measurements were made. Within a community, ecotypic differentiation seems unlikely, genetic variation is possible, but plastic and ontogenetic responses are likely. These differences do not bias our analyses of coexistence because we are using one mean and distribution for each species within each community, and they are implicitly taken into account in the niche overlap TSs. However, limiting similarity that results in plastic divergence in species traits when they co-occur will not be detected by our analyses, and plastic convergence will add noise and make our tests more conservative. Moreover, niche differentiation is only one of 12 possible mechanisms of species coexistence (Wilson 2011), and the operation of the other 11 will blur the co-occurrence patterns.

Environmental filtering and limiting similarity

Although our aim was to seek evidence of limiting similarity, in some cases, the opposite pattern was seen, i.e. the clustering of species with similar functional traits. Grime (1979) proposed that different environments would select for different traits, and many have documented this, even on a fine grain (e.g. Silva & Batalha 2009). Our finding of environmental filtering is to be expected because, in spite of our attempts to select traits that reflect the alpha niche, of our selecting environmentally similar areas and of our using a patch model, microenvironmental filtering is liable to intrude.

There was support for limiting similarity theory: species co-occurring locally should be less similar to one another than would be expected if species were drawn randomly from the community (evenly spaced in Table 3). The only significant result in the multi-trait test (TS1) was obtained at the area scale in the Shrub community. This was perhaps not surprising because such work is at an early stage in community ecology, so we do not know which traits are important in limiting species co-occurrence, let alone which are important in which types of community or which TS to use to find the resulting pattern.

We sampled at three spatial scales in the expectation that different coexistence patterns would be seen. Niche limitation should occur mainly at smaller scales (Watkins & Wilson 1992). This was clearly seen for succulence in the Shrub community – significant in six tests out of eight at the quadrat and point scales but in none at the area scale – and for root traits in the Rush community (see below).

Root trait differentiation

Evidence for root niche separation was seen in the Rush community, especially at the two smaller scales using TS2: horizontal root extent, rhizome diameter and root depth profile. For example, limiting similarity in root depth profile implies that shallow-rooted species such as Triglochin striata (arrowgrass, Juncaginaceae) would be associated with deep-rooted ones such as Ficinia nodosa.

In the Salt turf community, root depth profile was clustered in most analyses, but rooting depth has different implications in the Salt turf, where the lower soil layers are anaerobic, perhaps excluding deeper-rooted species in some patches.

Leaf traits

The general trend for light capture and photosynthesis traits in the Salt turf is for clustering at the two larger scales (Table 3). Thus, total chlorophyll was clustered in four analyses using TS2, TS3 and TS5, the species with high chlorophyll content in this community being the two grasses and Schoenus concinnus. However, there are two cases where a trait seems to cluster using mean values (TS3), but the species assort when within-community variation is taken into account (TS4). This reinforces Jung et al.’s (2010) point that it is important to take the local variation in traits into account, to establish niche width.

In the Rush community, there was evidence for limiting similarity (i.e. even spacing) in several leaf-related traits: leaf inclination, leaf thickness, leaf shape, chlorophyll a:b and succulence, with only one case of clustering. Leaf inclination separates the near-vertical PSUs (green stems) of the jointed rush Apodasmia similis and Schoenus concinnus (Cyperaceae) from the near-horizontal laminae of Apium prostratum and Samolus repens. Leaf thickness extremes were the thick PSUs (succulent leaf bases) of Sarcocornia quinqueflora vs. the thin laminae of the grass Lachnagrostis littoralis. Although placed in given different categories in Table 2, these traits contribute to several leaf functions and suggest that canopy interactions limit the coexistence of species in the Rush community.

Canopy interactions are also implicated in the Shrub community, with leaf lobation and succulence (fresh mass/dry mass quotient) evenly spaced whenever significant. This was relatively consistent for leaf lobation at all scales and also for succulence at the quadrat and point scales. Species with strong lobation include the trifoliate and toothed Apium prostratum and also the hastate-leaved Atriplex prostrata; those with low lobation include the green stems of Apodasmia similis and the Juncus-like Ficinia nodosa, as well as in the less frequent, parallel-leaved Triglochin striatum. The analysis suggests that there is a tendency for one of the Apium prostratumAtriplex prostrata group to occur with one of the Apodasmia similisF. nodosaT. striatum type. The species with low ‘leaf’ succulence are, as expected, the green-stemmed Apodasmia similis, the tussock grass Poa cita and other grasses Agrostis stolonifera, Elymus repens and Lachnagrostis littoralis. Succulent species comprised, again as expected, the obligate halophytes Sarcocornia quinqueflora and Suaeda novae-zelandiae and the facultative saltmarsh halophytes Selliera radicans, Apium prostratum and Atriplex prostrata (Partridge & Wilson 1987b). Overall, the evidence is that limiting similarity and hence species coexistence are related to competition for light in the Rush and Shrub communities.

Shoot architecture and nutrient content

There was clustering in these traits. Shoot/root ratio, expected to be a mechanism of niche differentiation and thus give evidence for limiting similarity, was instead clustered at the two larger scales in Rush community in terms of mean niche overlap (TS4) although evenly spaced at the point scale (TS5), again emphasising that assembly rules tend to operate at small scales. Nitrogen content, whenever significant, was clustered. We note that this cannot represent greater N uptake by species when they grow in a high-N patch, because species traits were held constant across each community. Because these two clustering effects, in shoot/root ratio and N content, occur at the two larger scales, there may be an environmental mosaic in the community, making quadrats different from each other; for example, N-poor patches being occupied by species with low shoot/root ratio and with tolerance of low N vs. patches of high-N requiring species such as Triglochin striata (Juncaginaceae) and Apium prostrata (native celery). Further clustering was seen in support fraction (clustered each time it was significant and that four times scattered over the three communities), indicating that there was a tendency for patches to comprise either low-support species such as Schedonorus arundinaceus (tall fescue) or high-support species such as the shrub Plagianthus divaricatus.

In contrast to N content, which was clustered across all communities whenever significant, P content was evenly spaced in the Rush community by the two distance statistics (TS2 and TS3) in four cases out of six.

Problems and methods in investigating limiting similarity

Several recent studies have used, as a substitute for traits, phylogenetic relatedness, which may represent traits that are not easily measured directly, or that may not be obvious. The problems are that (i) this is effective only if the alpha niche is well conserved in evolution and (ii) the phylogeny is only a hypothesis, and often alternative phylogenies are almost as parsimonious and as likely to be true. Phylogeny shares with multivariate analysis of traits the problems that (i) environmental filtering traits and those that limit coexistence may cancel each other (Swenson & Enquist 2009) and (ii) if there is a significant pattern, we do not know from the analysis which trait is causing it. In summary, either the phylogeny is closely related to the traits limiting coexistence, in which case it would be preferable to use the traits, or it is not, in which case analyses based on the phylogeny will be ecologically uninformative. However, phylogeny can now be extracted from data bases more easily than traits can be measured, and because there is correlation with traits, it can sometimes demonstrate limiting similarity. This has been reported within a taxonomic guild, e.g. oaks, hollies and pines separately (Cavender-Bares, Keen & Miles 2006) and within a clade in the Cyperaceae (Slingsby & Verboom 2006). Most phylogenetic studies on a broader taxonomic scale have failed to show assembly rules, e.g. Kraft & Ackerly (2010) with ‘free-standing woody species’ and Gonzalez et al. (2010) with tree species, although Kembel & Hubbell (2006) found even phylogenetic spacing in one of their four tropical rainforest habitats on Barro Colorado Island.

Many different TS have been used for traits, and there is little consensus on this. Studies that use only one test statistic avoid some problems of multiple testing, but may fail to discover patterns of species co-occurrence. Again, a compromise is needed; using too many statistics represent multiple testing and using too few possibly represents not seeking assembly rules thoroughly enough.

Two types of null model have been used for limiting similarity studies. Sometimes the quadrat/species occurrence matrix has been randomized, using the marginal totals fixed or as probabilities (e.g. Kembel & Hubbell 2006; Swenson & Enquist 2009; Kraft & Ackerly 2010; Thompson et al. 2010). The alternative is to retain the observed quadrat/species matrix and randomly assign traits to the species (e.g. Stubbs & Wilson 2004; Cavender-Bares et al. 2004; Schamp, Chau & Aarssen 2008; Ingram & Shurin 2009; this study). Because the aim is to examine how the traits relate to the co-occurrence pattern, not to examine the co-occurrence pattern itself, the latter procedure correctly follows the principle of constraining the null model to the observed data except in the feature it is intended to test. Hardy (2008) confirmed, using simulated data sets, that this is the preferable procedure. It does carry the problem of breaking the link between species’ traits and frequency. Hardy recommended that the traits be randomized only within frequency categories, as in Stubbs & Wilson (2004) and the present study. The number of groups has to be a compromise, but we believe the approach used here represents current best practice. As in much assembly-rule work, there is a danger that what one holds constant, e.g. the species frequencies, reflect the effects of interspecific interference. However, this seems far preferable to the Jack Horner effect (Wilson 1995b), demonstrating the obvious rather than the aspect of community structure it was intended to test.

Previous support for trait-based limiting similarity

Many attempts with animal communities to find the evidence of even spacing expected from limiting similarity theory have failed, finding only clustering of traits, presumably caused by environmental filtering (e.g. Mouillot et al. 2005 with parasites of fish in the Czech Republic, Mouillot, Dumay & Tomasini 2007 with lagoon fish in France, Mason et al. 2007 with lake fish in France). Some studies with plants have found no significant deviation from the null model in either direction: clustering or even spacing (e.g. Thompson et al. 2010 with a British roadside; Schamp, Chau & Aarssen 2008 in a 31-year Canadian old-field).

Even spacing has been found in some animal studies, e.g. Juliano & Lawton (1990) with water beetles in their larger, more stable sites; Nipperess & Beattie (2004) with ants in arid shrubland in Australia, although only at the smallest scale and where significance disappeared after a Bonferroni correction; Rabosky et al. (2007) with the body size of lizards in NW Australia and Ingram & Shurin (2009) with rockfish from north-west Pacific literature.

A limited number of studies have reported limiting similarity in plant communities. Weiher, Clarke & Keddy (1998) measured 11 morphological plant traits across a number of wetland habitats and claimed evidence for limiting similarity. However, they used a null model that combined the species from several species pools into one. The departures from null model that they found are therefore likely to be due to environmental pseudoreplication (Wilson 1999). Moreover, several of the TS used in Weiher, Clarke & Keddy’s (1998) study are affected by changing range size in the randomized communities (Stubbs & Wilson 2004) or do not measure the aspect of the species distribution that Weiher, Clarke & Keddy (1998) believed them to. It was a brave early attempt.

Work using the methods now accepted started with Armbruster, Edwards & Debevec’s (1994) study of reproductive organs in Stylidium in Western Australia, finding marginally significant even spacing using a patch model. Then, Stubbs & Wilson (2004), examining a New Zealand sand dune, found that departure from their null model was scattered, but almost always in the direction of even spacing. Cavender-Bares et al. (2004) examining co-occurrence of oaks in three Florida reserves, lumped into one null model, found significantly even spacing for four traits out of 20: acorn maturation time, embolism owing to freeing, leaf life span and vessel diameter. Although it is possible to make an argument that acorn maturation and leaf life span represent alpha niche differentiation via resource use in different seasons, or years with different conditions, that is difficult for leaf life span, it seems more like beta-niche differentiation. A patch model would have been useful to exclude possible environmental pseudoreplication.

More recently, Kraft & Ackerly (2010) found significant even spacing the 5 × 5 m scale for six out of seven traits in Amazon tropical rainforest. Strong evidence was seen in tree height and girth, suggesting the assembly restriction may be related to vertical stratification, comparable to the stratum guild proportionality of Wilson (1989). The same explanation may apply to the tropical dry forest in Costa Rica studied by Swenson & Enquist (2009), where there was overall clustering as seen in a PCA on six traits, but even spacing in seed mass and maximum height.


Several workers have failed to find evidence of limiting similarity and doubted whether it exists. Part of our approach was to choose areas as uniform as possible within each zone. This limits the generality of the results (Fisher 1966). However, the analysis of three communities here, together with the sand dune analysis of Stubbs & Wilson (2004), adds to the picture that is building up that limiting similarity structures some communities. We suspect it has a role in all, but that it is necessary to use powerful methods of analysis to demonstrate it above the signal of environmental filtering and of disturbance. A patch model of some kind must be used to reduce still further the danger of habitat pseudoreplication (e.g. Armbruster, Edwards & Debevec 1994; Stubbs & Wilson 2004). The null model should assign suites of traits to the observed quadrat/species occurrence matrix, within frequency classes (Hardy 2008). To demonstrate an assembly rule, the functional alpha niche must be characterized as best possible, with traits that are related to resource use at one point. Beta-niche traits that adapt to particular environmental conditions should be avoided, or similar species will be seen to occur together due to environmental filtering. Leibold (1998) made this point analytically and Mayfield & Levine (2010) with the example of adaptation to soils of different texture. Scheffer & van Nes (2006) suggested that species similar in resource use could cluster because they would ‘facilitate each other indirectly by suppressing a common competitor’, although it is hard to see how this would apply to plants. However, even although alpha- and beta-niche concepts are absolutely distinct – causing mutual exclusion vs. causing mutual co-occurrence – assigning traits to one or the other is difficult, because one trait can have many roles. It is somewhat frustrating that the wrong sampling, or traits, or TS, or null model can lead to a failure to demonstrate an assembly rule that exists or, more worryingly, provide apparent evidence for one that does not exist. Over these criteria, we believe no previous study has used these approaches as thoroughly, all together.

A saltmarsh, even in the zones least affected by salt, is a stressed habitat, as confirmed by the vegetation being herbaceous except for one specialized saltmarsh shrub. The prediction of C-S-R theory that competition is less intense in stressed habitats (Grime 1979) would suggest that assembly rules would be less in evidence. In contrast, Ruprecht et al. (2007) hypothesize, from their comparison of two sites, stronger constraints on species coexistence in stressed habitats. Here, the most stressed community, the Salt turf, gave little evidence for assembly rules. There was evidence for some traits in the Rush community, although the existence of clustering due to environmental filtering in other traits reduced the overall test to ‘marginal significance’. Overall, support for the theory of limiting similarity was found in only the Shrub community, the least stressed. Stubbs & Wilson (2004), in a similar study on a sand dune, concluded that limiting similarity there was owing to water and perhaps nutrient acquisition and use. Water is not likely to be limiting in these saltmarsh communities, but the results suggest that with the closed canopies of the Rush and Shrub communities the restrictions on species coexistence, i.e. the assembly rules, are based on canopy interactions, perhaps competition for light. In the Shrub community, root interactions may be involved as well.


We thank Dr Greg Collings and members of our research group for constructive comments on the manuscript. The late Professor Bannister provided advice during the project.