Assessment of temporal instability in the applied ecology and conservation evidence base

Outcomes of meta‐analyses are increasingly used to inform evidence‐based decision making in various research fields. However, a number of recent studies have reported rapid temporal changes in magnitude and significance of the reported effects which could make policy‐relevant recommendations from meta‐analyses to quickly go out of date. We assessed the extent and patterns of temporal trends in magnitude and statistical significance of the cumulative effects in meta‐analyses in applied ecology and conservation published between 2004 and 2018. Of the 121 meta‐analyses analysed, 93% showed a temporal trend in cumulative effect magnitude or significance with 27% of the datasets exhibiting temporal trends in both. The most common trend was the early study effect when at least one of the first 5 years effect size estimates exhibited more than 50% magnitude difference to the subsequent estimate. The observed temporal trends persisted in majority of datasets once moderators were accounted for. Only 5 datasets showed significant changes in sample size over time which could potentially explain the observed temporal change in the cumulative effects. Year of publication of meta‐analysis had no significant effect on presence of temporal trends in cumulative effects. Our results show that temporal changes in magnitude and statistical significance in applied ecology are widespread and represent a serious potential threat to use of meta‐analyses for decision‐making in conservation and environmental management. We recommend use of cumulative meta‐analyses and call for more studies exploring the causes of the temporal effects.


Highlights
What is already known • Temporal trends in effect magnitude and significance have been widely observed especially in medical fields.
• There is a disagreement about the prevalence of such temporal trends in ecology, and their potential impact on decision making in conservation and environmental management.

What is new
• Temporal trends in magnitude and statistical significance of cumulative effect sizes are explored using a large sample of meta-analyses published in applied ecology and conservation.• New methodology is applied to reduce risk of Type 1 error caused by inflated between study variance in cumulative meta-analysis, and to assess the effects of moderators more accurately.• 93% of examined datasets showed some form of temporal trend in cumulative effect with 27% of the datasets exhibiting temporal trends in both magnitude and significance of the cumulative effect.
Potential impact for Research Synthesis Methods readers • Methodology used in our study to test for effects of moderators on temporal trends in effect sizes can be applied in other fields.• Our study adds to the growing evidence of temporal trends in effect sizes across scientific disciplines which may be symptomatic of a reproducibility crisis.• Our findings indicate that temporal instability of evidence base can have significant implications for policy making and management since recommendations are often based on results of meta-analyses.

| INTRODUCTION
Evidence-based ecology and conservation are increasingly relying on results of meta-analyses of primary studies assessing the effectiveness of different management interventions. 1 In order for the recommendations derived from a meta-analysis to remain valid, the results of these analyses need to remain temporally stable, but a number of ecological meta-analyses have revealed changes in magnitude and statistical significance of the effects over time. 2 Recently, Koricheva and Kulinskaya (2019) argued that if widespread, such temporal trends represent a threat to use of meta-analyses in policy making in environmental management and conservation. 35][6] These may include reporting of extreme effects in early studies ('early study effect') or decrease in magnitude and/or loss of significance of the effect over time ('decline effect'). 5[9] The first review of temporal trends in ecology 7 examined 44 ecological and evolutionary meta-analyses using two methods.Firstly, Spearman's correlations were calculated for each meta-analysis between (i) year of publication of primary study and effect size, (ii) effect size and sample size, and (iii) year of publication of primary study and adjusted effect size (standardised for variance).The relationship between effect size and year of publication of primary study was also weighted to account for changes in sampling effort.Across all methods of analysis, the relationship between year of publication and both sample size and effect size were found to be significantly negative, indicating the potential presence of a declining temporal trend.The effect size relationship remained significant in the weighted analysis even once changes in sampling effort had been controlled for.Overall, this suggests the presence of a decline effect in the examined evidence, independent of any changes in sampling effort.
The second review 8 examined effect sizes across 52 ecological meta-analyses for various biases including temporal, by calculating correlation coefficients between effect size, year of publication, and bias-related variables such as journal impact factor and citation rate.This was then followed with a standard mixed effects metaanalysis.Ultimately, this review found evidence for many biases in ecology including temporal and concluded that earlier papers on a topic tended to have more extreme effect size estimates.
The most recent review 9 examined a larger sample of 466 ecology meta-analyses using both Pearson's correlation and permutation testing to analyse the relationship between absolute effect size and year of publication.Costello and Fox found that only around 5% of these metaanalyses showed any directional change in effect size and concluded that temporal trends in ecology are rare.The lack of broad consensus about the frequency and magnitude of temporal trends in ecology is particularly concerning not only because some see it as a sign of a broader reproducibility crisis of in science, 3,10,11 but also because ecology is considered especially prone to problems with the temporal stability of evidence due to small sample sizes, fairly small effect sizes, and meta-analyses often including a large number of contradictory primary papers. 3Additionally, the implications of out-of-date ecological meta-analyses are potentially very significant due to their immediate relevance to conservation and policy recommendations. 3revious attempts to study temporal trends in ecology have shared the same limitations in that they have relied on methods that are only able to detect linear, uniform trends.This does not represent the full range of temporal trends observed in other disciplines which may include gross overestimation of the effect in the initial studies on the topic, 11 reversal of effect sign over a period of time, 5 and the so-called Proteus phenomenon, where early replications of work frequently contradict (often in a very extreme manner) the initial reported findings, rather than lending support to the previously proposed theory. 12Temporal trends may also appear, disappear and change multiple times over the time span covered by a meta-analysis. 5Therefore, techniques such as cumulative metaanalysis (CMA), 13,14 which can reveal non-linear temporal trends may be more suitable for analysis of temporal trends in ecology.While CMA may be prone to inflated between-study variance, a new two-stage approach to CMA has been proposed that minimises these problems. 13otential causes of temporal trends in effect sizes are many, varied, and similarly poorly understood.[12][17][18][19] Other forms of publication bias may also cause temporal instability, for example 'paradigm-shifting' towards or away from an idea 20 and potential desire of journals to publish, 8 or to delay publishing 21 contradictory results.
Changes in sample sizes over time may also result in temporal trends in effect sizes, since studies with small sample sizes are more likely than larger ones to produce more extreme effect size estimates. 22,23These extreme effect sizes can occur by chance or because of biases, which may be more likely in smaller studies because they are often of lower methodological quality both experimentally and in terms of statistical power. 22,23nother potential cause of temporal trends in ecological effect sizes is changes in study characteristics (known as moderators in meta-analysis) such as geographic location or study species over time.If the proportion of studies exhibiting certain features changes over time and if these moderators have a significant impact on effect size, this can then cause a change in effect size or significance. 8,11,18,24Finally, some temporal changes may reflect genuine biological changes over time, for example in cases where climate change or resistance to biological agents like herbicide are reflected in effect estimates. 2,3ethods for unravelling the potential causes of temporal trends are complex and have not previously been attempted on a broad scale in ecology.
Previous reviews on temporal trends in effect sizes in ecology focussed on exploring the frequency of temporal trends in effect sizes in individual meta-analyses or in overall effects across a number of meta-analyses.Instead, the aim of our study is to assess how likely the results of a given meta-analysis in ecology are to change depending on the year when this meta-analysis is conducted.To address this aim, one needs to explore temporal trends in cumulative effects of individual meta-analyses.Therefore, we used cumulative meta-analysis to analyse and classify patterns of temporal trends in magnitude and statistical significance of cumulative effects in ecological metaanalyses in order to assess the extent of temporal changes in meta-analysis results.Additionally, we explored the potential causes of the observed trends such as changes in moderators, sample size or artefacts of traditional cumulative meta-analytic methodology.

| Data collection
A search of meta-analyses published in the areas of applied ecology and conservation was conducted on 5th January 2019 using the Web of Science Core Collection.First, we searched for all the meta-analyses published in the main applied ecology and conservation biology journals (Journal of Applied Ecology, Ecological Applications, Biological Conservation and Conservation Biology) by using the keyword 'meta-analysis' and the list of the above journals in the 'publication name' field.This resulted in 214 papers for screening.Then, a further 740 results were found by using the keywords 'management' and 'meta-analysis' in combination with search filters for specific Web of Science 'Categories' (e.g.Plant Sciences, Environmental Studies-for full list see Appendix 1).Finally, a further 872 results were found by searching for 'meta-analysis' and 'conservation' across all categories and publications (excluding those publications where results had already been extracted).This yielded 1826 abstracts for screening in total.
The inclusion criteria for the screening process (Figure 1) were as follows: 1.The publication contained at least one meta-analysis related to applied ecology or conservation.2. The meta-analysis included data from at least 10 separate primary studies published over at least a 10-year period.3. The publication provided either effect size and variance data from primary studies, or the raw data (sample sizes and means) for their calculation.
4. The effect size used in the meta-analysis was a standard effect size measure (such as log response ratio, Hedges' d or g, or correlation coefficient).
If a study reported several separate meta-analyses conducted on different response variables, we included them into analysis if they satisfied the above inclusion criteria.Authors of papers that did not provide either effect size and variance data or raw data for their calculation were contacted to request additional data.Of the 104 authors contacted, 9 responded with the requested data, the remaining studies were excluded.The final database consisted of 79 papers published between 2004 and 2018 which included 121 separate meta-analyses.The full list of the papers and their associated metaanalyses is available in Appendices 2 and 3.

| Data extraction
From each meta-analysis we extracted information on the number of primary studies included, publication years spanned by the primary studies, the effect size metric used, number of individual effect sizes across all showing the screening process.
primary studies and any moderator variables included in the analyses.It was also recorded whether the primary studies included in each individual meta-analysis overlapped with other meta-analyses from the same publication.
The majority (50 of 79) of studies contained only 1 meta-analysis suitable for inclusion, 17 studies contained 2 suitable meta-analyses, 10 studies contained 3 suitable analyses and 1 study yielded 8 suitable metaanalyses.Of the 35 papers that contributed more than one analysis, all but two used some or all of the same primary studies in all the analyses.That said, many of these primary studies provided completely different combinations of predictor and response variables for each metaanalysis, and may even have included data collected at different sites by different observers.A sensitivity analysis was conducted to check the impact of inclusion of more than one meta-analysis per study by repeating the analysis with only one meta-analysis with the largest (or random equally large) sample size included.
In some cases, not all of the required information was provided by the authors in their full-texts or supplementary data files, but the raw data was available to calculate the necessary values.The calculations used are outlined below.All analysis, including calculation of these values, was completed in R version 4.0.2(R Core Team, 2020).
Where a Hedges' g effect size and sample size information were available, variances were estimated according to the following equation 25 : where d denotes the effect size, and n₁ and n₂ are the sample sizes of groups 1 and 2 respectively.Where Hedges' g effect sizes and variances were available, but sample sizes were missing, these were calculated according to the following formula (note: here, n₁ and n₂ are assumed to be identical, and thus sample size for each group is simply N/2 25 : Where effect sizes were not included in the data file, or were calculated as log response ratios (but sample sizes, variances and means for the two groups were available), Hedges' g effect sizes were calculated using the escalc function from the metafor package (version: 3.8-1). 26This was done due to the inherent bias present in log response ratios, 27 and resulted in all but 13 logresponse ratio effect sizes and 2 Fisher's Z effect sizes being recalculated.

| Analysis and classification of temporal trends
Most meta-analyses included several effect size estimates within a study, and many studies published in the same year.Therefore, in order to examine temporal trends, effect sizes needed to be aggregated first by study, and then by year for each meta-analysis.This was done using nested random factors (individual study nested within publication year), and the multilevel approach included in the rma.mv () function included in the metafor package. 26This multilevel approach is described in detail in several publications 28,29 and in the metafor package documentation. 30Cumulative meta-analyses on effect size by year were then conducted using a new CMA method developed by Kulinskaya and Mah 13 and adapted from the rma.mv and cumul functions of the metafor package. 26This method is designed to reduce the type 1 error that can be inflated in CMA due to repeated testing and consecutive updating of between-study variance τ 2 and overall effect b θ.It involves adapting CMA into a two stage process: obtaining estimates of τ 2 from the first 5-10 studies, and then this estimate of τ 2 is kept fixed and a target value of the effect is monitored.Additionally, the 'method' argument specified in the CMA function in metafor was changed from Restricted Maximum Likelihood to Paule-Mandel since the latter requires less assumptions to be met. 31Results were then plotted on forest plots to allow for visual interpretation of temporal trends.We classified magnitude trends based on shifts in actual magnitudes of effects rather than in Cohen's benchmarks for small/medium/large effects, as these were not designed to be used in ecological studies 2,32 and were also not designed to be used when actual effect sizes and their percentiles are known. 33emporal trends in cumulative effect magnitude and significance were categorised based on classification adapted from Fanshawe et al. 5 and Trikalinos et al. 34 which was modified to encompass the wider variety of trends present in this dataset.
Temporal trends in magnitude of the cumulative effect were classified as following (Figure 2): 1. Flat: no effect size is more than 50% higher or lower than subsequent year estimates, and there is no overall change of more than 50% across the entire year span.2. Early study: one or more of the year estimates within the first 5 estimates is more than 50% higher or lower than subsequent estimates.This may be followed by another trend.3. Increasing: effect magnitude is continuously moving away from the zero line and the final effect estimate is more than 1.5Â the original estimate/the estimate where 'early study' criteria are no longer met.4. Crossing: The cumulative estimate crosses the zero line once or on multiple occasions, and this appears to be a continuous trend rather than an outlying year estimate or small fluctuations close to the zero line. 5. Diminishing: effect magnitude is continuously moving towards the zero line and the final effect estimate is less than 0.5Â the original estimate/the estimate where 'early study' criteria are no longer met.6. Outlying years: no visibly apparent continuous trend is present, but one or more consecutive or nonconsecutive year estimates differ by more than 50% from subsequent or previous ('outlying years') Temporal trends in significance of cumulative effect were classified as following (Figure 3): 1. Never reached: no statistically significant effect was observed at any point.2. Always significant: an effect estimate was statistically significant from the beginning and remained so throughout.3. Retained: a statistically significant estimate was reached in the first 5 estimates and remained significant up to and including the last meta-analysis.
4. Only at end: A statistically significant estimate was only observed in the final (or final 1-2) meta-analysis. 5. Lost, not regained: a statistically significant estimate became no longer statistically significant at some stage, and significance is not regained.6. Lost, regained: The final estimate was statistically significant and at some earlier stage of the meta-analysis, but at least one non-statistically significant estimate occurred in between these points.
Spearman's correlations between the mean effect sizes aggregated by publication year were also calculated in order to test for linear temporal trends.Additionally, all forest plots were plotted with 99% confidence intervals in order to counteract problems with inflated between-study variance in cumulative meta-analysis. 13

| Effects of moderators
To check for the potential impact of moderators on the temporal trends in effect magnitude, meta-regression analysis was conducted using original moderators included in each meta-analysis and model selection was completed using the dredge function in the R package MuMIn based on AIC value. 35In order to avoid overfitting, an approximate rule of 10 effect sizes per moderator was applied. 36In practice, this meant that interactions between moderators were rarely included in model selection.Where large numbers of moderators were tested in the original meta-analysis, judgement about inclusion was applied based on the number of levels to the moderator, the number of effect size estimates available, and the relationships between moderators (e.g., if latitude and longitude as well as study country were used as moderators, only country would be included).
To establish whether the moderators accounted for any of the temporal trends in effect sizes, a cumulative meta-analysis of residuals from the best meta-regression model was conducted using a new procedure written to be used in conjunction with functions from the metafor package (Appendix 1).This method allowed for incorporation of both the nested correlation structure of the meta-analyses (multiple effect size estimates from each study, and multiple studies within each meta-analysis), and the impact of moderators on effect sizes.The cumulative residuals were then plotted on a forest plot.Forest plots of cumulative effect sizes and cumulative residuals were compared for each dataset and patterns categorised according to the following criteria (Figure 4): 1.No trend: the temporal trend was absent in the effect size plot and in the residual plot.2. Trend remained: A trend was present in the effect size plot, and the same trend was also present in the residual plot.3. Appear: the original effect size plot was flat, but the residual plot has one or more years that have values that were significantly higher or lower than predicted by the model.4. Disappear: the original effect size plot exhibited a temporal trend; however, the residual plot is flat, indicating that moderators accounted for any temporal changes. 5. Other change: a temporal trend was present in the effect size plot, but this changed form in some way in the residual plot.In most cases this occurred where the effect size plot indicated an early study effect, but the residual plot had outlying years (or a continuous trend where consecutive year estimates were higher or lower than expected) at later year stages.6. N/A, no moderators: there were no moderators recorded in the full-text or data file provided, so the effects of these could not be tested.
The moderators selected by the best model were then categorised according to type in order to examine whether certain types of moderator were more likely to explain temporal trends.These moderator types were: climate (if not determined by study location), study location, study species, experimental conditions (e.g., lab or field), and study methodology (statistical or experimental).The association between the initial magnitude and significance categorisations, moderator types and residual trend (i.e., the likelihood of a temporal trend being explained by moderators) was then assessed using chisquared tests.This analysis was carried out without the two stage adaptation for type 1 error described by Kulinskaya and Mah since this is currently built around univariate meta-regression, 13 however using standard CMA methodology only affected overall presence/absence of magnitude trends in 13 of 121 meta-analyses.

| Other causes of temporal trends
The influence of various other potential causes of temporal trends was also assessed.The correlations between effect magnitude and significance trends (both presence or absence, and specific categorisation) and effect size measure, number and type of moderators, year of metaanalysis publication and publication year span were all tested using chi-squared or Spearman's analysis.The relationship between effect magnitude trends, the factors listed above, and residual trend categorisation were also assessed using chi-squared or Spearman's analyses as appropriate.To test whether sample size (and hence statistical power) of studies changes over time, median sample size per study was aggregated by year, and a Spearman's rank correlation test between median sample size and publication year was conducted.The Spearman's correlation between effect size and sample size was also estimated to see whether any changes in sample size would be likely to result in effect size changes.

| Characteristics of meta-analyses included in the database
The examined meta-analyses included between 10 and 437 primary studies published over the period of 10 to 61 years.The mean and median number of primary studies included per meta-analyses were 56 and 39, correspondingly, and the mean and median publication year span for primary studies were both 24 years.Multiple effect size estimates from different experimental observations were recorded for most of the primary studies, thereby resulting in the number of individual effect estimates per meta-analysis ranging from 10 to 8577 (with a mean of 382 and median of 118).Of the 121 metaanalyses, 41 either did not record which moderators were significant or did not find any of the moderators tested to be significant.It is notable that of these, 13 meta-analyses simply did not report which (if any) moderators had been tested, making exact replication of those meta-analyses impossible.A further 32 meta-analyses recorded only 1 significant moderator.The remaining meta-analyses reported F I G U R E 5 Frequencies of different magnitude and significance trends.
between 2 and 10 significant moderators.In terms of effect metric, initially these were very varied and included log odds ratios, log response ratios, correlation coefficients and both Hedges' d and g amongst others.After recalculation, 109 of the 121 meta-analyses had Hedges' g effect sizes, and the remainder had either Fisher's Z or log response ratio effect sizes.The meta-analyses covered a wide range of sub-topics within applied ecology including forestry, entomology, and agri-environmental schemes (see Appendix 2 for full table of MAs including topic coverage).

| Presence and types of temporal trends in effect magnitude
Of the 121 separate meta-analyses analysed, 109 (90%) showed some form of temporal trend in magnitude of effect size.The most common type of temporal trend was 'Early study effect', with at least one of the first 5 year effect size estimates exhibiting more than 50% magnitude difference to the subsequent estimate (Figure 5).The early study effect was observed in 31 of the 121 metaanalyses (26%).In 52% of cases the qualifying early study estimate was larger in absolute terms than the subsequent estimates, in 42% it was smaller, and in the remainder of cases either it was similar in absolute magnitude but reversed sign, or several year estimates qualified, and some were smaller and some larger.Effects that crossed the zero line were the second most common pattern observed in 25% of meta-analyses, followed by the diminishing trends (otherwise known as the decline effect) in 22% of studies.Only 9% of meta-analyses showed flat trend, that is, no temporal changes in the magnitude of the effects.When the analysis was repeated with only the largest (or random equally large) sample size meta-analysis from each paper, no change was seen in the proportion of meta-analyses showing temporal trends in magnitude, as 71 of the 79 meta-analyses (still 90%) exhibited trends.Consequently, the complete dataset was retained in subsequent analyses.

| Presence and types of temporal trends in significance of the effect
Temporal changes in statistical significance of the cumulative effect occurred much less frequently than changes in magnitude of the effect (Figure 5).The most common trend in terms of significance of the effect was 'retained' (37% of meta-analyses) where statistical significance of the effect was gained in the first 5 year estimates and retained throughout the whole time span, followed by 'never reached' (28%).In 5% of meta-analyses the cumulative estimate of the effect was always significant.However, the remaining 36 meta-analyses (30%) exhibited losses and/or gains of statistical significance of the cumulative effect through the time span (Figure 5).There was no significant association between presence/absence of trends in effect magnitude and significance (x 2 = 0.002, df = 1, P = 0.96), but the association between these variables at the specific category level was highly statistically significant x 2 = 47.47,df = 25, P = 0.004) driven primarily by comparatively large numbers of 'crossing, never reached' and 'flat, retained' categorisations.This association remained significant even when only those analyses which exhibited some kind of temporal trend in magnitude or significance were included (x 2 = 53.35,df = 25, P = 0.0008).As with the magnitude trends, little difference was seen between the proportion of meta-analyses showing significance trends in the complete dataset and the reduced one (30% and 33% respectively).Consequently as previously stated, the larger dataset was retained.

| Magnitude and significance trends combined
When changes in both magnitude and statistical significance of the cumulative effects were taken into account, only 9 of the 121 meta-analyses showed no trend in either effect magnitude or significance whereas 27% of the datasets exhibited temporal trends in both magnitude and significance (Table 1).
T A B L E 1 Frequencies of overall temporal trends in magnitude and/or statistical significance of the cumulative effects across 121 meta-analysis datasets.

Magnitude but not significance 76
Magnitude and significance 33 Neither magnitude nor significance 9 Significance but not magnitude 3 T A B L E 2 Frequencies of residual trend categories after moderator effects were taken into account.

Residual trend Frequency
Trend retained 68

Other change 31
No trend 11 Disappear 7 Appear 4 3.5 | Causes of temporal trends in effect magnitude and significance

| Moderators
Once moderators were accounted for, 99 of the 106 analyses exhibiting temporal trends in effect magnitude (when using standard 1 stage CMA method) either retained their original trend or showed a different or partly different temporal trend (Table 2).Only 7 of the initially present temporal trends in effect sizes disappeared once moderators were accounted for.In addition, 4 of the initial 15 analyses showing no temporal trend in effect magnitude showed a trend once moderators were accounted for.There was no significant effect of the initial effect size trend categorisation on the likelihood of this trend being explained by moderators (x 2 = 13.16,df = 8, P = 0.11, tested only on meta-analyses where an effect size trend was present since residual categorisation is different for analyses with no trend).When all datasets were included, only the methodology moderator a significant effect on the observed residual trend (methodology: x 2 = 9.99, df = 4, P = 0.04).This effect disappeared when only meta-analyses with effect size temporal trends were included (P = 0.18).None of the other moderator categories (climate, location, study taxa, experimental conditions) had any effect on the residual trend either with the inclusion of all data or with just datasets exhibiting trends included (climate: x 2 = 5.18, df = 4, P = 0.26, location: x 2 = 6.12, df = 4, P = 0.19, climate: location (since these may be related): x 2 = 9.96, df = 12, P = 0.62, study species: x 2 = 1.75, df = 4, P = 0.78, experimental conditions: x 2 = 3.05, df = 4, P = 0.55).This is largely driven by the fact that for datasets that had a methodology moderator included, residual trends were more likely to appear in previously flat datasets.However, this result should be interpreted with caution due to the overall very small number of datasets with flat magnitude trends.
The number of significant moderators included in the model had no significant effect on residual trend when tested against either all the data or just the analyses with trends present (all meta-analyses: adjusted R-squared = 0.04, F-statistic = 2.38 on 4 and 115 DF, P = 0.06 and trend only: adjusted R-squared = 0.002, F-statistic = 1.09 on 2 and 102 DF, P = 0.34).The effect of moderators on significance trends was not tested since significance trends cannot be seen on residual graphs.

| Sample size
Of the 109 analyses showing temporal trends in effect magnitudes, 5 showed significant correlation between sample size and year and sample size and effect size, indicating that the temporal trend in these analyses may be explained by sample size differences over time.One of these analyses is in the 'disappear' residual category (indicating the trend may also be explained by moderators) leaving only 4 analyses where the temporal trend is likely to be explained by sample size changes alone.The direction of this correlation varied between meta-analyses, with 3 negatively correlated (sample size decreases with publication year) and 2 positive.Similarly, only 2 of the 36 analyses exhibiting significance trends had significant sample size-year correlations (both negative).

| Year of publication of meta-analysis
Year of publication of meta-analysis had no statistically significant effect on presence of temporal trend in effect magnitude or on specific temporal trend category (Presence/absence of trend: binomial GLM -Null deviance: 73.53 on 119 degrees of freedom, Residual deviance: 71.44 on 118 degrees of freedom, Estimate: À0.19, Z value: À1.26, P value: 0.21; Specific trend: chi squaredx 2 = 55.13,df = 50, P = 0.29).
There was no significant association between year of publication and significance trend at the presence/absence level (Presence/absence: binomial GLM -Null deviance: 147.32 on 120 degrees of freedom, Residual deviance: 145.98 on 119 degrees of freedom, Estimate: À0.07, Z value: 1.16, P value: 0.24) but the association was statistically significant at the specific trend level (chi squaredx 2 = 95.52,df = 50, P = 0.0001).This association disappeared when only those datasets that exhibited temporally unstable significance trends were included (x 2 = 24.31,df = 16, P = 0.08) and is thus likely a result of expected narrowing of confidence intervals as evidence accumulates.
There was no effect of year of publication on residual trend categorisation (residual trend only: x 2 = 56.21,df = 68, P = 0.85; all MA: x 2 = 36.14,df = 40, P = 0.64), no correlation between year of publication and year span of meta-analysis (Spearman's correlation -S = 253,766, P = 0.12), nor were there were any statistically significant correlations between Year span and temporal trends in effect magnitude or significance, or on residual trend.There was no correlation between Year or year span and number of significant moderators.Our results indicate the widespread presence of temporal trends in cumulative effects in the applied ecology evidence base, mostly in the form of extreme early study effects, but also encompassing many other trends including diminishing and increasing magnitudes of effect.The majority of these temporal trends were not accounted for by the explanatory factors (moderators) tested.
It has previously been suggested that standard CMA methodology may be flawed to an extent due to inflated between-study variances when using cumulative analyses, 37 and that it may overestimate early study effects since it is harder for even extreme later estimates to move the effect average.Consequently, a new method of CMA was used that was more statistically robust. 13espite this, the vast majority of meta-analyses exhibited some temporal trend, and over a quarter of these were early study effects, indicating that such trends cannot be adequately explained by methodological artefacts alone.
The presence and declining and/or 'early study' nature of the majority of these trends aligns closely with previous evidence in ecology by Barto and Rillig 8 and Jennions and Møller 7 and indicate that temporal trends may be as prevalent in ecology as in other disciplines.While recent discussion of temporal trends in ecology focussed largely on prevalence of decline effects, 9 the diversity of forms of temporal trends that we have demonstrated indicates a much more complex structure to these trends than previously thought.In addition, previous reviews have not separated an 'early study' diminishing effect (where early estimates are extreme, but then the trend is flat) from a gradually declining effect over the entire time span, making the results less comparable.
6]34 Specifically, in the study by Fanshawe et al. 5 which explored temporal trends in Cochrane intervention reports using very similar methods to the ones in this paper, 57% of the meta-analyses showed an early study/diminishing effect on effect magnitude, 20% showed some form of crossing the zero line, and 23% showed an increasing trend.This compares to around 48% of the meta-analyses in our paper showing early study/diminishing effects, and 25% showing crossing trends, along with the 8% showing increasing trends and the 9% showing the additional categorisation of outlying years.Additionally, Fanshawe et al. found that approximately 27% of the meta-analysis exhibited temporal trends in effect significance compared to the 30% found in our study.As with our analysis, Fanshawe et al. were unable to clearly establish the causes of these trends, though they did find certain subject areas were more prone to temporal instability.This suggests that the prevalence of temporal trends in effect magnitude and significance in applied ecology may in fact be similar to those found in medicine.
This stands in contrast with a recent review by Costello and Fox 9 that stated that decline effects are rare in ecology as they were detected in around 5% of cases of 466 ecological meta-analyses examined.This may be due to methodological differences, as Costello and Fox's methods were only able to detect linear changes in effect magnitude, which as indicated by our results represent only a small proportion of the complex and evolving trends that may be present.In fact, a subsequent comment on Costello and Fox's paper by Yang et al. suggested that the methods used were potentially 'underpowered' to detect temporal trends, and when the results were re-analysed focusing on the magnitude of the decline (rather than the percentage of meta-analyses with P values <0.05), evidence was found that decline effects are 'persistent' and 'non-negligible' in ecology. 38imilarly, Yang et al. found no evidence that decline effects are becoming less common over time, 38 which is additionally displayed in our results with no statistically significant association present between year of publication of a meta-analysis, and the temporal trend or lack thereof it exhibited.Additionally, whilst Costello and Fox state the likely cause of early study effects to be inflated between-study variances, the form of CMA we used specifically addressed this issue, 13 and 26% of meta-analyses still exhibited early study effects.

| Potential causes of temporal trends
Barto and Rillig 8 and Jennions and Møller 7 suggested that temporal changes in sample size could offer a potential explanation for the temporal trends in effect sizes.However, in our dataset only 5 of the 106 meta-analyses with temporal trends in magnitude or significance had any significant temporal trend in sample size.Furthermore, even where sample size changes were significant, the direction of the correlation was varied, rather than positive as reviews suggest they would likely be if they explained the trend. 7,8he observed temporal trends also do not seem to be often explained by moderator changes, since only 7 of the 106 analyses with magnitude trends had 'disappear' residual categorisations.This is surprising as moderators have been suggested as a likely cause of temporal trends in several previous ecology studies, 17,39 but it is possible that the key moderators were missed by the authors in the preparation of the meta-analysis or during primary data collection.
With sample size, CMA methodology, and moderators not providing an adequate explanation for the temporal trends present in the ecology evidence base, two likely explanations of the trends remain: dissemination bias, or true biological change.Given the large number of trends observed, it seems unlikely that all of these are associated with genuine biological trends in magnitude of the effect, especially given the high proportion of extreme early study effects (biological changes would likely be more gradual).Consequently, some form of dissemination bias seems to be the most likely explanation for at least the early study portion of the temporal trends (as predicted by Jennions and Moller 7 ).This is especially likely since these 'early study' estimates were universally more extreme than subsequent ones, possibly suggesting a time-lag or selective reporting effect similar to that discussed in other reviews. 7,8,15,17,19,39ne important feature of this study to note is that due to the relatively long year span and number of studies (10 in both cases) required to adequately assess for temporal trends, many relevant and interesting meta-analyses did not qualify for inclusion.This reduced sample size in several groups, and made comparisons such as between specific research fields within applied ecology impossible.Additionally, it is concerning that so many meta-analyses took data from such small numbers of primary studies, since this increases the risk of inaccurate or temporally unstable conclusions.

| CONCLUSIONS AND RECOMMENDATIONS
Our results have significant management implications since they indicate widespread, varied, and unexplained temporal instability in the applied ecology and conservation biology evidence, the risk of which was outlined in Koricheva and Kulinskaya. 3 Given that the practical recommendations arising from results of a meta-analysis are based on magnitude and statistical significance of the effect, complex and often rapid temporal changes in cumulative effect magnitude and significance observed in this study suggest that results of meta-analysis on the same topic, and the resulting recommendations, might often depend on when the metaanalysis is conducted.Therefore, we recommend using cumulative meta-analysis in each ecological meta-analysis to test for presence of temporal trends which are likely to invalidate the results of analysis in the immediate future.
Our results also suggest that any management decisions made based on primary studies published in the first few years of a novel idea may be subject to early study effects and should be taken with caution until replicated.In addition, since these results seem to refute previous assertions that temporal instability in the ecology discipline is down to moderator or sample size changes, they may suggest a very concerning problem with publication or dissemination biases within the discipline.1][42] Finally, the potential presence of true biological changes causing temporal trends suggests the concerning possibility of unknown changes in the environment, which has major implications for management.
Clearly, despite these interesting results, many questions remain about temporal trends in the ecology evidence base, for example, exploring the role of publication/ dissemination bias in explaining (especially for early study trends), and in further exploration of management/policy recommendations.

F I G U R E 2
Examples of temporal trends (or lack thereof) in effect magnitude.

F I G U R E 3
Examples of temporal trends (or lack thereof) in effect significance.F I G U R E 4 Examples of residual trends.BRISCO ET AL.

4 | DISCUSSION 4 . 1 |
Prevalence and patterns of temporal trends in cumulative effects in applied ecology and conservation