Rapid evidence synthesis approach for limits on the search date: How rapid could it be?

Rapid reviews have been widely employed to support timely decision‐making, and limiting the search date is the most popular approach in published rapid reviews. We assessed the accuracy and workload of search date limits on the meta‐analytical results to determine the best rapid strategy. The meta‐analyses data were collected from the Cochrane Database of Systematic Reviews (CDSR). We emulated the rapid reviews by limiting the search date of the original CDSR to the recent 40, 35, 30, 25, 20, 15, 10, 7, 5, and 3 years, and their results were compared to the full meta‐analyses. A random sample of 10% was drawn to repeat the literature search by the same timeframe limits to measure the relative workload reduction (RWR). The relationship between accuracy and RWR was established. We identified 21,363 meta‐analyses of binary outcomes and 7683 meta‐analyses of continuous outcomes from 2693 CDSRs. Our results suggested that under a maximum tolerance of 5% and 10% on the bias of magnitude, a limit on the recent 20 years can achieve good accuracy and at the same time save the most workload. Under the tolerance of 15% and 20% on the bias, a limit on the recent 10 years and 15 years could be considered. Limiting the search date is a valid rapid method to produce credible evidence for timely decisions. When conducting rapid reviews, researchers should consider both the accuracy and workload to make an appropriate decision.


What is new
• When considering both accuracy and speed, under a tolerable accuracy of 5%-10%, limiting the search date to recent 20 years could be the optimal choice; under a tolerable accuracy of 15% or 20%, the limits of the recent 20, 15, and 10 years could be considered. Nevertheless, the results should be interpreted with more caution when a narrower limit is applied.
Potential impact for Research Synthesis Methods readers outside the authors' field • Rapid review is a valid method to produce credible evidence for timely decisions, while the accuracy of the results depends on how the rapid methodology is applied. When conducting a rapid review, the authors should consider both the accuracy and workload to make an appropriate decision. In practice, due to the large variance between topics, the authors should also consider the actual situation on a case-by-case basis to make an appropriate decision.

| INTRODUCTION
Credible evidence is the fundamental base for scientific decision-making regarding healthcare interventions. Systematic reviews synthesize all available evidence in a comprehensive and transparent manner to provide the best evidence source and are regarded as the highest level of evidence for decision-making. 1,2 To ensure the quality of the evidence, the process of systematic review needs to be conscientious and rigorous. As a result, it is often time-consuming and requires 6 months to 2 years to complete. 3,4 Such long periods may not be well-suited for the needs of decision-makers to make timely decisions, especially when they are facing urgent public health events. 5 The proposition of the rapid review approach provides an effective solution for decision-makers. [5][6][7] It employs streamlined processes for the components of the systematic review that enables researchers to synthesize evidence in a shorter period of time. 8,9 Rapid review is defined as "a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting a variety of methods to produce evidence in a resource-efficient manner." 10 There are many approaches in the rapid review methodology, such as narrowing the review question, limiting the literature search date, simplifying the screening and data exaction process, and simplifying or omitting the quality appraisal. [11][12][13][14][15][16][17][18] Limiting the literature search date is one of the most widely utilized approaches in published rapid reviews; it has been applied in 68%-89% of the rapid reviews. 19 However, the limit on search date in practice is generally subjective and arbitrary-some rapid review authors may limit the search date to the recent 10 years, while some may limit it to at most 7 years or 5 years. As a result, it is unclear whether the evidence under such limits is credible or subject to bias.
Marshall et al empirically evaluated the performance of several types of rapid review methods for binary outcomes among 2512 meta-analyses of the Cochrane Database of Systematic Reviews (CDSR). 20 They estimated the changes of point estimates and error estimates of "emulated" rapid reviews limiting search date to the recent 20 years, 15 years, 10 years, 7 years, and 5 years compared to the full meta-analyses. Their results suggest that about 19.1% (5 years) to 70.8% (20 years) of the rapid reviews had no important changes in the point estimates, and the changes in 41.1% (5 years) to 85.2% (20 years) were not statistically significant. 20 Based on the fact that the majority (78% or more 20 ) of published rapid reviews seldom quantitively synthesize the results, this important work supported that quantitative synthesis in rapid reviews is feasible. In addition, it implies that as the limit on search dates becomes less strict (i.e., from the recent 5 years to the recent 20 years), the results of rapid reviews are likely more concurrent to the full metaanalyses.
Despite these findings, the current evidence is far from well-established for rapid reviews in the context of limiting the search date. Many questions remain unanswered: (a) the performance of rapid reviews is unclear when limiting the search date to recent 25 years or more (e.g., 40 years); (b) the performance of rapid reviews is unclear when limiting the search date under different scenarios (e.g., outcome types, such as binary outcomes and continuous outcomes); (c) more importantly, it is unclear which timeframe limit is optimal when considering both the tolerance on the bias as well as the efficiency for the rapid purpose. In this article, we aimed to address the above three questions. These questions may be summarized as: how to limit the search date and how rapid could it be? We attempted to complete the evidence gap for rapid reviews on the approach to limiting the search date and provide some practical guidelines.

| Data source
We collected meta-analyses data through the CDSR as this database provides comprehensive systematic reviews with a standard implementation process. We obtained access to the CDSR through Florida State University. In detail, we collected rm5 (Review Manager version 5) files of all CDSRs from January 2003 to May 2018 using the R package "RCurl". 21 The reviews without performing a meta-analysis were identified and removed. Considering the potential impact of statistical properties on the results, we further removed the meta-analyses with less than five studies. [22][23][24] For a topic with limited evidence (with less than five studies), it is neither reasonable nor necessary to perform a rapid review; thus, a systematic review is more appropriate. The data files contained the general information in the meta-analyses, including aggregate data of each study in a meta-analysis, publication years of the included studies in a meta-analysis, data types (e.g., binary, continuous), analytic models for metaanalyses (i.e., fixed-effect [FE] and random-effects [RE] models), effect estimators (e.g., odds ratio [OR], risk ratio [RR], risk difference [RD], mean difference [MD], and standard mean difference [SMD]), among others. We removed the meta-analyses that did not contain the information on publication years for the included studies.

| Emulating rapid reviews
We emulated the rapid review approach by limiting the search date. To achieve this, we first ranked the studies in each meta-analysis by publication year, and the most recent one was treated as the reference date. We then emulated 10 scenarios that generated 10 rapid reviews: limiting studies published in the recent 40, 35, 30,25,20,15,10,7,5, and 3 years based on the reference date. The limits led to study loss, which further impacted the final synthesized results. For example, suppose a metaanalysis included 10 studies published from 2015 to 2019.
The reference date was set to 2019, and the limits generated 10 separate data sets for rapid reviews. Only one data set (i.e., with limits in the recent 3 years) had study loss, and the other 9 had no study loss. We set the widest limits as the recent 40 years because less than 5% of the studies were published before 1978 based on the CDSR dataset.
To measure how rapid the reviews could be performed by using such limits, we also repeated the literature search for the CDSRs that provided a full search strategy with the same limits: limiting search date in recent 40, 30,25,20,15,10,7,5, and 3 years based on the original search date reported by the CDSRs, instead of the reference date. Considering the large amount of workload for the current research, we drew a sample of 10% at random by stratified random procedure (year of publication) among the systematic reviews that provided a search strategy (see Data S1 for the details). In order to facilitate the process, the reproduction of the literature search was based on four electronic databases, Embase, Medline, PubMed, and the Cochrane central register of controlled trials, and we did not carry out additional searches (e.g., manual search). Although it is not 100% replication of the search, we expected it would not substantially impact the estimation of the relative workload.

| Data analysis
We re-analyzed all full meta-analyses: for meta-analyses with binary data, we estimated both pooled ORs and RDs for them; for meta-analyses with continuous data, we estimated both pooled MDs and SMDs. We considered these effect measures because they are the most commonly used ones in practice. We did not use RR for binary data because it may not be considered "portable" and thus a valid measure of effect. 25 Considering the potential between-study variance, we employed the RE model and inverse variance heterogeneity (IVhet) model. 26,27 The IVhet model has the same point estimates as the FE model based on the inverse variance weighting scheme, while it is expected to be more conservative than the RE model with respect to error estimation. 27 Studies with zero events in a single arm or both arms were handled with a continuity correction procedure. 28 We further applied the same meta-analytic methods to the 10 emulated rapid review scenarios.
The magnitudes and directions of the point estimates for the rapid reviews and the full meta-analyses were then compared. For the magnitudes, we pre-defined the following four cut-offs as maximum tolerance on bias of 5%, 10%, 15%, and 20%. The rapid reviews with relative biases, less than the defined cut-offs were considered of good accuracy (convergency). We used these cut-offs because a previous survey found that guideline developers and policy makers would tolerate a median risk of "biased" estimates of 10% (interquartile range [IQR]: 5%-15%). 29 The (relative) bias was estimated asθ Àθ r À Á =θ , whereθ was the point estimate for a full meta-analysis andθ r was the point estimate for a rapid review emulated from a full meta-analysis. For the directions of the full meta-analyses and the corresponding rapid reviews, we examined whether the change in ORs crossed 1, or changes in RDs, MDs, and SMDs crossed 0. 30 We conducted additional analyses to evaluate the impact of the amount of heterogeneity, event rate (ER), magnitude of point estimates, and publication bias on the results (see details in Data S1). 31 For the question of how rapid a review could be performed, we estimated the relative workload reduction for the literature screening of the rapid reviews compared to the original systematic reviews. The relative workload reduction was calculated as 1 À N r =N s , where N r is the number of records obtained from rapid reviews based on limits on search date and N s is the number of records from systematic reviews. This is expected to be a valid measurement to offset the impact of increasing publications over times. To further address the question of which of the above limits is the best choice considering both accuracy and workload, we obtained the Youden index (accuracy + relative workload reduction À 1) of the accuracy and relative workload reduction as well as the receiver operating characteristic (ROC) curves. 32 The data cleaning process was conducted by Stata/SE 14.0 (Stata, College Station, TX), and meta-analyses were conducted by R (version 3.4.2, R Development Core Team).

| RESULTS
Through the CDSR, we identified 21,363 meta-analyses of binary outcomes and 7683 meta-analyses of continuous outcomes that met our criteria. They were from 2693 CDSRs ( Figure S1, see Data S1).
3.1 | Accuracy of rapid reviews by limiting the search date Figure 1 presents the impact of date limits on the magnitude of effect sizes when a 5% change in magnitude was tolerable. Our results suggested that when limiting the search date to the recent 40, 35, 30, 25, and 20 years, there was a high proportion (> 80%) of meta-analyses achieving convergency, regardless of the effect estimator utilized. However, when limiting the date of publication to the recent 15 (78.39%), 10 (61.39%), 7 (46.78%), 5 s (36.29%), and 3 years (25.47%), the convergency rate dropped rapidly. There were similar trends when the tolerance cut-offs were set as 10%, 15%, and 20% ( Figures S2-S4, see Data S1). Figure 2 presents the impact of date limits on the direction of effect sizes. Again, when limiting the search date to the recent 40, 35, 30, and 20, a very small proportion (>2.5%) of meta-analyses changed the directions of the effect sizes. The proportion of meta-analyses that changed the directions sharply increased when limiting search date to the recent 10 (7.18%) to 3 years (18.11%). We noticed that among the two analytic models, the RE model had a slightly lower proportion that reached convergency and a higher proportion that changed the direction of effect sizes.
We further observed some impact of heterogeneity, event rate, magnitude of effect size, and publication bias on the accuracy (see Data S1, Figures S5-S13).

| Relative workload reduction by limiting the search date
From 2693 CDSRs, we identified 2177 (80.84%) that provided the full search strategy. A stratified (by publication year) random sampling procedure resulted in 219 systematic reviews to estimate the relative workload reduction by the limits on search date (Data S2). Table 1 presents the limits on the date and their influence on the loss of study, loss of sample size, and relative workload reduction. We can clearly see that when the limits on the search date ranged from the recent 40 to 20 years, the median loss of study and sample size was nearly zero, which explains why such limits reached good convergence and a low proportion that changed the directions of effect sizes. The median relative workload reduction ranged from 1.68% (recent 40 years) to 77.77% (recent 3 years). As the limits on the search date became narrower, there was an obvious increase in relative workload reduction. By limiting the search date from the recent 40 years to the recent 20 years, half of the rapid review authors can save about 1.68%-22.92% of workload on literature screening.                  15 years, the Youden index had the largest values for meta-analyses with both binary and continuous outcomes. Therefore, if at least 80% of the meta-analyses that reached convergency were acceptable, then the limit of the recent 20 years could be the best choice considering both the workload and accuracy. The results were similar when a 10% change was set to the tolerance cut-off ( Figure S14, Data S1). If we set the tolerance cut-off to 15% and 20% on convergency (Figures S15-S16, Data S1), the limits of the recent 10 and 15 years could also be considered after accounting for both the workload and accuracy for rapid reviews, with >80% reaching convergence.

| DISCUSSION
In this study, we used a large real-world dataset to investigate the impact of limits on search date for rapid reviews on the accuracy of the results and the potential relative workload reduction. Our results suggested that narrower search date limits in rapid reviews would lead to less accuracy of the results and a higher possibility to alter the conclusions of full systematic reviews, due to the loss of eligible studies. On the other hand, a narrower search date limit would bring an obvious reduction in the workload that makes the process more rapid. When considering both accuracy and speed, under a tolerable accuracy between 5% and 10%, limiting the search date to recent 20 years could be the optimal choice; under a tolerable accuracy between 15% and 20%, the limits of recent 20, 15, and 10 years could be considered. Nevertheless, the results should be interpreted with caution when a narrower limit is applied. Besides the limits on search date, we also observed the impact of heterogeneity, event rates, magnitudes of effect sizes, and publication bias on the accuracy. These results suggested that such factors should also be considered when appraising a rapid review. For example, when the outcome of interest is an adverse event, there would be a lower possibility to reach a bias-tolerable accuracy, and the results then should be interpreted with caution. Except for event rates, the other three factors were all post hoc statistics that could only be detected after data syntheses. As recommended by the Grading of Recommendations, Assessment, Development, and Evaluations (GRADE) working group, when substantial heterogeneity is detected, a small or moderate effect is observed, an imprecise result is obtained, or publication bias is detected, the level of evidence should be rated down. 33 In our study, there was a higher proportion of metaanalyses that reached the acceptable tolerance under different limits of search date than Marshall et al. 20 This may be due to the difference in the inclusion criteria; we excluded the meta-analyses with less than five studies, while such meta-analyses were included in the study by Marshall et al. This further suggests that for a topic with only few studies available, the use of the rapid approach by limiting the search date would have less accuracy compared to those with more studies. In such situations, as we mentioned earlier, a rapid review is unnecessary as it does not generate a differential amount of work (while it may increase the uncertainty) compared to a systematic review. Nevertheless, both studies confirm that rapid synthesis of the evidence by limiting the search date is feasible and a proper limit could generate a reasonable accuracy of the results.

| Implications for future rapid reviews
Based on the evidence from the current study, there are some important methodological recommendations for rapid review authors to consider when attempting to produce credible evidence.
• First, before starting a rapid review, the authors should specify the extent of tolerable bias. As shown in our results, a less strict cut-off of the tolerable bias (e.g., 20%) would result in a higher proportion of metaanalyses with the bias located within the criteria, and vice versa. This is important to further determine the limits of search date to conduct the review as rapidly as possible. For example, it is inappropriate to apply a limit of the recent 5 years for a rapid review while expecting a probability of at least 90% to obtain consistent results with a full systematic review. • Second, the authors should ensure an expected possibility of a rapid review having consistent results with a full systematic review under the defined tolerable bias. Although this is somewhat subjective and arbitrary, one may preferably set it as large as possible (e.g., ≥0.8).
• Third, the authors should consider both the accuracy and workload, and choose the limit scheme with good accuracy and relatively small workload. • Fourth, theoretically, employing multiple rapid review methodologies means higher bias that the results tend to be less accurate. Therefore, in practice, to trade-off the bias by additional rapid review methodologies (e.g., search only one database and language restrictions), it is suggested to relax the limits of existing rapid schemes. For example, if we plan to limit the databases, we may need to relax the limits on search data from the recent 10 years to the recent 15 years. • Fifth, for some new research topics (e.g., , due to the limited evidence and a narrow time period, there is mostly little difference in the workload for a rapid review and systematic review. In such situations, perhaps conducting systematic reviews (or as we prefer to call it "urgent reviews") with constant updates is more appropriate.

| Strengths and limitations
To the best of our knowledge, this is the first comprehensive study that investigated the accuracy and workload by different limits of search date. Our results were based on a large real-world dataset, so they are expected to be representative. We also investigated the accuracy of a rapid review under different scenarios (i.e., outcome types, event rates, heterogeneity, etc.). The findings will provide practical guidelines for future rapid review authors when applying the limit approach to search date. Nevertheless, several limitations of this study should be highlighted. First, 8% of the original meta-analyses failed to report information of publication years of the included studies, and they were excluded from our analyses. Such missingness might not be at random, potentially having some impact on our results. However, the impact may likely be negligible as the missingness proportion was small. Second, we used the most recent study as the reference to emulate the rapid reviews, but the original search date was used as the reference to repeat the literature search; these two dates did not always coincide. The latter is expected to be more recent than the former and might be a slight impact on the results of the ROC analysis and Youden index. However, the original search date could not be extracted from the rm5 files, and it would have been a large amount of work to manually check it from more than 2600 Cochrane reviews. An automatic data exaction tool to extract more relevant data could be useful and worth to be developed in the future. Third, when measuring relative workload reductions, we only used 10% of the meta-analyses to repeat and emulate the rapid review by limiting the search date. The estimated workload reductions may have some minor differences from the complete set of meta-analyses due to inevitable random error. However, we believe these limitations would not greatly change our results as their impact is likely to be small. Fourth, in the current study, we removed those meta-analyses with less than five studies because our simulation suggested that such meta-analyses were inconclusive, especially for heterogeneous meta-analyses. However, in practice, many systematic reviews or rapid reviews may conduct meta-analyses with less than five studies (although it is not recommended based on current evidence). Therefore, the findings of the current study may be only suitable for reviews with five or more studies. Fifth, for each full meta-analysis and rapid review, we employed the same analytic model to synthesize the data. However, in practice, due to the loss of study, the between-study variance may be changed, so the model used in the full metaanalysis may not be the best for every rapid reviews. This may have a potential impact on the results. Moreover, it should be noted that the results are only relevant for technologies for which a fixed date of invention or of entering the market (e.g., drug licensing) is not determinable.

| CONCLUSIONS
Based on our findings, we conclude that when conducting a rapid review, the authors should consider both the accuracy and workload, and choose a limit scheme with good accuracy and a relatively small workload. In general, for a rapid review, limiting the publications to be in recent 10 to 20 years would be a reasonable choice, accounting for both accuracy and workload. In practice, due to the large variance between topics, the authors should also consider the actual situation on a case-bycase basis to make an appropriate decision.