To perform an overview of literature search strategies in systematic reviews (SRs) published in periodontology and implant dentistry.
To perform an overview of literature search strategies in systematic reviews (SRs) published in periodontology and implant dentistry.
Two electronic databases (PubMed and Cochrane Database of SRs) were searched, independently and in duplicate, for SRs with meta-analyses on interventions, with the last search performed on 11 November 2012. Manual searches of the reference lists of included SRs and 10 specialty dental journals were conducted. Methodological issues of the search strategies of included SRs were assessed with Cochrane collaboration guidelines and AMSTAR recommendations. The search strategies employed in Cochrane and paper-based SRs were compared.
A total of 146 SRs with meta-analyses were included, including 19 Cochrane and 127 paper-based SRs. Some issues, such as “the use of keywords,” were reported in most of the SRs (86%). Other issues, such as “search of grey literature” and “language restriction,” were not fully reported (34% and 50% respectively). The quality of search strategy reporting in Cochrane SRs was better than that of paper-based SRs for seven of the eight criteria assessed.
There is room for improving the quality of reporting of search strategies in SRs in periodontology and implant dentistry, particularly in SRs published in paper-based journals.
Systematic reviews (SRs) of the dental literature have frequently been published in recent decades. The main advantage of SRs over narrative reviews is that the former may provide less biased estimates on treatment approaches. Nevertheless, for this assumption to be considered true, several steps related to the planning and conduction of SRs should be observed (Smith et al. 2011). One of these crucial steps is the strategy for searching the evidence, which should be searched in a highly sensitive fashion, to answer the proposed research question(s) by allowing the inclusion of all relevant information and reducing the risk of biased results (Robinson & Dickersin 2002).
Publication bias may occur when the literature used to support an estimate is not representative of the true available evidence (Easterbrook et al. 1991). For example, publication bias may exist when some relevant literature is not published because of a conflict of interest on the results of therapies. Depending on the level of publication bias, the presented estimates may not represent the numbers reported, completely misleading the guidance of therapies. Some data suggest that published literature may overestimate treatment effects compared with “grey literature” (i.e., studies that have not been formally published) (Hopewell et al. 2007).
Guidelines for search strategies have been developed over the years (Haynes et al. 1994, 2005, Montori et al. 2005, Wong et al. 2006), and some standard criteria should be respected to generate an optimal search strategy. For instance, the AMSTAR checklist for the methodological assessment of SRs and meta-analyses recommends the use of at least two electronic databases for a comprehensive literature search (Shea et al. 2007). Moreover, the literature search should involve hand-searching strategies to complement the electronic search (Shea et al. 2007). As more SRs are becoming available, it would be interesting to assess the standards of the different search strategies used in those reviews.
The primary objective of this study was to assess the standard literature search strategies in SRs with meta-analyses published in the fields of periodontology and implant dentistry. A secondary objective was to explore the potential differences between Cochrane and paper-based SRs in terms of literature search strategies.
SRs with meta-analyses on interventions in periodontology and implant dentistry were included. To be included in the study, the SRs should involve at least two therapeutic approaches to be compared by meta-analysis. SRs with diagnostic and prevalence studies were excluded from this study, as were SRs with only animal studies or in vitro studies.
This study is part of a project that evaluates the methodological quality of SRs with meta-analyses published in the fields of periodontology and implant dentistry. Although the previous project (Faggion et al. 2013) was focused only on the quality of reporting in the abstracts of SRs, the present project is focused on the search strategies described in the materials and methods section. The search for SRs was performed in the PubMed and Cochrane databases. It was limited to SRs published in the English language, with the last search performed in November 2012. The literature search strategy used is depicted in Table 1.
|Electronic databases search||Dental specialty journals||Hand search|
1) Periodontal debridementa
2) Root planninga
3) Furcation defectsa
4) Gingival recessiona
5) Root coverage
6) Sinus floor augmentationa
7) Sinus floor elevation
8) Peri-implantitisa OR periimplantitis
9) Periodontal diseasesa
11) Subgingival curettagea
12) Guided tissue regenerationa
13) Dental implantsa
14) Dental implant
15) #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14
17) #15 AND #16 (Filters: meta-analysis; review)
|Issues from November 1990 to November 2012 (Clinical Implant Dentistry and Related Research, Journal of Clinical Periodontology, Journal of Periodontology, Clinical Oral Implants Research, Journal of Periodontal Research, International Journal of Oral and Maxillofacial Implants, European Journal of Oral Implantology (2008–2012), Journal of Oral Implantology, International Journal of Periodontics and Restorative Dentistry, and Implant Dentistry)||Reference lists of included articles|
The literature search was performed independently and in duplicate by two authors. Any disagreements in the final selection of the SRs were resolved by discussion. In the case of persistent disagreement, a third author was consulted until consensus was achieved.
Eight questions related to the search strategy of the selected SRs were assessed. The questions were based on the recommendations of the Cochrane Handbook (Lefebvre et al. 2011) and a validated checklist for the methodological assessment of SRs and meta-analyses (AMSTAR) (Shea et al. 2007):
The selected questions were previously pilot tested on five SRs by one of the authors (CMF). To increase the precision of data extraction, two reviewers (CMF and SP) assessed the search strategies by retrieving the data of 25 SRs that were not included in the present sample (from fields other than periodontology and implant dentistry). Five rounds of assessment were carried out, with five SRs being assessed in each round. Between rounds, the results were intensively discussed between reviewers to improve the homogeneity of assessment.
All questions were answered with YES (authors adequately addressed the question), NO (authors inadequately addressed the question), and UNCLEAR (authors did not provide information to answer the question). Other relevant data, such as the type and the number of searched databases, were also descriptively reported. The assessment was performed independently and in duplicate by two authors (CMF and SP). All data were extracted and summarized into a standardized table. Any disagreements in the data extraction process were resolved by discussion. In the case of persistent disagreement, a third author was consulted until consensus was achieved. The full dataset was checked for precision by a third reviewer (MA).
The literature search strategies performed in Cochrane reviews were compared with those performed in paper-based journals. The frequency of questions answered with YES in both groups of reviews was measured by Fisher's exact test. Differences with p < 0.05 were considered statistically significant. All data were analysed with the Statistical Package for Social Sciences (SPSS 17.0, SPSS Inc., Chicago, IL, USA).
A total of 146 SRs with meta-analyses had their literature search strategies scrutinized (Data S1). Most of the included SRs (86% of the sample) reported the use of relevant keywords in their literature search. The searching processes used multiple databases, of which Medline was the most prevalent (Table 2). Authors of 40 SRs (27%) used one electronic database in their search, 30 SRs (21%) used two electronic databases, and almost half of the selected SRs (48%) searched three or more electronic databases. Six SRs (4%) did not report the names and number of electronic databases. The use of some kind of interface for searching electronic databases was reported in 83 SRs (57%).
|Electronic databases searched|
|1) Medline: 126 (86)|
|2) EMBASE: 56 (38)|
|3) CENTRAL: 54 (37)|
|4) Cochrane Oral Health Group Trial Register: 38 (25)|
|5) Cochrane Library: 14 (10)|
|6) Cochrane Database of SRs: 13 (9)|
|7) CINAHL: 12 (8)|
|8) Web of knowledge/science: 12 (8)|
|9) Cochrane Controlled Trial Register: 9 (6)|
|10) DARE: 9 (6)|
|11) LILACS: 7 (5)|
|12) Science Direct: 6 (4)|
|13) SCOPUS: 5 (3)|
|14) Other databasesa: 22 (15)|
|15) Not reported: 6 (4)|
A search of the grey literature was reported in 50 SRs (34%). Hand searching was performed by the authors of 121 SRs (84%). Hand searches of scientific journals and of the references of relevant papers were reported in 80 SRs (55%) and 106 SRs (73%) respectively. The literature search was performed in duplicate in 11 SRs (8%). Language restrictions were used in the search strategies of 73 SRs (50%), with 62 SRs (42%) being focused on English. However, 29 SRs (20%) did not report any information on language restriction. The authors of 42 SRs (29%) attempted to contact authors or manufacturers to retrieve unpublished data or to obtain further information on the primary studies. Three SRs (2%) explicitly reported that there was no attempt to contact authors/manufacturers to obtain information on published or unpublished studies, and 101 SRs (69%) did not provide any information on the attempt to contact authors/manufacturers.
The search strategies of Cochrane SRs met more criteria than the search strategies of paper-based SRs (Table 3). Seven of eight questions were completely answered in all Cochrane SRs (n = 19). However, one item, related to the duplication of the literature search, was adopted more often in paper-based SRs than in Cochrane SRs.
|Items||Cochrane (%)||Paper based (%)||df||p-valuea|
|Keywords reported (%)||19 (100)||107 (73)||1||0.076|
|Search in at least two electronic databasesc||19 (100)||81 (55)||1||0.001|
|Interface reported (%)||16 (84)||64 (44)||1||0.006|
|Grey literature performed (%)||19 (100)||31 (21)||1||<0.0001|
|Hand search performed (%)||19 (100)||110 (75)||1||0.13|
|Literature search in duplicate (%)||0b (0)||11 (8)||1||0.36|
|No language restriction (%)||19 (100)||24 (16)||1||<0.0001|
|Contact with authors (%)||19 (100)||23 (16)||1||<0.0001|
Our findings demonstrate that there is room for improvement in the search strategies of SRs with meta-analyses published in periodontology and implant dentistry. This is particularly the case for SRs published in paper-based journals. The lack of a comprehensive search strategy may introduce bias in the estimates of treatment effect and, consequently, may jeopardize the development of appropriate clinical practice guidance.
Reporting bias happens when the dissemination of research findings is influenced by the nature and the direction of results (Sterne et al. 2011). Due to the increasingly large number of SRs that are being published, reporting bias is an important issue in dentistry. A series of reporting bias is recognized in the literature (Table 4). With regard to search strategy, the most common types of bias are often related to publication (Easterbrook et al. 1991), location (Pittler et al. 2000), and language biases (Pham et al. 2005).
|Type of reporting bias||Definition|
|Publication bias||The publication or non-publication of research findings, depending on the nature and direction of the results|
|Time lag bias||The rapid or delayed publication of research findings, depending on the nature and direction of the results|
|Multiple (duplicate) publication bias||The multiple or singular publication of research findings, depending on the nature and direction of the results|
|Location bias||The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results|
|Citation bias||The citation or non-citation of research findings, depending on the nature and direction of the results|
|Language bias||The publication of research findings in a particular language, depending on the nature and direction of the results|
|Outcome reporting bias||The selective reporting of some outcomes but not others, depending on the nature and direction of the results|
Evidence suggests that trials with positive findings are published faster and more frequently than trials with negative results (Hopewell et al. 2009). A negative: positive trial publication proportion of almost 1:2 can be assumed. In other words, for each negative trial published, there will be two positive ones (Hopewell et al. 2009). Moreover, unpublished trials seem to present less positive treatment effects than published trials (Hopewell et al. 2009). Studies with more positive results are frequently published in journals with different hierarchical rankings (Kanaan et al. 2011), generating some sort of “location” bias. Although some data suggest that language restriction to only English language might have a limited effect on meta-analytic estimates (Jüni et al. 2002, Morrison et al. 2012), search strategies should not have any language restriction to increase the generalizability and applicability of the findings of therapeutic interventions (Jüni et al. 2002). Therefore, it is crucial that authors of SRs develop a sensitive search strategy that will include electronic databases, grey literature (Hopewell et al. 2007), and a “snowballing” hand-searching strategy, in the form of searching the reference lists of relevant papers (Greenhalgh & Peacock 2005).
Almost 30% of selected paper-based SRs did not report any keywords in the literature search strategies. The keywords used to perform the search and their combination with Boolean operators should be reported to allow the auditing and replication of the results. The careful use of a wide variety of synonyms (with both free text and controlled vocabulary terms) together with Boolean operators generates a very effective search strategy. The full description of the literature search strategy is important to the continuous development of search strategies for reviews on related topics (Yoshii et al. 2009). Furthermore, the initial search should be performed independently and in duplicate. Although this recommendation is not a standard criterion for the Cochrane and AMSTAR guidelines, the idea is to generate a more efficient search because two independent reviewers are more likely to retrieve more relevant data than a single one. Moreover, a duplicate search will open the discussion for the initial inclusion of papers and thus reducing the risk of not including relevant papers in the initial searching process.
The number of electronic databases searched varied across reviews (Table 2). In the present sample, 100 SRs (68%) performed the search in at least two electronic databases, a recommendation suggested by AMSTAR (Shea et al. 2007). The most searched database was Medline, followed by EMBASE and CENTRAL. Medline is one of the largest databases on health issues, and it makes the study abstracts freely available to the reader. These factors might have contributed to the popularity of this database. Most SRs reported the interface used to access the main databases (e.g., PubMed or OVID, in the case of Medline). This aspect is particularly important because these interfaces may generate heterogeneous results in terms of precision (De Groote 2000). Hence, it is important that the reader receives this information in case he or she needs to repeat or audit the results.
We included only SRs with meta-analyses on interventions because such SRs provide quantitative measurements and report the magnitude and direction of the treatment effect, which will guide the decision-making process. As the authors of certain included reviews had no stated intention to perform a systematic review, some may feel that these reviews are narrative reviews, instead of SRs. Nevertheless, we assume that, to provide reliable estimates, a meta-analysis should be based on the most representative sample of studies. In this case, a comprehensive and sensitive literature search strategy seems to be a “sine qua non” condition to capture all potential information to answer the research question. We may ask what the validity is of performing a meta-analysis based on information derived from a potentially biased narrative review.
Cochrane SRs have been proposed to have greater methodological rigour than paper-based SRs (Jadad et al. 1998, Delaney et al. 2007). Therefore, we compared a sample of Cochrane SRs with paper-based SRs. The search strategies in the Cochrane reviews had better standards than the paper-based reviews in seven of the eight topics assessed (Table 3). The topic “literature search in duplicate” was never reported, probably due to the Cochrane logistics in providing customized search strategies, from expert librarians to Cochrane authors. Regardless, information concerning whether the search strategies were performed in duplicate should be fully reported to the reader.
To the best of our knowledge, this study is one of the few reports assessing the standards of search strategies in a substantial sample of SRs with meta-analyses in dentistry. Two other studies evaluated the search and selection methods used in SRs: one used a limited sample of SRs published in the orthodontics field (Flores-Mir et al. 2006), and the other included SRs in several dental fields (Major et al. 2006). The authors of the latter study evaluated SRs published between 2000 and 2005 and also found that the strategies for searching and selecting studies need to be improved, although some improvements were reported in the sample of SRs published in 2005.
The sample of studies selected in this study is representative because it comprises virtually all SRs with meta-analyses published in the fields of periodontology and implant dentistry. Although the included SRs were published in English only, most of the reviews were published in dental journals with high impact factors, which normally have rigid criteria for publication. The impact factor has been suggested to be an indicator of quality for medical journals (Saha et al. 2003). One might consider the inclusion of SRs published in English only as a limitation of this study. Nevertheless, this study was designed to generate an overview of SRs on interventions in which meta-analytic findings might be influenced by language bias, instead of providing any meta-analytic estimate per se.
The small number of Cochrane reviews compared with paper-based reviews is a limitation of this study. We did not include other Cochrane reviews due to the strict guidelines of the inclusion and exclusion criteria. Therefore, the finding that the Cochrane SRs had more comprehensive search strategies than paper-based SRs should be considered with caution.
Authors of SRs should follow the suggested guidelines below, to develop a comprehensive and sensitive literature search strategy and to reduce the risk of reporting bias:
Scientific rationale for the study: Use of a comprehensive search strategy in SRs with meta-analyses is pivotal to avoid publication bias and to generate reliable results. An assessment of search strategies in SRs has never been reported in periodontology and implant dentistry.
Principal findings: In contrast to Cochrane SRs, paper-based SRs do not usually compliment the search of electronic databases with a grey literature search and contact with authors/manufacturers. Moreover, many paper-based SRs limit their searches to publications in English.
Practical implications: The present findings may guide authors of SRs to develop more comprehensive search strategies to generate less biased meta-analytic treatment effect estimates.