Search strategies in systematic reviews in periodontology and implant dentistry


  • Conflict of interest and source of funding statement

    The authors declare they have no conflicts of interest.

    The first author is the Colgate Senior Lecturer at the University of Otago. This position is partially supported by an unrestricted grant from Colgate-Palmolive New Zealand to the University of Otago. The authors received no special funding for conducting this study.


Clovis Mariano Faggion Jr

Faculty of Dentistry

University of Otago

PO Box 647, Dunedin 9054

New Zealand




To perform an overview of literature search strategies in systematic reviews (SRs) published in periodontology and implant dentistry.

Materials and Methods

Two electronic databases (PubMed and Cochrane Database of SRs) were searched, independently and in duplicate, for SRs with meta-analyses on interventions, with the last search performed on 11 November 2012. Manual searches of the reference lists of included SRs and 10 specialty dental journals were conducted. Methodological issues of the search strategies of included SRs were assessed with Cochrane collaboration guidelines and AMSTAR recommendations. The search strategies employed in Cochrane and paper-based SRs were compared.


A total of 146 SRs with meta-analyses were included, including 19 Cochrane and 127 paper-based SRs. Some issues, such as “the use of keywords,” were reported in most of the SRs (86%). Other issues, such as “search of grey literature” and “language restriction,” were not fully reported (34% and 50% respectively). The quality of search strategy reporting in Cochrane SRs was better than that of paper-based SRs for seven of the eight criteria assessed.


There is room for improving the quality of reporting of search strategies in SRs in periodontology and implant dentistry, particularly in SRs published in paper-based journals.

Systematic reviews (SRs) of the dental literature have frequently been published in recent decades. The main advantage of SRs over narrative reviews is that the former may provide less biased estimates on treatment approaches. Nevertheless, for this assumption to be considered true, several steps related to the planning and conduction of SRs should be observed (Smith et al. 2011). One of these crucial steps is the strategy for searching the evidence, which should be searched in a highly sensitive fashion, to answer the proposed research question(s) by allowing the inclusion of all relevant information and reducing the risk of biased results (Robinson & Dickersin 2002).

Publication bias may occur when the literature used to support an estimate is not representative of the true available evidence (Easterbrook et al. 1991). For example, publication bias may exist when some relevant literature is not published because of a conflict of interest on the results of therapies. Depending on the level of publication bias, the presented estimates may not represent the numbers reported, completely misleading the guidance of therapies. Some data suggest that published literature may overestimate treatment effects compared with “grey literature” (i.e., studies that have not been formally published) (Hopewell et al. 2007).

Guidelines for search strategies have been developed over the years (Haynes et al. 1994, 2005, Montori et al. 2005, Wong et al. 2006), and some standard criteria should be respected to generate an optimal search strategy. For instance, the AMSTAR checklist for the methodological assessment of SRs and meta-analyses recommends the use of at least two electronic databases for a comprehensive literature search (Shea et al. 2007). Moreover, the literature search should involve hand-searching strategies to complement the electronic search (Shea et al. 2007). As more SRs are becoming available, it would be interesting to assess the standards of the different search strategies used in those reviews.

The primary objective of this study was to assess the standard literature search strategies in SRs with meta-analyses published in the fields of periodontology and implant dentistry. A secondary objective was to explore the potential differences between Cochrane and paper-based SRs in terms of literature search strategies.

Materials and Methods

Inclusion and exclusion criteria

SRs with meta-analyses on interventions in periodontology and implant dentistry were included. To be included in the study, the SRs should involve at least two therapeutic approaches to be compared by meta-analysis. SRs with diagnostic and prevalence studies were excluded from this study, as were SRs with only animal studies or in vitro studies.

Literature search

This study is part of a project that evaluates the methodological quality of SRs with meta-analyses published in the fields of periodontology and implant dentistry. Although the previous project (Faggion et al. 2013) was focused only on the quality of reporting in the abstracts of SRs, the present project is focused on the search strategies described in the materials and methods section. The search for SRs was performed in the PubMed and Cochrane databases. It was limited to SRs published in the English language, with the last search performed in November 2012. The literature search strategy used is depicted in Table 1.

Table 1. Search strategy used to search for systematic reviews with meta-analyses in periodontology and implant dentistry
Electronic databases searchDental specialty journalsHand search
  1. a

    MeSH terms.

1) Periodontal debridementa

2) Root planninga

3) Furcation defectsa

4) Gingival recessiona

5) Root coverage

6) Sinus floor augmentationa

7) Sinus floor elevation

8) Peri-implantitisa OR periimplantitis

9) Periodontal diseasesa

10) Periodontitisa

11) Subgingival curettagea

12) Guided tissue regenerationa

13) Dental implantsa

14) Dental implant

15) #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14

16) Meta-analysisa OR metaanalysis OR mean difference OR risk difference OR risk ratio OR relative riska OR odds ratioa OR fixed-effects OR random-effects

17) #15 AND #16 (Filters: meta-analysis; review)

Issues from November 1990 to November 2012 (Clinical Implant Dentistry and Related Research, Journal of Clinical Periodontology, Journal of Periodontology, Clinical Oral Implants Research, Journal of Periodontal Research, International Journal of Oral and Maxillofacial Implants, European Journal of Oral Implantology (2008–2012), Journal of Oral Implantology, International Journal of Periodontics and Restorative Dentistry, and Implant Dentistry)Reference lists of included articles

The literature search was performed independently and in duplicate by two authors. Any disagreements in the final selection of the SRs were resolved by discussion. In the case of persistent disagreement, a third author was consulted until consensus was achieved.

Rationale for the assessment of search strategies

Eight questions related to the search strategy of the selected SRs were assessed. The questions were based on the recommendations of the Cochrane Handbook (Lefebvre et al. 2011) and a validated checklist for the methodological assessment of SRs and meta-analyses (AMSTAR) (Shea et al. 2007):

  • Did the authors of SRs explicitly report the terms used to search for primary studies?
  • Did the authors search in at least two electronic databases?
  • Did the authors report any interface to search in the electronic databases?
  • Did the authors search the grey literature, namely, information that is not published in easily accessible journals or databases, such as conference proceedings that include the abstracts of research presented at conferences or unpublished theses (
  • Did the authors perform “hand-searching,” which includes searching reviews, textbooks, reviewing the references of the selected studies, and contacting specialized registers or experts in the particular field of study?
  • Did the authors conduct the search in duplicate?
  • Did the search strategy include any language restriction; was the search strategy limited to one or two languages and, hence, insufficiently comprehensive to cover all languages?
  • Did the authors make any attempt to contact the authors of primary studies or manufacturing companies to obtain further relevant information?

Data extraction

The selected questions were previously pilot tested on five SRs by one of the authors (CMF). To increase the precision of data extraction, two reviewers (CMF and SP) assessed the search strategies by retrieving the data of 25 SRs that were not included in the present sample (from fields other than periodontology and implant dentistry). Five rounds of assessment were carried out, with five SRs being assessed in each round. Between rounds, the results were intensively discussed between reviewers to improve the homogeneity of assessment.

All questions were answered with YES (authors adequately addressed the question), NO (authors inadequately addressed the question), and UNCLEAR (authors did not provide information to answer the question). Other relevant data, such as the type and the number of searched databases, were also descriptively reported. The assessment was performed independently and in duplicate by two authors (CMF and SP). All data were extracted and summarized into a standardized table. Any disagreements in the data extraction process were resolved by discussion. In the case of persistent disagreement, a third author was consulted until consensus was achieved. The full dataset was checked for precision by a third reviewer (MA).

Comparison of Cochrane and paper-based reviews

The literature search strategies performed in Cochrane reviews were compared with those performed in paper-based journals. The frequency of questions answered with YES in both groups of reviews was measured by Fisher's exact test. Differences with p < 0.05 were considered statistically significant. All data were analysed with the Statistical Package for Social Sciences (SPSS 17.0, SPSS Inc., Chicago, IL, USA).


A total of 146 SRs with meta-analyses had their literature search strategies scrutinized (Data S1). Most of the included SRs (86% of the sample) reported the use of relevant keywords in their literature search. The searching processes used multiple databases, of which Medline was the most prevalent (Table 2). Authors of 40 SRs (27%) used one electronic database in their search, 30 SRs (21%) used two electronic databases, and almost half of the selected SRs (48%) searched three or more electronic databases. Six SRs (4%) did not report the names and number of electronic databases. The use of some kind of interface for searching electronic databases was reported in 83 SRs (57%).

Table 2. Search strategies (performed in electronic databases) in SRs with meta-analyses published in periodontology and implant dentistry. Numbers in parenthesis are the percentage of all included SRs (n = 146)
Electronic databases searched
  1. a

    Number of databases reported less than five times in the SRS.

1) Medline: 126 (86)
2) EMBASE: 56 (38)
3) CENTRAL: 54 (37)
4) Cochrane Oral Health Group Trial Register: 38 (25)
5) Cochrane Library: 14 (10)
6) Cochrane Database of SRs: 13 (9)
7) CINAHL: 12 (8)
8) Web of knowledge/science: 12 (8)
9) Cochrane Controlled Trial Register: 9 (6)
10) DARE: 9 (6)
11) LILACS: 7 (5)
12) Science Direct: 6 (4)
13) SCOPUS: 5 (3)
14) Other databasesa: 22 (15)
15) Not reported: 6 (4)

A search of the grey literature was reported in 50 SRs (34%). Hand searching was performed by the authors of 121 SRs (84%). Hand searches of scientific journals and of the references of relevant papers were reported in 80 SRs (55%) and 106 SRs (73%) respectively. The literature search was performed in duplicate in 11 SRs (8%). Language restrictions were used in the search strategies of 73 SRs (50%), with 62 SRs (42%) being focused on English. However, 29 SRs (20%) did not report any information on language restriction. The authors of 42 SRs (29%) attempted to contact authors or manufacturers to retrieve unpublished data or to obtain further information on the primary studies. Three SRs (2%) explicitly reported that there was no attempt to contact authors/manufacturers to obtain information on published or unpublished studies, and 101 SRs (69%) did not provide any information on the attempt to contact authors/manufacturers.

Cochrane versus paper-based SRs

The search strategies of Cochrane SRs met more criteria than the search strategies of paper-based SRs (Table 3). Seven of eight questions were completely answered in all Cochrane SRs (n = 19). However, one item, related to the duplication of the literature search, was adopted more often in paper-based SRs than in Cochrane SRs.

Table 3. Number of studies reporting the selected items to assess the literature search strategies of Cochrane and paper-based systematic reviews
ItemsCochrane (%)Paper based (%)dfp-valuea
  1. a

    Fisher's exact test.

  2. b

    One review (Eberhard 2008) duplicated the hand searching only.

  3. c

    AMSTAR recommends searching in at least two electronic databases.

  4. df, degree of freedom.

Keywords reported (%)19 (100)107 (73)10.076
Search in at least two electronic databasesc19 (100)81 (55)10.001
Interface reported (%)16 (84)64 (44)10.006
Grey literature performed (%)19 (100)31 (21)1<0.0001
Hand search performed (%)19 (100)110 (75)10.13
Literature search in duplicate (%)0b (0)11 (8)10.36
No language restriction (%)19 (100)24 (16)1<0.0001
Contact with authors (%)19 (100)23 (16)1<0.0001


Our findings demonstrate that there is room for improvement in the search strategies of SRs with meta-analyses published in periodontology and implant dentistry. This is particularly the case for SRs published in paper-based journals. The lack of a comprehensive search strategy may introduce bias in the estimates of treatment effect and, consequently, may jeopardize the development of appropriate clinical practice guidance.

Reporting bias happens when the dissemination of research findings is influenced by the nature and the direction of results (Sterne et al. 2011). Due to the increasingly large number of SRs that are being published, reporting bias is an important issue in dentistry. A series of reporting bias is recognized in the literature (Table 4). With regard to search strategy, the most common types of bias are often related to publication (Easterbrook et al. 1991), location (Pittler et al. 2000), and language biases (Pham et al. 2005).

Table 4. Types of reporting bias (Sterne et al. 2011)
Type of reporting biasDefinition
Publication biasThe publication or non-publication of research findings, depending on the nature and direction of the results
Time lag biasThe rapid or delayed publication of research findings, depending on the nature and direction of the results
Multiple (duplicate) publication biasThe multiple or singular publication of research findings, depending on the nature and direction of the results
Location biasThe publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results
Citation biasThe citation or non-citation of research findings, depending on the nature and direction of the results
Language biasThe publication of research findings in a particular language, depending on the nature and direction of the results
Outcome reporting biasThe selective reporting of some outcomes but not others, depending on the nature and direction of the results

Evidence suggests that trials with positive findings are published faster and more frequently than trials with negative results (Hopewell et al. 2009). A negative: positive trial publication proportion of almost 1:2 can be assumed. In other words, for each negative trial published, there will be two positive ones (Hopewell et al. 2009). Moreover, unpublished trials seem to present less positive treatment effects than published trials (Hopewell et al. 2009). Studies with more positive results are frequently published in journals with different hierarchical rankings (Kanaan et al. 2011), generating some sort of “location” bias. Although some data suggest that language restriction to only English language might have a limited effect on meta-analytic estimates (Jüni et al. 2002, Morrison et al. 2012), search strategies should not have any language restriction to increase the generalizability and applicability of the findings of therapeutic interventions (Jüni et al. 2002). Therefore, it is crucial that authors of SRs develop a sensitive search strategy that will include electronic databases, grey literature (Hopewell et al. 2007), and a “snowballing” hand-searching strategy, in the form of searching the reference lists of relevant papers (Greenhalgh & Peacock 2005).

Almost 30% of selected paper-based SRs did not report any keywords in the literature search strategies. The keywords used to perform the search and their combination with Boolean operators should be reported to allow the auditing and replication of the results. The careful use of a wide variety of synonyms (with both free text and controlled vocabulary terms) together with Boolean operators generates a very effective search strategy. The full description of the literature search strategy is important to the continuous development of search strategies for reviews on related topics (Yoshii et al. 2009). Furthermore, the initial search should be performed independently and in duplicate. Although this recommendation is not a standard criterion for the Cochrane and AMSTAR guidelines, the idea is to generate a more efficient search because two independent reviewers are more likely to retrieve more relevant data than a single one. Moreover, a duplicate search will open the discussion for the initial inclusion of papers and thus reducing the risk of not including relevant papers in the initial searching process.

The number of electronic databases searched varied across reviews (Table 2). In the present sample, 100 SRs (68%) performed the search in at least two electronic databases, a recommendation suggested by AMSTAR (Shea et al. 2007). The most searched database was Medline, followed by EMBASE and CENTRAL. Medline is one of the largest databases on health issues, and it makes the study abstracts freely available to the reader. These factors might have contributed to the popularity of this database. Most SRs reported the interface used to access the main databases (e.g., PubMed or OVID, in the case of Medline). This aspect is particularly important because these interfaces may generate heterogeneous results in terms of precision (De Groote 2000). Hence, it is important that the reader receives this information in case he or she needs to repeat or audit the results.

We included only SRs with meta-analyses on interventions because such SRs provide quantitative measurements and report the magnitude and direction of the treatment effect, which will guide the decision-making process. As the authors of certain included reviews had no stated intention to perform a systematic review, some may feel that these reviews are narrative reviews, instead of SRs. Nevertheless, we assume that, to provide reliable estimates, a meta-analysis should be based on the most representative sample of studies. In this case, a comprehensive and sensitive literature search strategy seems to be a “sine qua non” condition to capture all potential information to answer the research question. We may ask what the validity is of performing a meta-analysis based on information derived from a potentially biased narrative review.

Cochrane SRs have been proposed to have greater methodological rigour than paper-based SRs (Jadad et al. 1998, Delaney et al. 2007). Therefore, we compared a sample of Cochrane SRs with paper-based SRs. The search strategies in the Cochrane reviews had better standards than the paper-based reviews in seven of the eight topics assessed (Table 3). The topic “literature search in duplicate” was never reported, probably due to the Cochrane logistics in providing customized search strategies, from expert librarians to Cochrane authors. Regardless, information concerning whether the search strategies were performed in duplicate should be fully reported to the reader.

Strengths and limitations of the study

To the best of our knowledge, this study is one of the few reports assessing the standards of search strategies in a substantial sample of SRs with meta-analyses in dentistry. Two other studies evaluated the search and selection methods used in SRs: one used a limited sample of SRs published in the orthodontics field (Flores-Mir et al. 2006), and the other included SRs in several dental fields (Major et al. 2006). The authors of the latter study evaluated SRs published between 2000 and 2005 and also found that the strategies for searching and selecting studies need to be improved, although some improvements were reported in the sample of SRs published in 2005.

The sample of studies selected in this study is representative because it comprises virtually all SRs with meta-analyses published in the fields of periodontology and implant dentistry. Although the included SRs were published in English only, most of the reviews were published in dental journals with high impact factors, which normally have rigid criteria for publication. The impact factor has been suggested to be an indicator of quality for medical journals (Saha et al. 2003). One might consider the inclusion of SRs published in English only as a limitation of this study. Nevertheless, this study was designed to generate an overview of SRs on interventions in which meta-analytic findings might be influenced by language bias, instead of providing any meta-analytic estimate per se.

The small number of Cochrane reviews compared with paper-based reviews is a limitation of this study. We did not include other Cochrane reviews due to the strict guidelines of the inclusion and exclusion criteria. Therefore, the finding that the Cochrane SRs had more comprehensive search strategies than paper-based SRs should be considered with caution.

Recommendations for searching evidence

Authors of SRs should follow the suggested guidelines below, to develop a comprehensive and sensitive literature search strategy and to reduce the risk of reporting bias:

  • The initial search should be performed by at least two authors, independently and in duplicate. Ideally, one of the searchers should be a librarian or an expert searcher (Relevo & Balshem 2011).
  • The search strategy should at least include two electronic databases to increase sensitivity, and the hand searching of the reference lists of included papers should be conducted.
  • Grey literature should be sought, particularly on the specialized web pages for this type of evidence. Conference abstracts should be searched, and the authors of these abstracts should be contacted to obtain more detailed information (Hopewell et al. 2007). Dental/medical manufacturing companies may also need to be contacted to request potentially unpublished data (Relevo & Balshem 2011).
  • No language restriction should be used in the search strategies, mainly in SRs on interventions. Authors of SRs should declare how they dealt with the translation of articles published in non-English languages.
  • To improve the transparency and quality of SRs published in dental research, each dental journal may need to have an editorial board member that has the required expertise to review SRs. Such a member may require a detailed report of the whole process of conducting the SR, including the literature search process.

Clinical Relevance

Scientific rationale for the study: Use of a comprehensive search strategy in SRs with meta-analyses is pivotal to avoid publication bias and to generate reliable results. An assessment of search strategies in SRs has never been reported in periodontology and implant dentistry.

Principal findings: In contrast to Cochrane SRs, paper-based SRs do not usually compliment the search of electronic databases with a grey literature search and contact with authors/manufacturers. Moreover, many paper-based SRs limit their searches to publications in English.

Practical implications: The present findings may guide authors of SRs to develop more comprehensive search strategies to generate less biased meta-analytic treatment effect estimates.