Conflict of interest and source of funding statement
Search strategies in systematic reviews in periodontology and implant dentistry
Article first published online: 3 JUL 2013
© 2013 John Wiley & Sons A/S. Published by John Wiley & Sons Ltd
Journal of Clinical Periodontology
Volume 40, Issue 9, pages 883–888, September 2013
How to Cite
Search strategies in systematic reviews in periodontology and implant dentistry. J Clin Periodontol2013; 40: 883–888 doi: 10.1111/jcpe.12132., , .
The authors declare they have no conflicts of interest.
The first author is the Colgate Senior Lecturer at the University of Otago. This position is partially supported by an unrestricted grant from Colgate-Palmolive New Zealand to the University of Otago. The authors received no special funding for conducting this study.
- Issue published online: 6 AUG 2013
- Article first published online: 3 JUL 2013
- Accepted manuscript online: 10 JUN 2013 11:07AM EST
- Manuscript Accepted: 4 JUN 2013
- publication bias;
- search strategy;
- systematic review
To perform an overview of literature search strategies in systematic reviews (SRs) published in periodontology and implant dentistry.
Materials and Methods
Two electronic databases (PubMed and Cochrane Database of SRs) were searched, independently and in duplicate, for SRs with meta-analyses on interventions, with the last search performed on 11 November 2012. Manual searches of the reference lists of included SRs and 10 specialty dental journals were conducted. Methodological issues of the search strategies of included SRs were assessed with Cochrane collaboration guidelines and AMSTAR recommendations. The search strategies employed in Cochrane and paper-based SRs were compared.
A total of 146 SRs with meta-analyses were included, including 19 Cochrane and 127 paper-based SRs. Some issues, such as “the use of keywords,” were reported in most of the SRs (86%). Other issues, such as “search of grey literature” and “language restriction,” were not fully reported (34% and 50% respectively). The quality of search strategy reporting in Cochrane SRs was better than that of paper-based SRs for seven of the eight criteria assessed.
There is room for improving the quality of reporting of search strategies in SRs in periodontology and implant dentistry, particularly in SRs published in paper-based journals.
Systematic reviews (SRs) of the dental literature have frequently been published in recent decades. The main advantage of SRs over narrative reviews is that the former may provide less biased estimates on treatment approaches. Nevertheless, for this assumption to be considered true, several steps related to the planning and conduction of SRs should be observed (Smith et al. 2011). One of these crucial steps is the strategy for searching the evidence, which should be searched in a highly sensitive fashion, to answer the proposed research question(s) by allowing the inclusion of all relevant information and reducing the risk of biased results (Robinson & Dickersin 2002).
Publication bias may occur when the literature used to support an estimate is not representative of the true available evidence (Easterbrook et al. 1991). For example, publication bias may exist when some relevant literature is not published because of a conflict of interest on the results of therapies. Depending on the level of publication bias, the presented estimates may not represent the numbers reported, completely misleading the guidance of therapies. Some data suggest that published literature may overestimate treatment effects compared with “grey literature” (i.e., studies that have not been formally published) (Hopewell et al. 2007).
Guidelines for search strategies have been developed over the years (Haynes et al. 1994, 2005, Montori et al. 2005, Wong et al. 2006), and some standard criteria should be respected to generate an optimal search strategy. For instance, the AMSTAR checklist for the methodological assessment of SRs and meta-analyses recommends the use of at least two electronic databases for a comprehensive literature search (Shea et al. 2007). Moreover, the literature search should involve hand-searching strategies to complement the electronic search (Shea et al. 2007). As more SRs are becoming available, it would be interesting to assess the standards of the different search strategies used in those reviews.
The primary objective of this study was to assess the standard literature search strategies in SRs with meta-analyses published in the fields of periodontology and implant dentistry. A secondary objective was to explore the potential differences between Cochrane and paper-based SRs in terms of literature search strategies.
Materials and Methods
Inclusion and exclusion criteria
SRs with meta-analyses on interventions in periodontology and implant dentistry were included. To be included in the study, the SRs should involve at least two therapeutic approaches to be compared by meta-analysis. SRs with diagnostic and prevalence studies were excluded from this study, as were SRs with only animal studies or in vitro studies.
This study is part of a project that evaluates the methodological quality of SRs with meta-analyses published in the fields of periodontology and implant dentistry. Although the previous project (Faggion et al. 2013) was focused only on the quality of reporting in the abstracts of SRs, the present project is focused on the search strategies described in the materials and methods section. The search for SRs was performed in the PubMed and Cochrane databases. It was limited to SRs published in the English language, with the last search performed in November 2012. The literature search strategy used is depicted in Table 1.
|Electronic databases search||Dental specialty journals||Hand search|
1) Periodontal debridementa
2) Root planninga
3) Furcation defectsa
4) Gingival recessiona
5) Root coverage
6) Sinus floor augmentationa
7) Sinus floor elevation
8) Peri-implantitisa OR periimplantitis
9) Periodontal diseasesa
11) Subgingival curettagea
12) Guided tissue regenerationa
13) Dental implantsa
14) Dental implant
15) #1 OR #2 OR #3 OR #4 OR #5 OR #6 OR #7 OR #8 OR #9 OR #10 OR #11 OR #12 OR #13 OR #14
17) #15 AND #16 (Filters: meta-analysis; review)
|Issues from November 1990 to November 2012 (Clinical Implant Dentistry and Related Research, Journal of Clinical Periodontology, Journal of Periodontology, Clinical Oral Implants Research, Journal of Periodontal Research, International Journal of Oral and Maxillofacial Implants, European Journal of Oral Implantology (2008–2012), Journal of Oral Implantology, International Journal of Periodontics and Restorative Dentistry, and Implant Dentistry)||Reference lists of included articles|
The literature search was performed independently and in duplicate by two authors. Any disagreements in the final selection of the SRs were resolved by discussion. In the case of persistent disagreement, a third author was consulted until consensus was achieved.
Rationale for the assessment of search strategies
Eight questions related to the search strategy of the selected SRs were assessed. The questions were based on the recommendations of the Cochrane Handbook (Lefebvre et al. 2011) and a validated checklist for the methodological assessment of SRs and meta-analyses (AMSTAR) (Shea et al. 2007):
- Did the authors of SRs explicitly report the terms used to search for primary studies?
- Did the authors search in at least two electronic databases?
- Did the authors report any interface to search in the electronic databases?
- Did the authors search the grey literature, namely, information that is not published in easily accessible journals or databases, such as conference proceedings that include the abstracts of research presented at conferences or unpublished theses (http://www.cochrane.org/glossary)?
- Did the authors perform “hand-searching,” which includes searching reviews, textbooks, reviewing the references of the selected studies, and contacting specialized registers or experts in the particular field of study?
- Did the authors conduct the search in duplicate?
- Did the search strategy include any language restriction; was the search strategy limited to one or two languages and, hence, insufficiently comprehensive to cover all languages?
- Did the authors make any attempt to contact the authors of primary studies or manufacturing companies to obtain further relevant information?
The selected questions were previously pilot tested on five SRs by one of the authors (CMF). To increase the precision of data extraction, two reviewers (CMF and SP) assessed the search strategies by retrieving the data of 25 SRs that were not included in the present sample (from fields other than periodontology and implant dentistry). Five rounds of assessment were carried out, with five SRs being assessed in each round. Between rounds, the results were intensively discussed between reviewers to improve the homogeneity of assessment.
All questions were answered with YES (authors adequately addressed the question), NO (authors inadequately addressed the question), and UNCLEAR (authors did not provide information to answer the question). Other relevant data, such as the type and the number of searched databases, were also descriptively reported. The assessment was performed independently and in duplicate by two authors (CMF and SP). All data were extracted and summarized into a standardized table. Any disagreements in the data extraction process were resolved by discussion. In the case of persistent disagreement, a third author was consulted until consensus was achieved. The full dataset was checked for precision by a third reviewer (MA).
Comparison of Cochrane and paper-based reviews
The literature search strategies performed in Cochrane reviews were compared with those performed in paper-based journals. The frequency of questions answered with YES in both groups of reviews was measured by Fisher's exact test. Differences with p < 0.05 were considered statistically significant. All data were analysed with the Statistical Package for Social Sciences (SPSS 17.0, SPSS Inc., Chicago, IL, USA).
A total of 146 SRs with meta-analyses had their literature search strategies scrutinized (Data S1). Most of the included SRs (86% of the sample) reported the use of relevant keywords in their literature search. The searching processes used multiple databases, of which Medline was the most prevalent (Table 2). Authors of 40 SRs (27%) used one electronic database in their search, 30 SRs (21%) used two electronic databases, and almost half of the selected SRs (48%) searched three or more electronic databases. Six SRs (4%) did not report the names and number of electronic databases. The use of some kind of interface for searching electronic databases was reported in 83 SRs (57%).
|Electronic databases searched|
|1) Medline: 126 (86)|
|2) EMBASE: 56 (38)|
|3) CENTRAL: 54 (37)|
|4) Cochrane Oral Health Group Trial Register: 38 (25)|
|5) Cochrane Library: 14 (10)|
|6) Cochrane Database of SRs: 13 (9)|
|7) CINAHL: 12 (8)|
|8) Web of knowledge/science: 12 (8)|
|9) Cochrane Controlled Trial Register: 9 (6)|
|10) DARE: 9 (6)|
|11) LILACS: 7 (5)|
|12) Science Direct: 6 (4)|
|13) SCOPUS: 5 (3)|
|14) Other databasesa: 22 (15)|
|15) Not reported: 6 (4)|
A search of the grey literature was reported in 50 SRs (34%). Hand searching was performed by the authors of 121 SRs (84%). Hand searches of scientific journals and of the references of relevant papers were reported in 80 SRs (55%) and 106 SRs (73%) respectively. The literature search was performed in duplicate in 11 SRs (8%). Language restrictions were used in the search strategies of 73 SRs (50%), with 62 SRs (42%) being focused on English. However, 29 SRs (20%) did not report any information on language restriction. The authors of 42 SRs (29%) attempted to contact authors or manufacturers to retrieve unpublished data or to obtain further information on the primary studies. Three SRs (2%) explicitly reported that there was no attempt to contact authors/manufacturers to obtain information on published or unpublished studies, and 101 SRs (69%) did not provide any information on the attempt to contact authors/manufacturers.
Cochrane versus paper-based SRs
The search strategies of Cochrane SRs met more criteria than the search strategies of paper-based SRs (Table 3). Seven of eight questions were completely answered in all Cochrane SRs (n = 19). However, one item, related to the duplication of the literature search, was adopted more often in paper-based SRs than in Cochrane SRs.
|Items||Cochrane (%)||Paper based (%)||df||p-valuea|
|Keywords reported (%)||19 (100)||107 (73)||1||0.076|
|Search in at least two electronic databasesc||19 (100)||81 (55)||1||0.001|
|Interface reported (%)||16 (84)||64 (44)||1||0.006|
|Grey literature performed (%)||19 (100)||31 (21)||1||<0.0001|
|Hand search performed (%)||19 (100)||110 (75)||1||0.13|
|Literature search in duplicate (%)||0b (0)||11 (8)||1||0.36|
|No language restriction (%)||19 (100)||24 (16)||1||<0.0001|
|Contact with authors (%)||19 (100)||23 (16)||1||<0.0001|
Our findings demonstrate that there is room for improvement in the search strategies of SRs with meta-analyses published in periodontology and implant dentistry. This is particularly the case for SRs published in paper-based journals. The lack of a comprehensive search strategy may introduce bias in the estimates of treatment effect and, consequently, may jeopardize the development of appropriate clinical practice guidance.
Reporting bias happens when the dissemination of research findings is influenced by the nature and the direction of results (Sterne et al. 2011). Due to the increasingly large number of SRs that are being published, reporting bias is an important issue in dentistry. A series of reporting bias is recognized in the literature (Table 4). With regard to search strategy, the most common types of bias are often related to publication (Easterbrook et al. 1991), location (Pittler et al. 2000), and language biases (Pham et al. 2005).
|Type of reporting bias||Definition|
|Publication bias||The publication or non-publication of research findings, depending on the nature and direction of the results|
|Time lag bias||The rapid or delayed publication of research findings, depending on the nature and direction of the results|
|Multiple (duplicate) publication bias||The multiple or singular publication of research findings, depending on the nature and direction of the results|
|Location bias||The publication of research findings in journals with different ease of access or levels of indexing in standard databases, depending on the nature and direction of results|
|Citation bias||The citation or non-citation of research findings, depending on the nature and direction of the results|
|Language bias||The publication of research findings in a particular language, depending on the nature and direction of the results|
|Outcome reporting bias||The selective reporting of some outcomes but not others, depending on the nature and direction of the results|
Evidence suggests that trials with positive findings are published faster and more frequently than trials with negative results (Hopewell et al. 2009). A negative: positive trial publication proportion of almost 1:2 can be assumed. In other words, for each negative trial published, there will be two positive ones (Hopewell et al. 2009). Moreover, unpublished trials seem to present less positive treatment effects than published trials (Hopewell et al. 2009). Studies with more positive results are frequently published in journals with different hierarchical rankings (Kanaan et al. 2011), generating some sort of “location” bias. Although some data suggest that language restriction to only English language might have a limited effect on meta-analytic estimates (Jüni et al. 2002, Morrison et al. 2012), search strategies should not have any language restriction to increase the generalizability and applicability of the findings of therapeutic interventions (Jüni et al. 2002). Therefore, it is crucial that authors of SRs develop a sensitive search strategy that will include electronic databases, grey literature (Hopewell et al. 2007), and a “snowballing” hand-searching strategy, in the form of searching the reference lists of relevant papers (Greenhalgh & Peacock 2005).
Almost 30% of selected paper-based SRs did not report any keywords in the literature search strategies. The keywords used to perform the search and their combination with Boolean operators should be reported to allow the auditing and replication of the results. The careful use of a wide variety of synonyms (with both free text and controlled vocabulary terms) together with Boolean operators generates a very effective search strategy. The full description of the literature search strategy is important to the continuous development of search strategies for reviews on related topics (Yoshii et al. 2009). Furthermore, the initial search should be performed independently and in duplicate. Although this recommendation is not a standard criterion for the Cochrane and AMSTAR guidelines, the idea is to generate a more efficient search because two independent reviewers are more likely to retrieve more relevant data than a single one. Moreover, a duplicate search will open the discussion for the initial inclusion of papers and thus reducing the risk of not including relevant papers in the initial searching process.
The number of electronic databases searched varied across reviews (Table 2). In the present sample, 100 SRs (68%) performed the search in at least two electronic databases, a recommendation suggested by AMSTAR (Shea et al. 2007). The most searched database was Medline, followed by EMBASE and CENTRAL. Medline is one of the largest databases on health issues, and it makes the study abstracts freely available to the reader. These factors might have contributed to the popularity of this database. Most SRs reported the interface used to access the main databases (e.g., PubMed or OVID, in the case of Medline). This aspect is particularly important because these interfaces may generate heterogeneous results in terms of precision (De Groote 2000). Hence, it is important that the reader receives this information in case he or she needs to repeat or audit the results.
We included only SRs with meta-analyses on interventions because such SRs provide quantitative measurements and report the magnitude and direction of the treatment effect, which will guide the decision-making process. As the authors of certain included reviews had no stated intention to perform a systematic review, some may feel that these reviews are narrative reviews, instead of SRs. Nevertheless, we assume that, to provide reliable estimates, a meta-analysis should be based on the most representative sample of studies. In this case, a comprehensive and sensitive literature search strategy seems to be a “sine qua non” condition to capture all potential information to answer the research question. We may ask what the validity is of performing a meta-analysis based on information derived from a potentially biased narrative review.
Cochrane SRs have been proposed to have greater methodological rigour than paper-based SRs (Jadad et al. 1998, Delaney et al. 2007). Therefore, we compared a sample of Cochrane SRs with paper-based SRs. The search strategies in the Cochrane reviews had better standards than the paper-based reviews in seven of the eight topics assessed (Table 3). The topic “literature search in duplicate” was never reported, probably due to the Cochrane logistics in providing customized search strategies, from expert librarians to Cochrane authors. Regardless, information concerning whether the search strategies were performed in duplicate should be fully reported to the reader.
Strengths and limitations of the study
To the best of our knowledge, this study is one of the few reports assessing the standards of search strategies in a substantial sample of SRs with meta-analyses in dentistry. Two other studies evaluated the search and selection methods used in SRs: one used a limited sample of SRs published in the orthodontics field (Flores-Mir et al. 2006), and the other included SRs in several dental fields (Major et al. 2006). The authors of the latter study evaluated SRs published between 2000 and 2005 and also found that the strategies for searching and selecting studies need to be improved, although some improvements were reported in the sample of SRs published in 2005.
The sample of studies selected in this study is representative because it comprises virtually all SRs with meta-analyses published in the fields of periodontology and implant dentistry. Although the included SRs were published in English only, most of the reviews were published in dental journals with high impact factors, which normally have rigid criteria for publication. The impact factor has been suggested to be an indicator of quality for medical journals (Saha et al. 2003). One might consider the inclusion of SRs published in English only as a limitation of this study. Nevertheless, this study was designed to generate an overview of SRs on interventions in which meta-analytic findings might be influenced by language bias, instead of providing any meta-analytic estimate per se.
The small number of Cochrane reviews compared with paper-based reviews is a limitation of this study. We did not include other Cochrane reviews due to the strict guidelines of the inclusion and exclusion criteria. Therefore, the finding that the Cochrane SRs had more comprehensive search strategies than paper-based SRs should be considered with caution.
Recommendations for searching evidence
Authors of SRs should follow the suggested guidelines below, to develop a comprehensive and sensitive literature search strategy and to reduce the risk of reporting bias:
- The initial search should be performed by at least two authors, independently and in duplicate. Ideally, one of the searchers should be a librarian or an expert searcher (Relevo & Balshem 2011).
- The search strategy should at least include two electronic databases to increase sensitivity, and the hand searching of the reference lists of included papers should be conducted.
- Grey literature should be sought, particularly on the specialized web pages for this type of evidence. Conference abstracts should be searched, and the authors of these abstracts should be contacted to obtain more detailed information (Hopewell et al. 2007). Dental/medical manufacturing companies may also need to be contacted to request potentially unpublished data (Relevo & Balshem 2011).
- No language restriction should be used in the search strategies, mainly in SRs on interventions. Authors of SRs should declare how they dealt with the translation of articles published in non-English languages.
- To improve the transparency and quality of SRs published in dental research, each dental journal may need to have an editorial board member that has the required expertise to review SRs. Such a member may require a detailed report of the whole process of conducting the SR, including the literature search process.
- 2000) PubMed, Internet Grateful Med, and Ovid: a comparison of three MEDLINE Internet interfaces. Medical Reference Services Quarterly 19, 1–13. (
- 2007) The quality of reports of critical care meta-analyses in the Cochrane Database of Systematic Reviews: an independent appraisal. Critical Care Medicine 35, 589–594. , , , , & (
- 1991) Publication bias in clinical research. Lancet 337, 867–872. , , & (
- 2006) Search and selection methodology of systematic reviews in orthodontics (2000–2004). American Journal of Orthodontics and Dentofacial Orthopedics 130, 214–217. , & (
- 2005) Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. British Medical Journal 331, 1064–1065. & (
- 2005) Optimal search strategies for retrieving scientifically strong studies of treatment from Medline: analytical survey. British Medical Journal 330, 1179. , , , , & (
- 1994) Developing optimal search strategies for detecting clinically sound studies in MEDLINE. Journal of the American Medical Informatics Association 1, 447–458. , , , & (
- 2009) Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database of Systematic Reviews, 1, MR000006. , , , & (
- 2007) Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews 2, MR000010. , , & (
- 1998) Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. Journal of the American Medical Association 280, 278–280. , , , , , & (
- 2013) Quality of reporting in abstracts of systematic reviews with meta-analyses in Periodontology and Implant Dentistry. Journal of Periodontal Research doi: 10.1111/jre.12092. , , & (
- 2002) Direction and impact of language bias in meta-analyses of controlled trials: empirical study. International Journal of Epidemiology 31, 115–123. , , , & (
- 2011) The value of lesser-impact-factor surgical journals as a source of negative and inconclusive outcomes reporting. Annals of Surgery 253, 619–623. , , , , , & (
- 2011) Chapter 6: searching for studies. In: Higgins JPT, Green S (eds). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 (updated March 2011), The Cochrane Collaboration: Available from www.cochrane-handbook.org. Accessed at 01 March 2013 , & (
- 2006) An evaluation of search and selection methods used in dental systematic reviews published in English. Journal of the American Dental Association 137, 1252–1257. , & (
- 2005) Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. British Medical Journal 330, 68. , , , & (
- 2012) The effect of English-language restriction on systematic review-based meta-analyses: a systematic review of empirical studies. International Journal of Technology Assessment in Health Care 28, 138–144. , , , , , , , , & (
- 2005) Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. Journal of Clinical Epidemiology 58, 769–776. , , & (
- 2000) Location bias in controlled clinical trials of complementary/alternative therapies. Journal of Clinical Epidemiology 53, 485–489. , , & (
- 2011) Finding evidence for comparing medical interventions. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville, MD: Agency for Healthcare Research and Quality (US); 2008. Available at: http://www.ncbi.nlm.nih.gov/books/NBK53479/ (accessed on 15 February 2013). & (
- 2002) Development of a highly sensitive search strategy for the retrieval of reports of controlled trials using PubMed. International Journal of Epidemiology 31, 150–153. & (
- 2003) Impact factor: a valid measure of journal quality? Journal of Medical Library Association 91, 42–46. , & (
- 2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7, 10. , , , , , , , , & (
- 2011) Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology 11, 15. , , & (
- 2011) Chapter 10: addressing reporting biases. In: Higginsa, JPT & Green, S (eds). Cochrane Handbook for Systematic Reviews of Intervention. Version 5.1.0 (updated March 2011). The Cochrane Collaboration, Available from www.cochrane-handbook.org. Accessed at 01 March 2013. , & (
- 2006) Developing optimal search strategies for detecting clinically sound treatment studies in EMBASE. Journal of the Medical Library Association 94, 41–47. , & (
- 2009) Analysis of the reporting of search strategies in Cochrane systematic reviews. KE. Journal of the Medical Library Association 97, 21–29. , , , & (
Scientific rationale for the study: Use of a comprehensive search strategy in SRs with meta-analyses is pivotal to avoid publication bias and to generate reliable results. An assessment of search strategies in SRs has never been reported in periodontology and implant dentistry.
Principal findings: In contrast to Cochrane SRs, paper-based SRs do not usually compliment the search of electronic databases with a grey literature search and contact with authors/manufacturers. Moreover, many paper-based SRs limit their searches to publications in English.
Practical implications: The present findings may guide authors of SRs to develop more comprehensive search strategies to generate less biased meta-analytic treatment effect estimates.
|jcpe12132-sup-0001-SupportingInformation.docx||Word document||32K||Data S1. Included systematic reviews with meta-analyses.|
Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.