Description of the problem or issue
Systematic reviews of the literature have become vital decision-making aids for clinicians, researchers, policy makers and patients (Gough 2012a; Ligthelm 2007; Manchikanti 2009; Wilczynski 2007). They provide a formal synthesis of a large and ever increasing body of research literature. Systematic reviews typically address specified questions and can, as a result, help to (1) establish links between available information and potentially beneficial (or harmful) interventions, (2) compare and contrast conflicting results, and (3) identify gaps in medical knowledge (Manchikanti 2009; Wilczynski 2007).
In order to achieve their objectives, systematic reviews rely on the use of explicit strategies to search for relevant evidence and on methodological criteria against which to evaluate this evidence (Wilczynski 2007). When searching for relevant studies, researchers (including those conducting a systematic review) can make use of structured search strategies that can facilitate this process (Wilczynski 2007). However, searching for specific studies on a given topic can be challenging, particularly when searching for a specific study design. Wilczynski 2007 attributes this phenomenon to the spread of relevant papers across numerous scientific journals, the inherent limits in indexing, and the lack of search skills amongst database users.
Systematic reviews vary in many respects, including the types of research questions they ask (Gough 2012b). The review question will, in turn, determine the methods and the types of data that are most appropriate to answer the question itself (Gough 2012a; Gough 2012b). Systematic reviews assessing the effectiveness of interventions are typically best answered by data from randomised controlled trials (RCTs) (Glasziou 2001; Ligthelm 2007). Systematic reviews asking questions of aetiology and risk, prediction and prognosis, or frequency of rare outcomes or complications are usually best answered by data from observational studies (Furlan 2006; Glasziou 2001).
However, there are circumstances in which evidence from observational studies is needed in order to assess the effectiveness of interventions or safety outcomes: when data from RCTs are insufficient or when the findings of RCTs appear to be contradictory (Fraser 2006; Furlan 2006; Manchikanti 2009). Improvements in observational-study methods and statistical analyses have made observational studies an important source of evidence, particularly with regards to the side effects or adverse events associated with health interventions (Ligthelm 2007; Manchikanti 2009; Wieland 2005). As argued by Ligthelm 2007, observational studies can complement data from RCTs in order to provide an evidence base for clinical decision-making or for policy-making. While searching for RCTs has become a relatively simple task since the 1990s (Lefebvre 2013), limitations in indexing practices can make the identification of observational studies particularly challenging.
Description of the methods being investigated
The use of a search strategy in health-related bibliographic databases is the method required by The Cochrane Collaboration and other evidence-based healthcare organisations to identify relevant study reports for a systematic review. MEDLINE and EMBASE are the principal databases of biomedical scientific literature. Together, they contain abstracts for many millions of published articles in this field, the extent depending on the topics of interest. Records in these databases can be searched electronically for words in the title or abstract, and for assigned index terms. The latter are controlled vocabulary terms that indexers assign to each record after reviewing them (Higgins 2011). Searching these two databases is usually the minimum requirement for anyone wishing to conduct a systematic review; although they often overlap (between 10% and 87% of the indexed records) depending on the topic under consideration (Manchikanti 2009).
Search strategies can be complemented by including search filters. These refer to a predefined combination of terms that have been designed to retrieve a selection of records on the basis of a particular concept (CRD 2012). Filters used to retrieve records on the basis of their study design are often referred to as methodological filters. The combination of search filters with content terms will in turn determine the performance properties of a search strategy, namely, its sensitivity, precision (or positive predictive value (PPV)) and specificity (Doust 2005; Fraser 2006).
How these methods might work
Evaluating a search strategy relies on the availability of a reference standard against which to compare its performance, in this case the included studies of a systematic review (Sampson 2006). By comparing the records retrieved by a search strategy with a methodological filter with those retrieved by a search strategy without a filter, it is possible to calculate the performance properties of the filter. In the context of systematic reviews, the sensitivity and precision are the most relevant performance properties of a search filter (Sampson 2006). Sensitivity, also referred to as recall, is defined as the number of relevant records in a database identified by the search strategy as a proportion of the total number of relevant records in the database (Sampson 2006). The precision of a search strategy refers to the number of relevant reports identified by the search strategy as a proportion of the total number of records yielded by the search (Doust 2005; Furlan 2006; Sampson 2006).
Review authors should aim for search strategies that have both high sensitivity and high precision (Sampson 2006). In addition, authors should identify and include all possibly relevant reports (high sensitivity) in order to reduce the likelihood of bias in their systematic reviews, and to reduce random error in meta-analyses (Edwards 2002; Robinson 2002). At the same time, they should attempt to retrieve as few irrelevant records as possible (high precision) in order to minimise the burden on the resources available (Gough 2012a; Gough 2012b; Sampson 2006). However, in reality there are trade-offs between these two properties.
An ideal methodological filter could help review authors to achieve this balance by maintaining the sensitivity of a content-only search strategy while increasing its precision (Doust 2005; Fraser 2006). Applying methodological filters to a search strategy could in theory limit the number of records retrieved in a search, while avoiding the exclusion of relevant papers. At the same time, a methodological filter could limit the number of records that need to be evaluated for inclusion in the review. However, by reducing the number of hits methodological filters could increase the likelihood of missing relevant records that would otherwise be included in a systematic review.
Why it is important to do this review
Searching health-related literature by study design can identify the study type of primary interest in an efficient and time-saving manner (Littleton 2004). Specifying the types of study design is relatively easy for RCTs owing to initiatives such as the Cochrane Central Register of Controlled Trials (CENTRAL) database of trials, the introduction of the Consolidated Standards of Reporting Trials (CONSORT) statement (which is linked to better reporting of RCTs in the titles and abstracts), appropriate indexing terms in MEDLINE and EMBASE, and the publication of highly sensitive filters (Fraser 2006 ; Lefebvre 2013).
The situation is different when dealing with observational studies. Indexing using Medical Subject Headings (MeSH) intervention terms is limited; and when used, these terms are usually applied inconsistently (Fraser 2006; Wieland 2005). Despite the introduction of statements such as the Strengthening the Reporting of Observational studies in Epidemiology (STROBE) guidelines, reporting of methodological detail is still poor in observational studies, contributing to the problems in indexing and searching (Fraser 2006; Manchikanti 2009). The lack of appropriate search terms for observational studies has greatly contributed to the exclusion of methodological components from search strategies (Fraser 2006). As a consequence of this, searches often yield a large number of irrelevant records, leading to the inefficient use of resources and the time needed to complete a review increases (Doust 2005). For this reason, it is necessary to explore the literature for recent developments in search approaches that can lead to the efficient identification of observational studies.
In addition, there appear to be no agreed universal standard criteria for the creation of a search strategy (Lemeshow 2005); although guidelines are available to anyone thinking of undertaking a systematic review, particularly in relation to RCTs. In their Handbook for Systematic Reviews of Interventions (Higgins 2011), The Cochrane Collaboration presents their Highly Sensitive Search Strategy (HSSS) for identifying RCTs in MEDLINE. Similarly, The Cochrane Collaboration is working towards the creation of an objectively derived HSSS for identifying RCTs in EMBASE. The work of another group, the InterTASC Information Specialists' Sub-Group (ISSG), focuses on the identification, assessment and testing of search filters that are intended to select studies depending on their design or focus (CRD 2012). They offer various resources related to study designs such as RCTs, observational studies, diagnostic studies, and economic evaluations, among others.
Attempts have been made to appraise the evidence for search filters. A recent Cochrane systematic review (Leeflang 2013) evaluated the performance of search filters designed to retrieve diagnostic test accuracy (DTA) studies in MEDLINE and EMBASE. Similarly, McDonald 2013 attempted to assess search strategies to identify RCTs in MEDLINE (this protocol has now been withdrawn). However, we could not identify a similar protocol or review on search strategies for observational studies. A systematic review in this area could help to identify the specific features of a search strategy that can improve the identification of observational studies. As a result, this work could contribute to the creation of evidence-based standards for the formulation of search strategies.