Literature searching for social science systematic reviews: consideration of a range of search techniques

Authors

  • Diana Papaioannou,

    1. Information Resources, Health Economics and Decision Science, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
    Search for more papers by this author
  • Anthea Sutton,

    1. Information Resources, Health Economics and Decision Science, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
    Search for more papers by this author
  • Christopher Carroll,

    1. Information Resources, Health Economics and Decision Science, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
    Search for more papers by this author
  • Andrew Booth,

    1. Information Resources, Health Economics and Decision Science, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
    Search for more papers by this author
  • Ruth Wong

    1. Information Resources, Health Economics and Decision Science, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
    Search for more papers by this author

D Papaioannou, Information Specialist, Information Resources Section, School of Health and Related Research (ScHARR), University of Sheffield, Regent Court, 30 Regent Street, Sheffield S1 4DA, UK. E-mail: d.papaioannou@sheffield.ac.uk

Abstract

Background:  Literature for a systematic review on the student experience of e-learning is located across a range of subject areas including health, education, social science, library and information science.

Objectives:  To assess the merits and shortcomings of using different search techniques in retrieval of evidence in the social science literature.

Methods:  A conventional subject search was undertaken as the principal method of identifying the literature for the review. Four supplementary search methods were used including citation searching, reference list checking, contact with experts and pearl growing.

Results:  The conventional subject search identified 30 of 41 included references; retrieved from 10 different databases. References were missed by this method and a further 11 references were identified via citation searching, reference list checking and contact with experts. Pearl growing was suspended as the nominated pearls were dispersed across numerous databases, with no single database indexing more than four pearls.

Conclusions:  Searching within the social sciences literature requires careful consideration. Conventional subject searching identified the majority of references, but additional search techniques were essential and located further high quality references.

Introduction

The UK Higher Education Academy funded a project to investigate ‘enhancing the student experience of workplace-based e-learning’. This project involved a qualitative systematic review component and the creation of a best practice framework. The systematic review process required identifying relevant literature concerned with the student experience of e-learning, specifically trying to identify particular methods that are more favourable to students, for example, scenario-based learning or quizzes. The population being investigated were students in employment within any work sector who were involved in studying via e-learning. As with any systematic review, subject searches needed to be thorough, sensitive and transparent.1 An additional complication in searching for this particular systematic review was the diffusion of literature across different subject domains.

In health, specifically within the process for Health Technology Assessment, ‘gold standard’ systematic review search techniques are largely based on those published in the Cochrane Handbook.2 The Cochrane Handbook for Systematic Reviews of Interventions1 emphasises the need for searches to maximise recall rather than be high in precision. The objective is to ensure all relevant articles are located, therefore significantly reducing the introduction of bias into the systematic review.

The need for thorough in-depth searching is evident. Brettle et al.3 conducted three alternative search strategies to identify literature on health performance indicators. These were termed basic, intermediate and comprehensive; the first being the briefest and the last the most thorough. The authors conclude that the search strategy selected was determined by the requirements for the search results. Basic and intermediate searches are likely to miss several key papers but will suffice for quick clinical questions. However, more comprehensive search strategies are required for systematic reviews.

Seeking to minimise possible elements of bias, is both time and resource intensive. Indeed, the Cochrane Handbook acknowledges that ‘conducting a comprehensive, objective and reproducible search for studies can be the most time-consuming and challenging task in preparing a systematic review’.1 As a database is subject specific and only indexes a portion of the literature, more than one database should be searched. However, there may be some evidence that sources for a systematic review could be restricted. Royle and Waugh2 investigated whether searching a reduced number of databases for Health Technology Assessments was sufficient to find studies for clinical and cost effectiveness technology assessments. The authors found that little was gained in searching further databases beyond searching the Cochrane Library and four key healthcare databases (medline, embase and Science Citation Index, biosis: limited to meeting abstracts only).

Notwithstanding the evidence that sources for a systematic review could be restricted, the number of databases to search is largely determined by the nature of the topic area.4 Where there is input from multi-disciplinary staff (e.g. community-based interventions), it seems necessary to search a number of databases to reflect the wide spread of literature across different disciplines.

Stevinson and Lawlor5 investigated the benefit of searching additional databases (embase, cinahl and psycinfo) as well as subject-specific databases (cancerlit and sport discus) in a systematic review of exercise interventions for cancer patients. Three additional papers were located via cinahl, cancerlit and psycinfo that were not indexed on medline. A further six papers were retrieved by reference checking and one via contact with experts. Of these 10 additional papers that medline failed to retrieve, three were in fact indexed on medline but had not been retrieved by the search strategy. Similarly, searching at least six health and social science databases was deemed essential to identify literature for a systematic review of the efficacy of interventions in the rehabilitation of adults with severe mental illness;6 only 42% papers in this review were identified by a single database.

Cost and time are not the only considerations in systematic literature searching. Grayson and Gomersall7 discuss the various difficulties in ensuring that all the evidence for social science systematic reviews is found. To begin with, social science literature, like diffuse health topics, is diverse, incorporating a range of media such as books, practitioner journals, official publications, grey literature as well as peer-reviewed journals. It is very important to use a range of methods and sources for identifying social science literature for a systematic review, in addition to searching bibliographic databases.4,7 However, searching bibliographic databases is in itself challenging. There are an extensive number of databases each covering various sub-disciplines within the social sciences and each with variable search capabilities.6 Problems in terminology and indexing within databases make both searching with index and free-text terms problematic. Terms within social sciences are often ambiguous, poorly defined and constantly changing. Unfortunately, the use of controlled vocabularies and indexing is not applied across the social science databases with the same rigour as in medical databases.

For systematic reviews where literature is dispersed across a range of subject databases, it is possible that supplementary search techniques might prove more fruitful, particularly when concepts and themes are poorly defined, ambiguous or variability exists in their indexing. Furthermore, database coverage is not exhaustive. McDonald et al.8 found that 213 abstracting and indexing services indexed 52% of 977 psychiatry journals. Whilst biosis, embase, medline and psyclit indexed 90% of the 506 titles included in any of the abstracting and indexing services, this left a huge proportion of psychiatry literature difficult to access. Supplementary search techniques could provide a means of capturing this literature. Stevinson and Lawlor5 demonstrated that the use of reference list checking and contact with experts resulted in inclusion of seven additional papers for their systematic review; some of which were in fact indexed on medline but were not retrieved by the search strategy. Savoie et al.9 looked at the added value of extended systematic review search methods in identifying randomised controlled trials (RCTs) for two systematic review projects. Extended searches included specialised databases (e.g. trip), trial registries (e.g. National Research Register), the Web using search engines and meta-indexes, journal handsearching, reference list review and communication with experts. For the two projects, an additional 94 RCTs were identified via the extended methods. Seventy-four of these were indexed in medline but were not picked up by the main search. As well as searching a complement of six health and social science databases; citation searching, handsearching, grey literature and contact with experts enabled location of one-fifth of relevant papers in a systematic review for interventions in the rehabilitation of adults with severe mental illness.6

Research has also been undertaken into using other search techniques. Techniques such as traditional pearl growing (TPG) and comprehensive pearl growing (CPG) are believed to be advantageous when literature is dispersed across a range of subject areas and subject terms are variable or difficult to define.10 Pearl growing involves taking ‘gold standard paper(s)’ or ‘pearl(s)’ and using its characteristics such as index terms and keywords in an iterative process of searching. The process continues until the material found becomes less relevant. Searching is repeated throughout all relevant databases.10

Objectives

The literature in the subject area of e-learning is distributed across a range of sources in a variety of fields, including subjects such as education, library and information science, health, social sciences and psychology; posing problems in managing the systematic literature searching as one might find in diffuse health-related topics. This study examines the merits and shortcomings of different search techniques in the retrieval of evidence in the social science literature, i.e. a diffuse topic area. The quality of studies identified by the various methods is assessed to determine which method if any produced the better quality studies.

The search techniques include:

  • 1 conventional subject searching
  • 2 reference list checking
  • 3 contact with experts
  • 4 citation searching
  • 5 comprehensive pearl growing

Analysis will examine the result yields from each technique, with particular attention to the retrieval of unique references from techniques other than conventional subject searching for a systematic review.

Methods

In total, five techniques were employed to identify the literature for the systematic review.

Conventional subject searching

A full systematic search was conducted entailing the development of a search strategy around terms for e-learning (the intervention) and the workplace (the setting). Searches were carried out on 11 databases covering different disciplines, including education (eric, british education index), business (emerald management reviews), social sciences (ibss, social sciences citation index, assia), health (medline, psycinfo, cinahl), information technology (csa computer and information systems abstracts) and library and information science (lisa). Searches were restricted to English language publications and 1992 onwards (this was the date that saw increased use of the Web by Higher Education; Super JANET was rolled out in early 1990s).

Reference list checking

Reference lists from all included papers (41) identified by the various methods were checked. Any relevant references that had not been identified from the subject search were added to the overall search yield.

Contact with experts

At the project initiation, a working group of four e-learning experts all of whom had extensive knowledge in delivery of and research in e-learning, was established. Each was asked to identify key papers at the beginning of the study and after the literature searches were conducted.

Citation searching

Citation searching (CS) took place following the subject search for each paper identified for inclusion in the systematic review. Databases used for the CS were Google Scholar, science and social Science Citation Indexes (SSCI) and cinahl.

Comprehensive pearl growing

This process took place prior to the systematic searching, with a view to comparing the efficacy of comprehensive pearl growing (CPG) against conventional subject searching. A review of 13 studies of e-learning in the workplace was identified as a background resource for the review.11 To select the most appropriate databases to undertake the CPG, each study/pearl was checked in each of the databases specified for the conventional subject search to see if it was indexed. Those databases (or a single database) that were shown to index the highest number of pearls would be selected to carry out the CPG. Following this process, it was envisaged that CPG would be undertaken on the selected databases, generating a set of references to be compared with the output of the searches from the conventional subject search for the systematic review. However, the diffusion of nominated pearls across numerous databases, with no one database indexing more than four pearls made carrying out CPG problematic. Additionally, three pearls were not indexed by any databases. The use of CPG as an alternative search technique was suspended at this point.

Quality assessment of included studies

All studies included in the systematic review were appraised using standard checklists of quality assessment criteria, for different types of study design. Where appropriate, case study and survey or questionnaire critical appraisal checklists12,13 were used to assess the quality of a specific study design in more detail. If neither checklist was suitable, the quantitative and qualitative checklists from the Alberta Heritage Foundation for Medical Research were used.14 Scoring systems were not used in quality assessment as they had not been validated for any of the checklists. This process was undertaken to afford a basic idea of the quality of respective studies, while appreciating that the quality assessment of qualitative study designs may not always be appropriate.15 The quality of each study was examined in relation to the search technique that identified the study.

Results

Conventional subject searching

The systematic subject search yielded a total of 3476 references (following removal of duplicates). Of these, 30 were deemed appropriate to be included in the final review. Table 1 shows how the 30 studies were dispersed across 10 multi-disciplinary databases searched.

Table 1.   Sources of studies included in the systematic review from the conventional subject search for the systematic review
SourceNo. of included studies
  1. The combined total of included studies exceeds the final number of studies included in the review from the conventional subject search (30) as some papers were indexed on more than one database.

eric8
psycinfo7
cinahl6
lisa6
medline6
british education index4
assia2
ibss2
emerald1
ssci1

Reference checking

Four additional studies relevant to the review were identified from the reference checking of the 41 included studies.

Contact with experts

Two additional unique studies were identified by contacting experts within the field of e-learning.

Citation searching

The citation searching yielded 75 references (after duplicates were removed). Of the 14 potentially relevant papers, 11 were uniquely identified by this method of searching. Three of these 11 papers were subsequently included in the review (see Fig. 1). Two additional papers were identified serendipitously whilst carrying out the citation searches and consequently included in the review.

Figure 1.

 Results from the citation search

Comprehensive pearl growing

Three pearls were not indexed by any of the 14 databases and nine databases indexed no pearls. The remaining 10 pearls were dispersed across five databases: cinahl (1), emerald (4), eric (3), psycinfo (4) and ssci (1). Scholsser et al. 10-reference number indicate pearl growing be carried out when two or three databases index the majority of the pearls. None of the 14 databases indexed more than 30% of the pearls. This search method was abandoned at this point, due to the diffuse nature of our topic. A scaled-down version of the pearl growing method was also rejected as time and resources were considered better directed towards the other supplementary search techniques.

Quality of included studies

Table 2 shows the number of unique reference included in the review per search technique. Table 3 shows the levels of quality (high, medium or low) of studies identified by the different search techniques. Citation searching and reference list checking identified five of the nine high-quality articles and two of the seven medium-quality studies. The conventional subject search identified nine high- or medium-quality studies with the majority (21) being judged of low quality.

Table 2.   Number of unique references included in the systematic review per search technique
Search techniqueNo. of unique references included in the review
Conventional subject search for the systematic review30
Comprehensive pearl growing0
Citation searching and unstructured searching5
Reference list checking4
Contact with experts2
Table 3.   Quality of studies per search technique
Search techniqueNo. of high quality studiesNo. of medium quality studiesNo. of low-quality studies
Conventional subject search for the systematic review4521
Comprehensive pearl growing000
Citation searching and unstructured searching311
Reference list checking211
Contact with experts002

Discussion

Each of the four search techniques successfully used to identify the evidence yielded unique references for inclusion in the systematic review; the fifth technique, conventional pearl growing was abandoned. Whilst the conventional subject search identified the majority of references ultimately included in the review, supplementary searches were essential to locate further important references for inclusion. It was disappointing that the comprehensive pearl growing search technique could not be utilised. In hindsight, it was to be expected that it was unlikely for one or two databases to index the majority of the pearls, given the anticipated diffuse spread of the literature.

The conventional subject search for the systematic review generated over 3000 references (after de-duplication). Thirty (73%) of the 41 studies included in the review were identified by this method. This large yield and comparatively low number of included studies, is not uncommon for such a sensitive search. A very sensitive search increases the number of references to sift for inclusion or exclusion whilst providing some reassurance that all relevant references have been identified. However, we identified a further 11 references (27%) by other methods. Our findings are similar to those previously reported in the literature that demonstrates sensitive systematic review searches are not always exhaustive and unique references are identified via supplementary search techniques.6,9,16,17 In finding the evidence for a systematic review of access to healthcare for people with learning disabilities, McNally et al.17 located the majority of relevant papers through electronic database searching. However, by contacting experts, citation searching, personal bibliographies, current awareness bulletins and reference lists, this enabled identification of a further 219 relevant papers. Brettle et al.6 located 21% of relevant papers in a systematic review of rehabilitation for adults with severe mental illness through supplementary search techniques; papers that were missed from the main search due to poor indexing and lack of database coverage.

Locating relevant papers is a fundamental goal of searching for systematic reviews. By locating unique references that are included in the review, supplementary searches are essential. Our supplementary searches, like other studies5,16 not only located relevant papers but also high quality unique studies for inclusion in the systematic review. Furthermore, the 30 included references from the conventional subject search for the systematic review came from 10 different databases. This finding emphasises the dispersed nature of the literature for multi-disciplinary social science reviews4,7 and why our pearl growing search method would not work. Searching a number of sources appears essential in order to locate all relevant literature.

Citation searching, reference checking and contact with experts all proved very useful as supplementary strategies. An additional 11 references retrieved by these methods were included in the systematic review. In addition, CS facilitated serendipity of further papers by unstructured searching. However, it is correct to label these strategies as ‘supplementary’ as each of the three techniques could not be used in their own right to systematically identify the literature for this review. If supplementary techniques fail to retrieve further unique references, particularly the use of contacting experts, there remains the confidence instilled in the researchers that a good attempt has been made to identify all relevant studies4,18

Interestingly, reference list checking and citation searching/unstructured searching identified a greater proportion of higher quality articles than the conventional subject search. It is perhaps not surprising that studies of higher/medium-quality are more likely to be cited and included in reference lists. Nevertheless, it is a significant finding.

Devising an exhaustive search strategy for a diffuse topic is challenging. Matthews et al.19 searched for evidence for a systematic review on interventions which help health professionals to communicate with patients about risks. The authors found that this topic area was inconsistently indexed on the main databases with free-text terms such as ‘risk’ being used in many different contexts resulting in a considerable number of hits. To complicate the search process further, this broad topic required a number of databases to be searched and could not be limited by study design, e.g. RCTs, used in clinical effectiveness questions. After experimenting with different search strategies, 99 papers were included in the systematic review. Of 80 papers indexed on medline, only 54 were retrieved via the search strategy. This was mainly due to search term limitation in the medline search (for example, omission of cancer search terms) in order to make the search results more manageable. The remaining 26 papers were found by searching other databases. Nineteen further papers were identified by supplementary search techniques.19 In a review looking at chronic disease and work participation,20 another ‘diffuse’ topic, the authors discovered that 84% of the relevant publications retrieved from each database searched were unique to that particular source.

Our study similarly reflects difficulties in devising a truly exhaustive search strategy due to wide variability in terms, indexing and abstract content. Several databases do not index records, requiring the search strategy to rely on the accuracy and completeness of free-text terms within abstracts. A lack of detail and specificity in the content of abstracts reduced the likelihood of search terms matching with concepts in the abstracts. For example, commonly abstracts failed to include any terms around the employment or work status of the population being studied. As the search strategy retrieved only those papers that included terms around employment in addition to e-learning terms, potentially relevant papers may not have been retrieved. However, expanding our search to retrieve any papers that included terms simply around e-learning (as opposed to e-learning AND employment) terms, would have resulted in an overwhelmingly unmanageable result set. Unstructured abstracts of limited detail increase the difficulty of searching the social science literature.4 Golder et al.16 found that despite using a very broad search strategy with synonyms for the concepts used in the search, most of their searches did not retrieve all potential references. On closer inspection, poor indexing and the absence of abstracts accounted for this. McNally et al.17 had problems when constructing the electronic database search strategies for a systematic review of access to healthcare for people with learning disabilities. Using relevant free-text terms at title and abstract level resulted in unmanageable result sets. McNally et al.’s17 solution was to limit searching for free-text terms within the title field, making numbers more manageable and results more relevant. It is perhaps a point to reflect on whether it is more effective in terms of time and efficacy to limit electronic database searching by restricting terms to title fields; thus relying on other search techniques to identify potentially missed references.

McNally et al.17 and Ogilvie et al.18 noted during their searches for diffuse systematic review topics, that time might have been better spent in determining which sources were most valuable in terms of relevancy. McNally et al.17 suggest less useful sources be discounted at an early stage. In this way, there may be a role for pearl growing in determining the usefulness of sources; an area to be explored by further research. Perhaps a better approach and an area of future research might have been to locate a pearl within each subject area and use its characteristics within the appropriate database.

Ultimately, by employing a range of search techniques, the review team has attempted to capture the majority of relevant literature for this systematic review. Perhaps a more important point upon which to reflect is that as additional papers were identified and their individual findings incorporated into the review, the research team noted that data saturation had been reached with no new findings being discovered. With this being the case, the retrieval of further papers could be deemed unnecessary and indeed Petticrew and Roberts4 discuss instances when saturation is an indicator for stopping further literature searching. Overall, it is the effect of additional papers on the overall result of a systematic review that is ultimately important in the end.9 In the Health Technology Assessment field, Egger et al.21 demonstrated that the exclusion of trials that are difficult to locate, (unpublished trials and non-English trials), had relatively small impact on the overall result of the majority of meta-analyses examined in their study. This as yet has not been demonstrated in topics of more diffuse nature and making every effort to locate studies by supplementary means seems worthwhile.

Conclusion

Searching in the field of social science requires careful consideration. Papers included in the review were widely distributed across a number of multi-disciplinary databases. Problems in indexing, ambiguity of terms and limited abstract content resulted in reduced sensitivity for the main subject search. Follow-up techniques such as reference tracking and citation searching were minimally time intensive but productive, yielding unique, high-quality references for inclusion in the systematic review. Further research could determine if pearl growing is better suited to topic areas where terms are well defined, indexing is consistent and dispersal of literature is confined to a limited number of databases; or could be used as a precursor to conventional subject searching within diffuse topics to determine usefulness of sources.

Evidently, a range of search techniques are required to comprehensively identify social science literature. Perhaps, it is a case of compromising on the sensitivity of the main search strategy whilst supplementing this with additional search techniques to identify other useful references. A need for further investigations and comparison of these different techniques is indicated.

Key Messages

Implications for Practice

  •  Conventional subject searching may miss significant articles for inclusion in social science systematic reviews.
  •  Problems in indexing and abstract content make it difficult to devise a sensitive and exhaustive search strategy; inclusion of alternative search methods such as citation searching, reference list checking and contact with experts thus becomes essential.
  •  The number of higher quality studies identified by citation searching and reference list checking appears to be greater when compared with the proportion of higher quality studies found by database searching.

Implications for Policy

  •  Systematic searching of the social science literature requires a range of search techniques including citation searching, reference list checking and contact with experts

Acknowledgements

We acknowledge that the Higher Education Academy (HEA) funded this project.

Ancillary