Methodology Review

Checking reference lists to find additional studies for systematic reviews

  1. Tanya Horsley1,*,
  2. Orvie Dingwall2,
  3. Margaret Sampson3

Editorial Group: Cochrane Methodology Review Group

Published Online: 10 AUG 2011

Assessed as up-to-date: 23 JUN 2008

DOI: 10.1002/14651858.MR000026.pub2

How to Cite

Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database of Systematic Reviews 2011, Issue 8. Art. No.: MR000026. DOI: 10.1002/14651858.MR000026.pub2.

Author Information

  1. 1

    Royal College of Physicians and Surgeons of Canada, Centre for Learning in Practice, Ottawa, Ontario, Canada

  2. 2

    University of Manitoba, Neil John Maclean Health Sciences Library, Winnipeg, Canada

  3. 3

    Children's Hospital of Eastern Ontario, Library Services, Ottawa, Ontario, Canada

*Tanya Horsley, Centre for Learning in Practice, Royal College of Physicians and Surgeons of Canada, 774 Echo Drive, Ottawa, Ontario, K1S5N8, Canada. thorsley@rcpsc.edu.

Publication History

  1. Publication Status: New
  2. Published Online: 10 AUG 2011

SEARCH

 

Abstract

  1. Top of page
  2. Abstract
  3. Plain language summary

Background

Checking reference lists to identify relevant studies for systematic reviews is frequently recommended by systematic review manuals and is often undertaken by review authors. To date, no systematic review has explicitly examined the effectiveness of checking reference lists as a method to supplement electronic searching.

Objectives

To investigate the effectiveness of checking reference lists for the identification of additional, relevant studies for systematic reviews. Effectiveness is defined as the proportion of relevant studies identified by review authors solely by checking reference lists.

Search methods

We searched the databases of The Cochrane Library (Issue 3, 2008), Library and Information Science abstracts (LISA) (1969 to July 2008) and MEDLINE (1966 to July 2008). We contacted experts in systematic review methods and examined reference lists of articles.

Selection criteria

Studies of any design which examined checking reference lists as a search method for systematic reviews in any area. The primary outcome was the additional yield of relevant studies (i.e. studies not found through any other search methodologies); other outcomes were publication types identified and data pertaining to the costs (e.g. cost-effectiveness, cost-efficiency) of checking reference lists.

Data collection and analysis

We summarized data descriptively.

Main results

We included 12 studies (in 13 publications) in this review, but interpretability and generalizability of these studies is difficult and the study designs used were at high risk of bias. The additional yield (calculated by dividing the additional 'unique' yield identified by checking reference lists by the total number of studies found to be eligible within the study) of relevant studies identified through checking reference lists ranged from 2.5% to 42.7%. Only two studies reported yield information by publication type (dissertations and systematic reviews). No cost data were reported although one study commented that it was impossible to isolate the time spent on reference tracking since this was done in parallel with the critical appraisal of each paper, and for that particular study costs were not specifically estimated.

Authors' conclusions

There is some evidence to support the use of checking reference lists for locating studies in systematic reviews. However, this evidence is derived from weak study designs. In situations where the identification of all relevant studies through handsearching and database searching is difficult, it would seem prudent that authors of reviews check reference lists to supplement their searching. The challenge, therefore, is for review authors to recognize those situations.

 

Plain language summary

  1. Top of page
  2. Abstract
  3. Plain language summary

Examining reference lists to find relevant studies for systematic reviews

Systematic reviews are summaries of the information that is available on one topic. The most common way to find information for a systematic review is to search electronic literature databases. To increase the chances of finding important information, researchers can also search the tables of contents of journals, and they can contact experts or organizations for more information on the topic of the review. Another way to find more information is to check through the reference lists of relevant studies to see if these references include reports of other studies that might be eligible for the review. It is important to determine whether or not checking reference lists is a good use of time and resources when conducting systematic reviews.

We found 12 studies that explored whether or not checking reference lists was useful for systematic reviews. These studies reported a range of results, from identifying only a few additional studies (2.5%: 2 of 79 included studies) to identifying many additional studies (42.7%: 111 of 260 included studies) through checking reference lists. Unfortunately, none of the studies looked at how much time or money were spent on the process of checking reference lists, and it was suggested this would be almost impossible to estimate.

Unfortunately our findings are based on weak information. The data do suggest that in situations where researchers may have difficulty locating information, checking through the reference lists may be an important way to reduce the risk of missing relevant information.