Elaine Matthews, Research Team for Care of Elderly People, Academic Centre, Llandough Hospital, Penarth, Vale of Glamorgan, CF64 2XX, UK. E-mail: MatthewsEJ@cardiff.ac.uk
Using the example of communication about risk in a primary care setting, this paper puts forward a method of developing and evaluating a detailed search strategy for locating the literature for a systematic review of a ‘diffuse’ subject. The aim of this paper is to show how to develop a search strategy that maximizes both recall and precision while keeping search outputs manageable. Six different databases were used, namely Medline, Embase, PsychLIT, CancerLIT, Cinahl and Social Science Citation Index (SSCI). The searches were augmented by hand-searching, contacting authors, citation searching and reference lists from included papers. Other databases were searched but yielded no extra references for this subject matter. Of the 99 papers included, 80 were indexed on Medline. The Medline search strategy identified 54 of them and the remaining 26 were located on other databases. The 19 further unique references were found using the other databases and methods of retrieval. A combination of several databases must be used to maximize recall and to increase the precision of searches on individual databases, thus improving the overall efficiency of the search.
To demonstrate an efficient approach to locating the maximum amount of primary research on a diverse subject for a systematic review, using the example of communication about risk in primary care. This paper looks at how to develop and evaluate a search strategy that maximizes both recall and precision while endeavouring to ensure that the search is kept within manageable limits.
All systematic reviews strive to gather the full range of published and unpublished studies on the given topic. Most of the subjects that have so far been covered by the Cochrane Collaboration are narrowly defined and predominantly medical. For some subjects, such as epilepsy, it is a straightforward exercise to find papers for inclusion because there are a handful of key journals publishing the bulk of all research. 1 Thus, it is feasible to hand-search a small number of titles to complement a search on Medline and any specialist databases. Increasingly, systematic reviews are attempting to deal with much broader topics, where the nature of interventions are less easily specified and the disciplines involved are not simply medical, but ‘allied to medicine’, psychological sciences, social sciences and many others.
Communication in health care is one such example. Studies of risk topics also demonstrate multidisciplinary evaluation, and such studies have increased at ‘epidemic’ rates in the literature in recent years. 2 Our systematic review addresses communication about risk, examining ways in which people are informed of their risks for diseases such as coronary heart disease, HIV and cancer, or the risks of treatment and diagnostic procedures. The clinical areas are expansive and there are no key journals to enable efficient hand-searching, therefore a broad range of databases needs to be consulted.
Much of the research in more purely medical topics is in the form of randomized controlled trials (RCTs); searches can be limited to such papers using the search ‘filters’ developed by the Cochrane Collaboration. 3 Unfortunately, little research on risk communication in primary care uses RCTs, a broader range of study designs (levels of evidence) being appropriate to answer relevant research questions.4,5 In diffuse topics such as this it is not appropriate to limit searches to RCTs, thus further broadening the scope of the search.
Publication bias is another consideration. In some subjects there are comprehensive prospective research registers that can be checked against database search outputs to assess what research does not get published. An example is the National Research Register (NRR) produced by Update Software Ltd (Oxford, UK) on behalf of the Department of Health, which has been available on the Internet (www.update-software.com) since 1998. It is a register of ongoing and recently completed research projects funded by the National Health Service. There is a searchable database of more than 28 000 research projects, plus entries from the Medical Research Council’s Clinical Trials Register, and details of reviews in progress collected by the NHS Centre for Reviews and Dissemination. Risk communication is an emerging field within medicine so such registers contain little information about risk communication, although it is becoming more popular. Publication bias is therefore a significant concern; positive results are more likely to be published so the likelihood of finding a study may be influenced by its outcome.
Another issue concerns the geographical location of the research and the language in which it is written up. Many systematic reviews report limiting their searches to English language papers. In an emerging field, such as communication about risk, it may be uncertain at the outset from where most of the research will originate. This makes it inappropriate to limit searches by language or country of origin before an assessment of the literature has been made.
The primary objective of a search strategy is to obtain the maximum possible number of relevant papers, i.e. acceptable level of ‘recall’ (equivalent to sensitivity). A secondary aim is to maintain efficiency in retrieving these papers by achieving high precision (equivalent to specificity). The challenge is to achieve high recall without sacrificing precision. By using a range of databases we aimed to increase the precision of the Medline search even if this reduced its recall. This is because overall recall is improved by using a range of databases while at the same time keeping precision high (see Table 1).
Table 1. Definitions of recall and precision
Defining the search
One of the first considerations in designing a search strategy is to have a clear definition of what is being sought. A definition forms the basis of the search strategy and is used to develop specific inclusion and exclusion criteria to apply to the titles and abstracts identified.
We wanted to review the evidence on the effectiveness of interventions designed to help health professionals communicate with their patients about risks, particularly on a one-to-one basis. There is no established definition of risk communication in this context, however, and this leads to inconsistent indexing on the principal databases. Moreover, much of the primary research in this field concerns communication about environmental and toxicological hazards at a population level. These two features mean that running a simple search on Medline combining the MeSH (Medical Subject Headings) for risk and communication would yield few references that fitted the inclusion criteria. The free text phrase risk communication is similarly unhelpful as it almost exclusively refers mass communication when used in the PsychLIT database.
Another problem to overcome is the widespread use of the word ‘risk’. It is used commonly in many contexts. Searching for risk (truncated) as a free text word on Medline produces over 80 000 hits for the years 1993–December 1995. This is too broad and unusable as a search term, even in combination with other free text words. It was also more than some readily available computers could manage at the time, although the increasing availability of Medline via the Internet should go some way to solving these technical difficulties.
Developing the strategy
Pilot searches were developed and tested on Medline. It was unclear at the outset where the research on risk communication would originate from so it was decided not to limit by language and to consider all papers with an English abstract. Regarding geographical location, it was decided to include papers from all places where the health system has some similarity to the UK, i.e. mainly English speaking countries and Europe.
Due to the problems mentioned above, a range of related MeSH headings for the term ‘risk’ were therefore appropriate for the first section of the search (section A). A second set of terms related to ‘communication’ (section B) was created to select papers indexed for both risk and communication. As the MeSH heading for communication alone was of limited use, the concept was broadened out to include all terms related to health promotion, counselling and patient education. These two groups were then combined together using the Boolean operator ‘and’ to produce the first set of references for consideration against the inclusion/exclusion criteria. Initially these references had been further combined with subject headings covering primary care. The result of this, however, was to exclude the majority of relevant references, because primary care is rarely adequately indexed and many of the included papers were not set in the context of primary care. This element of the search strategy was therefore discarded.
This pilot strategy was adapted using comparable indexing terms specific for Embase, PsychLIT, CancerLIT, Cinahl, and Social Science Citation Index (SSCI) and again run for 1995.
All references retrieved were printed out with the abstract and subject headings and were assessed against the inclusion/exclusion criteria. The subject terms for the references retrieved were examined to see if any subject heading could be excluded, or others included. The result of this was the third section (C) made up of diseases and conditions combined with communication terms to locate additional references related to health promotion and communication with people with these conditions. Section A was also augmented with the MeSH headings for health behaviour and life style as these located further relevant references on Medline in combination with the communication terms.
The revised pilot searches were again run for the year 1995 and references printed out with abstracts. The included references were checked to see if they were on Medline but had been missed by the Medline search. If they were missed, the relevant MeSH terms were identified and incorporated into a new Medline search. A very broad Medline search was then piloted to ascertain whether this may be more efficient than relying so heavily on a range of databases.
Carrying out the search
The databases chosen for the full search were Medline, Embase, PsychLIT, CancerLIT, Cinahl and Social Science Citation Index (SSCI). Other databases piloted included International Pharmaceutical Abstracts and Applied Social Sciences Index (ASSIA), but these did not identify new references and were not used for further searches. This range of databases was necessary because it appeared from the pilot searches that if a reference did not come up on Medline (owing to inadequate indexing) there was still a possibility of finding it on another database with overlapping content but a different indexing policy. This assumption was further tested in the area of risk communication about cancer. Almost 8000 references were retrieved between 1993 and 1996 using the MeSH terms for prevention and control of cancer (exp Neoplasms/pc), so it was decided not to run them in Medline but to retrieve them with the CancerLIT search instead.
The finalized searches were run on all selected databases from 1985 to 1996. Bibliographic management software (Reference Manager, Research Information Systems (RIS), Carlsbad, CA, USA) was used to create a database of the references retrieved from the searches. The references from each database search were transferred into individual databases according to year. Each year was then combined to form a new database and duplicates deleted to produce all the unique references for that year. These titles and abstracts were then printed out and checked through by hand by two of the authors (E.M. and A.E.) against the agreed inclusion/exclusion criteria. For example, papers reporting studies set in prisons or schools or those entirely about mass communication were excluded. References meeting the inclusion criteria were put into one of 12 categories, described in detail in the final report. 6 Accepted references were obtained in full paper form.
In order to maximize the number of papers considered still further, a number of supplementary searches were carried out. First, the four most frequently encountered journals were ‘hand-searched’ for every issue from 1990 to 1996. These journals were: Medical Decision Making, Patient Education & Counseling, Health Psychology and Risk Analysis. Second, the most frequently quoted authors were asked to send their CVs to review their publications lists and ongoing research. Third, accepted papers were checked in the Science Citation Index and the SSCI to see which authors had cited them in subsequent papers. Finally, those papers that had earlier been identified as reviews relating to risk communication and those papers included in the systematic review had their references searched. All of these methods are accepted ways to assess and improve the recall of the original search strategy.
The Medline pilot search for 1995 found 47 papers suitable for detailed assessment out of 1040 references retrieved from Medline plus 107 from 1248 references in the other databases: a precision rate of 6.7%. The broader Medline search identified 61 of these papers from 3149 references: a precision rate of only 1.9%. The other databases would still have to be searched to locate the additional 93 references, resulting in a precision rate of only 3.5%. The pilot search strategy was judged successful for maximising recall and precision 7 and was applied to all years from 1985 to December 1996. The refined search strategy for Medline is shown in Table 2.
Table 2. Risk communication search strategy for Medline
Ninety-nine papers were finally accepted, passing through all stages from the initial search to data extraction and statistical analysis. Eighteen thousand one-hundred and fifty abstracts were examined and from these, 418 references were each placed into one of 12 categories to be assessed for inclusion in the main systematic review. The bulk of papers came from 1995 with a steady decline to 1985 and one paper each from 1982 and 1997 (see Fig. 1). The paper from 1997 was removed from the final analysis because it was outside the remit of the search.
Almost three quarters of the 99 papers were set in the USA (71 papers), 16 were from Canada, Australia or New Zealand, nine were from the UK and three were from other European countries. All the papers were in English.
Of the 99 papers for the systematic review, the search strategy identified 54 papers on Medline with the rest being found in other databases as shown in Table 3.
Table 3. Location of all references
Out of the 99 references, 80 were indexed on Medline but 26 of these were not identified by the Medline search strategy. These 26 references were found in other databases, as shown in Table 4.
Table 4. Location of references indexed on Medline but found using other databases
There are several reasons why these 26 papers were missed from the Medline search. First, cancer terms were not included in the Medline search to limit the overall number of references retrieved. Thirteen references on cancer were identified on CancerLIT and a further four papers were on cancer topics but were not found by either the CancerLIT or Medline searches. Embase, Cinahl, SSCI and reference searching each identified other unique references. Second, three papers were found on medical decision making; it had been agreed at the pilot stage not to use this as a MeSH term. The hand-searching of the journal Medical Decision Making found one reference, the other two were identified from reference and citation searching. Finally, a group of six papers were not found, either because they were in a clinical area that we had not included in section C (osteoporosis and contrast media) and/or because none of our communication terms in section B were used in the indexing. One of these papers had the word risk in the title but no MeSH term for risk. More than one-third (34) of the titles and abstracts did not include the word risk in the title, abstract or subject headings even though they involved communication about risk.
Had a Medline search been developed to find the 80 papers indexed on Medline, the remaining 19 unique references would have been identified from other databases as shown in Table 5.
Table 5. Unique references found by searches
Thus, each method of follow-up searching uncovered unique references in each database except CancerLIT. Medline had 22 unique references that were not indexed on any other database. No references were identified in CancerLIT that were not on Medline, nor were any new references found. This was to be expected as CancerLIT is a subset of Medline, with the inclusion of conference proceedings and other grey literature. No references were found using CancerLIT, therefore using CancerLIT did not contribute to recall, but precision was improved as a result of different indexing systems.
This analysis has been carried out with hindsight. We have found the needles in the haystack and we are now drawing a map of how to find them again. We have identified efficient strategies for researchers wishing to search literature on risk or communication topics and can identify general principles about searching in diffuse topics.
The combination of MeSH headings in the search strategy contributed to the recall (sensitivity) of the search. Medline contained the majority of the titles, but each database and method of searching found additional references. There was a peak in papers retrieved in 1995, declining steadily to 1985. The 1996 count was smaller than 1995 because the databases had not been fully indexed for 1996 when the searches were carried out.
A much broader search could have been developed for Medline alone. This could have found 80 references, but would still not have retrieved a further 19 papers not indexed on Medline, a significant proportion for the review as a whole. It was necessary to search the other databases in order to retrieve all 99 papers. For this topic PsychLIT, Embase, SSCI, Cinahl and the hand-searching all identified unique references beyond those identified from Medline. Within these sources there was further potential overlap if all the references had been found on all the databases where they were indexed.
Maximum recall from Medline can only be achieved with a substantial sacrifice of precision, rendering the search unmanageable. As well as containing unique references not on Medline, the range of other databases enables precision to be maintained even in diffuse topics such as described here. This is particularly so for CancerLIT where indexing allowed a much higher degree of precision, but produced no unique references. The same issue applies for the other databases—the maximum recall from PsychLIT can only be achieved by sacrificing precision, but using Embase, Cinahl and others enables precision to be maintained at manageable levels for improving the overall efficiency of the search.
The principles of this approach to identifying studies are pertinent to anyone wishing to search diffuse topics, which derive from a range of disciplines. It is likely that in many such topics the range of research designs is wide and standard search filters to restrict the output (e.g. for RCTs) will be inappropriate. A range of databases must be used to maximize recall and maintain acceptable precision.