Of the 30 respondents, 28 classified themselves as academics/researchers, two as research practitioners and one as an administrator. In 40% of cases, respondents reported fewer years practising qualitative research than totalled their years involved with research (Fig. 1).
Having confirmed that all respondents had a reasonable level of literature searching activity (defined as having initiated at least one literature search in the past 12-month period), clarification was sought on the precise nature of reported search practices.
Eighty per cent of respondents stated performing searches independently or with a colleague as their first preference, whilst requesting a Librarian to search with or on their behalf accounted for just 17% of first choice searches (Fig. 2). One respondent reported that ‘I would search with a Librarian for any major searches where I needed professional advice on strategies’.
Resources and search interfaces
All respondents reported searching electronic databases as part of their literature search, 67% supplementing their database searches with hand searches of printed indexes. Several respondents reported ‘the need for multiple pathways to achieve optimal quality’, whilst another noted that ‘searching databases is a very small part of what I call a literature search … I would [also] rely on hand searching libraries (‘pootling’) and talking to colleagues’.
Thirteen respondents employed additional search techniques as part of their literature trawl, 10 in supplement to searching printed indexes. These included: 17% using citation tracking (following up references listed in the bibliographies of papers and reports), 13% hand searching journals, and 10% either contacting colleagues or authors in the field, or utilizing specialist libraries or websites, e.g. Department of Health.
As might be anticipated, medline, the largest biomedical database available, was the most frequently searched database in the sample (97%), followed by the Social Science Citation Index (SSCI) (77%), cinahl (67%) and the Cochrane Library (67%) (see Fig. 3). Respondents also noted a range of alternative databases in addition to those named in the questionnaire. Included amongst these were the National Research Register (NRR), Science Citation Index (SCI), the databases of the NHS Centre for Reviews and Dissemination, locally available OPACs, and the Norwegian ‘BibSys’ system (Fig. 3).
The most frequently used interface to access these databases was BIDS (the Bath Information and Data Service facilitating access to: embase, the Science Citation Index (SCI) and the Social Science Citation Index (SSCI))—67%, SilverPlatter (facilitating access to cinahl and medline)—53%, PubMed (a free Internet-based medline service offering methodological search filters)—37% and Ovid (facilitating access to cinahl and medline)—30% (Fig. 4).
When asked to indicate how long was spent on their most recent literature search, 13 of the 30 respondents (43%) reported that they spent up to 3 hours searching. The majority of respondents reported a day or less spent searching for relevant literature (63%). Of those respondents professing to search for up to 1 month (10%), half made reference to searching in relation to either a literature or systematic review (Fig. 5). Several respondents qualified their answers, indicating that ‘[the time spent] will depend on the topic and the purpose for which I am searching’, ‘searching varies with the type of search’, and ‘this varies really so much depending on the reason for the searching and the subject’.
Figure 5. ‘Typical’ range of search times. Although respondents were requested to think back to the last time they searched, three respondents indicated that the time spent searching varied. Data were recorded as missing for one respondent, who reported that a Librarian, on their behalf, undertook their last search for published research. Additionally, the reply from a further respondent was ambiguous. However, it was not possible to clarify this reply, as the respondent had indicated that they did not wish to participate in any follow-up contact
Download figure to PowerPoint
The next section of the questionnaire was concerned with the purpose and type of searches undertaken. Respondents were asked to share their experiences—if any—of searching for specific types of research methodology. Seven of the 30 respondents had not undertaken methodologically based literature searches, with an eighth respondent choosing to search across methods and then categorize papers. Equal numbers (25%) had searched for qualitative studies as had experience of searching for randomised controlled trials (RCTs). Only one of these respondents stated they had used published optimal search strategies such as those developed by the Cochrane Collaboration, the Health Libraries Unit in Oxford, the NHS Centre for Reviews and Dissemination or those available on PubMed. The use of a hierarchy of evidence ‘in descending order from systematic reviews, randomised, double-blind, controlled trials, randomised trials, non-experimental studies’ was raised by one respondent in their pursuit of research literature.
To gain insight into respondents experiences and awareness of optimal search strategies (OSSs), respondents were questioned whether they had used, or considered using, an OSS. Whilst two respondents stated that they did not know OSSs existed, or that they had ‘no idea’ what they were, others stated that they had not known about them at the time of searching. Despite the provision of a definition of OSSs, one respondent proclaimed to be ‘unclear about this term’, whilst reporting elsewhere they had used all three levels of the Cochrane Collaboration's OSS.
There were enthusiastic supporters of OSSs, with some respondents stating ‘my experiences had been that the filters are effective for identifying relevant literature’, and that OSSs were ‘very good’ and ‘bloody marvellous’. A high level of awareness regarding search strategy development was demonstrated by one respondent, who acknowledged the need for search strategies to be modified to accommodate the idiosyncrasies of a particular database and interface as important. In particular, one respondent reported that existing OSSs need to be ‘modified by our information specialist so as to not crash our system’.
Whilst most respondents reported a willingness to use an OSS, 30% of respondents expressed reservations about using OSSs. Of these, two questioned being able to select papers without having sight of them, stating ‘[I] can’t think how you’d know what was relevant until you’d looked at it!’ This was a common concern, reflected in the following comment ‘I want to see all papers myself and make my own judgements’. These were issues echoed by those concerned with critical appraisal ‘reviewing often involves critiquing methodology, so I would do my own quality assessment’ and ‘[I] like to assess the appropriateness of the methods and the acceptability of the conclusions myself’.
The number of references potentially retrieved by OSSs was of general concern. One respondent stated ‘I generally want key references and find the volume produced by electronic searches to be largely irrelevant’. Whilst OSSs ‘can obviously be very helpful when a finely tuned search question is available’, three respondents expressed a desire to search widely, often across methodology, and were anxious not to exclude potentially relevant information. For example: ‘we didn’t use a filter as we wished to identify all relevant research regardless of methodology’ and ‘[the] philosophy we use re-searching is to do as wide a search as possible and then eliminate or include reports on an individual basis. We have found that the more prescriptive you are, the more studies of relevance we miss.’ This final point was reiterated by another respondent who stated that ‘we find that quality filters reduce the sensitivity of searches, especially when we are not interested only in RCTs … I don’t seek specific qualitative methodologies, but do a subject search and scan the abstracts’. More generally, one respondent commented that ‘I could see they would be useful just for assessing the evidence base in a subject area, but they would be less useful for more extensive literature reviews’.
The rigour of OSSs remained a concern for some who suggested that ‘the quality of the filters is dependent on the quality of the indexing of each particular database’, ‘sometimes I don’t use a filter because …[the database is] poorly indexed’ and ‘[OSSs] should not be used exclusively due to problems with keywords and mesh terms assigned to studies’.
Only three respondents did not perceive OSSs as relevant to their work. Two respondents stated that they had ‘never needed to’ use OSSs, a third respondent who was specifically seeking to find qualitative research evidence added ‘the quality filters seem to have been developed mainly for RCTs, as far as I know’.