gorecki c.a., brown j.m., briggs m. & nixon j. (2010) Evaluation of five search strategies in retrieving qualitative patient-reported electronic data on the impact of pressure ulcers on quality of life. Journal of Advanced Nursing66(3), 645–652.
Aim. This paper is a report of a study conducted to compare the effectiveness of qualitative methodology search strategies with subject-specific (health-related quality of life) search strategies in the retrieval of qualitative patient-reported data of the impact of pressure ulcers on health-related quality of life.
Background. Methods to locate qualitative patient-reported health-related quality of life research data electronically have undergone little replication and validation. A major problem in searching for this type of data is that it is reported in accounts of both primary qualitative research as well as mixed methods research.
Data sources. We combined five search strategies with terms for pressure ulcer and searched seven electronic databases from inception to October 2007.
Methods. The sensitivity, specificity, precision and accuracy for each search strategy were assessed.
Results. A subject-specific (health-related quality of life) search strategy, developed by us, had a high yield (100% sensitivity), but low specificity (<50%). The research methodology-based strategies had lower yields (sensitivity 72–83%) but high specificity (79–83%). Importantly, subject-specific search strategies identified all studies reporting qualitative patient-reported health-related quality of life data, whereas, research methodology-based strategies did not identify qualitative data reported in mixed method studies, making subject-based strategies more effective in retrieving qualitative patient-reported health-related quality of life research.
Conclusion. An important consideration in the health-related quality of life field is that qualitative data are reported in both qualitative and mixed methodology research and searching for this type data involves trade-offs between yield, sensitivity and specificity. Accurate indexing of subject-specific outcomes and methodology used in electronic databases and publications is also needed.
What is already known about this topic
- • One of the key elements of evidence-based practice is synthesis of research in the form of systematic reviews.
- • Methods for locating quantitative research have been developed and well-validated, including a combination of subject-specific search terms relevant to the clinical area of interest and methodological terms.
- • Robust methods for locating qualitative patient-reported quality of life research data have undergone little replication and validation.
What this paper adds
- • For some mixed method studies, indexing and key words gave preference to quantitative methods.
- • Subject-based search strategies appear more effective in retrieving qualitative patient-reported quality of life data than methodology-based (qualitative) strategies.
- • Comprehensive searching for qualitative patient-reported quality of life data involves trade-offs between yield, sensitivity and specificity.
Implications for practice and/or policy
- • A simple broad-based qualitative search strategy in CINAHL can be used when looking for a few sound studies on a topic.
- • A subject-specific (quality of life) strategy in CINAHL, MEDLINE and EMBASE should be used when carrying out a comprehensive systematic review.
- • There should be adequate indexing of subject-specific outcomes and methodological terms in all electronic databases and publications.
Evidence-based practice (EBP) is central when making decisions about the organization and delivery of health care and policy development (Lambert et al. 2006). Traditionally, there has been a focus upon evaluation of clinical outcomes through randomized controlled trials (RCT) to determine which interventions work best. Increasingly, importance is placed on the use of patient-reported outcomes (PRO) addressing constructs such as health-related quality of life (HRQL) and health status (Testa & Simonson 1996, Garratt et al. 2002). PRO measures provide feasible and appropriate methods for the assessment of issues important to patients in clinical trials and clinical practice research.
With mounting interest in PRO and HRQL assessment in clinical trials and clinical practice, PR data obtained by qualitative methods (i.e. patient interview or focus groups) are the accepted standard method applied in the development of PRO measures (US Department of Health & Human Services FDA 2006). Alternatively, if no existing PRO measure of HRQL is available for a given disease, obtaining PR information about the disease experience, related health care received and the impact of the disease and interventions on HRQL, qualitative methods would seem appropriate and useful for obtaining this information.
Qualitative research methodologies in clinical and healthcare research have expanded greatly, with growing interest in the synthesis of qualitative research data. Not only important for PRO development and evaluation, qualitative data contribute to healthcare and policy development as they provide direct patient accounts of the illness experience, its impact on human suffering, and the impact of related interventions received (Evans 2002). Because data are not reduced to numbers, qualitative research can yield a rich narrative account of patients’ experiences from their perspectives.
One of the key elements of EBP is synthesis of research in the form of systematic reviews. Robust systematic reviews are designed to advance knowledge through evaluation of the existing research and provide best-evidence, empirical answers to questions about health and healthcare issues, making them a valuable source of information to clinical practitioners. In addition, systematic reviews can identify areas where there is a lack of knowledge, so that the planning of new research can be focused around these areas. There are established scientific methods for systematic reviewing and collation of the best available evidence (Centre for Reviews and Dissemination 2001). To ensure that the evidence base is exhaustive, appropriate methods for locating the evidence need to be used. Effort has been made to establish searching methods through the development and validation of research methodology filters (Centre for Evidence-Based Medicine 2006; Dickersin et al. 1994, Higgins & Peacock 2005). Predominantly, methods for systematic reviewing have focused on locating and synthesizing data obtained from RCT (The Cochrane Collaboration 2007). As a result, a number of search strategies and methods for locating RCT have been developed, consisting of a combination of subject-specific search terms relevant to the clinical area of interest and RCT methodological terms, and this combination has been well-validated in retrieving RCT (Brettle et al. 1998, Higgins & Peacock 2005). The development of robust methods for locating qualitative research data has not kept pace with these wider developments, and as so existing qualitative methodology-based filters are fewer and have undergone less replication and validation (Dixon-Woods et al. 2005).
Systematic reviewing for qualitative research data is complicated by the use of a variety of qualitative methodologies, including ethnography, phenomenology and grounded theory (Ploeg 1999). The use of terminology to describe qualitative perspectives and methods is not standardized (Brookes 2007), and indexing in electronic databases is flawed in that qualitative research methods are not always clearly described and indexed in publications. Furthermore, PR-HRQL data are reported in accounts of both primary qualitative and mixed methods (qualitative and quantitative) research. Therefore the traditional use of subject-specific search terms and research methodology-based search strategies may not be the most effective method for locating qualitative PR-HRQL research data, particularly since existing methodology-based (qualitative) search strategies are not designed to identify qualitative data integrated within mixed methodology research.
What is needed are comprehensive and precise search strategies to identify all qualitative PR-HRQL data, regardless of research methodology, without missing key studies or retrieving excessive numbers of irrelevant studies (Brettle et al. 1998).
The aim of this study was to compare and evaluate the effectiveness of research methodology (qualitative)-based search strategies with subject-specific (HRQL) search strategies for use in electronic searching for the retrieval of qualitative PR-HRQL research data. The ability of each search strategy to locate qualitative HRQL research data relevant to pressure ulcers (PU) was evaluated across seven bibliographic databases.
The study emerged from a larger programme of research that has been established to develop a PRO measure of HRQL for use with patients with PU. The first part of this programme involved undertaking a systematic review of the HRQL and PU literature (Gorecki et al. 2009), which identified methodological issues when systematically searching for relevant papers reporting qualitative PR-HRQL data.
This evaluation involved replicating five search strategies: three research methodology (qualitative) search strategies (Shaw et al. 2004), one subject-specific (QL) strategy (InterTASC 2006, Fayter et al. 2007), and a refined subject-specific (HRQL) strategy developed by us (search strategy available from the corresponding author) for a systematic review where the intension was to identify all qualitative PR-HRQL data pertaining to the impact of PU.
The three methodology-based (qualitative) strategies developed by Shaw et al. (2004) for the identification of qualitative research within electronic databases are: a complex methodology-based (qualitative) strategy consisting of free-text terms; a complex methodology-based (qualitative) strategy consisting of thesaurus terms; and a simple methodology-based (qualitative) strategy consisting of broad-based terms (Shaw et al. 2004). As we were interested in locating qualitative PR-HRQL data within primary research, we anticipated that qualitative research methodology was likely to be used to address this topic. Therefore, the Shaw et al. (2004) search strategies were chosen as they include comprehensive terms for various qualitative methodologies. A review of existing qualitative methodology-based search strategies identified several strategies developed to enhance the retrieval of qualitative studies using various combinations of single terms (Grant 2004, Wong et al. 2004; Walters et al. 2006, Wilczynski et al. 2007). These were considered but excluded from evaluation as the single terms were also included in the research methodology (qualitative)-based search strategies developed by Shaw et al. (2004).
The subject-specific (HRQL) strategy was a complex QL strategy (Fayter et al. 2007) combined with an existing patient issues strategy (InterTASC 2006), consisting of free-text, thesaurus and broad-based QL terms, specific QL measurement terms and patient issue terms (see Table 1).
|Complex qualitative free-text terms (Shaw et al. 2004)||Complex qualitative thesaurus terms (Shaw et al. 2004)||Simple qualitative broad-based terms (Shaw et al. 2004)||Complex QOL and patient issues terms (InterTASC 2006, Fayter et al. 2007)||PR-HRQL|
|Forty-eight search terms that represent commonly used qualitative research methodologies e.g. ‘ethnonursing’, ‘grounded theory’, ‘lived experience’||Twenty terms that are keywords indexed within an electronic database and also termed subject headings. Each record in a database is categorized by keywords||Simplest search strategy involving three broad-based terms; ‘qualitative’, ‘findings’ and ‘interviews’||Fifty-three search terms consisting of 21 free-text, thesaurus, and broad-based HRQL terms, 12 specific HRQL instrument terms, and 20 terms to identify patient issues||Seventeen terms consisting of six broad-based and specific HRQL terms, two method terms and nine free-text patient experiential terms|
In addition to the existing search strategies, we had previously undertaken a systematic review where we developed a subject-specific (HRQL) search strategy for the retrieval of qualitative PR-HRQL research data pertaining to the impact of PU. This subject-specific (HRQL) strategy was constructed by reviewing the search terms from the existing complex QL search strategy and the patient views strategy (InterTASC 2006, Fayter et al. 2007), as well as reviewing abstracts, titles, and key words from relevant studies to identify the MeSH terms (different terminology for the same concept) and text word combinations most frequently used throughout all the studies related to qualitative PR-HRQL data. From these two sources, an initial list of search terms was compiled. Combinations of terms were compared for retrieval performance. Terms that returned results were retained. The PR-HRQL search strategy consisted of commonly-used free-text and broad-based QL terms, patient-specific psychological and experiential terms, and two method terms –‘interview’ and ‘questionnaire’.
Each search strategy was combined with specific MeSH headings for PU used by the Cochrane Wounds Group (e.g. ‘pressure ulcer’, ‘pressure sore’ and ‘bed sore’). Searches of AMED (1985–2007), British Nursing Index (BNI) (1985–2007), Ovid MEDLINE (1966–2007), EMBASE (1980–2007), PsychINFO (1967–2007), CINAHL (1982–2007), Cochrane Controlled Trials Register (CCTR) (1991–2007) were carried out in October 2007. These databases were chosen as they represent the disciplines of medicine, nursing, social sciences and psychology. Each search strategy was run in each database and in combined database searches.
In addition, known item searches were undertaken for all relevant papers identified from hand searching for the systematic review (Gorecki et al. 2009) but not retrieved by the searches electronically; this was done to determine whether the paper was actually missed by the searches or whether it was not included in the coverage of the database.
Retrieved studies were assessed by one reviewer as potentially relevant (i.e. whether the paper could be classified as relevant to HRQL and PR qualitative data). Potentially relevant papers were reviewed against the inclusion criteria by two independent reviewers, with the inclusion of papers agreed by consensus. Those assessed as relevant included adults with any grade PU, the study was clearly designed to assess HRQL by direct patient-reports, irrespective of methodology, and qualitative HRQL data were reported or provided by the author. Studies were excluded if HRQL was reported using only existing PRO measures.
The concepts of yield (the total number of records retrieved by the search after eliminating duplicates), sensitivity (the percentage of relevant papers retrieved), specificity (the percentage of non-relevant papers that were not retrieved), precision (percentage of retrieved papers that were relevant) and accuracy (the percentage of all citations classified correctly) were calculated to determine the retrieval performance of the strategy searches (Wong et al. 2004). Yield and number of relevant papers retrieved by each search within each database individually was determined to represent distribution of papers across database searches.
No patients or patient data were required for this study therefore ethical approval was not required.
Results by search strategy
The total yield of records retrieved by all the search strategies, searching seven databases, was 4620. From these, 18 studies were assessed as relevant as they contained qualitative PR-HRQL data. Three of these study reports described mixed methods in their design, and two described only quantitative methodology but reported qualitative HRQL data. The distribution of results across each search strategy is shown in Table 2.
|Search strategy||Yield* (no. relevant studies)||Sensitivity, % (calculation)||Specificity, % (calculation)||Precision, % (calculation)||Accuracy, % (calculation)|
|Complex qualitative free text||782 (14)||77·78 (14/18)||83·31 (768/4602)||1·79 (14/782)||83·29 (14 + 3834/4620)|
|Complex qualitative thesaurus||926 (13)||72·22 (13/18)||80·16 (913/4602)||1·40 (13/926)||80·13 (13 + 3689/4620)|
|Simple qualitative broad-based||962 (15)||83·33 (15/18)||79·42 (767/4602)||1·56 (15/962)||79·44 (15 + 3835/4620)|
|Complex HRQL||3038 (18)||100·00 (18/18)||34·38 (3020/4602)||0·59 (18/3038)||34·63 (18 + 1582/4620)|
|PR-HRQL||2502 (18)||100·00 (18/18)||46·02 (2484/4602)||0·72 (18/2502)||46·23 (18 + 2118/4620)|
The complex QL and the PR-HRQL subject-based search strategies had the best sensitivity (100%), retrieving all relevant papers, albeit poor specificity compared with the research methodology (qualitative) search strategies. Compared with the complex QL search, the PR-HRQL search had better specificity, precision and accuracy, making it the most effective subject-specific (HRQL) search strategy for locating qualitative PR-HRQL data, irrespective of research methodology. However, specificity was low, many irrelevant studies being retrieved.
From the three research methodology (qualitative)-based search strategies (Shaw et al. 2004), the simple qualitative broad-based search had best sensitivity (83·33%), but slightly reduced specificity compared with the other research methodology (qualitative)-based searches. This search was the most effective methodology-based (qualitative) search in retrieving relevant qualitative PR-HRQL data using qualitative research designs. However, it failed to identify studies that had used mixed methods. All the research methodology (qualitative)-based searches had better specificity than the subject-specific (HRQL) search strategies, correctly identifying studies as not relevant and producing a lower yield of records to be screened.
Results by database
The results retrieved by search strategy within each database are shown in Table 3. As the numbers include duplication of studies across databases, the nature of retrieval available in each database with each search strategy can be demonstrated.
|Database||Complex qualitative: free text*||Complex qualitative: thesaurus*||Simple qualitative: broad-based*||Complex QL*||PR-HRQL*|
|CINAHL||459 (14)||492 (13)||300 (15)||1034 (12)†||976 (15)†|
|MEDLINE||289 (8)||428 (9)||492 (10)||1452 (12)†||991 (15)†|
|EMBASE||220 (2)||178 (1)||151 (2)||1178 (3)†||664 (2)†|
|PsycINFO||29 (1)||36 (2)||47 (4)||84 (4)||187 (4)|
|BNI||11 (2)||19 (2)||55 [2)||128 (6)||55 (4)|
|AMED||30 (1)||46 (1)||42 (1)||174 (1)||121 (1)|
|CCTR||27 (0)||44 (0)||78 (0)||147 (0)||81 (1)|
CINAHL had overall best sensitivity in retrieving relevant studies across all search strategies. Using the subject-specific PR-HRQL search strategy in CINAHL or MEDLINE produced best yield of relevant papers, with 15 of the 18 relevant studies identified in each database. MEDLINE produced a higher yield of records when searched using either of the subject-specific (HRQL) searches compared with CINAHL. MEDLINE identified 15 relevant studies, including two that were not identified by any other database, but failed to identify three relevant studies (one retrieved in CINAHL; one in EMBASE; and one by both CINAHL and PsychINFO databases). CINAHL also produced a high yield, compared with the other databases, although it identified 15 relevant studies and retrieved one relevant study that was not identified in any other database. Despite good yield of relevant studies, CINAHL failed to identify three relevant studies that were identified by other databases (MEDLINE retrieved two studies and EMBASE one study). CCTR had the poorest yield of relevant studies, failing to identify all but one relevant study (Table 3).
Combined searching extended to incorporate CINAHL, MEDLINE and EMBASE databases produced the best yield of relevant studies, although it produced high overall yield, resulting in a large amount of irrelevant studies to screen (Table 3). In general, retrieval of relevant studies was relatively low for all databases. This means that a large number of irrelevant records were identified, with few relevant studies included.
Study strengths and limitations
A limitation of this study was the inclusion of a small sample of relevant studies despite our comprehensive searching methods. In addition, the PR-HRQL strategy was developed by comparing the performance of combinations of search terms for retrieval performance of qualitative PR-HRQL data on the impact of PU. As a result of these limitations, the validity of the strategy needs to be established by replicating it in other medical areas. However, the PR-HRQL search strategy should be generalizable for use in other medical areas when the aim of the research is to identify qualitative PR-HRQL data, as the strategy has been adapted from existing strategies currently used in systematic searching for HRQL research. The strategy could be strengthened by reviewing the non-relevant records retrieved by the PR-HRQL search to identify additional ‘but not’ search terms that would be beneficial for reducing yield and improving specificity and precision of the strategy.
A strength of this study was that we hand-searched relevant journals and reference lists of relevant papers to identify relevant studies that might not have been retrieved by the searches. This reduced the bias that would be introduced from relying only on data obtained electronically – we would not know what was missed.
Discussion and practical implications
The simple research methodology (qualitative) broad-based search strategy was most effective in retrieving relevant qualitative HRQL studies using qualitative research methodology. This concurs with finding by Flemming and Briggs (2007) and Wong et al. (2004) that broad-based qualitative terms such as ‘interview’ or ‘qualitative’ are effective in the retrieval of qualitative studies. Even though the simple qualitative broad-based search failed to identify studies using mixed methods, it was still an effective strategy for locating qualitative research studies. However, such a search may identify many irrelevant citations and possibly miss some relevant ones if they are not indexed as qualitative research, as reported by others (Shaw et al. 2004, Flemming and Briggs 2007, Dixon-Woods et al. 2005).
The problem with incorrect indexing in relation to finding RCT and qualitative research for systematic reviews has been highlighted previously (Dickersin et al. 1994, Dixon-Woods et al. 2001). The problem seems to lie with the classification of study designs and research methodology used (i.e. a phenomenological study was indexed as ‘case-report’ in the database). Upon inspection qualitative methods were not always clearly defined in electronic databases or described appropriately in publications. Thus, three relevant studies identified through hand-searching failed to be retrieved by either the complex or simple qualitative research methodology-based searches developed by Shaw et al. (2004). When we re-examined the initial search we discovered that these studies were incorrectly indexed in the database and therefore failed to be retrieved by research methodology (qualitative)-based searches alone; non-retrieval was because of the record being missed by the search strategy or because it was excluded in the database coverage. The other problem with using research methodology (qualitative)-based searches (Shaw et al. 2004) was that they were unable to locate mixed method research, as the search strategy was intended to be used as a methodological filter, and not for identifying quantitative methodology. Linked to this, for some mixed method studies, indexing and key words gave preference to quantitative methods.
The PR-HRQL strategy was most effective in retrieving qualitative HRQL data regardless of methodology, although specificity and precision were low, indicating that a large number of irrelevant records were also retrieved. This suggests that methodology-based (qualitative) searches may not be the most effective method for locating qualitative PR-HRQL data, although they are more efficient in terms of specificity. Similarly, the low specificity and precision of the HRQL-subject searches may have been as a result of inadequate reporting of the HRQL concepts and outcomes addressed. Quite often, when addressing HRQL, patients are asked about their experiences of health and the consequences of the illness and treatments received (Fitzpatrick et al. 1998), reporting their own views, feelings and experiences. This identifies the importance of using appropriate MeSH, text and key words when systematic searching and indexing qualitative PR-HRQL research regardless of the primary methodology, and the need for adequate indexing of subject-specific outcomes and methodological terms in electronic databases and publications.
The CINAHL database had the best yield of relevant qualitative research, which is similar to the findings of Flemming and Briggs (2007), despite failing to identify one relevant qualitative study. CINAHL had high yield across all the searches and failed to retrieve mixed-method research that included qualitative data. MEDLINE also identified all but one qualitative study, but had better retrieval of mixed-method research using qualitative data. Unlike Flemming and Briggs’ (2007) findings, systematic searching extended to incorporate CINAHL, EMBASE and MEDLINE databases may improve the yield of relevant studies when searching for qualitative PR-HRQL data, irrespective of methodology. When combined, these databases had the best yield of relevant studies. The reason for the inconsistency in findings may be because Flemming and Briggs’ (2007) research question was focused primarily on qualitative research specific to a nursing topic (living with a leg ulcer). This may have led to the increased topic relevance of searching in CINAHL as its focus is on nursing and allied health literature (Evans 2002).
Rather than using a methodology-based filter, searching using the PR-HRQL search strategy maximized yield of relevant studies reporting qualitative PR-HRQL data. For busy healthcare practitioners, the broad-based standard qualitative search strategy used in CINAHL was effective in identifying a majority of qualitative PR-HRQL research. This is a simple and easy method, although the disadvantage is that data reported in mixed methodology research will be missed. However, when undertaking a systematic review requiring a comprehensive and exhaustive search of the literature, our findings suggest that the subject-specific PR-HRQL search strategy run in CINAHL, MEDLINE and EMBASE is required. Although a large number of irrelevant studies will be retrieved, thus making the job of screening abstracts for inclusion in systematic reviews a laborious process, it does ensure that the literature search is exhaustive and that the evidence is based on all relevant information.
A simple broad methodology-based (qualitative) search strategy used in CINAHL was effective in identifying qualitative PR-HRQL research, although it failed to identify research reporting qualitative data within mixed method studies. This indicates the need for better indexing in electronic databases and publications of the methodology used and the outcomes measured in studies. The PR-HRQL search strategy identified all relevant studies reporting qualitative PR-HRQL data, regardless of methodology, suggesting that subject-based strategies are more effective in the retrieval of qualitative PR-HRQL data than methodology-based (qualitative) strategies. Rather than using methodology-based strategies, there is better yield of relevant studies when using a subject-based HRQL strategy, although the trade-off is high yield and low specificity, making comprehensive literature searching a laborious process.
This study had no external funding.
Conflict of interest
No conflict of interest has been declared by the authors.
CG, MB and JN were responsible for the study conception and design. CG performed the data collection and performed the data analysis. CG, JB, MB and JN were responsible for the drafting of the manuscript and made critical revisions to the paper for important intellectual content. JB provided statistical expertise. JN supervised the study. CG and JN provided other contributions.