A typology of reviews: an analysis of 14 review types and associated methodologies

Authors


Maria J. Grant, Salford Centre for Nursing, Midwifery and Collaborative Research, School of Nursing, University of Salford, Salford M6 6PU. E-mail: m.j.grant@salford.ac.uk

Abstract

Background and objectives:  The expansion of evidence-based practice across sectors has lead to an increasing variety of review types. However, the diversity of terminology used means that the full potential of these review types may be lost amongst a confusion of indistinct and misapplied terms. The objective of this study is to provide descriptive insight into the most common types of reviews, with illustrative examples from health and health information domains.

Methods:  Following scoping searches, an examination was made of the vocabulary associated with the literature of review and synthesis (literary warrant). A simple analytical framework—Search, AppraisaL, Synthesis and Analysis (SALSA)—was used to examine the main review types.

Results:  Fourteen review types and associated methodologies were analysed against the SALSA framework, illustrating the inputs and processes of each review type. A description of the key characteristics is given, together with perceived strengths and weaknesses. A limited number of review types are currently utilized within the health information domain.

Conclusions:  Few review types possess prescribed and explicit methodologies and many fall short of being mutually exclusive. Notwithstanding such limitations, this typology provides a valuable reference point for those commissioning, conducting, supporting or interpreting reviews, both within health information and the wider health care domain.

Background

The advent of evidence-based practice (EBP) in the early 1990s has seen the role of the health library and information worker in the ascendancy, with clinicians increasingly relying on health care literature in their decision making. With their knowledge of information sources and their skills to retrieve information to inform health care decisions, library and information sector workers have played,1 and indeed continue to play, an important role in assisting in the uptake of EBP principles and practice. It quickly became apparent that synthesized summaries of ‘all’ evidence within a particular domain would be required, in addition to the evidence from primary studies, if clinicians were to make truly informed decisions within a typical consultation. However, the review article of the time seemed ill-equipped to meet such a challenge. Medical review articles of the pre-EBP era were generally unsystematic and lacked formal statistical methods to derive best estimates of treatment effects from the available information. Consequently, they tended to reach conclusions that were biased and wrong.2

Archie Cochrane, a famous British epidemiologist, noted:

‘It is surely a great criticism of our profession that we have not organized a critical summary, by specialty or subspecialty, adapted periodically, of all relevant randomized controlled trials.’3

In answer to this challenge, the worldwide Cochrane Collaboration was formed in 1992 to provide an expanding resource of updateable systematic reviews of randomized controlled trials (RCTs) relating to health care. Thus began the modern incarnation of the review article, a tool that had for many centuries been the mainstay for updating scientific knowledge.

Rise of the review

While it is well established that, in the 18th century, James Lind was the instigator of the first reported RCT, a lesser-known fact records that he was probably the first to describe the systematic review method. Six years after his first RCT was published,4 Lind wrote:

‘As it is no easy matter to root out prejudices ... it became requisite to exhibit a full and impartial view of what had hitherto been published on the scurvy ... by which the sources of these mistakes may be detected. Indeed, before the subject could be set in a clear and proper light, it was necessary to remove a great deal of rubbish.’5

Gathering research, getting rid of rubbish and summarizing the best of what remains captures the essence of the science of systematic review. Nevertheless, although the need to synthesize research evidence has been recognized for well over two centuries, it was not until the 20th century that researchers began to develop explicit methods for this form of research. A fuller account of the rise of the discipline of research synthesis has been published by Iain Chalmers and other distinguished proponents.6 Recent years have seen recognition that the typical timescale commanded by the rigour of the systematic review process may be unsuited to the decision-making windows available to most policymakers. The need to trade off rigour and relevance has become a central theme to recent methodological developments and has led to a bewildering plethora of review designs developed to meet a variety of demands from the domains of research and policy.

An early example of a review was published in the non-health library and information sector,7 whilst the early 1980s saw the first of many manuscripts seeking to answer the eternal question of how best to undertake bibliographic instruction in the health sciences.8

Coinciding with the first Evidence-Based Library and Information Practice (EBLIP) conference in 2001, Booth proposed that the library and information science (LIS) sector could follow the trend within medical EBP by graduating to the development of more systematic reviews once a critical mass of rigorous studies has been attained.9 A recent evaluation of the evidence base has continued to sound such a call for establishing a solid evidence base within the LIS sector.10

To ‘review’ has been defined as: ‘To view, inspect, or examine a second time or again’.11

This definition broadly characterizes all review types now in existence. What remain largely unacknowledged are the subtle variations in the degree of process and rigour within the multifarious review types. Such variations are most clearly evidenced in the structures and methodologies that distinguish one review type from another. As more professions have adopted and adapted the systematic review method, the LIS sector, as elsewhere, has been afforded access to an ever-increasing variety of methods and techniques for summarizing the evidence base. The model of the systematic review of randomized controlled trials has limited potential within the LIS literature, given that no more than a score of such studies had been identified. This has necessitated the identification of a greater range of review types, opening up the prospect of summarizing case studies, qualitative research and even theoretical and conceptual published and unpublished outputs.

Different types of reviews

Analysis of download statistics from the Health Information and Libraries Journal electronic archive since 2006 indicates that reviews figure prominently amongst the most highly sought articles. This suggests that, as with medicine before it, the LIS sector values the opportunity to access already synthesized evidence in informing its practice. Indeed, in 2008, Ankem noted that there was evidence of systematic reviews making a ‘substantial contribution to medical library and information literature’.12

Ankem, in an evaluation of methods in systematic reviews and meta-analyses published between 1996 and 2006 in the library and information science sector, identified a total of eight manuscripts. These manuscripts, combined with reviews published in Health Information and Libraries Journal following an editorial commitment in 2007 to seek to publish a review in each future issue,13 provide an illuminating insight into the flourishing terminology synonymous with this genre. This terminology includes such terms or phrases as review of the evidence,14,15 comprehensive review,16 literature review,17 overview18 and systematic review.19–30 Given the importance evidence-based practice places upon the retrieval of appropriate information, such diverse terminology could, if unchecked, perpetuate a confusion of indistinct and misapplied terms.

Objective

The objective of this study is to provide a descriptive insight into the most common examples of review, illustrated by examples from health and health information domains.

Method

After initial scoping searches of the literature, the authors drew on their combined experience of 26 years of having worked with both the theory and practice of reviews in multifarious guises to examine the vocabulary used in the published literature, unpublished documents and other source material. The purpose was to determine the prevalent terminology; a process known as literary warrant.31 From this, common review types and their associated key attributes were identified and mapped against a Search, Appraisal, Synthesis and Analysis (SALSA) framework. Each review type was analysed, its characteristics were described and its perceived strengths and weaknesses were outlined. An example of each type of review was identified and selected, primarily for its usefulness in illustrating review characteristics. No judgement of quality is implied by each selection.

Results: characterizing types of review

Fourteen review types and associated methodologies were analysed against the SALSA framework (see Table 1). To inform the subsequent analysis, the same framework had been previously applied to 17 review manuscripts identified from combining those figuring in the Health Information and Libraries Journal review feature between 2007 and 2009 with papers cited in Ankem's Review of Reviews (see Table 2). A descriptive summary appears below.

Table 1. Main review types characterized by methods used
LabelDescriptionMethods used (SALSA)
SearchAppraisalSynthesisAnalysis
Critical reviewAims to demonstrate writer has extensively researched literature and critically evaluated its quality. Goes beyond mere description to include degree of analysis and conceptual innovation. Typically results in hypothesis or modelSeeks to identify most significant items in the fieldNo formal quality assessment. Attempts to evaluate according to contributionTypically narrative, perhaps conceptual or chronologicalSignificant component: seeks to identify conceptual contribution to embody existing or derive new theory
Literature reviewGeneric term: published materials that provide examination of recent or current literature. Can cover wide range of subjects at various levels of completeness and comprehensiveness. May include research findingsMay or may not include comprehensive searchingMay or may not include quality assessmentTypically narrativeAnalysis may be chronological, conceptual, thematic, etc.
Mapping review/ systematic mapMap out and categorize existing literature from which to commission further reviews and/or primary research by identifying gaps in research literatureCompleteness of searching determined by time/scope constraintsNo formal quality assessmentMay be graphical and tabularCharacterizes quantity and quality of literature, perhaps by study design and other key features. May identify need for primary or secondary research
Meta-analysisTechnique that statistically combines the results of quantitative studies to provide a more precise effect of the resultsAims for exhaustive, comprehensive searching. May use funnel plot to assess completenessQuality assessment may determine inclusion/exclusion and/or sensitivity analysesGraphical and tabular with narrative commentaryNumerical analysis of measures of effect assuming absence of heterogeneity
Mixed studies review/mixed methods reviewRefers to any combination of methods where one significant component is a literature review (usually systematic). Within a review context it refers to a combination of review approaches for example combining quantitative with qualitative research or outcome with process studiesRequires either very sensitive search to retrieve all studies or separately conceived quantitative and qualitative strategiesRequires either a generic appraisal instrument or separate appraisal processes with corresponding checklistsTypically both components will be presented as narrative and in tables. May also employ graphical means of integrating quantitative and qualitative studiesAnalysis may characterise both literatures and look for correlations between characteristics or use gap analysis to identify aspects absent in one literature but missing in the other
OverviewGeneric term: summary of the [medical] literature that attempts to survey the literature and describe its characteristicsMay or may not include comprehensive searching (depends whether systematic overview or not)May or may not include quality assessment (depends whether systematic overview or not)Synthesis depends on whether systematic or not. Typically narrative but may include tabular featuresAnalysis may be chronological, conceptual, thematic, etc.
Qualitative systematic review/qualitative evidence synthesisMethod for integrating or comparing the findings from qualitative studies. It looks for ‘themes’ or ‘constructs’ that lie in or across individual qualitative studiesMay employ selective or purposive samplingQuality assessment typically used to mediate messages not for inclusion/exclusionQualitative, narrative synthesisThematic analysis, may include conceptual models
Rapid reviewAssessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing researchCompleteness of searching determined by time constraintsTime-limited formal quality assessmentTypically narrative and tabularQuantities of literature and overall quality/direction of effect of literature
Scoping reviewPreliminary assessment of potential size and scope of available research literature. Aims to identify nature and extent of research evidence (usually including ongoing research)Completeness of searching determined by time/scope constraints. May include research in progressNo formal quality assessmentTypically tabular with some narrative commentaryCharacterizes quantity and quality of literature, perhaps by study design and other key features. Attempts to specify a viable review
State-of-the-art reviewTend to address more current matters in contrast to other combined retrospective and current approaches. May offer new perspectives on issue or point out area for further researchAims for comprehensive searching of current literatureNo formal quality assessmentTypically narrative, may have tabular accompanimentCurrent state of knowledge and priorities for future investigation and research
Systematic reviewSeeks to systematically search for, appraise and synthesis research evidence, often adhering to guidelines on the conduct of a reviewAims for exhaustive, comprehensive searchingQuality assessment may determine inclusion/exclusionTypically narrative with tabular accompanimentWhat is known; recommendations for practice. What remains unknown; uncertainty around findings, recommendations for future research
Systematic search and reviewCombines strengths of critical review with a comprehensive search process. Typically addresses broad questions to produce ‘best evidence synthesis’Aims for exhaustive, comprehensive searchingMay or may not include quality assessmentMinimal narrative, tabular summary of studiesWhat is known; recommendations for practice. Limitations
Systematized reviewAttempt to include elements of systematic review process while stopping short of systematic review. Typically conducted as postgraduate student assignmentMay or may not include comprehensive searchingMay or may not include quality assessmentTypically narrative with tabular accompanimentWhat is known; uncertainty around findings; limitations of methodology
Umbrella reviewSpecifically refers to review compiling evidence from multiple reviews into one accessible and usable document. Focuses on broad condition or problem for which there are competing interventions and highlights reviews that address these interventions and their resultsIdentification of component reviews, but no search for primary studiesQuality assessment of studies within component reviews and/or of reviews themselvesGraphical and tabular with narrative commentaryWhat is known; recommendations for practice. What remains unknown; recommendations for future research
Table 2. Reviews from Health Information and Libraries Journal review (2007–2009) feature or Ankem's review of reviews (2008)
Authors (year)DescriptionNo. of included studiesMethods described (SALSA)
SearchAppraisalSynthesisAnalysis
Ankem (2006)19Systematic review of the research literature110 studies3 databasesNoneNarrative and tabularMeta-analysis and descriptive statistics
Booth et al. (2009)21Systematic review2914 databasesStandard checklists of quality assessment criteria for different study designsQualitativeThematic using32
Boulos et al. (2007)18OverviewNot specifiedNot specifiedNoneNarrativeDescriptive
Brettle (2003)22Systematic review of the literature243 databasesInstrument developed by Health Care Practice R&D Unit (University of Salford)Narrative and tabularDescriptive
Brettle (2007)23Systematic review547 databasesNoneNarrative and tabularThematic and descriptive statistics
Brown (2008)24Systematic review20 peer reviewed, 19 magazine, 146 newspaper and 141 university newspaper articles23 databasesArticles from popular press, magazine and newspaper articles reviewed for types of information publishedNarrative and tabularChronological and thematic
Childs et al. (2005)25Systematic review of the literature578 databasesNoneNarrativeDescriptive
Davies (2007)14Review of the evidenceNot specified (34 from table)3 databasesNoneNarrative and tabularDescriptive
Fanner & Urquhart (2008)26Systematic reviewNot specified9 databasesNoneNarrativeDescriptive
Grant (2007)27Systematic review13 lisa NoneNarrativeThematic
Hall & Walton (2004)17Literature review237 databasesNoneNarrativeDescriptive
Koufogiannakis & Wiebe (2006)28Systematic review and meta-analysis5515 databasesGlasgow checklistNarrativeMeta-analysis and framework analysis
Rossall et al. (2008)15Review of the evidenceNot specifiedNot specifiedNoneNarrativeDescriptive
Wagner & Byrd (2004)29Systematic review355 databasesCriteria for medical informatics evaluative studies plus additional criteriaNarrative and tabularDescriptive
Ward et al. (2008)16Comprehensive review of the research literature7912 databasesNoneNarrativeThematic
Weightman & Williamson (2005)30Systematic review287 databasesInternationally accepted criteria from previously published literatureNarrative and tabularDescriptive
Beverley & Winning (2003)20Systematic review of the literatureSeventeen (16 unique) evaluative and 33 descriptive studies16 databasesCriSTAL: Critical Skills Training in Appraisal for Librarians ChecklistNarrative and tabularDescriptive

Critical review

Description.  A critical review aims to demonstrate that the writer has extensively researched the literature and critically evaluated its quality. It goes beyond mere description of identified articles and includes a degree of analysis and conceptual innovation. An effective critical review presents, analyses and synthesizes material from diverse sources. Its product perhaps most easily identifies it—typically manifest in a hypothesis or a model, not an answer. The resultant model may constitute a synthesis of existing models or schools of thought or it may be a completely new interpretation of the existing data.

Perceived strengths.  The ‘critical’ component of this type of review is key to its value. Under normal circumstances, conceptual innovation develops through a process of evolution or accretion, with each successive version adding to its predecessors. A critical review provides an opportunity to ‘take stock’ and evaluate what is of value from the previous body of work. It may also attempt to resolve competing schools of thought. As such, it may provide a ‘launch pad’ for a new phase of conceptual development and subsequent ‘testing’.

Perceived weaknesses.  Critical reviews do not typically demonstrate the systematicity of other more structured approaches to the literature. While there is considerable value in trying to identify all the available literature on a topic under review, there is no formal requirement to present methods of the search, synthesis and analysis explicitly. The emphasis is on the conceptual contribution of each item of included literature, not on formal quality assessment. While such a review does serve to aggregate the literature on a topic, the interpretative elements are necessarily subjective and the resulting product is the starting point for further evaluation, not an endpoint in itself.

Example.  Kulviwat, S., Guo, C. & Engchanil, N. Determinants of online information search: a critical review and assessment. Internet Research: Electronic Networking Applications and Policy 2004, 14(3), 245–53.

Literature review

Description.  According to the Medical Subject Headings (MeSH) scope note, a literature review describes ‘Published materials which provide an examination of recent or current literature. Review articles can cover a wide range of subject matter at various levels of completeness and comprehensiveness based on analyses of literature that may include research findings’.33 This is necessarily a very broad description making it difficult to generalize. However, common characteristics are that a literature review reviews published literature, implying that included materials possess some degree of permanence and, possibly, have been subject to a peer-review process. Generally, a literature review involves some process for identifying materials for potential inclusion—whether or not requiring a formal literature search—for selecting included materials, for synthesizing them in textual, tabular or graphical form and for making some analysis of their contribution or value.

Perceived strengths.  The literature review method seeks to identify what has been accomplished previously, allowing for consolidation, for building on previous work, for summation, for avoiding duplication and for identifying omissions or gaps.

Perceived weaknesses.  Literature reviews lack an explicit intent to maximize scope or analyse data collected. Any conclusions they may reach are therefore open to bias from the potential to omit, perhaps inadvertently, significant sections of the literature or by not questioning the validity of statements made. Additionally, authors may only select literature that supports their world view, lending undue credence to a preferred hypothesis.

Example.  Hall, A. & Walton, G. Information overload within the health care system: a literature review. Health Information and Libraries Journal 2004, 21(2), 102–8.

Mapping review/systematic map

Description.  This type of review has been developed and refined by the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre), Institute of Education, London, to map out and categorize existing literature on a particular topic,34 identifying gaps in research literature from which to commission further reviews and/or primary research. Mapping reviews can be distinguished from scoping reviews (see below) because the subsequent outcome may involve either further review work or primary research and this outcome is not known beforehand.

Perceived strengths.  Mapping reviews enable the contextualization of in-depth systematic literature reviews within broader literature and identification of gaps in the evidence base. They are a valuable tool in offering policymakers, practitioners and researchers an explicit and transparent means of identifying narrower policy and practice-relevant review questions. Systematic maps may characterize studies in other ways such as according to theoretical perspective, population group or the setting within which studies were undertaken. In addition to describing the research field, a systematic map can also provide the basis for an informed decision about whether to undertake the in-depth review and synthesis on all of the studies or just a subset. The map can show whether the total population of studies is sufficiently similar for a coherent synthesis. It can also establish whether these studies will help answer the review question and address pragmatic considerations about the resources available to complete the review.

Perceived weaknesses.  Mapping reviews are necessarily time constrained and lack the synthesis and analysis of more considered approaches. Studies may be characterized at a broad descriptive level and thus oversimplify the picture or mask considerable variation (heterogeneity) between studies and their findings—depending on the degree of specificity of the coding process. Mapping reviews do not usually include a quality assessment process; characterizing studies only on the basis of study design.

Example.  Gough, D., Kiwan, D., Sutcliffe, K., Simpson, D. & Houghton, N. A Systematic Map and Synthesis Review of the Effectiveness of Personal Development Planning for Improving Student Learning. London: EPPI-Centre, Social Science Research Unit, 2003.

Meta-analysis

Description.  Meta-analysis is ‘a technique that statistically combines the results of quantitative studies to provide a more precise effect of the results’.35 Although many systematic reviews present their results without statistically combining data in this way, a good systematic review is essential to a meta-analysis of the literature. For a meta-analysis to be valid requires all included studies to be sufficiently similar. This will include such characteristics as the population being studied, the intervention being explored and the comparison being made. Most importantly, it requires that the same measure or outcome be measured in the same way at the same time intervals.

Perceived strengths.  From its early origins in the social sciences meta-analysis has grown in popularity, primarily because of the facility to take individual studies, not in themselves sufficient to impact on practice, and to assimilate them into a composite evidence base. Small or inconclusive studies lacking in statistical significance can nevertheless make a contribution to the larger picture. In addition, such compilations are time efficient for decision makers, particularly when compared with the time taken to review scattered individual studies,

Perceived weaknesses.  Critics of meta-analysis argue at the inappropriateness of combining ‘apples and oranges’, i.e. studies that are not sufficiently similar. While such accusations persist, it must be acknowledged that this is not a criticism of meta-analysis per se but rather of the inappropriate use of meta-analysis. Nevertheless, one essential fact remains—a meta-analysis cannot be better than its included studies allow.

Example.  Saxton, M. L. Reference service evaluation and meta-analysis: findings and methodological issues. Library Quarterly 1997, 67(3), 267–89.

Narrative review

See Literature review.

Mixed studies review/mixed methods review

Description.  Generally speaking a Mixed Methods Review can refer to any combination of methods where at least one of the components is a literature (usually systematic) review. For example it might include a systematic review accompanied by interviews or by a stakeholder consultation. Within the specific context of this paper it most frequently refers to the bringing together of a quantitative effectiveness review and a qualitative review on attitudes to the intervention or on implementation issues. For example the EPPI-Centre at the University of London has successfully developed its own methods for bringing together outcome studies from health promotion with studies that describe the actual processes that were used.36 Such studies attempt to bring the ‘what works’ of the former together with the ‘how and why does it work’ of the latter to start to address the more complex issue of ‘what works under which circumstances’.

Perceived strengths.  The Mixed Methods Review is seen to capitalise on the corresponding weaknesses of the ‘what works’ effectiveness systematic review and alternative more theory driven approaches. For policy makers the attraction of being able to arrive at a more holistic understanding of a particular intervention or condition is compelling. This does however depend upon the review team being able to demonstrate the added value of the combined approach and also being equipped to meet the challenge of delivering a larger review enterprise within the same meaningful decision-making timeframe. Unlike single method reviews, such as most of those in this typology, mixed methods reviews also provide a potentially more complete picture of the research landscape in a specific topic area.

Perceived weaknesses.  Mixed Methods reviews may compound the methodological challenges of appraising and synthesising both quantitative and qualitative research with the added difficulty of integrating the resultant products. Currently no consensus exists on the point at which quantitative and qualitative components should be integrated. For example should the qualitative component precede the qualitative thus informing the resultant question? Alternatively should the qualitative component follow the quantitative to increase understanding of how the intervention works or of issues relating to its implementation or to adherence? Clearly if both components proceed in parallel there are issues as to when they should opportunely be brought together—either at a significant summative point or iteratively to help inform the ongoing conduct of both components. More significant than such pragmatic decisions are more complex issues regarding the theoretical and methodological challenges of bringing together differently structured studies, addressing different though related questions and conducted within different paradigms.

Example.  Shepherd, J., Harden, A., Rees, R., Brunton, G., Garcia, J., Oliver, S. & Oakley, A. Young people and healthy eating: a systematic review of research on barriers and facilitators. Health Education Research 2006, 21(2), 239–57.

Overview

Description.  An overview is a generic term used for ‘any summary of the [medical] literature’37 that attempts to survey the literature and describe its characteristics. As such, it can be used for many different types of literature review, with differing degrees of systematicity. In the early days of systematic reviews, the term ‘overview’ was used synonymously with ‘systematic review’ to describe that particular approach. As a consequence, the value of the term within a typology is debatable, notwithstanding the fact that the overall intent conveyed by the term continues to have considerable appeal to readers.

Perceived strengths.  Overviews can provide a broad and often comprehensive summation of a topic area and, as such, have value for those coming to a subject for the first time.

Perceived weaknesses.  As mentioned above, the term ‘overview’ is frequently used as a non-discriminant word for reviews of varying rigour and quality. For this reason, the Cochrane Collaboration has chosen to differentiate ‘systematic overview’, used as a synonym for ‘systematic review’ (see below), from other types of overview that typically lack both systematic methods and explicit reporting.

Example.  Boulos, M., Kamel, N., Hetherington, L. & Wheeler, S. Second Life: an overview of the potential of 3-D virtual worlds in medical and health education. Health Information and Libraries Journal 2007, 24(4), 233–45.

Qualitative systematic review/qualitative evidence synthesis

Description.  Qualitative systematic review is ‘a method for integrating or comparing the findings from qualitative studies. The accumulated knowledge resulting from this process may lead to the development of a new theory, an overarching “narrative”, a wider generalization or an “interpretative translation”’. It ‘looks for “themes” or “constructs” that lie in or across individual qualitative studies. The goal ... is not aggregative in the sense of “adding studies together,” as with a meta-analysis. On the contrary, it is interpretative in broadening understanding of a particular phenomenon’.38

There remains considerable confusion regarding the phrase ‘qualitative systematic review’. This is a historical legacy of the systematic review movement whereby ‘when the results of primary studies are summarized but not statistically combined, the review may be called a qualitative systematic review’.39 This definition (i.e. that of a systematic review where meta-analysis is not possible) continues to persist, particularly in the literature of analgesia and pain management. For this, and other reasons, the Cochrane Collaboration's Qualitative Research Methods Group promotes ‘qualitative evidence synthesis’ as the terminology of choice. Other terms encountered in the literature include the tautological ‘qualitative meta-synthesis’ and the misleading ‘meta-ethnography’ (describing a method that can be adapted to interpreting many types of qualitative research, not simply ethnographies).

Perceived strengths.  Qualitative systematic reviews can be used: to explore barriers and facilitators to the delivery and uptake of services; for an exploration of user views; to investigate perceptions of new roles, from the point of view of either those filling the roles or those with whom the post holder interacts; and to inform the prioritization of services where evidence on effectiveness is equivocal and preferences and attitudes thus become the determining factors.40 This type of review therefore possesses a considerable strength in complementing the research evidence with the other two essential components of evidence practice, i.e. user-reported and practitioner-observed considerations.41 Provided such insights are generalizable, findings from qualitative research may be more powerful than isolated comments from local questionnaires or surveys.

Perceived weaknesses.  Methods for qualitative systematic review are still in their infancy and there is considerable debate about when specific methods or approaches are appropriate. For example, opinions differ as to whether comprehensive search strategies are required, identifying as many relevant qualitative research studies as possible, or whether what is being sought is a holistic interpretation of a phenomenon. If the latter is the case, then a more selective search approach may be acceptable as long as the method of sampling papers for inclusion is appropriate. Such debates centre on whether the dominant model for qualitative evidence synthesis is the classic systematic review method or whether it is more appropriate to adapt and adopt concepts from primary qualitative research (e.g. grounded theory, theoretical saturation, purposive sampling etc.). Nevertheless, emerging guidance, now included in the Cochrane Collaboration's handbook42 and Centre for Reviews and Dissemination methodologies is gradually moving towards greater consensus.43

Example.  Duggan, F. & Banwell, L. Constructing a model of effective information dissemination in a crisis. Information Research 2004, 9(3). Available from: http://InformationR.net/ir/9-3/paper178.html.

Rapid review

Description.  The methods of rapid review, seen initially by some as an unwelcome concession to the need for evidence-based decisions within a policymaker's time frame, have recently gained legitimacy in the form of Rapid Evidence Assessments. This method is now proposed by the Government Social Research website as a means of providing an ‘assessment of what is already known about a policy or practice issue, by using systematic review methods to search and critically appraise existing research’.44

Perceived strengths.  Rapid reviews and rapid evidence assessments seek to be ‘Quick but Not Dirty’: ‘They aim to be rigorous and explicit in method and thus systematic but make concessions to the breadth or depth of the process by limiting particular aspects of the systematic review process’.45 The methodology identifies several legitimate techniques that may be used to shorten the timescale. These include carefully focusing the question, using broader or less sophisticated search strategies, conducting a review of reviews, restricting the amount of grey literature, extracting only key variables and performing only ‘simple’ quality appraisal. The reviewer chooses which stages to limit and then explicitly reports the likely effect of such a method.

Perceived weaknesses.  Curtailing the duration of the review process runs the risk of introducing bias. This is true for any review process, but this risk is accentuated when measures are fast-tracked or even sidestepped. Limiting the time taken to search may result in publication bias, limiting appraisal or quality assessment may place a disproportionate emphasis on poorer quality research, while a lack of attention to synthesis may overlook inconsistencies or contradictions. Producing the evidence within a rapid timescale has to be offset against this risk of increased bias. Documenting the methodology and highlighting its limitations is one way of militating against such biases. Furthermore, inadequate attention to the question to be addressed or the quantity and quality of literature that exists in a topic may result in a very precise answer to the wrong question or an inconclusive answer to an ill-conceived question.

Example.  Lacey Bryant, S. & Gray, A. Demonstrating the positive impact of information support on patient care in primary care: a rapid literature review. Health Information and Libraries Journal 2006, 23(2), 118–25.

Scoping review

Description.  This type of review provides a preliminary assessment of the potential size and scope of available research literature. It aims to identify the nature and extent of research evidence (usually including ongoing research).

Perceived strengths.  Scoping reviews are able to inform policymakers as to whether a full systematic review is needed. They share several characteristics of the systematic review in attempting to be systematic, transparent and replicable.

Perceived weaknesses.  Scoping reviews cannot usually be regarded as a final output in their own right, primarily because limitations in their rigour and limitations in their duration mean that they hold the potential for bias. In particular, they typically do not include a process of quality assessment. There is thus a danger that the existence of studies rather than their intrinsic quality is used as the basis for conclusions. As a consequence, their findings cannot be used to recommend policy/practice.

Example.  Weeks, L. C. & Strudsholm, T. A scoping review of research on complementary and alternative medicine (CAM) and the mass media: looking back, moving forward. BMC Complementary and Alternative Medicine 2008, 19(8), 43.

State-of-the-art review

Description.  State-of-the-art reviews are specifically mentioned by scope notes of the MeSH database for the entries under Review, Literature as a Topic33 and Review [Publication Type].46 As such they represent a subtype of the more generic ‘Literature Review’. They are characterized as follows: ‘State-of-the-art reviews tend to address more current matters’47 in contrast to the combined retrospective and current approaches of the “literature review”’. The review may offer new perspectives on an issue or highlight an area in need of further research.

Perceived strengths.  State-of-the-art reviews are of considerable value for those new to an area or for those seeking to identify potential opportunities for contemporary research. Instead of having to read multiple articles describing specific developments, the reader can derive a feel for the quantity and main characteristics of a topic from a single review article. An entire body of publishing output has developed around these acknowledged advantages both within Medicine (e.g. the Annual Review of ... Neurology, Cardiology etc.) and within Library and Information Science (e.g. the Annual Review of Information Science and Technology and the various New Review titles, including New Review of Information and Library Research and New Review of Academic Librarianship).

Perceived weaknesses.  Limitations of the state-of-the-art review are common to any ‘cross-sectional’ method of surveying a field. Such methods are time bound and may distort the overall picture of development of a field. For example, if a topic has been extensively covered by research in the past, but has temporarily gone into ‘remission’, its importance may be under-represented simply because it lies outside the agreed time horizon covered by the state-of-the-art review article. Alternatively, a subject expert may simply provide a particularly idiosyncratic and personal perspective on current and future priorities.

Example.  Bath, P. A. Data mining in health and medical information. Annual Review of Information Science and Technology 2004, 38, 331–69.

Systematic review

Description.  The best known type of review, a systematic review seeks to systematically search for, appraise and synthesis research evidence, often adhering to the guidelines on the conduct of a review provided by the Cochrane Collaboration47 or the NHS Centre for Reviews and Dissemination.43 It is transparent in the reporting of its methods to facilitate others to replicate the process.

Perceived strengths.  Systematic reviews seek to draw together all known knowledge on a topic area. In recent years, with the establishment of organizations such as Campbell Collaboration and the Cochrane Qualitative Methods Group, there has been a noticeable shift towards the inclusion of a wider range of study designs incorporating quantitative, qualitative and mixed method studies.

Perceived weaknesses.  Restricting studies for inclusion to a single study design such as randomized controlled trials, as practised in the early years of the Cochrane Collaboration, can limit the application of this methodology to providing insights about effectiveness rather than seeking answers to more complex search questions; for example, why a particular intervention is effective.

Example.  Weightman, A. L. & Williamson, J. The value and impact of information provided through library services for patient care: a systematic review. Health Information and Libraries Journal 2005, 22(1), 4–25.

Systematic search and review

Description.  Such a review combines the strengths of a critical review with a comprehensive search process. Typically, this type of review addresses broad questions and the result is a ‘best evidence synthesis’.

Perceived strengths.  The broad scope of this type of review means that it often incorporates multiple study types rather than focusing on a single preferred study design. They can thus provide a much more complete picture of the prevalence of research on a topic than a systematic review limited to randomized controlled trials. Subjecting the resultant literature to critical review or critique, albeit informally without the use of a standardized tool or checklist, provides a useful evaluative component. However, this presupposes that all included articles are being assessed and valued against the same underlying criteria, a fact that cannot usually be determined from the review's methods.

Perceived weaknesses.  While the initial search process may meet the exacting requirements of a systematic review, the subsequent critical review may be prone to some of the limitations of the traditional review. Without explicit inclusion and exclusion criteria and a clearly defined process of synthesis, the result may be a subjective selection of research to support a particular line of argument.

Example.  Carroll, L. J., Cassidy, J. D., Peloso, P. M., Garritty, C. & Giles-Smith, L. WHO Collaborating Centre Task Force on mild traumatic brain injury. Systematic search and review procedures: results of the WHO Collaborating Centre Task Force on mild traumatic brain injury. Journal of Rehabilitation Medicine 2004, 43(Suppl.), 11–4.

Systematized review

Description.  Systematized reviews attempt to include one or more elements of the systematic review process while stopping short of claiming that the resultant output is a systematic review. They may identify themselves parenthetically as a ‘systematic’ review. Systematized reviews are typically conducted as a postgraduate student assignment, in recognition that they are not able to draw upon the resources required for a full systematic review (such as two reviewers).

Perceived strengths.  Typically, the search stage possesses the most easily identified elements of systematicity and an author may conduct a comprehensive search but do little more than simply catalogue included studies. Conversely, the author might only search one or more databases and then code and analyse all retrieved results in a systematic manner. The resulting output ‘models’ the systematic review process and allows the author to demonstrate an awareness of the entire process and technical proficiency in the component steps, However, such a review necessarily falls short of being able to claim the comprehensiveness so fundamental to the systematic review method. Such reviews may form the basis for a more extensive piece of work either as a dissertation or a fully funded research project.

Perceived weaknesses.  For such reviews quality assessment and synthesis may be less identifiable. This means that these processes are not described, that they are modelled using a small set of eligible articles or that they are missing entirely. While the attempt at systematicity is to be welcomed, such reviews do possess a greater likelihood of bias than those that adhere more strictly to guidelines on the conduct of systematic reviews (see above). Completion of the academic requirements for the review is prioritized over methodological considerations.

Example.  Cornet, R. & de Keizer, N. Forty years of SNOMED: a literature review. BMC Medical Informatics and Decision Making 2008, 8(Suppl. 1), S2.

Umbrella review

Description.  The need for the umbrella review was first identified as a consequence of the activities of the Cochrane Collaboration. However, as systematic reviews become more plentiful, there is the potential for greater use of such overarching reviews as a mechanism for aggregating findings from several reviews that address specific questions. It specifically refers to a review ‘compiling evidence from multiple Cochrane reviews into one accessible and usable document’.48 Each umbrella review focuses on a broad condition or problem for which there are two or more potential interventions and highlights reviews that address these potential interventions and their results. To illustrate an umbrella review on virtual reference services might variously incorporate findings from several more specific systematic reviews on e-mail, chat or videoconference services.

Perceived strengths.  Umbrella reviews were initially conceived as a ‘friendly front end’ to The Cochrane Library, allowing the reader a quick overview (and an exhaustive list) of reviews relevant to the decision at hand. More generally, they are a response, and potential solution, to the perennial dilemma reviewers face regarding ‘lumping’ versus ‘splitting’, i.e. whether the needs of a particular field or area are best addressed by a broad review that covers multiple interventions at the cost of lost detail and specificity or by a succession of focused reviews that address specific comparisons at the risk of fragmenting the overall picture.

Perceived weaknesses.  The main weakness of an umbrella review is a logistic one. For an umbrella review to be truly useful requires the pre-existence of the narrower component reviews. As such, it is currently not feasible for many areas of library and information practice. However, the potential remains—for example, to combine reviews of the many types of health library outreach: clinical librarians; primary care outreach librarians; information skills trainers; to identify the circumstances under which a library commissioner favours one approach over a competing alternative.

Example.  Seida, J. K., Ospina, M. B., Karkhaneh, M., Hartling, L., Smith, V. & Clark, B. Systematic reviews of psychosocial interventions for autism: an umbrella review. Developmental Medicine and Child Neurology 2009, 51(2), 95–104.

Discussion

Three main approaches were identified when investigating the potential to characterize or typify reviews. The most obvious, and least satisfactory, is to rely on the labels and terminology used by the authors themselves. However, the considerable variation in use of terminology, with both overlap and the difficulty in distinguishing between review types, make such an approach unfeasible. A second approach, advanced by the Government Social Research's Rapid Evidence Assessment (REA) toolkit,44 focuses on inputs, specifically the time taken to complete each type of review. Again, this was not considered appropriate in the development of the typology presented in this paper, as the time taken for a review depends on numerous factors, beyond the type of review. These include, but are not exclusive to, the resources available, the quantity and quality of the literature, and the expertise or experience of the reviewers. The final approach, in contrast to a methodology based on inputs, is to focus on the tangible processes required in completing a review, namely search, appraisal, synthesis and analysis, embodied in the SALSA framework. This approach relies neither on terminology nor on inputs but, in a manner congruent with the principles of evidence-based practice, considers the nature and extent of the review processes as embodied in the description of the methodology.

Consideration of the 14 review types and associated methodologies reveals that, whilst labels may supply a pragmatic ‘shorthand’ for authors, there are frequent inconsistencies or overlaps between the descriptions of nominally different review types. Currently, there is no internationally agreed set of discrete, coherent and mutually exclusive review types. The authors contend that the only pragmatic way to identify to which of these various types a particular review belongs is to inspect closely the four main processes associated with that review's development. In this way, a clearer understanding of the distinguishing features of each review type can be built up within the systematic review community through both direct comparison and emerging precedent.

The rising value of reviews

Several drivers are leading to a growth in the number of reviews published within, and beyond, the health LIS sector. Firstly, there is a greater overall awareness of the general technique and its potential benefits in providing a synthesis of literature in a particular area, possibly with a view to informing local practice. Secondly, calls are increasingly being made at a national and international level to develop the LIS evidence base, both in terms of primary/original research and secondary/synthesized research in the form of reviews. Finally, there is mounting recognition that the systematic consideration of previously published research may be a more effective way of targeting funds than simply commissioning further primary research.

The typology presented in this paper provides an explicit basis for those involved in commissioning reviews, and those involved in delivering to such commissions, to gain a clear understanding of what is being requested and the resources that will be required to meet the specification.

Using reviews

For the LIS evidence base to be transformed from its current embryonic state to provide a firm foundation on which to base professional practice requires expansion of both the breadth and quality of the evidence being created. Whilst LIS practitioners may be more predisposed than other professions to the inherent advantage of using pre-existing evidence, the potential panacea of reviews and their associated methodologies remains a distant prospect. However, whether the evidence takes the form of primary or secondary studies, it is equally important to undertake an appraisal of quality. This should consider both its robustness (validity and reliability) and its relevance to the local context (applicability). Work on developing critical appraisal skills to inform the practice of LIS workers was first reported in 1999.49 At that time, it was noted that information workers face significant barriers with regard to lack of knowledge of research methodology and statistics. From such small beginnings has grown a more widely spread awareness of critical appraisal and ongoing development of these skills. One study undertaken in the north-west of England reports that, of 55 librarians who responded to their survey, a significant 86% had attended appraisal skills training.50

Tools for appraising review methodologies have been published by the Critical Appraisal Skills Programme (CASP).51 However, the diversity of review methodologies characterized in the typology above suggests that it is inappropriate to try to assess reviews using a ‘one size fits all’ approach. Mapping existing and future types of health information review against the SALSA framework can assist LIS workers in identifying the inputs, processes, strengths and deficiencies that accompany each review type.

For the scale of work to be manageable, particularly within the often uncharted territories of LIS research, reviews generally seek to address tightly defined research questions. This may not always be appropriate to the type of reviews most needed by the LIS profession. Thus, it is important that practitioners recognize and develop an awareness of the broader context and evidence base within which a particular review exists. Judgement could then be made on the basis of ‘fitness for purpose’ and not against a single ‘gold standard’ of what a review should or should not be. A case in point is the state-of-the-art review, that is, a review that offers new perspectives on an issue or highlights areas in need of further research. If an expert in a given field has written a state-of-the-art review, it is likely to provide a fairly accurate representation of current knowledge and future priorities for research. Depending on the extent of communication within the chosen field, it may also capture an informal consensus among fellow experts. Nevertheless, by virtue of its currency, a state-of-the-art review may in actuality be reporting a supposed gap in the evidence that is already being, or has already been, addressed, but has yet to be published. Clearly, a reliance exclusively on published literature in this specific context provides a significant limitation to the usefulness of an otherwise valuable type of review. ‘Fitness for purpose’ requires that, if such a review is being used in the context of planning and commissioning research, its use be supplemented by interrogation of research registers and contact with experts.

Notwithstanding the potential of many of the other types of review identified in the typology presented, the relative infancy of review activities within the LIS sector means that significant gaps in the evidence base remain to be addressed by now accepted ‘orthodox’ systematic review methods. In such instances, a LIS worker may choose to develop a systematic review, either singularly or within a team.

Developing reviews

A growing literature attests to the roles LIS workers can undertake within the review development process. This is particularly apparent in relation to systematic reviews, perhaps because their clear and structured methodology makes them prime candidates for input from information specialists. The US Medical Library Association (MLA) has acknowledged that librarians are increasingly invited to join research teams to provide an expert searching component to the team.52 However, although LIS professionals have adopted diverse roles within the systematic review context, as the MLA statement suggests, these are primarily of a supportive nature. Typically, LIS professionals contribute to the data collection and data management phases of a review, withdrawing from the team once the literature is identified and obtained in readiness for its appraisal, extraction and synthesis. One such collaborative venture is described by Swinkels et al.,53 who outline how the relationship between academic, clinician and librarian can lead to iterative searching. Elsewhere, Harris reports on extending the role of expert searcher to involve the LIS worker in preparing the outputs from reviews.54 Grant et al.55 extol the benefits of LIS worker involvement in a review, citing the opportunities for capability building to be gained by all involved through peer teaching and learning at all stages of a review cycle. They further identify the potential to transfer these skills within an evidence-based library and information practice context. Beverley et al. report a systematic review where information professionals participated effectively in all aspects of preparing a review56—a model repeated in a more recent review by Booth et al.21 Brettle exemplifies a natural progression of these latter approaches in describing how a group of health librarians in the north-west of England are using the process of undertaking a systematic review on the most effective methods of evaluating clinical librarian services as a means of developing their collective research skills.57

Conclusion

The librarian or information specialist who seeks to use or to develop reviews, either individually or as a team, encounters considerable variation in the associated terminology. Whilst it is easy to run the risk of stereotyping or over-simplification when attempting to characterize the main aspects of variation, at this point in time, any attempt at typology is necessarily descriptive. Only a handful of review types possess prescribed and explicit methodologies and many of the labels used fall short of being mutually exclusive. In lieu of internationally recognized review definitions, the typology reported here acknowledges that there is a lack of unique distinguishing features for the most common review types, whilst highlighting that some common features do exist.

LIS workers, whether involved in appraising published reviews or in the not insignificant undertaking of developing a review of their own, should pay particular attention to the importance of clear and transparent descriptions reporting individual methods. The typology of reviews presented in this paper provides a potentially valuable resource in informing how LIS workers may seek to approach such an undertaking. It will also provide a basis for those commissioning reviews, conducting reviews or supporting reviews to gain an improved understanding of what is required. In furthering the dual objectives of evidence-based library and information practice, in contributing both to the evidence-based practice of LIS workers themselves and to supporting the practice of others, the wide variety of review types identified holds the potential to further transform the health information landscape.

Key Messages

Implications for Policy

  • • Given the current overlap of methodologies, there is a need for an internationally agreed set of discrete, coherent and mutually exclusive review types.
  • • An agreed typology can provide an explicit basis for those involved in commissioning reviews, and those involved in delivering to such commissions, to gain a clear understanding of what is required and resources needed to meet the specification.

Implications for Practice

  • • Fourteen review types are presented together with their associated methodologies, an accompanying description, analysis of their perceived strengths and weaknesses and a reference to a selected example.
  • • The Search, AppraisaL, Synthesis and Analysis (SALSA) framework can help library and information science (LIS) workers to identify the inputs and processes, strengths and deficiencies, which characterize the main phases of each review type.
  • • This typology provides a valuable resource to inform how LIS workers might approach the appraisal or development of a health information review.

Ancillary