Qualitative synthesis and systematic review in health professions education
Margaret Bearman, HealthPEER (Health Professional Education and Education Research), Monash University, G08 Building 13c, Clayton Campus, Melbourne, Victoria 3800, Australia. Tel: 00 61 3 9902 0410; E-mail: email@example.com
Context Formal qualitative synthesis is the process of pooling qualitative and mixed-method research data, and then drawing conclusions regarding the collective meaning of the research. Qualitative synthesis is regularly used within systematic reviews in the health professions literature, although such use has been heavily debated in the general literature. This controversy arises in part from the inherent tensions found when generalisations are derived from in-depth studies that are heavily context-dependent.
Methods We explore three representative qualitative synthesis methodologies: thematic analysis; meta-ethnography, and realist synthesis. These can be understood across two dimensions: integrative to interpretative, and idealist to realist. Three examples are used to illustrate the relative strengths and limitations of these approaches.
Discussion Against a backdrop of controversy and diverse methodologies, readers must take a critical stand when reading literature reviews that use qualitative synthesis to derive their findings. We argue that notions of qualitative rigour such as transparency and acknowledgment of the researchers’ stance should be applied to qualitative synthesis.
qualitative (adj.): ‘relating to or measured by quality’1
synthesis (n.): ‘the combination of components to form a connected whole’1
Readers of health professional literature rely on qualitative synthesis within literature reviews to assist in understanding the implications of the collective research. Qualitative syntheses share the subjective or ‘critical’2 nature of a traditional ‘narrative’ review. However, they can be distinguished from traditional reviews, as more structured processes, and are often set within, or draw from, a systematic review framework. Systematic review methodology, which is over two decades old, uses structured and transparent processes for collecting, assessing and synthesising the literature.3 One of the key elements is the formal synthesis of the studies included, which is intended to support understanding of the combined implication of the ever-increasing evidence. Synthesis methodologies in systematic reviews were originally, and are still predominantly, numerical methodologies such as meta-analysis. Many within the health professional education community regard systematic review as the reference standard for a literature review in health professional education, and educators and academics look to systematic reviews to provide conclusions from the literature to inform teaching and research. Interestingly, for a technique that is commonly used within health professional education systematic reviews, discussion of qualitative synthesis methodologies is limited.
We define qualitative synthesis as any methodology whereby study findings are systematically interpreted through a series of expert judgements to represent the meaning of the collected work. In a qualitative synthesis, the findings of qualitative studies – and sometimes mixed-methods and quantitative research – are pooled. Judgement-based qualitative methodologies are used to draw conclusions regarding the collective meanings of this pool of research. This is necessarily a complex and multifaceted process, which often deals with widely variant sources, and the methodological challenge of conducting qualitative synthesis is well acknowledged in the health policy literature.4–6 There is a wide range of qualitative synthesis methodologies, which reflects the complexity of integrating diverse research studies.6–8 In addition, not all qualitative syntheses are derived from formal published processes, and many processes which would be covered by our definition might be called ‘integrative reviews’9 and ‘evidence synthesis’.7 Qualitative synthesis is also controversial and many of its methods are challenged by those who believe that evidence synthesis of educational literature should not be conducted, particularly within systematic review contexts.10,11
This paper aims to provide an overview of qualitative synthesis in health professional education. Its purpose is to introduce fundamental concepts and basic methodologies to both readers and researchers, who may have already encountered qualitative synthesis within systematic reviews of the literature. We will argue for the value of qualitative synthesis, describe three qualitative synthesis methodologies linked to three examples, and end with a discussion of the potential for improving the quality of qualitative synthesis.
Why conduct qualitative synthesis?
Making judgements about qualitative research requires a deep engagement with ‘rich, thick description’ and the context of the study. Qualitative synthesis is by its nature a subjective process. The themes presented in qualitative work may be summarisable, but their meaning is sometimes inseparable from the data and not usually generalisable beyond it. Two readers who independently engage deeply with the same text are likely to come to different conclusions. Hypothetically, many reviewers could independently extract the same list of themes from a qualitative study – this might be done by extracting the headings from a findings section – but such an engagement with the text is fundamentally superficial and represents an attempt to apply quantitative models to unsuitable data.
The nature of qualitative synthesis has provoked controversy and debate amongst the academic community. Traditionally, there has been strong criticism of qualitative synthesis in literature reviews because of its potential for bias.9 From this perspective, the synthesis method of choice is quantitative: meta-analysis. Also not without its critics,12,13 meta-analysis offers a quantitative way of synthesising the impacts of multiple randomised controlled trials on the basis that a large enough sample will effectively cancel out biases inherent in any particular study. By contrast, qualitative studies are interpretive and are drawn from data that are not intended to be generalisable nor without bias. These are studies which, through their access to in-depth understanding of nuanced relationships and authentic perspectives, illuminate rather than direct.
By contrast, a number of higher education researchers from the qualitative paradigm critique qualitative synthesis that draws from a systematic review methodology, particularly describing the difficulties of synthesising studies that come from diverse qualitative methodologies.14 They argue that such processes may discourage thoughtful analysis15 and are derived from positivist stances and audit-driven cultures.10 As an illustration of the depth of this sentiment, a recent scoping exercise indicated only three published systematic reviews within the general higher education literature, excluding health professional education studies.16
The controversy and debate from both qualitative and quantitative traditions perhaps represent the inherent philosophical tension between context-specific qualitative research and the notion of drawing more generalisable conclusions through synthesis. As Pope et al. write:
‘One of the strengths of this focus in qualitative research is that it can provide analytical depth and contextualised detail. The difficulty that this poses for any attempt at synthesis is whether it is appropriate to combine the results of several unique contextually rich studies…’5
However, precisely because of their in-depth focus within particular contexts, qualitative studies provide invaluable information that contributes towards our understanding of educational dilemmas and framing of educational decisions. As Pawson et al. write:
‘…we are dealing with complex social interventions which act on complex social systems … not “magic bullets” which will always hit their target, but programmes whose effects are crucially dependent on context and implementation.’4
In the study of health professional education and in health or education practice in general, which are complex, social and highly context-dependent, the value of qualitative synthesis cannot be understated. Despite the controversies, it is of great importance to gather what collective wisdom we can from a range of methodological perspectives. Moving past ‘does it work’ questions requires venturing beyond absolute mathematical systematicity13 to consider the possibilities of other types of syntheses, which can draw together qualitative, mixed-method and quantitative research. We also acknowledge the concerns of those in the higher education qualitative research community and note that synthesis processes may not do full justice to the originating studies. However, rather than adopt the all-or-nothing ‘false dualism’ of a qualitative-versus-quantitative dichotomy,17 we take the pragmatic perspective18 that different methodologies will yield different insights.
Three qualitative synthesis methodologies
One of the more daunting tasks facing those investigating qualitative synthesis methodologies is the number of approaches to qualitative synthesis.5,7,8 These are drawn from many disciplines, particularly the social sciences and health services, and reflect the diversity of the practices from which they originate. We describe three qualitative synthesis methods here, which are well utilised within the general literature and which provide a good representation of the types of variation in qualitative synthesis methodology. We explore thematic analysis, meta-ethnography and realist review, which collectively give a sense of the range of approaches, but we urge a thorough investigation of possible methodologies, which have been well described elsewhere.2,5,8,11
Thematic analysis is an umbrella term rather than a description of a process grounded in a particular formal protocol. In general, thematic analysis refers to the various processes of reading texts and refining the findings into key ‘themes’.7 These themes represent ways of understanding the combined meaning of the text. They can be derived by such informal means as reading the texts and describing key messages, or more rigorously by coding a meaning against a specific section of text and then iteratively grouping open codes into themes. It is always important to read the detail of thematic analysis processes to ensure that what the authors mean by this term is clear. In addition to being a qualitative synthesis methodology, thematic analysis is a commonly used approach to analysing qualitative data of all descriptions.
Meta-ethnography is a qualitative synthesis technique which involves synthesis of the findings of qualitative studies. Dixon-Woods et al.7 describe three key features to meta-ethnography: the mapping of key themes and concepts across studies (which may parallel a thematic analysis approach); the identification and resolution of any contradictions, and the building of a general interpretation based on the data. As with other qualitative synthesis techniques, there are often a number of variants on the exact processes. The distinguishing feature here is that this type of synthesis allows the authors to build an interpretative layer which may extend beyond the interpretations provided by the original included studies.
Realist review methodology was introduced by Pawson et al. in 2005.4 In this approach, explanatory theories are used to provide a framework for the phenomenon under review. The qualitative synthesis seeks to shed light on these frameworks and does so by trying to understand any or all of the following components: what it is about this kind of intervention that works, for whom, in what circumstances, in what respects and why.4 The realist methodology provides a way of conducting reviews that aim to ‘identify and explain the interaction between context, mechanism and outcome’.19
To further understand the differences among these three methodologies, it is worth considering two dimensions. Dixon-Woods et al.,7 drawing from Noblit and Hare,20 distinguish a continuum which ranges from methods that seek to summarise data (‘integrative’) to those that seek to develop concepts and theories (‘interpretive’). Another useful dimension is provided by Barnett-Page and Thomas,8 who consider the influence of the researchers’ epistemologies, that is, their views of knowledge and how it is constructed. At one end of the spectrum is the ‘idealist’, indicating a researcher who believes that all knowledge is subjective, and at the other end is the ‘realist’, indicating a researcher who believes that knowledge directly represents an external reality. Both these constructs assist in elucidating the potential findings of a particular methodology as they capture both the core philosophy underpinning the synthesis approach and the interaction of the synthesis process with the data. Other ways in which qualitative methodologies vary are according to their common uses, key discipline groups and types of literature included in the synthesis. Table 1 shows an overview of these elements.
Table 1. Overview of three qualitative synthesis methodologies
| Thematic analysis |
|Describing key recurrent messages from series of studies||Education researchers conducting systematic reviews, which summarise the current literature||Summarising the collective conclusions of the included studies||Qualitative or mixed-methods||Tends to realistic||Tends towards summarising||‘Inductive analysis’: an umbrella term like ‘thematic analysis’, involving establishing patterns within the data through examination of data without a priori frameworks28|
‘Content analysis’: term is used in different ways (sometimes as equivalent to thematic analysis), but in qualitative synthesis refers to a systematic, manner of coding themes, which is stable across raters; the themes can then be presented as ‘counts’7
| Meta-ethnography |
|Building a new theory from series of studies||Qualitative researchers, interested in theoretical or conceptual understandings of phenomena||Building theory, understanding relationships across studies||Tends to be only qualitative||Tends to idealistic||Tends towards developing theory||‘Grounded theory’: draws from the qualitative analysis methodology to develop a theory from the included studies, often included studies are only grounded theory studies2,8|
‘Thematic synthesis’: explicitly describes all ‘findings’ as data and qualitatively analyses these using thematic analysis, then establishes analytical framework including barriers, facilitators and implications for intervention development; designed for systematic reviews29
| Realist synthesis |
|Describing what the literature has to say about the influence of context upon findings||Health or health education researchers with a policy and change focus||Providing guidance for specific policy questions, in complex contexts||Tends to be mixed-method||Tends to realistic||Equal emphasis on summarising and developing||Framework synthesis: based on establishing an a priori framework, against which data are extracted and synthesised8, 30|
Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
Qualitative synthesis as reported in the health professions literature
Although there are many more than three qualitative synthesis methodologies, the reality is that very few explicit methodologies are reported within health professions education research. We conducted a scoping exercise to indicate the extent to which various methodologies are used within systematic review methodology in the health professions literature. We searched MEDLINE using the terms ‘systematic review’ and ‘education’ for literature published in the 3 years to May 2012. We found 75 of a possible 733 articles describe systematic reviews according to their titles and abstracts. Although at least 19 of these reviews clearly contain qualitative syntheses, eight mention a specific methodology for qualitative analysis. Two articles list realist synthesis as a methodology, three describe thematic analysis and three describe inductive analysis (which is another methodology loosely similar to thematic analysis). There are no mentions of meta-ethnography or any of the other commonly described qualitative synthesis methods in the health services literature. A further search in MEDLINE specifically for common methodologies did not indicate any additional reported usage in the health professions education literature outside of systematic review.
This brief exercise reveals two interesting points. Firstly, many excellent and frequently cited systematic reviews use qualitative synthesis without referencing a formal synthesis methodology (see Mann et al.21 and Issenberg et al.22 for examples). Secondly, qualitative synthesis often includes many types of quantitative data. That is, the notion of using rigorous, transparent but inherently individual and context-specific processes to combine study outcomes represents a useful way to synthesise all types of research findings.
We have selected two case studies from this scoping exercise, in addition to one from outwith the health professions education literature, which illustrate both the previously described methodological approaches and some of the practical difficulties and limitations.
Example 1: thematic analysis as an adjunct to meta-analysis
Cook et al.’s23 systematic review of the use of virtual patients in health professional education used qualitative synthesis to support a more thorough quantitative meta-analysis. Four qualitative studies informed the quantitative findings. Thematic analysis is a very broad term and therefore Cook et al. detailed their specific processes as:
‘…synthesised qualitative studies by identifying key themes and supportive statements, initially independently in duplicate and then by consensus, and iteratively revising and reclassifying these themes.’23
What does this thematic analysis produce? The authors23 described four common themes, which summarised the common points drawn across the included papers. No quotes or qualitative data were presented. These themes were framed in terms of the ‘effectiveness’ of virtual patients, highlighting the value of features such as natural case progression and working in groups rather than individually.23Table 2 shows further details for this and the other examples.
Table 2. A comparison of thematic analysis, meta-ethnography and realist review case studies
|Thematic analysis as adjunct to meta-analysis23||Virtual patients||Systematic review and meta-analysis||‘How effective are virtual patients in comparison with no intervention and alternate instructional methods, and what virtual patient design features are associated with higher learning outcomes?’||Thematic analysis; themes provided*||Qualitative synthesis is used to understand meaning of quantitative synthesis in conclusions||Process is clearly articulated; appropriate analysis within context of overall review||Limited conclusions/understandings could be drawn from rich qualitative data within the context of the overall review |
The qualitative data are overshadowed by the quantitative findings
|Meta-ethnography in higher education24||Faculty experience of problem-based learning (PBL)||Interpretive meta-ethnography||‘How does faculty thinking change as a result of moving from a more traditional teaching approach to an innovative educational approach?’||Meta-ethnography |
Detailed processes described*
|None||Deep, interrogative understanding of the phenomena |
Explicit transparent and reflexive processes
|Supporting specific decision making not covered by the in-depth focus of the study |
Authors query the value of this approach as it may be too reductionist
|Realist review variant, using qualitative synthesis with quantitative studies25||Journal clubs||Systematic review using realist synthesis||Is the (journal club) effective in supporting (evidence-based) decision making?||Adaptation of realist synthesis |
Role of theory differs from other realist reviews (e.g. Wong et al.19)†
|Studies are primarily quantitative but synthesised in a qualitative way||Findings are presented in a form useful for those wishing to establish journal clubs |
Clear links between findings and studies
|Theoretical frameworks which would assist in interpreting findings are limited |
Unclear reporting of quality assessment and synthesis processes impacts on understanding findings
Example 2: meta-ethnography in the higher education literature
Savin-Baden and Major24 reviewed the literature in higher education in general regarding the effects of problem-based learning (PBL) upon faculty staff. Their methodology24 broadly followed the processes described earlier in this paper. Four of the six qualitative studies included in the review comprised or included faculty staff from the nursing or physiotherapy disciplines. Only specific types of qualitative data were provided, which were ‘grounded in the faculty experience’.24 The qualitative synthesis methodology was meta-ethnography, with a focus upon the interpretive and ‘rich’ nature of the data.
Four cross-data themes raised insight into power and authority in teaching in higher education. In this qualitative synthesis, the original studies gave rise to an in-depth consideration about how conducting PBL changed educators’ perspectives about their own identities, the locus of authority and control in teaching, the nature of discipline-specific knowledge and their fundamental views on learning and teaching practice.24 These ideas represent themes that extend beyond the findings of the included studies, as Savin-Baden and Major24deliberately sought to build upon the studies through an in-depth qualitative comparison of their outcomes. In this instance, the findings provide a deeper understanding of the phenomenon rather than being overly concerned with effectiveness. Table 2 shows more details.
Example 3: realist review variant, using qualitative methods to synthesise quantitative studies
Harris et al.25 used the realist synthesis process in association with a more standard systematic review methodology. Eighteen primarily quantitative studies were included, the majority of which were before-and-after studies or questionnaire-based studies. After data extraction, the researchers determined that the studies were too heterogeneous to be quantitatively synthesised. Once studies with ‘similar approaches’ were identified, they were ‘lumped together as a case and determine[d] whether there was a core group of “active ingredients” that contributed to successful [journal clubs]’.25
The authors25 developed a logic model, which provided a theoretical framework to describe the value of journal clubs, as in realist synthesis frameworks are considered a priori. This logic model was used to identify elements (‘active ingredients’) which commonly occurred across the journal clubs. Studies were grouped together with respect to four basic outcomes in a kind of summarising phase. These outcomes described the positive outcomes of journal clubs, such as changes in reading behaviour, improved confidence in critical appraisal skills, improved critical appraisal practice, and the application of research to clinical practice. The authors discussed the variations leading to these ‘active ingredients’ and thus the conclusion and findings of the review included a discussion of ‘what worked for whom in what circumstances’.25 In this realist review,25 the authors found that certain supports, such as mentoring and the use of a structured review tool, were of assistance, but that a key mediating factor was level of experience. They later recommended that the design of a journal club should be built around the learners’ profile and experience.25Table 2 gives further details and a comparison across examples.
The three reviews indicate the value of qualitative synthesis. The collective understanding drawn from multiple studies assists in the broader understanding of the phenomenon under study. All of the studies have created value and meaning through their synthesis of the complex underlying original studies. The three examples reveal the diversity of the type of qualitative synthesis methods in the literature and the strengths and weaknesses of specific approaches to three different methodologies. They place varying degrees of emphasis on the contextually rich nature of qualitative data, but they show similarities in the way they draw the data together.
Rigour in qualitative synthesis
The examples also illustrate some general points about processes of qualitative synthesis. Two studies reported a team approach to synthesis, whereby judgements were made using the assessment of more than one reviewer. This use of multiple perspectives (‘triangulation’), in which disagreements are negotiated through consensus, is often used in qualitative analysis to add rigour to analysis processes and is a useful indicator of the types of checks and balances that build credibility in a research process. By contrast, with the exception of the meta-ethnography study,24 reports of the synthesis processes were neither detailed nor substantial and it was difficult to understand how judgements were made about the quality of particular research papers.
We anticipate that some will find the notion of rigour in qualitative synthesis difficult, given its subjective nature. Those with strongly quantitative backgrounds may be uneasy with a methodology which, if conducted by another team, would result in different findings. In a purely quantitative synthesis, if a separate review team were to follow methods as published, their process should result in the same outcome. Is it necessary for synthesis of qualitative research to be replicable? Is it even possible for qualitative synthesis to be reproducible? Qualitative synthesis is about making structured judgements. There is no need to make the judgements predetermined or reproducible because they are grounded in the contextualised expertise of the reviewers. As the examples illustrate, the outcomes of qualitative synthesis provide value through building a collective understanding of the data regarding a particular issue or phenomenon, not by establishing definitive causal links.
We think that qualitative synthesis should be conducted with the goal of achieving transparency of the process framework, not reproducibility. The structure or framework in which the judgements sit should be well described. This is consistent with both rigour in systematic review and rigour in qualitative research methodology. This means that many review publications, which do not reference a formal methodology, can still be read as rigorous approaches to qualitative synthesis. An example is Wearne et al.’s recent ‘integrative review’ of family doctors as supervisors, which clearly details the authors’ approach to qualitative synthesis.26
A clearly described framework enables the readers to make a critical assessment of the work, using their own expertise, drawn from their own contexts. Given the diversity of approaches to qualitative synthesis, readers need to acquire a breadth of understanding, arguably more than the requirements of the singular rhetoric of meta-analysis. Readers always make their own judgements about the value of a work and its applicability to their own context. Nuanced processes such as qualitative synthesis, which draw together evidence regarding complex phenomena, require nuanced complex thought from readers. We argue that criteria for rigour in the conduct of qualitative studies, such as declarations of the researchers’‘stance’,24 transparency24,27 and triangulation27, can also be applied to qualitative synthesis. We argue that a qualitative synthesis should particularly endeavour to report:
- • the researchers’ stance by providing a rationale for the choice of qualitative synthesis methodology, situated within a framework which describes the theories and views that inform the researchers’ approach to their topic, and
- • transparency by detailing the synthesis process, or providing a reference which clearly provides the processes used. Details should include how the text was read, how the team negotiated the synthesis, how any analysis was derived, and any checks and balances undertaken by the team to ensure rigour of analysis. An acknowledgement of limitations of the synthesis method should be given. Clear simple language should be used rather than specialised terminology, which may prevent ready understanding by health care providers or educationalists.
Qualitative synthesis combines rigorous processes and authorial judgement to present the collective meaning of research outputs. There are numerous illustrations of this within the health professional education literature, particularly within systematic reviews. We have provided three sample methodologies linked to three examples to illustrate both the value and the limitations of qualitative synthesis methodology. Qualitative synthesis already plays a very important part in our understanding of the literature. We believe this is appropriate, but we argue for a deeper engagement with synthesis methodologies on the part of both readers and authors.
Contributors: MB and PD met and outlined the conception and design of the article. MB and PD alternately drafted and revised the article. Both authors approved the final manuscript for publication.
Acknowledgements: We would like to acknowledge the insights of Associate Professor Elizabeth Molloy, which started the ball rolling.
Conflicts of interest: none.
Ethical approval: not applicable.