SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

Context  Formal qualitative synthesis is the process of pooling qualitative and mixed-method research data, and then drawing conclusions regarding the collective meaning of the research. Qualitative synthesis is regularly used within systematic reviews in the health professions literature, although such use has been heavily debated in the general literature. This controversy arises in part from the inherent tensions found when generalisations are derived from in-depth studies that are heavily context-dependent.

Methods  We explore three representative qualitative synthesis methodologies: thematic analysis; meta-ethnography, and realist synthesis. These can be understood across two dimensions: integrative to interpretative, and idealist to realist. Three examples are used to illustrate the relative strengths and limitations of these approaches.

Discussion  Against a backdrop of controversy and diverse methodologies, readers must take a critical stand when reading literature reviews that use qualitative synthesis to derive their findings. We argue that notions of qualitative rigour such as transparency and acknowledgment of the researchers’ stance should be applied to qualitative synthesis.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

qualitative (adj.): ‘relating to or measured by quality’1

synthesis (n.): ‘the combination of components to form a connected whole’1

Readers of health professional literature rely on qualitative synthesis within literature reviews to assist in understanding the implications of the collective research. Qualitative syntheses share the subjective or ‘critical’2 nature of a traditional ‘narrative’ review. However, they can be distinguished from traditional reviews, as more structured processes, and are often set within, or draw from, a systematic review framework. Systematic review methodology, which is over two decades old, uses structured and transparent processes for collecting, assessing and synthesising the literature.3 One of the key elements is the formal synthesis of the studies included, which is intended to support understanding of the combined implication of the ever-increasing evidence. Synthesis methodologies in systematic reviews were originally, and are still predominantly, numerical methodologies such as meta-analysis. Many within the health professional education community regard systematic review as the reference standard for a literature review in health professional education, and educators and academics look to systematic reviews to provide conclusions from the literature to inform teaching and research. Interestingly, for a technique that is commonly used within health professional education systematic reviews, discussion of qualitative synthesis methodologies is limited.

We define qualitative synthesis as any methodology whereby study findings are systematically interpreted through a series of expert judgements to represent the meaning of the collected work. In a qualitative synthesis, the findings of qualitative studies – and sometimes mixed-methods and quantitative research – are pooled. Judgement-based qualitative methodologies are used to draw conclusions regarding the collective meanings of this pool of research. This is necessarily a complex and multifaceted process, which often deals with widely variant sources, and the methodological challenge of conducting qualitative synthesis is well acknowledged in the health policy literature.4–6 There is a wide range of qualitative synthesis methodologies, which reflects the complexity of integrating diverse research studies.6–8 In addition, not all qualitative syntheses are derived from formal published processes, and many processes which would be covered by our definition might be called ‘integrative reviews’9 and ‘evidence synthesis’.7 Qualitative synthesis is also controversial and many of its methods are challenged by those who believe that evidence synthesis of educational literature should not be conducted, particularly within systematic review contexts.10,11

This paper aims to provide an overview of qualitative synthesis in health professional education. Its purpose is to introduce fundamental concepts and basic methodologies to both readers and researchers, who may have already encountered qualitative synthesis within systematic reviews of the literature. We will argue for the value of qualitative synthesis, describe three qualitative synthesis methodologies linked to three examples, and end with a discussion of the potential for improving the quality of qualitative synthesis.

Why conduct qualitative synthesis?

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

Making judgements about qualitative research requires a deep engagement with ‘rich, thick description’ and the context of the study. Qualitative synthesis is by its nature a subjective process. The themes presented in qualitative work may be summarisable, but their meaning is sometimes inseparable from the data and not usually generalisable beyond it. Two readers who independently engage deeply with the same text are likely to come to different conclusions. Hypothetically, many reviewers could independently extract the same list of themes from a qualitative study – this might be done by extracting the headings from a findings section – but such an engagement with the text is fundamentally superficial and represents an attempt to apply quantitative models to unsuitable data.

The nature of qualitative synthesis has provoked controversy and debate amongst the academic community. Traditionally, there has been strong criticism of qualitative synthesis in literature reviews because of its potential for bias.9 From this perspective, the synthesis method of choice is quantitative: meta-analysis. Also not without its critics,12,13 meta-analysis offers a quantitative way of synthesising the impacts of multiple randomised controlled trials on the basis that a large enough sample will effectively cancel out biases inherent in any particular study. By contrast, qualitative studies are interpretive and are drawn from data that are not intended to be generalisable nor without bias. These are studies which, through their access to in-depth understanding of nuanced relationships and authentic perspectives, illuminate rather than direct.

By contrast, a number of higher education researchers from the qualitative paradigm critique qualitative synthesis that draws from a systematic review methodology, particularly describing the difficulties of synthesising studies that come from diverse qualitative methodologies.14 They argue that such processes may discourage thoughtful analysis15 and are derived from positivist stances and audit-driven cultures.10 As an illustration of the depth of this sentiment, a recent scoping exercise indicated only three published systematic reviews within the general higher education literature, excluding health professional education studies.16

The controversy and debate from both qualitative and quantitative traditions perhaps represent the inherent philosophical tension between context-specific qualitative research and the notion of drawing more generalisable conclusions through synthesis. As Pope et al. write:

‘One of the strengths of this focus in qualitative research is that it can provide analytical depth and contextualised detail. The difficulty that this poses for any attempt at synthesis is whether it is appropriate to combine the results of several unique contextually rich studies…’5

However, precisely because of their in-depth focus within particular contexts, qualitative studies provide invaluable information that contributes towards our understanding of educational dilemmas and framing of educational decisions. As Pawson et al. write:

‘…we are dealing with complex social interventions which act on complex social systems … not “magic bullets” which will always hit their target, but programmes whose effects are crucially dependent on context and implementation.’4

In the study of health professional education and in health or education practice in general, which are complex, social and highly context-dependent, the value of qualitative synthesis cannot be understated. Despite the controversies, it is of great importance to gather what collective wisdom we can from a range of methodological perspectives. Moving past ‘does it work’ questions requires venturing beyond absolute mathematical systematicity13 to consider the possibilities of other types of syntheses, which can draw together qualitative, mixed-method and quantitative research. We also acknowledge the concerns of those in the higher education qualitative research community and note that synthesis processes may not do full justice to the originating studies. However, rather than adopt the all-or-nothing ‘false dualism’ of a qualitative-versus-quantitative dichotomy,17 we take the pragmatic perspective18 that different methodologies will yield different insights.

Three qualitative synthesis methodologies

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

One of the more daunting tasks facing those investigating qualitative synthesis methodologies is the number of approaches to qualitative synthesis.5,7,8 These are drawn from many disciplines, particularly the social sciences and health services, and reflect the diversity of the practices from which they originate. We describe three qualitative synthesis methods here, which are well utilised within the general literature and which provide a good representation of the types of variation in qualitative synthesis methodology. We explore thematic analysis, meta-ethnography and realist review, which collectively give a sense of the range of approaches, but we urge a thorough investigation of possible methodologies, which have been well described elsewhere.2,5,8,11

Thematic analysis is an umbrella term rather than a description of a process grounded in a particular formal protocol. In general, thematic analysis refers to the various processes of reading texts and refining the findings into key ‘themes’.7 These themes represent ways of understanding the combined meaning of the text. They can be derived by such informal means as reading the texts and describing key messages, or more rigorously by coding a meaning against a specific section of text and then iteratively grouping open codes into themes. It is always important to read the detail of thematic analysis processes to ensure that what the authors mean by this term is clear. In addition to being a qualitative synthesis methodology, thematic analysis is a commonly used approach to analysing qualitative data of all descriptions.

Meta-ethnography is a qualitative synthesis technique which involves synthesis of the findings of qualitative studies. Dixon-Woods et al.7 describe three key features to meta-ethnography: the mapping of key themes and concepts across studies (which may parallel a thematic analysis approach); the identification and resolution of any contradictions, and the building of a general interpretation based on the data. As with other qualitative synthesis techniques, there are often a number of variants on the exact processes. The distinguishing feature here is that this type of synthesis allows the authors to build an interpretative layer which may extend beyond the interpretations provided by the original included studies.

Realist review methodology was introduced by Pawson et al. in 2005.4 In this approach, explanatory theories are used to provide a framework for the phenomenon under review. The qualitative synthesis seeks to shed light on these frameworks and does so by trying to understand any or all of the following components: what it is about this kind of intervention that works, for whom, in what circumstances, in what respects and why.4 The realist methodology provides a way of conducting reviews that aim to ‘identify and explain the interaction between context, mechanism and outcome’.19

To further understand the differences among these three methodologies, it is worth considering two dimensions. Dixon-Woods et al.,7 drawing from Noblit and Hare,20 distinguish a continuum which ranges from methods that seek to summarise data (‘integrative’) to those that seek to develop concepts and theories (‘interpretive’). Another useful dimension is provided by Barnett-Page and Thomas,8 who consider the influence of the researchers’ epistemologies, that is, their views of knowledge and how it is constructed. At one end of the spectrum is the ‘idealist’, indicating a researcher who believes that all knowledge is subjective, and at the other end is the ‘realist’, indicating a researcher who believes that knowledge directly represents an external reality. Both these constructs assist in elucidating the potential findings of a particular methodology as they capture both the core philosophy underpinning the synthesis approach and the interaction of the synthesis process with the data. Other ways in which qualitative methodologies vary are according to their common uses, key discipline groups and types of literature included in the synthesis. Table 1 shows an overview of these elements.

Table 1.   Overview of three qualitative synthesis methodologies
Methodology and Summary descriptionCommonly used by:Useful for:Data sourcesIdealistic to realisticSummarising data or developing theoryExamples of qualitative synthesis methodologies with aligned utility and approach
Thematic analysis
Describing key recurrent messages from series of studiesEducation researchers conducting systematic reviews, which summarise the current literatureSummarising the collective conclusions of the included studiesQualitative or mixed-methodsTends to realisticTends towards summarising‘Inductive analysis’: an umbrella term like ‘thematic analysis’, involving establishing patterns within the data through examination of data without a priori frameworks28 ‘Content analysis’: term is used in different ways (sometimes as equivalent to thematic analysis), but in qualitative synthesis refers to a systematic, manner of coding themes, which is stable across raters; the themes can then be presented as ‘counts’7
Meta-ethnography
Building a new theory from series of studiesQualitative researchers, interested in theoretical or conceptual understandings of phenomenaBuilding theory, understanding relationships across studiesTends to be only qualitativeTends to idealisticTends towards developing theory‘Grounded theory’: draws from the qualitative analysis methodology to develop a theory from the included studies, often included studies are only grounded theory studies2,8 ‘Thematic synthesis’: explicitly describes all ‘findings’ as data and qualitatively analyses these using thematic analysis, then establishes analytical framework including barriers, facilitators and implications for intervention development; designed for systematic reviews29
Realist synthesis
Describing what the literature has to say about the influence of context upon findingsHealth or health education researchers with a policy and change focusProviding guidance for specific policy questions, in complex contextsTends to be mixed-methodTends to realisticEqual emphasis on summarising and developingFramework synthesis: based on establishing an a priori framework, against which data are extracted and synthesised8, 30

Thematic analysis, meta-ethnography and realist synthesis: examples from the literature

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

Qualitative synthesis as reported in the health professions literature

Although there are many more than three qualitative synthesis methodologies, the reality is that very few explicit methodologies are reported within health professions education research. We conducted a scoping exercise to indicate the extent to which various methodologies are used within systematic review methodology in the health professions literature. We searched MEDLINE using the terms ‘systematic review’ and ‘education’ for literature published in the 3 years to May 2012. We found 75 of a possible 733 articles describe systematic reviews according to their titles and abstracts. Although at least 19 of these reviews clearly contain qualitative syntheses, eight mention a specific methodology for qualitative analysis. Two articles list realist synthesis as a methodology, three describe thematic analysis and three describe inductive analysis (which is another methodology loosely similar to thematic analysis). There are no mentions of meta-ethnography or any of the other commonly described qualitative synthesis methods in the health services literature. A further search in MEDLINE specifically for common methodologies did not indicate any additional reported usage in the health professions education literature outside of systematic review.

This brief exercise reveals two interesting points. Firstly, many excellent and frequently cited systematic reviews use qualitative synthesis without referencing a formal synthesis methodology (see Mann et al.21 and Issenberg et al.22 for examples). Secondly, qualitative synthesis often includes many types of quantitative data. That is, the notion of using rigorous, transparent but inherently individual and context-specific processes to combine study outcomes represents a useful way to synthesise all types of research findings.

We have selected two case studies from this scoping exercise, in addition to one from outwith the health professions education literature, which illustrate both the previously described methodological approaches and some of the practical difficulties and limitations.

Example 1: thematic analysis as an adjunct to meta-analysis

Cook et al.’s23 systematic review of the use of virtual patients in health professional education used qualitative synthesis to support a more thorough quantitative meta-analysis. Four qualitative studies informed the quantitative findings. Thematic analysis is a very broad term and therefore Cook et al. detailed their specific processes as:

‘…synthesised qualitative studies by identifying key themes and supportive statements, initially independently in duplicate and then by consensus, and iteratively revising and reclassifying these themes.’23

What does this thematic analysis produce? The authors23 described four common themes, which summarised the common points drawn across the included papers. No quotes or qualitative data were presented. These themes were framed in terms of the ‘effectiveness’ of virtual patients, highlighting the value of features such as natural case progression and working in groups rather than individually.23Table 2 shows further details for this and the other examples.

Table 2.   A comparison of thematic analysis, meta-ethnography and realist review case studies
ExampleTopicAuthor-stated review typeReview question or aimQualitative thesis typeRelationship to quantitative (if any)Advantages of reported approach to synthesisLimitations of reported approach to synthesis
  1. * Multiple researchers conduct synthesis

  2. † Not reported for synthesis

Thematic analysis as adjunct to meta-analysis23Virtual patientsSystematic review and meta-analysis‘How effective are virtual patients in comparison with no intervention and alternate instructional methods, and what virtual patient design features are associated with higher learning outcomes?’Thematic analysis; themes provided*Qualitative synthesis is used to understand meaning of quantitative synthesis in conclusionsProcess is clearly articulated; appropriate analysis within context of overall reviewLimited conclusions/understandings could be drawn from rich qualitative data within the context of the overall review The qualitative data are overshadowed by the quantitative findings
Meta-ethnography in higher education24Faculty experience of problem-based learning (PBL)Interpretive meta-ethnography‘How does faculty thinking change as a result of moving from a more traditional teaching approach to an innovative educational approach?’Meta-ethnography Detailed processes described*NoneDeep, interrogative understanding of the phenomena Explicit transparent and reflexive processesSupporting specific decision making not covered by the in-depth focus of the study Authors query the value of this approach as it may be too reductionist
Realist review variant, using qualitative synthesis with quantitative studies25Journal clubsSystematic review using realist synthesisIs the (journal club) effective in supporting (evidence-based) decision making?Adaptation of realist synthesis Role of theory differs from other realist reviews (e.g. Wong et al.19)†Studies are primarily quantitative but synthesised in a qualitative wayFindings are presented in a form useful for those wishing to establish journal clubs Clear links between findings and studiesTheoretical frameworks which would assist in interpreting findings are limited Unclear reporting of quality assessment and synthesis processes impacts on understanding findings

Example 2: meta-ethnography in the higher education literature

Savin-Baden and Major24 reviewed the literature in higher education in general regarding the effects of problem-based learning (PBL) upon faculty staff. Their methodology24 broadly followed the processes described earlier in this paper. Four of the six qualitative studies included in the review comprised or included faculty staff from the nursing or physiotherapy disciplines. Only specific types of qualitative data were provided, which were ‘grounded in the faculty experience’.24 The qualitative synthesis methodology was meta-ethnography, with a focus upon the interpretive and ‘rich’ nature of the data.

Four cross-data themes raised insight into power and authority in teaching in higher education. In this qualitative synthesis, the original studies gave rise to an in-depth consideration about how conducting PBL changed educators’ perspectives about their own identities, the locus of authority and control in teaching, the nature of discipline-specific knowledge and their fundamental views on learning and teaching practice.24 These ideas represent themes that extend beyond the findings of the included studies, as Savin-Baden and Major24deliberately sought to build upon the studies through an in-depth qualitative comparison of their outcomes. In this instance, the findings provide a deeper understanding of the phenomenon rather than being overly concerned with effectiveness. Table 2 shows more details.

Example 3: realist review variant, using qualitative methods to synthesise quantitative studies

Harris et al.25 used the realist synthesis process in association with a more standard systematic review methodology. Eighteen primarily quantitative studies were included, the majority of which were before-and-after studies or questionnaire-based studies. After data extraction, the researchers determined that the studies were too heterogeneous to be quantitatively synthesised. Once studies with ‘similar approaches’ were identified, they were ‘lumped together as a case and determine[d] whether there was a core group of “active ingredients” that contributed to successful [journal clubs]’.25

The authors25 developed a logic model, which provided a theoretical framework to describe the value of journal clubs, as in realist synthesis frameworks are considered a priori. This logic model was used to identify elements (‘active ingredients’) which commonly occurred across the journal clubs. Studies were grouped together with respect to four basic outcomes in a kind of summarising phase. These outcomes described the positive outcomes of journal clubs, such as changes in reading behaviour, improved confidence in critical appraisal skills, improved critical appraisal practice, and the application of research to clinical practice. The authors discussed the variations leading to these ‘active ingredients’ and thus the conclusion and findings of the review included a discussion of ‘what worked for whom in what circumstances’.25 In this realist review,25 the authors found that certain supports, such as mentoring and the use of a structured review tool, were of assistance, but that a key mediating factor was level of experience. They later recommended that the design of a journal club should be built around the learners’ profile and experience.25Table 2 gives further details and a comparison across examples.

The three reviews indicate the value of qualitative synthesis. The collective understanding drawn from multiple studies assists in the broader understanding of the phenomenon under study. All of the studies have created value and meaning through their synthesis of the complex underlying original studies. The three examples reveal the diversity of the type of qualitative synthesis methods in the literature and the strengths and weaknesses of specific approaches to three different methodologies. They place varying degrees of emphasis on the contextually rich nature of qualitative data, but they show similarities in the way they draw the data together.

Rigour in qualitative synthesis

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

The examples also illustrate some general points about processes of qualitative synthesis. Two studies reported a team approach to synthesis, whereby judgements were made using the assessment of more than one reviewer. This use of multiple perspectives (‘triangulation’), in which disagreements are negotiated through consensus, is often used in qualitative analysis to add rigour to analysis processes and is a useful indicator of the types of checks and balances that build credibility in a research process. By contrast, with the exception of the meta-ethnography study,24 reports of the synthesis processes were neither detailed nor substantial and it was difficult to understand how judgements were made about the quality of particular research papers.

We anticipate that some will find the notion of rigour in qualitative synthesis difficult, given its subjective nature. Those with strongly quantitative backgrounds may be uneasy with a methodology which, if conducted by another team, would result in different findings. In a purely quantitative synthesis, if a separate review team were to follow methods as published, their process should result in the same outcome. Is it necessary for synthesis of qualitative research to be replicable? Is it even possible for qualitative synthesis to be reproducible? Qualitative synthesis is about making structured judgements. There is no need to make the judgements predetermined or reproducible because they are grounded in the contextualised expertise of the reviewers. As the examples illustrate, the outcomes of qualitative synthesis provide value through building a collective understanding of the data regarding a particular issue or phenomenon, not by establishing definitive causal links.

We think that qualitative synthesis should be conducted with the goal of achieving transparency of the process framework, not reproducibility. The structure or framework in which the judgements sit should be well described. This is consistent with both rigour in systematic review and rigour in qualitative research methodology. This means that many review publications, which do not reference a formal methodology, can still be read as rigorous approaches to qualitative synthesis. An example is Wearne et al.’s recent ‘integrative review’ of family doctors as supervisors, which clearly details the authors’ approach to qualitative synthesis.26

A clearly described framework enables the readers to make a critical assessment of the work, using their own expertise, drawn from their own contexts. Given the diversity of approaches to qualitative synthesis, readers need to acquire a breadth of understanding, arguably more than the requirements of the singular rhetoric of meta-analysis. Readers always make their own judgements about the value of a work and its applicability to their own context. Nuanced processes such as qualitative synthesis, which draw together evidence regarding complex phenomena, require nuanced complex thought from readers. We argue that criteria for rigour in the conduct of qualitative studies, such as declarations of the researchers’‘stance’,24 transparency24,27 and triangulation27, can also be applied to qualitative synthesis. We argue that a qualitative synthesis should particularly endeavour to report:

  • the researchers’ stance by providing a rationale for the choice of qualitative synthesis methodology, situated within a framework which describes the theories and views that inform the researchers’ approach to their topic, and
  • transparency by detailing the synthesis process, or providing a reference which clearly provides the processes used. Details should include how the text was read, how the team negotiated the synthesis, how any analysis was derived, and any checks and balances undertaken by the team to ensure rigour of analysis. An acknowledgement of limitations of the synthesis method should be given. Clear simple language should be used rather than specialised terminology, which may prevent ready understanding by health care providers or educationalists.

Conclusions

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References

Qualitative synthesis combines rigorous processes and authorial judgement to present the collective meaning of research outputs. There are numerous illustrations of this within the health professional education literature, particularly within systematic reviews. We have provided three sample methodologies linked to three examples to illustrate both the value and the limitations of qualitative synthesis methodology. Qualitative synthesis already plays a very important part in our understanding of the literature. We believe this is appropriate, but we argue for a deeper engagement with synthesis methodologies on the part of both readers and authors.

Contributors:  MB and PD met and outlined the conception and design of the article. MB and PD alternately drafted and revised the article. Both authors approved the final manuscript for publication.

Acknowledgements:  We would like to acknowledge the insights of Associate Professor Elizabeth Molloy, which started the ball rolling.

Funding:  none.

Conflicts of interest:  none.

Ethical approval:  not applicable.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Why conduct qualitative synthesis?
  5. Three qualitative synthesis methodologies
  6. Thematic analysis, meta-ethnography and realist synthesis: examples from the literature
  7. Rigour in qualitative synthesis
  8. Conclusions
  9. References
  • 1
    Concise Oxford English Dictionary, 12th edn. Soanes C, Stevenson A, eds. Oxford: Oxford University Press 2008.
  • 2
    Thorne S, Jensen L, Kearney MH, Noblit G, Sandelowski M. Qualitative metasynthesis: reflections on methodological orientation and ideological agenda. Qual Health Res2004;14 (10):134265.
  • 3
    Cochrane Collaboration. Cochrane Reviews. Oxford: Cochrane Collaboration 2012. http://www.cochrane.org/cochrane-reviews . [Accessed 1 June 2012.]
  • 4
    Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review – a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy2005;10 (Suppl 1):2134.
  • 5
    Pope C, Mays N, Popay J. Synthesizing Qualitative and Quantitative Health Evidence. Maidenhead: Open University Press 2007;78.
  • 6
    Ring N, Ritchie K, Mandava L, Jepson R. A Guide to Synthesising Qualitative Research for Researchers Undertaking Health Technology Assessments and Systematic Reviews. National Health Service Quality Improvement Scotland 2010. http://www.nhshealthquality.org/nhsqis/8837.html. [Accessed 23 December 2012.]
  • 7
    Dixon-Woods M, Agarwal S, Jones D, Young B, Sutton A. Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy2005;10 (1):4553.
  • 8
    Barnett-Page E, Thomas J. Methods for the synthesis of qualitative research: a critical review. BMC Med Res Methodol2009;9:59.
  • 9
    Cooper HM. Scientific guidelines for conducting integrative research reviews. Rev Educ Res1982;52 (2):291302.
  • 10
    Denzin NK. The elephant in the living room: or extending the conversation about the politics of evidence. Qual Res2009;9 (2):13960.
  • 11
    Dixon-Woods M, Bonas S, Booth A, Jones DR, Miller T, Sutton AJ, Shaw RL, Smith JA, Young B. How can systematic reviews incorporate qualitative research? A critical perspective. Qual Res2006;6 (1):2744.
  • 12
    Colliver JA, Kucera K, Verhulst SJ. Meta-analysis of quasi-experimental research: are systematic narrative reviews indicated?Med Educ2008;42 (9):85865.
  • 13
    Eva KW. On the limits of systematicity. Med Educ2008;42 (9):8523.
  • 14
    Suri H, Clarke D. Advancements in research synthesis methods: from a methodologically inclusive perspective. Rev Educ Res2009;79 (1):395430.
  • 15
    Maclure M. Clarity bordering on stupidity: where’s the quality in systematic review?J Educ Pol2005;20 (4):393416.
  • 16
    Bearman M, Smith C, Carbone A, Slade S, Baik C, Hughes-Warrington M, Neumann DLSystematic review methodology in higher education. High Educ Res Dev2012;31 (5):625–40.
  • 17
    Pring R. The ‘false dualism’ of educational research. J Philos Educ2000;34 (2):24760.
  • 18
    Badley G. The crisis in educational research: a pragmatic approach. Eur Educ Res J2003;2 (2):296308.
  • 19
    Wong G, Greenhalgh T, Pawson R. Internet-based medical education: a realist review of what works, for whom and in what circumstances. BMC Med Educ2010;10:12.
  • 20
    Noblit G, Hare R. Meta-ethnography: Synthesising Qualitative Studies. Newbury Park, CA: Sage Sage 1988.
  • 21
    Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ Theory Pract2009;14 (4):595621.
  • 22
    Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach2005;27 (1):1028.
  • 23
    Cook DA, Erwin PJ, Triola MM. Computerised virtual patients in health professions education: a systematic review and meta-analysis. Acad Med2010;85 (10):1589602.
  • 24
    Savin-Baden M, Major C. Using interpretative meta-ethnography to explore the relationship between innovative approaches to learning and their influence on faculty understanding of teaching. High Educ2007;54 (6):83352.
  • 25
    Harris J, Kearley K, Heneghan C, Meats E, Roberts N, Perera R, Kearley-Shiers K. Are journal clubs effective in supporting evidence-based decision making? A systematic review. BEME Guide No. 16. Med Teach2011;33 (1):923.
  • 26
    Wearne S, Dornan T, Teunissen PW, Skinner T. Family practitioners as supervisors in postgraduate clinical education: an integrative review. Med Educ2012;46:1161–73.
  • 27
    Mays N, Pope C, Popay J. Systematically reviewing qualitative and quantitative evidence to inform management and policy-making in the health field. J Health Serv Res Policy2005;10 (Suppl):620.
  • 28
    Patton M. Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage Publications 2001;41
  • 29
    Thomas J, Harden A. Methods for the thematic synthesis of qualitative research in systematic reviews. BMC Med Res Methodol2008;8:45.
  • 30
    Carroll C, Booth A, Cooper K. A worked example of ‘best fit’ framework synthesis: a systematic review of views concerning the taking of some potential chemopreventive agents. BMC Med Res Methodol2011;11:29.