Variations of mixed methods reviews approaches: A case study

Conducting mixed methods reviews is challenging. The aim of this article is to describe a range of rationales for and approaches to mixed methods reviews, with a particular focus on one research group. A case study was conducted to describe the mixed methods review process used at the Department of Health and Social Care Reviews Facility in England. The case study used document analysis. A total of 30 mixed methods reviews were identified and analyzed. The analysis revealed five key dimensions on which the reviews varied: review questions and purposes of the mixed methods questions, types of evidence and sources, reasons for using a mixed methods approach, synthesis methods and designs, and integration strategies. The questions in the included reviews addressed stakeholders' views, and intervention processes and/or intervention effectiveness. The mixed methods questions addressed four different purposes: comparing findings, identifying critical intervention features, quantifying effects, and making recommendations. Five main sources of evidence were used: formal evidence from primary studies, informal evidence, policy documents, systematic reviews, and work with stakeholders. Twelve reasons for conducting mixed methods reviews were identified: completeness, contextual understanding, credibility, different research questions, diversity of views, enhancement, explanation, process, triangulation, utility, development of a framework, and identification of promising interventions. Each review employed one or several integration strategies for comparing findings, connecting phases and/or assimilating data. It is hoped that the information garnered from this study will provide useful insights into mixed method review diversity and trigger new ideas for conducting this type of review.


| BACKGROUND
An increasing number of researchers are interested in combining qualitative and quantitative research in a systematic review. Various names have been given to designate this type of review such as mixed methods systematic review, 1-3 mixed methods research synthesis, 4 mixed studies review, 5 mixed methods-mixed research synthesis, 6 multi-method review, 7 multi-sources review, 8 and mixed knowledge review. 9 Several reasons can explain this increasing popularity.
First, decision makers often face complex issues that cannot be addressed using quantitative or qualitative research alone. As part of the evidence-based practice movement, systematic reviews have become an essential resource used by decision makers to inform practices, policies, and program development. 10 Traditionally, systematic reviews have mainly focused on the effectiveness of interventions and used meta-analysis. The methods used in these reviews were then adapted to consider complexity that can occur at different domains (eg, in an intervention, its implementation, the context, and the population). 11 To better inform decision-making, there is a need to provide a more complete and rich understanding of a phenomenon and incorporate evidence on aspects other than effectiveness such as intervention context, acceptability, and feasibility. 12 Second, over recent years, researchers have been more exposed to multimethod and mixed methods research, especially in primary research. 13,14 Since the beginning of the 21st century, this field has considerably grown with the creation of local and international associations (eg, Mixed Methods International Research Association (MMIRA), Méthodes Mixtes Francophonie (MMF), and Japan Society for Mixed Methods Research (JSMMR)), organization of regional and global conferences, development of graduate courses and workshops, 15,16 and publications of books 14,17 and specialized peer-reviewed academic journals (eg, Journal of Mixed Methods Research). Within this burgeoning field, researchers have been interested in applying the conceptual and methodological developments in mixed methods research to secondary research (ie, reviews of primary studies). 4,5 Third, in the literature, there is a wide diversity of research questions addressed and study designs used, which make it possible to conduct mixed methods reviews. Also, there is a growing number of scholarly documents that are available on the web. 18 The extent of existing evidence can greatly influence the methods that can be used in systematic reviews. 19 Conducting mixed methods reviews is challenging because of the heterogeneous nature of the included studies. Challenges may become more apparent when writing the review protocol since several questions can be raised such as: How to formulate the review questions?, Should different search strategies be developed for each study design?, Which critical appraisal tools should be used?, and How should the different data be synthesised? Moreover, when conducting mixed methods reviews, as with mixed methods research more generally, researchers face the "integration challenge," that is, the challenge of producing a whole that is greater than the sum of each individual part. 20 In mixed methods research, having only qualitative and quantitative components is not enough; its added value is in the mixing of these two components together. Fetters et al 21

have described three
What is already known • Mixed methods reviews are challenging to conduct since they require the combination of qualitative and quantitative research.

What is new
• This case study analyzed 30 mixed methods reviews conducted in one organization and revealed five key dimensions on which reviews varied: types of questions answered and purposes of the mixed methods questions, types of evidence and sources, integration strategies and reasons for using a mixed methods approach. • The included mixed methods reviews were not limited to incorporating qualitative and quantitative research. This study suggests broadening the conceptualization of mixed methods reviews to take into account a variety of sources and types of evidence.

Potential impact for RSM readers outside the authors' field
• This paper can provide useful information for researchers having to conduct mixed methods reviews by providing examples of mixed methods questions, types of evidence that can be combined, justifications for conducting mixed methods reviews, and integration strategies. different levels where integration can be achieved in a mixed methods study: (1) design (eg, convergent and sequential design), (2) methods of data collection and analysis, and (3) interpretation and reporting of results. Currently, the literature of mixed methods reviews has mainly addressed integration at the level of the review's design and few papers have focused on integration at the other levels. Several methodological papers on mixed methods reviews have suggested different synthesis designs, that is, different models within which a review can be conducted to combine quantitative and qualitative research. 4,[22][23][24][25] To our knowledge, at the methods level, one paper has presented five integration strategies for combining evidence from qualitative and implementation studies within intervention effectiveness reviews: juxtaposing findings, using a logic model or conceptual framework, analysing program theory, testing hypothesis derived using subgroup analysis, and qualitative comparative analysis (QCA). 26 More guidance is needed on how to conduct mixed methods reviews and how best to achieve integration between the different components in this type of review. 27,28 Also, few studies have addressed the reasons for performing mixed methods reviews. A better understanding of these reasons can help to raise awareness about the potential of this type of review and generate new ideas for conducting future reviews.
The aim of this article is to explore the rationales for and approaches to mixed methods reviews, by focusing on those used by a research group that has been operating in the field of research synthesis for some time. More specifically, the research questions were: 1. How have mixed methods reviews been performed by this group? 2. How were mixed methods questions formulated by this group and for what purpose? 3. What types of evidence were used in these mixed methods reviews? 4. What reasons were reported for using mixed methods in these reviews? 5. How was integration performed?

| METHODS
A single case study was conducted to describe the mixed methods review processes used in one organization. The case consisted in the Department of Health and Social Care Reviews Facility. This case was chosen because this facility has produced many mixed methods reviews since 1995. From its inception to 2013, the reviews from this facility were conducted by researchers from the Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI-Centre) at University College London (UCL). Since 2014, three collaborating centres of excellence are involved in this facility: EPPI-Centre, UCL; Centre for Reviews and Dissemination (CRD), University of York; and Public Health, Environments and Society (PHES), London School of Hygiene and Tropical Medicine.
The method used for the case study consisted of document analysis. The documents were taken from a set of all reports produced from the Department of Health and Social Care Reviews Facility and published between 1996 and 2019 (March). The data collected and analyzed in this study are from reports that are openly available at this website: http://eppi.ioe.ac.uk/cms/Default.aspx? tabid=75. From a list of 67 available reports, the first author worked with the co-authors to identify those that the co-authors considered to be mixed methods reviews. The co-authors of this article have between them been involved in most of reviews produced within this organization. Since the co-authors have written, presented and taught about mixed methods, no prior definition of mixed methods reviews was provided to them. This was done to understand their conceptualization of mixed methods reviews, which it was thought might differ from the definition most common in the literature (ie, reviews incorporating qualitative and quantitative evidence 4,23 ). Team meetings were then held to clarify the co-authors' definition of mixed methods and the approach used in the included reviews.
In particular, discussions were held to understand the reasons for considering reviews that were not limited to those mixing qualitative and quantitative research. The co-authors argued that reviews should be considered mixed when they included more than one kind of research question or more than one type of evidence. Out of scope for this analysis were any reviews that addressed a single question by including only qualitative studies or only quantitative studies. It was also decided not to look, for the purposes of this analysis, at the Facilities' reviews that brought together only systematic reviews (reviews of reviews), or at the reviews that mapped the literature without synthesizing study findings.
In addition to the reports on the reviews, several scientific papers on the mixed methods review process have been published by researchers from this team. 1,9,[29][30][31][32][33][34][35] These papers were consulted to better understand the review process used.
This study focused on the process and methods used for conducting mixed methods reviews. For each included review, the following data were extracted: review process, review questions, number of included studies, types of evidence included, synthesis methods used, reasons for mixing different types of studies, integration strategies used.
Descriptive analysis was performed to understand the review process and the characteristics of the included reviews. In addition, a deductive content analysis was conducted using existing frameworks from the literature on mixed methods research. The reasons provided for conducting mixed methods reviews were analyzed using the Bryman 36 coding scheme, which includes a list of 18 justifications for performing mixed methods studies: (1) triangulation, (2) offset, (3) completeness, (4) process, (5) different research questions, (6) explanation, (7) unexpected results, (8) instrument development, (9) sampling, (10) credibility, (11) context, (12) illustration, (13) utility, (14) confirm and discover, (15) diversity of views, (16) enhancement, (17) other, and (18) not stated. The synthesis methods and designs were respectively classified based on the taxonomies suggested by Gough et al 37 (aggregative and configurative synthesis methods), and Hong et al 23 (convergent and sequential synthesis designs). Also, integration was analyzed using the mixed methods integration framework including three main types of integration strategies: (1) connection of phases (ie, the results of a first phase inform the data collection/ analysis of a second phase), (2) comparison of results (ie, the results of the qualitative and quantitative components are compared), and (3) assimilation of data (ie, the data of one component are transformed and then merged with those from the other component). 38 For each main type, three integration strategies are suggested (see Table 1).

| RESULTS
A total of 32 reports representing 30 different reviews were retained in this study. The reports were published between 1999 and 2019. Appendix 1 presents the characteristics of the included reviews (online supplementary file).

| Review process
The majority of reviews (n = 17) included two main stages: (1) systematic descriptive map to describe the nature and breadth of research activities and identify priority questions, and (2) in-depth systematic review to address the priority questions (see Figure 1). All included reviews followed the typical steps of systematic reviews; that is, searching for evidence in multiple sources, screening and selecting studies using specific inclusion and exclusion criteria, data extraction, evidence quality assessment, and synthesis usually performed by two and more reviewers. The quality assessment of included studies used pre-existing tools for each study design included in the reviews, such as the Cochrane Risk of Bias for randomized controlled trials, 39 AMSTAR for systematic reviews, 40 QATSO for observational studies, 41 the Assessment Form by Hawker et al 42 for qualitative studies, as well as tools developed by Facility researchers (eg, tools presented in O'Mara-Eves et al 43 ).
Most reviews (n = 25) involved stakeholders in some active way. Stakeholders were involved using different mechanisms such as a scientific advisory group, a steering group and individual or group consultations of various types. A range of stakeholders were involved in the reviews, such as policy specialists, review commissioners, practitioners, patients, representatives of groups advocating for patients or service users, clinicians, local authority staff, teachers, as well as researchers. This involvement work aimed to ensure that a review was aligned with stakeholders' needs and emerging policy requirements. Stakeholders helped to identify priority research areas and inform the scope of a review; provided advice on the questions, concepts, methods, strategies for dissemination and impact; helped interpret initial findings; and provided contextual understanding.

| Review questions
The majority of reviews (n = 25) included overlapping or complementary questions that addressed more than one of the team's broad type of evidence (see below-views of people on a phenomenon, intervention process, or intervention effectiveness). As an example, one review included a question about views (What are people's views about the barriers to and facilitators of effective workplace health?) as well as a question about effectiveness (What is the evidence available from systematic reviews for the effectiveness of workplace health interventions in improving health outcomes?). 44 These reviews were all considered mixed because of the different types of evidence they brought together. Four reviews formulated questions about only one of the broad type of evidence (in all four the focus was on people's views). The questions either: (a) addressed the views of one group of persons on different issues (eg, What are children's views about the meanings of obesity or body size, shape, or weight [including their perceptions of their own body size], and what experiences do they describe relating to these issues? 45 ); or (b) the views of different groups of persons (eg, What are patients', clinicians', and researchers' perspectives and experiences of diagnosis of Lyme disease? 46 ). These four reviews were all considered mixed because, despite having a focus on the same broad type of evidence, they had included both qualitative studies and quantitative studies (eg, interviewor focus-group studies were mixed in the review with surveys with fixed response questions).
One review asked questions about associations between factors. This review looked at the relationship between childhood obesity and educational attainment in the scientific literature (eg, What do we know about the relationship between childhood obesity and educational attainment, from the research literature?) and at the same time, asked stakeholders-young people and teachers-whether or not they believe there is an association, what factors might explain an association, and how important they might be (primary-level evidence). 47 This review was considered mixed because, while it focused only on one of the broad types of evidence, it combined evidence from research studies with evidence from the review team's work with stakeholders.
During the analysis, particular attention was paid to mixed methods questions. These were found in 21 reviews. In the literature on mixed methods research, mixed methods questions are defined as "questions that embed both a quantitative research question and a qualitative research question within the same question" (p. 483). 48 Applied to mixed methods reviews, a question was considered mixed methods when it drove the necessity of combining different types of evidence together. The questions could be overarching or overlapping other questions. Overarching mixed methods questions combine different components into one question. For example, a typical overarching question is "what is known about the [effectiveness, views, process,…] of an intervention?." Also, other reviews asked specific questions for each component (eg, What is the effectiveness of an intervention? and What are the views of users of that intervention?) and a mixed methods question combining both (eg, How do the users' views match the intervention evaluated?). These mixed methods questions addressed different purposes: 1. To compare findings from the views or process studies with those on effects of an intervention. 2. To identify critical intervention features based on findings from views or process studies. 3. To quantify the effect of critical intervention features from findings of views or process studies.
To make recommendations about future intervention evaluation research based on views or process studies. Table 2 presents examples of mixed methods questions. In these questions, the findings from the synthesis of views and/or process studies were compared with those from effectiveness studies.

| Types of sources of evidence
Five main sources of evidence were used in the included reviews: 1. Formal evidence from primary studies (n = 30): All reviews included formal evidence, that is, evidence from primary studies that directly addressed the questions in a review. The majority of the reviews included qualitative and quantitative study designs (n = 27). However, they did not always explicitly use the terms "qualitative" or "quantitative." Some reviews categorised studies with study design terms, such as surveys, quasi-experimental studies, correlational studies and trials, or by using umbrella terms such as "intervention studies" (used to signify any study that focused upon an intervention). In most reviews, the types of studies were classified in terms of their focus (on participants' views, intervention process, and/or intervention outcomes). Table 3 presents the different constructs related with each type of study. Even within these three overarching study types, different designs could be included. For example, views studies sometimes included qualitative studies and surveys using open ended or fixed response questions, and outcome evaluation studies could use randomized and non-randomized study designs. 2. Informal evidence from primary studies (n = 3): Three reviews included informal evidence, 46,54,55 that is, evidence that is not derived from formal data collection and analysis methods. One review extracted informal evidence on intervention processes, and two reviews extracted informal evidence on views of researchers from quantitative studies. 3. Policy documents (n = 2): Policy documents were used in two reports 44,56 to provide information on key characteristics of successful intervention, important components to inform mechanisms, and contextual information. 4. Systematic reviews (n = 5): Systematic reviews were used in five in-depth syntheses. 44,51,[57][58][59] Most of the systematic reviews were on the effectiveness of interventions.
5. Evidence from reviewers' work with stakeholders (n = 5): In five reviews, 46,47,54,60,61 stakeholders' perspectives on a topic were sought using data collection methods (recording of meetings, online or emailed questionnaires). Analyses of these data were presented in the review report and, to differing degrees, were then integrated with other types of evidence.

| Justifications for conducting a mixed methods review
Several justifications for why the review combined several types of evidence were mentioned. Using the coding scheme from Bryman, 36 10 reasons were identified: completeness (n = 16), contextual understanding (n = 4), credibility (n = 2), different research questions (n = 5), diversity of views (n = 5), enhancement (n = 12), explanation (n = 7), process (n = 5), triangulation (n = 3), and utility (n = 5). In addition to these justifications, two additional reasons were found: combining different types of studies was helpful to develop a framework and to identify promising interventions to further study. Table 4 presents the list of justifications as well as explanation and examples. These justifications are not mutually exclusive.

| Synthesis methods and designs used
The majority of reviews (n = 20) used more than one synthesis method. Two main categories of synthesis methods were used: configurative (arrange the findings from primary studies to generate or explore new understanding of a phenomenon) and aggregative (adding up the findings from primary studies to test hypotheses). 9 The reviews used predominantly configurative methods to synthesise both quantitative and qualitative evidence such as thematic summaries, thematic synthesis, narrative synthesis, framework synthesis, realist synthesis, and QCA. Some reviews (n = 9) also combined configurative methods with aggregative synthesis methods such as meta-analysis and meta-regression.
Regarding the synthesis designs used, the majority of the reviews conducted results-based convergent designs where the components were synthesized independently and then combined them into a cross-study synthesis (n = 22). Three reviews analyzed all the data using a same synthesis method (data-based convergent design). 56,57,68 Sequential design, in which the findings from one component were used to inform a second component, was used in five reviews. 43

| Integration strategies
Integration has been defined as the "explicit interrelating of the quantitative and qualitative component in a mixed methods study" (p. 33). 72 Applied to mixed methods reviews, integration consists in combining different types of evidence. The integration strategies used in the reviews were analyzed based on a framework presenting three general types and nine integration strategies (see Table 1). 38 Based on this framework, all three types and four strategies were found. Table 5 presents terms extracted from the included reports that referred to integration.

| Integration strategy for comparison of results
The most common type of integration strategies used in the reviews consists in comparing the findings from at least two different syntheses (integration 2.1 in Table 1). The comparison was mainly done by juxtaposing the findings from different syntheses in a table. This approach is referred to "matrix" in the literature. 26 A matrix allows for side-by-side comparison of the findings to identify matches and mismatches.
An example of comparison is presented in Figure 2. In this review, three syntheses were performed independently; the first two aimed at identifying factors that influence the decision to self-care and the third one aimed to study the effectiveness of interventions for promoting self-care for minor ailments. 62 Then, the findings from these syntheses were compared at two instances.
First, to answer the first question of the factors, they compared the findings from the qualitative and survey studies to identify the most important themes and subthemes. Second, the findings of the three syntheses were compared in a matrix consisting in a table presenting in a first column the barrier/facilitator and then three other columns presenting respectively the findings from the synthesis of qualitative, survey and evaluation studies. Each line presented one barrier/facilitator and the "We examine 'outcome evaluations' which are designed to establish whether or not an intervention works and 'views studies' which use qualitative and other types of methods to study young people's perspectives and experiences." 65

Diversity of views
Combine different perspectives and angles through different types of evidence.
"The comprehensive overview of PME that this systematic review provides enables new insights both through the up-to-date assessment of each issue and by bringing together evidence examining the issue from different angles." 55 6. Enhancement Augment quantitative findings with qualitative data (or the opposite).
"Qualitative research is often capable of opening up the so-called 'black box' of quantitative trials and epidemiological work on risk factors, to expose why interventions work or do not work." 50

Explanation
One synthesis is used to help explain the findings generated by the other.
"It draws on the views of users and providers of weight management services, taking these as a starting point to explore explanations for the varied effectiveness of different WMPs that have been evaluated." 60 8. Framework development Combine different types of evidence to develop or adapt a framework.
"We iteratively developed a conceptual framework for community engagement in public health interventions to address health inequalities based on an integration of findings from previous theoretical literature, outcome evaluations, process evaluations and cost/resource evaluations." 43 9. Process Provide information on process to understand the results on effectiveness.
"Formal process evaluation provides a context for understanding the results of experimental or policy interventions more fully." 66 10. Promising intervention Identify promising interventions to be developed and tested.
"When considered in conjunction with findings about the effectiveness of interventions, such corresponding findings from the syntheses. By juxtaposing the barriers and facilitators identified from the qualitative and survey studies against the findings from the evaluation studies, it was possible to analyze the extent to which the interventions matched the recommendations from syntheses 1 and 2, and whether the interventions meeting these recommendations were more effective or not.

| Integration strategy for connection of phases
The second type of integration strategies consists of connecting phases. In this strategy, the findings of a first synthesis will inform the data extraction and/or synthesis of a second one. This strategy is often seen in sequential synthesis design. Thus, when using this strategy, the syntheses cannot be done independently since the results of one synthesis influence the other. Figure 3 presents an example of connection. In a first synthesis, the researchers performed a thematic synthesis using data from qualitative studies to identify critical features of weight management programmes. Then, the findings from this first synthesis on critical features that were considered important by children, parents and providers were used to inform the second synthesis that aimed at identifying the most and least effective features using QCA on trials on service evaluations (integration strategy 1.1 in Table 1). This connection allowed the review to better take into account experiential evidence, to sensitise to conditions that may have been underemphasized in trials, and to provide more fine-grained evidence. 61

| Integration strategy for assimilation of data
The third type of integration strategy is assimilation of data. Assimilation refers to merging data together. This integration strategy is mainly observed in review where all the data are analyzed using a same synthesis method. Assimilation can require that data transformation be performed prior to the synthesis, either quantitizing or qualitizing (respectively, integration strategies 3.1 and 3.2 in Table 1). For example, when using QCA, data from qualitative studies may be converted into values of 0 (absent) and 1 (present). Conversely, when using a qualitative synthesis method such as thematic synthesis, the data from quantitative studies (eg, effect sizes, percentages) will be transformed into words that will be used to identify themes.

Justifications
Explanation Example views highlight a number of promising ways in which to develop and test future mental health promotion interventions." 59

Triangulation
Seek corroboration of findings from different evidence.
"…each of the inductively derived themes identified was corroborated in some way by evidence from the effectiveness synthesis." 55 12. Utility Combine different types of evidence to inform policy and practice.
"Contrasting the findings of research based on people's own descriptions of their lives with those from more 'expert-driven' research can raise important issues for policy, practice and research." 67  Figure 4 presents an example of assimilation strategy used in a review on no-fault compensation schemes. 56 In this review, the data from all empirical studies and policy reviews were analyzed using one similar synthesis method, realist synthesis (integration strategy 3.2 in Table 1). The data were used to generate theoretical frameworks on the mechanisms that might influence engagement in no-fault compensation schemes.

| Multiple integration strategies
In some reviews, several types of integration strategies were used. For example, a review on interventions to promote healthy eating among children 53 used two integration strategies ( Figure 5). First, they used a matrix approach in which they juxtaposed the findings from the synthesis of views studies against those of trials, and F I G U R E 2 Example of comparison integration strategy 62

F I G U R E 3 Example of connection integration strategy 61
identified the matches and gaps between these findings (comparison of results, integration strategy 2.1 in Table 1). Second, they used the results of this comparison to reanalyze the data from the trials (connection of phases, integration strategy 1.3 in Table 1). For each intervention that had a sufficient number of matches, they conducted statistical subgroup analysis. This allowed exploring statistical heterogeneity, and identifying promising directions for the development and testing of interventions to promote the consumption of fruits and vegetables among children.
In summary, a variety of strategies have been used to integrate the different components of the included mixed methods reviews. The most common category is comparison, where the results from different syntheses are compared using mainly the matrix approach. The other types F I G U R E 4 Example of assimilation integration strategy 56 F I G U R E 5 Example of comparison and connection integration strategies 53 (connection of phases and assimilation of data) were less frequently seen and have mainly been used in the past few years. Some reviews used more than one integration strategy.

| DISCUSSION
This case study analyzed 30 mixed methods reviews and focused on how and why to perform integration. Four integration strategies and 12 reasons for conducting mixed methods reviews were identified. The review process included mainly a systematic descriptive mapping stage followed by an in-depth review stage. 32 A mapping stage is recommended for systematic reviews of complex interventions to have an idea of the amount and type of evidence available in the literature and the types of interventions studied. 11 Also, stakeholders were involved at different stages of the review process, which can ensure greater relevance and uptake of results in decisionmaking. 73 One challenge when conducting this case study was about defining and conceptualizing mixed methods reviews. The included reviews were bespoke and driven by the questions that can be addressed based on the needs and concerns of the policy-makers, as well as the available evidence, resources and time. Most of the reviews involve multiple components covering different types of questions (eg, what, how, why), types of evidence (eg, views, effectiveness, process), sources (eg, primary studies, policy documents, systematic reviews), perspectives (eg, children, parents, clinicians), and synthesis methods (eg, QCA, metaanalysis, thematic synthesis). From the included mixed methods reviews, four general categories of combinations could be identified: (1) different types of studies, (2) different synthesis methods, (3) primary-and secondary-level evidence, and (4) formal and informal data. One or several combinations could be used in a review.
The first category consists of combining different types of studies. This is coherent with the literature on mixed methods research synthesis and mixed studies reviews that defines this type of reviews as reviews combining qualitative, quantitative and mixed methods primary studies. 22,30,74,75 However, the studies included in the reviews that were analyzed in this case study were not always categorized as qualitative and quantitative. The research team mentioned they tend to avoid categorizing studies as "qualitative" and "quantitative" research since their distinction is not always neat and precise; both types of research can include features of either type. 1,30 They suggested using instead "numerical" and "textual" data and limiting the use of the qualitative and quantitative labels to the type of synthesis. 30 Moreover, the studies used in the reviews were often classified as view, process, and outcome evaluation studies. Outcome evaluation studies have been the most frequent type of studies included in systematic reviews to address the effectiveness of interventions. More recently, process evaluation and views studies have been used in systematic reviews to help in understanding how an intervention was delivered 76 and what are the individuals' perspectives and experiences of an intervention. 29 Several designs (qualitative and quantitative) can be used in process evaluation and views studies. Combining these types of evidence together allowed for better understanding of which interventions are most effective or promising. Also, the included reviews were not always limited to primary studies. Depending on the available literature and the research questions, some of the analyzed mixed methods reviews also included systematic reviews. Using systematic reviews can be more efficient in terms of time and resources. For example, in some fields, the literature on effectiveness can be more abundant than those on views or process. Some reviews used both a secondary (ie, systematic review of primary studies on views) and a tertiary (ie, review of systematic reviews on effectiveness) level of research analysis. 9 The second category is combining different synthesis methods. Using mixed synthesis has been mentioned in several conceptual papers on mixed methods reviews. 1,4,75 Mixed synthesis consists of using "qualitative" or configurative methods (eg, thematic synthesis) and "quantitative" or aggregative methods (eg, statistical meta-analysis) in a review. Also, recent reviews have used QCA, which is a synthesis method that aims at identifying the necessary and sufficient intervention components that can be associated with given outcomes. 77 This method has been considered as a mixed methods approach to synthesize evidence since it transcends the qualitative/quantitative divide. 77 Indeed, QCA combines both qualitative synthesis (to identify important features for successful interventions) and quantitative synthesis (to test the association between features and the effectiveness of the intervention). 26 In the literature, new synthesis methods for integrating qualitative and quantitative evidence have been suggested. For example, Thompson Coon et al 78 have developed the interweave synthesis approach that rely on team work using intersubjective questions and immersion in the entirety of the evidence base during the final stages of the synthesis of each review. Another example is provided in van Grootel 79 that quantified findings from qualitative studies and used Bayesian meta-analysis to match findings from qualitative and quantitative studies.
The third category combined primary-and secondary level evidence. Primary-level evidence was collected from policy documents and work with stakeholders to provide complementary evidence on the perspectives of stakeholders from local settings. 44,46,47,54,56,60,61 The research team emphasised that work with stakeholders are underpinned by two aspirations. The first is ethical: to enable the voices of often marginalised patient and public groups to be heard and to address health and social inequalities. The second is practical: to ensure the reviews are relevant and useful to stakeholders. Incorporating the views of those directly affected by policies, interventions and strategies means the reviews address current concerns which in turn increases the likelihood that they will be useful for informing health and socialcare practice. The primary-level evidence can be useful to provide contextual information such as understanding how interventions have been developed and implemented. 80 A better understanding of the contextual evidence can also contribute to help decision-makers address real-world challenges. 73 This category of combination is comparable to multi-sources synthesis in which the use of interviews with patients were found useful to improve the local relevance of results from a systematic review, find unidentified needs, and served as "contextspecific triangulation." 8 Multi-sources synthesis integrates data from both primary and secondary research. 8 A last category is about combining formal and informal data from scientific evidence. This is new in the literature on mixed methods reviews. One review including only quantitative studies was considered mixed because it used informal evidence on process. 55 Informal process data are usually reported in the discussion of studies and address the experience of developing and implementing an intervention. 81 A methodological approach was developed to analyze informal evidence, the intervention component analysis (ICA) approach. 81 ICA is used to explore the features of an intervention to better understand how and why interventions might work by uncovering aspects or new configurations that are important for their effectiveness. 81 Informal data on views were also extracted from quantitative studies in two reviews, which allowed to compare the views of different stakeholder groups. 46,54 These informal data can be considered "qualitative" or "textual" data since they are derived from line-by-line coding and analyzed using inductive thematic analysis.
One core characteristic of mixed methods research is the integration of qualitative and quantitative data and results. 14 Integration was seen in all included reviews. Four integration strategies were used covering the three types of integration seen in mixed methods research: connection of phases, comparison of results, and assimilation of data. 38 The main integration strategy used in the included reviews is the matrix approach, that is, juxtaposing the results of the syntheses to allow for comparison of results of syntheses (strategy 2.1 in Table 1). 30 This approach is similar to what is named "joint display" in the mixed methods research literature, which consists in "a visual means to both integrate and represent mixed methods results to generate new inferences" (p. 555). 82 In this case study, over 80% of the analyzed reviews used a convergent synthesis design. This is coherent with a review that found that more than 95% of mixed studies reviews used convergent designs. 23 In the recent JBI Reviewer's Manual, due to the paucity of usage of sequential synthesis design, methodological guidance is provided exclusively for convergent designs. 3,83 In this case study, the reviews that used sequential synthesis design mainly aimed at identifying intervention components and then analyzing evidence on their effectiveness. Other uses of sequential synthesis design can be found in the literature. For example, in a mixed methods reviews on the barriers to hypertension control, the qualitative studies were first analyzed to develop a conceptual framework of barriers that then served to organize the quantitative studies and calculate proportion of participants for each identified barrier. 84 The mixed methods reviews analyzed in this case study used different critical appraisal tools for each study design. This approach can be time and resources consuming since it requires searching, selecting and learning how to use different tools. In the literature, some tools have been developed to appraise the quality of studies included in mixed methods reviews such as the QualSyst Assessment tool, 85 86 and Mixed Methods Appraisal Tool (MMAT). 87 Such tools could facilitate the quality assessment process in mixed methods reviews since they provide criteria that could be applied to several study designs. However, further validity and reliability testing of these tools is still needed.
Beyond the divide and technical aspect of mixed methods research (qualitative vs quantitative methods), Greene 88 suggested focusing attention on ontology and epistemology. She defined mixed methods inquiry as a "planned and intentional incorporation of multiple mental models" for the "purposes of generatively engaging with difference toward better understanding of the phenomena being studied" (p. 30). 88 The included reviews dealt with evidence of different nature for greater depth and breadth in understanding of complex phenomena, which would not have been possible with only one type of evidence. The reasons for conducting the reviews (Table 4) are very similar to those found for mixed methods studies. In light of the literature on mixed methods research and the results of this case study, the conceptualization of mixed methods reviews could be broadened. It is about making the most of the available evidence by combining a diversity of sources, types of evidence and/or methods for the purpose of gaining a more complete and deeper understanding of a complex phenomenon.

| Strengths and limitations
In addition to the synthesis designs, this study focused on the integration strategies used. Integration designs present ways of structuring reviews (eg, sequential vs convergent designs as seen in mixed methods research 23 ), while integration strategies can provide practical information on how to perform integration. Although some integration strategies are found in specific designs (eg, connection of phases is usually used in sequential design), designs and strategies are not necessarily bounded since several combinations of strategies can be used within one study design. 38 Focusing on strategies allows for better understanding of how integration was achieved in the reviews.
The results presented in this paper were from a single case study. The case consisted in one organization with extensive experience in conducting mixed methods reviews. The process described in this paper represents general steps that were found in most included reviews and was mainly used for reviews on interventions in health policy. Variations were seen; for example, some reviews did not include a systematic mapping step, the level of stakeholders' involvement varied, and the types of evidence and questions differed. This process is flexible and should be adapted as needed. Several factors should be taken in consideration when developing a protocol for a mixed methods review such as the needs of the stakeholders, the research questions, and the time and resources available.
The analysis was performed by one researcher who was not involved in the production of the included reviews. This allowed for an external and independent view of the review process. Discussions with the research team were needed to clarify how they conceptualized mixed methods review and the steps involved.

| CONCLUSION
This study described the process used to conduct mixed methods reviews in one setting and provided examples of mixed methods questions, constructs that can be combined, justifications for conducting mixed methods reviews, and integration strategies. This study also suggested broadening the conceptualization of mixed methods reviews to take into account the variety of sources and types of evidence. The analysis used existing frameworks from the literature on mixed methods research to better understand how and why mixed methods reviews have been produced. It would be interesting to repeat this method to analyze the mixed methods review processes from other research teams and in other fields, and compare them together. It is hoped that the information garnered from this study will provide useful information for reviewers embarking on mixed methods reviews and trigger new ideas for conducting this type of review.

CONFLICT OF INTEREST
The author reported no conflict of interest.

DATA AVAILABILITY STATEMENT
The data collected and analyzed in this study are from reports that are openly available at this website: http:// eppi.ioe.ac.uk/cms/Default.aspx?tabid=75.