Over the course of the past year, the Journal has received a number of manuscripts that purport to use ‘mixed methods’, whether in the form of sequential mixed methods, amalgamated mixed methods or integrated mixed methods using both qualitative and quantitative data. For a number of us, it is exciting that we have moved beyond the restrictive paradigms of the qualitative-quantitative divide. Unfortunately, what often comes out in the Results section is what usually happens when you mix rich soil and pure bottle water: you end up with mud.
The need to use both qualitative and quantitative methods is generally based on the question one wishes to answer.1 In mixed methods research the question being posed should require the use of both methods. More usually, the researchers have taken the more generalist approach of using both methods because it provides more information.
A review of the submitted manuscripts that state they have used a mixed method approach indicates that many have been rejected on sound reasons by reviewers who are both quantitative and qualitative methodologists. They typically identify a lack of understanding by the researchers of the level of rigour required to successfully execute one or other type of research. Common criticisms are inappropriate integration of the two methods and a ‘half baked' approach in the use of either or both. As noted in a previous editorial,2 a good method is the best guarantee of success in the Journal.
A proposal for funding can be written in such a manner that reviewers of that proposal instinctively believe there will be two studies for the price of one. Unfortunately, whether during the course of the study or in an error by the researcher, what is written up is neither a quality quantitative study nor a quality qualitative study.
To execute a mixed methods study well is exceptionally difficult. Mixed method researchers have to address the same validity and reliability issues with their quantitative data as with their qualitative data, including the need for appropriate sample sizes and study populations, biases and confounding, validity and reliability of their instruments, and design of their analysis plan. The principles of research science must be maintained for both the quantitative and the qualitative data.
On the rare occasions when they are done well, these are exciting studies and provide excellent additions to the literature in an area.
Mixed methods studies submitted to this Journal fall into several groups, each facing their own challenges. The first group of studies involve the merging of data, where the researcher uses qualitative data and quantitative data to a convergence on a result.3 The idea the researchers are exploring is that both the qualitative data and quantitative data support the findings. This is one of the more challenging types of mixed methods to report because, at the end, both sets of data need to stand on their own. One issue is that a sampling framework must be developed that meets the requirement of both methods. Generally, if one sampling framework is used, one or other method will be compromised. The challenge is not only conducting the two studies but also reporting both qualitative reporting and quantitative reporting of the data methods and results within a Journal's word limits. Of all the mixed methods designs this is one the strongest.
The second group of mixed methods studies use sequencing,3 where a qualitative study influences the design of a subsequent quantitative study (or vice versa) to build and explore a sequence of ideas to yield a result. This sequential method also has challenges in that each phase of the overall study needs to stand on its own and be able to be so reported. However, as one study influences the next, the reporting of the data is also sequential. The quantitative data influences the qualitative data that is collected in study two, which then influences an either qualitative or quantitative design in study three, and so on. This mixed method design can provide robust and rich results but is a challenge to report succinctly.
The third type of mixed methods3 is where researcher uses an amalgamated method. In reporting this type of study, qualitative data sit within the quantitative data to yield a result. This can be a challenge as, again, not only both data types must be able to stand on their own but the authors must also explain this amalgamation. For example, a particular dichotomised variable might have a complex set of explanations for one of the response options which warrant further explanation. Commonly, this process is not clear to the reviewers and the editors.
One of the greatest challenges for qualitative researchers has always been word limits.4 This Journal has accommodated outstanding qualitative research by extended word limits, and a similar approach is taken with mixed methods papers. However, while a paper may be novel, if the contents are not succinct, it cannot be supported. All researchers have the challenge of distilling and highlighting the key elements of their results, and with dual results mixed methods researchers face a greater challenge.
Dealing with a plethora of data is always a challenge, but the common problem of data mining in mixed methods papers tends to make them redundant and superfluous. Data which provide the same information reported in a previous publication are not of interest and also create an academic redundancy problem. For mixed methods researchers, reporting the value adding of the dual data is the important and interesting challenge, not just repeating the same result.
While a challenge, mixed methods research provides exciting and rewarding methodological challenges for researchers. It takes training and experience in both research traditions and clarity on the researcher's part to be able to apply appropriate analytical and interpretative techniques of both methods to support their results of either one method. The risk is that what comes out is the muddy mixing of water with dirt so that the scientific integrity that Journal reviewers require for acceptance of the manuscript has been lost.