• Open Access

Clear as mud: results and the use of mixed methods


  • John Lowe

Journal contact details
Mail:Australian and New Zealand Journal of Public Health, PO Box 351, North Melbourne, Victoria 3051.
Street deliveries:c/- SUBStitution Pty Ltd, 1st Floor, 484 William Street, Melbourne, Victoria 3003.
Phone:(03) 9329 3535Fax:(03) 9329 3550

Over the course of the past year, the Journal has received a number of manuscripts that purport to use ‘mixed methods’, whether in the form of sequential mixed methods, amalgamated mixed methods or integrated mixed methods using both qualitative and quantitative data. For a number of us, it is exciting that we have moved beyond the restrictive paradigms of the qualitative-quantitative divide. Unfortunately, what often comes out in the Results section is what usually happens when you mix rich soil and pure bottle water: you end up with mud.

The need to use both qualitative and quantitative methods is generally based on the question one wishes to answer.1 In mixed methods research the question being posed should require the use of both methods. More usually, the researchers have taken the more generalist approach of using both methods because it provides more information.

A review of the submitted manuscripts that state they have used a mixed method approach indicates that many have been rejected on sound reasons by reviewers who are both quantitative and qualitative methodologists. They typically identify a lack of understanding by the researchers of the level of rigour required to successfully execute one or other type of research. Common criticisms are inappropriate integration of the two methods and a ‘half baked' approach in the use of either or both. As noted in a previous editorial,2 a good method is the best guarantee of success in the Journal.

A proposal for funding can be written in such a manner that reviewers of that proposal instinctively believe there will be two studies for the price of one. Unfortunately, whether during the course of the study or in an error by the researcher, what is written up is neither a quality quantitative study nor a quality qualitative study.

To execute a mixed methods study well is exceptionally difficult. Mixed method researchers have to address the same validity and reliability issues with their quantitative data as with their qualitative data, including the need for appropriate sample sizes and study populations, biases and confounding, validity and reliability of their instruments, and design of their analysis plan. The principles of research science must be maintained for both the quantitative and the qualitative data.

On the rare occasions when they are done well, these are exciting studies and provide excellent additions to the literature in an area.

Mixed methods studies submitted to this Journal fall into several groups, each facing their own challenges. The first group of studies involve the merging of data, where the researcher uses qualitative data and quantitative data to a convergence on a result.3 The idea the researchers are exploring is that both the qualitative data and quantitative data support the findings. This is one of the more challenging types of mixed methods to report because, at the end, both sets of data need to stand on their own. One issue is that a sampling framework must be developed that meets the requirement of both methods. Generally, if one sampling framework is used, one or other method will be compromised. The challenge is not only conducting the two studies but also reporting both qualitative reporting and quantitative reporting of the data methods and results within a Journal's word limits. Of all the mixed methods designs this is one the strongest.

The second group of mixed methods studies use sequencing,3 where a qualitative study influences the design of a subsequent quantitative study (or vice versa) to build and explore a sequence of ideas to yield a result. This sequential method also has challenges in that each phase of the overall study needs to stand on its own and be able to be so reported. However, as one study influences the next, the reporting of the data is also sequential. The quantitative data influences the qualitative data that is collected in study two, which then influences an either qualitative or quantitative design in study three, and so on. This mixed method design can provide robust and rich results but is a challenge to report succinctly.

The third type of mixed methods3 is where researcher uses an amalgamated method. In reporting this type of study, qualitative data sit within the quantitative data to yield a result. This can be a challenge as, again, not only both data types must be able to stand on their own but the authors must also explain this amalgamation. For example, a particular dichotomised variable might have a complex set of explanations for one of the response options which warrant further explanation. Commonly, this process is not clear to the reviewers and the editors.

One of the greatest challenges for qualitative researchers has always been word limits.4 This Journal has accommodated outstanding qualitative research by extended word limits, and a similar approach is taken with mixed methods papers. However, while a paper may be novel, if the contents are not succinct, it cannot be supported. All researchers have the challenge of distilling and highlighting the key elements of their results, and with dual results mixed methods researchers face a greater challenge.

Dealing with a plethora of data is always a challenge, but the common problem of data mining in mixed methods papers tends to make them redundant and superfluous. Data which provide the same information reported in a previous publication are not of interest and also create an academic redundancy problem. For mixed methods researchers, reporting the value adding of the dual data is the important and interesting challenge, not just repeating the same result.

While a challenge, mixed methods research provides exciting and rewarding methodological challenges for researchers. It takes training and experience in both research traditions and clarity on the researcher's part to be able to apply appropriate analytical and interpretative techniques of both methods to support their results of either one method. The risk is that what comes out is the muddy mixing of water with dirt so that the scientific integrity that Journal reviewers require for acceptance of the manuscript has been lost.

In this issue

Recent years have seen a shift towards a systematic review of the literature as a stand-alone research project. Systematic reviews come in a variety of forms and the editorial by John McCall and Jennie Connor sets out procedures for conducting such a review.

Food and drink is of increasing importance in public health but the papers in this section demonstrate that there is a range of concerns. We need to educate people to drink water to beat heat stress (Tracey Oakman and colleagues); given that Australia is now fortifying bread-making flour with folate, we need a baseline measure against which we can assess impact on neural tube defects (Samanthi Abeywardana and colleagues); and we need data on habitual salt intake so that we can assess salt reduction programs (Karen Charlton and colleagues).

The section on Alcohol and tobacco starts with a study by Paul Dietze and Michael Livingston on risky drinking by older adolescents in which we learn that alcohol sourced only from parents is safer than previously thought. Michael Livingston and colleagues argue that we need to deal with alcohol harm, which is increasing substantially, especially in small, high-risk groups. Bruce Bolam and Leonard Sunil Piers focus on what would seem to be an eminently avoidable risk factor: smoking in pregnancy. Here, the contribution of the traditional high-risk groups is less evident so they recommend a whole-of-population approach to prevention. Jae Cooper and colleagues analyse the success of the ban on smoking in licensed venues although this does not mean lower smoking rates elsewhere.

Public health faces a rising rate of Chronic illnesses, which is particularly difficult because people with chronic illnesses commonly suffer from multiple morbidities. Clive Aspin and colleagues analysed health policy in Australia and New Zealand to show that their demands on the health systems are being ignored and now require explicit policies. People with chronic illnesses can monitor their own health states but, argue Anne Taylor and colleagues, high blood pressure and cholesterol are often under-estimated in comparison with clinic readings. Chronic illnesses sometimes requires a number of simultaneous medications. Penny Buykx and colleagues show that when people present to an emergency department with an overdose, the most common culprit is benzodiazepines and the most common source of the medication used is from the patient's usual doctor. Prevention strategies therefore need to focus on the time of prescription.

Andre Renzaho and Gery Karantzas introduce the section on Inequalitites with a study that shows the importance of a healthily functioning family in fostering empathic, caring and responsible children. Programs need to target stressors on the family. Mridula Bandyopadhyay and co-authors analysed the post-childbirth experience of immigrant women in Victoria. Immigrant women fared remarkably well but when they had poor command of English they faced numerous additional challenges and needed practical and emotional support. If access to health care is difficult for some population groups, the idea of using the internet for health information is appealing. Nova Reinfeld-Kirkman and colleagues report increasing health information-seeking on the Internet. But there are inequalities with those employed, for example, accessing information almost twice as often as the unemployed. Less access to the Internet is compounded by illiteracy, pointing to the need for a greater variety of health information on the web.

Acquiring a tan has been important Lifestyle issue for many people, young and old, but it carries the risk of skin cancer. Kate Francis and colleagues show that use of the easy route to a tan, the solarium, is relatively low in Australia and appears to be declining.

Lastly, our Letters to the Editor contain short, sharp gems and the books reviews are always worth reading.