Qualitative method and the curse the illustrative quotation
Article first published online: 6 OCT 2009
© 2009 The Authors. Journal Compilation © 2009 Public Health Association of Australia
Australian and New Zealand Journal of Public Health
Volume 33, Issue 5, pages 405–406, October 2009
How to Cite
Daly, J. (2009), Qualitative method and the curse the illustrative quotation. Australian and New Zealand Journal of Public Health, 33: 405–406. doi: 10.1111/j.1753-6405.2009.00419.x
- Issue published online: 6 OCT 2009
- Article first published online: 6 OCT 2009
A biomedical researcher, a cynic, once said to me, “Of course, it is easy to do qualitative research. All you have to do is take a recorder and round up three or four friends.” He added that a qualitative research addition to a well-designed randomised controlled trial did produce nice quotations and personal insights that could illuminate an otherwise rather dry presentation of tables and figures. And there we have it, the lure of the illustrative quotation.
My colleague saw the use of direct quotations in reporting qualitative research as comparable with the use of ‘sound bites’ in the media where quotes are grabbed from a member of the public who has just happens to witness an accident or other newsworthy event. To qualitative researchers this is an alarming thought. Qualitative research produces evidence for practice and policy making; a media report based on sound bites has a different function. So, let us consider the difference between the two.
In media reports, quick sound bites from three or four bystanders have an immediate impact when all we know about an event is what these people have just witnessed. The opportunity for quick sound bite grabs does not last long. Within hours the media has to present the event in a more thoughtful manner. Media reports may continue to use what look like sound bites but these are more often quotations from people identified by particular roles that give them access to more detailed knowledge of the situation: the chief of police, the local member of parliament, etc.
The problem for this Journal is that some qualitative methods articles submitted to us make extensive use of illustrative quotations but they read much like media reports at the sound bite stage. The quotations used are evocative and they appear to project authenticity. Usually we are told that ‘many’, ‘most’ or ‘some’ of the participants held certain views, followed by the quotation that is identified by an anonymous demographic factor such as ‘50 year old man’.
The problem with this reporting of a research project is that it gives little indication that research has actually been conducted. The splat of data in a quotation can illustrate a view, illuminate a problem but it does not amount to analysis. We cannot tell whether it is a superficial sound bite or the quotation represents generalisable views expressed in a properly conducted and analysed study. In short, authors need to persuade us by the methodological detail provided that a rigorous study has been conducted in a manner that excludes bias and impressionistic description.
Let us take an hypothetical example of the way in which things can go wrong in a paper. Imagine reviewing a paper where the introduction tells of the legal and moral problems surrounding termination of pregnancy. The aim of the paper is therefore to give the perspective of women who have undergone the procedure. The authors conducted 39 interviews with women from community groups selected to give a diversity of class and ethnic backgrounds. Study participants were recruited until data were ‘saturated’, that is, until no new information was generated. Under results, the paper reports that ‘many’ women undergoing termination of pregnancy suffered longlasting trauma. This point is illustrated by a series of quotations like this one:
“After I left the hospital I was crippled by mental anguish. I bled and bled and when I went to the toilet I thought that bits of my baby were still coming out. Now I am so depressed. I will never get over the experience. Nothing will ever persuade me that this is the right thing for women.”
The paper concludes that public health should oppose the legalisation of abortion.
When such a paper is sent for review, an experienced qualitative researcher would want a substantial revision (if kind) or recommend a rejection. The questions asked would start with the sampling procedure. The reviewer would want to know which social groups were identified as central to the study. Given that the quantity of data collected from each research participant is large, qualitative research methods use relatively small samples so research participants have to be selected to obtain the best, most relevant information for addressing the research problem. A review of the literature on abortion and a review of relevant social theory would be the most common sources. In the paper we need to be told from which community groups the study participants were drawn and how well the resultant sample represented the range of issues identified in the literature. In other words, was this an appropriate sample or could it be biased, likely to produce one view of the issue only?
We then need to know how the data were analysed. If ‘many’ of the women held these views, how well does each quotation represent the views of the many? Did the women all hold views this strong or has the quotation been selected for its knockout qualities? If ‘many’ women held this view, what about the others? If there were one or more other groups, what were their views and what distinguishes these groups from the ‘many’? What effort was made to identify and interview groups with divergent views?
Importantly, we need to have enough information from each of the groups to be able to explain the different views held. Qualitative research analysis is not only about the views of a majority but about presenting the different views held and then helping us to understand how these differences come about. In the old adage, research aims to describe, then compare, then explain. The same is true of research using qualitative methods.
Since this is fundamental to the recommendations made, how might we explain the strong views held against termination? Were the women in the sample, for example, likely to hold religious views or have cultural values opposed to termination? If so, was a determined effort made to sample from groups with different views? If the researchers have failed to include opposing views, how limited are the findings of the study and how cautious should we be about recommendations for policy change?
This is an example of what a reviewer needs to know when assessing the evidence generated by a study using qualitative methods. The decontextualised illustrative quotation does not allow us to judge whether the selection of quotations is motivated by the search for impressive sound bites or by rigorous analysis. On the other hand, evocative quotations are an important feature of qualitative methods and we certainly want to retain them in our articles. They make our research come alive. But we need to locate the quotation in the overall analysis. There is certainly room for selecting an outstandingly vivid quotation but then we need to be told that this has been done and, if possible, explain why we have chosen this ‘outlier’. To return to our example, a strong view against the legalisation of abortion may be explained by the strong views held by particular cultural or religious groups and may not be evident in others. Can we tell under what circumstance these views come to dominate?
If a qualitative paper presents quotations as an integral part of the analysis, the article will rapidly exceed the low word count accepted by many health and medical journals. One response is to present a series of extracted quotations separately in a box which, according to the rules, is excluded from the word count. This process compounds the decontextualisation of the data, the analysis suffers accordingly, and it is not generally recommended (but there are always exceptions).
In 2008, we revised our Journal procedures and changed the word limit for long articles to 7,000 words, including in this count the abstract, references, tables, boxes and figures. Our hope was that this would be of benefit to researchers reporting studies using qualitative methods. They should now be able to report on the full range of views of study participants, with proper attribution of quotations, and integration of the quotations into the analysis itself.
In this issue
As we go to press we are absorbing the Preventive Health Taskforce first report, Australia: the healthiest country by 2020 available from http://www.preventativehealth.org.au/internet/preventativehealth/publishing.nsf/Content/national-preventative-health-strategy-1lp. The report emphasises many of the issues that have emerged as important in articles published in this Journal. We will be monitoring Australia's response to this report and will report on it in the next issue.
The editorials in this issue address the critical issues of the importance of collaboration in Indigenous health and research, and the ever-important issue of methodological rigour in public health research. Methodological rigour is also the focus of the first section, including a paper in which the authors talked to researchers and community liaison officers to find out about obstacles to effective research when ways of conducting research in disadvantaged (particularly Indigenous) communities There are also papers about how to measure changes in falls in older people, and, from rural Victoria, a paper describes a method for developing a score for measuring access to services applicable to a rural setting.
A number of papers consider issues raised in the Preventive Health Taskforce Report. The Food and Nutrition section includes papers addressing the connection between SES and food (especially fruit) consumption, and a review of the literature concerning retail concludes that a lack of competition is a negative public health effect, both outcomes of considerable importance in Indigenous communities.
A report of a trial of up-skilling the dental therapist workforce is presented as a practical way to improve rural dental service access, and mammography costs using different screening procedures and work-related musculoskeletal injury complete our papers on Improving Health.
We include four papers that address important and controversial ethical issues, starting with risk factors for, and demonstrated hepatitis C virus in, a cross-section of endoscopy patients; a team from New South Wales has thought about the needs of refugee children, comparing those attending a clinic with the number of children known to be residing in the areas, and demonstrate some important shortfalls. A paper and accompanying commentaries debate important issues around an analysis of different ways of obtaining consent from relatives for organ donation.
With summer approaching, Westmead Hospital's entomology department provide some evidence about the implications of joint use of sunscreen and insect repellent, showing that application of sunscreen over repellent reduces its efficacy; and the Cancer Council provides the results of an audit of solariums, showing a fall in numbers following legislative changes. The Burnet Centre tells us about other changes, this time in risky sexual behaviour in cross-sections of young people attending the Melbourne Big Day Out showing an improvement in protective actions.
An interesting letter and four book reviews complete our Journal.