• Open Access

Evidence pyramids, rigour and ethics review of public health research


Journal contact details
Mail: Australian and New Zealand Journal of Public Health, PO Box 351, North Melbourne, Victoria 3051.
Street deliveries: c/- SUBStitution Pty Ltd, 1st Floor, 484 William Street, Melbourne, Victoria 3003.
Phone: (03) 9329 3535 Fax: (03) 9329 3550

As Bob Dylan once sang, “Ah, but I was so much older then, I'm younger than that now”.

At the start of my training in public health I was presented with the classic evidence pyramid for scientific studies, portraying strength of research evidence. This construction has been an important part of my epidemiological epistemological framework, and without it I would have undoubtedly been the poorer.

Since then, I have practised and taught public health, including research methods, and developed and supervised various research projects. I also sit on a human research ethics committee (HREC). I am often reminded of the evidence pyramid, with blinded clinical trials at the pinnacle, moving downwards through case control and cohort studies, to case series and case studies, and on to ever-weaker designs. The pyramid enables us to assess the strength of clinical evidence and there is little debate about using this hierarchy in assessing the effectiveness of drugs, vaccines, and clinical interventions.

The Introduction to the 2007 NHMRC Australian Joint Statement on Ethical Conduct in Human Research states: “Unless proposed research has merit, and the researchers who are to carry out the research have integrity, the involvement of human participants in the research cannot be ethically justifiable.”1 Problems arise when members of ethics committees equate the most scientific design – meaning as close to a blinded trial as possible – with scientific merit, which somehow becomes synonymous with the most objective, and ethical, way to do research. This misappropriation of the evidence pyramid is a problem: even within the realm of quantitative studies, this assumption cannot possibly be correct.

Clinical trials are the best evidence sometimes– but at other times trying to answer a question with this study design is impossible, even unethical. We need different quantitative designs to provide an overview of the size and shape of public health problems, enabling us to pass a lens over quantitative data so that we can select groups for additional attention, intervention, or special assistance. Today my evidence framework for what qualitative researchers often call ‘positivist’ studies looks more like a continuum, with various designs of equal importance for answering the questions we ask.

Frameworks used in quantitative methods, however, only answer some public health research questions. For example, these cannot tell us why our population thinks the way it does about climate change, why women seek or avoid HRT or screening programs, what Australians think about mental health issues, or how new parents find the responsibilities – and demands – of parenthood. We need qualitative study designs to find out about all those ‘what’ and ‘why’ questions. Qualitative data provide us with a deeper understanding of an issue, usually from the perspective of one or more groups of people most affected by a phenomenon. Critical, ecological and constructivist non-mathematical designs are seeking the answers to different problems, an explanation for an identified effect, or an implied consensus or explanation.

We can, in fact, learn a great deal about rigour from one another. Logically, context provides evidence, and the need for a particular kind of evidence drives the research context.2

A common point of argument, relevant for ethical review is about the number and method for recruiting study participants. There seems to be a belief ipso facto that quantitative studies have large sample sizes and qualitative studies have small ones. Most quantitative public health studies require a sample size calculation based on statistical power as part of the design, so that research resources are not wasted and to avoid Type II error, and need a clear recruitment strategy to ensure that the people in the study are representative of the various populations under study. But not all quantitative studies are large and complex – some work has involved very few people and the December 2005 issue of this journal includes studies of important communicable diseases with sample sizes of one or very few. Conversely, not all qualitative studies are conducted using small numbers of participants. Excellent examples exist which, by comparison with the above studies, include large sample sizes.3 The number also depends on the context.

Which design questions should apply to non-mathematical study designs when researchers are seeking to investigate a question of population importance? HREC members reading the application need to be sure that the evidence resulting from the study will truly represent the population the researcher says it will. In qualitative designs the rationale for recruitment, including the participant groups and the likely numbers of participants, should clearly be derived from the literature review and the theoretical framework for the study.4 The design of qualitative studies requires possible or likely sample size justification to allow recruitment for ‘data saturation’ (if there can be said to be such a thing), and include clear strategies for recruiting these participants. Over the past few years, several useful papers on the rigorous conduct, analysis and interpretation of qualitative research studies have been published in this Journal (see in particular October and December 2007 and April 2009), tackling these issues in various ways not addressed in standard public health research methods textbooks.

HREC members need to be able to apply criteria for rigour in assessing study designs regardless whether quantitative or qualitative. As public health questions become more complex, study design becomes increasingly important. It is high time for public health researchers to abandon the ‘them and us’ of yesteryear and think about rigour in all research. By having and using frameworks for assessing rigour in both qualitative and quantitative methods they achieve equal importance. At last qualitative research might stop being regarded as ‘soft’ research, just as quantitative studies might stop being regarded as ‘just numbers’. So here's to ‘youth-ing’.

In this issue

The aim of public health is to forestall disease, morbidity or death through Health Promotion. Our first challenge is to be clear about what is causing the harm and that is what the first four articles address. Ross Bailie and colleagues show that inadequate fluoridation of community water supplies poses a significant health risk in the form of dental caries for children living in remote Northern Territory communities. They call for water fluoridation in remote Northern Territory locations. Peter Buzzacott and colleagues tackle a very different problem. They show that certified recreational scuba divers have fewer breaches of safe diving practices. Requiring certification of divers can reduce recreational scuba diving fatalities. In a study of hospital admission due to interpersonal violence, Jesia Berry and colleagues show that Indigenous people, particularly women and people in remote setting, were more likely to be injured. They recommend culturally appropriate interventions to tackle social and economic issues associated with remote living. A different opportunity for health promotion is identified in the study of recent criminal activity in regular injecting drug users in Australia by Stuart Kinner and colleagues. Contact between injecting drug users and the criminal justice system is a marker of risk, presenting an opportunity for harm reduction interventions.

Once we know there is a problem that needs addressing, public health is concerned with activities in Primary Care. Shelley Wilkinson and colleagues conducted a study of health behaviour (such as smoking, diet and physical activity) in pregnant women in Queensland and showed a clear need for primary prevention programs in maternal and infant health services. Helen Calabretto shows that university students who have unprotected sexual intercourse are poorly informed about the use of emergency contraception and that it is underutilised. They recommend improved information from health professionals as well as information campaigns in the community. Tracy Comans and colleagues argue for a multidisciplinary falls prevention team for older adults and show that it can be cost-effective in a centre-based model of care. Increasing patient co-payments for prescription drugs may seem like a sensible way of reducing the cost of health care but Anna Hynd and colleagues show that increased cost reduces the use of prescription drugs like asthma medicine, proton pump inhibitors and statins, particularly among vulnerable groups. Evaluation of the impact of the 2005 changes to the Pharmaceutical Benefits scheme should be evaluated as a priority.

Research Issues are an important part of ensuring that public health practitioners and policy makers act on sound evidence. Kathy Chapman and colleagues analysed studies of food marketing to children and found consistent evidence that television food marketing focuses on unhealthy food. They found a lack of research on the effect of such marketing and recommend a more strategic approach to identifying research priorities. Armita Adily and colleagues are concerned that public health research in Australia may not be reaching its full potential because researchers are not sufficiently engaged with the ultimate users of the research. They call for a more systematic study of knowledge translation to ensure better research utilisation. Sanjay Sharma has reviewed the social science literature on living in remote mining towns, an important issue given the role of the mining industry in the Australian economy. Atypical working and living conditions seriously affect family health and psychological wellbeing, including through domestic violence. Clinicians working in these towns need to understand these problems but we also need better research to devise policies for dealing with these problems. Low response rates create analytical problems in surveys and Luke Wolfenden and colleagues have devised strategies for that tricky problem of obtaining parental consent for school-based research. Our interventions in Indigenous health need to be informed by an accurate assessment of any health problem. Pamela Bradshaw and colleagues show that the coding of Indigenous status in the Western Australia Hospital Morbidity Database can be used with confidence to compare health outcomes between Indigenous and non-Indigenous people.

Viruses are much in the news at present. In any Journal, studies of epidemics must lag well behind media reports of high risk but we welcome studies like that of Allison Salmon and colleagues which shows that Australia has successfully averted an epidemic of HIV among injecting drug users, although concerns persist about homosexual male injecting drug users. Monitoring infectious disease is central to concerns about climate change. Craig Williams and colleagues have devised models for defining which variations in conditions predict increased Ross River virus activity in the River Murray valley in South Australia. Melissa McLeod and colleagues show that the Wellington Emergency Department Respiratory Surveillance System could have provided an early warning of the 2005 influenza outbreak in Wellington schools in 2005, leading to an earlier response.