Qualitative research in nutrition and dietetics: assessing quality

Authors


Dr J. A. Swift, Division of Nutritional Sciences, School of Biosciences, The University of Nottingham, Sutton Bonington Campus, Loughborough, Leicestershire, LE12 5RD, UK.
Tel.: +44 (0)115 951 6178
Fax: +44 (0)115 951 6122
E-mail: judy.swift@nottingham.ac.uk

Abstract

In recent years, qualitative research has become much more widely used in healthcare settings and undoubtedly has much to offer nutrition and dietetics. Its value is, however, still sometimes called into question and, for those unfamiliar with qualitative approaches, it can be difficult to grasp what distinguishes ‘good’ qualitative research from that which has been less rigorously conceived and conducted. This review, the fourth in the series, aims to highlight some of the key scientific debates around the quality criteria that can be applied to qualitative research, and offers some flexible guidelines that may be used both in producing and assessing qualitative health research, including studies in nutrition and dietetics. Other reviews in this series provide a model for embarking on a qualitative research project in nutrition and dietetics, an overview of the principal techniques of data collection, sampling and analysis and some practical advice relevant to nutrition and dietetics, along with glossaries of key terms.

Introduction

As qualitative methods become more widely used and accepted in health research, so funding bodies, readers and users of research need to be able to assess the quality of this work. Traditionally, quantitative research has dominated health care (Broom & Willis, 2007), including nutrition and dietetics (Fade, 2003), and there are many common preconceptions about qualitative work; that it is ‘soft’, ‘subjective’, ‘anecdotal’‘unscientific’ or ‘speculative’ (Abusabha & Woelfel, 2003). Partly, this may be seen as a problem of audience: much early qualitative work was written, for example, by sociologists for other sociologists, which assumes a certain shared epistemological position and assumptions that are taken for granted, and may result in a lack of explicit statement about study design, methods, etc. However, qualitative researchers themselves must also be held responsible; in any field of study, there exists published work which is of less than exemplary quality. There is now a growing awareness that qualitative research intended to be read and used by a healthcare audience may require a different presentation, and also an acceptance that it is the responsibility of researchers to demonstrate, rather than assert, the quality of their research (Murphy & Dingwall, 2003). This review, the fourth in the series, aims to highlight some of the key scientific debates around qualitative research, and offer some flexible guidelines that may be used in producing and assessing qualitative health research, including studies in nutrition and dietetics. Other reviews in this series provide a model for embarking on a qualitative research project in nutrition and dietetics (Swift & Tischler, 2010) and an overview of the principal techniques of data collection, sampling (Draper & Swift, 2010) and analysis (Fade & Swift, 2010). In addition, all reviews in the series provide some practical advice for those wishing to engage with qualitative research in nutrition and dietetics and a glossary of key terms.

Criteria of quality

One key debate in qualitative research is whether it should be judged by the same criteria as quantitative work (Malterud, 2001). Some researchers would argue that conventional measures of validity, reliability and generalisability are inappropriate for qualitative research, which assumes a relativist ontology (see Glossary) (Swift & Tischler, 2010). If one takes the view that what qualitative research produces is one of many possible perspectives on the world that has been constructed in and through the research process, then these traditional measures clearly become redundant. Although some researchers feel that this is sufficient cause to reject formal methods of assessing the quality of qualitative research (Buchanan, 1992), most recognise the value in such critique (e.g. Seale & Silverman, 1997; Mays & Pope, 2000).

Indeed, qualitative research can be assessed using criteria of quality that do not look very different from those used for assessing quantitative research albeit operationalised differently (Mays & Pope, 2000). Before examining these, however, it is worth noting that establishing properties such as reliability and validity is not a problem unique to qualitative research (Broom & Willis, 2007). In a randomised controlled trial, for example, potentially confounding variables must be correctly identified and controlled for to preserve validity, which is often much more complex and imprecise than it sounds. As Oakely (2000) helpfully points out,

‘… the distinguishing mark of all good research is the awareness and acknowledgment of error’ (Oakley, 2000: 72)

Validity

Validity refers to the issue of whether the researcher’s account truly reflects what actually happened (i.e. is it accurate?). In the natural sciences, findings only become validated when they are replicated. However, as Bloor (1997) points out, in social research, validation cannot occur through replication because identical social circumstances can never be recreated. Instead, there are elements of social life that can be generalised across settings (e.g. Perakyla, 1995; Silverman, 1997), whereas others remain particular to that setting (e.g. the physical organisation of the clinic and where the parties sit).

In the absence of replication, social researchers have developed two main techniques for validation: triangulation and respondent validation. Triangulation is an attempt to replicate research within the same settings, rather than across different ones. Different methods of data collection (e.g. interviews, vignettes and observation) are used and findings are judged valid when all of these yield identical findings (e.g. Cant, 2009). Respondent (or member) validation is a technique in which the researcher’s account is compared with the accounts of those individuals who participated in the research to see if they correspond. It often involves taking results back to those who have participated in the research and asking them to comment on the adequacy of the descriptions that have been produced.

It is important to recognise, however, that the use of these two techniques is not necessarily straightforward. Where triangulation produces different results through different methods, there is the problem of what weight should be attached to the findings of each of these methods. Respondent validation can be particularly difficult for the researcher. For example, there may be conflicts over issues that are of major importance to one individual in the sample but not to others, or there may be political reasons rather than reasons of accuracy for a participant’s objections to the researcher’s account. In other words, as Bloor (1997) highlights, there is a need to bear in mind that the researcher’s purposes are not the same as the participants and that, ultimately, this may require difficult decisions to be made in terms of whether members should have privileged status as commentators on their own actions.

Clearly, then, the problems of the particularities of social life apply just as much to the methods for validation as they do to the generation of the original data. However, this is not to say that these approaches are without their uses, although it is probably best to consider them as techniques to generate another source of data for comparison rather than producing a definitive answer about accuracy. Comparing these sources of data means that some of the problems associated with the partiality of data drawn from a single source can be overcome, and increase the comprehensiveness of a study (Murphy & Dingwall, 2003). Inconsistencies can encourage researchers to revisit their conclusions and explore their own assumptions. The end result should therefore be an analysis which represents reality for the participants, rather than attaining some absolute truth (Mays & Pope, 2000).

Reliability

Reliability

‘refers to the degree of consistency with which instances are assigned to the same category by different observers or by the same observer on different occasions’ (Hammersley, 1992: 67)

In other words, the idea that the results of the analysis would be the same if carried out by different observers or by the same observers on different occasions. Internal reliability, the issue of consistency of application of concepts to the data, can be addressed by the use of inter-rater reliability checks. Different researchers’ applications of the concepts can be compared and adjustments made if and where necessary (e.g. Thomas et al., 2008). This can be a time consuming, labour intensive and expensive process, although, as Seale (1999) notes, it can be facilitated by the use of Computer Assisted Qualitative Data Analysis software (CAQDAS). The issue of external reliability (see Glossary) is more difficult for qualitative research: once you accept the view that identical social circumstances can never be recreated, it becomes irrelevant. What is possible, however, is for researchers to provide clear and explicit definitions of the concepts they have used and how they have used them. This process makes the analysis explicit and allows, as far as is possible, for the reproducibility of the research. It also enables the reader to make their own evaluations of the claims that are made.

Generalisability

The aim of most healthcare research is to inform or influence policy or practice in some way, and this means that researchers generally need to demonstrate that their findings are likely to be applicable not just in this clinic or to this group of patients (Swift & Tischler, 2010). By their nature, qualitative studies often involve small sample sizes and/or single settings, and this can create concerns about their generalisability. However, although not empirically generalisable, qualitative research findings can be theoretically generalisable (Swift & Tischler, 2010). One of the key ways to enhance generalisability is through proper consideration of sampling at the design stage of the study; although it may not be possible, or desirable, to obtain a representative sample of whatever population is under investigation, there should be good theoretical reasons for the sample that is selected (Draper & Swift, 2010).

Guidelines and checklists for assessing qualitative research

Given the vast range of methods that can be incorporated into qualitative research and their different philosophical underpinnings, it is difficult to produce a quality check list that can be universally applied. Previously published checklists (e.g. Secker et al., 1995; Blaxter, 1996; Malterud, 2001) have stimulated much discussion and debate. What is offered in this review is a set of general criteria to guide authors as well as readers of qualitative health research, although these are not intended to be treated as inflexible or definitive. They are produced in support of the view that the quality of any study, whether quantitative or qualitative, has to be demonstrated rather than asserted (Murphy & Dingwall, 2003).

Clarity of methods of data collection and analysis

Murphy & Dingwall (2003) suggest that a key concern in assessing the quality of qualitative research is whether the methods that have been used are appropriate to answer the question at hand. They suggest asking the question

‘Are the methods used in this study fit for the purpose to which they have been put?’ (Murphy & Dingwall, 2003: 181)

To make a judgement on this, an article needs to contain sufficient information on how the data were collected and analysed and why they were collected and analysed in the chosen way. Although this should include the study’s connection with and/or location in existing bodies of theory or knowledge, qualitative research reporting is frequently insufficient (Carter & Little, 2007). Unfortunately, Fade (2003) has demonstrated a lack of detail regarding qualitative research articles published in the Journal of Human Nutrition and Dietetics. An illustration of the information required to clarify the research methods is presented in Swift & Tischler’s (2010), Draper & Swift’s (2010) and Fade & Swift’s (2010) worked example of how to developing a qualitative research strategy from the hypothetical research question ‘What is the experience of obese adolescents who access adult weight management services?’

It is worth noting that this guideline can be a particular issue for qualitative work published in medical or health-oriented journals, which typically have a word limit of approximately 3000 words, compared to a typical limit of 8000 in social science journals. There is, therefore, a difficult balance to be struck between giving the required space and emphasis to the findings of the study, and providing sufficient information on method. One way around this can be to reference methodological information which has been published elsewhere.

Reflexivity

As discussed in Swift & Tischler (2010) and Draper & Swift (2010), reflexivity is recommended as a key strategy to enhance the credibility of qualitative research. There are two components to reflexivity. The first is recognising the impact of the researcher on the research setting, as well as the professional, intellectual and personal baggage the researcher brings to their analysis. This may include, for example, the age, sex and social class of researcher (Manderson et al., 2006), as well as what is sometimes described as the ‘distance’ between the researcher and the group under study (Mays & Pope, 2000). This distance may be relatively small if the study group are other healthcare professionals, in the sense that cultural values and norms may be shared by both the researcher and their participants, although it is potentially much larger if they are, for example, young women with an eating disorder and the researcher is a middle-aged male doctor.

The second component involves recognising research participants as actively involved and purposefully engaged in producing the activities studied as part of research. Why people behave as they do in the context of a research project is in itself a key analytic question (Murphy & Dingwall, 2003). Unfortunately, there is no simple test to assess whether research has been reflexively carried out. Instead, a judgement has to be made, on the basis of the information provided, as to whether and how researchers have considered their findings within the context of their production. Have they, for example, been explicit at the outset about any possible source of bias, be it professional, personal or intellectual? Bishop (2007) gives an interesting and revealing reflexive account of her analyses of two historical qualitative data sets to explore the beliefs and practices from which current uses of convenience food may have emerged. These kinds of considerations demonstrate that reflexivity has been taken seriously and add to the credibility of the research.

Dealing with negative cases

Much qualitative research illustrates findings by including direct quotes from participants, and this can lead to charges of anectdotalism. In other words, the presentation of disembodied and isolated snippets of talk or text to illustrate a point potentially undermines the validity of much qualitative research (Silverman, 2000). As Silverman (2000) states, researchers need to convince their audience that research findings are genuinely based on critical investigation of the whole corpus of data, rather than a selective assembling of the examples that illustrate one aspect of it. One of the dangers of any kind of research is ‘finding what you look for’, in the sense that the researcher is drawn to instances or examples that align with their preconceived ideas, political views, previous research, etc. To guard against this, the representativeness of the data presented needs to be addressed, and particular attention needs to be paid to those negative or ‘deviant’ cases that do not appear to fit the prevailing pattern or the overall argument. Considering why they do not fit is vital to a rigorous approach: it may be that there is something different about the case itself (e.g. participants, setting, etc.) that could explain the deviation, so that the existing argument still holds. On the other hand, it may be an indication that the existing argument provides an unsatisfactory or only partial explanation of the data.

Fair dealing

This principle is designed to make sure that all those studied are dealt with even-handedly, and that one group of participants’ responses is not privileged over another’s. For example, in a study designed to investigate the information given to pregnant women to allow them to make a decision about antenatal screening, it would be easy to take women’s reports that they were not given sufficient information about choice at face value. In the same study, midwives who were interviewed were adamant that they dealt explicitly with issues of choice. Careful examination of the consultation data, however, reveals a much more complex situation that not only supports both sets of participants’ views, but also highlights that when and how the information is delivered by midwives becomes as significant as what is said (Pilnick, 2008). Researchers must therefore be resistant to the temptation to select the most extreme or memorable quotes, or favour particularly articulate participants (Hancock et al., 2009).

Worth or value

One way to judge worth or value is the addition to existing knowledge that a particular study makes; for example, new research should build upon the existing foundations of knowledge in a particular topic. Research may produce new findings but may also increase confidence in existing findings by confirming them, or expanding the range of settings to which they can be applied. Research that ignores existing work in the field is problematic because there is no way of knowing whether the current study is cumulative, novel, or typical. In a review of qualitative research relating to patients’ experience of diabetes, seven studies were identified, with none of them referring to another (Campbell et al., 2003). Morse (2005) rightly suggests that qualitative articles should earn their space in research journals by being concise and by only providing new and relevant information.

Getting qualitative research published

As already noted, condensing qualitative research into a 3000 word article can present challenges. Individual journals often have very specific requirements and the starting point is to identify the journal that you are writing for and to adhere to their stylistic guidelines. However, there are some general principles that apply and the reader is directed to texts such as Holliday (2002) for further information.

  • • Make sure the abstract contains the essential message of the article.
  • • State the research question clearly.
  • • Give an account of the approach underpinning the study and why this is appropriate to tackle the research question(s) [for some useful examples, see Holliday (2002: 49–52)].
  • • Include both methodology (a brief theoretical discussion of the conceptual framework) and methods (aspects of design, sample size, setting, etc.).
  • • Give a detailed description of how the analysis was performed to ensure transparency.
  • • Organise the results section in terms of thematic headings, and illustrate these with carefully selected examples. Remember to consider how each of these themes are linked, and how they contribute to the overall argument. Do not overstate claims based on small sample sizes or limited data.
  • • In the conclusion, be clear about the overall message of the article. Consider the implications that the findings have for clinical practice or public health.
  • • In line with the discussion above, re-read the article and make sure you have been as transparent as possible about the process of the research. Ultimately, this is what will allow reviewers to judge it.


Glossary

AnectdotalismThe presentation of disembodied and isolated snippets of talk or text to illustrate a point, potentially undermines the validity of qualitative research (Silverman, 2000)
External reliabilityThe extent to which re-analysis would produce similar findings
GeneralisabilityThe extent to which research findings have meaning outside the research context (i.e. to what other groups, settings or activities can the results obtained be generalised?) Empirical generalisability refers to the extent to which research findings can be used to infer about the characteristics of a wider population (Mason, 2002). Theoretical generalisability refers to the extent to which research findings can be used to develop concepts, understand phenomena and theoretical propositions that are relevant to other settings and other groups of individuals (Draper, 2004)
Internal reliabilityThe extent to which concepts have been consistently applied to the data
Negative (deviant) casesInstances when the data does not appear to fit the prevailing pattern or the overall argument
ReflexivityConsideration of the researcher’s own role in the research process via critical self-scrutiny (Mason, 2002)
Relativist ontologyBelief in a socially constructed reality. Relativism accepts that how people perceive the world and their thoughts about it are always influenced by social factors such as culture, history and language (Willig, 2008)
ReliabilityThe extent to which the results of the analysis would be the same if carried out by different observers or by the same observers on different occasions
Respondent (or member) validationA technique in which the researcher’s account is compared with the accounts of those who participated in the research to see if they correspond. It often involves taking results back to those who have participated in the research and asking them to comment on the adequacy of the descriptions that have been produced
TriangulationAn attempt to replicate research within the same settings, rather than across different ones. Different methods of data collection (e.g. interviews, vignettes and observation) are used and findings are judged valid when all of these yield identical findings
ValidityRefers to the issue of whether the researcher’s account truly reflects what actually happened (i.e. is it accurate?)

Conclusions

Qualitative methods have already contributed much to the understanding of health and the organisation and delivery of health care. Although it has yet to be fully embraced, it undoubtedly also has much to offer nutrition and dietetics (Swift & Tischler, 2010). To ensure that future qualitative research is credible and ultimately useful, attention must be paid to issues of validity, reliability and generalisability. Although it can be difficult to produce quality check lists that can be universally applied across all qualitative research, this review offers a flexible set of general criteria that are relevant for studies in nutrition and dietetics. This covers issues of clarity, reflexivity, negative cases, fair dealing and worth. Ultimately, it is the responsibility of the researcher to demonstrate rather than asserted the quality of their study.

Conflict of interests, source of funding and authorship

Both authors were involved in the preparation of this manuscript. The authors report no conflict of interest.

Ancillary