How to … assess the quality of qualitative research

As a clinician interested in qualitative research, you are likely to have pondered whether a particular study is trustworthy. How do you know whether qualitative research findings are valid and can be applied in your setting? The quality standards you would look for in quantitative research do not apply. Furthermore, qualitative research is conducted within a number of paradigms, or ways of understanding the nature of reality and knowledge, each associated with different ways of defining, understanding and reporting quality. This ‘How to …’ article aims to support health care practitioners, educators and researchers to recognise some of the essential characteristics or ‘markers’ of high‐quality qualitative research. We hope that this article will support those reflecting on the quality of their own research, those justifying research design to funders and those reviewing qualitative research for journals or for inclusion in reviews.

How to … SUMMARY As a clinician interested in qualitative research, you are likely to have pondered whether a particular study is trustworthy. How do you know whether qualitative research findings are valid and can be applied in your setting? The quality standards you would look for in quantitative research do not apply.
Furthermore, qualitative research is conducted within a number of paradigms, or ways of understanding the nature of reality and knowledge, each associated with different ways of defining, understanding and reporting quality. This 'How to …' article aims to support health care practitioners, educators and researchers to recognise some of the essential characteristics or 'markers' of high-quality qualitative research. We hope that this article will support those reflecting on the quality of their own research, those justifying research design to funders and those reviewing qualitative research for journals or for inclusion in reviews.

How to … assess the quality of qualitative research
A high-quality qualitative study, just like any other study, should have a clear and justified research question that implies that the study is timely, original, rigorous and relevant INTRODUCTION B efore you start conducting your own health professions education research you are likely to read many published studies. The abstract at the very beginning of a qualitative research article can be seen as the first marker of high quality. An abstract should clearly explicate the research problem and the aim of the study, and should skillfully condense the contents of the study, as well as indicate its novelty and contribution to research and practice in health professions education. This is typically elaborated upon further in the Introduction to the article, which should motivate and guide the reader to explore the article further. Typically, the authors then move into presenting a comprehensive literature review, which should map the existing and up-to-date research knowledge and explicate the research problem that the study at hand will address, as well as its significance to the audience. 1 A high-quality qualitative study, just like any other study, should have a clear and justified research question that implies that the study is timely, original, rigorous and relevant. 2 A further important marker for assessing the quality of a qualitative study is that the theoretical or conceptual framework is aligned with the research design, the research question(s) and the methodology used in the study, as well as with the reporting of the research findings. High-quality qualitative research necessitates critical reflection and a justification of the selected framework underpinning the study.
To elaborate on the many markers for assessing the quality of qualitative research, we now move on to discussing these in relation to the research process.

HOW TO ASSESS THE QUALITY OF THE RESEARCH METHODOLOGY?
Four criteria are widely used to appraise the trustworthiness of qualitative research: credibility, dependability, confirmability and transferability. 3 In Table 1 we define these criteria along with an additional marker of quality, reflexivity. We also provide a summary of how you can recognise these criteria in a research article.
Credibility of the research demands that the methodology chosen should be well explicated and justified. For example, authors might say that they chose to use grounded theory or a phenomenological approach and accompany that with a clear explanation of why they made that choice. The data collection methods and the volume of data should be justified and appropriate to the chosen methodology. Depending on the research question, observations (as well as other methods) might be an alternative or complement to interviews, or individual interviews may be more appropriate than group interviews or focus groups. 4 One key marker for assessing the quality of qualitative research is the selection criteria used to recruit study participants. As described in our earlier article in this 'How to …' series, there are a number of ways to select your sample and the sample size is also linked to the methodology. 4 The number of respondents or the length of observations are not necessarily the key markers to indicate high-quality research. Instead, when assessing quality, the focus needs to be directed towards the depth, richness and appropriateness of the data, and whether, when analysed, the data provide enough evidence to answer the research question(s). Morse proposes the following factors to be taken into consideration: the scope of the study; the nature of the topic; what is already known about the studied phenomenon; the study design; the quality of the data; the amount of useful information obtained from each participant; and the use of shadowed data (i.e. the respondent's awareness of others' understanding of the phenomenon being explored). 5 Moreover, it may not be possible to analyse a huge volume of data in the appropriate depth required for certain methodologies, and a less rich data sample (for example very short interviews) is likely to be of less value to the research than fewer in-depth interviews that provide rich insight. Taken together, this makes it very difficult to know beforehand how many respondents will be needed, and in qualitative studies there is flexibility regarding this point.
In some studies, the credibility of the analysis can be enhanced through member reflections, which means that the preliminary findings are presented to the participants for input and elaboration. Similarly, saturation or data sufficiency are commonly used terms to indicate credibility, suggesting that enough data were gathered to identify all relevant aspects to answer the research question. This concept is widely debated, however, and Varpio et al. provides a great description of why information power might be a better concept. 6 They suggest that researchers should explain how their data are adequate in terms of allowing for transferability, being able to answer the research question and being aligned with their methodological orientation. Varpio et al. further explain how the concepts of triangulation or crystallisation are used as a quality marker in somewhat different ways, depending on the paradigm. 6 Depending on the paradigm or scientific tradition you are working in, the analytic phase can be High-quality qualitative research necessitates critical reflection and a justification of the selected framework underpinning the study … focus needs to be directed towards the depth, richness and appropriateness of the data, and whether, when analysed, the data provide enough evidence to answer the research question(s) described in different ways. For some researchers, perhaps leaning towards a more post-positivist approach, several researchers may conduct the analysis independently of each other and then compare their findings, calculating an inter-rater relationship as a quality marker. Others, with perhaps a more constructivist approach, acknowledge that the findings are a co-construction between the researchers and the data, and suggest that it is unlikely that any other researcher would create an exact replica as previous experiences will influence the conclusions inferred. In any case, applying qualitative analytic methods requires an open mind, and the usage of an explorative approach, but this does not mean that that the researcher does not have knowledge that already influences the analysis. Most importantly, the process of data collection and data analysis should be described in enough detail for someone to follow the same steps. This supports the dependability of the research. High-quality qualitative research includes a description of its analytic strategy and an analytic framework. In this, a rationale can be given for coding decisions, leading to the presentation of a coding scheme with clearly defined steps. The ability of the researcher to perform analysis that generates a theory or develops a novel conceptual framing also evidences sound qualitative research.
In qualitative research, the person collecting the data plays a central role (compare the roles of a researcher conducting a personal interview versus a researcher collecting data through an online questionnaire). An important marker of quality is that the researcher reflects his or her role in the study (e.g. their relationship to the respondents). This process, reflexivity, is a key marker of quality. 7 A teacher interviewing his or her students is likely to get different answers than a student asking peers the same questions. Whether the interviewer is an expert on the topic is also likely to affect the interview, as well as other factors such as the interviewees' and the respondents' respective ages, gender and professions. Disclosing such information in the methodology section is important and increases the transparency and trustworthiness of the research findings.
As for all research, ethical considerations are central in qualitative research. These should be discussed, as well as a clear statement made on formal research ethics committee approval or waiver. Gaining appropriate consent and protecting participant identity are important. Consideration must be given to the possibility of coercion to participate, particularly when the researcher is in a position of power relative to the participants: e.g. teacher and students.

HOW TO ASSESS THE QUALITY OF THE RESEARCH FINDINGS?
In the write-up of the findings, confirmability is enhanced by the inclusion of quotes or similar research data. It may also be tempting to quantify qualitative findings by expressions such as '80% of all respondents agreed that' or 'only one of the interviewees mentioned that'; however, Monrouxe and Rees provide a compelling argument for why this may not be strengthening the research. 8 Strong markers of high-quality qualitative research can be seen in the discussion and conclusion part of a journal article, which states how the findings add to the existing knowledge through new theoretical or practical interpretations, and how the findings link to previous relevant research. In the discussion, the authors should

QUALITY CHECKLISTS AS TOOLS FOR ASSESSING QUALITY
To facilitate the process of assessing the quality of a qualitative study there are several checklists available. Many researchers oppose the use of these, however, suggesting that they may even be counterproductive if used uncritically and without careful consideration of the research context. We suggest that such guidelines may help to better identify markers indicating high-quality qualitative research in journal articles as well as in conducting one's own research or when writing grant proposals and reviews. We would caution, however, that a high score on such a list may say little about whether the findings can be applied in your setting, bearing in mind that the authors' theoretical and philosophical frameworks and the standards of journals vary immensely. The perhaps most commonly used framework today is Consolidated Criteria for Reporting Qualitative Research (COREQ), and some journals now require authors to use this when submitting an article. 9 Another checklist, developed especially for medical education, is Standards for Reporting Qualitative Research (SRQR), whereas Tracy and Critical Appraisal Skills Programme (CASP) present criteria for qualitative research applicable across genres and methodologies. [10][11][12] For novice researchers, a combination of checklists might be appropriate.

CONCLUSIONS
In this article, we have introduced some of the essential 'markers' that indicate high-quality qualitative research. In sum, a detailed description of the research process, including the research context, research aims, questions and design, theoretical underpinnings, and the methods of data collection and data analysis, results, discussion, and their careful alignment, usually increases the quality of a qualitative study. It is important to note that these markers differ between qualitative research and quantitatively oriented natural sciences, albeit that variation also exists between different qualitative methodologies.