Differentiating the Scientific Endeavors of Research, Program Evaluation, and Quality Improvement Studies
Article first published online: 23 JAN 2012
© 2012 AWHONN, the Association of Women's Health, Obstetric and Neonatal Nurses
Journal of Obstetric, Gynecologic, & Neonatal Nursing
Volume 41, Issue 1, pages 1–3, January/February 2012
How to Cite
Lowe, N. K. and Cook, P. F. (2012), Differentiating the Scientific Endeavors of Research, Program Evaluation, and Quality Improvement Studies. Journal of Obstetric, Gynecologic, & Neonatal Nursing, 41: 1–3. doi: 10.1111/j.1552-6909.2011.01319.x
- Issue published online: 23 JAN 2012
- Article first published online: 23 JAN 2012
Nursing and other health care journals receive a wide range of manuscripts for publication consideration that are labeled as research, yet reviewers and editors often ask the question “is this research?” and if it is not, “should it be published?” During the past decade in nursing, this question has been asked more frequently because of three primary developments: (a) the evidence-based practice (EBP) movement focusing attention on the clinical application of research findings and the evaluation of care outcomes; (b) the development of the doctorate of nursing practice (DNP) as the highest level of clinical education with an emphasis on practice studies such as quality improvement (QI) and program evaluation (PE); and (c) growing attention to applied data in the research community through the National Institute of Health's emphasis on type 2 translational science (Chesla, 2008), and more recently, comparative effectiveness research (Wilensky, 2009).
Clinical questions arising from EBP traditionally have been answered through QI and PE initiatives (Bellin & Dubler, 2001), whereas the emerging fields of translational and comparative effectiveness science promote the application of research findings in the complex, uncontrolled environment of community health care and the evaluation of what treatments work best for which patients under what circumstances. Together, these emphases have led to more data collection and analysis, using more sophisticated methods, in applied clinical settings—a good thing by any measure. However, they have also blurred historical distinctions that existed between QI, PE, and research.
|Generalizability – per the federal definition in the Common Rule for human subjects research, research is “designed or intended to create generalizable knowledge.” But …|
|o QI and PE have limited-scope generalizability (e.g., methods, lessons learned)|
|o Research may not generalize because of overly limiting selection criteria|
|o Some research modalities (e.g., community-based participatory research, qualitative research) may not be intended to generalize|
|Methodology – scientific goals can be achieved using different methods:|
|o QI is generally pre/post; PE is often nonrandomized|
|o High-quality QI or PE uses matching and/or statistical controls to address common threats to validity within a nonrandomized design|
|o Research often (but not always) uses experimental controls|
|Assumption of Benefit – an application of best practices, or something unproven?|
|o QI interventions delivered to all patients are presumed effective, not experimental|
|o PE is presumed effective, although nonbeneficial programs may be discontinued|
|o In research, no benefits are assumed: “If we knew what we were doing we wouldn't call it research” (attributed to Albert Einstein)|
|o Random assignment is one way to judge investigators’ intent, because it requires an assumption of equipoise—that is, that a treatment is not expected to be beneficial|
|Opt-In or Opt-Out? – is the activity external or intrinsic to providing care?|
|o QI may be ethically required if it leads to better care|
|o PE is clearly better than making changes in care without evaluating|
|o Patients’ consent to treatment is sufficient for QI or PE when benefit is assumed|
|o Research requires consent because it is ethically optional, not promising benefits|
|Other Indicators of Investigator Intent – is generalizability the goal, or just a by-product?|
|o QI involves rapid-cycle protocol changes; research is standardized|
|o Research primarily serves investigator goals; QI or PE serves organizational goals|
|o Research has limited stakeholder involvement; QI or PE has extensive involvement|
|Role of Theory – is a program being tested, or the operational definition of a theory?|
|o QI – it's assumed the program will continue; the question is how to make it better. Organizational theory may be used to support the implementation of changes.|
|o Evaluation – evaluation assigns value, to decide if a program should continue. Theory may be used to design a program, but testing the theory is not the goal.|
|o Research – the program itself is not the point; it's just one possible operational definition of a theory. The goal is to prove or disprove the underlying theory.|
All of these efforts are systematic investigations that involve data, use scientific methods, and can be considered science from the broad perspective of science as an endeavor to describe, predict, or influence phenomena in our world. Despite the similarities of research, PE, and QI within this view of science, there are some critical differences reflected in the following definitions:
- Research, as defined by U.S. federal statue is “a systematic investigation, including research development, testing and evaluation designed to develop or contribute to generalizable knowledge” (Protection of Human Subjects, 1991).
- Evaluation has been described as “the systematic collection of descriptive and judgment information necessary to make effective … decisions related to the selection, adoption, value, and modification of … activities,” such as clinical care programs (Goldstein, 1980, p. 237).
- Quality improvement involves “systematic, data-guided activities designed to bring about immediate improvements in health care delivery in particular settings” (Lynn et al., 2007, p. 667).
Table 1 shows elements proposed to operationalize the differences between QI, PE, and research. One critical distinguishing element is the role of theory in these three types of systematic investigations (Cook, 2010). This can be termed the intent of the study, summarized in the statement of whatever the project was “designed to do.” The goal of research is to develop and/or test theory and theoretical propositions for the purpose of generalization, whereas the focus of a PE project is to evaluate a particular program that may or may not be based on a specific theory and that may or may not generalize to other situations. In QI, the focus is on improving the program rather than evaluating the underlying theory. Because QI projects are an extension of clinical care with the intent to improve care delivery, they must involve all levels of the clinical decision-making organization and may be based on an organizational theory that helps to structure the purposes, design, and measures of the project. But the goal of the QI project is to use rather than to test its theory of organizational change. In addition to the role of theory in the study, other questions that can be asked to help differentiate research, PE, and QI studies are in Table 1.
The question for scientific journals, like JOGNN, is “are all of these types of science appropriate for dissemination through publication?” Although scientific methodological rigor and the writing ability of the author(s) are fundamental to publication for all studies, the purpose of dissemination is to make the data and findings available to others. Scientific journals exist to provide generalizable information to the scientific and practice communities. Rather than thinking of generalizability as a dichotomy, it is probably best conceptualized along a continuum (Lynn et al., 2007). If a study produces insights applicable to the majority of similar patients in similar situations, it is most likely research. If, however, the scope of generalizability becomes progressively narrowed to fewer and fewer patients or more restrictive situations, such as in a particular institution, the study is more likely to be PE or QI. In this case, the investigators must focus less on the results of their individual project and instead ask themselves, “what are the lessons learned or pearls for practice” that warrant broad dissemination through publication (Newhouse, Pettit, Poe, & Rocco, 2006)?
JOGNN's Editorial Advisory Board recently grappled with this issue and decided that we will differentiate research, PE, and QI by publishing PE/QI studies under the “Principles and Practice” section of the journal and research studies in the “Research” section. This will help to differentiate these studies for reviewers and readers alike by providing a clearer categorization. JOGNN's Guidelines for Authors have been revised to reflect this change and can be accessed online at http://www.edmgr.com/jognn/
Systematic inquiry is a hallmark of professional practice and health care delivery. Nursing's social contract with the public mandates that we practice on the basis of best available evidence, implement effective programs, and continuously work to improve the care that we provide. This mandate requires studies of all types including research, PE, and QI. JOGNN's goal is to present you with the best science in each of these areas.
- 2001). The quality-improvement-research divide and the need for external oversight. American Journal of Public Health, 91, 1512–1517. , & (
- 2008). Translational research: Essential contributions from interpretive nursing science. Research in Nursing and Health, 31, 381–390. doi:10.1002/nur.20267. (
- 2010). Research, program evaluation, or QI? Mapping the edges in adherence studies. Communicating Nursing Research, 43, 1–14. (
- 1980). Training in work organizations. Annual Review of Psychology, 31, 229–272. doi:10.1146/annurev.ps31.020180.001305. (
- 2007). The ethics of using quality improvement methods in health care. Annals of Internal Medicine, 146, 666–673. , , , , , , … (
- 2006). The slippery slope: Differentiating between quality improvement and research. Journal of Nursing Administration, 36, 211–219. , , , & (
- Protection of Human Subjects, 45 C.F.R. 46.102(d) (1991).
- 2009). The policies and politics of creating a comparative clinical effectiveness research center. Health Affairs, 28, 719–729. doi:10.1377/hlthaff.28.4.w719. (