Conflict of interest: There is no conflict of interest to declare.
Validation of the Quality of Diagnoses, Interventions, and Outcomes (Q-DIO) Instrument for Use in Brazil and the United States
Article first published online: 26 MAR 2014
© 2014 NANDA International, Inc.
International Journal of Nursing Knowledge
Volume 26, Issue 1, pages 19–25, January 2015
How to Cite
Linch, G. F. d. C., Rabelo-Silva, E. R., Keenan, G. M., Moraes, M. A., Stifter, J. and Müller-Staub, M. (2015), Validation of the Quality of Diagnoses, Interventions, and Outcomes (Q-DIO) Instrument for Use in Brazil and the United States. Int Jnl Nurs Knowledge, 26: 19–25. doi: 10.1111/2047-3095.12030
- Issue published online: 13 JAN 2015
- Article first published online: 26 MAR 2014
- Fundo de Incentivo à Pesquisa e Eventos (FIPE) do Hospital de Clínicas de Porto Alegre (HCPA)
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES)
- Electronic health record;
- nursing diagnosis;
- nursing record;
- terminology as topic;
- validation study
To assess the reliability and validity of Quality of Diagnoses, Interventions, and Outcomes (Q-DIO) in Brazil and in the United States.
The sample comprised 180 records: centers 1 (electronic records and standardized language) and 2 (paper-based records without standardized language in Brazil, and center 3 (electronic records without standardized language) in the United States. Reliability and discriminant construct validity was analyzed.
Cronbach's alpha for all 29 Q-DIO items were greater than or equal to 0.70 for all centers; construct validity was significantly different between the three study centers.
These results indicate that Q-DIO is valid and reliable for assessing the quality of nursing records.
Implications for Nursing Practice
The Q-DIO may be useful to assess the quality and accuracy of nursing records.
In clinical practice, nurses are required to systematize patient care as to ensure patient safety and quality care (Paans, Sermeus, Nieweg, & Van der Schans, 2010; Saranto & Kinnunen, 2009). A key aspect of this process is that nursing records be kept in a comprehensive and proper fashion and, especially, that this documentation be understood and valued. Within this context, the use of standardized terminology and electronic systems for nursing documentation have gained ground and helped enhance the quality of nursing records (Linch, Müller-Staub, & Rabelo, 2010).
In Brazil, although nursing documentation is a routine practice, nurses rarely use standardized language for this purpose (Ochoa-Vigo, Pace, & Santos, 2003). In the United States, 64% of nursing records are still paper based. Data from a U.S.-based integrative review of the literature designed to examine the relationship between electronic nursing documentation and quality of patient care suggest that these records lack the objectivity and reliability required to ensure their excellence (Kelley, Brandon, & Docherty, 2011). The authors of this review also suggest that the extent to which electronic nursing documentation improves the quality of care—an area in which research is lacking—needs to be explored.
The Quality of Diagnoses, Interventions and Outcomes (Q-DIO) instrument, a reliable and valid tool that can be used to assess the quality of nursing records regardless of documentation format, bridges this knowledge gap (Müller-Staub, Lunney, et al., 2008; Müller-Staub, Needham, et al., 2008; Müller-Staub et al., 2009). Within this context, in view of the heterogeneity of nursing documentation systems in Brazil, the United States, and, consequently, in similar countries, excellence in patient care requires assessment of the consistency of nursing documentation. Although the original Q-DIO instrument was developed in English, it has only been validated for use in Swiss hospitals.
To validate the Q-DIO for use in three different settings in Brazil and in the United States.
This methodological instrument validation study was carried out at three centers (two in Brazil and one in the United States). The data collection period was from November to December 2011 in Brazil and from May to June 2012 in the United States.
Participants and Sampling Procedure
Three separate groups were established: (a) electronic records and standardized terminology for nursing diagnoses (NDs; center 1); (b) paper-based records and no standardized terminology (center 2); (c) electronic records and no standardized terminology (center 3). The sample comprised all nursing records of patients post cardiac surgery who had spent at least 48 hr in one of the study units and whose records contained history, progress notes, and nursing prescriptions for a period of at least 4 days, as recommended by the developers of the original Q-DIO instrument (Müller-Staub, Lunney, et al., 2008; Müller-Staub, Needham, et al., 2008; Müller-Staub et al., 2009).
The minimum sample size required for this study was calculated as 145 observations, as the literature recommends 5–10 observations per instrument item for calculation of Cronbach's alpha (Hair, Tatham, Rolph, & Black, 2009). The study sample ultimately comprised 60 patient records from each of the participating centers for a total of 180 records.
The sampling technique consisted of compilation of a list of all eligible patients who had undergone cardiac surgery within the year preceding data collection at each of the participating centers. This list was then randomly sampled (taking into account 20% sampling loss, e.g., incomplete records) with the aid of the SPSS 18.0 software (SPSS, Inc., Chicago, IL, USA). Furthermore, 20% of records (24 patients) from each center were randomized for assessment of intraobserver reliability (test and retest) and interobserver agreement.
Psychometric Test of the Brazilian Version of the Q-DIO
- Reliability and stability: The reliability of the Q-DIO was assessed by Cronbach's alpha for the 29 items by the test and retest (intraobserver three weeks later).
- Interobserver agreement: The interobserver agreement was assessed by two raters trained in the same records independently.
- Construct validity: The divergent construct validity was assessed in the comparison between three groups of nursing records: electronic records and standardized terminology for NDs; paper-based records and no standardized terminology; and electronic records and no standardized terminology.
Validated Instrument: Q-DIO
The main purpose of the Q-DIO is to assess the accuracy of NDs, effectiveness of interventions, and the quality of patient outcomes in documentation, with a focus on issues recorded in the nursing history and on progress notes as a record of the nursing process, regardless of the use of standardized language. It is applicable to electronic and paper-based records alike (Müller-Staub, Lunney, et al., 2008; Müller-Staub, Needham, et al., 2008; Müller-Staub et al., 2009). The Q-DIO consists of 29 items divided across four domains: (a) NDs as process, (b) NDs as product, (c) nursing interventions, and (d) nursing-sensitive patient outcomes.
Both in the original instrument and in the Brazilian version, items are graded on a three-point Likert-type scale, with 0 being “not documented,” 1 being “partially documented,” and 2 being “comprehensive documentation.” Assuming that each of the 29 items receives a maximum score of two, the highest possible score for the total of the Q-DIO instrument is 58 points.
Assessing correctness and accuracy was defined in the Q-DIO based on the concepts and definitions of standardized terminology as presented in the nursing diagnosis classification NANDA-I, in the Nursing Interventions Classification (NIC), and Nursing Outcomes Classification (NOC).
Analysis of the Q-DIO for comparisons among different strata was based on assessment of means scores; the higher the score, the higher the quality of the nursing documentation assessed. Analysis can also be conducted separately for each Q-DIO domain or for the sum of all domains for the instrument as a whole.
Translation and Adaptation of the Instrument for Use in Brazil and the United States
In view of differences in culture and language, validation of the Q-DIO for use in Brazil first required translation and cross-cultural adaptation of the instrument, which have been published elsewhere (Beaton, Bombardier, Guillemin, & Ferraz, 2000; Linch, Müller-Staub, Moraes, Azzolin, & Rabelo, 2012). No translation or adaptation was required for use of the instrument in the United States, as the Q-DIO was originally developed in English (Müller-Staub, Lunney, et al., 2008).
The study was approved by the institutional review boards of the participating centers. All authors signed a data use agreement affirming their commitment to safeguarding the confidentiality of patient data.
All analyses were carried out in SPSS 18.0. The reliability of the Q-DIO and its four domains was assessed by means of Cronbach's alpha coefficient. Stability and agreement and interobserver were analyzed by means of intraclass correlation coefficients (ICCs). Discriminant construct validity among the three documentation groups was assessed by analysis of variance with Tukey's post-hoc test (Hair et al., 2009).
The sample comprised 180 nursing records of patients' status post cardiac surgery (60 records from each of the study centers).
Cronbach's alpha values for all 29 items were ≥0.70 at all centers (0.70, 0.89, and 0.82 at center 1, center 2, and center 3, respectively). Table 1 shows item–total correlation values and Cronbach's alpha if item excluded for each center.
|Center 1||Center 2||Center 3|
|Cronbach's alpha = 0.70||Cronbach's alpha = 0.89||Cronbach's alpha = 0.82|
|Item-total correlationa||Cronbach's alpha if item excluded||Item-total correlationa||Cronbach's alpha if item excluded||Item-total correlationa||Cronbach's alpha if item excluded|
|1. Actual situation, leading to the hospitalization||0.354||0.679||0.062||0.889||0.309||0.812|
|2. Anxiety and worries related to hospitalization, expectations, and desires about hospitalization||0.555||0.658||0.000||0.886||−0.009||0.820|
|3. Social situation and living environment/circumstances||0.399||0.671||0.000||0.886||0.064||0.818|
|4. Coping in the actual situation/with the illness||0.454||0.668||0.000||0.886||0.045||0.821|
|5. Beliefs and attitudes about life (related to the hospitalization)||0.324||0.681||0.000||0.886||0.144||0.824|
|6. Information of the patient and relatives/significant others about the situation||0.372||0.677||0.000||0.886||0.064||0.825|
|7. Intimacy, being female/male||0.351||0.676||0.000||0.886||0.000||0.818|
|8. Hobbies, activities for leisure||0.434||0.669||0.000||0.886||0.000||0.818|
|9. Significant others (contact persons)||0.371||0.674||0.000||0.886||0.420||0.814|
|10. Activities of daily living||0.177||0.692||0.095||0.886||0.121||0.818|
|11. Relevant nursing priorities according to the assessment||0.070||0.705||0.095||0.886||0.394||0.809|
|12. Nursing diagnosis label is formulated||0.000||0.696||0.670||0.877||0.000||0.818|
|13. Nursing diagnosis label is formulated according to NANDA-I and is numbered||0.000||0.696||0.670||0.877||0.000||0.818|
|14. The etiology (E) is documented||0.084||0.696||0.463||0.881||0.364||0.810|
|15. The etiology (E) is correct, related/corresponding to the nursing diagnosis (P)||0.064||0.699||0.496||0.881||0.529||0.801|
|16. Signs and symptoms are formulated||0.312||0.681||0.711||0.875||0.240||0.815|
|17. Signs and symptoms (S) are correctly related to the nursing diagnosis (P)||0.353||0.678||0.749||0.872||0.619||0.797|
|18. The nursing goal relates/corresponds to the nursing diagnosis||0.361||0.677||0.650||0.876||0.655||0.796|
|19. The nursing goal is achievable through nursing interventions||0.192||0.691||0.803||0.870||0.610||0.797|
|20. Concrete, clearly named nursing interventions—according to Nursing Interventions Classification—are planned (what will be done, how, how often, who does it)||0.000||0.696||0.275||0.887||0.143||0.817|
|21. The nursing interventions effect the etiology of the nursing diagnosis||0.276||0.684||0.738||0.872||0.598||0.798|
|22. Nursing interventions carried out are documented (what was done, how, how often, who did it)||−0.318||0.728||0.367||0.883||0.143||0.817|
|23. Acute, changing diagnoses are assessed daily or from shift to shift/enduring diagnoses are assessed every fourth day||0.098||0.695||0.736||0.872||0.000||0.818|
|24. The nursing diagnosis is reformulated||0.225||0.691||0.756||0.872||0.254||0.815|
|25. The nursing outcome is documented||−0.123||0.699||0.000||0.886||0.514||0.803|
|26. The nursing outcome is observably/measurably documented||0.000||0.696||0.000||0.886||0.599||0.800|
|27. The nursing outcome shows||0.158||0.692||0.689||0.874||0.530||0.804|
|28. There is a relationship between (or connection of) nursing outcomes + nursing interventions||0.173||0.693||0.446||0.882||0.682||0.795|
|29. Nursing outcomes and nursing diagnoses are internally related||0.214||0.689||0.591||0.877||0.672||0.799|
Stability and Interobserver Agreement
ICCs ranged from 0.64 to 0.85 for intraobserver reliability and from 0.68 to 0.82 for interobserver agreement (Figure 1). Box plots A, B, and C in Figure 1 show the intraobserver ICC values for centers 1, 2, and 3, respectively, and plots D, E, and F show their interobserver counterparts.
Regarding discriminant construct validity, there were statistically significant differences between the three study centers in means for the sum of all 29 Q-DIO items (Figure 2), with mean scores of 36.8 ± 4.5 (95% confidence interval [CI] 35.6–37.9) at center 1 (electronic records with standardized language), 11.5 ± 6.2 (95% CI 9.9–13.1) at center 2 (paper-based records and no standardized language), and 31.2 ± 5.3 (95% CI 29.8–32.6) at center 3 (electronic records and no standardized language). Assuming that each of the 29 items receives a maximum score of two, the highest possible score for the total of the Q-DIO instrument is 58 points.
This was the first study designed to validate the Q-DIO for use in Brazil and in the United States. The Q-DIO has been used to assess the accuracy of NDs, effectiveness of interventions, and quality of patient outcomes in records (regardless of the use of standardized language), and to assess the impact of educational programs to implement standardized NDs, interventions, and outcomes into practice, as well as within the context of audit systems. Although it was originally developed in English, it had only been validated in Swiss hospitals thus far (Müller-Staub, Needham, et al., 2008; Müller-Staub et al., 2009).
Regarding the reliability of the Q-DIO, Cronbach's alpha coefficients were ≥0.70 at all three study centers when the 29 instrument items were assessed. These values suggest that the 29 component items of the Q-DIO are correlated and homogeneous in that they measure the same attribute. However, we observe that, especially at center 2 (domain—NDs as product), the correlations between eight items were zero, indicating that these items are not related. Conversely, if these items were to be excluded, internal consistency should remain unchanged. The lack of standardized language and use of paper-based records do not favor a situation in which these items are covered in the interview and duly recorded by the nurse.
Similar values were obtained during the pilot study phase of the original version of the instrument when the 29 instrument items were assessed, which Cronbach's alpha values ranged from 0.80 to 0.98 (Müller-Staub, Needham, et al., 2008). The first study to assess the reliability of the original instrument was conducted on a random sample of 60 nursing records from 12 units of a single hospital in Switzerland. Of these, 30 were described as high-quality records and 30 as low-quality records, with and without standardized language, respectively. Despite the high Cronbach's alpha values calculated, the investigators noted that further research with larger sample sizes was required, and they reiterated the pilot nature of the study (Müller-Staub, Needham, et al., 2008).
Reliability was also assessed during pretesting of the Brazilian Portuguese version of the Q-DIO, in which 40 nursing records (50% electronic and compliant with the NANDA-I terminology and 50% paper-based and not using any standardized language). The Cronbach's alpha coefficient calculated in this sample was 0.97 (Linch et al., 2012). These findings corroborate those of prior studies in which the component items of the Q-DIO were found to measure the same attribute of nursing documentation quality, regardless of the use of standardized terminology.
Regarding stability, ICC values at centers 1, 2, and 3, respectively, were 0.74, 0.64, and 0.85 for intraobserver reliability, and 0.70, 0.68, and 0.82 for interobserver agreement. Coefficients for centers 1 and 2 corresponded to satisfactory agreement, whereas those found in center 3 corresponded to excellent agreement. Centers 1 and 2 also differed with respect to the type of nursing records. The lowest ICC values were found at center 2, where nursing records were paper based and used nonstandardized language. In this setting, free text nursing records may have contributed to variation in inter- and intraobserver assessment due to the subjectivity of individual interpretations. Conversely, center 3 (electronic records and no standardized language) had the highest coefficients of agreement. At this center, even though assessment was conducted by a U.S. investigator (native speaker of English) and by a Brazilian observer (native speaker of Portuguese), comparison between their assessments demonstrated the stability of the instrument. The findings of inter- and intraobserver assessment in the original Q-DIO study also confirmed the stability of the instrument when administered by different raters (Fayers & Machin, 2000).
In this study, discriminant construct validity was used to assess the ability of the Q-DIO to discriminate between different types of nursing records (paper-based versus electronic, with versus without standardized language). This analysis showed significant differences in means for the sum of all 29 instrument items among the three study centers.
In a prior study performed by the instrument developer to assess the discriminant ability of the Q-DIO, a randomized clinical trial was conducted to assess improvement in the quality of nursing records after an educational intervention supporting NDs implementation into clinical practice. The intervention group took part in 1.5-hr weekly sessions, led by a nurse specialized in clinical reasoning and using NANDA-I NDs, the NOC, and the NIC, for a period of 5 months. These sessions included discussion of actual cases of hospitalized patients to facilitate critical thinking when using standardized terminology. Instead of this educational program, the control group received the conventional training provided by the institution, which consisted of case discussions with no questions or reflection; textbooks were provided. At the end of the study, 225 nursing records were assessed using the Q-DIO. The study results revealed highly significant differences in quality in all three Q-DIO domains—accuracy of NDs, effectiveness of nursing interventions, and quality of nursing-sensitive patient outcomes (Müller-Staub, Needham, et al., 2008). These findings suggest that the Q-DIO was able to discriminate between the quality of records kept by nurses in the intervention group and those kept by nurses in the control group.
Among other proposed uses for the Q-DIO, the instrument has also been employed as an audit indicator to ascertain the quality of nursing records. The first study, conducted at a Swiss hospital, sought to assess the impact of implementation of an educational program on the quality of NDs, interventions, and outcomes. A total of 36 nursing records were randomly selected from 12 units of the study hospital and assessed before and after an educational program that involved implementation of NANDA–NOC–NIC terminology. Among several implications for nursing practice, this study found a significant improvement in the quality of nursing records, as measured by the Q-DIO, after the implementation of the standardized terminologies (Müller-Staub, Needham, Odenbreit, Lavin, & van Achterberg, 2007).
The Q-DIO was developed for applicability to clinical practice settings, and it has proved its value as such. Studies conducted in a variety of different contexts have validated the Q-DIO as a feasible instrument for application in real-world practice.
Having tested the psychometric properties of the Q-DIO on nursing records kept in Brazil and the United States, we conclude that the Brazilian Portuguese version of the instrument has good reliability (internal consistency and stability) as compared with the original instrument, which, in turn, had good reliability (as assessed by the same factors) when used in a U.S. setting.
Furthermore, the Q-DIO was able to discriminate quality across different modalities of nursing documentation, including paper-based and electronic records and records documented with and without standardized language.
This work was supported by the Fundo de Incentivo à Pesquisa e Eventos (FIPE) do Hospital de Clínicas de Porto Alegre (HCPA) and Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES).
- 2000). Guidelines for the process of cross-cultural adaptation of self-report measures. Spine, 25(24), 3186–3191. , , , & (
- 2000). Quality of life: Assessment, analysis and interpretation. England: John Wiley & Sons. , & (
- 2009). Análise Multivariada de Dados (6th ed.). Porto Alegre, Brazil: Artmed. , , , & (
- 2011). Electronic nursing documentation as a strategy to improve quality of patient care. Journal of Nursing Scholarship, 43(2), 154–162. , , & (
- 2012). Cross-cultural adaptation of the Quality of Diagnoses, Interventions and Outcomes (Q-DIO) Instrument into Brazilian Portuguese. International Journal of Nursing Knowledge, 23(2), 1–8. , , , , & (
- 2010). Quality of nursing records and standardized language: Literature review. Online Brazilian Journal of Nursing, 9(2), 01–06. , , & (
- 2009). Evaluation of the implementation of nursing diagnoses, interventions, and outcomes. International Journal of Nursing Terminologies and Classifications, 20(1), 9–15. (
- 2008). Testing the Q-DIO as an instrument to measure the documented quality of nursing diagnoses, interventions, and outcomes. International Journal of Nursing Terminologies and Classifications, 19(1), 20–27. , , , , , & (
- 2009). Development of an instrument to measure the quality of documented nursing diagnoses, interventions and outcomes: The Q-DIO. Journal of Clinical Nursing, 18(7), 1027–1037. , , , , , & (
- 2007). Improved quality of nursing documentation: Results of a nursing diagnoses, interventions, and outcomes implementation study. International Journal of Nursing Terminologies and Classifications, 18(1), 5–17. , , , , & (
- 2008). Implementing nursing diagnostics effectively: Cluster randomized trial. Journal of Advanced Nursing, 63(3), 291–301. , , , , & (
- 2003). Retrospective analysis of nursing records at a specialized unit. Revista Latino-Americana de Enfermagem, 11(2), 184–191. , , & (
- 2010). Prevalence of accurate nursing documentation in patient records. Journal of Advanced Nursing, 66(11), 2481–2489. , , , & (
- 2009). Evaluating nursing documentation—research designs and methods: Systematic review. Journal of Advanced Nursing, 65(3), 464–476. , & (