Closing the patient experience chasm: A two‐level validation of the Consumer Quality Index Inpatient Hospital Care

Abstract Background Evaluation of patients’ health care experiences is central to measuring patient‐centred care. However, different instruments tend to be used at the hospital or departmental level but rarely both, leading to a lack of standardization of patient experience measures. Objective To validate the Consumer Quality Index (CQI) Inpatient Hospital Care for use on both department and hospital levels. Design Using cross‐sectional observational data, we investigated the internal validity of the questionnaire using confirmatory factor analyses (CFA), and the generalizability of the questionnaire for use at the department and hospital levels using generalizability theory. Setting and participants 22924 adults hospitalized for ≥24 hours between 1 January 2013 and 31 December 2014 in 23 Dutch hospitals (515 department evaluations). Main variable CQI Inpatient Hospital Care questionnaire. Results CFA results showed a good fit on individual level (CFI=0.96, TLI=0.95, RMSEA=0.04), which was comparable between specialties. When scores were aggregated to the department level, the fit was less desirable (CFI=0.83, TLI=0.81, RMSEA=0.06), and there was a significant overlap between communication with doctors and explanation of treatment subscales. Departments and hospitals explained ≤5% of total variance in subscale scores. In total, 4‐8 departments and 50 respondents per department are needed to reliably evaluate subscales rated on a 4‐point scale, and 10 departments with 100‐150 respondents per department for binary subscales. Discussion and conclusions The CQI Inpatient Hospital Care is a valid and reliable questionnaire to evaluate inpatient experiences in Dutch hospitals provided sufficient sampling is done. Results can facilitate meaningful comparisons and guide quality improvement activities in individual departments and hospitals.


| INTRODUCTION
Evaluation of patients' health care experiences has become central to measuring quality in health care and, as a result, health care providers are more often held responsible for monitoring and improving patients' care experiences. 1 Patient care experiences reflect the degree to which care is patient-centred (ie care that is respectful and responsive to patients' preferences, needs and values). 2 In addition to its intrinsic value as an indicator of quality, a growing body of evidence points to the positive associations between positive patient experiences and clinical processes of care 3,4 as well as better patient adherence to treatment, improved clinical outcomes and decreased utilization of health care services. 5 Even though improving patient care experiences is increasingly being incorporated in both local and global health agendas, 6 patient feedback remains largely underutilized in local hospital improvement plans. 7 One of the main reasons for this is lack of specific and timely feedback that is easily translatable to improvements on the frontline. 8,9 Current instruments used to collect patient experience data mostly collect data on hospital-wide level for identification of larger national trends and contracting of hospital services. In order to bridge the gap between external reporting and internal quality assurance, some have recommended to use different instruments for different purposes. 9,10 This is, however, not desirable due to lack of standardization of measures, a lack of common language and possible disconnect between local improvement efforts and hospital-wide measurements. Implementation of instruments is also costly and can potentially lead to duplication of work and unnecessary use of valuable resources.
An alternative approach is to adapt existing instruments to reflect their multiple purposes. In this study, we attempt to address these problems using the Dutch version of the American Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, which was imported into the Netherlands in 2006 by Arah et al. for use within the Dutch health care system. 11 This has led to the development of nationally used standardized questionnaires and protocols called the Consumer Quality Index (CQI), wherein the Dutch HCAHPS is known as the CQI Inpatient Hospital Care. 12 Efforts to adapt this questionnaire for multiple purposes, including external accountability and internal quality assurance, have resulted in different versions of the questionnaire to be produced. 13,14 However, no extensive validation of the CQI Inpatient Hospital Care has occurred since the original validation study by Arah et al. (2006). As the results are consequentially used by patients, hospital staff, health insurers, the inspectorate and researchers for different purposes, it is imperative that the questionnaire can evaluate and differentiate patient care experiences across hospitals, specialties as well as departments reliably and validly. We aimed to assess the internal validity and reliability of the CQI Inpatient Hospital Care on both hospital and department levels. Additionally, we investigated whether the questionnaire measured similar domains of patient experiences across four specialties, namely surgery, obstetrics and gynaecology, internal medicine and cardiology.

| Setting and study population
We analysed CQI Inpatient Hospital Care questionnaire data from 23 Dutch hospitals including four academic centres, 515 department evaluations in 17 specialties (nine surgical and eight medical) collected between 1 January 2013 and 31 December 2014. Eligible patients 16 years or older who were hospitalized for at least 24 hours with a discharge within the previous 12 months were identified using hospital admission lists. Eligible participants were invited to evaluate their experiences of hospitalization using either online or paper-based CQI Inpatient Hospital Care (Appendix S1). Evaluations collected in 2013 were used for national benchmarking among 43 hospitals in four specialties, namely surgery, internal medicine, cardiology, obstetrics and gynaecology. Therefore, we focused on these specialties in this study. The hospitals and clinical departments that re-evaluated their inpatient hospital care using the same questionnaire in 2014 for own internal quality assurance purposes were considered to be independent evaluations and were, therefore, also included in the analysis. We analysed the results both for 2013 and 2014 together and separately, and if there was no change, reported the combined results only.

As retrospective research does not fall under the Dutch Medical
Research Involving Human Subjects Act (WMO), an official ethical review was not required for this study. Nonetheless, we obtained permissions from individual hospitals to use anonymized questionnaire data for research purposes. Furthermore, we consulted a privacy officer at our institution to ensure that the data provided for this research complied with Dutch Personal Data Protection Act. Participating hospitals were recruited through the Miletus Foundation (www.stichtingmiletus. nl), a coordinating body of all CQI evaluations within the Netherlands.
A detailed research proposal was sent to all hospitals and subsequently discussed at the general meeting. Hospitals interested in participating in the study gave informed consent either via the Miletus Foundation or by directly contacting the primary researcher (AS). MediQuest (home. mediquest.nl), a company that processes patient evaluation data from these evaluations, provided the final data set for the study.

| CQI Inpatient Hospital Care questionnaire
The CQI Inpatient Hospital Care questionnaire has been developed in co-operation with patient and consumer organizations based on three existing instruments used to measure patient care experiences: the CAHPS Hospital Care questionnaire, the Dutch Hospital Association inpatient satisfaction questionnaire and the Hospital Comparison

| Statistical analysis
First, respondents and non-respondents were described using descriptive statistics. Questionnaires were excluded if they had a negative or no response to the question whether or not the patient had a hospital admission within the last 12 months or if less than half of core items were completed. Evaluations with missing data were imputed using multiple imputation technique to create 10 complete data sets. 16 Multiple imputation was preferable to single-imputation methods such as maximum-likelihood approaches because it better reflected the inherent uncertainty due to missing data in the sample. 17 Convergence of the imputations was assessed by examining trace plots and calculating the Rhat statistic. 18 In order to maximize convergence, we increased the number of maximum iterations to 200.
We then calculated the subscale scores for each imputed data set by averaging the scores for the items within each subscale.
The internal validity of the questionnaire was evaluated by assessing the fit of the pre-identified 9-factor structure of the questionnaire.
In order to assess the overall fit of the model, we performed a confirmatory factor analysis (CFA) on all imputed data sets and combined the final results using Rubin's rules. For categorical variables, weighted least squares with mean and variance adjusted (WLSMV) estimator was preferred to account for the categorical nature of the answers.
The WLSMV estimator is a robust estimator that does not assume normally distributed variables and is preferred for modelling categorical or ordered data. 19 We assessed the global model fit using the comparative fit index (CFI), Tucker-Lewis index (TLI) and root mean square error of approximation (RMSEA). 20 The following cut-off values indicated a good fit: CFI≥0.95, TLI≥0.95 and RMSEA≤0.06. 19 The overall fit was deemed acceptable if at least two of the three criteria of fit indices were met. 21 In order to establish whether the questionnaire measured similar patient experiences across various medical specialties, CFA was then repeated in four subgroups: surgery, obstetrics and gynaecology, internal medicine and cardiology. These specialties were chosen because these specialties were included in the national benchmark. Same cut-off points were used to evaluate the fit of the factor structure as for the overall sample. Finally, we repeated the CFA on the department level by aggregating the scores of each variable to the department level.
Internal consistency of the subscales was evaluated by calculating

| RESULTS
Of the distributed 74090 questionnaires, 23476 were returned (gross response rate 31.7%). Table 1 reports characteristics of respondents and non-respondents. In total, 552 questionnaires were excluded due to negative or no response to the question whether or not they had a hospital admission within the last 12 months or less than half of core items completed. The resulting sample size was 22924 (net response rate 30.9%), including 23 hospitals, 17 different specialties and 515 department evaluations. Table 2 further describes the demographic characteristics of the included respondents.
As the results did not differ between 2013 and 2014, we report only combined results below.

| Psychometric properties
CFA showed a good fit for surgical, obstetrics and gynaecology, internal medicine, cardiology specialties and all specialties combined (  Table 4).
Communication of treatment did not predict global ratings of either the hospital or the department, while explanation of treatment was a significant predictor of the rating of the hospital but not the global rating of the department (Table 4).
5% or less of total variance in scores was attributable to the department or the hospital ( Table 5)  that communication with nurses was the strongest predictor of overall ratings of the department as well as the hospital. This is not surprising as nurses are the primary providers of care in the hospital environment. Furthermore, research has shown that factors related to nursing work such as nursing work environment, nurse-to-patient ratios 28 and missed nursing care 29 and nurse-patient interaction 30 can influence patient satisfaction ratings. A new finding, however, is that higher scores on the subscale discharge information significantly contributed to patients' global ratings of both the hospital and the department.

| DISCUSSION
This is different from the findings by Elliott et al. 26 , in which discharge information was one of the least valued aspects of inpatient care and was important for only half of hospitalization types. This is not surpris- ing as there appears to be a gap in communication between patients and providers at discharge. A survey of hospitalized patients showed that more than half of patients 70 years or older did not receive instructions about how to care for themselves after hospitalization. 31 Our findings suggest that discharge information may be more import- summative judgements about the quality of care. 34 We, therefore, recommend using the generalizability results of this study (shown in Appendix S2) to adjust the cut-off criteria based on the proposed use of the questionnaire.
In interpreting the results, several limitations should be mentioned. Patient surveys suffer from low response rates. Our response rate of 31% was similar to those previously seen in this setting. 14 Reasons for non-response were not collected during the original data collection process, which made a non-responder analysis impossible.
Although we tried to account for non-respondents by including sex and age as covariate in regression analyses, this may not have been sufficient because respondents and non-respondents may also vary based on other characteristics that we have not been able to account for, such as country of origin, language spoken at home or level of education. For example, we did not have any data on how many patients were invited to fill out online or paper-based questionnaire.
Furthermore, in this study we aggregated the individual scores to the level of the department, because this is how typically the scores may be used. Other methods can be tried, such as using median or factor scores, but these may be difficult to interpret. Also, we did not test alternative models on the department level or factor equivalency between different specialties or respondents groups. Finally, we did not investigate the external validity of this questionnaire by studying the relationship between aspects of inpatient hospital care and other important process or outcome measures. Nonetheless, this study also has several strengths. One strength of this study is its use of more than 22000 patient evaluations and over 500 department evaluations from multiple specialties in multiple hospitals including academic and nonacademic centres, which supports the generalizability of our results.
Another strength in this study is the use of multiple imputation for handing missing data, which accounts for the uncertainty associated with imputation of missing data. 17 With this study, we contribute evidence for validity of the CQI Inpatient Hospital Care questionnaire and its utility for use in different settings and for both quality assurance and summative purposes.
We recommend that stakeholders including hospitals, clinical departments and health insurers using this questionnaire use appropriate sample sizes based on its purpose and level of use. Considering the response rate is 31%, much larger samples may be required to arrive at recommended numbers of evaluations. Low response rates have become worrisomely common in survey research, 35 with many studies now reporting rates as low as or lower than ours. 36 Low response rates may indicate low levels of receptivity of the instrument by patients. Improvements in response rates, for example by identifying and addressing reasons for non-response, are needed to ensure optimal use of resources as well as appropriate sample sizes.
Although this questionnaire has originally been imported to facilitate standardization of the instrument for international comparisons, 11 38 we recommend using multilevel models for longitudinal and hierarchical data analyses, rather than using average department or hospital scores.
In conclusion, the CQI Inpatient Hospital Care questionnaire can provide valid and reliable data on patient experiences of inpatient hospital care on both department and hospital. The resulting data can be used to facilitate meaningful comparisons and guide quality improvement activities. Future research can focus on improving reliability of the scales, wording of the individual items to reflect specific provider or clinical settings better, and validating the structure on the department level and for different specialties.