A validation study: how effective is the Hospital Morbidity Data as a surveillance tool for heart failure in Western Australia?
Tiew-Hwa Katherine Teng, School of Population Health (M431) University of Western Australia, 35 Stirling Highway, Crawley, WA 6009. Fax: (08) 6488 1188; e-mail: email@example.com
Objective: To determine the accuracy of the hospital discharge coding of heart failure (HF) in the Western Australian (WA) Hospital Morbidity Data (HMD).
Methods: A retrospective medical chart review of a sample of 1,006 patients with a principal diagnosis code indicating HF in the WA HMD was undertaken. Validation was reported against a written diagnosis of HF in the medical chart and using Boston criteria score as a gold standard.
Results: The positive predictive value (PPV) of the HMD coding of HF as the principal diagnosis was 99.5% when compared to the medical chart diagnosis and 92.4% when compared to the Boston score criteria for ‘definite’ HF and 98.8% for a combined ‘possible’ and ‘definite’ HF Boston score.
Conclusions: With the high predictive accuracy, the WA HMD can be used with confidence to monitor trends in the epidemiology of in-hospital HF patients.
Heart failure (HF) is a major public health problem in Australia. More than $1 billion of the $5 billion per year attributed to chronic cardiovascular disease in Australia1,2 is related to heart failure, with 70% of the total health expenditure related to hospitalisation costs.3,4 Despite its significance, there is a paucity of information about the incidence and prevalence of the disease, especially at a population level.
In Western Australia (WA), all hospital admissions and separations in all rural/metro, private/public hospitals are recorded in an administrative database known as the WA Hospital Morbidity Data (HMD).5 The WA HMD has the potential to identify and monitor heart failure incidence, prevalence, trends and outcomes over time, if the validity of the coding of heart failure in the HMD can be established.
In this study we sought to validate the diagnostic coding of heart failure in the WA HMD against two different criteria, namely: 1) a medical diagnosis of ‘heart failure’ written in the patient's medical chart; and 2) the well-established Boston criteria.6
A validation study involving the retrospective review of the medical charts of patients identified in the WA HMD as having a primary diagnosis of HF. Ethics Approval for the study was obtained from each of the Hospital's Ethics Committees and the WA Confidentiality of Health Information Committee (CHIC).
The WA HMD was used to identify a random sample (n=1,006) of patients with a principal (separation) diagnosis of HF from any of the three tertiary metropolitan hospitals (Sir Charles Gairdner Hospital, Royal Perth Hospital and Fremantle Hospital), between 1996 and 2006. HF was defined on the basis of the diagnostic codes used in the International Classification of Diseases (ICD), and included any of the following codes: ICD9: 428x, 402.01, 402.11, 402.91, 404.1, 404.3, 425x, 518.4, 514, 391.8, 398.91; and ICD10:I50x, II 1.0, 113.0,113.2,142x, J81,101.8,1020.
The study was restricted to non-elective admissions. Patient records with missing chest radiography reports were excluded, due to the inability to score category III of the Boston diagnostic criteria.
The Medical charts were reviewed by two research staff and detailed clinical information extracted directly into an Access database. Of the three most commonly used diagnostic criteria for HF, namely, Framingham,7 Boston6 and the European Society of Cardiology8 (ESC), we chose the Boston score as the ‘gold’ standard based on its relative ease of use and better construct validity9 The Boston diagnostic score6 was calculated as shown in Table 1, with ‘possible’ HF (scores 5-7) and ‘definite’ HF (scores ≥8) used as the gold standard.
Table 1. The Boston Diagnostic Criteria.
|CATEGORY I – HISTORY|
|Paroxysmal nocturnal dyspnea||3|
|Dyspnea while walking on level area||2|
|Dyspnea while climbing||1|
|CATEGORY II – PHYSICAL EXAMINATION|
|Heart rate abnormality||1-2|
| 1 point if 90-110 beats/min||1|
| 2 points if >110 beats/min||2|
|Jugular venous elevation||2-3|
| 2 points if >6 cm H2O||2|
| 3 points if 6 cm H2O & hepatomegaly or edema||3|
| 1 point if basilar||1|
| 2 points if more than basilar||2|
|Third heart sound||3|
|CATEGORY III – CHEST RADIOGRAPHY|
|Alveolar pulmonary edema||4|
|Interstitial pulmonary edema||3|
|Bilateral pleural effusion||3|
|Cardiothoracic ratio >0.5||3|
|Upper zone flow redistribution||2|
Concordance between the WA HMD principal diagnosis of HF and both gold standards was reported as proportions and positive predictive values (PPVs).
Categorical variables were presented as proportions and compared using the Chi-square test. Continuous variables were reported as means (standard deviations) and compared using the Student's t-test. A p value ≤0.05 was considered statistically significant.
Is the diagnosis of HF in the WA HMD accurate?
The review of the medical charts (including diagnostic test results) of the 1,006 patients with a principal diagnosis of HF in the WA HMD found only five cases without a ‘true’ diagnosis of heart failure. Four of these were clearly not HF and one had a secondary diagnosis of HF. The positive predictive value (PPV) of the HMD versus medical chart HF diagnosis was 99.5% (1001/1006).
Based on the clinical data extracted from the medical charts, a Boston HF score was calculated for each of the 1,006 patients. Only 12 had a low score (≤4); with 65 classified as ‘Possible HF’ (score= 5-7) and 929 classified as ‘Definite HF’ (score ≥8). Therefore the PPV of a HMD diagnosis of HF against a ‘Definite’ HF Boston score was 92.4% (929/1006); and against a combined ‘Possible’ and ‘Definite’ HF Boston score was 98.8% (994/1006).
Was there inter-rater reliability?
Thirty-one medical charts were cross reviewed by both data extractors, with perfect agreement found. (Kappa score=1.0)
Was the study sample representative of the total HF population?
The characteristics of the study sample were found to be similar to those of the total population of patients with a principal diagnosis of HF in the HMD, between 1996 and 2006 (see Table 2). Similarly, there was no significant difference in the crude case fatality between the study sample and the total HF population, in terms of in-hospital case fatality (6.2% vs 6.1%) or 5-year fatality (52.1% vs 47.8%).
Table 2. Age group distribution by gender in our random sample of HF patients versus the HF population.
|60-64 years||47 (9.2)||26 (5.1)||73 (7.2)||516 (9.2)||263 (4.7)||779 (7.0)|
|65-69 years||65 (12.8)||41 (8.1)||106 (10.5)||752 (13.4)||449 (8.1)||1201 (10.7)|
|70-74 years||83 (16.3)||56 (11.1)||139 (13.7)||945 (16.8)||698 (12.5)||1643 (14.7)|
|75-79 years||117 (22.9)||79 (15.7)||196 (19.3)||1137 (20.2)||1030 (18.5)||2167 (19.3)|
|80-84 years||96 (18.8)||125 (24.8)||221 (21.7)||1157 (20.5)||1197 (21.5)||2354 (21.0)|
|>85 years||102 (20.0)||177 (35.1)||279 (27.5)||1122 (19.9)||1938 (34.8)||3060 (27.3)|
|Total||510 (50.3)||504 (49.7)||1014 (100)||5629 (50.2)||5575 (49.8)||11204 (100)|
Few studies have examined the validity of the coding of HF in hospital administrative databases. We found the positive predictive value (PPV) of the HMD coding of HF as the principal diagnosis to be 99.5% when compared to the medical diagnosis written in the medical charts and 92.4% when compared to the relatively ‘strict’ Boston score criteria for ‘definite’ HF The PPV was even higher if only principal diagnosis codes of 428x and I50x were considered (99.6% when compared to medical charts and 94.6% when compared to ‘definite’ HF). Contrary to reports that hospital discharge coding for HF are unreliable,10 our results confirm the accuracy of the principal diagnosis of HF in the WA HMD.
With more than 1,000 patient records reviewed, our study is one of the largest reported. Moreover, there was good inter-rater reliability between the two data abstractors and the validation sample was shown to be representative of the population from which it was drawn.
Our results are more favourable than a 2001 WA study,11 involving 844 admissions of HF for 379 patients from the Perth MONICA register. It was found that cardiac failure was mentioned in the written discharge summary in the medical record in 93% of the 844 admissions. However, the study sample of post-AMI patients included all cases where there was mention of HF in any of the diagnostic fields – not just the principal diagnosis.
Our results are comparable to a Canadian study in 200512 where the PPV of the principal diagnosis of HF in the administrative records from the Canadian Institute of Health Information was 94.3% using the Framingham criteria and 88.6% using a scoring system similar to the Boston criteria.
The high accuracy of the HMD coding reported in our study might reflect the intensity of training of clinical coders in WA hospitals that accompanied the migration of the ICD-9-CM to the ICD-10-AM coding in 1998 and the implementation of the case-mix (Diagnosis-Related Groups) based classification system. It may be that the quality of coding in administrative databases has improved in more recent years.
One limitation of our study is that we were unable to determine the magnitude of the ‘false negative’ classification of heart failure in the WA HMD, and as such were not able to calculate the negative predictive value. This would have required sampling a very large number of patients without a principal diagnosis of HF and undertaking medical chart review to ascertain whether there was any mention of heart failure. Cost and logistics prohibited this. Our results have confirmed that if there is a principal diagnosis of HF in the WA HMD, we can be confident that this is truly a hospitalisation relating to HF. Thus, we can be assured that any incidence or prevalence measures of hospitalisations for HF made using the WA HMD represent a true minimum population value. We recognise that the HMD only captures HF patients presenting to hospital, and not the total number of HF patients in the community. However, population-based cohort studies have shown that a large percentage (59%)13 of newly diagnosed HF patients require hospitalisation within two years of diagnosis and more than 80% of incident cases of HF require hospital admission.14
The restriction of the study to patients presenting to only three tertiary metropolitan hospitals in WA might have threatened the representativeness of the sample – however, as shown in Table 2, this does not appear to have been the case.
Heart failure, as a principal diagnosis in the WA HMD has a high positive predictive value, when compared to the Boston diagnostic criteria and written diagnosis in the medical charts.
Implications for health practice
The results of this validation study confirm the accuracy of the diagnostic coding of HF in the WA HMD and lends confidence in the capacity of the WA HMD to be used to investigate the epidemiology and cost of (hospitalised) HF patients in WA. Such information can assist with the planning of health service requirements and development of health policy regarding the prevention and management of HF.
Special thanks to Aqif Mukhtar for the database establishment and Louise Schreuders for assisting with the medical chart reviews. The assistance of staff from Royal Perth Hospital, Sir Charles Gairdner Hospital, Fremantle Hospital, with accessing the medical charts, is much appreciated.