SEARCH

SEARCH BY CITATION

Keywords:

  • Case fatality rates;
  • emergency admission rates;
  • funnel plots;
  • gynaecology;
  • outcome indicators

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

Objective  To identify suitable outcome measures for comparing gynaecology performance between hospitals.

Design  Analysis of routinely collected statistics.

Setting  England.

Population  A total of 1.45 million gynaecology admissions in 1999/2001.

Methods  The database used was a linked file of English NHS hospital admission statistics and death certificate data. Case fatality rates (CFRs)and emergency readmission (ERA) rates were calculated for different components of gynaecology workload. Funnel plots, using age–sex standardised measures, were displayed to compare the outcomes.

Main outcome measures  CFRs and ERA rates.

Results  The CFR within 30 days after admission for patients with cancer was 5.1%. These patients accounted for only 3% of all the admissions but for 73% of all 30-day deaths. All other 30-day CFRs were extremely low—below 0.5%. The 30-day ERA rates ranged from 1.8% after day case care to 17.4% after emergency admissions for people who did not have an operation. Funnel plots showed considerable variation between hospitals for ERA after day case care but not after elective abdominal hysterectomy.

Conclusions  There are no measures of mortality that could be used routinely and meaningfully to compare the performance of gynaecology units. We suggest that two suitable comparative measures of outcome, derivable from routine hospital statistics, are 30-day ERA rates after day case admissions and after elective abdominal hysterectomy, excluding those records with a cancer diagnosis. These measures are relatively homogeneous with respect to their likely rates of adverse events and have sufficient numbers to produce potentially useful comparative results.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

The objective of our study was to determine whether there are outcome indicators for gynaecology, derivable from routinely collected statistics, which are potentially suitable for making comparisons between hospitals. Two main principles were adopted. First, although the clinical data in routine statistical systems are limited, if analysed appropriately, they may include information of interest to clinicians. Second, any measures used to compare hospitals must be based on large enough numbers of adverse events for the comparisons to be statistically meaningful. Almost invariably, this means considering clinical conditions that are treated in large numbers or that have high rates of adverse outcomes. If such conditions are few, one may consider grouping conditions together, providing that a group so formed includes conditions that are likely to be homogeneous in respect of the outcomes studied.

Patients and methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

Gynaecology workload was grouped into broad clinical categories. Judgements were then made about those groups to study in detail based on the numbers in each group and their clinical homogeneity in respect of the outcomes being investigated.

The database used was a linked file of hospital episode statistics (HES) for hospital admissions (technically NHS trust spells) in England (population of 50 million). HES includes a short statistical record of all inpatient and day case admissions to hospitals in the NHS. In the linked file, successive admissions for the same person were brought together. The hospital data were linked to mortality data derived from death certificates from the Office for National Statistics. This ensured that all deaths were identified, including those that occurred after discharge or after transfer from the hospital. Initial admissions included in the analysis were for the period 1 April 1999 to 31 March 2001 (1999/2001), and there was a further follow up of data until 30 June 2001 to allow the identification of all events of interest up to 90 days after admission.

The initial admissions were NHS trust spells, starting with a ‘finished consultant episode’ in gynaecology. Admissions including a cancer diagnosis (ICD-10 codes C00-97, D37-48 and Z51.1) were excluded from the indicator specifications except when stated otherwise.

Admissions were divided into groups according to the method of admission as day case care, elective ordinary admissions (the HES term for elective admissions with an overnight stay), emergency admissions, transfers from another hospital and method of admission not known. The elective and emergency groups were further divided into admissions that included an operation and those that did not. In specifying the groups that had operations, admissions which only included certain minor and mainly diagnostic operative procedure codes, such as urinary catheterisation, were not counted as operations but were included in the nonoperation group.

Analyses were also performed for elective admissions in which an abdominal hysterectomy was coded as the main procedure (OPCS-4 code Q07 abdominal excision of uterus). This is the only major operation in the specialty that is recorded as the main procedure in more than 20 000 admissions nationally per year, and it accounts for 22% of all elective admissions which had an operation, when patients with cancer are excluded. Two-third of these patients also had unilateral or bilateral removal of their adnexae, but there was no difference in case fatality rates (CFRs) or emergency readmission (ERA) rates between those who had their adnexae removed and those who did not.

CFRs were calculated including deaths from any cause within 30 and 90 days after the initial admission (which we termed the ‘index’ admission). ERA rates were calculated using the first ERA occurring within 30 and 90 days after discharge from the index admission. All first ERAs were included regardless of diagnosis. Patients were excluded from the readmission analyses if they died during the index admission or during the time over which the measure was being derived (30 or 90 days). For the purpose of analysing readmission rates, index admissions were specified to include only those patients who were discharged to their usual place of residence. CFRs and ERA rates were age and sex standardised using the indirect method of standardisation.

Results are shown graphically as funnel plots, which show standardised ERA rates on the y-axis plotted against expected numbers of ERAs on the x-axis [in a scatter plot]. The horizontal line in the middle of each plot shows the national overall ERA rate around which the standardised ERA rates for each trust are distributed. The outer lines show the 95 and 99% confidence intervals for each value of the ‘expected’ number of ERAs (which is a function of the national average ERA rate and the total number of patients in the clinical category treated in each hospital trust). As the expected number of ERAs gets larger, the lines denoting the successively smaller confidence intervals get narrower, leading to the appearance of a funnel shape.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

Table 1 shows the number of admissions, deaths and ERAs, CFRs and ERA rates for the different groups of index admissions making up the gynaecology case mix.

Table 1.  Number of gynaecology admissions, deaths and ERAs, death and ERA rates 1999/2001
Type of caseNumber of admissionsDeathsERAs
 30 days90 days30 days90 days
No.RateNo.RateNo.RateNo.Rate
All admissions1 452 82827410.265770.583 2945.7118 4218.2
Admissions with cancer39 69320065.1467511.830237.6541613.6
Admissions without cancer1 413 1357350.119020.180 2715.7113 0058.0
Day cases686 79547>0.1181>0.112 2901.821 9383.2
Ordinary elective with operation280 082134>0.13870.178622.811 9684.3
Ordinary elective without operation40 390880.22520.620345.032428.0
Emergency with operation112 052910.11980.285457.612 62911.3
Emergency without operation280 7783400.18090.348 94017.462 24822.2
Transfer5779220.4410.72905.04347.5
Method of admission not known7259130.2340.53104.35467.5
Elective abdominal hysterectomy61 654330.1520.135145.742346.9

The CFR after admission for patients with cancer was 5.1% within 30 days and 11.8% within 90 days. All other 30-day CFRs were below 0.5%. The 30-day ERA rates ranged from 1.8% after day case care to 17.4% after emergency admissions that did not have an operation. Abdominal hysterectomy accounted for 22% of the elective admissions with an operation but was associated with 42% of the ERAs for this group.

To produce statistically relevant information comparing hospital outcomes, there must be an adequate number of index admissions and adverse events. For the purposes of this study, it was decided to continue to investigate measures with at least 20 000 admissions nationally per year and adverse event rates of more than 1%. None of the CFR measures met these criteria in gynaecology. With respect to ERA rates, analyses were performed for all groups except for transfers and method of admission not known groups. Funnel plots for all these measures have been produced. The two indicators judged to have least confounding from case-mix variation were the ERA rates for day cases and abdominal hysterectomy, and their funnel plots are shown in Figures 1 and 2. The other funnel plots are posted on the web.1

image

Figure 1. Gynaecology day cases in English hospitals 1999/2001: Standardised 30-day ERA rates plotted against expected number of ERAs. Each plot is a hospital.

Download figure to PowerPoint

image

Figure 2. Elective admissions for abdominal excision of uterus in English hospitals 1999/2001: Standardised 30-day ERA rates plotted against expected number of ERAs. Each plot is a hospital.

Download figure to PowerPoint

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

The linked file allows the calculation of readmission rates and mortality rates regardless of where and when deaths occurred. In-hospital mortality rates, only counting deaths in the index admission, miss a substantial percentage of deaths within 30 days of admission.2 However, with the exception of admissions for patients with cancer, mortality rates in gynaecology, even including out-of-hospital and 90-day deaths, were very low. There were less than 90 deaths per annum within 30 days of admission for day case care and for elective admissions with an operation, taken together. There is no statistical measure of mortality in noncancer gynaecology that is useful, as a routine, for comparing trust or consultant outcomes. However, it may be worth treating any such deaths as a sentinel event and reviewing the circumstances in which they occur. This is the approach taken by national confidential enquiries.3

We recommend that hospital statistics should not be used to study outcomes after admissions with a diagnosis of cancer. Nationally organised audits are the most clinically useful way of producing measures about cancer outcome as they enable the indicators to be risk adjusted for a number of factors, such as cancer stage, not normally collected by routine data systems such as HES. Furthermore, because cancer diagnoses are associated with a relatively high rate of deaths during or shortly after admission, their inclusion in analyses of other gynaecology workload would swamp CFRs from the other gynaecological conditions. When gynaecology workload as a whole is considered, admissions with a cancer diagnosis accounted for only 3% of all admissions but for 73% of the 30-day deaths in the specialty.

A clinically useful indicator should permit comparison to be made of ‘like with like’ with respect to case mix. The use of comparative indicators at specialty and individual consultant level is problematic because of the different case mix that will exist in different hospitals. Indeed, a key determinant of a gynaecology department’s or gynaecologist’s CFR is likely to be the volume of surgery carried out on patients with cancer. The use of CFRs, based on total caseload, to compare the performance of individual surgeons or the specialty as a whole is thus clinically inappropriate and could be very misleading. The two measures studied by us that are most likely to contain reasonably homogeneous populations with respect to clinical outcomes are, after excluding any patient with a cancer diagnosis, 30-day ERA rates following day case care and following elective admissions for abdominal hysterectomy. Day cases by definition are all planned and the patients are intended to be admitted and discharged without an overnight stay. A key clinical issue in day case care is the appropriate selection of patients, and it can be assumed that a major selection criterion is a low risk of postoperative mortality and morbidity. Accordingly, although heterogeneous in case mix, day cases are homogeneous in expected outcome in that their postoperative adverse event risk is or should be negligible. It is therefore appropriate to study day cases as a single group for the purposes of comparing ERA rates. Elective admissions for abdominal hysterectomy are unlikely to differ greatly between departments in severity of the gynaecological condition once patients with a cancer diagnosis have been excluded, although they may vary with respect to the presence of other comorbidities.

Process control charts such as funnel plots allow hospitals to be split into the following three groups: those whose performance is unremarkable, those whose performance is of concern and those that seem to be performing well. The funnel charts used for this study show those that were outside 95 and 99% confidence limits. Data on about 200 hospitals were analysed and one would expect about two hospitals to fall outside the 99% confidence intervals by chance. The readmission funnel plot for day case care (Figure 1) shows ten hospitals above and about 30 below these limits. Much lower levels of variation for elective abdominal hysterectomy are shown in Figure 2.

In using routinely collected administrative data like HES, consideration must be given to the likely or known accuracy of coding. Published large-scale studies of validation, checking codes against original case notes, are few in England. In recent years, furthermore, privacy concerns have made such studies difficult or impossible to undertake. It is clear, however, that it is hard to generalise about reliability. Gough et al.,4 who compared their surgical audit system with routine hospital statistics, reported that there was very close agreement between the two for ‘straightforward procedures such as appendicectomy, cholecystectomy, hernia repair and breast operations’. They reported, however, that there was a shortfall of recording of endoscopies in the routine hospital statistics. Campbell et al.5 undertook a systematic review of coding accuracy as reported in 21 published studies. They concluded that ‘coding accuracy on average is high’, at least for principal diagnoses and operations, but that there is variation in accuracy between studies, centres and different clinical conditions.

We focused on readmission and mortality rates. Other measures could be constructed from routine data, some but not all requiring linked data, such as lengths of hospital stay and total time spent in the hospital6 (to compare hospitals in respect of prolonged hospitalisation), return-to-theatre rates and postoperative infection rates. Harley et al.,7 in a study to determine whether routine data can be used to identify extreme cases of poor performance, considered not only measures like these but also measures of overuse of operations that should be uncommon in particular groups of patients. These included numbers of hysterectomies in women younger than 30 years and numbers of sterilisations on women younger than 25 years.7

The measures we have recommended, although potentially useful indicators, should not be used to make definitive judgements about hospitals. Well-chosen indicators provide pointers as to where further investigation of the services may be worthwhile, but they do not give definitive answers about whether services are good or poor. Any indicators derived from routine national statistics must be used with care and for purposes for which the data are of sufficient quality to be suitable. The gynaecology workload has been defined by broad categories derivable from a linked hospital admission and mortality database. Clinical and statistical criteria have been used to identify those groups that might provide the most useful comparative indicators. The two recommended measures minimise case-mix differences and have sufficient numbers to produce comparative analyses. Their funnel charts have identified a few hospitals in 1999/2001 that would probably have merited further investigation.

Acknowledgements

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

The work programme of the Unit of Health-Care Epidemiology to analyse the linked data was funded by the NHS National Centre for Research Capacity Development. This work was also funded in part by the Department of Health through its funding of the National Centre for Health Outcomes Development.

Ethical approval

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References

Ethical approval for analysis of the database has been obtained from the Central and South Bristol Research Ethics Committee in its role as a multicentre research ethics committee (04/Q2006/175).

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Patients and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. Ethical approval
  9. References
  • 1
    Mason A, Seagrott V, Meddings D, Goldacre M. Clinical Series Report CR17. Gynaecology: Case-fatality and Hospital Readmission Rates. Oxford, UK: National Centre for Health Outcomes Development, 2005[www.uhce.ox.ac.uk/hessepho]. Accessed 22 March 2006.
  • 2
    Goldacre MJ, Griffith M, Gill L, Mackintosh A. In-hospital deaths as a fraction of all deaths within 30 days of admission for surgery: analysis of routine statistics. BMJ 2002;324:106970.
  • 3
    National Confidential Enquiry into Patient Outcomes and Deaths. [www.ncepad.org.uk]. Accessed 22 March 2006.
  • 4
    Gough MH, Kettlewell MGW, Marks CG, Holmes SJK, Holderness J. Audit: an annual assessment of the work and performance of a surgical firm in a regional teaching hospital. BMJ 1980;281:91318.
  • 5
    Campbell SE, Campbell MK, Grimshaw JM, Walker AE. A systematic review of discharge coding accuracy. J Public Health Med 2001;23:20511.
  • 6
    Ferguson JA, Goldacre MJ, Henderson J, Gillmer MDG. Audit of workload in gynaecology: analysis of time trends from linked statistics. Br J Obstet Gynaecol 1991;98:7727.
  • 7
    Harley M, Mohammed MA, Hussain S, Yates J, Almasri A. Was Rodney Ledward a statistical outlier? Retrospective analysis using routine hospital data to identify gynaecologists’ performance. BMJ 2005;330:92941.