RO1: has the overall average of hospital standardised mortality ratios increased or decreased during the period 2005–2012?
From 2005–2012, there has been a 6% increase in mean HSMRs for all providers.
What exactly does this mean?
It means that if we are to believe that the Dr Foster HSMRs are true representations of the ratio between how many people die compared with how many people should die at hospitals, then more people are dying now in hospitals compared with how many should, compared with 2005.
It also means that we may doubt the data coding techniques of Dr Foster. Mohammed et al. (2009) published an influential paper titled ‘Evidence of methodological bias in HSMRs: retrospective database study of English hospitals’. This paper is heavily critical of HSMRs as an accurate measure, because of the coding and risk predictions that make them up. This article goes on to conclude that ‘variations in HSMRs from Dr Foster Unit reflecting differences in quality of care are less than credible’ (2009: 228; 780). Thus, it is possible that this increase in HSMRs over time does not truly represent a worsening of hospital quality.
In appendix 9 of the Francis Enquiry (vol. 1), there is a detailed review of mortality statistics produced by two Harvard academics. This also points out that there is no fool-proof method for quantifying health care quality. It is stated that
We accept that there is no single, perfect mechanism for assessing health care quality. We also agree that every statistical quality monitoring algorithm, including Dr Foster, should be critically examined by experts to determine its validity. (2010, p.440)
There is, however, significant evidence in this paper, which means that Dr Foster HSMRs should not be ignored or written off completely because of the methodological flaws. The conclusion on HSMRs is as follows:
The HSMR is a summary figure, designed to give an overview of mortality within a trust, and we accept it will hide a considerable number of differences in the risk profiles across different factors in the model, but we do not see why this should decrease the value of the HSMR as a summary figure used in conjunction with other measures. (2010, p.442)
Specifically referring to the Mohammed et al. (2009) research, it is stated that
We are disturbed by the final sentence summarising the author's conclusions: “In other words, quality of care should remain innocent until proven guilty”. This is a hospital-centric admonition, but certainly not one that would be acceptable to most patients or to the regulators entrusted with ensuring the quality of their care. (2010, p.446)
Despite the disclaimers by Mohammed et al. (which were in effect discussed by Francis), it still raises the question: Could HSMRs even at the time of the Mid Staffordshire National Health Service (NHS) Foundation Trust debacle (2005–2009) not have shown trends over time, or hospital to hospital, as cause for worry? This could perhaps be considered as a middle way between Mohammed's purism and the view that Mid Staffordshire did indeed have 1200 deaths too many. It is possible to concede that HSMRs do not show the whole picture, but they do certainly show an important part. The Francis report into the failures at Mid Staffordshire has identified that there were three types of data at the time that should have signalled the issues. This could not have helped matters, in terms of Mid Staffordshire being authorised as a foundation trust (which are supposed to be models of best practice). The three warning signs should have come from HSMRs, CQC (or Healthcare Commission previously) data and Monitor data. The balance of hard verses soft data has evidently not been right, and the mechanisms for triangulating the three types of data need to be enhanced, if these failures are to be avoided.
Clearly, Mid Staffordshire has been an example where HSMRs increasing was not noticed quickly enough and resulted in far too many people dying. This deficiency still remains a wide concern, as there are 14 trusts that have been identified by the Department of Health as having over than expected death rates, and one of the main possible causes is medical staffing levels.
The NHS Commissioning Board has identified the trusts, following the publication of the Francis report, and they are as follows: North Cumbria University Hospitals, United Lincolnshire Hospitals, George Eliot Hospital, Buckinghamshire Healthcare, Northern Lincolnshire and Goole Hospitals, The Dudley Group of Hospitals, Sherwood Forest Hospitals, Medway, Burton Hospitals, Colchester Hospital, Tameside Hospital, Blackpool Teaching Hospitals, Basildon and Thurrock University Hospitals and East Lancashire Hospitals (2013, p.1).
Bearing the validity of the HSMRs in mind, the fact that HSMRs have deteriorated each year apart from 2010 is worrying. A 6% increase in this period (2005–2012) is troublesome for policymakers even when the potential failures of quantifying health quality are taken into consideration.
RO2: are providers with poor hospital standardised mortality ratios becoming worse?
Not significantly, ‘bad’ providers are becoming worse at a slightly greater rate than ‘good’ providers, but there is not really much difference. More concerning than this is the fact that they have both gone up and not down.
RO3: is there a statistically significant correlation between hospital standardised mortality ratios and Care Quality Commission data on quality?
There is no significant correlation between HSMRs and any of the four CQC quality related data sets. This means that one of the two variables is not accurately measuring what they are supposed to and portraying quality in a useful way.
This is based on the rationale that if safety and quality outcomes in hospitals are measured accurately, then this should correlate with the amount of people dying there compared with how many should die (if measured accurately). On the basis of the previous discussion on HSMRs, it is assumed that these do represent quality to some extent at least. If we accept that HSMRs at least ‘give an overview of mortality’, then the CQC data may not properly represent quality in their measurements.
The five key standards include safety, care, respect, staffing and management. It must be the case that these sorts of measures, if accurate, would correlate with HSMRs. They do not however at any level of significance. The CQC accident and emergency, and outpatient and inpatient surveys all do not correlate with HSMRs. These are national surveys, which are undertaken periodically over hundreds of locations. It again seems logical to assume that if these measurements accurately portray quality, they would correlate with the HSMR (even with the HSMR critique considered). If these things are not obtaining the information they are supposed to, information that is relevant on quality and relevant for the regulation of hospitals for policy makers, then different techniques for data collection in this area should be deployed. It is possible that the CQC relies too heavily on ‘soft’ data on quality such as surveys, observations and self-reported information. HSMRs are based on actual numbers of people dying; this can be considered ‘hard’ data on quality (criticisms on the expected death algorithm considered). If the soft data sets collected on quality do not accurately portray quality, this renders the whole process pointless. If quality is not measured with hard data, it seems inevitable that providers in England will deteriorate without the CQC noticing.
RO4: is there a significant correlation between hospital standardised mortality ratios and Monitor data on finances and governance?
All the logic seems to point to there being an inextricable link between finances and governance on the one hand and quality on the other. But on the basis of the data sets, it seems that neither governance nor financial scores positively or negatively correlate with HSMRs. In this case, the evidence on finances and governance is less likely to be unreliable. Many of the elements of the Monitor weightings on risk are quantifiable variables, which are not open to interpretation (as with some of the CQC and Dr Foster data). Returns surpluses and liquidity ratios, for example, are unlikely to be false. The lack of correlation between HSMRs, finances and governance shows that it is quite possible to be performing well financially and governance wise but be performing poorly HSMR wise. The converse is equally true.
It is perhaps disappointing that there is no correlation. If organisations that were performing well financially and are effectively governed performed favourably on their HSMR scores, then this would add clout to the argument that the work of Monitor was driving up standards. There is, however, another argument that the Monitor assessing regime is merely a box-ticking process. If we are to doubt the reliability of the Monitor data at quantifying finance and governance robustness, then perhaps, there is an inseparable link between quality and finance, as well as quality and governance, but this is not clear from the data because the nature of Monitor regulation is bureaucratic and does not measure what it is supposed to.
RO5: is there a significant correlation between Care Quality Commission data on quality and Monitor data on finances and governance?
There is not, as with the Monitor data and the HSMRs. The reasons discussed previously explain why the CQC data may not accurately quantify quality, which may explain the lack of correlation. But equally, it may be that quality does not actually correlate with either finances or governance.
RO6: to ascertain using linear regression whether it is possible to predict the values of the Care Quality Commission and Monitor data sets based on the hospital standardised mortality ratio data set.
The linear regression between the CQC outpatient surveys and the Dr Foster HSMRs was revealing in several ways. The model predicts that for an increase in the CQC outpatient survey of one unit, there will be an increase in HSMR by 7.3. This is very troublesome because it is the opposite of what would be anticipated to be the case.
It is logical to assume that a better score for quality based on outpatients' surveys would be a predictor for the number of people dying and that as the outpatient scores increase (get better), the amount of people dying will decrease (improve). The fact that the converse is true suggests that there are serious limiting factors with the methodologies deployed by the CQC in their outpatient surveys, and there may well be issues with the HSMRs also (as has been discussed already). The 95% confidence intervals do illustrate that the figure of 7.3 may not be entirely accurate, but with a range of −0.132 to 14.8, it is appropriate to suggest that the opposite of what should be occurring is almost certainly occurring: As scores for outpatients improve, scores for death rates worsen, seriously limiting the validity and usefulness of the data sets involved.
What does it all mean for the arm's length bodies and their effectiveness at positively influencing health policy?
The fact is there are reasons to doubt all attempts of quantifying quality in any capacity. There are important lessons to be learnt for the NHS in England as well as internationally. Many countries around the world deploy regulation of a similar nature in some capacity at least, and these countries should take note of the trends of the NHS in England. There are certainly many deficiencies associated with the current regulatory regime in English health policy, which have been confirmed by the lack of the data sets to correlate with one another. If arm's length bodies are not measuring what they are supposed to, the service is likely to suffer. There is an overreliance on soft rather than hard data, which limits the effectiveness of regulation. There is also too much box ticking, which has no value to health policy, and actually creates additional bureaucracy.
A tempting conclusion may be that the composite regime of health regulation in England is having a neutral effect on quality at best. There is approximately no evidence of any link between providers graded highly by regulators and the normalised deaths among their patients. Indeed, it could be argued that the cost of the regulation, both in the regulators and servicing their requirements in the providers, uses resources that could be better allocated elsewhere. But this may be too superficial a view. The existence of the regulatory bodies and the public visibility of their findings may be exerting an overall upwards pressure on service quality, without which the HSMR figures would have worsened even faster. This possibility cannot be assessed through the kind of analysis contained in this paper. The jury is still out.