High-quality health care maximises the benefit of evidence-based interventions while minimising harm. It is widely acknowledged that having the right systems in place to implement effective and safe care gives patients a better chance of achieving their desired outcomes. This truism appears to be the driver behind quality assessment methods currently in vogue, methods that place greater emphasis on structures and processes, while seeming to pay little heed to clinical outcomes. If a system is not working well, outcomes will not be good. It follows that assessment of quality needs to include clinical outcomes (including patient experience), as discrepancies between aspects of quality, particularly between process and outcome, are well documented and continue to be highlighted today.
Please cite this paper as: Draycott T, Sibanda T, Laxton C, Winter C, Mahmood T, Fox R. Quality improvement demands quality measurement. BJOG 2010;117:1571–1574.
Evidence for weakness of certification using process measures but not outcomes
Quality schemes using self-certification based on process measures alone should no longer be considered satisfactory. We need to know when we are not doing well, and certification based solely on process measures can mask poor clinical care. This has been highlighted by several events in the UK.
Recently, there were delays in identifying problems in at least two hospitals where mortality rates were contrary to the green status the units had been awarded, without any reference to measures of outcome.1 Assuming the structures and processes were in place (as certified), the failure of those two hospitals to achieve excellent outcomes, and for the assessment to miss those problems, is a cause for concern.
The Clinical Negligence Scheme for Trusts (CNST) is an insurance scheme in England and Wales that rewards maternity units that meet their strict criteria for risk management standards with a discount of between 10 and 30% on the insurance premium.2 CNST does not measure clinical outcomes per se, but relies on a certification based on process measures. Since its inception, there has been evidence of inconsistency between the CNST level attained and clinical outcomes: achieving a higher CNST level (a presumed indicator of a safer unit), does not relate to reduced payments on obstetric claims3,4 (Walker S, Personal communication, 2010).
In 2000, following recommendations by CNST, two similar UK units commenced training for shoulder dystocia. In subsequent analyses of their outcome data, one unit found an associated decrease in the brachial plexus injury rate (70% reduction),5 whereas the other unit had a 100% increase.6 During the same period, both of these units satisfied CNST with their risk management processes, including having training in place, and were awarded the same CNST level.
In a second example, a maternity unit was awarded level-2 CNST (which currently defines the 50% better performing maternity units in the UK), but a national review of maternity services, which assessed the service against a range of 25 indicators, rated the trust’s maternity service as falling within the ‘least well performing’ hospitals in England (a category that contained just 21% of all hospital groups).1
A way forward
Ten years ago there was a real need to encourage hospital groups to develop risk management. Today, there is a risk that maternity units will focus too heavily on process and be diverted from more meaningful quality improvement. To improve quality, and to sustain that improvement, we need to move the main emphasis away from process measurement, to include monitoring of clinical outcome as part of a standardised set of quality indicators. Outcome is a more direct measure of quality, and ultimately is what matters to patients. Poor outcomes should be a flag to alert units to examine their processes in more detail. Monitoring outcomes could have rapidly identified the adverse trends encountered by some of the units discussed, allowing them to scrutinise clinical care directly and undertake timely corrective action, thereby preventing further harm.
Which quality indicators?
Currently there are 290 maternity outcomes in 96 clinical categories published by just four professional bodies: Royal College of Obstetricians & Gynaecologists (RCOG), American College of Obstetricians & Gynecologists (ACOG), Royal Australian and New Zealand College of Obstetricians & Gynaecologists (RANZCOG) and Society of Obstetricians and Gynaecologists of Canada (SOGC). Within these documents there up to 18 definitions for a single quality indicator (e.g. postpartum haemorrhage), and some outcomes that cannot be improved by best care, for example, the incidence of shoulder dystocia. Clearly 290 outcomes and 18 definitions for a single condition are unusable, and rationalisation is required.
The RCOG and the UK’s National Health Service Connecting for Health (CfH) programme have recently produced a prototype set of intrapartum clinical outcomes that can be measured and are alterable by best practice, in line with the Good Indicators Guide.7 This could be used as a pilot national data set for intrapartum outcomes. Once a national data set has been established, it must be made easy to produce and use. The differing case mixes of units would need to be taken into account, so that each unit can act as its own control.
Experience of labour should be assessed, but it is difficult to measure and may be related to outcome. A number of national reviews have been completed, with variable results.8,9 A simple three-part patient perception score has recently been validated for use after instrumental delivery,10 and this may be sufficiently user-friendly for regular feedback.
Efficient data management
Measurement without rapid feedback and accurate interpretation for frontline staff is a wasteful activity. We need to measure performance in a way that can help inform and improve our ongoing practice in a timely way.
The Maternity Dashboard has been produced to present ongoing performance statistics for a maternity service over time.11 It utilises the traffic lights coding system (red, amber, green – RAG status) to alert users of the occurrence of, or changes in the frequencies of, selected quality indicators. As well as providing quality assurance for trust boards, external regulators and the public, it is an early warning tool. At present, dashboards often have to be populated manually, as the data cannot be easily retrieved from current maternity databases. Added to this is the lack of consensus on clinically useful thresholds to inform the RAG status.
A statistically informed dashboard is under development.12 Evaluation of an early prototype version demonstrated that prospective continuous monitoring of clinical outcomes using an automated system was feasible. It led to the detection of an otherwise unrecognised adverse trend, which was investigated and corrected by a training intervention, resulting in an improvement in performance.13
Connecting for Health and the RCOG have been collaborating on an automated solution using the standard maternity databases currently used in the UK. Any solution needs to be cheap, accessible to all, and easy to implement and use. More information is needed of the level of work generated by such monitoring schemes, and what resources they require.
Future role for insurance incentive schemes
Insurers also want the best possible outcomes for women and babies. Several units on both sides of the Atlantic have demonstrated improved patient outcomes after training, and one of the common characteristics of these units was a financial incentive to train, usually through a reduction in insurance premiums.14 Insurance incentivisation that focusses on the measurement of process rather than outcomes cannot provide the whole answer, however.
We should incentivise good performance and embrace more imaginative reward policies. One innovative method has been proposed by the medical insurer for the state of Victoria, Australia (VMIA): they have proposed setting aside some of the maternity income to reward the 20% best performing units, with this additional income being contingent on a commitment to share training, skills, ideas, etc., with those below them, as well as investing more resources in the least well performing units (Battilana C, Personal Communication, 2010).
We can only continuously improve the things that we can measure.15 To achieve this, we must make measurement easier, more timely and more understandable, in order to make rapid quality improvement feasible. We must rely less on the self-assessment of risk processes, and begin to prioritise what matters most: clinical outcomes and patient experience. We should collect and produce a standard, relevant set of quality indicators, ideally from our routinely collected data, and present these in a manner that facilitates ongoing quality improvement. Once issues are identified, we must be able to respond rapidly by undertaking further investigations and corrective action.
Monitoring quality must be a continuous process in order to detect and inform when performance changes. This way, we will be able to quickly identify processes that we may have incorporated with good intentions, but turn out to be ineffective or harmful. After all, much care in obstetrics still lacks a sound evidence base. It therefore remains questionable whether simply applying the processes in place (some developed from a poor evidence base) truly leads to the desired outcomes. Where good evidence exists, the correct application or use of one process or intervention in one small part of the dynamic process that is pregnancy and labour, does not preclude its misapplication elsewhere. Process guides implementation; outcomes define quality.
Insurers should take this into account when they tailor future incentive schemes in order to support and facilitate quality improvement.
Disclosure of interests
None of the authors have any relevant interests to disclose.
Contribution to authorship
TD, RF and TS conceived idea, co-authored and edited manuscript. TM, CW and CL co-authored the manuscript and approved final version.
Details of ethics approval
No ethical committee approval required.
There are no funding conflicts of interest.
We thank Dimitris Siassakos and Jo Crofts from the Research into Safety & Quality Group at Southmead Hospital for their help with the article.
Commentary ‘Quality improvement demands quality measurement’
Patient safety has become an increasing focus in maternity care. There are many reasons for this, including a greater appreciation of the harm that errors can cause to women, and medico-legal issues (Clark et al. Obstet Gynecol 2008;112:1279–83; Studdert et al. JAMA 2005;293:2609–17). Tim Draycott and his colleagues have been at the forefront of the developing safety culture (Draycott et al. BJOG 2006;113:177–82; Crofts et al. BJOG 2005;112:997–9). In the attached paper, they have highlighted the importance of measurement to improving care.
Two measures are particularly relevant to safety: process and outcome. Process measures quantify the degree of adherence to guidelines or protocols. An example is ‘the frequency with which mothers who deliver prior to 34 weeks have received antenatal corticosteroids’. Low adherence to evidence-based guidelines is a signal for organisations to investigate why, and how, adherence can be improved.
However, as Draycott et al. point out, measuring process alone cannot tell us if optimal outcomes are being achieved. Just because desired care pathways are followed does not guarantee that adverse events do not occur. Outcome measures (e.g. frequency of peripartum hysterectomy), on the other hand, provide direct evidence of whether the care provided is associated with fewer adverse events. Draycott et al. rightly note that over-reliance on process measures can be misleading and may not provide sufficient insight into the quality of care.
However, outcome measures alone may not always reflect patient safety better than process measures alone (Lilford et al. Lancet 2004;363:1147–54). First, an outcome measure may not reflect the quality of care. As Draycott et al. note, to reflect quality, an outcome must be able to be influenced by the care provided within an institution. For example, although preterm birth is an adverse event, the measure of preterm birth frequency has no relation to quality of intrapartum care because we do not have any effective interventions. Also, even when outcomes can be affected by the care within an institution, an outcome measure may be misleading if it is also strongly dependent upon patient factors. For example, if ‘frequency of haemorrhage’ were used as an outcome measure, without risk adjustment or stratification, institutions who care for women with a higher than average risk of haemorrhage (such as women with invasive placentation) may appear to be less safe if allowance for risk status is not made. Finally, even when poor outcomes are identified, the corrective actions needed may not be immediately obvious.
Pronovost et al. (Pronovost et al. Crit Care Clin 2005;21:1–19) noted that no single type of measure will be sufficiently comprehensive to reflect all aspects of patient safety and they compared the representation of safety with the representation of a person’s life as a collage. A single picture, or even several pictures, focusing on similar events would be insufficient. Understanding the safety of care within a system requires the measurement of a variety of process and outcome measures. Draycott et al. have explained why outcome measures are particularly crucial in the assessment of patient safety. As they acknowledge, further work needs to be done to establish a consensus regarding the outcome measures best suited to our needs.
Disclosure of interests
Dr Grobman is co-principal investigator of the NICHD study ‘Assessment of Perinatal Excellence’, which is designed to investigate measures of obstetric safety. No funding was provided in support of this manuscript.
Department of Obstetrics and Gynecology and the Institute for Healthcare Studies, Northwestern University Feinberg School of Medicine, Chicago, IL, USA