Valid and reliable measures of trauma system performance are needed to guide improvement activities, benchmarking and public reporting, future investment and research. Traditional measures of in-hospital mortality fail to take into account prehospital and posthospital care, recovery after discharge, and the nature and costs of long-term disability.
Drawing on recent systematic reviews, an overview was conducted of existing and emerging trauma care performance indicators. Changes in the nature and purpose of indicators were assessed.
Among a large number of existing, mostly locally developed performance indicators, only peer review of deaths has evidence of validity or reliability. The usefulness of the traditional performance measure of in-hospital mortality has been challenged. There is an emerging shift in focus from mortality to non-mortality outcomes, from hospital-based to long-term community-based outcome assessment, and from single measures of trauma centre performance to measures better suited to monitoring the performance of systems of care spanning the entire patient journey. As a result, a new generation of indicators is emerging that are both feasible and potentially more useful for commissioners and payers of population-based services.
Systems of trauma care are necessary to resuscitate, repair and rehabilitate injured people in order to reduce mortality, improve the quality of survival and lessen the overall burden of injury. Measurement and feedback of performance are integral to the concept of a system of care.
Realization that death due to suboptimal trauma care was preventable has led to the development of trauma systems and, in particular, the designation of major trauma centres in the USA, since the 1970s1. Preventable death rates and risk-adjusted mortality rates have since become standard methods of measuring and comparing trauma centre and trauma system performance. With improvements in trauma care, the mortality rate among severely injured people treated in mature trauma centres has declined from one in three to around one in ten, and preventable death rates have declined from one in three to less than one in ten, and in some centres as low as one in 502.
Inpatient mortality is an easy outcome to define and feasible to collect. However, as preventable mortality has diminished, so too has the usefulness of mortality as an indicator of performance; it is unlikely that future improvements in care will result in substantial further reductions in mortality.
Furthermore, inpatient mortality says little about the burden of injury that is felt both in and outside of hospitals, the cost of which is borne by patients, governments, insurers and the community. Although historically useful, mortality alone is no longer an adequate performance measure for a system of trauma care, nor is it acceptable to limit the measurement of the impact of an entire trauma system to that of a major trauma centre.
This overview highlights the main developments and challenges in measuring improvements in care in established trauma systems. It describes the limitations of inpatient mortality as a trauma care performance measure, and the paucity of validity and reliability evidence for existing structure, process and outcome indicators. Furthermore, the indicator requirements of various stakeholders are considered, ranging from patients and their families, to clinicians and managers and payers. In addition, an outline is presented of why it is important to pursue performance indicators that include quality of survival as well as survival, that go beyond the acute hospital episode to reflect the long-term results of acute and rehabilitation care and the stable long-term disability needs of the patient, and that reflect performance of important aspects of a trauma system before, during and after hospital care.
The purpose of performance indicators
Performance monitoring can serve several purposes, and appropriate indicators are best chosen according to the intended purpose (Fig.1).
Internal use for quality improvement
Driven by clinicians or by managers, performance indicators are useful within a system for monitoring outcomes or compliance with guidelines and particular aspects of care, and benchmarking performance over time. Prehospital time, rates of pulmonary emboli and compliance with venous thromboembolism prophylaxis are examples. Within a stable system these indicators do not always need risk adjustment.
External presentation for benchmarking purposes
Risk adjustment becomes particularly important when comparing centres or systems of care. Differences in patient case mix and delivery of trauma care, over time or across trauma systems, have an impact on the capacity to achieve valid benchmarking. Valid benchmarking requires detailed knowledge of factors that affect the outcome of interest (such as patient age, injury severity, co-morbid status), and adjustment for these factors in any analysis or presentation of data, to provide an accurate estimate of the association between trauma centre (or trauma system) and outcome.
Commissioners' and/or payers' use to determine future investment
In systems in which investment is tied to performance—whether through performance-based payments or funders aiming to get value for money more generally—indicators need to be chosen carefully to reflect important care goals. Although performance-based payments can be useful to elicit change, undesired and unexpected effects can result and should also be monitored.
Particularly when standardized in format and data collection, indicators can enable higher-quality comparative studies of interventions such as compensation systems that are unsuitable for randomization.
Approaches to measuring quality and performance
To examine trauma system performance in providing quality care to injured patients, a conceptual model such as Donabedian's framework of quality of care is helpful. In Donabedian's framework, the three components of healthcare quality are structure, process and outcome3. Their relationship is depicted in Fig.2. The structure is the environment in which healthcare is provided, and includes material and health resources, operational factors, and organizational characteristics of the healthcare facility. For example, designation of trauma centres, trauma centre volume, prehospital notification and presence of a quality improvement programme have been identified as independent determinants of Canadian trauma centre in-hospital mortality4. The process is the method by which healthcare is provided, and includes the giving and receiving of care by the providers and healthcare system. The outcome is the consequence of healthcare, and includes the health status of patients and communities. Prehospital, hospital, posthospital and prevention performance indicators can be chosen to capture the spectrum of care from the moment of injury to recovery; examples are shown in Table1.
Table 1. Conceptual model of performance measurement in trauma care
Phase of care
Table shows examples of performance measures. Some measures, such as pain, may cut across all domains. For example, a protocol for pain scoring and management is a prehospital, hospital and posthospital structural indicator; compliance with protocol is a process measure; and pain score at 6 months after discharge is an outcome measure. GOS-E, Glasgow Outcome Scale–extended.
Total prehospital time with component parts
Massive transfusion protocol
Appropriate activation of massive transfusion protocol
Deaths due to haemorrhagic shock
Head injury protocol
Risk-adjusted mortality for head injury
GOS-E at 6 and 12 months after injury
Proportion of eligible patients receiving protocol-based management
Rehabilitation facility in community
Time to rehabilitation consultations
Return to work, adjusted for severity
Standardized rehabilitation protocols
Injury prevention activities
Proportion of eligible patients receiving alcohol screening and brief intervention
Proportion of patients returning with new alcohol- or drug-related injuries
A very large number of such indicators have been defined and used in trauma care around the world. A scoping review identified 1572 indicators that could be classified into eight categories: non-American College of Surgeons Committee on Trauma (ACS-COT) audit filters (42·0 per cent), ACS-COT audit filters (19·1 per cent), patient safety indicators (13·2 per cent), trauma centre/system criteria (10·2 per cent), indicators measuring or benchmarking outcomes of care (7·4 per cent), peer review (5·5 per cent), general audit measures (1·8 per cent), and guideline availability or adherence (0·8 per cent). Measures of prehospital and hospital processes (60·4 per cent) and outcomes (22·8 per cent) were the most common, and posthospital and secondary injury prevention indicators accounted for less than 5 per cent5.
Based on appraisal of the validity and reliability of identified indicators6, 7, strengths and weaknesses of existing performance measurement are apparent. First, a common set of clearly defined, evidence-based and broadly accepted performance measures for evaluating the quality of trauma care does not exist. Instead, many different indicators are used. It is not possible to have a systematic data-driven performance measurement and feedback system if there are no generally agreed measures. Second, originating from surgical audit, quality improvement efforts in trauma care have historically been institution-specific. Performance measures should also serve state or national goals. Third, current performance measures have been developed largely to address the measurement needs of providers, but have failed to capture patient care along the journey from injury to recovery. For example, most published performance indicators examine prehospital and hospital processes and outcomes of care. There has been limited development of measures to examine posthospital care or injury prevention.
Further, there is little evidence to support the use of most current performance indicators in trauma care. For example, only a single measure—peer review for preventable death—has both reliability and validity evidence7. In paediatrics, even less evidence is available, with most measures originating from the adult literature and subsequently applied to paediatric care6. Clinicians caring for patients with traumatic injuries do not have the same evidence base to draw upon in developing quality indicators.
Finally, developing performance indicators for diseases or disorders that are homogeneous is easier than for those that are heterogeneous. Trauma is not a disease, but a collection of injuries; performance indicators may therefore need to focus on common pathways and outcomes that are important to patients.
Choosing important, usable, scientifically sound and feasible indicators
Through consultation with developers and users, for the Agency for Health Research and Quality, Hussey and colleagues8 identified four broad attributes for evaluating performance indicators: importance, usability, scientific soundness and feasibility. Important indicators are those that are relevant to stakeholders, and show potential for improvement. The structures and processes measured should be those that contribute, or are likely to contribute, to desired outcomes. Things that make indicators usable, for example, include standardized templates and clear guidance for public reporting or pay-for-performance programmes. Scientifically sound indicators have a strong evidence base, are reliable and valid, allow for patient/consumer perspectives, and together form a comprehensive evaluation of the quality of trauma care. Feasible indicators are those for which clearly specified numerator and denominator data are readily available from administrative data, trauma registries, linked databases or feasible purpose-designed data sources.
In-hospital mortality is a feasible and reliable outcome, but there are significant problems with interpreting this outcome for hospital and system benchmarking purposes. Trauma deaths may occur before reaching hospital, during interhospital transfer or during convalescence from the acute hospital stay; therefore, in-hospital deaths represent only part of the overall picture. Measurements of severity of injury are not coded in a standard way, the whole spectrum of Abbreviated Injury Scale versions are used, coding may be translated from International Classification of Diseases data sets, and training and auditing are inconsistent. Attempts at risk adjustment using the Trauma and Injury Severity Score and other scores have been shown to be unreliable9, 10 because of the coarse categorization of risk (for example age over 55 years) and inconsistent attempts at managing missing data11. If in-hospital mortality is to be used as a performance measure for benchmarking, international agreement is needed on standardization of reporting12.
Moving from mortality to non-mortality outcomes
The burden of traumatic injury is multidimensional; Lyons and co-workers13 have described 20 groups of injury consequences: 12 related to the individual, three relevant to family and close friends, and five to the wider society (Table2). Although consensus statements exist, the recommendations for selection of instruments for measuring important outcomes and time points differ, and the relevance to system performance and quality improvement has not been addressed14–16.
Table 2. List Of All Deficits (LOAD) framework of injury consequences13
Death, including fetal
Pain and discomfort
Reduced short-term physical activity
Long-term physical disability
Development of secondary conditions
Behavioural change and secondary health loss
Fear of repeated injury
Decreased quality of life
Societal fear of injury
Psychological consequences in observers
Direct medical costs
Function and quality of life
Although arguments have been made that functional and quality-of-life (QoL) outcomes should have a higher priority than mortality17, the uptake of routine collection of these outcomes by trauma registries and trauma systems has been poor. The Functional Capacity Index remains the only disease-specific measure of function developed for a general trauma context18, 19, but its validity for predicting long-term outcome has been questioned20 and none of the consensus statements has recommended its inclusion in injury outcome studies14–16. The Functional Independence Measure (FIM), commonly used in rehabilitation settings, may be a useful measure of function at hospital discharge21, but lacks the responsiveness to change of more comprehensive measures of function such as the Glasgow Outcome Scale (GOS) and its extended version (GOS-E)22. The GOS scales consider higher levels of functioning and participation, such as community participation, transport, return to work, relationships, and participation in social and leisure activities, factors important to patients, caregivers and society that are not included in the FIM.
Generic measures of health status or health-related QoL are recommended for use in trauma populations16, 23. These measures enable comparison of functioning of injured patients with that of other disease groups and the general population. There are numerous instruments available, including the 12- and 36-item Short Form Health Surveys (SF-12® and SF-36®; QualityMetric, Lincoln, Rhode Island, USA), the EQ-5DTM (EuroQol Group, Rotterdam, The Netherlands), the Quality of Wellbeing Scale and the Health Utilities Index. For each instrument, societal and cultural differences, and the availability of local population norms may limit comparisons. For example, the consensus statement by van Beeck and colleagues24 recommended the use of the EQ-5DTM for injury outcome studies, but its utility is unknown in populations that lack appropriate population norms. In contrast, population norms for most countries are available for the SF-36® and SF-12® instruments, although they may not cover particular population subgroups, including those in which injury is more prevalent25. The purpose of benchmarking should be considered before selecting generic health status measures and, in particular, which groups will be compared and how comparisons will be made.
Benchmarking of trauma centre or system performance on non-mortality outcomes has been limited to short-term morbidity (such as complications) and function at discharge26–28. Function at discharge will vary widely depending on the instrument chosen. Instruments performing well at discharge provide limited capacity to measure change and recovery21, 22. Studies using function at discharge to evaluate quality of care should be interpreted cautiously given the wide variation in discharge practices, availability of rehabilitation beds and patient case mix21. Benchmarking of functional or health-related QoL outcomes should be performed at standard time points after injury, rather than at poorly standardized time points such as hospital or rehabilitation discharge, as length of stay varies considerably.
Moving from short-term to long-term outcome measurement
The use of inpatient outcome measures is inadequate for several reasons. First, severely injured patients rarely achieve a steady state by the time they are discharged from hospital, and status at discharge fails to take into account the care provided and improvements made after discharge from an acute hospital. The limited available data to date suggest that improvement has slowed and most patients have reached a steady state by 12 months after injury29–31.
Second, inpatient mortality and function at discharge are affected by discharge practices, availability of rehabilitation beds and patient case mix21. Mortality over 1 year after injury is likely to be more meaningful than inpatient mortality. For example, among 124 421 patients treated in a mature trauma system and coded in the Washington State Trauma Registry between 1995 and 2008, 5·8 per cent died during the initial hospital admission; however, by 1, 2 and 3 years, 9·8, 13·2 and 16·0 per cent respectively had died32. This represented excess postinjury mortality at 3 years of 10 per cent above age- and sex-adjusted norms. Furthermore, although the in-hospital mortality rate decreased from 8 to 4·5 per cent between 1995 and 2008, mortality after discharge increased commensurately, such that the overall 1-year mortality rate after injury was unchanged.
A third reason for extending the period of outcome measurement is that inpatient mortality correlates poorly with costs to payers. The least costly patients are those who recover well and regain independence, and those who die early. The most expensive are those who survive but have permanent severe disabilities, do not return to work and require prolonged attendant care. Systems that compensate for the healthcare costs, and lifetime healthcare and disability costs of severely injured people have found that the majority of long-term liabilities relate to the disability care of permanently incapacitated clients33.
Numerous studies have described the long-term functional and health-related QoL outcomes following trauma for individual cohorts34. Mackenzie and co-workers35 compared trauma centre care with non-trauma centre care using 12-month functional outcomes. Yet few jurisdictions routinely use such measures for monitoring performance and improvement. Questions of feasibility of collecting this information have been raised, such as the selection of instruments for follow-up and the potential for bias related to loss to follow-up. However, these challenges should not be considered insurmountable36.
Recently, Gabbe and colleagues37 described the development and implementation of population monitoring of functional and QoL outcomes at 6, 12 and 24 months after major trauma through the Victorian State Trauma Registry (VSTR). The key elements of the VSTR methodology are listed in Table3.
Table 3. Key elements of a successful registry that includes long-term functional and quality-of-life outcomes of injured patients
Characteristics derived from the Victorian State Trauma Registry37. QoL, quality of life.
An opt-out consent process whereby all eligible patients are included, and sent a letter and brochure describing the registry's purpose, that patients will be followed up by telephone, and details of how to opt-out of the registry
Collection of both patient and next-of-kin contact details
Telephone interview of patients or their next of kin, rather than mailing self-administered questionnaires, to limit loss to follow-up
Brief (approximately 10–15 min) follow-up interview including functional and health-related QoL measures, and a proxy version in which health-related QoL measures are excluded
Centralization of telephone interviewing to ensure supervision and quality control of interviews
Telephone interviewers with patient contact and/or clinical experience
The VSTR approach has been implemented at relatively low cost (approximately 30 000 Australian dollars per time point for 2000 patients who suffered major trauma), low opt-out rates (less than 1 per cent) and high follow-up rates (about 85 per cent)37. Importantly, across the system, there is early evidence of improvement in patient function over time, as well as continued reductions in mortality, reinforcing the importance of benchmarking on outcomes other than mortality38. The experience of the VSTR has shown that the collection of long-term functional and QoL outcomes across a trauma system is feasible, relatively inexpensive and useful, if investment is made in the integration of such monitoring into the trauma system.
Moving from hospital performance to system performance
Measuring performance in one acute hospital involving multiple services (for example emergency, intensive care, surgery, anaesthetics) is difficult. Measuring and monitoring processes and outcomes across a system of care that involves prehospital care, acute care, rehabilitation and community services is even more complicated. However, to assess truly whether outcomes for injured patients are improving over time and where the opportunities for improvement lie, systemwide monitoring is essential. This can happen effectively only with linkage of databases containing crash data, ambulance service data, administrative and discharge data from hospitals within the region, data on rehabilitation services, coroner findings and insurance data. This information should be linked with registry data that include validated long-term disability outcome measures.
Once a system monitoring process is in place, it is possible to measure the effects of key processes such as trauma triage, transfer protocols, new resuscitation procedures, compensation payments and rehabilitation. Given the cost of providing this care and the potential improvements in outcome, the cost of systemwide monitoring is relatively low. It is important that there is proper analysis and governance of this data collection process to ensure that the data drive system change. Carefully selected indicators show promise in the goal of system-level monitoring. Examples include composite indicators, such as time to haemorrhage control, that bring together several elements of the patient journey and allow for various context-specific ways of achieving the same goal.
In established trauma systems the need to pursue developments in quality indicators is pressing; without doing so, the effects of quality improvement initiatives on long-term patient outcomes, the requirements of health and other services, and costs to the system will remain unknown. Settings with reasonable quality data sources and sufficient resources for follow-up should aim to develop the capacity to collect long-term outcome data on survivors, link hospital data to other data sets, and work with their jurisdictions to determine key indicators and how they will be used.
An urgent international effort is required to agree on standard indicators and risk adjustment techniques. A suite of structure, process and outcome indicators is needed that can facilitate system performance monitoring and benchmarking everywhere. Efforts are under way to foster such agreement.
As highlighted, there is also a need to improve the science behind the development, validation and use of indicators. It is no longer acceptable for institutions to rely solely on locally developed indicators that have not been subjected to rigorous validity and reliability testing. Although structure and process indicators are very useful in understanding key aspects of care and in their more rapid turnaround compared with long-term outcome measures, their relationship to important outcomes must be defined and validated. For the measurement of system performance there may be a place for development and testing of composite process indicators, such as time to haemorrhage control for patients with haemorrhagic shock, or compliance with condition-specific best practice care bundles.
In jurisdictions without developed trauma systems or established hospital-based data systems, performance improvement will rely on morbidity and mortality review, limited audit of specific evidence-based processes, and peer review of preventable deaths and other adverse events. Such methods are globally applicable trauma quality improvement techniques that can be instituted in even the least developed settings39. Excellent examples exist of the use of these techniques to begin effective quality improvement programmes in low- and middle-income countries40, 41, although they give little or no opportunity for benchmarking of performance within or between centres and systems of care. The use of local preventability audits should, therefore, not preclude efforts to establish a trauma registry that links with population-based injury data, and can capture prehospital information and postdischarge outcomes. This type of measurement requires concerted efforts at city, state and national levels, and cannot be left solely to individual acute care facilities, although they may be initiators and drivers of the process.
With a standardized international approach to the development of well constructed registries, a system of meaningful and useful performance indicators that facilitate comparison, improvement and resource allocation decisions is possible. Doing so will also help to build an evidence base about effective ways of improving trauma care given that many potential systemwide interventions are unsuitable for randomized trials, and instead rely on valid comparisons from high-quality registry data.
R.L.G. is supported by a Career Development Award from the National Health and Medical Research Council. B.J.G. is supported by a Postgraduate Award from the National Health and Medical Research Council. H.T.S. is supported by a New Investigator Award from the Canadian Institutes of Health Research. P.A.C. is supported by a Practitioner Fellowship Award from the National Health and Medical Research Council.
Disclosure: The authors declare no conflict of interest.