Developing a process to measure actual harm from medication errors in paediatric inpatients: From design to implementation

The potential harm associated with medication errors is widely reported, but data on actual harm are limited. When actual harm has been measured, assessment processes are often poorly described, limiting their ability to be reproduced by other studies. Our aim was to design and implement a new process to assess actual harm resulting from medication errors in paediatric inpatient care.

goal to reduce severe, avoidable harm related to medications by 50% in 5 years, there is a lack of consistent data on the incidence of actual harm from medication errors in clinical care, and particularly in a paediatric setting. 1,2Methods to identify errors vary widely, ranging from retrospective medical record reviews to the use of trigger tool methods, which impact the type of errors detected. 3,4ce errors are identified, there are extensive data on the classification and frequency of different error types, and several scales for measuring potential harm. 2 Whether such errors reach patients and cause actual harm in children is less well known. 5Various tools have been used, including the National Coordinating Council on Medical Error Reporting and Prevention (NCC-MERP) in the United States, 6 which apply different definitions making it difficult to compare the incidence and impact of medication errors in clinical care across different studies. 7,8 additional challenge is understanding the impact of medication errors on patients and their families.The National Cancer Council in the United States compiled a severity scale (Common Terminology Criteria for Adverse Events, 'CTCAE') for measuring medication adverse events using clinician-based severity assessments, 9 with patient-reported outcomes available for children aged seven and above. 10However, the CTCAE is focused on harms resulting from adverse events in oncology trials and these are not necessarily the equivalent to harms resulting from medication errors. 113][14] In order to develop an economic evaluation model, we wanted to measure not only medication errors, but the actual harm from errors and the impact of these harms on patients, their families, 15 clinical staff and the health system.Our aim for this paper was to report on the design and application of a new process to assess the incidence and severity of actual harm resulting from medication errors in paediatric inpatient care.

| METHODS
We developed and implemented a systematic process to identify errors, assign a level of potential harm severity, determine whether actual harm occurred, and the severity of that harm.The study used patient record data relating to patients admitted to nine wards at an acute tertiary paediatric hospital in New South Wales (NSW), Australia. 12We conducted retrospective chart reviews to identify prescribing error data, 14 and used direct observational methods to obtain medication administration error data. 13

| Definitions of harm
Our harm assessment process used the following definitions adapted from the NCC-MERP 6

| Identifying and classifying medication errors
3][14] For the prescribing errors, a team of clinical pharmacists retrospectively reviewed all medication orders prescribed on the study wards during the study period.The pharmacists classified the errors as clinical (e.g.incorrect medication) and procedural (e.g.use of incorrect abbreviation) errors. 14For

What is already known about this subject
• The data on harm from medication errors largely reflect potential harm to the patient.
• Methods for identifying actual harm are often not clearly described and therefore difficult to replicate.

What this study adds
• We describe a detailed process using multidisciplinary panels to assess actual harm.
• Our structured case study approach allows systematic consideration of clinical and therapeutic issues that impact harms.
• Differences between potential and actual harm highlight the importance of measuring both outcomes to reduce errors that cause the most actual harm in hospitalized children.
medication administration errors, a team of research nurses used a purpose-designed observation tool to record (by direct observation) the details of medication preparation and administration on the study ward. 16The observational medication administration data were reviewed then compared to the documented medication orders in patients' records to determine whether an error had occurred (i.e.whether the administered medications matched the patient's medication orders, manufacturers' instructions and hospital policies).The teams developed error classification tools for both the prescribing and administration errors (see Table S1) and undertook inter-rater reliability testing to ensure consistency in error identification and classification.The pharmacist reviewers obtained a minimum Cohen's kappa score of 0.77 on error type and 0.76 on severity level before they commenced independent data collection. 12-14

| Information sources used to identify medication errors
Medication use in paediatric populations does not have the same evidence base as in adult populations.We used a variety of sources to establish the accepted medication dose ranges, and hospital guidelines for approved administration processes.The main reference sources used by the teams were accessed through the Clinical Information Access Portal (http://www.clininfo.health.nsw.gov.au).This website is available to clinical staff at NSW public hospitals through the hospital intranet and comprises a wide range of evidence-based point-of-care clinical information and support including: internal guidelines (Meds4Kids), the Paediatric Injectable Handbook, network policies and procedures, and the Australian Medicines Handbook Children's Dosing Companion (AMH-CDC). 17Other specific sources included the British National Formulary for Children 18 and the Australian Injectable Drugs Handbook. 19

| Rating errors for potential harm
After identifying and categorizing both prescribing and administration errors, the pharmacist and nursing teams rated the potential harm associated with each medication error.We used a modified version of the NCC-MERP by condensing the number of severity categories, from nine to five (Table 1) for ease of classification. 6All errors were classified according to this modified potential harm severity scale.Given the large number of errors identified, we focused on assessing actual harm in those errors with a high 'potential' harm severity score.We aggregated all errors with a potential harm severity rating of 'Temporary harm requiring intervention', 'Permanent harm requiring intervention', and 'Potential death' (see Table 1) and labelled these as errors with 'serious potential harm'.

| Developing the harm assessment process
We convened a working group to design a process for identifying and categorizing actual harm from medication errors.The working group comprised paediatricians, pharmacists, clinical pharmacologists, nursing staff, clinical governance executives and other research team members.The group determined that a structured case study approach should be used to review the evidence of harm, 20 and that a multidisciplinary panel would be most appropriate to assess actual harm and would include a paediatrician, a nurse, and either a pharmacist or clinical pharmacologist/therapeutics specialist with a working knowledge of paediatric medication processes and contemporary practices on paediatric wards.

| Guidelines for determining which errors would be sent to panel review
Criteria to determine which prescribing and administration errors were to be reviewed by the harm assessment panels were prepared (see Table S2).Initial criteria included whether errors: were rated as serious potential harm (see Table 1); were administered or prescribed on the study wards during the study period 12 ; and reached the A-D

Temporary harm requiring monitoring
An error occurred which has the potential to cause temporary harm to a patient and may require monitoring. E-F

Temporary harm requiring intervention b
An error occurred which has the potential to cause temporary harm to a patient and would require an intervention. E-F

Permanent harm requiring intervention b
An error occurred which has the potential to cause permanent harm to a patient and would require intervention. G-H

Potential death b
An error occurred which has the potential to result in the patient's death or would require intervention that is necessary to sustain life.patient.For example, a wrong dose error was considered to have reached the patient when one or more doses of the order were signed as administered or observed to have been administered by the research team.The most common reason for errors not being presented to the harm assessment panel was that the wide therapeutic dose range of the drug meant that even if the dose errors were outside our 10% error range, the errors would not necessarily result in harm.These medications were not included if the daily dose was within the daily dose limits and guidelines approved by an expert group of clinical and academic pharmacists, clinical pharmacologists and other clinicians.Medications that fell into this category mainly comprised opioid analgesics (56%) and paracetamol (33%).The next most common reason for case studies not being reviewed was a duplicate order where a patient was prescribed a regular dose of the drug concurrently with an order for the same drug to be 'given as required' (prn), leading to potential double dosing of the drug.These errors were not sent to the panel where careful review of the medical records identified that the patient did not receive additional doses.
The most common medications in this category were oxycodone and paracetamol.For administrations, case studies involving drugs being administered orally where the patient had a nasogastric, gastric or jejunal tube, were excluded from panel review where the patients were taking oral solids and liquids and there was no evidence of a neurological deficit or dysphagia that would create a risk of aspiration pneumonia.

| Tools for assessing actual harm from medication errors
The research team reviewed existing tools to determine which classification would provide the most consistent and replicable results for assessing actual harm severity and the level of plausibility that the error was linked to the harm.We piloted the assessment process with the working group and reviewed the NCC-MERP, 6 the Naranjo Scale 21 and the World Health Organization Uppsala Monitoring Centre (WHO-UMC) tools. 22The working group found the NCC-MERP difficult to apply as it rates any permanent harm, however mild, above any temporary harm.The working group also found that although the Naranjo Scale and WHO-UMC tools both provided a good assessment of causality and plausibility, many of the metrics collected were not relevant to medication errors, which meant comparisons with previous studies using these scales would be difficult.The harm severity score developed by Dean and Barber used multidisciplinary clinician reviews, but, although validated using actual harms, was designed to assess potential harm and did not include plausibility or confidence measures. 23The 5-point scale developed by Overhage and Lukes was also reviewed, but, although this method contains a detailed description of the harms studied, it does not address clinical aspects of actual harm. 24us, the research team developed the Harm Assessment Associated with Medication Errors Classification (HAMEC) and included a plausibility scale to link the error with the harm, and a confidence level for when the panellists determined that no harm had occurred. 8

| Preparing case studies for panel review
To ensure a systematic approach to extracting clinical information from medical records, we designed a purpose-built tool to detail the possible harms resulting from errors leading to, for example, higher or  2).Each guide was reviewed by a paediatrician with specialist knowledge of the medical conditions underlying standard use of the drug, such as for pain (for paracetamol and opioid error PHAME guides) or infection (for gentamicin and other antibiotics).The guides were approved by the working group and provided a checklist for identifying clinical and/or laboratory features that could be expected in the event of harm from the associated medication error (see Table S3).
Researchers with clinical backgrounds prepared the case studies (see Table S4).The team used a combination of clinical experience and the PHAME guides to review the medical record for each patient with a medication error that met the criteria for being presented to the panel.To ensure presentation consistency, we followed the PHAME guide structure, including the relevant timings and components for identifying harm relating to the errors.
T A B L E 2 Data items extracted from patients' records for case studies.
• De-identified demographic data: age, gestational age (no actual dates or ward data were included), height, weight and length of stay.
• Reason for current admission.
• Current diagnosis and a summary of the patient's past medical history.
• A summary of the error(s) identified as having a serious potential harm.
• Evidence as to whether the errors had been detected during admission.
• Relevant symptoms expressed by the patient or parent.
• The clinical signs noted by the care team and whether these signs were noted during a clinical review or rapid response unit.
• Medications that were given or withdrawn.
• Tests relating to the error for monitoring, diagnosis or treatment.
• Actions by the care team; additional actions that would normally have been carried out if the drug event had been detect.
• Ιtems in the care record, such as incident reporting, disclosure with patient and family, clinical reviews, and any changes to the care record.
To ensure we captured the full extent of a patient's exposure to errors we presented all case studies relating to either prescribing orders or administrations that occurred within the same admission.
For example, looking at each error separately would miss the cumulative effect of a wrong dose over several days, and reviewing each administration case study separately would miss an ongoing issue with incorrect preparation methods.This approach also meant that the clinical picture of the patient would only need to be reviewed once by the panel members.

| Identifying actual harm using multidisciplinary review panels
The decision tree used during the harm review panels for determining actual harm is outlined in Figure 1.The panels comprised the panel chair, the multidisciplinary panellists, and at least one member of the research team whose main task was to record decisions, note when discussions occurred during the harm assessment process, and provide additional details from the patients' records as required.The panel chair (a researcher with a paediatric clinical background) was responsible for ensuring a consistent process occurred with each panel and in reaching consensus decisions.For each meeting, the panel chair would summarize the process to be followed during the panel and introduce the panellists and other team members.
Panellists were also advised they could request further details from the patients' medical records at any time.
The panel chair provided a summary of the case study and then asked the panellists to determine whether there was any evidence that the patient experienced any harms that would be expected from the errors described in the case study.If they determined this was the case, the panel chair asked them to describe the harms (e.g.respiratory issues or nausea) and assess the severity of actual harm on a four-point scale: minor, moderate, serious or severe (see Table S5). 8Panellists were then asked to determine whether there was a plausible link between the harm and the error (certain, probable, possible or unlikely).If the panel determined no harm had occurred, they were asked to assess how confident they were in that decision (very, moderately, somewhat or slightly confident).Panellists were given a third option 'un-assessable' if there was not enough information in a patient's medical record to assess the presence or absence of harm (see Figure 1 and Table S4).Research team members recorded whether there was immediate consensus or whether the panel discussed the cases during the assessment process.Panel members were aware that they were reviewing cases with serious potential harm but were not told which category of potential harm severity the errors related to.
During pilot testing, our working group determined that presenting the cases only during the panel meetings created limited time for discussion; we therefore sent a proportion of cases to the panellists F I G U R E 1 Decision tree for assessing harm. 8or independent review and rating ahead of the panel.Cases where the panellists were in independent agreement were not discussed at length during the panel meetings except to determine if additional data for measuring the impact of harm on the health system was required.

| Additional information collected
Errors have an impact on the health system in addition to the direct and indirect impact on patients and their families.The panellists were asked to identify additional information about actions resulting from the error, such as filing an incident report, whether a family or clinical case conference was held, additional outpatient care and changes to expected length of stay, as these can be used in evaluating impacts and costs of harm from medication errors. 15,25

| Ethical considerations
The research was approved by the Human Research Ethics Committee of the Sydney Children's Hospitals Network (HREC/15/ SCHN/370) and by Macquarie University.We were granted a waiver of individual consent as we were using retrospective record reviews.
All nurses observed during the administration data collection were consented and de-identified.Case studies presented to harm panels did not include any patient identifiers such as name, date of birth or medical record number.
Researchers observing nurses administering medications did not view patient records at the time and generally could not assess whether an error was made in real time.However, we developed a Serious Error Protocol and algorithm (see Table S6) to guide the nurse observers collecting medication administration data on actions to take should they observe an administration error which was potentially dangerous.Further details of these methods have been published elsewhere. 26

| RESULTS
A total of 28 panel meetings were held, each lasting between 1.5 and 2.5 h, reviewing an average of 20 case studies per meeting.

| Prescribing case studies prepared for the harm assessment panels
The research team reviewed 26 369 orders and identified 19 692 errors, of which 1473 (7.5% of errors) were categorized as leading to serious potential harm, including 308 procedural errors and 1165 clinical errors (see Figure 2). 14Further assessment showed that 1095 errors did not meet the criteria for panel review (see Table S2).The panel reviewed the remaining 173 case studies, comprising 378 errors.

| Outcomes of the prescribing harm assessment panels
The multidisciplinary panels determined that 22 case studies (out of 173) were associated with harm including 21 related to clinical prescribing errors and one to procedural error.The panel was not able to assess six case studies due to a lack of relevant information in the patients' medical records (see Figure 2) and found no evidence of harm in 145 case studies.There was one case of serious actual harm (using our HAMEC tool) 8 and no cases of severe harm, the highest severity rating (see Table 3). 8e most common error types in the case studies reviewed by the panel involved wrong route (31%), wrong dose (22%) and dose unverified (17%).This last category includes the off-label use of medications that have only been explicitly approved for use in the adult population.The errors reviewed by the panels involved 125 different medications, with paracetamol and midazolam as the most common single medications (n = 17, 6% each).The most frequent drug groups involved in errors were antibiotics (n = 75, 19%) and anticonvulsants/ sedatives (n = 52, 14%).

| Medication administration case studies prepared for harm assessment panels
The research nurses observed 5137 administrations and identified a total of 3663 errors (1899 administrations) with 1105 errors (489 administrations) rated as serious potential harm (see Figure 2).
The team prepared 393 case studies (comprising 525 errors) for each administration with serious potential harm that met the criteria of: dose levels greater or less than 10% of the prescribed dose, errors in preparation or delivery/route and/or were administered at a rate 15% above or below the recommended rate (see Table S2).These guidelines were based on a systematic review and approved by the working group. 27

| Administration harm assessment panels
Harm from medication administrations was found in 67 case studies involving 56 patients.The average number of errors per case study was 1.3 (median = 1, 95% confidence interval [CI] 1.27-1.40)with a maximum of five errors (see Table 4).The three most frequent error types included: intravenous (IV) medications being administered more rapidly than recommended (33%) and where doses given were either lower (21%) or higher (16%) than the prescribed dose.The most common classes of drugs involved were antibiotics (36%) and the most common single drug was paracetamol (6% of all cases).The panel found evidence of serious actual harm in seven case studies but none with severe actual harm.

| Panel discussion metrics
A member of the research team attending the panels recorded whether panel members discussed their findings at different stages of the harm assessment process (see Table 5) including whether harm occurred, harm severity, plausibility and confidence of no harm (see Figure 1).Overall, discussions regarding whether harm occurred were held more frequently for administration errors (37% of case studies reviewed) than for prescribing errors (22% of case studies).For cases of actual harm, discussions of whether harm occurred were noted in 41% (9/22) of prescribing case studies and in 60% (40/67) of administration case studies.Where no harm was determined, a discussion over whether harm had occurred was noted in 17% for prescribing case studies (n = 24) and 33% of administration case studies (n = 105); however, the most common discussion point for cases with no harm related to determining the confidence level that no harm occurred (prescribing errors = 68 [47%], administration errors = 238 [74%]).For the prescribing case study assessed as involving serious actual harm, the panel discussed whether harm occurred and the harm severity level but no discussion was recorded for plausibility as all F I G U R E 2 Summary of errors and harms.three assessors were in agreement and certain that the harm was related to the error.For the seven administration case studies with serious harm, discussions were held at different points of the process with the most common discussion (six out of seven case studies) relating to harm severity.

| Comparing potential and actual harm assessments
Our review of potential vs. actual harm outcomes highlights the very different results between the two assessments (see Table 6).For example, the panels did not find evidence of actual harm in 328 prescribing errors and 436 administration errors where serious potential harm had been identified by the pharmacist and nursing research teams.In addition, five prescribing errors were rated at the highest level of potential harm (i.e.potential death) but were deemed by the panel to have not caused actual harm to the patient.In one case of serious actual harm, the potential harm rating was lower.Two of the cases with higher potential than actual harm ratings related to 1000 times dosing errors where the prescriber had written mg (milligrams) instead of the correct dose of mcg (micrograms).Although the errors would potentially have been extremely serious, in both cases the panel felt confident that the errors did not reach the patient.In one case the medication was uncommon and would have required several times the hospitals' entire supply, making it unlikely that administration had occurred.In addition, routine blood level tests indicated the correct dose was given, and there was written clarification that the dose had been given according to protocol which stated the dose in micrograms.
The administration data showed a different picture to the prescribing data with six errors that were given a potential harm rating of 'Temporary harm requiring an intervention' (the lowest of our serious potential harm ratings) but were given a serious actual harm rating by the panel.In one case the drug was administered too quickly and resulted in the patient's intravenous drip needing to be re-sited.The attending doctors were not able to do this on the ward, despite several attempts, and the child had to be taken to theatre, requiring administration of an anaesthetic.
T A B L E 3 Actual harm associated with prescribing errors including plausibility and confidence ratings.We designed and applied an iterative process to identify and categorize actual harm from medication errors using a structured case study approach that was familiar to the medical, nursing and pharmacy staff involved in the panels.Our PHAME guides were essential for the team to build case studies that incorporated aspects of harm resulting from both higher and lower doses or blood levels of T A B L E 5 Panel discussion metrics.the medications involved.Our multidisciplinary panels provided a range of specialized expertise which is reflected in the level of discussion concerning the presence of harm, the severity of harm and the level of plausibility that the error caused the harm.Even where no harm was found, the panel members discussed different aspects of the process in 55% of prescribing cases and 80% of administration cases.The most common topic of discussion was how confident the panel members were that no harm had occurred.
Our pilot panel noted that panel members found it difficult to differentiate the severity of harm from the severity of the error.Our final actual harm assessment process therefore contained a three-step process to assess harm by first asking whether there was any evidence from the case study that the patient experienced the type of harm that would be expected from the error.If there was evidence of harm, the next step was to determine the severity of the harm and only then to assess the level of plausibility that the harm was linked to the error.
Only one of the serious harms was deemed 'certain' to have been linked with the error, with the remaining serious harms classified as 'probable' (n = 6) or 'possible' (n = 1).This variation in plausibility highlights the challenge of determining harm in patients who often have multiple co-morbidities which may mimic or mask harms, for example, changes in breathing patterns in patients with underlying lung disease who are given drugs orally rather than through an existing feeding tube.
The comparison between ratings of potential harm and actual harm illustrates the different information about the impact of medication errors that is produced by the different rating scales.For example, we identified two 1000 times dose error cases which were given a high level of potential harm, but the panel found no evidence of actual harm.Potential harm ratings provide an accurate assessment of 'risk' to the patients, whereas the additional step of assessing whether this risk translates into actual harm provides different information and may indicate both the resilience of the organization and individual patients to handle that risk.At present actual harm assessment rarely occurs and many studies continue to only consider potential harm, which is a limitation in understanding how both risk and harm can be reduced.
A limitation of our harm assessment process is that we used retrospective audits to identify prescribing errors.Although a common approach, 28,29 this meant that harms may be under-reported due to the high number of errors where there appeared to be no evidence in patients' records that clinical staff had detected the errors and hence no monitoring or investigations related to the error occurred during the child's admission.One study, using retrospective trigger tool audits, reported only 3.7% of errors resulted in an incident report. 30Conducting audits closer to the events, for example, daily, would enable staff to be informed of those errors and immediately enact required testing and monitoring pathways. 31,32Immediate feedback to staff is also more likely to generate learning from error situations; however, this would be more resource-intensive as the research team would need to identify the errors within a tight timeline.
Further analysis of the results will be conducted to detail the types of harms resulting from prescribing and administration errors, and the health system costs of the additional actions associated with those harms, such as compiling incident reports.These costs have been reported from an additional length-of-stay perspective, 33 and from a parent perspective in terms of the non-medical out-of-pocket costs incurred when a child is in hospital. 15volving patients and parents is critical to determine the impact of those harms, and the next phase of the project will measure how patients, their parents and clinical teams assess relative harm severity.
Our goal is to create a database that incorporates data from other harm assessment studies and includes information about the impact of harm from medication errors on patients and their families, as well as the health system.
We describe our methods in detail in order to facilitate replication of our harm assessment process in different care settings (such as primary care, community care and aged care) and in different patient groups.Including a range of clinicians with recency and currency of relevant practice domains in the assessment process was critical as our panellists were able to contribute different clinical perspectives to detailed discussions on the often-complex clinical picture, normal patterns of care delivery, and the very specific therapeutic impact of age and weight in this patient group.We note that the process we describe for identifying and categorizing actual harm is resource-intensive but believe this can be mitigated by using a clear and consistent process to identify errors and assess harm severity.

| CONCLUSION
This paper addresses the outcome of medication errors, by identifying actual harm, providing a direct measure of patient safety which can be used to measure effectiveness of medication safety initiatives in a healthcare setting. 34Classifying actual harm from medication errors is therefore an important step in understanding the impact of errors on patients and their families, and it is necessary to capture both risk of potential harm and actual harm in order to reduce the errors that cause the most harm to hospitalized children.
on the interpretation of findings.Virginia Mumford prepared the first draft of the manuscript, and all authors contributed to, and approved, the final version.

K
E Y W O R D S medication safety, paediatrics 1 | INTRODUCTION Despite the World Health Organization's Patient Safety Challenge

:
Harm: Impairment of the physical, emotional or psychological function or structure of the body and/or pain resulting from the drug event, and any associated actions such as monitoring* or other interventions.***Monitoring: any change in care pattern from the usual standard level of care, including assessing urine output, general level of consciousness, or vital signs (heart and breathing rate) to ensure no intervention is required.**Interventions: any active treatment, including tests (such as blood tests or x-rays), administering a drug, or general medical/ surgical treatment.

T A B L E 1
Potential harm categories and definitions.with no, or minimal, potential to cause harm to the patient and no need for a change in monitoring or intervention.
Serious Potential Harm category used to assess actual harm.
lower doses than recommended for a drug: the Paediatric Harm Assessment of Medication Error (PHAME) guides.Using the 10 most commonly administered medications and the 10 medications most frequently associated with errors, as reported to the study hospital's voluntary incident reporting system, a series of PHAME guides was prepared by clinicians and pharmacists in the research team.The PHAME guides summarized the relevant symptoms, signs, medications, tests, actions by the clinical care teams and other relevant items from the medical records of patients in whom an error had occurred (see Table Administration harm summary including plausibility and confidence ratings.
Values in bold emphasis emphasize the no harm and total harm in the summary columns-with the minor, moderate and serious contributing to the total harm column.Comparison of actual vs. potential harm ratings.
a Includes unassessable.