Implementation of a computer-assisted monitoring system for the detection of adverse drug reactions in gastroenterology
Dr H. Dormann, Medizinische Klinik I mit Poliklinik, Friedrich-Alexander Universität Erlangen-Nürnberg, Ulmenweg 18, D-91054 Erlangen, Germany.
Aim : To investigate the effectiveness of a computer monitoring system that detects adverse drug reactions (ADRs) by laboratory signals in gastroenterology.
Methods : A prospective, 6-month, pharmacoepidemiological survey was carried out on a gastroenterological ward at the University Hospital Erlangen-Nuremberg. Two methods were used to identify ADRs. (i) All charts were reviewed daily by physicians and clinical pharmacists. (ii) A computer monitoring system generated a daily list of automatic laboratory signals and alerts of ADRs, including patient data and dates of events.
Results : One hundred and nine ADRs were detected in 474 admissions (377 patients). The computer monitoring system generated 4454 automatic laboratory signals from 39 819 laboratory parameters tested, and issued 2328 alerts, 914 (39%) of which were associated with ADRs; 574 (25%) were associated with ADR-positive admissions. Of all the alerts generated, signals of hepatotoxicity (1255), followed by coagulation disorders (407) and haematological toxicity (207), were prevalent. Correspondingly, the prevailing ADRs were concerned with the metabolic and hepato-gastrointestinal system (61). The sensitivity was 91%: 69 of 76 ADR-positive patients were indicated by an alert. The specificity of alerts was increased from 23% to 76% after implementation of an automatic laboratory signal trend monitoring algorithm.
Conclusion : This study shows that a computer monitoring system is a useful tool for the systematic and automated detection of ADRs in gastroenterological patients.
Adverse drug reactions (ADRs) cause a significant prolongation of hospital stay and rank amongst the top 10 causes of death in US hospitals.1, 2 The implementation of drug surveillance systems in hospitals has been suggested by several authors.2–5 However, permanent intensive pharmacovigilance systems are expensive and time consuming, whereas spontaneous, voluntary reporting by medical staff only provides limited information and underestimates the rate of ADRs.4–6 The most important, but unresolved, problem during the course of drug therapy is the early identification of potential ADRs before they cause serious damage to the patient. The availability and use of large computerized clinical databases linked to electronic medical records could provide a tool for the early detection of ADRs and thus help clinicians to react appropriately in time.7 It has been shown in several studies that computerized monitoring, using alerts based on laboratory data, can facilitate ADR detection.8–12 This study was undertaken with the following aims: (i) to provide data on the type and incidence of ADRs in gastroenterology; (ii) to modify the generation of computerized alerts in order to increase the specificity of the system and to evaluate its effectiveness; (iii) to compare chart reviews and modified generation of computerized alerts, representing two different methodological approaches to the identification of ADRs; and (iv) to analyse the impact of ADR detection on the behaviour of physicians.
During a 6-month period (September 2000 to March 2001), a prospective survey was carried out on a 29-bed gastroenterological ward at the University Hospital Erlangen-Nuremberg, Germany. Only male patients were admitted to this ward. The main outcome measure was the appropriateness of computerized ADR alerts, generated as automatic laboratory signals (ALSs), for ADR monitoring. An ADR was defined according to the World Health Organization definition as: ‘a response to a drug which is noxious and unintended, and which occurs at doses normally used in man for the prophylaxis, diagnosis, or therapy of disease, or for the modification of physiological function’.13
Two independent methods were used in parallel to identify ADRs.
(a) ADRs were identified through chart reviews conducted by one of two pharmacoepidemiological teams, each consisting of one physician and one pharmacist. The probability and severity of the identified ADRs and the type of affected target organ were classified using the Naranjo score and the World Health Organization Adverse Reaction Terminology System— Organ Classes.10, 14, 15 The consensus of both teams in the evaluation of ADRs was reached in each case prospectively. Both physicians in the teams were specialized in gastroenterology.
(b) ADRs were identified through a computer monitoring system, which generated alerts as ALSs. These ALSs were defined for 25 laboratory tests and automatically assigned to three different types by the following rules (Table 1): (i) a new ALS was the first value of a laboratory test outside the range defined as normal in the study (usually the physiological range); (ii) a delta ALS was a new value of a laboratory test, which differed significantly from the previous value (the absolute value of delta ALS was defined by the pharmacoepidemiological team for each laboratory test; for example, for γ-glutamyltransferase, delta ALS was defined as 20 U/L and, for serum creatinine, as 0.3 mg/dL, etc.); (iii) a temporary ALS was defined as a repetitive abnormal value, which differed from the previous value by less than delta ALS and thus was not used as an alert.
Table 1. Normal ranges (lower, upper and delta limit) and positive predictive value (PPV) of automatic laboratory signals (ALSs) used as alerts
|Hepatotoxicity|| || || || ||1255||64/263||21||470||28/130||28|
| Bilirubin||mg/dL||0||1|| ||196||17/64||33||76||11/41||54|
|Renal toxicity|| || || || ||139||16/40||29||12||3/6||50|
| Uric acid||mg/dL||0||7||1||87||6/21||24||3||–/1||33|
|Electrolytes|| || || || ||174||35/65||37||15||7/10||67|
| Na+||mmol/L||132||155|| ||55||12/22||40|| || || |
| Ca2+||mmol/L||2.3||2.8|| || || || || || || |
|Haematological toxicity|| || || || ||207||14/60||29||25||2/15||25|
| Hb||g/dL||9||20|| ||33||3/10||30||1|| || |
| Pla||× 103/µL||140||600|| ||174||11/50||29||24||2/15||63|
| Leucocytes||× 103/µL||2.5||20|| || || || || || || |
|Coagulation|| || || || ||407||48/116||29||54||18/26||48|
| INR|| ||0.15||1.2||0.15||295||32/79||28||24||7/11||46|
|Metabolic disturbances|| || || || ||129||5/27||21||4||–/2||50|
| Triglyceride||mg/dL||0||200|| ||71||3/15||21|| || || |
| Glucose||mg/dL||70|| || ||9||2/3||33|| || || |
| IgE||U/mL||0||100|| ||8||0/1||13|| || || |
|Total|| || || || ||2328||184/574||25||580||59/189||32|
In addition to all generated ALSs, the patient's database contained individual demographic characteristics, history (e.g. allergies), laboratory findings, diagnoses (International Classification of Diseases, 10th Revision) and a list of drugs (AHFS — Pharmacologic Therapeutic Classification) given during hospitalization.16
All ALSs were evaluated by a pharmacoepidemiological team regardless of whether or not they were associated with an ADR.
The detection of ADRs by new and delta ALSs as alerts was tested vs. delta ALSs alone.
Each alert (new and delta ALSs) was analysed according to whether or not it was definitely diagnostic for ADRs. The ALS alerts definitely diagnostic for particular ADRs were assigned to the ‘LabADR’ category by the pharmacoepidemiological team physicians. For example, a decrease in haemoglobin due to gastrointestinal bleeding caused by non-steroidal anti-inflammatory drugs was defined as a LabADR, as was hypokalaemia due to loop diuretics or hypoglycaemia caused by sulphonylureas in diabetes. Alerts which were associated with ADR-positive admissions, but were not definitely diagnostic for an ADR in the manner mentioned above, were defined as ‘associated with ADR’. For example, new or delta ALSs of hypokalaemia in antibiotic-induced diarrhoea were defined as associated with ADR.
For each type of alert (new and delta ALSs), the positive predictive value, sensitivity and specificity were assessed. The positive predictive value was defined as the number of alerts associated with ADRs out of the total number of alerts. The sensitivity was defined as the number of ADR-positive admissions, detected through a given type of alert, out of the total number of ADR-positive admissions. The specificity was defined as the number of ADR-negative admissions, without a given type of alert, out of the total number of ADR-negative admissions.
In addition, the awareness of staff physicians to ADRs was evaluated and classified by the pharmacoepidemiological team. If the physician's chart noted a change in drug regimen, additional laboratory tests or other diagnostic actions, subsequent and related to a specific ADR, the ADR was categorized as ‘recognized’.
Characteristics of the study sample
One hundred and nine ADRs occurred in 474 male admissions, representing 377 patients [mean age, 54.5 years; standard deviation (s.d.), 16.7 years; minimum, 16 years; maximum, 90 years]. The admissions included 88 emergencies, 212 scheduled admissions and 32 transfers from other clinical departments or hospitals (45 not documented). The average length of hospital stay was 9.3 days (s.d., 9.6 days; minimum, 1 day; maximum, 60 days). Three hundred and sixty-six patients were regularly discharged, two were transferred to other hospitals and nine died during hospitalization.
Characteristics of ADRs by probability and severity
According to the Naranjo algorithm, 62 of the 109 ADRs were categorized as possible and 47 as probable. All ADRs were judged to be of clinical relevance. Sixty-six were regarded as mild, 42 as moderate and one as severe (generalized seizure after the application of antibiotics). In 50 cases, additional drug therapy and laboratory diagnostic measures were necessary. In 23 ADRs, the drug involved had to be withdrawn. In 40 cases, drug therapy was continued due to a positive benefit to risk ratio or a lack of therapeutic alternatives (e.g. chemotherapy), but the dosages were reduced. In 10 ADRs, diagnostic procedures (e.g. endoscopic intervention) were needed to treat the adverse reaction. In a further five cases, intensive care was essential and, in one case, irreversible damage (corticosteroid-induced osteoporosis) was observed.
Drugs causing ADRs and the target systems and organ classes involved
Central nervous system agents (n = 23) represented the prevailing ADR-inducing drug class of all the prescriptions used (n = 3742). Opiate agonists caused gastrointestinal disorders, whereas anxiolytics, sedatives and hypnotics exerted liver or biliary tract toxicity. Taking into account the total number of prescriptions in each drug class, hormones and synthetic substitutes (8%) (anti-diabetic agents and adrenals), followed by electrolytic-, caloric- and water-balancing agents (6%), were the prevailing ADR-inducing drugs. These ADRs resulted predominantly in metabolic and nutritional disorders. Anti-neoplastic agents (6%) were responsible for red blood cell or platelet, bleeding and clotting disorders. Cardiovascular drugs (4%) led to a wide variety of symptoms. Anti-infective agents (3%) resulted in gastrointestinal (e.g. diarrhoea), liver, biliary, central nervous system (e.g. convulsive seizure) and skin disorders.
Laboratory tests and ADR alerts (Table 2)
Table 2. Number of laboratory tests, automatic laboratory signals (ALSs) generated and alerts in adverse drug reaction (ADR)-positive admissions and all admissions
|ADR-positive admissions||10417||1584|| 385||189||1010|
In 460 of the 474 admissions, 39 819 laboratory tests were determined during the study period. The frequency and distribution of all new, delta and temporary ALSs in ADR-positive vs. all admissions are shown in Table 2. In 74 of the 76 ADR-positive admissions, 10 417 laboratory values generated 1584 ALSs, 574 of which (385 new ALSs and 189 delta ALSs) were associated with ADRs. One hundred and eighty-four alerts produced by new ALSs and 59 alerts produced by delta ALSs were categorized as LabADRs (Table 1).
Types of alert (Table 1)
The largest numbers of alerts were those indicating hepatotoxicity (1255) and coagulation disorders (407). The positive predictive value of alerts categorized as new ALSs varied between a minimum of 13% (allergy) and a maximum of 40% (electrolytes). Using delta ALSs alone as alerts, the overall positive predictive value was 32%, varying between 18% (alkaline phosphatase) and 67% (serum potassium).
Comparing the positive predictive values of the alerts produced by new ALSs, on the one hand, and by delta ALSs, on the other, the platelet count, cholesterol and potassium showed a significant increase to 67%. For alkaline phosphatase, the positive predictive value was independent of the type of alert and remained stable at 18%. Delta ALSs for sodium or triglyceride did not occur.
Sensitivity and specificity (Table 3)
Table 3. New and delta automatic laboratory signals (ALSs) as adverse drug reaction (ADR) alerts
|Alert (new ALSs)*||69||306||375|
|No new ALSs|| 7|| 92|| 99|
|Alert (delta ALSs)†||31|| 94||125|
|No delta ALSs||45||304||349|
Overall, the computer monitoring system generated alerts for 69 of the 76 ADR-positive admissions.
The sensitivity and specificity of alerts generated by new ALSs were 91% and 23%, respectively. Ninety-two of the 398 ADR-negative admissions were negative for new ALSs. Delta ALSs were associated with only 31 ADR-positive admissions out of a total of 76. Three hundred and four of the 398 ADR-negative admissions received no delta ALS alert. The sensitivity and specificity of delta ALS alerts were 40% and 76%, respectively.
ADR detection methods (Table 4)
Table 4. Adverse drug reaction (ADR) detection methods and alert definitions
|CMS + physician|| 61/44|| 31/19|
|Physician only|| 6|| 6|
|CMS only|| 39|| 15|
Of the 109 ADRs, the attending physicians recognized 67 (61.5%) in 76 admissions. New ALSs generated 184 specific alerts (LabADR) in 72 (66.0%) of the 109 ADRs. Overall, 385 new ALSs were associated with 100 (91.7%) ADRs in 69 admissions. Of the 109 ADRs, delta ALSs were associated with 51 (46.8%) ADRs in 31 admissions, whereas LabADR ALSs were present in 34 (31.2%) ADRs, as shown in Table 4. Sixty-one ADRs were alerted by the computer monitoring system and the attending physicians. Six ADRs were recognized by physicians only. Thirty-nine ADRs were alerted by the computer monitoring system and three through chart reviews alone. Fifteen ADRs were alerted by delta ALSs alone.
The use of computer-based decision support tools based on electronic medical records in the management of adverse events has been shown to aid the detection of adverse events, improve patient outcome and enhance clinician competence in drug therapy.7, 17–19 Classen et al. demonstrated an 800% increase in the number of adverse drug events identified compared with spontaneous reporting.11 Therefore, the need for monitoring hospital wards electronically for ADRs has been stressed repeatedly.3, 11, 20–22 With regard to the actual drug therapy of patients, computerized surveillance systems usually compare laboratory values with pre-determined reference parameters.11, 22–24 The effectiveness of computer monitoring systems in alerting for potential ADRs depends on the predictive value of the pre-defined study rules (border values) in both sensitivity and specificity.3, 8, 11, 22 We attempted to improve the existing study rules in order to select early indices of the development of ADRs and to increase the predictive value of the alerts and specificity without losing the sensitivity of the computer monitoring system.10 To achieve these goals, our rules (new ALS, temporary ALS) were supplemented with time-related delta ALSs, which demand the physician's attention when relevant trends in laboratory values occur. After introducing delta ALSs, we observed an improvement in the overall predictive value from 13% in former studies to 32% in this study. Delta ALSs proved to be most efficient when based on electrolyte shifts (e.g. new ALS if potassium > 4.8 mmol/L; delta signal if shift > + 0.5 mmol/L). In this particular group, two of three alerts were indeed related to ADRs.
Without the dynamic modification, only one of seven alerts was related to a real ADR.10 Even if the upper reference value was fixed on a high level, e.g. potassium > 6.5 mmol/L, the positive predictive value was no higher than 12% without implementing delta signals.22 On the other hand, alkaline phosphatase and γ-glutamyltransferase were of less predictive value. One explanation for this may be the high number of patients with pancreatic diseases and bile duct stones in this study. Nevertheless, the sensitivity of the system increased from 62% to 91% and the specificity from 42% to 76% compared with other studies.9, 10, 12
Using new ALSs as alerts, the computer monitoring system was highly efficient in screening for ADRs: only nine ADRs were not alerted by the computer monitoring system. However, the main disadvantage of this procedure was the difficulty in discriminating between ADR-positive patients and those without an ADR. This problem was solved partially by introducing delta ALSs, which alerted 51 ADRs in 31 patients. By this procedure, the specificity increased to 76%.
However, our system is still not optimally suited to achieve both an early detection of ADRs and a high predictive value of the alerts. Further improvement may be achieved by modification of the rule definitions. For example, by excluding laboratory parameters (alkaline phosphatase and γ-glutamyltransferase) from the study rules, the positive predictive value of delta ALSs would increase to 47% without a significant change in the sensitivity or specificity of the system in gastroenterology. In contrast, by including further parameters, such as heparin-induced antibodies or microbiological data (Clostridium difficile and its toxins), a further improvement of the overall sensitivity and specificity of the system may be obtained. Greater improvement will be possible in the future by the contextual linkage of new and delta ALSs to individual medication data in drug databases, allowing real-time decision support.25, 26 However, using computerized drug databases without linkage to individual patient parameters (ALSs, symptoms, diagnoses, etc.) will produce a large number of false positive alerts and will not support physicians in time-saving drug surveillance.27 With the present improvements of our computer monitoring system, 24 clinically relevant ADRs were detected by the system alone, clearly increasing the overall detection rate (Table 4). This finding indicates that it is feasible to detect ADRs by a computer monitoring system in real time, and alerted physicians may intervene before serious harm is inflicted.28
The specificity of the computer monitoring system was improved by the implementation of ‘trend monitoring’ in laboratory values (delta ALS). This study shows that the computer monitoring system is a useful tool for the early and automated detection of ADRs in patients with hepatic or gastrointestinal diseases.
This study was supported by grants from ELAN No. 98.07.30-1, the German Israel Foundation (GIF) No. G 690 221.9/2000 and the Marohn Foundation No. Bru/2001.