A learning health system approach to the COVID‐19 pandemic: System‐wide changes in clinical practice and 30‐day mortality among hospitalized patients

Abstract Introduction Rapid, continuous implementation of credible scientific findings and regulatory approvals is often slow in large, diverse health systems. The coronavirus disease 2019 (COVID‐19) pandemic created a new threat to this common “slow to learn and adapt” model in healthcare. We describe how the University of Pittsburgh Medical Center (UPMC) committed to a rapid learning health system (LHS) model to respond to the COVID‐19 pandemic. Methods A treatment cohort study was conducted among 11 429 hospitalized patients (pediatric/adult) from 22 hospitals (PA, NY) with a primary diagnosis of COVID‐19 infection (March 19, 2020 ‐ June 6, 2021). Sociodemographic and clinical data were captured from UPMC electronic medical record (EMR) systems. Patients were grouped into four time‐defined patient “waves” based on nadir of daily hospital admissions, with wave 3 (September 20, 2020 ‐ March 10, 2021) split at its zenith due to high volume with steep acceleration and deceleration. Outcomes included changes in clinical practice (eg, use of corticosteroids, antivirals, and other therapies) in relation to timing of internal system analyses, scientific publications, and regulatory approvals, along with 30‐day rate of mortality over time. Results The mean (SD) daily number of admissions across hospitals was 26 (29) with a maximum 7‐day moving average of 107 patients. System‐wide implementation of the use of dexamethasone, remdesivir, and tocilizumab occurred within days of release of corresponding seminal publications and regulatory actions. After adjustment for differences in patient clinical profiles over time, each month of hospital admission was associated with an estimated 5% lower odds of 30‐day mortality (adjusted odds ratio [OR] = 0.95, 95% confidence interval: 0.93‐0.97, P < .001). Conclusions In our large LHS, near real‐time changes in clinical management of COVID‐19 patients happened promptly as scientific publications and regulatory approvals occurred throughout the pandemic. Alongside these changes, patients with COVID‐19 experienced lower adjusted 30‐day mortality following hospital admission over time.

changes in clinical practice (eg, use of corticosteroids, antivirals, and other therapies) in relation to timing of internal system analyses, scientific publications, and regulatory approvals, along with 30-day rate of mortality over time. In addition to the need for coordinated global approaches to pandemics, 1 individual health care delivery systems must seek to give equitable, evidence-based care across institutions regardless of geographical region or hospital type. 4 In this realm, a learning health system (LHS) is an ideal organizing principle to inform evidence-based responses to public health emergencies like COVID-19. 4 The LHS concept is characterized as an environment in which "science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience." 5 Seeking to embrace the LHS model, the UPMC health system leveraged its science, data, and analytics capabilities and established the multidisciplinary COVID-19 Therapeutics Committee in early 2020. The purpose of this Committee was to evaluate any possible COVID-19 treatment option and rapidly disseminate updated guidelines to all institutions within the system. The Committee also coordinated with information technology specialists to build forcing functions into several electronic medical records (EMRs) to enforce practice guideline recommendations and also collaborated with research teams to integrate clinical practice with clinical trial enrollment across the enterprise. This LHS process, coupled with regular internal COVID-19 analyses from the UPMC Clinical Analytics Team (described in Methods), formed the basis for establishing, disseminating, and documenting data-driven clinical recommendations to all UPMC outpatient and in-patient facilities caring for patients with COVID-19.
We describe the UPMC LHS approach to the COVID-19 pandemic since March 2020. We share processes on the development and dissemination of clinical guidelines that occurred in a near real-time manner across the entire UPMC system. We also share quantitative results of how such changes mirrored credible findings and information from key scientific publications and regulatory approvals. This is followed by temporal assessment of the 30-day rate of mortality of hospitalized patients with COVID-19.

| QUESTION OF INTEREST
How have hospital patient volumes, patient clinical management, and 30-day mortality changed since the onset of the COVID-19 pandemic within a large, multihospital LHS? 3 | METHODS

| Influence of UPMC COVID-19 Therapeutics Committee
The UPMC COVID-19 Therapeutics Committee was appointed by health system senior leadership in March 2020. This Committee was a subcommittee of the System Pharmacy and Therapeutics Committee (P&T); however, it was given ability to vote and implement therapeutics guideline changes in real-time rather than going through traditional P&T pathways. The Committee met weekly at first and then biweekly. Membership included physicians, pharmacists, hospital leadership, and other stakeholders from academic and community hospitals with clinical, operational, and research experience. A pharmacist and two physicians co-chair the Committee. An internal communications representative attended all meetings and worked, in real time, to update the system intranet as relevant and help draft system-wide communications for clinicians. Information technology specialists also served as Committee liaisons. Finally, an intensive care unit (ICU) service center collaborated with the Committee and provided system-level recommendations, ICU surge provider staffing algorithms, and tele-medicine support for patients throughout the system to limit hospital transfers. 6 We evaluated the influence of the UPMC COVID-19 Therapeutics The system-wide dissemination of treatment guidelines to all physicians and other clinicians affiliated with UPMC occurred through email notifications, computer screensavers, educational webinars, and formal directives from the chair of the Committee. A COVID-19 therapeutics webpage was built into the system intranet.  Therapeutics Committee also created continuous, updated recommendations on the use of monoclonal antibodies for ambulatory COVID-19 patients beginning in November 2020; however, the present analysis is restricted to treatment of hospitalized patients and omits that intervention.

| Patient Population
Within this LHS, there were 5 large, academic hospitals (2474 licensed beds), 8 large, community hospitals (2293 licensed beds), and 9 small, community hospitals (1107 licensed beds) (Table S1) An additional 1875 COVID-19 patients were hospitalized at a UPMC hospital with testing performed outside the UPMC system, resulting in a total of 11 429 hospitalized patients for analysis ( Figure S1).

| Explanatory variables
For assessment of temporal changes, we categorized the study analysis period into 4 discrete "waves" based on empirical change in hospital admissions within the UPMC system. We chose the 4-wave classification scheme ( Figure S2) based on the start and nadir of individual waves. However, because Wave 3 (September 29, 2020 -March 10, 2021) had dramatically higher hospital admissions and discharges, we split this wave at its zenith to assess its impact during rapid acceleration and deceleration. For assessment of variation between waves, we considered demo-  (Table S1).

| Statistical methods
We describe changes over time in COVID-19 hospital admissions  Table S2). Our study received formal ethics approval by the UPMC Ethics and Quality Improvement Review Committee (Project ID Project ID 2882).

| Temporal changes in clinical practice
The COVID-19 Therapeutics Committee published 45 iterations of the clinical practice guideline during the study period. Among patients who received any form of supplemental oxygen, there was rapid system-wide implementation in the use of dexamethasone immediately around the date in which initial positive results of the RECOVERY trial were published as a preprint 11 (Figure 1). Of note, subsequent peer-review publication 12 did not trigger an added uptake in the use of dexamethasone. A steep increase in the use of remdesivir among patients on oxygen therapy occurred after Emergency Use Authorization (EUA) granted by the FDA 13 and subsequent public announcements and regulatory actions [14][15][16] (Figure 2). There was no appreciable variation in the use of dexamethasone or remdesivir by volume or type across the 22 UPMC hospitals (Figures S3 and S4).
Remdesivir was allocated via an ethical lottery system from May 15 through August 1, 2020, during times drug supply was scarce. 17 In contrast, despite widespread publicity, 18 Figure S6). Use of noninvasive ventilation did not vary appreciably across waves, whereas use of mechanical ventilation was markedly lower after wave 1 ( Figure S6).

| Temporal changes in patient characteristics and 30-day mortality
Hospitalized patients in waves 1 and 3a/3b were significantly older than patients in wave 2 (about 3-4 years), and the most recent wave 4 patients were the youngest with mean (median) age of 59.5 (62) years and about a quarter (27%) being age 50 years or younger (Table S3). In aggregate, patients in waves 1 and 3 generally presented with more comorbidities, higher estimated 90-day probability of mortality, and higher neutrophil to lymphocyte ratio (NLR) and systemic inflammatory index than patients in waves 2 and 4 (Table S3).

| DISCUSSION
In 2009, the National Academy of Medicine (NAM) called for development of an LHS, setting a goal that by 2020, "…90 percent of clinical decisions will be supported by accurate, timely, and up-to-date clinical information, and will reflect the best available evidence." 25 The importance of this LHS goal is emblematic with the COVID-19 pandemic. 26 While lacking the ability to demonstrate cause and effect, the fact that the adjusted risk of in-hospital mortality among hospitalized COVID-19 patients at UPMC hospitals has decreased monthly by an average of 5% suggests a consistent learning effect to improved patient care. its EUA, given the existing data. 32 The time between clinical evidence arising and uniform implementation of use was in days-to-weeks, rather than months-toyears, which has been the traditional gap for implementation of findings from RCTs into clinical practice. 33 While desirable, no formal criteria or certification process exists for an institution to be designated as an LHS. 4 One component we believe is essential is embedding of randomized controlled trial procedures into routine care processes using existing institutional infrastructure and electronic health records. 8 This approach defines broad eligibility criteria and aims to enroll as many "real-world" patients as possible to continuously evaluate therapies believed to be potentially efficacious. The key is avoiding "research" and "care" schisms, but rather use all care as an opportunity to learn about care improvement. Randomization is an added tool for some efforts, allowing adaptation as the trial evolves such that subjects are preferentially randomized to receive better performing arms based on interim analyses-termed "response adaptive randomization." 36 This was accomplished at our hospitals by embedding REMAP-CAP enrollment into the EMR, screening all patients with COVID-19 at all hospitals for trial eligibility, and integrating trial enrollment with Therapeutics Committee treatment guidelines. 37 Similarly, when treatment resources are limited and equitable lottery systems are implemented (eg, Remdesivir), this "natural experiment" can be analyzed against nontreated controls.
There are some limitations to our study; because this is the experience in one, albeit large, integrated healthcare system in Western Pennsylvania, the generalizability of our findings may be questioned.
However, the fact that we saw similar findings across our different sites suggests that our findings are applicable across academic, community, and rural hospitals. In addition, we cannot determine the extent to which the therapeutic interventions implemented uniformly by the UPMC COVID-19 Therapeutics Committee contributed to lower adjusted mortality over time, as opposed to other less well-documented clinical practices that may have been implemented over time (ie, mechanical ventilation). Moreover, we cannot directly compare our lower adjusted mortality risk over time to similar findings that have been reported among studies with more hospitals and wider geographic distribution. 27, 28 The LHS description and results presented herein are not meant to be content-or institution-specific, but rather to illustrate some of the processes that can be used to support the NAM imperative for clinical decisions that are supported by accurate, timely, and up-todate clinical information that reflects the best available evidence. 25 On a broader level, we support the stated advocacy for a learning health network that promotes collaboration among health systems, community-based organizations, and government agencies, especially during public health emergencies. 4

| CONCLUSION
Other institutions have qualitatively described their respective LHS processes employed in response to the COVID-19 pandemic, 38,39 with limited quantitative temporal assessment of clinical outcomes. 40 We believe our analysis and description is the first to empirically document how COVID-specific processes employed within an LHS were actually implemented to achieve timely changes in clinical practice on a system level. We recommend that institutions in describing their respective LHS do so by linking (and presenting) processes and sources of information that were used in the establishment and dissemination of clinical care guidelines with data-documented temporal changes in clinical practice and patient outcomes.