A Medical Staff Peer Review System in a Public Teaching Hospital—An Internal Quality Improvement Tool



Peer review of the quality of care of the medical staff in a healthcare delivery system, properly executed and utilized, can bring about changes that improve the quality and safety of patient care, enhance clinical performance, and augment physician education. Although all healthcare facilities are mandated to conduct peer reviews, the process of how it is conducted, reported, and utilized varies widely. In 2007, our institution, a large public teaching acute care facility, developed and implemented an electronic Medical Staff Peer Review System (MS-PRS) that replaced the existing paper-based system and created a centralized database for all peer review activities. Despite limited resources and mounting known challenges, we have developed and implemented a system that includes 100% mortality reviews, an ongoing random review for reappointment and operative procedures, and morbidity peer reviews. Parallel to the 4-year implementation of the system, we observed a steady, significant downward trend in the medical malpractice claim rate, which can be attributable in part to the implementation of MS-PRS. In this paper, we share our experiences in the development, outcomes, challenges encountered, and lessons learned from MS-PRS and provide our recommendations to similar institutions for the development of such a system.

Peer review of the quality of care of the medical staff in a healthcare delivery system, properly executed and utilized, can bring about changes that can improve the quality and safety of patient care, enhance overall clinical performance, and augment physician education (Antonacci, Lam, Lavarias, Homel, & Eavey, 2009; Keroack et al., 2007; Olcott, Mitchell, Steinberg, & Zarins, 2000). In the past two decades, there have been external forces to monitor the performance of the medical staff. The most obvious one is the Joint Commission's requirement that a healthcare delivery system must have in place a peer review system to monitor the quality of care of the medical staff on an ongoing basis.

Although all healthcare facilities are mandated to conduct peer reviews, the process of how they are conducted, reported, analyzed, and utilized varies widely allowing for significant inconsistencies. Although positive results have been found in some peer review systems (Jarvi, Sultan, Lee, Lussing, & Bhat, 2002; Nolan, Burkard, Clark, Davidson, & Agan, 2010; Owens et al., 2010; Williams, Mechler, & Akins, 2008), many faced multiple challenges impeding the progress in the field (Livingston & Harwell, 2001; Marren, Feazell, & Padock, 2003; Spaeth, Pickering, & Webb, 2011). Edwards (2009a, 2009b) developed and published a Peer Review Program Self-Evaluation Tool, which can be used to rate current peer review programs. Using this validated tool, it was found that its score is strongly associated with perceived quality impact (Edwards, 2010). Data from 296 acute care hospitals show that the majority of programs rely on an outmoded and dysfunctional process model and that adoption of best practices informed by the continuing study of peer review program effectiveness has the potential to significantly improve patient outcomes (Edwards, 2011).

In 2007, our institution has developed and implemented an electronic Medical Staff Peer Review System (MS-PRS), which has replaced the paper-based system in order to expedite the review process and create a database of all peer review activities including outcomes and improvement actions. In this paper, we present the development, implementation, and outcomes of MS-PRS as well as the challenges encountered and lessons learned.

Materials and Methods

The Healthcare Delivery System

Founded in 1878, our institution is the largest public teaching hospital in the west coast serving the indigent population. Originally established as a 100-bed hospital with 47 patients, it now is licensed for 724 beds. In November 2008, our Medical Center transferred to its new state-of-the-art facilities. We train approximately 1,500 medical professionals daily, including more than 870 medical residents in nearly all clinical specialties.

The Medical Staff Peer Review System (MS-PRS)


The development of the MS-PRS began in July 2006 with the recruitment of a computer engineer whose sole responsibility was the research and development of an electronic peer review system. After development of the peer review guideline and training of all involved parties, the mortality module was fully implemented in January 2007. As of January 1, 2011, the modules for mortality, morbidity, and random peer review are operational including a front-end web-based entry of peer reviews. This paper reports our 4-year experience and findings from mortality reviews.

Processing the reviews

The procedure of processing mortality reviews included the following steps:

  • MS-PRS automatically downloads all deaths on a weekly basis from the hospital system.
  • Quality Improvement Managers (QIMs) and supportive staff ensure medical records are available either in the paper chart or in the hospital's computer systems.
  • The cases ready for peer review are uploaded to MS-PRS’ front-end Intranet web page.
  • Peer reviewers perform and record peer reviews online.
  • Cases that require referrals to or consults from other departments are forwarded for further review by the QIMs or support staff.
  • When a case has no opportunity for improvement by all departments, the case is closed.
  • When a case is deemed to have an opportunity for improvement by any one of the departmental reviewers, the case advances to the next levels of review when final disposition of the case is made.

Peer review levels and bodies

Each mortality review progresses through three levels of peer review: departmental, intermediate, and final levels.

Departmental level review

The departmental level review begins with the review by an attending of the discharge department. The attendant who is privileged in the same specialty serves as the peer reviewer of the case, except his or her own case. The MS-PRS blocks all cases that belong to an attending who signs on to review. A standardized peer review form is filled out by the peer reviewer. If the case requires further review by another department, the peer reviewer designates as such in the review form. When all departmental reviews are completed for the case and if the case is deemed to have no opportunity for improvement by all departmental reviews, then the case is closed. No further action is needed. If the case is deemed to have opportunity for improvement in one or more departmental reviews, it will be forwarded to the intermediate-level review.

Intermediate-level review

The intermediate-level review is performed by the Small Peer Review Committee (SPRC), which consists of a group of voluntary and senior clinicians and surgeons. This intermediate level of review was found necessary to relieve the backlog of cases to be reviewed by the Executive Peer Review Committee (EPRC). The SPRC was formed in July 2010. It meets twice a month. There is a medical leader and a surgical leader. A minimum of three clinicians must be present to have a quorum. All cases deemed to have opportunity for improvement at the departmental level are brought to this committee for review. The SPRC identifies opportunities for improvement, evaluates the appropriateness of the level of care provided, and determines whether or not the improvement actions are acceptable. Once a case is completed, SPRC forwards the case to the last level of review and for further follow-up if necessary by EPRC.

Final-level review

The EPRC serves as the final decision body for the medical staff peer review process. It meets every 2 months. The committee is composed of the Chief of Staff who serves as the Chair of the EPRC, the Chief Medical Officer (CMO), the QI Director, and at least five physicians from different specialties or departments. The Committee's primary responsibilities include: (a) evaluating the appropriateness of follow-up actions taken by clinical and hospital departments and SPRC, (b) identifying opportunities of improvement and forwarding recommendations to responsible parties or departments for system-wide actions, and (c) resolving discrepancies regarding the outcome of peer reviews. The EPRC reviews all cases that have opportunities for improvement and makes its final decisions on preventability and necessary improvement actions. Improvement actions may include but are not limited to conducting root-cause analysis, revising existing, or adding new policies. Letters to the providers are sent out notifying them of the Committee's decision.

Feedback and reporting

To enhance the processing, accuracy and utility of the peer reviews, MS-PRS has developed several feedback functions and interim reports to the clinical departments through the QIMs and to all parties involved in processing the reviews. The following are the major built-in functions:

  • If a case requires referral to another department, MS-PRS automatically generates a new review form for the receiving department. Multiple reviews by different clinical services for each death are captured and linked in the System.
  • A tracking system is in place to improve the processing of each review including when the medical record is successfully retrieved, when the medical record is given to the peer reviewer, when the peer review is done, and when the peer review information is entered into the System.
  • A list of incomplete reviews requiring further action is produced in real-time in the system so that the staff and the QIMs can access it anytime. The list provides the reasons why the review is incomplete.
  • Weekly reports are generated and uploaded in the system to show the number and percent of reviews by department as a progress report to each department to monitor review activity.
  • Quarterly reports are generated and provided to the departments and leadership regarding the responses to the questions, the level of care provided, the opportunity for improvement, and improvement actions taken.
  • Cases requiring the review of the EPRC are generated for discussion from the system.
  • Decisions by the SPRC and EPRC on the preventability of the death and the acceptability of improvement actions are recorded directly into the system during the meetings.
  • Letters to the department Chairs and providers are generated electronically after decisions from EPRC are made.
  • Periodic reports summarizing the outcome and process measures are generated by the system both for the Medical Center as a whole and for each clinical department.
  • The system includes the review of the care provided by residents. Corrective actions for residents are also captured in the system. The mortality and morbidity modules are utilized by departments for discussion of cases at the weekly or monthly M&M conferences for educational purposes.
  • The system is continuously monitored, maintained, and improved by the QI Department.

Confidentiality of peer reviews

The MS-PRS is developed for the use as an internal QI tool. Therefore, peer review information in the system is only accessible to the persons involved in the QI. Both the name of the provider under review and the name of the peer reviewer are blacked out once the information is entered. Feedback reports are without provider or peer reviewer identifications.


The core staffing for the development of the system consists of a full-time computer engineer, half-time project coordinator, and part-time programmer for generating reports from the system. They work under the guidance of the Director of Biostatistics and Outcomes Assessment. This office is responsible for the development of the MS-PRS, coordination of the measurements, and compilation of summary reports. The implementation team includes all the QIMs, under the direction of the Director of QI, who are responsible for communicating with the departments regarding the peer review system, reports, follow-up actions, and orientation of new members in the department. There is an administrative support team in the QI Department who assists in entering the peer reviews that are done on paper into the MS-PRS. At this point in time, some peer reviews are still being recorded on paper due to the restriction of the online Intranet web entry capabilities to the main hospital facility only. Medical staff whose offices are located in other buildings will not be able to access the online Intranet web page.


Peer Review Activities Centralized

After 4 years of development and pilot testing, we have in place an electronic MS-PRS to process both random and focused peer reviews. We have created a database that (a) provides accurate data on outcomes to both clinicians and hospital administration, (b) monitors outcome and process measures over time by individual provider or clinical department/division, and (c) tracks the adequacy and timeliness of improvement actions taken. We have observed improvements in educational opportunities and practice patterns of the medical staff, the delivery of healthcare, and hospital environment and policy. Further, MS-PRS has brought together the fragmented peer review activities throughout the healthcare network and allowed peer reviews from all departments to be centralized, standardized, and quantified as a whole.

Reports and Feedback Provided

Feedback to the clinical departments are provided in the form of monthly operational reports and quarterly outcomes reports. The operational reports provide each department the progress of the peer review activities in the different modules in MS-PRS. The quarterly outcomes reports provide the process outcomes in terms of appropriateness or adequacy of the medical history, physical examination, formulation of diagnostic impression and assessment, monitoring of patient's progress and follow-up, supervision of residents or medical students, use of tests, medications, blood or blood products, invasive procedures, and documentation. Quarterly reports are also provided for peer review outcomes in terms of level of care met, opportunity for improvement, and improvement actions taken.

The Mortality Peer Reviews

The mortality module in MS-PRS has been operational since January 2007 and can provide sufficient information for a more complete picture of the process and outcome. The random review modules were implemented at a much later date in July 2010. Thus, we did not include their evaluation at this point. During 2007 and 2008, mortality reviews were recorded on the paper form by reviewers and later entered into MS-PRS by the supportive staff. Beginning in 2009, the front-end Intranet web entry was operational and the peer reviewers could enter their reviews online into MS-PRS.

Opportunities for Improvement Identified

Over the 4-year period, 2007 through 2010, a total of 3,099 deaths were peer-reviewed, 133 (4%) were considered to have an opportunity for improvement at the first-level review. Of the 133 cases, 5 (4%) have not been peer-reviewed by the EPRC at this time, 93 (70%) cases were finalized as having no opportunity for improvement, and 35 (26%) cases were considered as having opportunities for improvement (Table 1). Of the 35 cases that had a final determination to have an opportunity for improvement by the EPRC, 84 improvement actions were taken. Among them, 51 (61%) were provider-related and 33 (39%) were system-related. The improvement actions are listed in Table 2.

Table 1. Mortality Peer Reviews Activities
Opportunities for Improvement2007200820092010
Number of mortalities reviewed842785726746
No opportunity for improvement at level 1 (%)798 (94.8)752 (95.8)693 (95.5)723 (96.9)
Had opportunity for improvement at level 1 (%)44 (5.2)33 (4.2)33 (4.5)23 (3.1)
Number of mortalities reviewed at levels 2 and 344333323
Final status not determined yet (%)0 (0.0)0 (0)3 (9.1)2 (8.7)
No opportunity for improvement (final) (%)32 (72.7)19 (57.6)23 (69.7)19 (82.6)
Had opportunity for improvement (final) (%)12 (27.3)14 (42.4)7 (21.2)2 (8.7)
Table 2. Improvement Actions Taken
Type of Improvement ActionsNumber (%)Number (%)
  1. *Of the 133 cases that had opportunity for improvement from level 1 review, 93 (70%) were finalized to have no opportunity for improvement, 35 (26%) were finalized to have opportunity for improvement, and 5 (4%) are still pending the last level of review.

Total number cases that had opportunity for improvement*35 
Total number of improvement actions taken84 
Provider-related subtotal51 (61) 
Education including Mortality and Morbidity Conferences 18 (35)
Verbal counseling including review of procedures 17 (33)
Continuous monitoring of performance 7 (14)
Written counseling 5 (10)
Improved supervision 4 (8)
System-related subtotal33 (39%) 
Generated new policies 9 (27)
Improved hospital environment, safety and cleanliness 6 (18)
Improved communication within and between departments 6 (18)
Increased staffing 3 (9)
Purchased new equipment 3 (9)
Increased ICU beds or operating room operations 2 (6)
Clarified existing policies 2 (6)
Generated new form to capture needed information 2 (6)

Measures of Potential Impact

It is difficult to measure the direct impact of MS-PRS as it is an integral part of the quality improvement effort at our institution. As an overall quality indicator on which MS-PRS might have an indirect effect, we examined the trend for the number of active medical malpractice claims at the beginning of the month per 100,000 patient encounters during the month from 2006 through 2010 (Figure 1). We used a patient day as an inpatient encounter and an outpatient visit as an outpatient encounter. The total patient encounters for the month is the total of the inpatient days and outpatient visits for the month. It can be observed that there has been a steady significant downward trend since 2006. To assess the relevance of the decline at our institution, we compared the number of our annual medical malpractice claims with our national and California's number of medical malpractice payment reports against physicians published in a national database (DHHS, 2011) and with California's number of complaint cases against physicians published by the Medical Boards of California. It can be observed from Table 3 that the number of medical malpractice claims on our institution declined 58% from FY 06/07 to FY 09/10, significantly higher decline than that experienced in the United States or in California. Additional measures of impact would be ideal such as serious patient safety events, provider satisfaction with the new process. Efforts will be continued to include these measures in our future evaluations.

Table 3. Comparison of the Number of Medical Malpractice Claims Opened During Year Between Our Institution and California and the United States
 United StatesCaliforniaOur Institution
 Number ofNumber ofNumber of 
 Medical MalpracticeMedical MalpracticeComplaint CasesNumber of
 Payment ReportsPayment ReportsOpened AgainstMedical Malpractice
Year*Against PhysiciansAgainst PhysiciansPhysiciansClaims Opened
  1. *Data from Medical Board of California and our institution are reported by fiscal years and data from the National Practitioner Data Bank are reported by calendar year.

  2. Data from the National Practitioner Data Bank Combined Annual Report 2007, 2008, and 2009. U.S. Department of Health and Human Services, Health Resources and Services Administration, Bureau of Health Professions, Division of Practitioner Data Bank, September 2011, pp. 66–67. Data from Medical Board of California: http://www.mbc.ca.gov/statistics_complaint-investigate.pdf. Bolded results show significant findings.

FY 05/06, 200514,0061,1911,33177
FY 06/07, 200612,4901,0751,18251
FY 07/08, 200711,4759991,13356
FY 08/09, 200811,0219621,12341
FY 09/10, 200910,7721,0031,31232
p-Value from chi-square test comparing 5-year experience to our institution0.0270.0110.008—-
Percentage decrease from FY 05/06−23%−16%−1%−58%
or 2005 to FY 09/10 or 2009(3,234/14,006)(188/1,191)(19/1,331)(45/77)
p-Value from chi-square test comparing the change from 2005 to 2009, to our institution0.0040.001<0.0001—-


Despite limited resources and mounting known challenges, we have developed and implemented an electronic MS-PRS that includes mortality reviews of 100% of deaths, an ongoing random review for reappointment and operative procedures, and morbidity peer reviews.

Figure 1.

Active Medical Malpractice Claims at Beginning of Month per 100,000 Patient Encounters During Month (January 2006 through December 2011)

We evaluated the quality of our peer review system using the Peer Review Program Self-Evaluation Tool developed and validated by Edwards (2009a, 2009b, 2010). The tool has 13 aspects of evaluation and has a total score of 100. Based on a recent national survey of 330 hospitals, the total scores ranged from 0 to 85 with a mean of 45. Using this tool, our mean score is 69, which was derived from the independent rating by each of the authors. Using this tool, the major challenge we identified is the lack of timely performance feedback. Currently it can take from 1 month to 1 year to complete a mortality review. The standard used in the tool considers a timely review to be completed in less than 3 months. Another challenge we identified is the use of a 7 or more rating scale to measure each performance element. Our current instrument uses different scale for different measures. We use a dichotomous scale for process measures, four levels for quality of care and three levels for opportunity for improvement. Other challenges identified using this tool include standardization of process and review structure and recognition of excellence. We believe that we have made significant achievements but many challenges lie ahead. Based on our experience we offer the following lessons learned and recommendations.

Lessons Learned and Recommendations

  • When budgeting for such a development, adequate staffing and adequate infrastructure must be included to assure proper implementation, monitoring, and dissemination of the findings.
  • Must be supported by a centralized electronic medical records system. Because of the need to pull cases from different hospital information systems, the integrity of the peer reviews is dependent on integrated, complete, and accurate information. In addition, peer reviewers must be provided universal electronic access to record the peer reviews online.
  • Must have continuous in-service, monitoring, and improvement of the system. All departments must have continuous in-service. A peer review system should monitor the level of care received by the patient, any system breakdown associated with the care of the patient, and the follow-up actions needed to achieve closure for each case.
  • Must balance the peer review workload among departments. Each department has a different average number of morbidities, mortalities, providers, and operative procedures.
  • All clinical departments must buy in. Without full departmental cooperation, peer review will not be fair or balanced, reports on providers cannot be compared, and follow-up actions will not be consistent.
  • Must improve the peer review instrument to reduce variation in judging the quality and appropriateness of care and reviewer biases as found by others (Kadar, 2010).
  • Must be endorsed and supported by the medical staff and the institution. The Attending Staff Association must ensure compliance with, and thus utility of, the peer review system.
  • The culture of peer review must be changed from a punitive to a rewarding philosophy. If we truly want to improve quality, it has to be internal, nonthreatening, confidential, and yet accurate and effective.
  • Overall, no amount of science, technology, or knowledge is sufficient to bring about good without proper implementation, monitoring, and dissemination; and, no amount of support of a peer review system is sufficient to bring about good without wide endorsement by the medical staff and the institution.

We have addressed the challenges for establishing an MS-PRS in a large teaching hospital. We must also address challenges encountered in measuring the impact of the system. The most direct measures of impact are the number of preventable deaths and the corrective actions taken by the institution as we have reported here. The other measure of impact, the medical malpractice claims, reported here cannot be considered as directly related to the MS-PRS as many other QI programs or management changes have occurred over the same time period. Efforts are underway to evaluate the impact of the system by studying indicators that may be more directly related to the system such as serious patient safety events, provider satisfaction, related readmissions, and related mortalities.


This project is supported in part by the Kaiser Permanente Southern California Region Community Benefit Program.


  • Linda S. Chan, PhD, is the director of Biostatistics and Outcomes Assessment at the Los Angeles County University of Southern California Medical Center and is professor of Research at the Keck School of Medicine of the University of Southern California. She directs the development and implementation of the medical staff peer review system.

  • Manal Elabiad, MS, is a computer engineer at the Los Angeles County University of Southern California Medical Center who develops and maintains the system.

  • Ling Zheng, MD, PhD, is an epidemiologist and biostatistician at the Los Angeles County University of Southern California Medical Center who is in charge of the reporting of findings.

  • Brittany Wagman, BS, was the research coordinator at the Los Angeles County University of Southern California Medical Center during the early years in charge of the implementation and in-service of the system.

  • Garren Low, MS, is the research coordinator at the Los Angeles County University of Southern California Medical Center in charge of the implementation and in-service of the system.

  • Roger Chang, MA, is a doctoral candidate in Biostatistics at the Los Angeles County University of Southern California Medical Center who assists with the integration of data systems and compilation of reports.

  • Nicholas Testa, MD, is the associate medical director for quality improvement at the Los Angeles County University of Southern California Medical Center who oversees the implementation of the system.

  • Stephanie L. Hall, MD, is the chief medical officer at the Los Angeles County University of Southern California Medical Center who initiated the project and oversees the development, implementation, and utility of the system.