In a supplement to its inaugural issue, the Journal of Hospital Medicine published core competencies for hospitalists covering 3 areas: clinical conditions, systems in health care, and procedures.1 Completion of a traditional internal medicine residency may not provide hospitalists with the skills necessary to safely perform necessary procedures such as thoracentesis. A recent article reported that most internal medicine residents surveyed were uncomfortable performing common procedures, and their discomfort was higher for thoracentesis than for central line insertion, lumbar puncture, or paracentesis.2 This confirmed a previous report that family practice residents had low confidence in performing thoracenteses.3 Thoracentesis also carries the risk of the potentially life-threatening complication of pneumothorax, which may be increased when performed by physicians-in-training.4
One method for improving training and assessment is the use of simulation technology. Simulation has been used to increase knowledge, provide opportunities for deliberate and safe practice, and shape the development of clinical skills.5, 6 Simulation has also been advocated for assessing competence in procedures including carotid angiography,7 emergency airway management,8 basic bronchoscopy,9 and advanced cardiac life support (ACLS).10, 11
Recently, we used simulation technology to help residents reach mastery learning standards for ACLS.11 Mastery learning,12 an extreme form of competency-based education,13 implies that learners have acquired the clinical knowledge and skill measured against rigorous achievement standards. In mastery learning, educational results are equivalent, whereas educational practice time differs. To demonstrate mastery learning, we first documented a 38% improvement in skill after a simulation-based educational intervention10 and used a multidisciplinary panel to determine mastery achievement standards for ACLS skills in 6 clinical scenarios.14 These standards were used in a study in which the amount of time needed to achieve skill mastery was allowed to vary while the skill outcomes of the residents were identical clinically.11
The present study had 4 aims. The first was to assess the baseline skill and knowledge of third-year residents in thoracentesis. The second was to compare the thoracentesis-related knowledge and skills of residents before and after an educational intervention. The third was to assess the correlation of medical knowledge and clinical experience with performance on a clinical skills examination after simulation training. The last was to document the feasibility of incorporating simulation-based education into a training program.
- Top of page
All residents consented to participate and completed the entire training protocol. Table 1 presents demographic data about the residents. Most had limited experience performing and supervising thoracenteses.
Table 1. Baseline Demographic Data from 40 Internal Medicine PGY3 Residents Participating in a Simulation-Based Training Program on Thoracentesis
|Age (years), mean (SD)||28.88 (1.57)|
|African American||1 (2.5%)|
|U.S. medical school graduate||39 (97.5%)|
|Foreign medical school graduate||1 (2.5%)|
|Number of thoracentesis procedures|| |
| Performed as an intern|| |
| Performed as a PGY-2 and PGY-3 resident|| |
| Supervised others as a PGY-2 and PGY-3 resident|| |
Interrater reliability for the thoracentesis checklist data was calculated at pretest. Across the 25 checklist items, the mean kappa coefficient was very high (κn = .94). The MPS used as the mastery achievement standard was 80% (eg, 20 of 25 checklist items). This was the mean of the Angoff and Hofstee ratings obtained from the first judgment of the expert panel and is displayed in Figure 1.
Figure 1. Performance on thoracentesis written exam and clinical skills exam performance (MPS, minimum passing score).
Download figure to PowerPoint
No resident achieved mastery at pretest. However, 37 of the 40 medicine residents (93%) achieved mastery within the standard 4-hour thoracentesis curriculum. The remaining 3 residents (7%) needed extra time ranging from 20 to 90 minutes to reach mastery.
Figure 1 is a graphic portrait with descriptive statistics of the residents' pretest and posttest performance on the thoracentesis written and clinical skills exams. For the written exam, the mean score rose from 57.63% to 89.75%, a statistically significant improvement of 56% from pretest to posttest (t = −17.0, P < .0001). The clinical skills exam also showed a highly significant 71% pretest-to-posttest gain, as the mean score rose from 51.70% to 88.3% (t = −15.6, P < .0001).
Results from the regression analysis indicate that neither pretest performance, medical knowledge measured by local or USMLE examinations, nor thoracentesis clinical experience was correlated with the posttest measure of thoracentesis clinical skills. However, the need for additional practice to reach the mastery standard on the posttest was a powerful negative predictor of posttest performance: b = −.27 (95% CI = −.46 to −.09; P < .006; r2 = .28). For those residents who required extra practice time, the initial clinical skills posttest score was 20% lower than that of their peers. Although the need for extra deliberate practice was associated with relatively lower initial posttest scores, all residents ultimately met or exceeded the rigorous thoracentesis MPS.
The responses of the 40 residents on a course evaluation questionnaire were uniformly positive. Responses were recorded on a Likert scale where 1 = strongly disagree, 2 = disagree, 3 = uncertain, 4 = agree, and 5 = strongly agree (Table 2). The data show that residents strongly agreed that practice with the medical simulator boosts clinical skills and self-confidence, that they received useful feedback from the training sessions, and that deliberate practice using the simulator is a valuable educational experience. Residents were uncertain whether practice with the medical simulator has more educational value than patient care.
Table 2. Course Evaluations Provided by All Residents (n = 40) after Simulation-Based Educational Program
|Practice with the thoracentesis model boosts my skills to perform this procedure.||4.3||0.8|
|I receive useful educational feedback from the training sessions.||4.0||0.6|
|Practice with the thoracentesis model boosts my clinical self-confidence.||4.1||0.9|
|Practice with the thoracentesis model has more educational value than patient care experience.||2.3||1.0|
|The Skills Center staff are competent.||4.3||0.6|
|Practice sessions in the Skills Center are a good use of my time.||3.7||1.0|
|Practice sessions using procedural models should be a required component of residency education.||3.8||0.8|
|Deliberate practice using models is a valuable educational experience.||4.0||0.9|
|Practice sessions using models are hard work.||2.1||0.7|
|Increasing the difficulty of simulated clinical problems helps me become a better doctor.||3.9||0.7|
|The controlled environment in the Skills Center helps me focus on clinical education problems.||3.9||0.8|
|Practice with the thoracentesis model has helped to prepare me to perform the procedure better than clinical experience alone.||4.0||1.0|
- Top of page
This study demonstrates the use of a mastery learning model to develop the thoracentesis skills of internal medicine residents to a high level. Use of a thoracentesis model in a structured educational program offering an opportunity for deliberate practice with feedback produced large and consistent improvements in residents' skills. An important finding of our study is that despite having completed most of their internal medicine training, residents displayed poor knowledge and clinical skill in thoracentesis procedures at baseline. This is similar to previous studies showing that the procedural skills and knowledge of physicians at all stages of training are often poor. Examples of areas in which significant gaps were found include basic skills such as chest radiography,27 emergency airway management,8 and pulmonary auscultation.28 In contrast, after the mastery learning program, all the residents met or exceeded the MPS for the thoracentesis clinical procedure and scored much higher on the posttest written examination.
Our data also demonstrate that medical knowledge measured by procedure-specific pretests and posttests and USMLE Steps 1 and 2 scores were not correlated with thoracentesis skill acquisition. This reinforces findings from our previous studies of ACLS skill acquisition10, 11 and supports the difference between professional and academic achievement. Pretest skill performance and clinical experience also were not correlated with posttest outcomes. However, the amount of deliberate practice needed to reach the mastery standard was a powerful negative predictor of posttest thoracentesis skill scores, replicating our research on ACLS.11 We believe that clinical experience was not correlated with posttest outcomes because residents infrequently performed thoracenteses procedures during their training.
This project demonstrates a practical model for outcomes-based education, certification, and program accreditation. Given the need to move procedural training in internal medicine beyond such historical methods as “see one, do one, teach one,”29 extension of the mastery model to other invasive procedures deserves further study. At our institution we have been encouraged by the ability of simulation-based education in ACLS to promote long-term skill retention30 and improvement in the quality of actual patient care.31 In addition to studying these outcomes for thoracentesis, we plan to incorporate the use of ultrasound when training residents to perform procedures such as thoracentesis and central venous catheter insertion.
Given concerns about the quality of resident preparation to perform invasive procedures, programs such as this should be considered as part of the procedural certification process. As shown by our experience with several classes of residents (n = 158), use of simulation technology to reach high procedural skill levels is effective and feasible in internal medicine residency training. In addition, our residents have consistently enjoyed participating in the simulated training programs. Postcourse questionnaires show that residents agree that deliberate practice with simulation technology complements but does not replace patient care in graduate medical education.5, 10
An important question needing more research is whether performance in a simulated environment transfers to actual clinical settings. Several small studies have demonstrated such a relationship,8, 9, 31, 32 yet the transfer of simulated training to clinical practice requires further study. More work should also be done to assess long-term retention of skills30 and to determine the utility and benefit of simulation-based training in procedural certification and credentialing.
This study had several limitations. It was conducted in 1 training program at a single medical center. The sample size (n = 40) was relatively small. The thoracentesis model was used for both education and testing, potentially confounding the events. However, these limitations do not diminish the pronounced impact that the simulation-based training had on the skills and knowledge of our residents.
In conclusion, this study has demonstrated the ability of deliberate practice using a thoracentesis model to produce high-level performance of simulated thoracenteses. The project received high ratings from learners and provides reliable assessments of procedural competence. Although internists are performing fewer invasive procedures now than in years past, procedural training is still an important component of internal medicine training.29, 33 Attainment of high procedural skill levels may be especially important for residents who plan to practice hospital medicine. We believe that simulation-based training using deliberate practice should be a key contributor to future internal medicine residency education, certification, and accreditation.