Experience Curves as an Organizing Framework for Deliberate Practice in Emergency Medicine Learning


  • The authors have no relevant financial information or potential conflicts of interest to disclose.

Address for correspondence and reprints: Martin Pusic, MD, PhD; e-mail: Martin.pusic@nyumc.org.


Deliberate practice is an important skill-training strategy in emergency medicine (EM) education. Learning curves display the relationship between practice and proficiency. Forgetting curves show the opposite, and demonstrate how skill decays over time when it is not reinforced. Using examples of published studies of deliberate practice in EM we list the properties of learning and forgetting curves and suggest how they can be combined to create experience curves: a longitudinal representation of the relationship between practice, skill acquisition, and decay over time. This framework makes explicit the need to avoid a piecemeal, episodic approach to skill practice and assessment in favor of more emphasis on what can be done to improve durability of competence over time. The authors highlight the implications for both educators and education researchers.


Las Curvas de Experiencia como Marco Organizativo para el Ejercicio Reflexionado en el Aprendizaje de la Medicina de Urgencias y Emergencias

El ejercicio reflexionado es una estrategia importante en el entrenamiento de habilidades en la formación en medicina de urgencias y emergencias. Las curvas de aprendizaje muestran la relación entre la práctica y la competencia. Las curvas del olvido muestran lo contrario, y demuestran cómo la habilidad decrece con el paso del tiempo cuando ésta no es reforzada. Tomando como ejemplo los estudios publicados sobre el ejercicio reflexionado en medicina de urgencias y emergencias, se enumeran las propiedades de las curvas de aprendizaje y del olvido, y se sugiere cómo pueden combinarse para desarrollar curvas de experiencia: una representación longitudinal de la relación entre la práctica, la habilidad de adquisición y el deterioro a lo largo del tiempo. Este marco hace evidente la necesidad de evitar una aproximación episódica o parcelada a la valoración y ejercicio de la habilidad en favor de un mayor énfasis sobre lo que puede hacerse para mejorar la durabilidad de la competencia a lo largo del tiempo. Los autores destacan las implicaciones tanto para los docentes como para los investigadores sobre la formación.

Emergency physicians (EPs) intervene cognitively or physically to ameliorate any of a number of health conditions ranging from simple laceration repairs to managing massive multitrauma disaster situations. One of the fundamental tasks in medical education is the teaching and learning of these cognitive and kinesthetic skills. To gain these skills, novices use a number of instructional strategies, ranging from mental imagery to practicing on high-fidelity simulators. Some of these skills are amenable to deliberate practice, in which the learner repeats the given skill with informative feedback, in the hopes of progressive improvement to a recognized level of mastery.[1, 2] Once mastery is achieved, issues of skill retention supervene.[3] Skills not practiced can degrade at a startling rate.

In this article, we present experience curves, which are combinations of learning and forgetting curves that can demonstrate a single, longitudinal representation of knowledge and skill acquisition and decay over time. We will focus on discrete skills that are amenable to deliberate practice as defined by Ericsson.[1] Deliberate practice consists of repetition with tailored feedback, is effortful, and is best done over an extended period. Examples could be cognitive (interpreting a radiograph, doing a calculation, applying knowledge to a medical decision, etc.), kinesthetic (performing a cricothyroidotomy), or mixed (leading a resuscitation).

Informed by selected research in the medical and general education literatures, we first describe the educational properties of learning curves and forgetting curves.[4-6] We then show how these concepts can be united to make up experience curves.[7] Our goal in writing this concept article is to promote a longitudinal approach to how these sorts of skills could be practiced in emergency medicine (EM) simulation environments. This framework makes explicit the need to avoid a piecemeal, episodic approach to skill practice and assessment in favor of more emphasis on durability of competence over time.

Learning Curves

Learning curves represent the relationship between units of practice and level of performance.[5, 8] With increasing deliberate practice, performance reliably improves; however, as the individual moves up the learning curve, the law of diminishing returns supervenes such that each unit of practice is rewarded with progressively less increase in performance[9] (Figure 1). That is, the relationship is not linear, but rather follows a negative power distribution.[5, 9]

Figure 1.

Generic learning curve. From Pusic M, Pecaric M, Boutis K. How much practice is enough? Using learning curves to assess the deliberate practice of radiograph interpretation. Acad Med. 2011;86:731–6.[9]

At some point in his or her improvement, the trainee will cross a competency threshold in which he or she can reliably perform the skill independently. Learners may then diverge according to their goals and motivations, with some continuing deliberate practice in pursuit of ever greater skill, even in the face of diminished return on each invested unit of practice. Others find their competence level acceptable and decrease their practice repetitions, sometimes stopping altogether. These latter practitioners may allow the skill to become automatic, where the skill is compiled, with many aspects becoming subconscious.[1, 10] This automaticity can be adaptive in that it makes the procedure less effortful for the practitioner; however, it has the disadvantage of a leveling off of the skill. The practitioner no longer improves with each repetition. To fight the natural tendency to automaticity, practitioners can engage in regular deliberate practice, in which they work on related tasks that “are initially outside their current realm of reliable performance.”[11] This practice is effortful and not inherently pleasant, especially at the expert level where incremental improvements are difficult to discern. The essential part of this idea is that a learner is not “locked” into a given trajectory, but it requires deliberate effort to keep improving.

To demonstrate how graphical learning curves can illustrate these principles, consider Figure 2. This figure was generated from our prior research on deliberate practice of radiograph interpretation.[12] We recruited a convenience sample of 46 medical participants who deliberately practiced the interpretation of pediatric ankle radiographs taken for the purposes of ruling out fractures. The participants spanned the medical education spectrum (20 medical students, six residents, 12 fellows, five EM attending physicians, and three pediatric radiologists). Each participant reviewed 234 cases that were in diagnostic proportions equivalent to those seen in actual practice, with approximately one-third showing evidence of fracture. The cases were done online in a serial fashion with immediate feedback for each case in the form of a marked-up image and the radiologist's report. The methods are outlined in detail in the original report.[12] The explicit graphing of the learning curves by learner level reveals several ways in which these longitudinal expressions of performance can be useful. First, the curves of the medical students, residents, and fellows showed the classic learning curve pattern with rapid initial learning followed by slowly progressive learning out to the last case. Second, their curves allow prognostication: it would take the average fellow or resident about another 150 cases to reach the performance level of an EM attending physician. Third, we can also comment on the efficiency of the learning intervention: clearly the greatest gains across groups occur within the first 100 cases; thereafter, learning becomes more effortful. Residents and fellows learned faster than medical students, represented by the slopes of the learning curves (greater for residents). Finally, the asymptotes are also revealing. The attending EPs were no longer learning after the first 100 cases. Specifically, we can see that this learning intervention, for the EM attending physicians, is not sufficient to bridge the gap between themselves and the radiologists.

Figure 2.

Learning curves for radiograph interpretation. From Boutis et al.[12] Five groups of medical participants (n = 46) each read 234 pediatric ankle radiograph cases in a serial fashion. The nature of the curves differed between groups, as described in the text.[12] Area under ROC curve is an index of radiograph interpretation performance. ROC = receiver operating characteristic.

More generally, learning curves have been used to provide such insights in a wide variety of medical endeavors such as other radiology tasks,[13] cricothyroidotomy insertion,[14] and laparoscopic surgery.[5]

Forgetting Curves

Some skills are so well encoded during training, or so frequently practiced during work, that the individual's performance will remain competent for the whole of his or her career. Other skills, often those of higher complexity with limited opportunities for workplace practice, will degrade in the absence of refresher practice.[3] Skill degradation follows a curve that is inverse to the learning curve.[7, 15] It too has a consistent slope and, while performance skill may eventually hit zero, it is more likely to also follow an (inverse) power function in which the drop-off in skill slows with time to some asymptote of functioning still above that of a novice.[7] For complex skills, the drop-off may be dismayingly steep.[16, 17]

Consider an EM example. Kovacs and colleagues[17] ran an EM airway course and then measured content-specific knowledge at four time points afterward, comparing a control group with an intervention group that received refresher training. Their results are shown in Figure 3, illustrating that the refresher training blunted the forgetting seen in the control group.

Figure 3.

Forgetting curves after an airway course. From Kovacs et al.[17] Two groups of learners participate in an airway management course at time 0. Their scores on a knowledge test are measured four times total for two groups, one that received no further training (control) and another that received regular refresher training (refresher education). The refresher education blunted the decay in knowledge seen in the control group.[17]

A number of factors can affect the slope of a skill degradation, or forgetting, curve. The nature of skill to be retained is important.[18] Some studies have found that cognitive tasks are less easily retained than kinesthetic ones, although there is likely a large interaction with task complexity.[6] Regular deliberate practice on simulated patients or task trainers (including appropriate difficulty and feedback) could prevent skills decay.

A large number of experiments across fields show that the rate of degradation is proportional to the strength of the initial encoding.[6, 19] Thus, during an individual's training period, a successful instructional strategy might be to train the individual well beyond any minimal competency threshold with the reliable expectation that, in addition to attaining a higher level of excellence, the skill would also likely be more durable.[20] Other instructional strategies such as interweaving (mixing practice types and methods) and spaced learning (distributing practice over time instead of massing it) have also been shown to have outsized effects on skill degradation compared with assessments on immediate measures.[21, 22] Reviewing this large literature is outside the scope of this review. However, the idea that the choice of instructional design can affect the skill degradation rate provides the impetus to join learning curves to forgetting curves.

Experience Curves

Learning curves can be usefully joined to forgetting curves to form what are known as “experience curves”[4, 7] (Figure 4). As before, the individual's performance (vertical axis) improves with increased deliberate practice (horizontal axis). With enough practice and feedback, the individual achieves a performance that allows the community to declare her competent. She may continue to practice and achieve even higher performance. Once the individual decreases or stops training, her performance may decay in accordance with a forgetting curve such that the performance may eventually fall below the competency standard. The nature of the initial training, the strength of the encoding, and the workplace affordances may all influence the “durability” of the individual's competence. We could also represent an individual's ability to sustain excellence by the area bounded by the experience curve and the line of competence, although some might rightly argue that excellence is characterized by performance at a level considerably higher than the minimal line of competence.[1]

Figure 4.

Prototypical experience curve. Notable features include the presence of both a learning component (upward slope during training, with horizontal axis in units of deliberate practice such as number of simulator repetitions) and a forgetting component (where time units are in units of elapsed time since last training). We show the potential difference associated with a greater amount of deliberate practice for Group 1 than Group 2.

In attempting to reflect in one diagram both the learning and forgetting experiences of an individual with a given skill, we point out some constraints on the axes. For the horizontal axis, we need to represent two different conceptualizations of time: effort invested in deliberate practice during the training-learning phase and its opposite for the decay curve—elapsed time spent without practice. During the learning phase, elapsed time, in say a residency program, can be poorly correlated with the actual amount of time spent in true deliberate practice that is effortful and that incorporates critical feedback. Thus, while learning trajectories based on elapsed time might naturalistically describe a trainee's experience in a residency program, the explanatory power of these trajectories can be improved by basing them on measures of time spent in deliberate practice. In his famous studies of German violinists, Ericsson[11] found that hours spent per week in deliberate practice was reliably predictive of final attainment, whereas years in the activity was not. For medical procedures, the hours spent deliberately practicing a procedure reflects the cost that has been paid to achieve a given level of performance. Beyond the initial latent phase, counts of the number of times a procedure is done under deliberate practice conditions may be a more easily obtained proxy measure. The time scale used for the experience curve needs to change when describing the degradation portion. Here we are representing a context where the clinician is no longer learning but rather forgetting. Elapsed time on a scale that can capture the fall below a competency standard is the appropriate metric. Time spent in refresher training is represented as a bolus at a point in time, although more sophisticated methods can be used.[7]

For the vertical axis representing degree of skill, it is ideal if these units are the same for both the learning and the forgetting portions. Ideally, the same measure of performance is used all along the scale from novice to expert so that valid developmental comparisons can be made on an intraindividual basis.

Implications for EM Education

The principal benefit of the experience curve conceptual model for EM educators would lie in its explicit emphasis on longitudinal assessment of important skills. This has important consequences both during training and beyond. Assessments have more meaning when you know where you have been and where you are going. This notion of taking into account the learning path and not just isolated snapshots in time has been called for in several recent reviews of medical education research.[23]

Simulation-based skills training, whose dominant instructional strategy is deliberate practice, has penetrated EM residency curricula to a great extent.[2, 24-26] Working out the most effective simulation training curricula and schedules is one of the most important education tasks of this generation of EM educators.[27] Experience curves make clear that the development of skills during a training program can set the context for the use of simulation-based instruction and assessment for maintenance of competence during professional practice. For example, an intervention that “costs” two additional trainee hours, but saves one attending physician refresher training hour, might be advantageous considering the higher opportunity cost of attending physician time. Greater understanding of experience curves might also help predict those trainees who may not meet milestones. This might allow for individual learning plans and allow program directors to target interventions to milestones not necessarily tied to the date of residency graduation.

In the general education research literature, the longitudinal perspective on learning has led to the several fruitful lines of research. For example, the optimal learning technique is different when the emphasis is on skill durability rather than immediate posttest scores. In this regard, mixed practice (varied order and topic) is better than blocked practice, where one topic is “completed” before another is begun,[21, 28] and distributed practice is better than massed practice.[21, 29] EM simulation research that uses skill durability as an endpoint instead of short-term outcomes will be more consonant with the goals of educators. Recent advances in simulation center design and linkages with longitudinal databases of education data will greatly simplify the collection of long-term data.

We argue that experience curves can be useful as a conceptual framework that helps educators plan and assess training with an emphasis on skill durability. Despite the increasing standardization offered by simulation centers, differing interinstitutional contexts may make it difficult to map out precise slopes and parameters for every deliberately practiced EM skill. However, we could learn a great deal from investigating important representative skills. Additionally, on an intraindividual basis, mapping out experience trajectories can inform the type of self-regulation that has frequently been found lacking in physician continuing medical education programs.[30]


For clinical skills that can be deliberately practiced, we have presented the experience curve framework as a way of explicitly addressing longitudinal aspects of learning, while at the same time taking into account the decay of some types of knowledge with disuse. Research into the proper place of experience curves in EM education will be facilitated by the increasing use of simulation education with longitudinal tracking of outcomes.