SEARCH

SEARCH BY CITATION

Keywords:

  • learning curve;
  • education;
  • urology

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

Objectives

  • To describe how learning curves are measured and what procedural variables are used to establish a ‘learning curve’ (LC).
  • To assess whether LCs are a valuable measure of competency.

Patients and Methods

  • A review of the surgical literature pertaining to LCs was conducted using the Medline and OVID databases.

Results

  • Variables should be fully defined and when possible, patient-specific variables should be used. Trainee's prior experience and level of supervision should be quantified; the case mix and complexity should ideally be constant.
  • Logistic regression may be used to control for confounding variables.
  • Ideally, a learning plateau should reach a predefined/expert-derived competency level, which should be fully defined.
  • When the group splitting method is used, smaller cohorts should be used in order to narrow the range of the LC.
  • Simulation technology and competence-based objective assessments may be used in training and assessment in LC studies.

Conclusions

  • Measuring the surgical LC has potential benefits for patient safety and surgical education. However, standardisation in the methods and variables used to measure LCs is required.
  • Confounding variables, such as participant's prior experience, case mix, difficulty of procedures and level of supervision, should be controlled.
  • Competency and expert performance should be fully defined.

Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

The concept of the ‘learning curve’ (LC) was first described in aircraft manufacturing where the amount of man-hours required to produce a single unit decreased at a uniform rate as the production quantity was doubled [1]. In other words, performance improves with time and experience, and productivity is increased as a result (Fig. 1). A similar idea has been adopted in medicine and surgery, that learning a practical skill becomes easier with time, with an initial period of difficulty followed by a rate of improvement and stabilisation in performance.

figure

Figure 1. Graphical illustration of a LC with reference to number of times a task is performed and productivity.

Download figure to PowerPoint

The LC is particularly important in surgery where a constant stream of new skills must be acquired safely and efficiently. It would be useful to know how many procedures a surgeon may have to carry out before reaching a safe and competent level of performance. The issue of patient safety was highlighted by the UK General Medical Council Enquiry into the Bristol Paediatric Surgical Unit where concerns were raised about patients being exposed to surgeons in the early phase of the LC [2]. Knowledge about LCs could also enhance surgical training [3]. A surgeon may have to perform a procedure a certain number of times under supervision or in a simulated environment before performing it independently. Training programmes could use LC data to map progress of trainee surgeons [4]. Furthermore, an understanding of the LC is crucial in randomised control trials comparing new procedures with older interventions [5]. The effect of the LC of acquiring the new procedure must be considered in order to reach valid conclusions.

Whilst many studies have measured the LC of various surgical procedures, there is large discrepancy in the methods and variables used. Questions arise as to whether the LC is a valid measure of competency. This review of available literature aims to critically discuss:

  1. The variables used to measure the LC.
  2. Methods and statistical analysis of the LC.
  3. Whether LCs are a valid measure of competency.
  4. Recommendations will be provided based on findings and discussion.

Which Variables Should be Used?

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

An important aspect of measuring the LC is choosing the right variables. There are two main types of variables: measuring the surgical process or measuring patient outcomes [6, 7]. Measures of surgical process includes variables such as time to complete the procedure, the success or completion rate of the procedure, conversion rate from laparoscopic to open surgery, resection and margin involvement in cancer surgery etc [8, 9]. Measures of patient outcomes include amount of blood loss, length of hospital stay, intraoperative/postoperative complications, mortality etc [10, 11].

One issue is the discrepancy in the definition of variables between studies. For example, operating time is the most common variable used to measure LCs. When comparing studies investigating the LC of the same procedure (laparoscopic colectomy), two studies define it as the time from incision to completion of wound closure [8, 12], whilst one study describes it as time from incision to application of dressing [13]. Many studies did not define operating time at all [14-16]. The problem is greater with more complex patient outcome variables, e.g. complication rate. When comparing two studies investigating intraoperative complications in laparoscopic colectomy, both included visceral injury or bleeding, but one study only considered it only when additional procedures or transfusions were required [12] and another study included additional complications, e.g. anastomotic leak and failure to harvest an adequate number of lymph nodes [8]. Thus, the differences in the definition of variables make it harder to compare LCs between studies of the same procedure.

There is also debate as to which of the two types of variables: patient outcome or surgical process, should be measured. Some variables are easier to measure than others, but not all variables may be reliable measures of learning. For example, it is easier to measure and statistically analyse data on operating time than patient complications. Yet, a LC based on operating time alone may not be a useful indictor of good practice. It has been suggested that many variables must be considered to reliably measure a LC [17]. Also, variables should be tailored to the type of procedure. For example, operative mortality may not be a good indictor in low-risk procedures where patient satisfaction and quality-of-life measures may be more important. In cancer surgery, it may be more meaningful to measure long-term survival etc [17]. There may be no ideal measure of learning and a multi-variable approach should be used.

Finally, with any variable, confounding factors must be minimised to draw valid conclusions. Organisational factors (facilities, equipment), surgical team (experience, co-operation), case mix, complexity of cases and surgeon's characteristics (previous experience, natural abilities, motivation etc.) can all affect the LC of a procedure [5, 7]. Simulation and inanimate trainers may have a role in minimising some of these factors (e.g. organisational, case mix/complexity, surgical team etc.).

Another issue is that with increasing experience, surgeons may take on more difficult procedures, making LC analysis difficult. Surgeon's previous experience is often poorly defined in many studies [5]. Furthermore, many trainees are supervised by senior surgeons but it is not always clear as to how much supervision is being provided and whether procedures are being taken over by supervisors when performance is deemed less adequate. Awareness and full reporting of these confounding factors is crucial.

Methods and Statistical Analysis

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

It is vital to quantify the surgical LC using valid methods and statistical analysis. Graphical representation is the most popular method of displaying LC data. A simple plot of outcome against experience is commonly used but does not use any statistical analysis. Simple linear regression and various curve fitting methods have been used to derive a ‘learning curve’. The LC has a slope (representing the rate of learning), which may eventually plateau, suggesting that performance has reached a steady level. However, there is much variation and no particular rationale in the type of regression analysis or curve fitting model used (e.g. least-squares regression, logarithmic or negative exponential curves etc.) [18]. Also, reaching a plateau on the LC does not necessarily ensure competence; i.e. stability in performance does not equate to safe performance. Some studies have used an expert derived proficiency level as a standard in competency. If the LC plateaus at the line of expert performance, the trainee can be assumed to have reached competence at that predefined level. However, as discussed later, there can be problems with defining expert performance and competency. Another issue is that two trainees may have completely different LCs, making comparisons difficult. Furthermore, the LC may not reach a plateau if case numbers are not large enough to allow stability in performance.

Other methods include chronologically dividing the case number into consecutive groups to compare learning with increasing time and experience. For example, dividing 100 cases into quartiles and comparing performance in each quartile over time. The means between the groups are statistically compared using anova, chi-squared test etc. However, this group splitting method also has its limitations. The splitting of consecutive cases is arbitrary with no rationale for the cut-off points. Although it is ideal for comparing large case numbers, the size of groups may hinder LC interpretation. For example, the larger the group or cohort, the more difficult it becomes to pinpoint the exact number of cases required to overcome the LC. For example, if 100 cases were divided into quartiles and improvement in learning was seen in the second quartile, the LC may lie anywhere between 25 and 50 cases.

As mentioned previously, another issue in designing a LC study is accounting for confounding variables, such as variety in case mix and complexity; as well as organisational, patient, group and surgeon characteristics. It is very difficult to restrict the analysis or to use stratification to control for the large number of confounding variables in a dynamic setting such as the operating theatre. Multivariate logistic regression analysis could be used to control for multiple confounding factors, given a large enough sample size and dichotomous outcome measure [18]. For example, one LC study used multifactorial logistic regression analysis to control for patient, surgeon and procedure-related factors associated with conversion of laparoscopic to open surgery [19]. However, these statistical methods are not optimally used in the LC literature.

Finally, another statistical method that is gaining popularity within the LC literature is cumulative sum (CUSUM) analysis. This is a very useful sequential analysis method that detects change in the individual surgeon's performance. The trend in outcome is graphically represented as a straight line with acceptable performance and an upward slope with less than optimum performance. A trainee surgeon is expected to show a rising curve (the rate of learning), which may eventually plateau when performance stabilises [20]. Although, this is an excellent statistical method for detecting change and the possibility of adverse events, it is less good for comparing interoperator differences.

Are LCs a Valid Measure of Competency?

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

The main purpose of overcoming a LC is to achieve an acceptable level of competence. However, issues arise with defining, assessing and setting standards in competency. Firstly, the definition of competency is variable. Competency has been defined as not only being able to carry out a task but also the behaviour, personal characteristics and attributes that are developed to perform that task [21]. Some authors have distinguished ‘competent’ from ‘expert’. Competent individuals are able to consciously plan and carry out a task, whereas experts can carry out the task with flexibility and speed as well as adapt to various circumstances [22]. As well as technical skills acquisition, a competent individual must also combine cognitive, integrative, affective/moral, communicational and personal attributes [23]. Many studies of LCs do not assess competency in such a holistic manner and do not consider these additional features of competency. Thus, there is no universal definition of competency in surgery, which makes it difficult to gauge whether adequate competency has been achieved in overcoming a LC.

Also, how do we set the standard in competency when measuring LCs? Many studies have used the experience of senior surgeons as the expected standard of competency. One way of quantifying learning is to assume that a trainee surgeon has overcome the LC when they achieve a predefined level of expertise set by a more experienced surgeon [24, 25]. However, many studies do not report clearly the level of experience (e.g. number of procedures) of the senior mentors and there is huge variability in who is considered an expert [6]. This makes LC data difficult to compare. Studies that do not use an expert-derived benchmark can also be criticised because reaching a plateau on the LC may not be valid unless a baseline or control is provided for comparison.

We have discussed the lack of a universal definition of competency and the difficulty in judging whether LC studies assess competency. One of the main issues is that few outcomes or endpoints in LC studies will assess true competency; and fewer still will measure competency in a holistic manner, considering both technical and non-technical skills acquisition. For example, endpoints, e.g. operating time and blood loss, do not satisfactorily show that the surgeon has achieved competency. Complications and mortality may be better outcomes for assessing competency, as they show that the surgeon can perform the procedure safely. However, confounding variables, such as complexity of the case mix, may affect these outcomes. There may be no ideal endpoint in the current LC literature to assess true competency.

However, there may be a way to overcome the LC and show competency for a procedure. Most LC studies involve real patients, which in itself has ethical implications. Perhaps learning should shift from the operating theatre to a more controlled setting. In this way, inanimate trainers and virtual reality models can be used in overcoming the LC, followed by competency based assessments, e.g. checklists, objective structured clinical examinations (OSCEs) and objective structured assessment of technical skills (OSATS) [26]. Simulation is gaining increasing popularity in surgical training. Many types of high and low fidelity simulators are now available including synthetic, animal and cadaveric organ models to mechanical and virtual reality simulators. They allow the trainee to overcome LCs for complex surgical skills in a controlled environment, without jeopardizing patient safety. As well as training, these simulators can also act as tools for competency assessment in structured examinations. For example, an OSTATS may evaluate the trainee's performance on a simulator by the use of checklists and global scoring sheets. One study has used a competency based checklist to evaluate the learner's performance of flexible laryngoscopy on a mannequin [27]. Thus, simulators and objective structured assessments can be used to overcome LCs and assess competency. Although, disadvantages with simulation is the level of validity and skills transfer to the real environment. In addition, high fidelity simulators can be costly.

Assessment of progress of individual surgeons also has medico-legal relevance. There has been recent interest in tracking performance of individual surgeons with a move towards publishing performance data within the next 2 years in England [28]. The change is fuelled by concerns over inadequate performance as in the Bristol Paediatric Unit enquiry, where there were unacceptably high numbers of deaths [2]. It is hoped that publicising performance data will encourage individual surgeons to keep abreast of new skills and seek help from colleagues when necessary. However, the ‘league table’ style assessment may deter surgeons from taking on more high-risk cases and allowing junior surgeons to assist on complex surgeries. Whatever the verdict, the importance of LC analysis and competency assessment is especially highlighted in this debate. Increased availability of LC data and valid assessment of LCs is likely to be highly relevant in the future.

Recommendations

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

The following recommendations are made based on the above discussion:

  • Variables should be fully defined and when possible, patient-specific variables should be used.
  • Trainee's prior experience should be quantified; the case mix and complexity should ideally be constant; the level of supervision received should be fully described.
  • Ideally, a learning plateau should reach a predefined competency level. When this is expert derived, previous experience of experts should be fully quantified and the study should include a detailed outline of how expert performance was measured.
  • When the group splitting method is used and large numbers of cases are compared by cohorts, smaller groups should be used in order to narrow the range of the LC.
  • Statistical analysis, e.g. logistic regression, may be used to control for confounding variables, given a large population and dichotomous outcome measure.
  • Simulation technology and competence-based objective assessments may be used in training and assessment in LC studies.

Conclusions

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References

Measuring the surgical LC has potential benefits for patient safety and surgical education. However, standardisation in the variables, methods and statistical analysis is required. Confounding variables, e.g. participant's prior experience, case mix, difficulty of procedures and level of supervision etc., should be controlled. Competency and expert performance should be fully defined. There is concern that endpoints currently used in the LC literature do not measure competency. There is an emerging role for simulation technology in the training and assessment of surgical skills and structured objective assessments may have a role in assessing competency. Recent concerns over tracking and publicising individual surgeon's performance makes LC analysis a very relevant issue.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Which Variables Should be Used?
  5. Methods and Statistical Analysis
  6. Are LCs a Valid Measure of Competency?
  7. Recommendations
  8. Conclusions
  9. Conflict of Interest
  10. References
Abbreviations
LC

learning curve

OSATS

objective structured assessment of technical skills