SEARCH

SEARCH BY CITATION

Keywords:

  • clinical competence;
  • education;
  • graduate;
  • medical;
  • teaching;
  • ureteroscopy

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

Aim:  Virtual reality (VR) simulators are now commercially available for various surgical skills training. The Uro Mentor VR Ureteroscopy Simulator by Symbionix is one system that may revolutionize the way we assess and teach surgical residents. Surgical educators may no longer have to depend on the operating room as the sole venue for teaching residents technical skills. We validated performance on this new system with previously developed assessment tools and compared it to performance on a high fidelity ureteroscopy bench model.

Methods:  Urology residents (n = 16) were assessed on their ability to perform cystoscopy, guidewire insertion, semirigid ureteroscopy and basket extraction of a distal ureteric stone on the VR simulator. A blinded examiner assessed subject performance using a checklist, global rating scale and a pass/fail rating. In addition, computer-generated parameters including time to complete task, scope and instrument trauma and the number of attempts to insert a guidewire were analysed. Performance on the VR simulator was compared to performance on a high fidelity ureteroscopy bench model.

Results:  Senior residents (n = 8) scored significantly higher on their global rating scale (29.4 ± 2.5 vs 20.8 ± 0.9, P = 0.005), checklist (19.1 ± 1.1 vs 15.2 ± 0.9, P = 0.02), pass/fail rating (χ2 = 7.3, P = 0.007) and required less time to complete the task (352.9 ± 55.7 s vs 576.8 ± 67.4 s., P = 0.02) than the junior residents (n = 8) on the VR simulator. Junior residents also had a significantly higher incidence of scope trauma (4 vs 0.6, P = 0.02). No significant differences were noted in instrument trauma and the number of attempts to insert the guidewire. Global rating scale performance on the VR simulator correlated well to performance on the high fidelity ureteroscopy bench model (r = 0.7, P = 0.002) as did time to complete task (r = 0.7, P = 0.004).

Conclusions:  The Uro Mentor VR Ureteroscopy Simulator is a useful tool in assessing resident endourological skills. Performance on the VR simulator is comparable to a validated high fidelity ureteroscopy bench model. Future studies will assess the utility of VR simulators in surgical skills training.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

With increasing financial and time constraints, surgical educators are beginning to consider other methods of teaching surgical skills to maximize resident operating room experience.1 Laboratory surgical skills training has been viewed as an option for some of the more complex skills such as laparoscopy and endourology where learning curves tend to be steeper than with open surgery.2,3 Both low fidelity and high fidelity bench models have been used for surgical skills training in a laboratory environment.4,5 More recently, with technological advances, virtual reality simulators have been introduced to the commercial market as teaching tools. Analogous to flight simulators for pilots, virtual reality surgical simulators have been viewed as the future for training and evaluating surgical skills6 and we are now beginning to see the impact of such trainers in urology.7,8 Initial studies have demonstrated the relative merits of virtual reality (VR) training on ureteroscopic skills in medical students. The objectives of our study were to validate the Uromentor VR Ureteroscopy Simulator by comparing it to a previously validated ureteroscopic bench model and to see whether urology resident experience level can be discerned by performance on this particular simulator.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

Over a 3-day period (13–15 December 2001), 16 residents recruited from the University of Toronto’s urology training program participated in this study to validate a VR ureteroscopy model. All residents gave informed consent prior to their participation and were asked to submit their prospectively completed ureteroscopy log form, which documented the number of ureteroscopic cases they had done during their urologic training to date.

Equipment

A new virtual reality suite was created at the Kidney Stone Center at St. Michael’s Hospital (Toronto, Canada). The simulator utilized for the study was the Simbionix (Tel Aviv, Israel) Uromentor VR ureteroscopy simulator. At the time of the study, this model was the latest model available from Symbionix (Fig. 1). The VR simulator consisted of a central processing unit, a touch-sensitive liquid crystal display flat screen, a sensor module with latex penis and various instruments including a rigid cystoscope, semirigid ureteroscope, flexible scope, basket, grasper and wires.

image

Figure 1. Figure of a participant performing flexible ureteroscopy using the Uromentor VR Simulator (Simbionix, Tel Aviv, Israel). System consists of a male genital with sensors, sensor box, central processing unit and liquid crystal display monitor. Two scopes are available: a flexible scope and a rigid scope. Virtual instruments include: wires, laser fibers, baskets and graspers.

Download figure to PowerPoint

Orientation phase

All residents had participated in earlier studies at the Surgical Skills Center and were familiar with ureteroscopy and the instruments used during a stone manipulation.1 All residents underwent an exercise drill on the simulator called ‘basketball’ because of the novel technology being used by the VR simulator The drill consisted of picking up basketballs in a virtual reality court and dropping them through a basket and allowed the trainee to become familiar with the virtual reality environment.

Evaluation phase

The residents were given case number 10 from the training scenarios. The task they were asked to complete was the endoscopic management of a mid-ureteric stone. They were expected to do a complete cystoscopy, insertion of a guidewire, ureteroscopy and basket manipulation of a mid-ureteric stone. This was similar to a task completed on a high-fidelity ureteroscopy bench model during an earlier surgical skills study. This allowed good comparison of performance on the two different modalities of training. The high-fidelity ureteroscopy model (Limbs and Things, Bristol, UK) consisted of a latex penis, urethra, bladder, trigone, ureter and renal pelvis (Fig. 2). During the evaluation, an examiner, blinded to the level of resident training, assessed the performance of each resident using a validated checklist, global rating score and pass/fail rating (see Appendices I and II).3 The checklist was modified by eliminating the ‘shuts off water’ during the cystoscopy and ureteroscopy phase as the VR model itself did not require the irrigation to run continuously in order to visualize endoscopically.3 In addition to the objective structured assessment of performance, virtual reality parameters were also analysed. These parameters included: scope trauma, instrument trauma and the number of attempts to insert the guidewire. Time to complete the task was measured by the VR system.

image

Figure 2. Example of a high fidelity non-virtual reality ureteroscopy bench model.

Download figure to PowerPoint

Residents were also asked to complete a poststudy questionnaire consisting of visual analog line assessing the realism and the effectiveness of the VR model.

Statistical analysis

Statistical analysis was performed using SPSS 10.0 (SPSS, Chicago, IL, USA). Student’s t-test was used to compare virtual reality ureteroscopy performance between junior and senior residents. The χ2-test was used to analyse pass ratings between these two groups. Pearson’s r was used to correlate number of ureteroscopic procedures logged and performance on the VR ureteroscopy model. Pearson’s r was calculated to determine the correlation between performance on the VR ureteroscopy model and the high-fidelity latex ureteroscopy bench model. Means were shown with standard error.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

Eight junior residents (postgraduate year 1–3) and eight senior residents (postgraduate year 4–5) participated in the study. The 14 male and two female subjects were all right-handed and aged 26–31.

The junior residents had logged an average of 1 ± 0.6 and the senior residents had logged an average of 6.4 ± 2.3 ureteroscopic cases prior to the study. There was a good correlation between the number of ureteroscopic cases logged and VR global rating scores with Pearson’s r = 0.63 (P = 0.008).

There was a significant difference between junior and senior urology residents’ VR scores with respect to the global rating score, checklist scores, pass/fail rating and time to complete task (Table 1).

Table 1.  Junior versus senior resident performance on the Uromentor VR simulator
 Junior residentSenior residentP-value
  • t-test;

  • χ2-test (±standard error). VR, virtual reality.

Checklist score15.3 (±0.9)19.1 (±1.2)0.02
Global rating score20.8 (±0.9)29.0 (±2.5)0.002
Time to complete task (s) 578 (±67) 353 (±56)0.02
Scope trauma 4.0 (±1.2) 0.6 (±0.4)0.02
Guidewire insertion attempts 5.9 (±3.0) 2.0 (±0.8)0.23
Instrument trauma0.25 (±0.2)  0 (±0)0.15
Percent passing38%100%0.007

Junior residents registered a higher incidence of scope trauma as measured by the VR system. There were no significant differences in the incidence of instrument trauma or the number of attempts to insert a guidewire as measured by the VR system (Table 1).

We compared performance on the VR system to performance on a validated high fidelity bench model. Residents had previously performed an ureteroscopic extraction of a mid-ureteric stone on a Limbs and Things ureteroscopy bench model. There was good correlation between the VR model and the high fidelity bench model with respect to the global rating score (Pearson’s r = 0.72, P = 0.002) and time (Pearson’s r = 0.67, P = 0.004).

Residents were also asked if the UroMentor VR Trainer was a better teaching model than the high fidelity bench model: 12.5% percent strongly agreed, 31.3% agreed, 25% felt no difference and 31.3% disagreed. Residents were also asked if the UroMentor Endourology VR Simulator was a more realistic environment than the high fidelity ureteroscopy bench model: 50% strongly agreed, 37.5% agreed, 12.5% felt no difference and 0% disagreed.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

With advances in computer technology, the VR surgical simulator is no longer in the realm of science fiction but is a useful tool that can be used by surgical educators both for the assessment and teaching of technical skills. The strengths of the VR simulators are repetitive practice without time constraints, no patient involvement and the opportunity for trainees to learn from mistakes in a safe environment. In addition to expert feedback which may occur during training, the VR simulators also offers an advantage by providing formative feedback by summarizing performance parameters including: time to complete task, number of instrument trauma and scope trauma. This type of feedback allows the trainee to measure their performance and make the necessary changes to their technique. More importantly, it allows training in a setting that is not supervised by a more senior surgeon. Bench model training frequently requires a mentor to be present to orient trainees to the instruments, model, and to troubleshoot problems as they arise. The VR simulator facilitates unsupervised training by providing such feedback and monitoring of complications without a mentor present.

Our study has demonstrated that the Uromentor VR ureteroscopy simulator trainer is a valid instrument that can be used to assess the ureteroscopic skills of urology residents. From a psychometric standpoint, we were able to demonstrate construct validity between junior and senior residents using the Uromentor VR ureteroscopy simulator as an assessment tool. Furthermore, there was good correlation between the number of ureteroscopic cases logged and the VR global rating score. This finding is in keeping with other studies which have demonstrated the importance of global rating scores in assessing surgical performance.9 Presently, global rating scores have been demonstrated to be the most powerful measure of surgical skills and supports the concurrent use of global rating scales during performance assessment on the Uromentor VR ureteroscopy system. Performance on the Uromentor VR ureteroscopy simulator also correlated well with performance on a previously validated ureteroscopy bench model, which further demonstrates the value of this system as an assessment tool. Furthermore, this supports the predictive validity and content validity of the virtual reality trainer. This study did not explore the utility of the Uromentor VR ureteroscopy simulator as a training device; however, future studies are planned for correlating performance on this virtual reality model and actual performance in the operating room.

There are some limitations associated with the current VR simulators. The first is haptic feedback. This refers to the application of tactile feedback and control through an interface, such as a grasper, endoscope or basket, to interact with the virtual environment. Complex modeling calculations are necessary in order to simulate the tactile sensation of instruments interacting with virtual tissue. To achieve this, these calculations have to be translated into a mechanical output via servos and rollers attached to the instrument or user interface which provides the appropriate tactile sensation. A second limitation to VR simulators is the display system. Endoscopic procedures lend themselves well to virtual reality endeavors through the use of a video monitor display system. Simulators of this nature are simply replicated through the use of a computer monitor. Display systems become more complicated for open procedures which require a 3-D display system. Such display systems are being developed; however, the display system usually requires a bulky headpiece in order to simulate an open surgical field. Unlike endoscopic procedures which utilize up to four or five movement axes, open surgical procedures are complicated by a greater number of axes due to the interaction of the instrument, fingers, hand and wrist. The final limitation to be discussed is the fidelity or ‘realism’ of the simulator. The fidelity is dependent on several factors including, but not limited to, programming, processing power, memory and video card performance. One of the goals of VR simulation is to provide the user with the most accurate rendering of anatomy and to simulate realistic responses to instrumentation. Performance on a VR simulator should be compared to performance in patients in an operating room environment. However, variables such as anatomical variation and more importantly patient safety limit such comparison. Our study did compare performance to a previously validated surgical bench model and we have shown similarities in performance between models.5 From an anatomical simulation standpoint, 87.5% of the subjects felt that the Uromentor VR ureteroscopy simulator was better than the bench model; however, only 43.8% of the subjects felt that the Uromentor VR ureteroscopy simulator was better than the bench model in terms of a teaching tool. Subjective comments included ‘feedback needs to be improved’ and ‘would like more tactile sensation when using the instruments’.

Virtual reality simulators are appealing from a surgical educator’s perspective. Training on a simulator will allow a trainee to become familiar with the pertinent steps of the procedure and the instruments necessary to carry out the procedure. This should help prepare the resident for the operating room and maximize their learning experience. The current ‘apprenticeship teaching model’ has had residents perform their first procedure on live patients for the first time, albeit in a well-supervised environment, which can potentially increase operating time and risk to the patient. Well-designed training in a surgical skills center with appropriate feedback will minimize error when it comes time to operate on a real patient. We feel that this model of surgical training is a step up from the classic ‘Halsteadian’ model, and from a patient’s perspective, provides an increased margin of safety.

Conclusions

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

Despite the need for ongoing refinement, the Uromentor VR ureteroscopy simulator will fulfill an important role in assessing and teaching endoscopic skills. In our study, we have demonstrated that the Uromentor VR ureteroscopy simulator is a valid tool for assessing resident ureteroscopic performance and that this new virtual reality model compares favorably to a previously validated ureteroscopy bench model.

Acknowledgments

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

We would like to thank Patrick Harty and Cook Urology Canada for their resources, which made this study possible.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

Appendices

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusions
  8. Acknowledgments
  9. References
  10. Appendices

Appendix I

Checklist. Note: items 8 and 19 not utilized during virtual reality assessment
Ureteric Stone Checklist
Candidate Identification No:
Instructions to Candidate: you are asked to remove a (left or right) mid-ureteric stone
Items Not done /Done correctlyDone incorrectly
Cystoscopy
1.Inserts cystoscope while holding penis at 45–60 deg. angle (if not please advise or assist)01
2.Supports cystoscope at the lens and camera connection. (if not please advise or assist)01
3.Keeps scope in the centre of the urethra, and does not bump into wall. (if not advise or assist)01
4.Visualizes verumontanum and levels out scope prior to entering bladder. (if not please advise or assist)01
5.Enters bladder without bumping into wall. (if not please advise or assist)01
6.Identifies ureteric orifices01
7.Examines all quadrants of bladder by rotating scope01
8.Shuts off water flow to scope. (if not please advise or assist)01
9.Passes guidewire through scope via working port and intubates the correct ureteric orifice (intubates in less than 3 tries)01
10.Advances guidewire until there is slight resistance01
11.Removes cystoscope without dislodging guidewire01
Examiners – Please detach camera and water tubing from the cystoscope and attach them to the ureteroscope. Please make sure the camera is in focus before beginning next part
Ureteroscopy
12.Inserts ureteroscope while holding penis at 45–60 deg. angle. (if not please advise or assist)01
13.Supports cystoscope at the lens and camera connection. (if not please advise or assist)01
14.Keeps scope in the centre of the urethra, and does not bump into wall. Keeps guidewire in view. (if not advise or assist)01
15.Visualizes verumontanum and levels out scope prior to entering bladder. (if not please advise or assist)01
16.Identifies ureteric orifice by following guidewire01
17.Inserts ureteroscope into correct ureteric orifice. (in less than 3 attempts)01
18.Keeps ureteroscope centered with guidewire in view while maneuvering up to stone. (if not please advise or assist)01
19.Finds stone and shuts off water flow01
20.Passes basket until sees the tip01
21.Pulls scope back and advances the basket, then opens and closes the basket to make sure it is functioning01
22.Closes basket and passes tip of basket past the stone, keeps scope approx. 2 cm from the stone when working01
23.Opens the basket and traps the stone01
24.Closes basket and removes stone by withdrawing scope and basket together. Keeps basket in view and follows the contour of ureter and urethra while withdrawing. (if not advise or assist)01

Appendix II

Global rating scale

Please circle the number corresponding to the candidate’s performance in each category, irrespective of training level.

Respect for tissue12345
Scope frequently pushed into urothelial wall. Used unnecessary force with guidewire and/or basket Scope occasionally pushed into urothelial wall. Careful handling of guidewire and/or basket for the most part No trauma to urothelial wall with scope. Consistent and careful handling of guidewire and/or basket
Time and motion12345
Many unnecessary moves Made some unnecessary moves but time more efficient No unnecessary moves and time is maximized
Instrument handling12345
Needed to repeatedly attempt guidewire insertion and/or basketing of stone Able to insert guidewire and basket stone within first few tries. Occasional awkward maneuver Able to insert guidewire and basket with fluid motion and no awkwardness
Handling of endoscope12345
Frequently had scope pointing away from the center of the urethra or ureter. Scope poorly aligned during procedure Had scope centered for the most part. Guidewire in view for the most part. Better use of scope angle during procedure Scope always centered and guidewire always in view. Scope always set at a good angle throughout procedure
Flow of procedure and forward planning12345
Frequently stopped or need advice or assistance from examiner Demonstrated the ability to think forward with relatively steady progression of procedure Obviously planned procedure from beginning to end with fluid motion
Use of assistants12345
Failed to have assistants help with guidewire insertion and/or stone basketing Appropriate use of assistants most of the time Strategically used assistants to the best advantage at all times
Knowledge of procedureDeficient knowledge. Needed specific instruction at most operative steps Knew all important aspects of operation Demonstrated familiarity with all aspects of operation
Pass rating

Would you feel confident in allowing this trainee to perform this procedure in the operating room?

YES NO