Development and validation of non‐guided bladder‐neck and neurovascular‐bundle dissection modules of the RobotiX‐Mentor® full‐procedure robotic‐assisted radical prostatectomy virtual reality simulation

Abstract Background Full‐procedure virtual reality (VR) simulator training in robotic‐assisted radical prostatectomy (RARP) is a new tool in surgical education. Methods Description of the development of a VR RARP simulation model, (RobotiX‐Mentor®) including non‐guided bladder neck (ngBND) and neurovascular bundle dissection (ngNVBD) modules, and assessment of face, content, and construct validation of the ngBND and ngNVBD modules by robotic surgeons with different experience levels. Results Simulator and ngBND/ngNVBD modules were rated highly by all surgeons for realism and usability as training tool. In the ngBND‐task construct, validation was not achieved in task‐specific performance metrics. In the ngNVBD, task‐specific performance of the expert/intermediately experienced surgeons was significantly better than that of novices. Conclusions We proved face and content validity of simulator and both modules, and construct validity for generic metrics of the ngBND module and for generic and task‐specific metrics of the ngNVBD module.

Simbionix Products, Cleveland, OH, USA), is a robotic surgery VR simulator that has been developed to train surgeons using the da Vinci® Surgical System (Intuitive Surgical, Sunnyvale, CA, USA). The simulator platform consists of a height adjustable headset containing stereoscopic visors, free-floating hand controls, and adjustable foot pedals integrated into a single console. It has been proven by Whitaker et al. 3 to be effective for training using the FRS curriculum, a basic VR module. Currently, very few full procedure VR simulations exist to train trainees in performing all aspects of a robotic procedure. Development of non-guided procedural training is likely to become an integral and essential part of modern urological robotic training curricula. Developments in VR techniques and affordable hardware have led to the production of full procedural VR simulations. Full procedural VR simulations have the potential to train surgeons in advanced technical, nontechnical, and cognitive skills, as well as in operation-specific surgical steps that cannot be realized in basic VR modules. Before implementing full procedural VR modules into a robotic urological training curriculum, face, content, and construct validity of each step within a module need to be investigated and evaluated. Recently, Harrison et al. 4 could proof construct validity of the RobotiX Mentor®'s guided bladder neck dissection (BND) and the urethrovesical anastomosis (UVA) step of the robotic-assisted radical prostatectomy (RARP) module.
The aim of this prospective study was to describe the development of the non-guided bladder neck dissection (ngBND) and nonguided neurovascular bundle dissection (ngNVBD) in a VR simulation model (RobotiX Mentor®), and to perform the first assessment of face, content, and construct validation of the ngBND and ngNVBD in the full procedural RARP training module of the RobotiX Mentor® robotic surgery VR simulator.

| Study design
The project was carried out in two phases: (1) the design and development of the VR RARP simulation and (2) its evaluation and validation. In the development phase, multiple high definition (HD) videos of RARP, carried out by one expert high-volume surgeon from Karolinska University Hospital, Stockholm, Sweden, were recorded.
The procedure was divided into phases, one of which included the bladder neck dissection (BND) and one the neurovascular bundle dissection (NVBD). Using task deconstruction, each phase was broken down into defined tasks to be completed and the associated anatomical landmarks. Then, the tasks were defined in an objective 'binary' manner, and within each task, important visual cues, surgical errors, and events to be avoided were identified. The VR simulation was then built, taking into account these defined metrics. Table 1 describes the important visual cues to be identified in the VR simulation, as well as errors and events identified as important metrics for surgical performance. These metrics were integrated into the automated scoring system of the RobotiX Mentor® that rates the surgeon's performance.
In the next phase, we aimed to evaluate both the RobotiX Mentor® console and its modules of ngBND and ngNVBD as parts of the full procedural VR RARP training module. This evaluation and validation was performed as a prospective, observational, and comparative study.

| Participants
Subjects were categorised into three groups (novices, intermediates, and experts). Categorisation was based on the number of procedures required to reach proficiency in RARP.
Experts were individuals who had performed at least 100 RARP independently, were experienced as trainers, and who were presently and regularly performing RARP. 5 The intermediately experienced group included residents and fellows receiving surgical training who had performed less than 100 cases. The novice group consisted of individuals who had no previous console operative experience, but may have had tableside assistance experience. Participants were recruited at Karolinska University Hospital and Medical School, Stockhom, Sweden, and at the 13th edition of the meeting of the EAU Robotic Urology Section (ERUS).

| Power analysis
The power analysis was calculated with a two-tailed test, with α ¼ 0.05 and power (1 À β) ¼ 0.80, and an intended reduction of 30% in time taken to complete each of the various tasks and stages for highly experienced robotic surgeons (more than 500 independent RARP cases), versus moderately experienced surgeons (20-50 RARP cases), versus robotic naïve medial students, based on data from previous studies. [6][7][8] This calculation revealed that 8 subjects per group would be sufficient for finding statistically significant differences, which we increased to at least 15 per group to allow for the possible occurrence of dropouts and technical malfunctions of the simulator, and to compensate the less restrictive definition of expert level with at least 100 prostatectomies performed.  � "Fundamentals of Robotic Surgery -Vessel Energy Dissection." � Non-guided BND (full bladder-neck sparing) for study purposes (one time).

| Curriculum
� Non-guided NVBD (full nerve-sparing) for study purposes, both sides, start with right side (one time).
The study protocol was in compliance with the Declaration of Helsinki and in accordance with the regulations of the local Ethics Committee of the Karolinska Institutet, which approved the study; it was discussed and decided at the local institutional ethical review board that no formal ethical assessment is required as it does not involve patients. However, all participants were given written information on the study and asked to sign a participation consent form.
All data collected was anonymised with no individual reports.
All novice surgeons initially underwent basic skills training on the RobotiX Mentor® console, which consisted of guiding the novices through the controls and teaching basic robotic skills.
Intermediates and experts were offered the opportunity to complete a "familiarisation" task prior to commencing the procedural study modules. No data was collected from the familiarisation tasks. All participants then performed the ngBND task, aiming for full bladder neck-sparing ( Figure 1(A-C)). The participants were asked to accurately find the right plane between prostate and bladder before starting with the ngBND; they should clear the lateral periurethral parts (Figure 1(A)) before opening the urethra (Figure 1(B)), and were told to control the task for bleeding by using the suction mode and bipolar coagulation. Monopolar energy was not provided for the scissors. A fourth arm was provided to control the catheter with a forceps (Figure 1(C)). The ngBND task was followed by the ngNVBD task, aiming for full nerve-sparing ( Figure 2 Likert scale was used for scoring (1 ¼ worst ranking, 5 ¼ best ranking).
All data are presented as median with interquartile range (IQR).

| RESULTS
A total of 51 participants completed the study; their demographics are shown in Table 2.

| Evaluation of the ngBND task performance:
In general, the intermediately experienced surgeons performed the task most accurately, followed by the experts and the novices, but these differences were not statistically significant (97. 5  The number of movements of the right, left, and fourth instruments decreased significantly with increasing degrees of expertise, and was therefore lowest in the expert group. Additionally, the total moving distance of the right and left instruments became significantly smaller with increasing expertise. Experts  Table 3.

| Evaluation of the NVBD task performance:
The numbers of movements of the right, left, and fourth instrument were lowest in the expert group; statistically significant differences were found for the movements of the right instrument (p ¼ 0.01) and fourth instrument (p ¼ 0.02). The total moving distance of each instrument arm was smallest in the expert group, but the difference between the groups was statistically significant only for the fourth arm (p ¼ 0.045). In addition, no significant differences were recorded for the number of instrument collisions.

| Evaluation of realism of the RobotiX Mentor® simulator and modules
Overall, participants in all three subgroups assessed the realism of the RobotiX Mentor simulator, the ngBND, and the ngNVBD task as comparable to the da Vinci® Robot.
The overall ratings and the subgroups' ratings for the RobotiX Mentor® simulator and both modules are presented in detail in Table 5.
The ratings for realism of the ngBND task were significantly different among the three participating groups concerning: bleeding/  Dissection into prostate, median (IQR)

| Validation of the RobotiX Mentor® simulator and console:
Overall, the acceptance rate of the RobotiX Mentor® simulator was high and it was rated as a valuable and feasible training tool by all three groups, but novices tended to award higher ratings in almost every category than expert and intermediate candidates.
In addition, the usability of the console received high ratings from all participating groups. However, experts appraised the use of the hand pedal and the diathermy pedal significantly worse than intermediates and novices. Also, the realism of the graphics was rated lower by the experts, but the difference was not statistically significant.
These results, along with the importance of several tasks for robotic surgical training, are shown in Table 6.

| Validation of robotic simulator training:
The importance of robotic simulator training was mentioned by all three groups, with no significant difference. Of those having answered the question, all participants in each group agreed with the statement that simulation should be implemented into training programs.  Table 7.

| DISCUSSION
We Nonetheless, the training effect that can be achieved by the simulator certainly reaches its limits. This is reflected by the finding that novices showed a stronger, and experts a less pronounced, improvement in their robotic skills, as compared to intermediates.
This result supports face validity of the simulator system.
For the ngBND task, significant differences between the differently experienced groups was mainly seen in the generic automated performance metrics (APM) (e.g. less instrument collisions, less movements, and smaller moving distance by instruments with increased expertise), and also by the total time to finish the module, which decreased with the level of expertise, but not in task-specific performance metrics. These findings are similar to APM assessments on the actual da Vinci® Robot. 9 Interestingly, the total moving distance of the fourth arm was much smaller but was moved more often by the novices, representing an inefficient/more uncontrolled usage of the fourth arm in this group. Recently, APMs, defined as instrument motion tracking metrics and synchronized surgical footage, captured with a novel datarecording device, the "dVLogger" (Intuitive Surgical), directly from the da Vinci® robotic system in real time during the actual live surgical procedure, were investigated concerning their ability to assess surgical performance and patient outcomes. The data revealed that experts showed more efficient camera manipulation and smaller total distance between the two instruments (dominant and nondominant) in selected steps of RARP (BND and NVBD not investigated). 10 Another study on robotic VR simulation exercises also suggested that expert surgeons had significantly less total camera moving time but higher frequency of camera movement than new robotic surgeons. 11 Correspondingly, in our study, the total distance between the right and the left instrument was smallest in the expert group in both the ngBND task and ngNVBD task. The distance moved by camera did not differ significantly between the three groups in requirements, especially in situations where tissue starts moving. As described for the urethrovesical anastomosis task, 4 some participants described "floating clips" and described the tissue as "jelly-like." Consequently, problems with the setting of the clips to control the vascular pedicles in the NVBD module might have been the reason for the low rating of this metric in all three groups, but the lowest in the expert group. However, the realism of moving tissue and the realistic behaviour of tissue towards other objects, like clips, needles, or instruments, is demanding and represents a remarkably weak point with a potential for improvements in advanced and VR medical simulators, not only in the RobotiX Mentor® simulator. However, future products will allow for better processing, and new graphic cards will allow manufacturers to improve tissue behaviour.
A wide variety of procedures are conducted in urology, and thus a variety of simulation modalities have become available. 12 Simulation-based training is increasingly recognised as a valuable adjunct to training in urology and other disciplines. However, studies investigating the validity and usefulness of procedural VR simulations are rare and mainly limited to endourological and laparoscopic procedures. 13 To our knowledge, 3D Systems Simbionix is the only company that provides complete robotic procedure simulated cases, which enables the performance of critical steps of various procedures, including RARP. 3,4 This structured method of training, called modular training, is supported by a good evidence base in urology, 14 and numerous suggested training modules exist for minimally invasive procedures. 13 In real live surgery, the recent development of a modular assessment tool for RARP allows for evaluation of learning curves for individual steps of the procedure, which provides mentors with an objective way to assess progression through the modules. 16 In VR, full procedure modular training competency can be assessed by developing benchmark scores based on experts' performance. 17 Consequently, the data of this study can be further used to establish a benchmark-based modular training curriculum of full procedural radical prostatectomy modules. In addition, the data collected from the robot naïve medical student group will also provide further information on the variation in baseline "inherent" skills in VR robotic simulation. In future, utilizing machine learning and APMs to evaluate RARP performance in live surgery and predict patient outcomes 9,18 might influence the assessment of candidates performing VR full procedural RARPs.
However, the assumed additional benefit of VR full procedure radical prostatectomy modules in comparison to basic VR training needs to be further investigated.
The role of nontechnical skills in surgery is increasingly understood and represents an expanding area within the literature. It is now known that deficiencies in this area are a major source for surgical errors, and it is increasingly recognised that these skill sets are not peripheral but rather should be seen, alongside with technical skills, as being of core importance. 13 The ngNVBD module of the RobotiX Mentor® simulator uses a virtual laparoscopic assistant arm, which is currently guided through a functional icon wheal by the console surgeon. The virtual assistant arm was highly rated in terms of realism, especially by the expert group. Nevertheless, we recommend implementing a voice control of the virtual assistant arm. This would provide the opportunity to also simulate a very important nontechnical skill, defined as "social skill," including communication, teamwork, and leadership. 13 Furthermore, the study participants agreed to implement simulation into training programs, whereas they disagreed on it being part of accreditation/(re)-certification. This reflects the strong need to implement advanced VR full procedural prostatectomy modules, beside basic skills modules, into RARP surgical training curriculums, but at the same time raises the question of where to adequately place them to gain the most benefit in the surgical education and quality management process. The fact that some specific construct validity results, like the "achieved percentage of nerve sparing," were already EBBING ET AL. In our study, no guiding was applied during the tasks, but while performing the familiarisation tasks, all participants were informed about the relevant characteristics of the coded metrics in the study to prevent from coding-related bias. However, the fact that, apart from the total performance time, no statistically significant differences in the ngBND task-specific performances were found between the groups might be explained by this circumstance. On the other hand, Harrison et al. 4 who validated the guided BND module of the RobotiX Mentor® simulator, had experienced problems to prove construct validity of the guided BND task, too. They argued that some intermediates and experts questioned the computer's technique for the task and were hesitant to perform the task as per the instructions provided by the simulator, whereas the novices followed the instructions readily. 4 Experts and intermediates were defined as surgeons who had independently performed at least 100 or less than 100 RARP, respectively. Actually, the median (IQR) number of performed RARP results, but it also reflects the long learning curve that is necessary to achieve good oncological and functional results in RARP. 5,9,19,20 However, we performed the same analysis with expert level defined as 50 RARP, and found no relevantly different results.
In summary: novice, intermediate, and expert RARP surgeons evaluated the ngBND and ngNVBD full procedural VR training module of the Simbionix MentorX simulator and approved it as a realistic, feasible, and acceptable component for a robotic surgical training program. Construct validity was proved for generic performance metrics but not for task-specific metrics in the ngBND module, and for both metrics in the ngNVBD module. Novices showed an increase in the percentage of nerve sparing between the right and the left side of the NVB, which indicates a significant learning effect.
Since validity has been described as a continuing argument, 21 we believe our work makes an important contribution to the ongoing validation of the RobotiX-Mentor® full-procedure RARP VR simulation by describing important metrics identified to define surgical performance during VR RARP simulation, by analysing the performance of the study participants, and by comparing our results with previous validation results. However, validation is determined by evidence. 21 Thus, further implementation of these modules into the curriculum and continued analysis would be beneficial to gauge its overall usability.