Development and content validation of the percutaneous nephrolithotomy assessment score

To develop and content validate a percutaneous nephrolithotomy assessment score, taking into consideration the procedure‐specific risks.


Introduction
PCNL remains the first-line treatment for patients with large renal calculi (>20 mm) or when alternative treatments, such as ureteroscopy or ESWL, have failed. 1,2 With increasing challenges facing surgical training, including working time restrictions, there is increasing demand for objective assessment of surgical competence. Checklists are objective assessments that can be utilized to ascertain whether the important steps of a procedure have been accomplished. Checklists should also address key safety items that, if omitted, could lead to an adverse outcome.
At present, there is no PCNL-specific intraoperative assessment tool available to evaluate the performance of trainees. Procedure-specific checklists have been implemented in other high-risk urological procedures, such as LRN 3 and RARP, 4 with the RARP assessment tool being utilized to identify the learning curves of the procedure. These tools can be used to monitor the progress of trainees and identify the learning curves of a procedure, which will ultimately guide the focus of their training going forward to ensure they are competent to complete all aspects of the procedure.
Failure mode and effect analysis has been applied in numerous high-risk industries, including the aerospace industry, to identify issues within a process before an adverse event occurs; this technique has now been adapted for use in healthcare. 5 Application of HFMEA to a procedure is a systematic method of highlighting the hazardous steps of an operation for inclusion in a procedure-specific assessment tool.
The aims of this study were: (i) to develop a PCNL assessment score utilizing HFMEA for use in assessing trainees during surgical training; and (ii) to content validate and revise the PCNL assessment score through multi-operator, multi-institutional discussion.

Methods
The international, multicenter, prospective and longitudinal observational study was carried out in two phases: phase 1, development of the PCNL assessment score through application of HFMEA; and phase 2, content validation of the PCNL assessment score.

Phase 1: Development
HFMEA was applied to the PCNL procedure to highlight the steps most likely to fail or cause adverse outcomes (Fig. 1). Six surgeons were observed carrying out PCNLs for 25 h across four institutions, enabling process mapping of the procedure. The processes and subprocesses of PCNL were outlined, and hazard analysis was then carried out, highlighting failure modes (the way in which a step of the procedure could fail) and failure mode effects (the result of a step failing). For example, during insertion of the needle into the kidney before tract dilatation, a failure mode would be trauma to the surrounding tissues, subsequently the failure mode effect would be that the PCNL procedure was suspended until this was repaired.
A HFMEA hazard analysis table was constructed and distributed internationally to 11 expert surgeons from eight institutions. Each surgeon systematically reviewed the failure modes, and assigned them a severity and probability score (Fig. S1). The median of the 11 probability and severity STEP 1 -Define the HFMEA Topic  scores was calculated to give a final score for each failure mode. Where the median value was not an integer, it was rounded up to the nearest whole number. The probability and severity scores for each failure mode were then multiplied in accordance with the HFMEA hazard analysis decision tree (Fig. S1), to provide a hazard score for each mode. The hazard score assigned could be between 1 and 16, higher numbers indicated a more hazardous stage. Subsequently, each failure mode and effect was assessed to identify if they were single point weaknesses (the steps that would result in the entire procedure failing if they did not occur). The stages of PCNL assigned a hazard score ≥4 or defined as a single point weakness were deemed most hazardous and included in the PCNL assessment score.

Phase 2: Validation
Content validation was carried out throughout the design and development of the PCNL assessment score. This validation was carried out by reviewing the content of the tool with six surgeons after initial process mapping and hazard analysis scoring with. Further along in development, the PCNL assessment score was examined by 20 expert surgeons and their teams from 17 centers and 10 countries to ensure international consensus. Feedback was utilized to adapt the tool and produce the final PCNL assessment score.
All surgeons included in the study were deemed experts in PCNL, as they had completed their training in the procedure and were carrying out PCNLs independently. They had all performed ≥100 procedures (average 1316, range 100-4000) and were currently working in a PCNL center.

Results
Development and content validation produced a 10 phase, 21 process and 43 subprocess PCNL assessment score (Fig. 2).

Phase 1: Development
PCNL was defined as the topic to be assessed using HFMEA. The patient pathway was observed from entry into the anesthetic room until the handover to the recovery ward. The procedure was discussed with surgeons and other members of the multidisciplinary team. The discussions highlighted the areas of heterogeneity in PCNL technique dependent on the institution, surgeon preference and the PCNL subtype being carried out. Next the process map for PCNL was created acting as a visual representation of the procedural steps (Fig. S2). The hazard analysis part of the project used the process map to identify the phases (key stages of the operation e.g. stone fragmentation), processes (further breakdown of the actions included in each phase) and subprocess (further breakdown of the processes to provide step-by-step instruction for the procedure) of PCNL. There were 10 phases, 26 processes and 51 subprocesses identified. The failure mode for each subprocess was then established; 64 failure modes were found.
Hazard analysis by 11 surgeons assigned each failure mode a hazard score. A total of 37 failure modes had a hazard score ≥4. If the 27 failure modes with a hazard score <4 were found not be a single point weakness, they were excluded. Failure modes with a hazard score ≥4 or considered a single point weakness were included unless it was deemed that they had an existing control measure in place to prevent the failure from occurring. For example, patient identification is not  included, as there is an existing control measure in the form of the World Health Organization safety checklist to ensure correct patient identity. The 47 failure modes proceeding to the next stage of the HFMEA process were deemed essential for inclusion in the PCNL assessment score, as their omission would result in non-completion of the operation or completion to an unacceptable standard.
The failure mode column was removed and several subprocesses merged to provide a simplistic, aesthetically pleasing form. Consensus was gathered on the difficulty of each subprocess to assign each a rating of I (least difficult) to V (most difficult). This was carried out to provide the trainee with guidance as to which sections of the procedure might be more challenging and where they might expect to initially see lower scores.

Phase 2: Validation
Content validation ran throughout the HFMEA process beginning with the procedure mapping stage. Process mapping was carried out during the observation phase, at which six PCNL surgeons and other members of the multidisciplinary team were consulted for content validation. Validation this early in development of the tool ensured all information included in subsequent hazard analysis and in the final tool was correct.
Validation was also carried out during hazard analysis, particularly associated with the failure modes identified. A total of 11 surgeons completed hazard analysis and identified any errors or areas for improvement. Throughout the HFMEA process, experts highlighted the need for flexibility in the assessment tool to allow for operator/institute preference, as well as PCNL technique variation; therefore, a number of subprocesses are optional.
Finally, the PCNL assessment score was circulated to 16 experts from 10 countries for content validation. Discussion highlighted that the tool should require the assessor to assign the trainee a score for each of the subprocesses. Therefore, scoring columns were added along with a definition of the scores. The experts agreed that the final PCNL assessment score contained all the pertinent procedural steps (Fig. 2).

Discussion
This longitudinal, international, multicenter study produced a PCNL assessment score to be utilized by urology trainees and assessors. The application of HFMEA has enabled the systematic assessment of risk in the PCNL procedure, highlighting steps in the technique that are most hazardous, as well as integral for success of the operation. The tool has been content validated by 20 PCNL experts and consensus on the final assessment tool has been obtained. The PCNL assessment score comprises of 10 phases, 21 processes and 43 subprocesses, and is formatted to allow scoring of the urology trainee intraoperatively with the aim of quantifying operator progress over time.
PCNL was established >40 years ago; 6 the procedure is still considered challenging for surgeons to master. This is supported by training programs across the world; for example, in the UK, PCNL is only included in the Intercollegiate Surgical Curriculum Program syllabus for senior urology registrars, showing it is a technically challenging skill that requires a certain amount of previous experience. 7 Furthermore, although PCNL is an efficacious method for calculi removal, it is still associated with higher complication rates than other endoscopic urolithiasis techniques, particularly when carried out by less experienced team members. 8 Therefore, it is essential to develop effective training programs for PCNL to reduce the risk of harm to patients.
There has been a movement to create procedure-specific checklists that focus predominantly on technical skills. This has been shown in a number of studies; for example, Ahmed et al. developed a safety checklist for robotic surgery, which identified hazards in an operating theater with robotic technologies through the application of HFMEA. 9 Subsequently, the first procedure-specific safety tool was developed; HFMEA was utilized to highlight risk-associated steps in RARP and develop a RARP assessment score. The present study also went on to utilize the training tool to identify learning curves for this particular procedure. 4 More recently, HFMEA has been applied to LRN to create the LRN assessment tool. 6 The present study took a similar approach and produced the only assessment tool tailored specifically to PCNL. The PCNL checklist encompasses all technical aspects of the surgery. It acts to ensure that the steps of the procedure are completed and to an acceptable standard to prevent adverse events while simultaneously providing trainees with quantitative assessment of their performance of the procedure.
The present study had a number of strengths; it was a multicenter project that ensured international consensus on the PCNL assessment score. The tool was developed so that it can be adapted to different PCNL techniques dependent on operator preferences.
It is acknowledged that the proportion of urologists or interventional radiologists who gain percutaneous renal access or place nephrostomies varies between centers and countries; for example, a study in the USA found that 11% of urologists gained access, 10 whereas in the UK this was 34%. 11 For trainees who do not complete all parts of the procedure, they can exclude these; for example, they can omit the needle and guidewire insertion phase. The total maximum score available for theses trainees would be lower; however, their scores from each PCNL attempt would be comparable to their previous scores, therefore still enabling identification of operator progression.
Content validation of the tool resulted in inclusion of a scoring system for each subprocess. Use of the scoring system will enable trainees to quantify their performance during PCNL with the aim of observing an increase in this score over time. Furthermore, the scoring system allows assessors to highlight specific parts of the procedure in which the trainee is attaining a low score.
At present, the PCNL assessment score is applicable to various PCNL techniques; however, surgical techniques are continuously evolving. As recently as 2013, there was the introduction of the ultramini PCNL. 12 Future adaptations of PCNL might require amendments to the PCNL assessment score.
The limitations of the present study include that the tool does not take into account a number of factors that may might the trainees' scores. There are several patient factors that might make the procedure more challenging, such as anatomical variation, patient size and case complexity. Carrying out a PCNL on an obese patient is associated with issues when positioning the patient and locating the stone. 13 Additionally, the distance from the skin to the kidney when puncturing the renal collecting system is greater, leaving more space for error. 14 Stone location, size and density might also influence trainee scores. Stones with low or high density have been associated with lower rates of treatment success and longer operating times. 15 The present study developed the initial PCNL assessment score and content validated it. The next stage for the tool is further validity studies, including construct validity and realtime scoring feasibility. Ultimately, the developed PCNL assessment score can be incorporated within a full PCNL training curriculum alongside e-learning and simulation training.
The PCNL assessment score can be used in the early learning phase of PCNL training to provide structured feedback and quantify operator progress. For example, if a trainee is consistently obtaining low scores during tract dilatation and nephroscope insertion, they could complete additional training in this area and their mentor could provide advice on how to improve this area. The hope would be to produce surgeons competent in all aspect of PCNL. The tool can also be used to identify learning curves of each component, in conjunction with patient safety and operative efficacy outcome measures. This will allow for a more objective assessment of each phase of PCNL, and correlate the hazard scoring or difficulty level granted to each process with actual learning curves.
To conclude, the demand for intraoperative assessment is increasing; assessment tools should account for risks within a procedure and be content validated. The PCNL assessment score has been created, with 10 phases, 21 processes and 43 subprocesses. The hazardous steps of the procedure have been highlighted through application of HFMEA and included in the final tool. Content of the checklist has been validated throughout the process to ensure international consensus on the final PCNL assessment score. It is recommended to utilize the assessment tool in modular operating room training to help achieve trainee competence in PCNL, a keystone in the delivery of gold standard urolithiasis treatment.