Development of a Hybrid Training Simulator for Structural Heart Disease Interventions

To address the expanding needs to acquire the necessary skill sets for managing a wide array of transcatheter interventions, a 3D visualization system that integrates into the training platform would significantly enhance the trainee's capacity to comprehend the spatial relationships of various cardiac structures and facilitate the learning process. In addition to procedural training, the same technology may potentially help formulate treatment strategies in preprocedural planning especially in complex anatomy. Herein, a hybrid simulator for structural heart disease interventions is demonstrated by using the combination of a mixed reality (MR) display and 3D printing. The system consists of a 3D printed phantom heart model, a catheter with real‐time tracking using electromagnetic sensors, and the stand‐alone MR display for rendering 3D positions of the catheter within the heart model, along with quantitative feedback. The phantom heart model is generated by 3D printing technology using a segmented geometry from a human cardiac computed tomography (CT) scan. The catheter is coupled with electromagnetic sensors that allow real‐time tracking of their 3D positions and orientations. Custom software and algorithms to coregister and display the catheter's position relative to the phantom heart model are developed to interface with commercial software provided with the tracking sensors and MR display such that updates occur seamlessly in real time. Prespecified target crossings in the fossa ovalis during a transcatheter septal puncture procedure are demonstrated in the training scene. This hybrid training system will be used for training and educating transcatheter septal puncture procedure and other structural heart interventions.

To address the expanding needs to acquire the necessary skill sets for managing a wide array of transcatheter interventions, a 3D visualization system that integrates into the training platform would significantly enhance the trainee's capacity to comprehend the spatial relationships of various cardiac structures and facilitate the learning process. In addition to procedural training, the same technology may potentially help formulate treatment strategies in preprocedural planning especially in complex anatomy. Herein, a hybrid simulator for structural heart disease interventions is demonstrated by using the combination of a mixed reality (MR) display and 3D printing. The system consists of a 3D printed phantom heart model, a catheter with real-time tracking using electromagnetic sensors, and the stand-alone MR display for rendering 3D positions of the catheter within the heart model, along with quantitative feedback. The phantom heart model is generated by 3D printing technology using a segmented geometry from a human cardiac computed tomography (CT) scan. The catheter is coupled with electromagnetic sensors that allow real-time tracking of their 3D positions and orientations. Custom software and algorithms to coregister and display the catheter's position relative to the phantom heart model are developed to interface with commercial software provided with the tracking sensors and MR display such that updates occur seamlessly in real time. Prespecified target crossings in the fossa ovalis during a transcatheter septal puncture procedure are demonstrated in the training scene. This hybrid training system will be used for training and educating transcatheter septal puncture procedure and other structural heart interventions. current methodologies using animals, cadavers, stock anatomical models, and 2D fluoroscopic images.
New devices are used more frequently in medical training to visualize 3D medical applications using virtual reality (VR) or augmented reality (AR). [5] VR refers to a digital environment in which the user interacts as if it takes place in the real world. [6] However, the focus of the interaction remains in the digital environment. AR differs from VR because the focus of the performed task lies within in the real world instead of the digital environment. [7] VR is mainly useful for educational purposes [8] or in training modules for surgeons. [9] Many studies show that the number of applications of VR or AR in the field of medical education/training has been increasing in the last 20 years. [5,10] Mixed reality (MR) is a new concept that has emerged in recent years, as a combination between VR and using cutting-edge devices for control. [11] MR combines AR glasses with a system of tracking cameras and sensors. Through this novel head-mounted device, the virtual 3D objects are added in the screen within the ability to be registered with the physical world. [12] The medical applications using MR are increasingly more numerous especially in medical education, computed tomography (CT) image visualization, or medical procedures, [13] offering another supportive tool for 3D visualization and manipulation.
Extended realities (VR/AR/MR) have been successful in medical education and training. The VR/AR courses for human anatomy were developed in medical school to enable students to examine the human anatomy in totality by using Microsoft HoloLens or Oculus Rift VR displays. [14] The Vimedix simulator enabled echocardiogram training using MR. [15] AR and MR have also been developed to act as a Digital Imaging and Communications in Medicine (DICOM) viewer for 3D echocardiogram, CT, magnetic resonance imaging (MRI), and Positron emission tomography (PET) scans and help preoperative planning. [16] AR has been successful in providing enhanced efficacy and precision for electroanatomical mapping procedures for arrhythmia. [17] However, extended reality technologies have not been applied in the rapidly developing field of structural heart disease interventions.
It is important to develop a real-time guiding system for both procedure planning/guidance in the catheterization laboratory and medical training for interventional procedures. Most of the cardiac interventions are primarily guided by X-ray fluoroscopy. [18] However, the heart is transparent to fluoroscopy, and therefore radiopaque contrasts are used to visualize the relative position of the catheter to surrounding tissue. Furthermore, fluoroscopy only provides a 2D projection of the catheter and the device without any information of depth provided. [19] Other imaging modalities, such as CT and MRI, provide detailed anatomic information preoperatively and are often displayed on separate screens or overlaid on real-time imaging modalities to improve image-guided interventions. [20] However, this hybrid imaging often obstructs the view of the real-time image during the procedure, thus offering limited improvements. Furthermore, all these images are displayed on 2D screens, which fundamentally mitigate the ability to perceive depth and orientation. [21] Transesophageal echocardiography (TEE) is the most common real-time guiding system for transcatheter procedures due to its improved image resolution and ability to see relevant cardiac structures. [22] Despite the contributions of TEE to image guidance, its widespread use is still debated in the cardiology community because TEE requires general anesthesia; its invasiveness can potentially cause serious complications (e.g., esophageal perforation, aspiration, oropharyngeal damage); it requires experienced echocardiographer; [22] and it uses equipment not readily available in all hospitals. [23] Therefore, there is an unmet need for educational or training system and development of real-time procedure-guiding system in structural heart disease interventions.
In an effort to develop an efficient training system for interventional cardiology, we have built a novel training system that combines MR holograms and 3D printing technologies together to mimic catheter-based interventions for treating structural heart diseases. This combination provides physicians in training physical feedback, which is not available in a visualization-only training system. Preoperative cardiac CT images were used for generation of 3D computer-aided design (CAD) and 3D prints of a phantom heart model. We report our experience using both holographic applications and 3D printed models for training navigation inside the right side of the heart, specifically for a transseptal puncture procedure.

Hybrid Simulator System
The overall process of generating the hybrid simulator is shown in Figure 1. We have developed algorithms to register the catheter tracking position to the 3D model in the MR environment, and also to determine its relative position to specific targets of interest. The system is composed of physical hardware (electromagnetic (EM) tracking system, HoloLens goggles, and 3D printed heart model) and software (3D CAD from CT scans, custom recording and registering software, and Unity program for MR design and animation). The 3D CAD and 3D printed model are generated from CT scans of a patient. EM tracking sensors, which can detect the relative positions and angle from the transmitter, are inserted into the catheter. The real-time tracking data from the EM tracker are recorded through custom web-based recording software. Custom C# code performs an affine transformation of the recorded positions of fiduciary markers and catheter positions. The transformed coordinates of the catheter are utilized in the 3D rendering software to register the catheter image into the 3D heart model. Finally, the MR holograms and training applications are deployed to the Microsoft HoloLens display.

Generation of the 3D Printed Training Model
From the cardiac CT scans of patients, 3D CAD and 3D printed heart models are developed ( Figure 2). The use of the medical images in this study was approved by the institutional review board (IRB) at Weill Cornell Medicine. The informed consent requirement was waived as the patient's information was deidentified prior to the use of these image sets. Each slice of the CT images is first binarized by an appropriate threshold value for blood volume to generate a rough segmentation that included the left/right atrial and left/right ventricular blood, aorta, pulmonary arteries/vein, superior vena cava (SVC)/inferior vena cava (IVC), and bones. After trimming the undesired structures (e.g., ribs, sternum, or small blood vessels), the atrial and ventricular blood volume, large blood vessels including the SVC, IVC, and ascending aorta are all selected as a single object, along with the spine, which is selected as a separate single object. The selected masks are modified by smoothening and filtering and are then exported as stereolithography (STL) files. For the heart model, a hollow part with a 2 mm-thick shell is created from the blood volume. Supporting structures for connecting the heart model and spine model are added. The slots for external fiduciary markers are also added to the periphery of the supporting structures to provide the reference for coordinate transformation into the MR environment.

Development of Navigation System Using EM Tracking System
For the real-time tracking of the catheter in the model, an EM tracking system (3D Guidance Trakstar) is utilized ( Figure 1). EM sensors are inserted into the 13.8 F deflectable steerable guiding catheter (Destino Twist, Oscor). To centralize the Figure 1. Flowchart of hybrid simulator for transcatheter septal puncture. Components consist of both physical hardware (black boxes) and software (red boxes). 1) CT scan from the deidentified patient is prepared. 2) The 3D CAD of heart, spine, and connecting parts for 3D printing and MR visualization is prepared. 3) Physical heart model for the procedure training is 3D printed. 4) Steerable catheter is prepared together with three intraluminal EM tracking sensors. 5) The real-time tracking data from the EM tracker-equipped catheter in the 3D printed heart model are recorded through custom web-based recording software. 6) Fiduciary marker positions are recorded using additional EM sensor and utilized to make a transformation matrix to register the catheter positions in the transformed coordinates by using custom C# programming. 7) Unity software generates 3D heart model and visualizes the registered catheter positions as 3D-rendered catheter design. 8) The holograms and training modules are deployed to the Microsoft HoloLens MR display. Figure 2. Generation of 3D printed heart model from cardiac CT images. Human CT scan is processed into a 3D CAD model that is both rendered in MR and 3D printed for use in the hybrid training system. www.advancedsciencenews.com www.advintellsyst.com position of the EM sensor at the tip of the catheter (sensor 1), a custom-designed 3D printed plug is inserted into the inner diameter of the catheter tip ( Figure 3A-D). Additional sensors are located at the center (sensor 2) and the beginning (sensor 3) of the distal arc of the catheter ( Figure 3E).

Coordinate Transformation and Image Registration
To synchronize the 3D space of EM tracker (EMT) and MR with higher accuracy considering translation, rotation, and scaling between each coordinate, an affine transform function is utilized ( Table 1, Equation (1)). To this end, the positions of four predefined fiduciary markers are used for the calculation of the affine transformation matrix between the EMT and MR 3D coordinate systems. The 3D CAD model along with fiduciary markers in the EMT 3D coordinate system is loaded into the Unity software, where we manually rotated the model to a desired orientation ( Figure 4A). Considering the marker positions in EMT space, Similarly, in the MR space, we introduce the basis Table 1). Affine transform function contains four basic functionalities as the following. First, all 3D points are converted to a homogeneous coordinate (Equation (3) of Table 1), where the fourth coordinate is named as the w coordinate. Then, the matrices shown in Equation (4) of Table 1 are used to perform the basic affine transforms in 3D. Furthermore, to address and include the rotation about each of the individual axes of the coordinate system, the sets of three basic rotations in 3D are used (Equation (5) of Table 1).
After the coordinate transformation, we move into the catheter registration where three EM sensors are implemented at the tip of the catheter and two other locations, allowing to record the 3D positions of the catheter, in the form of a comma-separated values (CSV) file, in the EMT coordinate system, Figure 4B shows the sequence of catheter registration in MR space. Utilizing the calculated transformation matrix T from the coordinate transformation, the positions of the three sensors are transformed in the MR environment, thereby registering the position of the catheter in the MR 3D space, S 0 1 ¼ To render a smooth catheter shape in the MR environment, additional catheter reference positions, which are connected with short cylinder objects, are calculated by spline interpolation from the given three sensor positions using the "Mathf. InverseLerp" function in Unity. The EMT-MR matching system described earlier is implemented in the high-level programming language Microsoft C#, enabling the use of all capabilities of the MR platform for the generation of the rendered images. This object-oriented programming language gives access to all the variables of the program and all the 3D objects within the virtual world. To reduce latency, the C# program extracts catheter movements from a data stream generated by the EMT device. The data stream is uploaded, by an EMT OEM Microsoft Windows software, to a CSV text file containing all 3D information of the EMT coordinate system. In the C# program, we read the stream in batches of the size that corresponds to the value of the frequency of data generation. We read the stream generated for the markers at the same frequency. We apply the affine transformation as introduced earlier and load the transformed positions into the MR environment.

MR Visualization of the Transcatheter Septal Puncture Procedure
MR visualization is provided by rendering the patient's heart, spine, and catheter as holograms on the HoloLens display www.advancedsciencenews.com www.advintellsyst.com (Microsoft Corporation). The Hololens display is a pair of video see-through smart glasses that do not block the surgical view of the medical practitioner, and therefore does not disturb the normal interventional procedures. It is integrated with a wireless communication module that can be used to seamlessly receive the processed results from the server computer via a shared WiFi network. The 3D model of the heart and spine is imported into the HoloLens system using Unity software ( Figure 5). The 3D catheter model is instantiated at the positions transformed from custom C# code in real time at a refresh rate of 60-80 frames s À1 . The generated catheter is destroyed (i.e., deleted) with an optimized time gap (%0.02 s) to maintain real-time tracking visualization. The main hologram is generated at the center of the scene in the anterior-posterior (AP) projection view. The distance from the tip to the target and the angle between centerline of the catheter and connection line to the target are displayed in real time. Furthermore, text instructions are displayed at the top box of the scene. Right anterior oblique 30-degree view (RAO 30) and left anterior oblique 30-degree view (LAO 30) are displayed as additional panels on left side of the scene. The virtual endoscopic catheter view from the tip of the catheter with the view aligned with the directional vector of the catheter's tip is also provided in the right top panel. A magnified view for the current target site is displayed in the right bottom panel. These holograms visualized in the physical world are shown in Figure S1, Supporting Information.
In this representative training scene, four target sites (i.e., upper SVC, lower SVC, upper limbus of the fossa ovalis, and lower posterior of the fossa ovalis), which are commonly identified during transcatheter septal puncture procedures, were prepared in the MR environment. Each target is activated sequentially in the order following the sequence of a typical (2) Determining transformation matrix using the positions of four predefined fiduciary markers in physical (EMT) and MR space.
(3) Converting all 3D points to the homogeneous coordinates system.  www.advancedsciencenews.com www.advintellsyst.com transcatheter septal puncture procedure. First, the catheter is advanced as instructed into the upper SVC after passing through the IVC and right atrium ( Figure 5A). Second, the catheter is pulled back to the lower SVC, where it can enter into the right atrium ( Figure 5B). Third, the catheter is placed at the upper limbus of the fossa ovalis in the interatrial septum ( Figure 5C). The final target is located at the lower posterior quartile of the fossa ovalis, which is generally considered the optimal puncture site for left atrial appendage occluder implantation procedure ( Figure 5D). An example of a recorded training session is available in Video S1, Supporting Information. This hybrid training system has several advantages over other training models: 1) it provides both 3D visual and physical feedback to the trainee in real time, 2) there is no exposure of X-ray or biological hazards during the training, and 3) patient-specific models for complex anatomy can be generated for preprocedural planning. A brief educational session using this model may deliver the key learning points of the complex procedure more efficiently for training interventional cardiologists.

Precision and Accuracy of the MR Simulator in the 3D Printed Model
For the measurement of precision and accuracy of this navigation system, an additional phantom target model was 3D printed ( Figure 6A-D). As we intended to measure more reliable accuracy data of the registration algorithm, we used a visually accessible open target model rather than complex and closed heart model. Four slots were made for fiduciary markers to allow for registration of the physical model to the MR coordinate system. Each fiduciary marker slot was located at the corners of the model within the thick pillars, having different heights (0, 4.5, 9, and 13.5 cm from the bottom surface) to reduce the unnecessary error from the vibration and distortion. Additional ten slots at various locations in 3D space were made as target positions. The overall size of the target model was 25 Â 25 cm and the ten targets were distributed randomly with different heights (ranging from 1 to 9 cm). Although the manufacturer of the EM sensor does list a root mean square (RMS) accuracy value of 1.4 mm, this value represents a precision of the system in the manufacturer's environment. The precision (static accuracy) of the EM sensor in our experimental environment is measured by inserting an EM sensor into the fixed positions for 10 s. The precision is defined as the RMS deviation of a true measurement of the magnetic center of a single sensor with respect to the magnetic center of a single transmitter measured over the specific translation range with different quality numbers. The quality number is calculated by the algorithm provided by Northern Digital Inc., which relies on the known behavior and properties of an idealized 3 Â 3 matrix (9 raw signal values). Deviation from this ideal state are combined together to yield this indicator. The accuracy of targeting is measured by comparing the transformed coordinates of ten targets obtained during experiments and the reference positions of the ten targets as predefined in the 3D CAD of the MR environment.
The tracking precision or static accuracy for the fixed sensor positions is shown in Table S1, Supporting Information. In position 1 of the 3D printed target model, which has the quality number 1 (best performance), RMS of the measurements for 10 s (800 measurements) was only 0.048 mm in X direction, 0.060 mm in Y direction, 0.044 mm in Z direction, and 0.089 mm in 3D Euclidean distance. Even in position 4, which had the highest quality number 4 (worst among the four www.advancedsciencenews.com www.advintellsyst.com positions), RMS was 0.170 mm in X direction, 0.217 mm in Y direction, 0.158 mm in Z direction, and 0.318 mm in 3D Euclidean distance. The accuracy of image registration was evaluated by quantifying the registration errors between the target positions of the 3D CAD and the affine transformed catheter positions engaged to those targets ( Figure 6E-H). The mean distance from reference targets (n ¼ 10) was À0.2 AE 1.3 mm in X, 0.0 AE 0.8 mm in Y, À1.5 AE 1.7 mm in Z, and 2.4 AE 1.2 mm in Euclidean distance. Considering the diameter of fossa ovalis is about 13-15 mm, this accuracy level is in a reasonable range which can be accepted by an interventionalists. Table S2, Supporting Information, summarizes the accuracy for each target in the 3D printed phantom model; note that the target accuracy can be variable among different targets in the 3D printed model. When the EM measurement was tested by moving the EM sensor on a straight ruler, it can be seen that the accuracy of the sensor deviates as you move away from the transmitter, suggesting that only small volumes (<100 mm 3 ) can reliably provide accurate results ( Figure S2, Supporting Information).

Conclusion
In this study, a hybrid training system that combines MR guidance with 3D printed phantom models for training structural heart disease interventions is demonstrated. This approach provides a truly 3D visualization that can allow interventionalists to better orient themselves while interacting with a physical model. Traditional 2D screen displays are limited in providing an intuitive 3D visual experience in education/training. Headmounted MR displays, such as Microsoft's HoloLens, allow the user to manipulate digital content using voice control and hand gestures in the physical environment. Furthermore, the headsets are portable and allow the renderings to be positioned anywhere, which provides needed flexibility due to the crowded nature of a catheterization lab. Also, the real-time quantitative tracking can provide a means for the user to assess their accuracy and precision in catheter manipulation, which could improve the acquisition of new skills, such as procedural steps and catheter payload delivery. Although the focus of this and future studies is the use of this system for training and preprocedural planning, this system has the potential for providing intraprocedural guidance to interventions without further imaging (i.e., fluoroscopy) in real time. However, the clinical implantation of this system will require other developments, such as integration of the EM sensors into the clinical catheter and motion compensation for the registration algorithms. Limitations of this study that will be addressed in the future include cardiac motion from both cardiac contractions and respiration, use of rigid materials for the phantom model, and lack of simulated flow within the model. MR simulators, along with VR and AR simulators, must meet certain quality standards prior to their approval for application in medical training. The system should be tested for validation, effectiveness of skill transfer, learning curve, and retention of the skill. Combination of a holographic training application and 3D printed heart model would provide intuitive, memorable, and hands-on experience and customized models to specific pathologies or procedures. Future studies will pursue the quantitative comparison of the hybrid training system versus the standard fluoroscopy-based guidance using multiple heart models in a pilot study, which medical residents, fellows, and interventionalists will participate in.

Experimental Section
Materials: The 3D printing of the model used the rigid and semitransparent material, named Vero Clear (Stratasys) in Objet 260 Connex1 system (Stratasys). The supporting material (SUP705, Stratasys) for the 3D print was removed manually and with a high-flow water jet cleaner.
3D CAD: End-diastolic cardiac CT images were imported as a deidentified DICOM format into a 3D image processing software (Materialize-Mimics Research software 21.0). Basic STL files for the heart and spine and supporting structure were generated by Solidworks software 2018 (Dassault Systemes). Further smoothening of the meshwork and generation of the shell structure were done by Geomagic Wrap (3D Systems Corporation).
EM Tracking System: We used the 3D Guidance Trakstar system for EM tracking. The midrange transmitter and the Model 90 sensors with six degree of freedom (6DOF) (X, Y, Z, Azimuthal angle, Elevational angle, and Rotational angle) are used. The reported static RMS accuracy of the tracking system was 1.4 mm in position and 0.5 in orientation. The quality of the measurement was provided by the system's self-reported quality number, which indicates the degree of which the position and angle measurements are in error. The quality number represents a level of accuracy degradation resulting from either movement of the sensor or environmental noise. All the measurements are performed inside the recommended performance motion box where the lowest quality numbers are achieved (20-36 cm in X, AE20 cm in Y, and AE10 cm in Z). For all of our applications, we have found these numbers are sufficient.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.