Multimodal immersive digital twin platform for cyber–physical robot fleets in nuclear environments

The nuclear energy sector can benefit from mobile robots for remote inspection and handling, reducing human exposure to radiation. Advances in cyber–physical systems have improved robotic platforms in this sector through digital twin (DT) technology. DTs enhance situational awareness for robot operators, crucial for safety in the nuclear energy sector, and their value is anticipated to increase with the growing complexity of cyber–physical systems. The primary motivation of this work is to rapidly develop and evaluate a robot fleet interface that accounts for these benefits in the context of nuclear environments. Here, we introduce a multimodal immersive DT platform for cyber–physical robot fleets based on the ROS‐Unity 3D framework. The system design enables fleet monitoring and management by integrating building information models, mission parameters, robot sensor data, and multimodal user interaction through traditional and virtual reality interfaces. A modified heuristic evaluation approach, which accounts for the positive and negative aspects of the interface, was introduced to accelerate the iterative design process of our DT platform. Robot operators from leading nuclear research institutions (Sellafield Ltd. and the Japan Atomic Energy Agency) performed a simulated robot inspection mission while providing valuable insights into the design elements of the cyber–physical system. The three usability themes that emerged and inspired our design recommendations for future developers include increasing the interface's flexibility, considering each robot's individuality, and adapting the platform to expand sensor visualization capabilities.

The use of robots in the nuclear energy sector has become increasingly prevalent since its first implementation in the 1950s (Moore, 1985), with advanced robotics technology playing an essential role in various aspects of nuclear plant operations, including inspection, maintenance, and decommissioning (Bogue, 2011;Smith et al., 2020).At present, robotics plays a growing role in this sector by augmenting the capability of its stakeholders to manage important and high-risk assets.As the nuclear industry continues to prioritize safety, efficiency, and cost-effectiveness, robots have emerged as a valuable tool for improving plant performance while reducing the risks associated with human intervention in radioactive environments (Vitanov et al., 2021).
Most of the challenges the nuclear industry faces are aimed towards improving the overall safety, time, and cost of operations (Smith et al., 2020).Robots can help address such challenges through the following means: (1) remote handling of radioactive materials to reduce the risks associated with human exposure to radiation (Marturi et al., 2016), (2) inspection, characterization, and maintenance of areas of no-man-access which may contain radioactive hazards (e.g., silos) (Cheah et al., 2022;Tsitsimpelis et al., 2019), (3) decommissioning of nuclear plants, which is a complex and hazardous process that requires careful planning and execution (Bogue, 2011), (4) emergency response such as nuclear accidents, by performing tasks that are too dangerous for humans to undertake (Nagatani et al., 2013;Zhang et al., 2018), and (5) cost reduction by automating routine tasks and reducing the need for human intervention (Smith et al., 2020).
Recently, there has been a growing interest in the scalability of robots and their functions for efficient industrial applications (Buerkle et al., 2023;Mitchell et al., 2022).For example, the nuclear energy industry can benefit from implementing multirobot teams that can collaborate to complete tasks more quickly and efficiently than individual robots or human workers.This can reduce the time required for routine inspection, maintenance, and repair operations within the facility (Mitchell et al., 2023).Furthermore, heterogeneous robot fleets can broaden the scope of work as they have varying sizes, shapes, locomotion, and sensor capabilities and can be deployed in a single mission (Khamis et al., 2015).
Despite the advantages of multirobot fleets, various concerns limit their current applications.There are often issues in the integration and compatibility of robots with different software architectures (Harper et al., 2023;Rouček et al., 2020), so it is important to design systems that can overcome this.Before fully autonomous systems can be implemented, there is also a need for systems to be usable by human operators (Mitchell et al., 2021).The DARPA Subterranean Challenge is an initiative to develop innovative technologies for heterogeneous robot fleets in extreme and hazardous environments, with some benefits impacting the nuclear sector.
One common failure in recent implementations was the overburdening of a single human operator when managing a fleet of systems (Agha et al., 2021;Scherer et al., 2022;Tranzatto et al., 2022), highlighting the need for system designs that can address this challenge.
While different robots can be specialized for specific tasks, such as capturing images, mapping unknown environments, or handling hazardous materials, they should be able to work together to complete a more significant task, especially in hazardous environments (Lunghi et al., 2019).Ensuring that the robots in the fleet are compatible with each other and can communicate effectively can be a challenge, especially if the robots are made by different manufacturers or use different software.The Robot Operating System (ROS), a type of middleware that can unify platforms, was made to address this.Robot operators would also need to be trained to operate and interact with the robots in the fleet, which can require additional time and resources.Additionally, robot operators must be able to track each robot's location, status, and mission status in the fleet to ensure that the robots are operating effectively.
We believe these challenges could be addressed by integrating strategies in digitalization and Industry 4.0 (Tahmasebinia et al., 2023), creating a robot cyber-physical system involving the human operator in the loop.In robotics, digitalization refers to integrating digital technologies and data processing capabilities into the robots to enhance their functionality and improve their performance (Goel & Gupta, 2020).This involves using sensors, processors, and communication networks to collect and analyze data from the environment, the robot, and its interactions with other systems (i.e., humans and robots).As a result, the digital twin (DT) in a robot cyber-physical system is created with a detailed and real-time representation of its physical twin's characteristics, behavior, and performance.The robot's DT can be used to simulate and analyze its operation in different scenarios.
In the context of human-robot interaction, DTs can be used as an interface to improve the situational awareness of humans operating robots by providing them with a more complete and accurate understanding of the robot's behavior and capabilities (Gallala et al., 2022;Malik & Bilberg, 2018).By granting humans access to this information (i.e., via traditional flat screens or virtual reality [VR] platforms), DT interfaces can help them make more informed decisions about the mission, ensuring that it is safe and efficient.It can also test and validate new control algorithms and operating strategies before implementing them in the real world.
Ensuring safe, secure, productive, and reliable operations of multirobot fleets poses significant challenges due to the complexity of coordinating robots with varying capabilities.To address this, integrating digitalization and Industry 4.0 principles is crucial in creating a comprehensive robot cyber-physical system involving human operators.Human-centered design principles are essential in developing efficient human-robot interfaces, such as VR, and realworld/mock trials are vital to optimize these interfaces for highconsequence environments like the nuclear industry (Sato et al., 2020).These trials gather valuable feedback for refinement, aligning interfaces with human cognition and preferences to enhance mission safety and efficiency.By integrating insights gained from human feedback, researchers can develop practical strategies for robot fleets in nuclear environments, fostering increased efficiency in various industrial applications.This work details the development of a multimodal immersive DT platform (a type of data and asset DT) that enables near real-time monitoring, control, and optimization of cyber-physical robot fleets operating in nuclear environments, aiming to improve the safety, efficiency, and reliability of nuclear operations.We created a system that utilizes a ROS-Unity 3D framework that enables real-time management of a heterogeneous robot fleet for nuclear environments and scenarios.This allows robot operators to: 1. Manage a fleet of heterogeneous robots on a mission scenario for nuclear environments, effectively communicating crucial information between individual fleet members and their operators.
2. Operate a live mobile robot with a new user interface designed for nuclear environments.
3. Visualize data and identify hazards through a traditional flatscreen display or a VR headset.
The proposed system will be particularly useful in most asset management and routine operations in nuclear environments, such as facility maintenance, hazardous waste handling, and radiation monitoring.Furthermore, the effective management of a nuclear robot fleet, as provided by the DT interface, can help improve such operations in the context of safety and risk management, accuracy of reports, and efficiency/performance.
The novel contributions of this work to the field are as follows: • Modified heuristic evaluation method: To better understand the usability of the DT platform, we employed a modified heuristic evaluation method, which involved assessing the user interface with a group of nuclear robot operators.In this approach, we introduced the practice of accounting for positive and negative comments from participants regarding the interface, which accelerates the iterative design process of such systems (see Section 3).
• Design recommendations for DT interfaces of cyber-physical robot fleets: The results of the modified heuristic evaluation have provided significant implications for designing effective and efficient DT interfaces for heterogeneous robot fleets in a nuclear environment.We have enumerated and visualized our design recommendations for future developers to take inspiration from when designing similar systems for demanding industries (see Section 4.2).
The next sections of this paper are organized as follows: Section 2 discusses the system design and architecture of the cyber-physical robot fleet, while Section 3 presents the background, methodology, and results of the heuristic evaluation performed with the system's user interface, Section 4 provides a discussion of the implementation, design recommendations, and its limitations, and finally, Section 5 concludes the work and provides a perspective on the future of the research.

| SYSTEM DESIGN
This section discusses the system design, components, and architecture of the cyber-physical robot fleet system for nuclear environments.We first present the members of the heterogeneous robot fleet and their assigned functions for the mission scenario.We then describe the cyber-physical system architecture, based on the ROS-Unity 3D software framework, focusing on the flow of data from the physical sensors to the visualization approaches in the virtual environment.Finally, we provide a deeper discussion of the elements in the multimodal DT interface.
A Supporting Information video of the interface and its features can be accessed here: https://youtu.be/0EJ8Y8esQz0.The Clearpath Husky with a single UR5 arm (Figure 1b) is a mobile manipulator platform that features a 6-axis Universal Robots UR5 robotic arm with a payload capacity of 5 kg and a reach of 850 mm (Husky UGV-Clearpath Robotics, n.d).The robot arm is mounted on a rugged, all-terrain mobile platform that can traverse rough terrain and carry payloads of up to 15 kg.A battery pack powers the robot for up to 4 h of operation.

| Heterogeneous robot fleet
The CARMA2 (Figure 1c) is a mobile robot that is based on the Clearpath Jackal platform and is designed to continuously inspect a map of the ground for radioactive contamination from fixed or migrating sources (Bird et al., 2019;Nouri Rahmat Abadi et al., 2023).
CARMA2 uses the following off-the-shelf sensors: two 20-m LiDARs from Hokuyo, an Orbbec depth camera, two Thermo Fisher Scientific DP6, and a Radeye SX radiation sensor.The robot is also equipped with a frequency-modulated continuous-wave (FMCW) radar sensor (SiversIMA RS3400K) (RS3400K/00 24 GHz Radar Sensor, 2023) that can be used to detect aqueous areas on the ground.The robot is powered by a battery pack for up to 8 h of operation.
The DJI/Ryze Tello UAV Figure 1d is a small, lightweight drone designed for aerial photography and videography (Tello, n.d).It features a camera that can capture 720p HD video and 5-megapixel photos.The drone is controlled via an Android-based smartphone app with a range of up to 100 m.It can fly for up to 13 min on a single battery charge and has a maximum flight speed of 8 m/s.The drone also features several advanced flight modes, which can aid the user in capturing images.

| System architecture using ROS-Unity
The general software architecture of the cyber-physical system is based on the ROS-Unity 3D framework.Figure 2 presents the system architecture and data flow for the cyber-physical robot fleet system.As discussed in Section 2A, each mobile robot platform provides a distinct function that is vital to the success of the mission.In the following paragraphs, we describe these functions in the context of sensor capabilities and the flow of data within the system.The Scout 2.0 is tasked to scan and generate a 3D map of the environment through the Velodyne LiDAR sensor.Point cloud data were generated and localized in the virtual 3D environment.The point cloud is then converted into 3D voxels with the use of the Octomap ROS package (Hornung et al., 2013), resulting in an optimized 3D occupancy grid (occupied_cells_vis_array) in the virtual world.On average, the point cloud data had an update frequency of 0.724 Hz (SD = 1.06), while the voxels generated by the Octomap package yielded an update rate of 0.361 Hz (SD = 1.82).As the goal of this stage is to generate a 3D map of the environment (and not account for any fast-moving objects), the update rate was deemed acceptable for the user's overall experience.
The Husky mobile manipulator serves as a robot that can perform remote handling.Aside from this, the Husky provides a realtime camera feed (image) for teleoperation by the user.It is important to note here that no remote handling tasks were performed in this scenario, as the scope of the work revolves around the overall operation of the fleet in the DT interface.However, future work will involve testing the interface's usability during remote handling operations.
The CARMA2 UGV provides specialized sensor readings related to radiation (Thermo Fisher Scientific DP6 and Radeye SX radiation sensors) and other hazards (FMCW for aqueous detection).As a result, CARMA2 transmits a costmap of the hazards in the form of an occupancy grid (costmap) (West et al., 2022).
Among the heterogeneous mobile robot fleet members, the Tello UAV is the only one not natively operated in ROS.Tello UAV provides a live camera feed from an aerial perspective, giving the operators vital information during the mission.To integrate with the DT platform, the images transmitted by Tello UAV can be converted to ROS topics through its driver package (tello_driver, n.d).

| Unity 3D
Here, we discuss the progression of robot and environmental sensor data from ROS to the Unity 3D platform.This involves creating and visualizing various DT elements such as building information models (BIMs), robot models, and sensor data used in the platform.Figure 3 presents the various assets and elements of the DT environment as seen in Unity 3D.
From the ROS Master node, we compiled and transmitted the required ROS topics to build our DT platform in Unity 3D.The Unity 3D platform is a popular game engine that has also become a powerful tool for creating interactive simulations and VR experi- cost maps, etc.) from the robot.Furthermore, a Unified Robotics Description Format (URDF) model of each robot can be imported into Unity 3D via the URDF Importer package to generate its digital shadow, reflecting the robot model and its joint states in the virtual world (see Figure 3b).
The DT environment and its BIM are based on the RAICo1 Laboratory Facility, located in Whitehaven, Cumbria, UK.RAICo1 is a collaboration between the University of Manchester, Nuclear Decommissioning Authority, Sellafield Ltd., and the United Kingdom Atomic Energy Authority to advance research and innovation in nuclear robotics and artificial intelligence.The virtual environment (see Figure 3a) was built using existing building plans, measurements, and related information, simplifying some elements from the real world to optimize performance in simulation.Photorealistic textures were then generated from photographs and media files of existing objects and components in the building.All assets and game objects were created using Blender version 3.4.1, a free and open-source 3D computer graphics software tool.
We created a visualization solution for radiation data in the DT platform, aimed at providing information about specific areas of interest known as "radiation hot spots" (see Figure 3c).For this application, radiation hot spots were created by simulating a radiation source inside the ROS Gazebo simulation environment through a package called gazebo_radiation_plugin (Wright et al., 2021).This plugin provides similar behavior to that of the CARMA2 UGV radiation sensor during real-world deployments.
Radiation hot spots were presented as meshes made of points, where the color of each point is determined by mapping radiation data values to hue, saturation, and value (HSV) color.The values are scaled to fit a specific range useful for heat map visualization (i.e., red for high radiation and blue for low).A Unity mesh was created, where the vertex locations are the Cartesian coordinates of the radiation values received as an ROS message from the robot.The color value for each point is calculated by converting the calculated HSV color to RGB values for rendering.A colored vertex shader with configurable vertex size is used to render the procedurally generated mesh.Vertex size is determined empirically to make the radiation hot spot clearly visible without overtly obscuring the environment.The data to create the meshes is received as an array of floating precision radiation values, twinned with metric information from the MapMetaData ROS message from the ROS Navigation Stack.A mesh is created per received message, and the meshes can be either left for visualization or deleted as new messages are received.

| Digital twin interface
Information from the DT environment is presented to the user via two modalities: (1) traditional flat-screen display (default mode) and (2) VR.This section discusses the process flow of using and switching between these modalities and their respective user interaction elements.Figure 4 presents the default mode of the interface via a traditional flat-screen display.

| Traditional flat-screen display
This mode provides the user with mission-related information and allows interaction with the robot DTs and the DT environment through the computer's mouse and keyboard.The elements of the user interface can be categorized into seven features:

Camera view and navigation
The DTs of robots and their operating virtual environment are viewed through a virtual camera in Unity 3D.A Unity C# script was integrated into the camera game object, allowing the user to navigate and fly through the virtual environment.For this feature, the user can translate/pan through the virtual world by simultaneously clicking the right mouse button with the following keys: W (forward), S (backward), A (left), and D (right).The user can change the camera's Combining both translation and orientation is possible to have a flythrough effect.The camera is attached to a Unity User Interface canvas, which allows the interactive elements (e.g., buttons) to be viewed and clicked at all times during flat-screen mode.

Robot fleet member information
Each member of the robot fleet has an attached hovering world space canvas that displays individual information (see Figure 4a).The details include the assigned name of the robot, network connectivity status, and battery life.The robot's name is displayed as white text with a condensed sans-serif typeface, allowing the name to be visible in the interface at most times.The network connectivity is presented as a circle icon that changes its color according to its status: fully connected (green) and not connected (gray).The battery life is presented as a green "health bar" that is reduced as the battery percentage gets lower.We took inspiration from visualizing the health points of agents in various computer games.In addition, the battery percentage is also displayed as text in the middle of the slider, giving numerical and visual information.

Unity robotics hub's ROS dashboard
The Unity Robotics Hub has a dedicated dashboard to display ROS topics and status information (see Figure 4b).This includes Unity's connectivity status with ROS, as displayed by the color-changing bidirectional arrow icon: connected (blue) and disconnected (red).
Other elements in the ROS dashboard include ROS internet protocol information, buttons to toggle 2D and 3D ROS topics, ROS topic meta-data, view transforms, and buttons to change the layout of the dashboard.

Toggle sensor views
The interface also allows the user to toggle views related to the BIM (e.g., structural components, walls, existing 3D models of assets/ mounted equipment, and floor plans) (see Figure 4c).This was developed as a feature to allow unobstructed sensor views of the DT environment.

Spawn hazard tags
The ability to spawn various hazard tags is integral to the routine inspection of nuclear facilities (see Figure 4d).This was developed to increase the usability of the proposed DT platform in inspection and maintenance scenarios.By integrating a "click spawner" C# script, the DT platform can spawn five different types of hazards or incidents: (1) moved objects, (2) radiation detected, (3) flammables, (4) corrosives/ caustics, and (5) a general caution tag.Through Unity 3D's ray-casting capabilities, a hazard tag is spawned in the virtual world by clicking the left mouse button immediately after selecting a particular hazard in this panel.It is important to note that the click spawner script can only be triggered once to avoid overspawning unnecessary tags.

Initiate VR mode
This button allows the user to switch between traditional flat-screen and VR modes (see Figure 4e).Upon clicking this button, the current camera game object is switched off while the VR camera assigned to the VR head-mounted device (HMD) is turned on.The user then wears the VR HMD and interacts with the DT platform using the same user interaction elements but with a different input system (i.e., VR controllers).The following subsection discusses the mode's VRbased user interaction elements.

Generate hazard report
Upon completion of the inspection mission, and when all necessary hazard tags have been placed, the software platform is able to compile all placed and labeled tags in a list and generate a hazard report (see Figure 4f) that can be used by different members and stakeholders of the nuclear facility.

Robot first-person view
Each member of the heterogeneous robot fleet provides their own camera feed (first-person view) during the mission.This allows the user to view each member's activities and is used during remote teleoperation missions.We designed a heads-up display (HUD) that presents battery, connectivity, and mission information during individual robot viewing.Figure 5 presents the robot's first-person view with the HUD elements.In this configuration, the screen is divided into three panels: (1) left: camera feed with HUD, (2) top right: 3D navigation view similar to the default mode, and (3) bottom right: a robot "mini-map" to help with the navigation and avoidance of obstacles during teleoperation.This is achieved by placing a camera above and facing down the robot to capture a top view of its surroundings (at least 2 × 2 m) while superimposing sensor-detected obstacles.

| Virtual reality
VR can enhance the situational awareness of workers in nuclear facilities by offering immersive and interactive simulations of such environments.In VR, users can experience the DT environment, which closely resembles the real industrial environment, and interact with it as if they were physically present.This experience helps users better understand the layout, features, and potential hazards of the industrial environment.
In this work, we implemented a second modality for the DT interface through VR.The motivation is to take advantage of the benefits discussed above while complementing the traditional flatscreen mode of the DT platform.We used the HTC Vive Pro VR system, which offers a display resolution of 2880 × 1600 pixels (1440 × 1600 pixels per eye), a 90-Hz refresh rate, and a 110°field of When in VR mode, the user is situated in the DT environment through a virtual camera that mirrors head and hand movement from the tracking data provided by the HTC Vive Pro headset.As a result, the user can view the elements visualized in the DT environment but with a first-person point of view.Figure 6 presents the DT interface's VR mode and its interaction capabilities.
The VR interface uses tools from SteamVR to enable interaction with the virtual environment, including its assets.The HTC Vive Pro VR controllers consist of a grip, buttons, triggers, touch pads, and motion sensors that detect the user's movements.The following features were integrated into the DT interface for this VR implementation.

VR teleportation
The left-hand controller was programmed to perform teleportation within the VR space and was implemented using a SteamVR preset package.In this preset, the floor plane of the virtual environment is mapped in reference to the VR user (wearing the headset and holding the controllers).The user can select the area they want to teleport to (in the form of a game object visualization, see Figure 6b) and press the trigger to initiate the process.This allows users to navigate and explore the VR environment more efficiently.

VR spawn hazard tags
The right-hand controller includes features that enable users to view and select features from a wrist menu.By viewing their hands in VR, they can access the wrist menu, allowing them to choose between the different hazard tags described in the traditional flat-screen mode by pressing the appropriate button.Once a hazard tag is selected, they can spawn it by pointing a laser at a surface in the VR environment.This is made possible through ray tracing, which allows the user to spawn the hazard tag accurately and precisely in the desired location.

| HEURISTIC EVALUATION
This section describes the methodology and results of a usability study to evaluate the DT interface for a heterogeneous robot fleet in a nuclear environment.Heuristic evaluation is a usability engineering technique used to assess the usability of a user interface design by systematically identifying and analyzing potential problems (Nielsen & Molich, 1990).It involves a selected group of evaluators with expertise in usability principles examining the interface design and evaluating its adherence to established heuristics.Any identified usability issues can be resolved as an integral part of the iterative design process.
The study aimed to initially identify usability issues and improve the overall user experience of the DT interface in its current state.
Using this methodology ensured that the next iteration of the interface would be more intuitive, efficient, and easy to use, ultimately leading to increased productivity and improved user satisfaction.
It is important to note here that the nature of a heuristic evaluation study is not to assess whether the developed interface passes or fails a specific test to be deemed usable.Instead, it is an inspection method where evaluators systematically examine a user interface based on predefined usability principles (heuristics).Furthermore, we have implemented a slightly modified approach such that evaluators can comment on the positive aspects of the interfaceallowing us to assign positive or negative points to each entry to inform our analysis (see Section 3.2).
In this study, the researchers followed the University of Manchester Research Ethics Committee's (UREC) guidelines on ethical approval for studies involving working with professionals.

| Digital twin interface heuristics
We first established our usability test objectives to generate a set of suitable heuristics.These are used to define the design inconsistencies and problem areas within the DT interface.
During the operation of the software application, potential sources of error may include: • Navigation errors: Failure to locate functions, excessive keystrokes or button pushes to complete a function, and failure to follow the recommended screen flow.
• Presentation errors: Failure to locate and appropriately act upon the desired information in the displays and selection errors due to labeling ambiguities.
• Control usage problems: Improper toolbar or entry field usage.
In this work, we give importance to the usability of the DT interface as a tool to increase the safety and efficiency of the system for nuclear environments.Therefore, the objectives arising from these considerations are: • To exercise the software application under controlled test conditions with representative users.The data will be used to assess whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.
• To establish a baseline user performance and user satisfaction levels of the user interface for future usability evaluations.
As a result, the heuristics presented in Table 1 were generated.
We have adopted the current set of heuristics from existing methodologies in human-computer interaction literature (Forsell & Johansson, 2010;Nielsen & Molich, 1990;Sutcliffe & Gault, 2004) and previous work (Bremner et al., 2020).This set of heuristics may not apply to each feature at all times.Instead, they were intended to provide direction and structure to the participants' feedback.

| Experimental setup
Heuristic evaluation sessions were conducted in a controlled environment inside a well-lit room with minimal noise and visual distractions.
For this study, we implemented the virtual (i.e., DT) versions of the aforementioned robotic platforms (see Section 2).A personal computer and VR system with specifications presented in Sections 2B and 2C, respectively, were used during the evaluation sessions.
Figure 6c presents the experimental setup of the user operating the DT interface.The user is seated on a chair facing a 24-in.display monitor to operate the DT interface during the flat-screen mode (default mode).
They will then have to click (using a mouse) on the initiate VR mode button and wear the VR HMD when switching to the VR mode.

| Participants
We recruited mobile robotics experts as evaluators from two leading institutions involved in nuclear decommissioning and robotics: Sellafield Ltd. and the Japan Atomic Energy Agency (JAEA).A total of six experts (N = 6), three from each institution, were recruited to participate in the study.We have chosen this sample size based on Nielsen and Landauer's (1993) recommendations for usability tests: the optimum number of participants that can cover a majority of usability problems while taking into account resources is within three to five.The main criteria for inclusion in the study involve experience with either designing, testing, and implementing a mobile robot for a simulated or actual nuclear facility.This ensures that the evaluation data the experimenters receive will be viable in improving the usability of the DT interface in real-world nuclear applications.

| Experimental design
Each heuristic evaluation session had a duration of approximately 40 min and was facilitated by one or two experimenters.The session was divided into three parts.
• Part 1 involved an orientation, preparation, and pre-experiment interview, which lasted about 10 min.This was also the part when the heuristics were discussed in detail before commencing the session.A print-out paper containing the set of heuristics was presented beside the evaluator throughout the session.
• Part 2 used the usability evaluation of the DT in which the experts performed three tasks using the system (see Section 3.1.5Task Design).
• Part 3 involved a posttask interview to probe into the issues for each heuristic and feature.This lasted for about 5 min.
During Part 2, the expert evaluator was advised to think aloud during the performance of the tasks to cover their thought processes and problems encountered with system usability.The experimenters also noted the experts' behaviors and problems encountered while doing the tasks.All participants have signed an informed consent form and have agreed to have their conversations with the experimenters audio recorded.

| Task design
We have created three scenarios to demonstrate our DT platform's capabilities and interface.As discussed in the experimental setup, a fully virtual robot fleet was implemented.Still, it has preserved the input and interactive controls of the cyber-physical system to provide a realistic and seamless experience for the participants.The three scenarios for the experiment are presented in the following paragraphs.
Scenario 1: Manage a fleet of autonomous, heterogeneous robots in a nuclear environment: The participant is situated in an ongoing routine inspection of a nuclear facility undergoing postoperational clean-out and decommissioning.The current mission involves observing an autonomous fleet of mobile robots while being able to explore and interact with the software interface freely.The user interface elements and features assessed in this scenario were: camera view and navigation, robot first-person view, robot fleet member information (e.g., battery life bar, connectivity status), and toggle sensor views (e.g., building model information, 3D voxel map, and radiation map).Figure 4 shows an overview of this scenario.
Scenario 2: Teleoperate a live robot during routine inspection and mapping: The participant's view is then shifted from observing an autonomous operation of the robot fleet to controlling an individual live virtual robot member.In this scenario, a virtual CARMA2 robot was used to perform routine inspection via video streaming and 3D mapping.The participant was asked to map the nuclear facility by T A B L E 1 Heuristics for the digital twin interface.

Heuristic Description
Information coding Perception of information directly depends on mapping data elements to visual objects.This should be enhanced by using realistic characteristics/techniques or the use of additional symbols.

Minimal actions
Comments about workload with respect to the number of actions necessary to accomplish a goal or a task.

Flexibility
Flexibility is reflected in the number of possible ways of achieving a given goal.

Spatial organization
Comments about users' orientation in the information space, the distribution of elements in the layout, precision and legibility, efficiency in space usage, and distortion of visual elements.

Consistency
The way in which design choices are maintained in similar contexts and are different when applied to different contexts.

Recognition rather than recall
The user should not have to memorize a lot of information to carry out tasks.Minimize the user's memory load by making objects, actions, and options visible.

Remove the extraneous
Concerns whether any extra information can be a distraction and take the eye away from seeing the data or making comparisons.

Compatibility with the user's task domain
The virtual environment and behavior of objects should correspond as closely as possible to the user's expectation of real-world object, their behavior, and affordances for task action.

The natural expression of action
The representation of the self/presence in the virtual environment should allow the user to act and explore in a natural manner and not restrict normal physical actions.

Close coordination of action and representation
The representation of the self/presence and behavior manifest in the virtual environment should be faithful to the user's actions.Response time between user movement and the update of the virtual environment display should be less than 200 ms to avoid motion sickness problems (Sutcliffe & Gault, 2004).

Faithful viewpoints
The visual representation of the virtual world should map to the user's normal perception, and the viewpoint change by head movement should be rendered without delay.

Navigation and orientation
The users should always be able to find where they are in the virtual environment and return to known, preset positions.

Consistent departures
When design compromises are used, they should be consistent and clearly marked, for example, cross-modal substitution and power actions for navigation.

Match between the system and the real world
The system should communicate the users' language with words, phrases, and concepts familiar to the user rather than system-oriented terms.Follow real-world conventions, making information appear in a natural and logical order.going through the dark corners and passageways with a single light source coming from the robot's headlights.They will have to rely on the data presented by the software interface (i.e., the robot's firstperson view from the camera with a HUD overlay).Other user interface elements and features assessed in this scenario were the mini-map and the 3D view map. Figure 5 presents an overview of this scenario.
Scenario 3: Visualize data and identify hazards through a traditional flat-screen display or a VR platform.Upon accomplishing the first two scenarios, it was established that the robots had successfully surveyed the facility and collected the data needed for the mission (i.e., 3D voxel map, radiation map, images, videos, etc.) and compiled through a ROS bag.The data can then be used for offline viewing and assessment.As the final stage in the mission, a ROS bag is played and streamed to the software application for replaying and reviewing.The objective in this scenario was for the participant to visualize and identify potential hazards through the collected data on a flat screen or a VR platform.In both modes, the participant could toggle sensor views and place tags for the hazards introduced in Section 2.3 (i.e., moved objects, radiation detected, flammables, corrosives/caustics, and general caution tag).The participant can then view a separate scene where a completed inspection mission with hazard tags and annotations is present in the virtual world (see Figure 7) alongside a printed hazard inspection report.
After accomplishing these three scenarios, the experiment's main part is completed, and a posttask interview is performed.

| Results and analysis
We have compiled a total of 55 entries comprising comments and observations from the six expert evaluators due to their participation in the heuristic evaluation sessions.After organizing the entries according to scenarios and features, we assigned heuristic points, which can be positive (+) or negative (−).Each entry can have more than one heuristic point assigned to it, and there is no limit on how many points an evaluator can give for each feature.Table 2 summarizes the compiled heuristic points, organized according to the different features and heuristics adapted for the current DT interface.
A total of 65 negative and 37 positive heuristic points were generated from the evaluators' entries.The feature that received the most negative heuristic points was the hazard tagging and report generation (18).In contrast, the feature that received the most positive heuristic points was the mini-and 3D maps (15).
From the 14 heuristics, flexibility was the one that received the most negative entries ( 21), followed by information coding (10), and spatial organization (6).Meanwhile, the heuristics that received the greatest number of positive points were: navigation and orientation (9), information coding (8), and faithful viewpoints (7).
Here, we present a summary of the comments and experiences received from the expert evaluators for each of the features being evaluated:

| Camera view and navigation
Five of the six participants found it challenging and confusing to navigate the 3D scene using a combination of right-clicking the mouse button and the WSAD keys.One participant, familiar with strategic and first-person video games, mentioned that they are accustomed to using the SZXC keys for in-game navigation.
However, three participants noted that the current navigation method could be adapted with sufficient training and consistent use.An alternative suggestion for navigation is to incorporate a "click-to-move" function, where the user can click on an area or object to zoom in for a closer look.Another suggestion is to utilize the mouse scroll and display a cross-hair to indicate the camera's central point.

| Robot fleet member information
Several entries regarding the individual robot fleet member information focused on two main areas: proper information coding (N = 4) and flexibility (N = 3).One common issue was that most participants initially did not grasp the meaning of the circle icon next to the robot name, which represents the robot's connectivity status.To address this, a participant proposed using a WiFi symbol instead, making it clearer that it indicates wireless connectivity.
For the battery indicator, suggestions were made to enhance visual attention by changing the color of the battery bar, such as turning it red or making it flash when the battery is running at a critically low level.Finally, one participant suggested introducing a kill switch for each robot to introduce another layer of safety from the interface during the mission.

| Toggle sensor views
When evaluating this feature, participants encountered usability issues mainly concerning flexibility in designing and organizing a grid of buttons within a software application window.Several suggestions were made to address these issues.One suggestion was to display the names of functions when hovering the mouse over their respective buttons, providing clear information about their purpose.
Another suggestion involved implementing a dropdown list as the number of sensors to be visualized increased, ensuring more efficient use of screen space.Additionally, participants emphasized the importance of being able to toggle the visualization of an individual robot's sensors.For instance, in multirobot mapping scenarios, it T A B L E 2 Compiled heuristic points for each feature based on the results of the evaluation.would be beneficial to toggle between the 3D maps generated by each robot to analyze any variations.
It was also proposed to include icons for each robot within the software space, enhancing the overall user experience and facilitating navigation.Participants encountered confusion when interpreting the symbols used in this feature.For instance, the eye icon, meant to represent sensor data visualization, proved puzzling and bothersome for some participants.In the case of evaluators from Sellafield Ltd., they highlighted that their robots utilize various types of sensors, such as optical, thermal, acidity, radiation, and more, which are essential for capturing and recording the presence of diverse hazards in their operations.Another instance of confusion arose with the building icon, which was intended for toggling BIMs and assets.The icon depicted a building with traditional Western-style pillars and a triangular roof, which was unfamiliar to the Japanese evaluators from JAEA due to cultural differences in symbolizing everyday elements.
To alleviate these issues, participants highlighted the importance of incorporating textual information when hovering the mouse over symbols.Additionally, it was suggested to use a more universally recognizable icon for icons instead of the specific depiction previously used.

| Spawn hazard tags
Participants faced challenges interpreting icons without clear labels, similar to the information coding issues observed in the previous feature.For example, one participant confused the "moved objects" icon as a function to move objects within the DT environment physically.In light of this, a suggestion was made to introduce proper labels or textual cues to enhance icon comprehension.Another suggestion similar to the feature above was to incorporate a dropdown list as a solution for handling an increasing number of potential hazards in future interface iterations.
Concerns were raised regarding the clutter and confusion caused by numerous tags placed within the DT environment.Two participants proposed a feature allowing users to view, minimize, and hide tags and annotations selectively, preventing them from simultaneously cluttering the environment.Additionally, participants found it beneficial to have the ability to preview the location of hazard tags before they are spawned in the DT environment.This would give users an overview of where the tags will appear and allow for better planning and positioning within the space.
The ability to identify and tag hazards was most beneficial when using the VR mode.One participant expressed that their awareness of the virtual environment is enhanced by VR, attributing it positively to the heuristics of flexibility, spatial organization, natural expression of action, and faithful viewpoints.Moreover, the participant reacted positively to the navigation and orientation style of the VR mode, which was integrated using an existing SteamVR plugin for the VR headset.Another participant commented on their ability to identify more hazards and other related incidents in VR, which would not be possible when using the default flat-screen mode alone.

| Generate hazard report
In nuclear decommissioning sites like those in Sellafield Ltd., the conventional method of reporting inspection missions involves manually generating reports using a template.The current software application has introduced a feature that allows reports to be generated from hazard tags and annotations, which the participants found useful.Several suggestions were made to enhance this feature.First, it was proposed to relocate the virtual camera to the hazard's location when clicked from the generated report.This would provide a more immersive and focused view for further investigation.
Participants have also recommended implementing the ability to upload an image of the hazard, which could be viewed in the annotations and included in the report.Furthermore, participants suggested expanding this feature to include tagging incidents, such as repairs needed, equipment issues, or areas requiring special attention.They also emphasized the importance of keeping the generated reports simple, avoiding excessive complexity or cluttered information.Participants noted that a more detailed report could be generated using alternative methods, but a straightforward and concise report through this feature would be beneficial.

| Robot first-person view
The robot fleet's first-person camera view, accompanied by a HUD showing the robot's name, battery status, and connectivity information, allows users to monitor individual robot activities closely.In the first scenario, where managing an autonomous robot fleet is the focus, most participants appreciated the option to view each robot's camera feed through their respective channels.However, four evaluators expressed the need for a camera feed grid to be displayed either as a side panel or in a separate window.This would enable users to supervise assigned tasks without losing sight of the overall mission.Additionally, two participants suggested implementing the ability to focus on a specific camera feed if desired, as well as the option to switch between the grid view and individual robot feeds, which would cater to different application requirements.The camera grid would be beneficial for general tracking, while the per-robot camera feed would be helpful for focused activities.
In the teleoperation scenario, all participants voiced difficul-

| Mini and 3D map
In the teleoperation scenario, the robot's camera feed was accompanied by two additional windows: a mini-map and a 3D map view (shown in Figure 5).These windows provided the operator with extra information from LiDAR and sensor scans, helping them navigate around obstacles in the facility.Most participants found the mini-map and 3D map view beneficial, and they assigned positive heuristic points to aspects, such as "recognition rather than recall," "faithful viewpoints," and "navigation and orientation."This feature received 15 positive heuristic points and no negative heuristic points.The mini-map, in particular, was frequently used by participants to avoid obstacles and perform complex maneuvers in tight spaces due to its top-down perspective.Our newly developed user interface facilitates the adaptability of cyber-physical robot fleets in high-stakes industrial environments, specifically the nuclear sector, by incorporating immersive human-robot interaction modalities, such as VR (Roldán Gómez et al., 2019;Stadler et al., 2023).VR technology offers the potential to significantly enhance the situational awareness of nuclear workers, enabling them to experience the nuclear site environment without being physically present.However, it is crucial to recognize that not all tasks necessitate VR operation, and the continued utilization of traditional flat-screen interfaces is essential.With this in mind, we have implemented a versatile solution that allows users to seamlessly switch between these two modes, accommodating diverse operational requirements and preferences.
In addition to the multimodal approach, we have provided modular visualization options to the platform by enabling the addition and toggle of robot sensor data and DT assets into a single operating virtual environment.With the inclusion of nuclear-specific capabilities (e.g., visualizing radiation hot spots, spawning hazard tags, and generating reports that provide more information to stakeholders in nuclear settings), more value is added to using this DT platform compared to simple teleoperated robots with a camera feed.
The current academic database indicates that this was the first implementation of a cyber-physical robot fleet operating within a digitized mock-up nuclear environment-the RAICo1 Facility in Whitehaven, West Cumbria, UK, which accurately depicts a realworld scenario.By utilizing the RAICo1 DT environment, we enable realistic simulations, optimize performance, and facilitate effortless modifications of digital assets and game objects in a controlled environment.Our approach to data visualization incorporates a huebased color mapping system to effectively relay information on radiation hot spots using sensor data from our robots.This enables the users of our platform to interpret radiation levels with clarity and precision.Drawing inspiration from various first-person and strategic video games, we have also developed a new HUD for individual teleoperation of live robots, which includes features such as robot information overlaid on the camera feed, battery life (health bar), network connectivity, mini-map view, and 3D map view.
To assess the usability of our DT platform, we opted for a heuristic evaluation approach, which allowed us to investigate the usability of our system through expert evaluation.While there are inherent limitations to using this approach, such as potential subjectivity and reliance on evaluators' expertise (Nielsen & Molich, 1990), it provided us with a valuable means to pilot the usability of our platform in a controlled manner.
The results of our heuristic evaluation have provided significant insights derived from the experts' entries and the cumulative heuristic points assigned to each feature.These data have highlighted three major themes concerning the usability of our platform: (1) the necessity for a flexible software interface, (2) the importance of maintaining the individuality of each robot in the fleet, and (3) the potential for expanding the interface's sensor visualization capabilities.

| Flexible software interface
The ability to navigate the 3D scene using various methods was identified as a crucial requirement.The current navigation method, involving a combination of mouse clicking and keyboard keys, proved challenging and confusing for some users.To address this, participants proposed alternative navigation options, such as incorporating a "click-to-move" function or utilizing the mouse scroll with a central point indicator.These suggestions emphasize the importance of offering users multiple ways to interact with the interface based on their preferences and familiarity with different navigation methods.
This theme supports the findings by Daniel et al. (2014), where an industrial robot's more flexible graphical user interface promotes an intuitive human-robot interaction, reducing the duration of task execution and decreasing the number of required interactions.

| Individuality of each robot in the fleet
Evaluators emphasized the need to distinguish between robots and provide relevant information specific to each one.This individuality allows users to track and monitor the activities of different robots more effectively.A suggestion was made to incorporate buttons for toggling sensor views for each robot to enhance the overall user experience and facilitate navigation.Furthermore, a collective suggestion was to arrange the individual camera views in a grid layout, enabling users to easily identify and differentiate between robots and their respective tasks.This layout ensures efficient management of the fleet.

| Expanding sensor visualization capabilities
The current version of the interface has limitations when it comes to visualizing a large number of sensors.Evaluators suggested revising this aspect to accommodate a broader range of sensors.
One proposed solution is to incorporate a dropdown list, which allows for more efficient use of screen space as the number of sensors increases.This revision addresses the usability issues raised by participants regarding the design and organization of the sensor visualization grid.By providing a dropdown list, users can easily select and visualize specific sensors based on their requirements.

| Design recommendations
Through an iterative approach, we present the findings and insights from our comprehensive heuristic evaluation into a design recommendation framework, represented by the conceptual interface shown in Figure 8.
This framework includes consideration of the three major themes that were brought up in the usability study: 1. Flexible camera and scene view navigation options: -Implement a way to pan and scroll the camera through the mouse, keyboard, or VR controller.
-Set a waypoint or implement a "click-to-move" function.
2. Individuality of each robot in the fleet: -Improve options to toggle views for each sensor (e.g., point cloud, thermal, and nuclear radiation types).
-Introduce a function to toggle multiple sensor views for each robot.
-Arrange camera views in a grid layout within the main window or as a separate one.

Expanding sensor visualization capabilities:
-Introduce a dropdown button for additional views, spawning hazards, hazard reports, and more.
-Enhance the capability to visualize nuclear-specific challenges addressed by additional sensors.
The integration of ROS and Unity often involves the use of that it is feasible (Kivrak et al., 2022).Techniques such as compression, streaming, and asynchronous processing can be further employed in future work to minimize latency.
The resulting recommendations aim to optimize the user interface design, ensuring a seamless interaction between operators and the DT platform in nuclear environments.

| Limitations
Several limitations were identified in our system design and its evaluation.First, our cyber-physical system encompasses only a narrow spectrum of the DT evolution.Cyber-physical systems and their DT components contain various stages of autonomy, data representation, and cybersecurity (IBM, 2022;Rao, 2022).Currently, our system focuses on incorporating a robot's location and mapping data, anchoring it to a fixed frame in the digital world, and relying on the robots' positioning in both the physical and virtual worlds.In future iterations, we plan to expand the system's capabilities by incorporating a wide range of predictive and cognitive algorithms, leveraging artificial intelligence and machine learning.These enhancements will enable the system to address critical aspects of industrial asset management applications, such as autonomous operation, data processing, and cybersecurity.
Another limitation of this work pertains to the heuristic evaluation approach employed to assess the system's usability.We acknowledge that our approach in recruiting robot operators working in nuclear environments may lead to expertise bias.As with any heuristic evaluation approach, the evaluators in the study may possess biases or expertise that could influence their feedback.For instance, if the evaluators have extensive experience in robotic design or human-computer interaction, which is common in research institutions, their perspectives may differ from those of other target users, such as manual inspection workers.Consequently, the broader applicability of the study's findings may be limited.Nonetheless, we determined that the heuristic evaluation approach was the most suitable method at this development stage.For future usability studies, we aim to implement a quantifiable method to assess the DT platform's performance, operability, and robustness.This assessment should be conducted separately for the flat-screen and VR modes within the same use cases.Overall, this multimodal immersive DT platform has the potential to revolutionize the management of cyber-physical robot fleets in nuclear environments and make a significant impact on the nuclear industry.

Figure 1
Figure 1 presents the members of the heterogeneous robot fleet with their specialized features.A fleet comprising four heterogeneous robots was considered for the mission: (1) AgileX Scout 2.0 unmanned ground vehicle (UGV) for general three-dimensional (3D) mapping, (2) Clearpath Husky UGV with Universal Robots UR5 manipulator for remote handling, Continuous Autonomous Radiometric Monitoring Assistant (CAR-MA2) UGV for radiation and aqueous detection and mapping, and DJI/Ryze Tello unmanned aerial vehicle (UAV) for aerial inspection and image capture.The AgileX Scout 2.0 (Figure 1a) is a versatile mobile robot designed for agile exploration and inspection tasks (SCOUT 2.0-Agilex Robotics, n.d).It is equipped with an NVIDIA Jetson Xavier NX processor and an integrated inertial measurement unit sensor for precise navigation.The robot's modular design allows for easy customization and expansion with additional sensors and payloads.It features a 360°Velodyne VLP16 Light Detection and Ranging (LiDAR) sensor for obstacle avoidance and mapping and a RealSense depth camera for visual inspection.The robot can travel at speeds of up to 1.5 m/s and has a battery life of up to 6 h.
2.2.1 | Robot operating systemROS is a widely used software framework for robot development.It involves a collection of software packages that provide a wide range of functionality, such as communication between systems and processes (ROS nodes), hardware abstraction, sensor data processing, and visualization tools.In this implementation, we used the Melodic Morenia version of ROS (called ROS Melodic) on Ubuntu 18.04 (Bionic Beaver) as middleware to integrate members of the robot fleet into a single operating network.Aside from the Android-based Tello UAV, the rest of the robot fleet members (i.e., Husky, Scout 2.0, and CARMA2) use ROS Melodic in their onboard computers.The ROS Master Node is operated from a virtual machine (VMWare Workstation 16 Player) installed on a Windows 11 × 64 personal desktop computer with a 24-in.display monitor, 3.60 GHz Intel Xeon W-2123 processor, and a 32-GB of random access memory.Furthermore, the machine has a dedicated F I G U R E 1 Members and general functions of the cyber-physical robot fleet: (a) AgileX Scout 2.0 for general 3D mapping, (b) Clearpath Husky with UR5 manipulator for remote handling, (c) CARMA2 for radiation and aqueous detection and mapping, and (d) DJI/Ryze Tello for aerial inspection.3D, three-dimensional; CARMA2, Continuous Autonomous Radiometric Monitoring Assistant; FMCW, frequency-modulated continuous-wave; LiDAR, Light Detection and Ranging.[Color figure can be viewed at wileyonlinelibrary.com] graphics processing unit: NVIDIA GeForce GTX 1080 Ti.This machine serves as the central base station of the cyber-physical robot fleet and is where the operator can interact with the DT interface.Each robot instance was then connected to the master node via the ROS_MASTER_URI through a wireless network.Communication between the fleet members and the master node (central base station computer) was initiated and confirmed through their unique ROS_IP addresses.
ences.Our work used the 2021.3.0fsversion of the platform to develop the DT environment.We also utilized the Unity Robotics Hub, a central repository for tools, resources, and documentation for robotic applications in Unity 3D.Unity Robotics Hub communicates with ROS via the Transmission Control Protocol (TCP).From the ROS side, a ROS-TCP-Endpoint package is launched to publish data from the relevant topics contained in the integrated ROS network.A ROS-TCP-Connector package from the Unity 3D side is then imported to the Unity scene file with the identification of the ROS_MASTER_URI address.Upon running the Unity 3D scene, a successful connection can be confirmed through the in-game ROS dashboard by displaying a blue two-way arrow.Through the ROS-TCP-Connector and the Unity Robotics Visualizations packages, the DT platform can relay ROS messages and visualize sensor data (e.g., point cloud, occupancy grid, F I G U R E 2 System architecture and data pipeline of the cyberphysical robot fleet based on the ROS-Unity 3D framework.Each fleet member presents a specialized function in the mission and relays data wirelessly to the base station.Data are then processed and visualized in Unity 3D to translate useful information to the user via traditional flat-screen display or VR.3D, three-dimensional; CARMA2, Continuous Autonomous Radiometric Monitoring Assistant; HMD, head-mounted device; VR, virtual reality.[Color figure can be viewed at wileyonlinelibrary.com]

F
I G U R E 3 The digital twin environment's assets and elements in Unity 3D: (a) building information model of the RAICo1 Facility constructed from existing building plans, (b) Octomap voxels generated from point cloud data and the imported robot models, and (c) visualization of radiation cost maps from the ROS radiation plugin which simulates the radiation sensor of the CARMA2 UGV.3D, three-dimensional; CARMA2, Continuous Autonomous Radiometric Monitoring Assistant; RAICo, Robotics and Artificial Intelligence Collaboration Hub; ROS, Robot Operating System; UGV, unmanned ground vehicle.[Color figure can be viewed at wileyonlinelibrary.com] orientation by moving the mouse while the right button is pressed.

F
I G U R E 4 The digital twin interface of the cyber-physical robot fleet on a traditional flat-screen display.The user interface includes interactive elements such as (a) display information from robot fleet members, (b) ROS topics dashboard, (c) toggle building information and sensor views, (d) spawn hazard tags, (e) switch to VR mode, and (f) generate hazard report.BIM, building information model; CARMA2, Continuous Autonomous Radiometric Monitoring Assistant; ROS, Robot Operating System; VR, virtual reality.[Color figure can be viewed at wileyonlinelibrary.com] view.The device is equipped with SteamVR Tracking 2.0, a G-sensor, a gyroscope, and a proximity sensor, which accurately tracks the user's movements.The SteamVR plugin provides a software development kit for Unity 3D that enables the creation of VR experiences for HTC Vive through its tracking and input modules.Furthermore, SteamVR provides a framework for optimizing performance and minimizing latency in VR applications, which is essential for creating immersive and comfortable experiences for users.F I G U R E 5 Robot first-person view consisting of three panels: camera feed with heads-up display (left), 3D map view and navigation (top right), and robot "mini-map" (bottom right).3D, three-dimensional; CARMA2, Continuous Autonomous Radiometric Monitoring Assistant.[Color figure can be viewed at wileyonlinelibrary.com] Each participant was provided with a participant information sheet F I G U R E 6 The digital twin (DT) interface in virtual reality (VR) mode: (a) radial menu for placing hazard tags using the right-hand controller, (b) teleportation within the DT environment using the left-hand controller, (c) VR mode user setup, and (d) DT interface in VR.HMD, head-mounted device.[Color figure can be viewed at wileyonlinelibrary.com] and privacy notice statement and signed an informed consent form before participating in the evaluation session.All documents containing personally identifiable information and anonymized audio recordings have been securely stored by the data handler in compliance with UREC and the UK's implementation of the Data Protection Act 2018-General Data Protection Regulation.
Sellafield Ltd. is a nuclear decommissioning site license company in West Cumbria, United Kingdom.Initially established in the 1940s, it has since engaged in various nuclear activities, including fuel manufacturing and reprocessing.The site's extensive experience in the nuclear industry has made it a global leader in waste management, site remediation, and nuclear decommissioning.Sellafield Ltd. has also become a hub for nuclear robotics and automation, collaborating with universities and industry leaders to advance state-of-the-art technology for the safe and effective decommissioning of nuclear facilities.JAEA is a governmental research organization focused on nuclear science and technology, including decommissioning and robotics, due to Japan's many aging nuclear facilities.JAEA has decades of experience in these fields, including the decommissioning of the Fukushima Daiichi Nuclear Power Plant after the 2011 nuclear accident.JAEA has developed robots capable of operating in highly radioactive environments to improve safety.

7
Hazard tags and annotations in the digital twin interface.FMCW, frequency-modulated continuous-wave.[Color figure can be viewed at wileyonlinelibrary.com] ties maneuvering the mobile robot through tight spaces and narrow passageways within the RAICo1 DT environment.They struggled with navigation and encountered challenges in driving the robot effectively.A common suggestion to address this issue was to implement the principles of shared autonomy by utilizing waypoint navigators that already exist in certain mobile robot platforms equipped with simultaneous localization and mapping capabilities.By implementing this feature, users could guide the robot along predetermined paths, improving navigation in complex environments.
General discussionThis work presents the design and implementation of a multimodal immersive DT platform for heterogeneous cyber-physical robot fleets operating in nuclear environments.Our primary objective is to enhance the safety, efficiency, and reliability of deploying robot fleets within the nuclear industry by offering a near real-time monitoring and management tool applicable to various commercial or customized robotic platforms.Additionally, we conducted a usability study by employing heuristic evaluation methods and engaging experienced robot operators the nuclear energy sector.
middleware and communication protocols.ROS is commonly used for managing hardware abstraction, device drivers, communication between processes, package management, and more on the physical robot.Unity, on the other hand, provides a platform for building digital simulations and interfaces.For dynamic sensor data like F I G U R E 8 Proposed user interface redesign for the digital twin platform based on the heuristic evaluation involving nuclear robot operators.[Color figure can be viewed at wileyonlinelibrary.com] images from cameras and point clouds from LiDAR, achieving near real-time communication (i.e., within milliseconds) can be challenging.Still, it is currently an active area of research in the field.Our previous work investigating the factors affecting the real-time network communication of environmental sensor data from robots suggests This article presented a multimodal immersive DT platform for cyber-physical robot fleets operating in nuclear environments, focusing on enhancing the safety, efficiency, and reliability of their deployments in the nuclear industry.The system design enables seamless integration with diverse robotic platforms through the ROS-Unity 3D framework, allowing near real-time monitoring and management of fleets with commercial and customized robots.The methods and results introduced in the modified heuristic evaluation, conducted with robot operators from the nuclear energy sector as expert evaluators, have been the main contribution of this article to the field, providing valuable insights into the usability of the platform and accelerating our design process by accounting for both the positive and negative aspects of the DT platform.The three usability themes arising from this study were the flexibility of the software interface, the individuality of each robot in the fleet, and adaptation with expanding sensor visualization capabilities.Other contributions discussed in this article refer to developing cyber-physical systems for high-stakes industry applications.First, the DT interface addresses the specific challenges of nuclear environments, offering a comprehensive and intuitive interface for managing and controlling robot fleets.Integrating near real-time data, immersive visualization, and interactive control mechanisms enables operators to make informed decisions and respond effectively to changing scenarios and hazards.In addition, participants' reliance on features such as the mini-map and the nuclear-specific sensor visualizations demonstrates how this DT significantly increases the safety of deploying semi-autonomous robot fleets in nuclear environments.Second, the platform's multimodal nature, incorporating VR and traditional flat-screen interfaces, accommodates diverse operational requirements and user preferences.This adaptability enhances the situational awareness of nuclear workers and allows for seamless switching between different modes of operation.The approach of providing a flexible and modular platform by allowing expansion with new sensing capabilities caters to the diverse needs of nuclear facilities, spanning different use cases, including operational stages and decommissioning processes.As a result, the DT platform becomes a versatile tool that adapts to the specific requirements of each nuclear facility, ensuring optimal performance and effective utilization of assets and resources.Future work can focus on integrating advanced features such as artificial intelligence algorithms for autonomous decisionmaking, enhanced data analytics for predictive maintenance, and further integration with existing nuclear facility systems.More extensive field trials and user studies in operational nuclear facilities will provide valuable insights into the platform's performance, adaptability, and overall impact on nuclear operations.By addressing these areas, the platform can further contribute to advancing nuclear facility management and protecting human workers in hazardous industrial environments.
Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/rob.22329by Welsh Assembly Government, Wiley Online Library on [11/04/2024].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License Downloaded from https://onlinelibrary.wiley.com/doi/10.1002/rob.22329by Welsh Assembly Government, Wiley Online Library on [11/04/2024].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License