A Computer‐Aided Teleoperation System for Intuitively Controlling the Behavior of a Magnetic Millirobot within a Stomach Phantom

Untethered magnetic millirobots with a characteristic length of a few millimeters can be wirelessly controlled. They exhibit promising potential in a wide variety of applications, particularly for tasks in clinic workspaces. However, magnetically controlling these robots is counter‐intuitive and requires a steep learning curve, hindering their wide adoption. Herein, a computer‐aided teleoperation platform is developed to operate a soft millirobot, with its feedback control being conducted behind‐the‐scenes, bridging the user's inputs directly with the millirobot's actions to offer an intuitive control. This system enables untrained users to conveniently control the position and actions of the millirobot inside a human stomach phantom by pointing‐and‐clicking on a real‐time video monitor or using a keyboard. The platform automatically materializes the user's instructions by maneuvering a robotic arm with a tip‐mounted magnet to exert a magnetic field to induce the desired response from the millirobot. Experiments show that the system allows the user to intuitively operate the millirobot and deliver its cargo without splitting their attention to monitor the workspace or to calculate the constantly changing control parameters. This platform can lower the barrier for healthcare practitioners without engineering expertise to adopt miniature robotic systems into their workflow and realize these systems’ promising potential.


Introduction
[17][18] The millirobots not only assist doctors to complete high-precision and highquality surgical operations but also reduce the risk of surgery and shorten the operation time. [19]In the practical application of millirobots, the common examples include an endoscope that is directly inserted using an external tube connection and an external line that is directly connected to a robot that supplies energy. [20]For minimally invasive surgery, tiny magnetic robots can reach areas that are difficult to reach with traditional surgical methods.It has great potential to improve the surgical efficiency and safety of the operation. [21]However, those robots required a complex manipulation system and an experienced operator to complete the corresponding operation.The high degree of complexity in the peripherals to drive these robots increases the cost and the difficulty of the manipulation process, and an inapposite driving method or inappropriate control signals may cause a robot to damage the biological tissues that are in its vicinity.Therefore, the efficient and safe control of these novel miniature robots is a problem that is as challenging as their initial design.
Magnetic control can be easily tuned by changing the parameters relevant to its working principles, such as the magnet's size and strength, the distance between the driving magnet and the robot, and the robot's magnetic moment.Varying the strength of the control magnetic field or the magnetic moment of the robot can regulate the force applied by the robot upon the biological tissues. [22]Utilizing magnetic fields has become a popular choice in controlling miniature robots in recent years.For example, some magnetic miniature robots can deliver drugs within the natural cavities and ducts of organs, such as intestines, [23,24] or during a biopsy within the stomach. [25]Thanks to the fact that magnetic control can instantaneously and simultaneously exert both force and torque on the robot while safely penetrating biological substances, [26] most of these robots do not need to carry onboard power sources or control units, leaving ample space to transport other essential loads, such as therapeutic drugs or imaging contrast agents for diagnostic purposes.In addition, the magnetic field can be conveniently generated using permanent magnets and electromagnetic coils, and there are many parameters (i.e., direction, strength, and spatiotemporal variance) to be manipulated. [27] magnetic field can be generated by an electromagnetic coil system (EMCS) or a permanent magnet system (PMS).Both of these systems have been employed to control magnetic miniature robots, each with distinct advantages and limitations.For an EMCS, a three-axis Helmholtz coil has been used to drive magnetic spiral microswimmers, [28] and four magnetic coils with magnetic cores have been used to drive a magnetic wireless capsule robot. [29]Also, an eight-pole electromagnetic system has been developed to generate a three-dimensional magnetic field to control a clamp-shaped micromachine. [30]However, the magnetic field from an EMCS has a limited working space and requires continuous power to sustain a magnetic field.Efforts have been made to partially relax the workspace constraint, but the high power consumption is so far still unavoidable. [31][34] Nevertheless, the magnetic field generated by a PMS is highly nonuniform, and its direction, strength, and spatiotemporal variance are all coupled together.As a result, most previous studies have reported that controlling a PMS is counter-intuitive and requires manual actuation by experienced operators, in which case a high-precision actuation is still rather challenging, and the operator needs to stop from time to time to carefully consider the next maneuver. [35,36]An alternative to the direct manual control of a PMS is fixing the PMS to the tip of a robotic arm for accurate and precise movement and rotation.With the introduction of a robotic arm, the accuracy of the magnet's movement can be greatly increased, and the risk of injuries related to handheld magnets can be reduced. [37]As an example, a millirobot in a blood vessel was operated using a robotic arm and a PMS system. [21]With the help of the robotic arm, the highly heterogeneous magnetic field near the magnets could be turned into an advantage to achieve any desired magnetic field configuration.However, in an actual clinic workspace, maneuvering the robotic arm at a small distance away from a patient poses serious safety concerns.Thus, carefully designed methods and programming of the robotic arm are urgently needed to utilize the advantages of a PMS while simultaneously minimizing these risks. [38,39]t has been shown that it is feasible to apply dynamic forces and/or torques to control the translational and rotational motion of a magnetic millirobot by changing the direction of a single external permanent magnet (EPM). [40,41]Direct usage of magnetic gradient field dragging as the primary movement method is likely to cause excessive attractive forces, resulting in undesired damage to tissue in the workspace, whereas a regularshaped robot requires relatively little torque to rotate, and it is easy to control the position of the robot.44] Using a permanent magnet to drive a magnetic robot is counter-intuitive considering the highly nonuniform response of the magnetic field created by the magnet.For example, if the user rotates the driving magnet, the robot will roll correspondingly, but in the opposite rotating direction of the magnet. [45]As a result, this kind of robot maneuvering requires a fundamental understanding of magnetism and extensive training in magnetic manipulation.However, most people working in real-world healthcare lack the required theoretical background in magnetic control principles, and there is a steep learning curve for doctors and nurses to become proficient in controlling magnetic robots directly.Due to the steep learning curve, doctors with limited practical experience after short-term learning are more prone to making mistakes during practical applications. [46]ven after the operators have been appropriately trained, a high probability of making mistakes in time-sensitive surgical operations still exists due to the inherent counter-intuitiveness and inconvenience of magnetic control.But in the use of a magnetic flexible endoscope test, an intuitive operating system led by a computer and a robotic arm enables users with no relevant background to achieve proficiency with less practice. [47]Therefore, it is necessary to develop a system that operators can intuitively manipulate to control the robotic behavior without requiring extensive engineering knowledge or intensive training.As discussed by Martin et al. [48] there are three layers of control in a system that includes a robotic arm.Each layer has different concerns in terms of system control, and the system autonomy increases as the number of layers increases.In the first layer, the user controls the movement of the robotic arm instead of directly controlling the miniature robot's movement, meaning this layer of the system has little autonomy.In the second layer, the user directly controls the position of the miniature robot and does not need to be aware of the robotic arm's presence and actions.Users can intuitively control the robot in real time in this layer.Compared with the first two layers, the third layer provides the most autonomous control.The operator only needs to set the target position, and then the system will calculate the required route and method to move the robot to the target position.This layer greatly reduces the user access barrier and training costs.Considering this three-layer framework and the fact that the most intuitive way for an operator to perceive the robot's position is to directly observe the robot inside the patient, [49] this current research has developed a third-layer control system to reduce the difficulty of controlling the magnetic robot inside the body for users who lack knowledge of magnetism.
Considering some drawbacks of the existing magnetic actuation methods, as shown in Figure 1a,b, for the electromagnetic coil control system, the robot is positioned at the geometric center of the system, enclosed by the electromagnetic coils.A magnetic field in an arbitrary 3D direction can be generated by modulating the electric currents running in each of the coils.However, the main shortcomings of this system are a constrained workspace and a large power consumption.The magnetic field generated by the electromagnetic coil at the center is proportional to the current strength passing through, and it is inversely proportional to the center position and the distance from the coil.To generate a corresponding magnetic field at the center requires increasing the current intensity or reducing the robot's working space.Therefore, the millirobot placed in the center of the solenoid can be activated freely but mostly only in a restricted space.Such as some EMCSs can generate magnetic fields in any 3D direction, but the working space is restricted. [50,51]For the handheld permanent magnet setup to manually control a robot, this method has safety and accuracy issues.The magnetic field variation trend of the driving magnet varies greatly with distance.And human hand cannot accurately control the distance between the magnet and the millirobot, which is likely to cause the wrong distance between the magnet and the robot.When the distance is too small, it causes too much attraction to harm human tissue, or when the distance is too large, it cannot generate enough magnetic force to drive the robot.The handheld permanent magnet cannot be pinpointed a location on a millimeter scale resulting in the inability to drive the robot to the correct position.It also requires users to have relevant knowledge and experience to overcome the counterintuitive nature of magnetic maneuvering.In this article, we designed a novel system that can overcome the counter-intuitive control issues of using permanent magnets to directly control tethered soft millirobots with simple operation procedures and a high degree of accuracy.Compared with the existing magnetic control system, the newly proposed system overcomes the problem that the working space is restricted while avoiding the high energy consumption required by electromagnetic coils.Besides, it establishes an intuitive "direct linkage" between user input and millimeter-scale magnetic robots.This system combines computer vision, robotic arm movement principles, the rotary motion of a motor, and the properties of permanent magnets to form a platform with direct point-to-point and intuitive control between the operator and the millirobot.The user's attention only needs to focus on the robot's behaviors and does not even need to be aware of the existence of the robotic arm.Thus, the system can reduce the requirement of the engineering and magnetism knowledge of users.We also propose a new soft cuboidal magnetic gripper (SCMG) that can conduct cargo delivery tasks under the control of the externally exerted magnetic field.This SCMG can be effectively maneuvered to transport and release cargo on the inner stomach wall via the magnetic field created by a driving a permanent magnet.In an imaginary medical intervention scenario, the patient would swallow the SCMG and lie face down on his or her stomach on the operating table; then the doctor (operator) would simply click on the screen to set the target point for the SCMG within the patient's stomach, and this platform would automatically drive the gripper to the target point to release its cargo (drugs) with an error range of less than 4 mm.

System Development
This section introduces the basic components of the system, the process of the system's construction, and the design of the newly proposed SCMG.

System Design
The key goal of this system is to create an intuitive and direct linkage between the user's inputs and the millirobot's actions.As shown in Figure 1c, the system consists of three main parts: the user console, workspace, and connecting computer.A motor is installed at the end of the robotic arm, which not only overcomes the rotation-angle restriction of the distal joint of the robotic arm but also independently controls the rotation of the driving magnet to deal with different situations during the manipulation.To establish a direct connection between the millirobot and the user's inputs, the connecting computer is used to directly connect the mechanical equipment in the workspace and the operating equipment in the user console.
In the workspace, the robotic arm controls the precise position of the driving magnet, and the motor controls the magnet's rotation.The control signals required by the two mechanical devices are not directly inputted by the user; instead, the connecting computer converts the user's command into the electric signals for the motor rotation and the coordinates for the robotic arm's movement according to the magnetic field required to drive the robot under different preset conditions, thus driving the magnet to complete the action.During the driving magnet rotating process, the computer will compare the coordinates of the robot and the target point in real time.First, the position of the driving magnet was moved by the robotic arm when the user chose the target point.Then, by comparing the coordinates of the millirobot and the target point, the drive magnet is tilted 20°to the left or right along the Y-axis to better rotate the robot.For instance, the magnet tilts to the left when the target point is at the bottom left and top right of the robot, and tilts to the left when the target point is at the bottom right and top left.The real-time coordinate comparison determines the rotation direction (opposite direction from the millirobot to the target point) of the driving magnet and it can be changed by the code real time.When the millirobot reaches the targeted region, the computer immediately sends a signal to stop rotating, and the magnetic field at this time will anchor the robot in the designated position.The real-time feedback received by the user comes directly from a top-view camera mounted directly above the workspace.The millirobot's working area is located above the robotic arm and is separated from it by a partition board.According to the user's control and view, the behaviors of the robot directly correspond to the user's input, and the route-planning and movement details are calculated by the computer in real-time behind-the-scenes.The robotic arm's movement area is completely separated from the space reserved for the patient to establish the third layer control system [48] between the user and the millirobot and, at the same time, to achieve a high safety standard. [52]2.The Design and Fabrication of the SCMG Due to the complex and uneven surface conditions of the inner stomach wall, rotating the whole body of the SCMG is an effective method to drive the millirobot across it.The SCMG is made from a mixture of Ecoflex-50 and permanent magnetic microparticles (NdFeB, Magnequench, average diameter of 5 μm) with a mass ratio of 1:2.The SCMG is soft after curing.A magnetic soft body makes the robot can be easily deformed to release the cargo under the magnetic field.In contrast to the tough material millirobot, soft material can deform their shape to adapt the contact surface to increase the contact area.This results in lower pressure compared to the tough material under the same magnetic field.In this work, to use magnetic torque to drive the magnetic robot under a weak magnetic field and to ensure the millirobot can move across obstacles without slipping, the SCMG was designed as a cuboidal structure since the edges of the cube can fix the SCMG on a slop or obstacle with a fixed magnetic field.The SCMG always maintains a closed state when moving due to its small magnetic moment.The body length of the SCMG is 6 mm, and it has eight movable hands: four on the upper surface and another four on the bottom surface.Figure 2a shows the SCMG and the movement of its hands.The eight hands can freely rotate along the connecting edge of the gripper to form an open or a closed state due to the soft properties of the materials.As shown in the closed state of Figure 2a, there are certain angles between the magnetic moment of each moveable hand (the small red arrow) and the central axis of the SCMG to ensure the correct direction of the moveable hands' rotation.The direction of the magnetic moment (the large blue arrow) of the whole SCMG points from the center of the bottom surface to the center of the upper surface, and the inside chamber volume is 64 mm 3 .The SCMG can rotate along the central axis under a weak rotating magnetic field to move over obstacles.Under a strong magnetic field, the SCMG will be opened, and the inside chamber will be compressed by the side walls as shown in the open state of Figure 2a.The deformed shape of the SCMG in its open state is also the shape used during the magnetizing step.To obtain the magnetic moment distribution of the active hand in the closed state, the moveable hands are not totally flat during the magnetizing process.
The behaviors of the SCMG are shown in the Figure 2b.When the external magnetic field applied in the direction of the magnetic moment of the SCMG is 13 mT, the movable hands of the SCMG begin to open.When the magnetic field intensity reaches 45 mT, the SCMG becomes to open state.The angle represents the angle between the magnetic moments of the SCMG and the driving magnet.
The fabrication of the SCMG is shown in Figure 3a.The first step was the mold casting process, which involved 3D printing of the casting mold and injecting the liquid NdFeB mixture.The next step was assembling the three casting molds and curing the mixture for 4 h.Once the mixture was fully cured, the upper and bottom casting molds were separated, revealing the four movable hands on each of the upper and bottom surfaces of the SCMG that were formed due to the design of the center casting mold.Next, the center mold was pulled out, and the SCMG was magnetized in its open state.The end result was an SCMG with a 6 mm body length, as shown in the finished product in Figure 3a, where the white dash lines represent the gaps between the moveable hands.
Figure 3b presents the snapshots of a preliminary experiment to verify the functionalities of the SCMG.The SCMG was put inside a medical stomach phantom and was successfully driven to move and release its cargo in the targeted region using a handheld permanent magnet.The aim of this experiment was to establish the foundation for a direct linkage between the desired action of the SCMG and the actions of the robotic arm and motor.It configured the specific magnetic fields and the corresponding hardware control signals that are required for different SCMG actions.

Analysis of the Magnetic Driving Method
In this subsection, the magnetic force and torque between the driving magnet and the millirobot are calculated, and the range of distances appropriate for effective driving is determined.The path of the driving magnet will be planned based on these results.
Rolling is one of the most efficient ways to move millirobots over obstacles. [53]The magnetic millirobot will roll due to the applied torque when the driving magnet rotates.Magnetic force and torque vary significantly with a change in the working distance.An overly small actuating distance is likely to cause damage to the workbench and injuries to the patient's body, while an overly large actuating distance will result in insufficient magnetic force to drag the robot over some obstacles.Thus, strict distance control is necessary when using the magnetic field to drive the magnetic robot.Applying torque at a specific controlled distance and direction allows the robot to move to the targeted point with high precision.The locomotion capability of the millirobot proposed in this project can be achieved by magnetic torque, and the opening and closing functionalities of the SCMG can be easily controlled by decreasing or increasing the distance between the driving magnet and the millirobot.
The upper figure of Figure 4a is the schematic figure of locomotion.The red dashed lines are the magnetic moment of the SCMG and the moveable hands, and the blue dashed lines show the direction of the magnetic field generated by the driving magnet.The EPM moves first, rotating in the direction determined by the detection function.During this process, the SCMG keeps itself in the closed state, and the magnetic moment of the whole robot follows the direction of the magnetic field generated by the driving magnet at the SCMG's position, so it can move over the rugae inside the stomach.The lower figure of Figure 4a shows how the SCMG turns to the open state from the closed state to release its cargo under wireless magnetic field control.When the SCMG is moved closer to the EPM, the magnetic force applied on the moveable hands of the SCMG increases, and it will begin to open and release its cargo when the magnetic force is larger than the elastic force that maintains its closed state.

The Magnetic Interaction
The force and torque exerted by a magnetic field B on the SCMG are denoted as F and τ, respectively.For the calculation of the magnetic field of a permanent magnet, the magnets used in this article are considered to be magnetic dipoles.The driving magnet selected for the experiment is an EPM with a side length of 38.1 mm, its magnetic moment m 1 is measured to be 60.4A m 2 , and its magnetization is 1.09 Â 106 A m À1 .The millirobot has a cubical shape with a side length of 6 mm.Its magnetic moment m 2 is 1.2 Â 10 À3 A m 2 , and its magnetization is 1.03 Â 104 A m À1 .The external magnetic field B 1 generated by m 1 upon m 2 can be calculated using Equation (1): where μ 0 is the vacuum permeability (H m À1 ) and r is the distance vector between the EPM and the SCMG (m).Then, the magnetic force exerted by the EPM with the magnetic moment m 1 on the SCMG with the magnetic moment m 2 is calculated using Equation (2): During the manipulation process, the EPM is always located below the working area in the vertical direction.The attraction force on the SCMG always has a downward component, which does not facilitate the wireless drive and control of the SCMG.To make the SCMG move effectively on the uneven surface of the stomach and roll and climb across obstacles to reach the targeted position, it is necessary to calculate the effective rotation driving distance between the EPM and the SCMG.The torque exerted by the EPM on the SCMG can be calculated by Equation (3): (3)

Effective Driving Method
The internal tissues of pigs are similar to those of humans, and when considering the penetration force that human stomach tissues can withstand, reference is made to the force of using a needle to penetrate the pig tissues, which is 1 N. [54] That is, the force exerted by the SCMG on the inner surface of the stomach model (the magnetic force exerted by the SCMG) should not exceed 1 N at its maximum.At a driving distance of 55À60 mm, the SCMG begins to deform, and it is fully open at a driving distance of 30 mm, which is much greater than the thickness of an average stomach wall of 4-6 mm. [55]The left-hand figure of Figure 4b shows the comparison of the calculated force, and measured force with the puncturing force from 1.2 to 4.5 cm to determine a safe operating distance.Theoretically, the magnetic force applied by the SCMG will exceed 1 N when the driving distance is less than 14 mm.Correspondingly, we conducted a force measurement experiment by fixing the EPM on a position and using a load plate to fix the SCMG on the spring tension meter above the EPM to measure the magnetic force on the SCMG at different distances from the central point of the EPM to the SCMG.The error of measured force became bigger when the distance decreased, and due to the deformation property and small magnetic moment of the SCMG, the magnetic force only exceeded 1 N when the distance was less than 12 mm.The distance at which the force exceeds 1 N is hereafter named as the critical distance OR safe distance, and the deformation distance is much bigger than this critical distance.
The central point between the EPM and the SCMG is called the driving distance.The right-hand figure of Figure 4b shows the theoretical comparison of force and torque trends when the SCMG is perpendicular to the direction of the driving magnet.The variation trend of torque is gentler than that of force with the distance increase, which is more suitable for controlling robot movement in this system.Moreover, the torque required to drive the SCMG is very small, it provides the basis for torquedriven SCMG rolling.However, when the drive distance exceeds 80 mm, the drive efficiency will be reduced due to the reduction of magnetic force.As a result, to ensure that the millirobot will not be deformed and opened during the locomotion, the driving distance is always greater than 60 mm, which is much larger than the safe distance of 12 mm.To ensure that there is enough magnetic torque to roll the SCMG, the driving distance during the locomotion cannot be greater than 80 mm.During the cargo-releasing process, when the drive distance is 30 mm, the SCMG will be fully opened.At this point, the pressure on the surface is 0.3 N, less than the force of 1 N that causes damage to human tissue.

System Composition
Based on the calculation and measurement results, Figure 4c shows the experimental setup to demonstrate the accuracy and success rate of the proposed system.The interior of the phantom is uneven, and the overall thickness is 18 mm.The distance between the center of the EPM and the SCMG at the initial point was set to be within 70À80 mm.A top-view camera was used to monitor and gain the coordinates of the SCMG.The experimental area was set up using a life-size human stomach phantom made of Ecoflex-0050.In an expected future scenario where the system will be used in medical treatment, the patient will swallow the magnetic millirobot and then lie face down on his or her stomach on the platform above the robotic arm, as shown in Figure 1c.After the top optical camera acquires the real-time image of the robot, the operator will use a mouse to click on the corresponding position at the user interface on the monitor, and the robotic arm and motor will drive the millirobot to the targeted point to perform the corresponding task.
The system comprises three main components that serve different functions: 1) two windows are displayed on the monitor in the user console-the operation command window and the video window-as shown in Figure 4c.The upper part is a video window for the user to observe the real-time situation of the millirobot, and the user can select the targeted point by pointingand-clicking the mouse.Then the robotic arm and the motor drive the robot to move to the targeted position inside the stomach phantom using real-time path planning and the rotation of the EPM in different directions.Besides, in the current state, the video window will also show the path of millirobot's movement, and the path will disappear after when the robot reaches the targeted point.The lower part is the command window, where the computer will check the present position and condition of the millirobot, and then print corresponding text in the command window in real time; 2) the workspace contains a 7-degrees-of-freedom robotic arm with a motor mounted at its distal end, a driving EPM with a side length of 38.1 mm, an optical camera, a human stomach phantom, and the wireless SCMG within it; and 3) the connecting computer serves as a hub that connects the user's operating platform, the robotic arm, the motor, and the optical camera.After receiving the commands from the user console, the computer converts and sends the corresponding movement and rotation commands to the robotic arm in real time.The optical camera in the working area provides real-time feedback to the system and users.In addition, the system built in this project uses computer vision to incorporate the function of detecting the position of the robot to determine whether it has reached the targeted point and can proceed to the next action or switch to the manual mode.In the whole system, the driving of the robotic arm and the motor is conducted behind-the-scenes and isolated from the user.Thus, from the user's perspective, his or her operations correspond directly to the movement of the SCMG.

System Working Process
The system's image processing and coordinate acquisition for the camera are based on the OpenCV library in Cþþ.This process and the modules are shown in Figure 5.The two main functions are locomotion and cargo release, and they can be controlled independently.The yellow, red, and blue parts represent the three main modules of the system.Ideally, all the work by the user (setting the target, deciding to release the cargo, and checking the command window) is done directly from the user console; the rest of the process is conducted automatically by the connecting computer, which controls the robotic arm and the motor.If the millirobot does not reach the designated position within 5 min, it is a deviation or an unexpected situation.When a deviation or an unexpected situation happens during the locomotion process, the system will display "Please switch to manual mode" in the user interface and the drive magnet will be moved directly below the robot.Then user can choose another path selection to drive the robot to the target point or use the manual mode (keyboard movement) to drive the robot.

Locomotion
After the system was started, the first action was aligning the X and Y coordinates of the SCMG and the EPM, and the adjustment of the Z coordinates of the EPM was performed.The computer grabbed the coordinates of the SCMG and transferred the data to align the EPM with the SCMG in the vertical direction and adjusted the Z coordinate to a preset value.This process is called "Catch the robot".Afterward, the user set the targeted point in the real-time monitor and the computer processed the coordinates of the targeted point, which were compared with the coordinates of the SCMG in the present situation to plan and select the path.When the EPM reached the waypoint, the detection and determination function sent the signal to the motor to rotate the EPM in a certain direction, then the SCMG was rotated to move to the targeted point.The detection function continuously checked and compared the position of the SCMG and the targeted point and sent updated rotation signals to the motor.Once the center of the SCMG reached the region within a 4 mm radius of the targeted point-meaning that half of the SCMG's body was already inside the targeted point regionthe motor stopped rotating immediately and the command window would print "Move finish!"

Cargo Releasing
Based on the soft and magnetic properties of the SCMG, approaching the EPM can transform the SCMG from a closed state to an open state.After performing the "Catch the robot" Figure 5. Flowchart of the developed computer-aided teleoperation system to enable intuitive control of a magnetic millirobot within a stomach phantom.The entire driving process of the system operates based on the relationship between the position of the SCMG, the targeted point, and the coordinates of the distal end of the robotic arm.and locomotion processes, the user could press "C" on the keyboard to start the "Cargo releasing" function.When this function was activated, the EPM approached the workspace and then rotated, bringing the distance between the EPM and SCMG to 35À50 mm (the thickest place of the stomach model is 15 mm).The SCMG then opens to release its cargo and then the EPM will be rotated clockwise once and then counterclockwise once to move the SCMG to leave the release position.

Manual Mode
If the SCMG did not reach the target point after 5 min, the command window would display "Please switch to manual mode", and the user could press either the up, down, left, and right keys or the "A", "D", "Q", and "E" keys on the keyboard to adjust the position of the SCMG.Q: tilt the SCMG 25°to the left along the X-axis.E: tilt the SCMG 25°to the right along the X-axis.W: tilt the SCMG parallel to the X-axis.A: roll the SCMG to the left.D: roll the SCMG to the right.S: stop rolling.C: drug release process.Ese: quit the platform.Up, down, left, and right keys: moving the underneath driving magnet to the corresponding directions.In the system, the installation direction of the driving magnet was rotated along the Y-axis, and the user was required to directly observe the direction of the wrinkles on the screen in manual mode.Figure 6 shows the SCMG's movement when it was maneuvered in the manual mode and the corresponding keys pressed on a keyboard.In this example, pressing the key "A" rotated the robot to the left.Pressing the key "D" rotated the robot to the right.When driving the SCMG to move downward along the X-axis, we needed to first move the SCMG down and left (press the "Q" and then "A" keys), and then move the SCMG to the left along the Y-axis (press the "E" then "D" keys).When driving the SCMG upward along the X-axis, we needed to first move the SCMG to the right (press the key "D"), then move the SCMG up and left (press the "E" and then "A" keys).When moving the robot up and down on the X-axis, the tilting magnet depends on the real-time situation of the robot's position.In this example, the robot was operated in the manual mode to move up and down in the same area.Moving downward required tilting the magnet first while moving upward did not.The movement of the SCMG in the manual mode is shown in the Movie S1, Supporting Information.

Results
The whole user interface is divided into two windows, with the upper window showing the real-time camera feed.The blue circle is the initial point of SCMG.The user can click the target position, and a blue circle will appear at the point of the click.The lower window is the command window, which shows the results of the movements.
In this experiment, the starting point of the SCMG was chosen near the cardia of the stomach phantom to simulate the likely position of the millirobot after being swallowed by the patient.To measure the success rate of individual moves to different area points, the entire stomach model was divided internally into 38 block areas.A successful example of this experiment is shown in Figure 7a.During the whole process, the camera would always reveal the real-time coordinates of the center point of the SCMG.This successful process is shown in the Movie S2, Supporting Information.
Before the user set the targeted point, the "Catch the robot" process was performed first.Then the user clicked to set the targeted point, and the command window displayed "Move start" as shown in Figure 7a at 0 s.After setting the targeted point, the computer compared the coordinates of the start and targeted points to select horizontal or diagonal movements.This example was a diagonal movement to the lower left.First, the EPM tilted 20°horizontally to the left, and the condition of the SCMG is, as shown in Figure 7a, at 4 s.Next, the EPM moved to the targeted point, and the detection function compared the coordinates of the SCMG and the targeted point.The motor was then turned on to rotate the EPM, and the SCMG was rotated to move.The SCMG did not reach the targeted point within 4 mm and moved slightly past the target point, as shown in Figure 7a, at 48 s.For the precise movement, the EPM continued rotating to move the SCMG, as shown in Figure 7a, at 55 s.Once the SCMG moved two blocks past the target point, the detection function would rotate the EPM in the opposite direction to move the SCMG back to the targeted point, as shown in Figure 7a, at 60 s.Finally, in Figure 7a at 64 s, the SCMG reached the target point within the 4 mm error window, the circle turned to green, the command window displayed "Move finish!" and for better observation of drug delivery, the locomotion trajectory was turned off.After the user pressed the "C" key to start the cargo-releasing function, the EPM began to move back to the original position in the horizontal direction, the command window displayed "Cargo release start" and the condition of the SCMG was shown in Figure 7a at 77 s.After the EPM moved to the right position, it approached the SCMG, which opened, as shown in Figure 7a, at 179 s.To ensure the bottom hands of the SCMG were totally open, the SCMG was rotated slightly, as shown in Figure 7a, at 180 s.Finally, the SCMG was driven to a nearby point, the cargo (a cuboidal red PLA) was released inside the target circle, and the command window displayed "Cargo release finish" as shown in Figure 7a, at 186 s.During the whole process, the user only needed to set the targeted point and activate the cargo-releasing function-all the rotations of the motor were decided automatically by the connecting computer.
The internal surface of the stomach is extremely sophisticated, with an inner wall full of folds.The left-hand side of Figure 7b shows the success rate of the experiments.The entire area of the stomach model was divided into 38 squares with 8 mm sides, and each area was set as a targeted point for 10 repeated experiments.The success rate was lower for areas that tend to be red and higher for areas that tend to be blue.The blue areas were the areas with an acceptable success rate.The folds inside the stomach phantom are marked out using black lines in the figure.The denser lines indicate larger folds, which contributed to a lower success rate of the SCMG reaching its target.This phenomenon was more pronounced at the edges of the stomach.Most of the working area could be reached by this system in the experiments.In general, the fewer folds there were in the area, the more likely the targeted area was reached successfully, and the error distance did not exceed 4 mm.
The right-hand side of Figure 7b indicates the number of regions with different arrival success rates as a percentage of the total number of experimental regions.The highest success rate value of these regions was 100% and the lowest was 0%, which occurred in the two regions located at the edge of the stomach wall that has large folds.Except for these regions, all other regions had at least a 60% success rate.This result demonstrates the effectiveness of the developed system.Most areas could be reached while using this system to drive the millirobot inside the stomach phantom, with over half of the areas considered easy to reach (a greater than 80% success rate).Figure 7c reports the time it took to reach each region of the stomach phantom.The red blocks indicate the regions that could not be reached.All the accessible regions could be reached in 150 s or less, and the arrival time to most areas did not exceed 120 s.

Conclusion
This study develops the framework of a computer-aided teleoperated platform that provides a direct linkage between the user's inputs and an untethered millirobot's actions within a human stomach phantom.With the help of this system, a nonexpert user can intuitively control the locomotion and cargo delivery actions of a millimeter-scale SCMG by simply mouse-clicking on a live camera feed displayed on a monitor.The system employs computer vision techniques to extract the real-time position of the SCMG, compares it with the targeted position, and plans a path to reach the destination while rolling the robot to overcome obstacles on the stomach wall.This system sends automatically calculated commands to the robotic arm to control the position, pose, and movement of the EPM mounted at its tip, generating the required magnetic fields to induce the desired behavior of the SCMG.Experimental characterization of this prototype system shows that the movement error of the SCMG is less than 4 mm and the time taken is less than 150 s.Most regions inside the stomach phantom have a 60% or higher rate of being successfully reached by the SCMG under the automatic maneuvering of the proposed system.In addition, the SCMG proposed in this work is simple to fabricate and agile in locomotion under magnetic control.It can carry a large cargo load using its internal chamber.
The proposed system has been successfully demonstrated to dramatically lower the barrier to magnetically maneuver miniature robots inside a workspace mimicking the human stomach, which has been acknowledged as a counter-intuitive and arduous task, especially for nonexpert users.Making the magnetic control of a miniature robot easier and more automatic not only reduces the training costs needed for healthcare practitioners to adopt these miniature robotic systems into their workflow but also, more importantly, it allows the users to focus on medical tasks without getting distracted by the engineering system during operations, benefiting the safety and efficiency of such robotic healthcare systems in medical interventions.In this project, to ensure the force on the surface of tissue will not be stronger than 1 N to cause the damage, the force of the robot under different magnetic field intensities was characterized and measured before experiments.As the result, the distance between the millirobot and the driving magnet is always larger than 30 mm during the whole process of the system working, it is much larger than the distance which will cause damage to human tissue.This project aims to simulate a real surgical situation in an experimental environment, including the use of a 1:1 stomach model of an adult male made of silicone close to the hardness of human tissues, and the distance between the magnet and the millirobot during actuation is greater than the distance from the stomach to the epidermis.In practical application to surgery, due to individual differences in the internal environment of the stomach, it may be more difficult to drive the robot across obstacles.However, the current experimental results show that the robot can reach most of the areas outside the vertical surface of the stomach, and if the target area is located at the edge, it can also be solved by turning the patient's body.
At its current stage, the developed teleoperation system works with a custom-built SCMG for cargo delivery within a stomach phantom.Nevertheless, with proper modifications in the program, the framework of this platform could be adapted to control other magnetic miniature robots in different workspaces relevant to medical interventions.Future research can focus on extending the capability of the developed teleoperation system toward the goal of making it a general-purpose control setup for a variety of magnetic miniature robots in clinic workspaces, which will promote the real-world adoption of novel miniature robotic solutions to healthcare challenges by minimizing the required personnel training and ensuring the safety of patients during their operation.

Figure 1 .
Figure 1.Schematics comparing the existing control methods of magnetic miniature robots with the proposed teleoperation platform.a) Schematic illustration of a 3D electromagnetic coil control platform and the main shortcomings.b) Schematic illustrations of a handheld permanent magnet setup to manually control a robot and the main shortcomings of this system.c) Schematic illustrations of the proposed teleoperation control platform.It works with an open and large workspace with low power consumption and high precision.More importantly, a direct linkage is established between the user's inputs and the millirobot's motions, enabling intuitive control of the millirobot by nonexpert users.

Figure 2 .
Figure2.The design and the behaviors of the SCMG under the magnetic field.a) Schematics (the left column) of the proposed millirobot are juxtaposed with corresponding photographs (the right column).In a close state, the small red arrows are the magnetic moment of the moveable hands, and the big blue arrow is the magnetic moment of the whole SCMG.b) The behaviors of the SCMG under the different magnetic fields.The SCMG began to deform when the magnetic field intensity was 13 mT, and the SCMG opened completely at 45 mT.

Figure 3 .
Figure 3.The fabrication process of the SCMG and preliminary experiment results.a) Schematics explaining the fabrication process of the SCMG, and the SCMG was magnetized in the open state.b) Photographs of the SCMG conducting the preliminary locomotion and cargo delivery experiment within a stomach phantom, controlled by a handheld permanent magnet.

Figure 4 .
Figure 4. Working principles, the determination of the control distance of the SCMG, and components of the proposed system.a) Schematics explaining the working principles of the maneuvering of the millirobot to move and release cargoes.b) Characterization results to determine the allowable control distance between the EPM and the millirobot.Error bars represent the standard deviation of five measurements.c) Photographs of the experimental setup.

Figure 6 .
Figure 6.Demonstrations of exemplar manual maneuvering of the millirobot using a keyboard.Four composite top-view frames observed by the user during direct maneuvering of the millirobot using a keyboard are shown, together with a schematic of the relevant keyboard keys.The millirobot's motions are mapped to different keystrokes.

Figure 7 .
Figure 7. Characterization results of experimentally moving the millirobot to different regions within the stomach phantom.a) A chronological sequence of top-view frames of the millirobot being maneuvered to move toward targets, and release its cargo.The targeted position was specified by a user via clicking the mouse on the real-time video feed.b) Quantitative characterization results of the success rate of the experiment demonstrated in (a) when the targeted position was specified in different locations within the stomach phantom.c) Qualitative characterization results of the time taken to reach different locations within the stomach phantom.The black curves in (b) and (c) depict the wrinkles on the inner surface of the stomach phantom.