Image‐based real‐time feedback control of magnetic digital microfluidics by artificial intelligence‐empowered rapid object detector for automated in vitro diagnostics

Abstract In vitro diagnostics (IVD) plays a critical role in healthcare and public health management. Magnetic digital microfluidics (MDM) perform IVD assays by manipulating droplets on an open substrate with magnetic particles. Automated IVD based on MDM could reduce the risk of accidental exposure to contagious pathogens among healthcare workers. However, it remains challenging to create a fully automated IVD platform based on the MDM technology because of a lack of effective feedback control system to ensure the successful execution of various droplet operations required for IVD. In this work, an artificial intelligence (AI)‐empowered MDM platform with image‐based real‐time feedback control is presented. The AI is trained to recognize droplets and magnetic particles, measure their size, and determine their location and relationship in real time; it shows the ability to rectify failed droplet operations based on the feedback information, a function that is unattainable by conventional MDM platforms, thereby ensuring that the entire IVD process is not interrupted due to the failure of liquid handling. We demonstrate fundamental droplet operations, which include droplet transport, particle extraction, droplet merging and droplet mixing, on the MDM platform and show how the AI rectify failed droplet operations by acting upon the feedback information. Protein quantification and antibiotic resistance detection are performed on this AI‐empowered MDM platform, and the results obtained agree well with the benchmarks. We envision that this AI‐based feedback approach will be widely adopted not only by MDM but also by other types of digital microfluidic platforms to offer precise and error‐free droplet operations for a wide range of automated IVD applications.


| INTRODUCTION
In vitro diagnostics (IVD) refers to medical tests that are done on biological samples extracted from bodily fluids or solid tissues, in order to detect the presence of pathogens, identify medical disorders, and monitor a person's health status or response to treatment. 1 IVD plays an essential role in patient care and public health; billions of molecular IVD tests (PCR tests) and immunodiagnostic IVD tests (antibody tests) have been performed to contain global COVID-19 pandemic. [2][3][4][5][6] One of the riskiest factors involving IVD is sample handling, which may cause accidental exposure to contagious pathogens among healthcare workers. Automated IVD can reduce such risks by minimizing manual interventions required to handle hazardous biological samples.
Digital microfluidics provides an effective way of IVD automation. [7][8][9][10] Digital microfluidics is a type of liquid handling platform that manipulates microliter-sized droplets on an open substrate.
Digital microfluidic platforms employ droplets as reaction chambers for IVD, and handle the liquids by actuating these droplets that contain biological samples and assay reagents. [9][10][11][12] The droplet actuation mechanisms include magnetic force, electrowetting, and surface acoustic wave. 9,13,14 Magnetic droplet actuation, hence the name magnetic digital microfluidics (MDM), attracts particular attention because it is relatively cost-effective, straightforward to implement, and well-suited to perform biological assays due to the bio-affinity of magnetic particles used also for actuation. 9,10,[15][16][17][18][19] The pros and cons of electrowetting and magnetic droplet actuation are discussed in detail in Reference 9. However, it remains challenging to create a fully automated IVD platform based on the MDM technology because of a lack of effective feedback control system to ensure the successful execution of various droplet operations required for IVD. On MDM platforms, droplets are transported, merged and extracted via the added magnetic particles, and these droplet operations are a result of highly complex interactions that are influenced by the volume of droplets, quantity of particles, strength of magnetic field, moving speed, and surface tension of the substrate and liquids. 15,20 Currently, automated droplet manipulation on MDM platforms mainly relies on open-loop control algorithms, assuming that the desired droplet operations could be successfully accomplished with a given set of predetermined parameters. 21 However, these droplet operations fail at times due to surface imperfection, operating under borderline conditions, or inconsistent performance of the control equipment, but no mechanism is in place to rectify the failed operations. Therefore, to create a fully automated IVD platform based on MDM, a closed-loop feedback system must be in place to monitor droplet operations in real time and guide the control system to rectify the problems before moving on to the next step. So far, only a limited number of works on electrowetting-based digital microfluidic platforms demonstrated the capability of feedback control by sensing the change in capacitive or resistive electrical signals when the droplets moved onto an electrode. [22][23][24][25][26][27] However, these sensing mechanisms require additional circuits, which increases the complexity and cost of the already complicated electrowetting-based control system. Further, these sensing methods are not applicable to MDM because the substrate of MDM is free of electrodes due to the different actuation mechanism. On the other hand, imaging offers a way of contactless signal acquisition without the need to modify existing MDM platforms. The key challenge to implementing image-based feedback for MDM is how to rapidly identify droplets and magnetic particles and determine the operation status based on this information. Conventional droplet detectors in electrowetting-based digital microfluidic platform usually rely on Hough transform and edge detection algorithms. [28][29][30] However, these algorithms are time-consuming and ineffective in recognizing droplets on MDM platforms because the transparent water droplets do not show a strong contrast against the background and also do not always appear in regular shapes during movement. In contrast, artificial neural network (ANN) does not solely rely on edge features for object detection and has shown excellent performance in recognizing transparent droplets. 28,29,31 Several closed-channel emulsion droplet platforms used image feedback to control droplet generation and sorting by employing an ANN-based object detector to identify droplets. [32][33][34] Indeed, the ANN model for MDM needs to perform more complex tasks beyond just identifying droplets.
In this work, an artificial intelligence (AI)-empowered MDM platform with image-based real-time feedback control is presented. An ANN object detector based on NanoDet, 35 an ultrafast and lightweight target detection model, is trained to recognize droplets and magnetic particles, measure their size, and determine their location and relationship. [36][37][38] This automated MDM platform shows the ability to rectify failed droplet operations based on the feedback information, ensuring that the entire IVD process is not interrupted due to the failure of liquid handling. Two IVD assays for protein quantification and antibiotic resistance detection are demonstrated on this platform, and both assays are fully automated to accomplish desired droplet operations without the need for human intervention in case of failure. To our best knowledge, this is the first smart MDM platform with AI-empowered real-time feedback, and we believe that this technology can be readily adopted by other types of digital microfluidic platforms to greatly broaden their applicability for IVD. due to the following two reasons: first, the particles appeared as a tiny dot in the images when they were present in a small quantity, which may be overshadowed by the reflection and refraction patterns of the droplets; second, the particles may spread and occupy the entire bottom surface of the droplet when they were present in a large quantity, in which case the droplets may also be identified as particles. This reflects the strong generalizability of our platform. Once the droplet reaches the first destination, the AI confirms the successful completion of the droplet transport operation before continuing with the next operation ( Figure 3ciii). The entire operation ends when the AI confirms that the droplet has reached the final destination ( Figure 3civ). During droplet transport, the magnet is at the leading edge of the droplet. An "overshoot" function is programmed to allow the particles to move beyond the destination along the ydirection by 20 pixels and then move back by the same distance in order to position the center of the droplet and particles close to the coordinate of the destination, which could facilitate subsequent operations. This demonstration proves that the ANN object detector is able to identify droplets and particles and monitor their locations in real time for feedback control. Based on this feedback information, the AI judges whether the droplet transport operation is successful and commands the electromagnet to repeat the operation in the case of magnet disengagement to rectify the failed operation.

| Particle extraction
Particle extraction is another fundamental operation that separates the solid phase from the liquid phase when performing heterogeneous IVD assays on MDM platforms. The control algorithm for the particle extraction operation is shown in the flowchart in Figure 4a. Before the operation starts, the AI identifies the droplet and particles in live streams, and the CP guides the electromagnet to where the particles are located and switch on the electromagnet. After the user specifies towards which direction the particles are extracted in the GUI, say "EAST," the CP guides the electromagnet towards right at a high speed to extract particles from the droplet (Figure 4b and Video S2).
The particle extraction operation may fail if the quantity of the particles is too large and the moving speed is not high enough. 15,20 During this operation, the AI monitors the location of the droplet and particles (Figure 4bi). If the two still overlap after the movement, which suggests that the particle extraction operation is unsuccessful, the AI sends a command to bring the electromagnet back to where the particles are and restart the particle extraction process at a higher moving speed (Figure 4bii). Surface energy traps (SETs) may be included to facilitate particle extraction by anchoring the droplet on the substrate. 15,39 In the end, the AI confirms that the particle extraction operation is successful if the particles and droplet are at two distinctive locations (Figure 4biii). After a successful extraction, a small number of residual particles may be left in the original droplet. However, the amount of these left-over particles is usually too small to be identified as "Particles" by the AI, and hence they do not affect subsequent operations.
In this operation, AI-based closed-loop control again demonstrates the ability to monitor the operation, examine whether it is successfully executed, and rectify the problem if the operation fails, which is unattainable on conventional MDM platforms.

| Droplet merging
Droplet merging is an operation that combines different reactants to initiate bioreactions required for IVD assays on MDM platforms. The control logic for the droplet merging operation is shown in the flowchart in Figure 5a. After the AI identifies the droplet and particles in the live stream, the user is asked to specify which two droplets to merge (Figure 5b  of the bounding box increases by a certain threshold value, typically at least three pixels, the droplet merging operation is deemed a success, and the AI continues with the next operation; otherwise, the CP guides the electromagnet to where the particles are and restart the merging operation.

| Passive mixing
Once two droplets are merged, the content of the merged droplet is homogenized by moving the particles and droplet on the surface, which promotes passive mixing on the MDM platform. 15,40 The mixing operation is essentially a combination of a series of droplet transport operations (Figure 6b and Video S4). The user may define customized moving paths for the passive mixing or use the predefined cross-shaped path with a single click of button. Another parameter that requires user input is the number of loops for mixing. The center of the cross-shaped path is set to the current location of the particles.
The CP guides the electromagnet to move the particles and droplet along each arm of the path and then return to the center. The  moving along the cross-shaped path for five loops and incubated for 10 min (Figure 6biii). During incubation, the electromagnet moved back to the HOME position and was switched off (Figure 6biv). After incubation, the electromagnet moved to the location of the particles and was switched on (Figure 6bv). In the last step, the particles were extracted from the reaction droplet so that the reaction solution could be retrieved from analysis. To perform particle extraction, the reaction droplet was transported to the SET (Figure 6bvi). As the electromagnet drove the particles towards right, the droplet was anchored to the SET, which facilitated particle extraction (Figure 6bvi). After AI confirmed that the droplet and particles were at two distinctive locations, which indicated successful particle extraction, the electromagnet was switched off and returned to the HOME position. To conduct the reaction in triplicates, the CP guided the electromagnet to Areas 2 and 3 in sequence to complete the droplet merging and mixing while the first reaction droplet was being incubated. After incubation, the particle extraction operation was performed sequentially from Areas 1 to 3 to complete the triplicates (Video S4). The protein concentration was determined by measuring the absorbance of the reaction droplet at 480 nm. The same batch of sample was also analyzed in a microwell plate as a benchmark. The standard curve of the absorbance versus the protein concentration is shown in Figure 6c.
The results obtained on the AI-empowered MDM platform agree well with the benchmark ( Figure S3). The limit of detection of the MDMbased assay is 163.2 μg/ml, which is comparable to that of the benchmark of 181.0 μg/ml. Only a single reaction is shown in the figure for clarity. The operation for triplicates is shown in Video S4.

| Detection of carbapenemase-producing Enterobacteriaceae (CPE)
Carbapenem is a group of antibiotics that is considered the last line of defense against Gram negative bacterial infections. 43 One of the main mechanisms of bacterial resistance to carbapenem is through the production of carbapenemase, an enzyme that hydrolyzes carbapenem. The Carba NP test is a rapid IVD assay that identifies CPE by sensing pH changed induced by the hydrolysis of carbapenem by carbapenemase. 44,45 In the Carba NP test, each sample is analyzed with a testing reaction that contains imipenem (a type of carbapenem) and a control reaction without imipenem. If the bacterial strain is CPE, it hydrolyzes imipenem in the testing reaction, which reduces the pH and changes the color of the pH indicator from red to yellow/orange (Figure 7a). In contrast, non-CPE strains are unable to hydrolyze imipenem, and the color of the testing reaction would remain red. In both cases, the color of the control reaction must remain red for the assay to be valid.  (Figure 7biii). The AI monitored the operation in real time to ensure that the merging operation was completed before moving on to the next step. After merging, the electromagnet moved to the location of the merged droplets and mixed the merged droplets one at a time (Figure 7biv,v). Both the test reaction and control reaction droplets were incubated for 1 h after mixing, during which the imipenem was hydrolyzed if the strain was CPE. To facilitate the observation, the particles were removed from the reaction droplets in the end. To do so, the electromagnet transported the first reaction droplet to a SET where it was anchored to the substrate for easy particle extraction (Figure 7bvi,vii). The same was done for the second reaction droplet (Figure 7bviii,c). Once the particles were extracted, the color of the reaction droplets was observed. Strain No. 2 is a CPE strain that belongs to the KPC type; therefore, the color of the test reaction droplet (on the right) turned into yellow (Figure 7c). The color of the control reaction droplet remained red, indicating that the assay was valid. The RGB values of the droplets in the images were mapped into the color space for objective determination of the droplet color ( Figure 7d). A total of eight strains, including six CPE and two non-CPE strains, were tested on both the AI-empowered MDM platform and in microwell plate that served as a benchmark ( Figure S4). The results obtained with MDM agree with the benchmark, and all eight strains are correctly identified (Table 1).

| ANN model and feedback control system
The ANN object detector used for droplet and particles identification was trained based on Nanodet, an ultrafast and lightweight model optimized for implementation on mobile platforms. 35 The data set containing 769 images were manually labeled with a total of 3647 labels. All images were converted to the COCO format. The data set was divided into 70%/30% for training/testing so that the training data set contained 538 images and 2604 labels, and the testing data   respectively. The operation conditions, such as P/D ratio and moving speed, were selected to favor droplet movement. As a result, few failures were encountered when performing droplet movement, and most operations were completed with a single trial. The amount of particles was increased to 1 μl for particle extraction to increase the difficulty of particle extraction so that we could demonstrate how the AI rectify a failed operation. A higher moving speed or the assistance of SETs was used to facilitate the particle extraction in case of failure.
In droplet transport, the droplet was considered having reached the destination when the droplet was within a distance of 10 pixels from the destination; in particle extraction, the operation was considered successful when the bounding box of the particles fell outside the bounding box of the droplet; in droplet merging, two droplets were considered merged when the size of the droplet sized increased by at least 3 pixels.

| Carba NP test
Detailed protocol for bacteria subculture and the preparation of Carba NP reagents were reported in our previous work. 16 Nonetheless, this study is a preliminary proof-of-concept, and we notice several limitations with the current implementation. The main issue we have faced is caused by the misidentification of droplets. This is particularly problematic for small transparent droplets that frequently suffer from misdetection, in which case the AI is unable to make the correct decision. A potential solution to this problem is to improve the illumination conditions so that the droplet features are more prominent in the images. Another limitation is the relatively small data set used for training. Increasing the size of the training data set by including more droplet-particle combinations will improve the mAP of the ANN-based object detector for more accurate feedback control.

DATA AVAILABILITY STATEMENT
Data available on request from the authors.