Acoustic Manipulation of Particles in Microfluidic Chips with an Adaptive Controller that Models Acoustic Fields

Acoustic manipulation is a technique that uses sound waves to move particles, droplets, or cells. Closed‐loop control methods based on complex, time‐varying acoustic fields have been demonstrated, but usually require accurate models of the acoustic fields or many training experiments for successful manipulation. Herein, a new adaptive control method is proposed for the acoustic manipulation of single and multiple particles inside microfluidic chips. The method is based on online machine learning of the acoustic fields. Starting with no knowledge of the fields, the controller can manipulate particles even on the first attempt, and its performance improves in subsequent attempts, yet can still readapt if the models are invalidated by a sudden change in system parameters. The controller can generalize: it can use information learned from one task to improve its performance in other tasks. Despite the machine‐learning nature of the controller, the internal models of the controller have a physical interpretation and correspond to the experimentally observed acoustic fields. The online adaptiveness of the controller should make it easier to use in practical applications, such as particle and cell sorting, microassembly, labs‐on‐chips, and diagnostic devices, as the method does not require extensive training or prior models.

The contactless nature of the acoustic force simplifies the fabrication of the chip.In the case of bulk acoustic waves, the chip itself can be a passive glass device, with the ultrasonic waves produced by a piezoelectric transducer attached to the chip. [11]In the case of surface acoustic waves, the driving electrodes can be located away from the channels.The chip channels can be also closed, unlike in manipulation involving tools, such as pipettes [22,23] or needles, [24,25] that need to be in direct or close contact with the manipulated object.Acoustic manipulation is wellsuited for applications involving cells and living organisms, [26,27] because the acoustic force exerted on a particle scales with its volume, preventing applying excessively large forces on tiny particles. [28]][31][32][33] The range of manipulations that can be achieved with such a field is fairly limited; generally, the particles just migrate toward the nodes of the standing wave acoustic field. [1,9,34]More recently, methods based on more complex fields [20,[35][36][37] have been put forward, including methods based on time-varying fields [38] and fields from multiple transducers. [3,20,37]The challenge with such methods is that they require accurate modeling of the acoustic fields and calibration of the acoustic parameters, including knowledge of the acoustic properties of the materials involved.The experimentally observed field shapes often deviate significantly from theoretical models, [39] and the properties of the materials can change, e.g., the speed of sound and viscosity both depend on the temperature. [40]herefore, recent works and our group have explored methods based on machine learning to overcome the difficulty of using first principles to predict particle motion in acoustic fields.In Zhou et al., [41] our coauthors reported a data-driven control method for the 2D manipulation of particles on a flexural vibrating plate (Chladni plate) and demonstrated the manipulation of single and multiple particles.The method was based on modeling the particle motion based on a large dataset, collected by using machine vision to monitor the particle motion in response to different frequencies (1-30 kHz).The dataset consisted altogether of %390 000 data points.During manipulation, the algorithm only varied frequency, because that determines the overall shape of the Chladni pattern, and therefore the direction of motion of the particles.The amplitude and duration of the driving signal were kept constant for each frequency, as these mostly determine the magnitude of particle displacement per step, as was demonstrated in the article.
Despite a large amount of data, the method suffered from having no adaptivity; any change in system parameters (e.g., a change in how the plate was mounted to the transducer or additional damping on the plate) could cause discrepancies between the system and the data-driven model, requiring an entirely new dataset to be collected for the model.A model discrepancy often caused the controller getting stuck, because it kept on choosing frequencies solely based on the model, but the particles did not move as the model predicted.Therefore, control methods with online learning and adaptivity would still be needed.
Later, Latifi et al. [42] explored a control method based on reinforcement learning, where neural networks were used to estimate the Q-function, the expected reward for each action taken.In their case, the Q-function is an estimate of how many pixels or micrometers a particle travels toward its targeted destination if a particular frequency is actuated.The method was based on using multiple training episodes, with each training episode consisting of manually placing the particle at its starting point and then using the controller to try to complete the manipulation task.Here also, there was no adaptivity within each training episode; the learning was done offline between the episodes.The controller was also trained using a simulator, for which pre-existing data-driven models of the particle motion were anyway needed.With a single particle, the controller was not able to complete the manipulation task typically for the first 10 episodes or so.For the successful manipulation of three particles, 50 000 training episodes were done.Changing the target points or the manipulation task required retraining of the controller, and the number of training episodes is very large if all the experiments would have been done on a real system without the existence of a simulator.
Raymond et al. [43] showed that deep-learning neural networks can learn and predict acoustic field shapes from surface traveling waves within chambers of various shapes.The network was trained using a large data set, generated by simulating acoustic field shapes in various chamber geometries.The predictions of the network were validated both against experimental and simulated results.
In our previous works, we reported reinforcement learning methods for manipulating particles [1] and droplets [9] inside closed-channel microfluidic chips with bulk acoustic waves.The methods were derived from the family of multiarmed bandit algorithms; specifically, the UCB-1 and ε-greedy algorithms. [44]he algorithms start with no prior knowledge of the device parameters (e.g., device dimensions, or models of the acoustic fields), but are able to perform the assigned manipulation tasks even on the first attempt, including: 1) particle and droplet transport through a defined route; 2) manipulation of three particles simultaneously; 3) merging multiple droplets; and 4) transporting a particle to a defined outlet of the chip (particle sorting).The algorithms were learning online how to manipulate the particles/ droplets and were able to adapt in sudden system changes during the experiment.However, the problem with that method was the lack of long-term memory; the algorithms did not have any internal model of the acoustic fields.The algorithms only learn how the particles move locally and forget this by the time the particle has traveled sufficiently far away from the current position.Therefore, in every manipulation experiment, the controller tries frequencies by trial and error, to find which frequency moves the particles toward their current waypoint.Consequently, the controller suffers from long manipulation times, and poor path-following accuracy.
In this article, we develop the ideas of our previous multiarmed bandit-based algorithms further, by incorporating a long-term memory to them.The long-term memory is based on a regression model of the acoustic field shapes, but instead of using black box models like neural networks, we develop our models based on physical insights of the acoustic waves inside the chamber.Importantly, the controller is still fully learning and adapting online.The models are not based on data or simulations obtained prior to the manipulation, and there are no training episodes.Instead, model parameters are initialized as zeros, and after every control step, the controller observes how particle(s) move in response to a particular frequency and updates its model parameters for that frequency, so that the models start converging toward the field shapes.We call the proposed new controller the acoustic-model-based adaptive controller, or AMA controller for short.
For each frequency, we use a regression model to estimate the Gor'kov [45] potential for that frequency and assume that the particle motion is driven by the gradient of the Gor'kov potential.The specific regression method is a sparse regression method, with sinusoidal basis functions.This can be viewed as the regression of the 2D Fourier series coefficients of the Gor'kov potential.The sparsity requirement is justified by the insight that even though multiple modes of the chamber could be excited simultaneously, most of the time only a few modes near the driving frequency or its harmonics should be dominant.
The controller proposed in this article has similarities to model predictive controllers and reinforcement learning algorithms: 1) the controller maintains a model to estimate how the particles move if a particular action (frequency) is chosen, given the current positions of the particles; 2) has a rule to update the model after every control cycle, based on the observed motions of the particles; and 3) balances exploration and exploitation, exploiting the frequencies known to produce best results while occasionally exploring alternatives by testing new frequencies.
Using the experimental setup in Figure 1a, we show that the proposed method can successfully manipulate single particles without any prior knowledge of the acoustic fields, and the manipulation is faster in subsequent repetitions.The controller completes the first attempt of a single particle manipulation task in %50 min, while after five repetitions, the task took only %10 min.The same task takes %140 min to complete with the εgreedy controller from our previous work.In addition, just by performing a single training task with a single particle, the proposed control method can generalize and shows improved performance in a manipulation task it has never completed before.Comparing the acoustic field models of the controller, obtained from single particle manipulation experiments, to the experimental patterns formed by many particles accumulating in static acoustic fields (Chladni patterns), we see a clear correspondence between the models of the controller and the Chladni patterns.Despite the long-term memory, the controller remains still adaptive and can relearn all the acoustic field models even when the models are invalidated, e.g., when we change the physical chip to another chip design or when we artificially shift all the manipulation frequencies by 10%, both without explicitly telling the controller about the change in the system.
In short, the advantages of the new control method are: 1) manipulations are generally successful even on the first try; 2) the time to complete the manipulation and how accurately the particles follow the designated paths improve as the controller accrues information about the acoustic field shapes inside the chip; 3) the controller can generalize information learned from one task toward completing other tasks; 4) the field models can be validated by comparing them to acoustic field shapes obtained using other means; 5) it still has the online learning nature of our previous controller and can correct its models even after the properties of the system have dramatically changed.
This paves the way toward the use of acoustic manipulation in practical microfluidic applications, such as particle and cell sorting, microassembly, labs-on-chips, and diagnostic devices, where it is likely unfeasible to expect the user to conduct the large number of training experiments every time the chip or the chip parameter has changed and where it is difficult to derive the acoustic field shapes from first principles.The algorithm is also suitably general that it should be in principle adaptable to other acoustic manipulation setups, e.g., manipulation of particles on a flexural vibrating plate, [41] which produces similar Chladni patterns as bulk acoustic waves even though the boundary conditions are different.The algorithm should be adaptable even to multitransducer surface acoustic wave devices, [14] where instead of controlling only the frequency of the signal, the algorithm would explore different combinations of phases and relative amplitudes of the signals used to drive the transducers.When adapting the algorithm to other manipulation platforms, alternative regression models that correspond better to the physics of the manipulation setup should be tested (e.g., models predicting uniform displacement across the field for each combination of parameters).

Control Algorithm
Details and pseudocode of the proposed acoustic-model-based adaptive (AMA) controller are given in the Note, Supporting Information; here we develop the main ideas behind the algorithm.

Control Cycle
The basic control cycle is shown in Figure 1b.At the beginning of each control cycle, the controller has the machine vision-detected positions of the particles and must decide which frequency to actuate next.The frequency is chosen from a discrete set A of 100 linearly spaced frequencies in the range of 65-700 kHz.The choice between exploration and exploitation is made randomly, similar to the ε-greedy algorithm [44] ; with the probability of ε, the frequency is chosen completely randomly, while otherwise, the controller chooses the frequency that takes the particles closest to their current targets, according to its internal acoustic model.
In exploitation, the controller greedily chooses the frequency which minimizes a cost function that embeds the manipulation task (greedy choice means here we do not predict the particle positions more than one control step into the future).The cost function J is typically of the form where t j and p j are, respectively, the current target and the current position of particle j, in 2D.When manipulating multiple particles, to avoid clustering and agglomeration, a penalty was added to the cost function when the distance between two particles Δp was less than r.The penalty term was proportional to ðjjΔpjj=rÞ À12 À 1, which increases sharply if two particles are closer than r i.e., the cost function was of the form P j jjt j À p j jj þ P i>j Cð½jjp i À p j jj=r À12 À 1Þ, where C and r are constants setting the strength and cut-off distance of the "repulsion" between the particles.When manipulating multiple particles simultaneously, we used r ¼ 1.5mm and C ¼ 1mm.
Using the models for each frequency, the chosen frequency a Ã is found with where d a ðpÞ are the predicted particle displacements for the frequency a, given their current positions.From now on, we will drop all subscripts of a for brevity, but it should be kept in mind that all the acoustic field models and corresponding predictions of particle motions are always specific to a particular frequency a.
After choosing a frequency, the controller actuates the frequency with a sinusoidal driving signal for a duration of t = 0.5 s and a voltage of U piezo ¼ 25 V.In our setup, the machine vision algorithm takes 0.1 s to detect the particles from the captured image, so the actuation time was chosen to be short but still significantly longer than the machine vision detection time, so that the actuator would be driven and particles moving for most of the time.The actuation voltage was chosen as a decent trade-off between accuracy and manipulation time (see Results).The new positions of the particles are then detected using machine vision and the internal acoustic model of the controller is updated with the new information of the particle displacements.The control cycle is repeated until the manipulation task is finished.

Modeling the Acoustic Fields
The primary acoustic force F on the particle can be found with where U is the Gor'kov potential [45] of the acoustic field.Gor'kov developed a closed-form solution for U, which depends on the time-averaged squared pressure and velocity fields, and material and particle size-dependent constants.Here, the closed-form solution is unimportant, because we have no prior knowledge of the acoustic field shapes, but we use data-driven regression to directly model U.
Our particles are small and in a viscous liquid (water), so we assume the particles reach the terminal velocity v during each manipulation cycle, and that the acoustic force is perfectly countered by the Stokes drag, so that We then assume that the actuation cycle is short, so that displacements d of the particles are much shorter than the wavelength.Therefore, the Gor'kov potential gradient can be considered constant within one actuation cycle and d % tv (5) Putting Equation ( 3)-( 5) together, we have where U $ is the scaled Gor'kov potential (U $ $ U, sign flipped for brevity), multiplied by the factors of time, and drag, which are independent of the frequency.Note also that magnitude of the Gor'kov potential and drag depend on the particle size, but this scaling is again the same for all frequencies.
We now assume that U $ in Equation ( 6) can be approximated by a finite 2D Fourier series where m and n are integer indices from the finite set M, c mn are the Fourier coefficients, p x,y are, respectively, the x and y components of the position of particle p, and L and W are the chamber length and width, L ¼ 7 mm and W ¼ 6 mm in our case.The Fourier coefficients were constrained so that the regressed Gor'kov potential remained always a real-valued function; in practice, this was done by writing the Fourier transform in its trigonometric form [46] with real-valued coefficients (see Note, Supporting Information), but here the equations are given in complex form for compactness.It is reasonable to assume that c mn % 0 for frequencies much higher than the highest frequency we ever use to actuate the chamber, so M can be chosen to roughly cover the frequencies we use during manipulation.The justification for using a Fourier series is that Equation (7) has a clear relationship to the acoustic modes of a rectangular chamber.The pressure fields P of standing wave modes inside a hard-walled rectangular chamber are given by where m and n are now the mode numbers.Note that the Gor'kov potential depends on the time-averaged square of the pressure field, [47] but the effect of squaring of the pressure is that the wavelengths in the Gor'kov potential fields are essentially half the wavelengths of the pressure fields.In practice, this detail is unimportant to us, because we anyway find the Gor'kov potential using a regression method.Also note that Equation (7) gives functions that are periodic with a period of ½2L, 2W.Theoretically, with infinite Fourier series, we could have considered Fourier series with a period of ½L, W.However, with finite Fourier series, this would artificially force the function to be continuous when wrapping around chamber walls, which might not be true in practice, because our chambers are not ideal rectangles and chamber boundaries are not infinitely hard.This is why we considered functions with a period of ½2L, 2W.

Adaptive Update of the Model
For each frequency, we collect a dataset of observed particle displacements D and starting positions of the particles P, with P k denoting the row of the data and P x k its x-component, and our goal is to find the coefficients c mn in Equation (7).Combining the Equation ( 6) and ( 7), we have for the x-direction where the superscripts x and y denote the x and y components of a vector and e is the modeling error.The equation for the y-direction is similar.These equations give a regression problem of the form where d now has all the x and y displacements stacked into a column vector, c has all the coefficients c mn stacked into a column vector, and X is a matrix of the explaining variables, calculated using Equation ( 9) or its y-direction counterpart.
There could be many ways to solve the regression problem in Equation (10), including the least square method [48] and Ridge regression, [49] among others.Here, we use further physical insights to guide us: even though the acoustic field might be a superposition of multiple different modes, it is still reasonable to assume that it is dominated only by few modes, near to the actuation frequency or its harmonics.Therefore, most of the coefficients c mn should be zeros.In other words, we want to find a solution to (10) that is sparse.We do this by finding c with where K is the number of datapoints.Such a problem can be solved efficiently with the LASSO algorithm. [50]The efficiency is important here as the problem needs to be solved once every control step.In the problem, λ is a parameter controlling the sparsity of the solution; with λ ¼ 0, the problem is just a classical least squares regression problem, while λ !∞ gives c mn ¼ 0 ∀ m, n.
In Equation (10), all observations regardless of their age are given equal importance.However, because the acoustic properties of the system can change, we want to emphasize more recent observations.This was done by weighting the observations by a factor of γ Às , where γ is the forgetting factor, γ ∈ ð0, 1, and s is the number of times the same frequency has been actuated since this observation.In the practical implementation, very old data points with small weights can be discarded from the data, as they do not significantly affect the results of the regression.This avoids the memory requirements of the algorithm growing infinitely.

Manipulation of a Single Particle
To test the proposed AMA controller in acoustofluidic manipulation, we used a glass microfluidic chip with a piezoelectric transducer attached to its backside.A schematic of our experimental setup is shown in Figure 1a.The design of the chip is similar as in our previous work [1] and the design of the chip is shown in Figure S1, Supporting Information.The chip was wet etched from a fused silica glass wafer.Inside the chip, the manipulation takes place in a rectangular chamber, with inlet and outlet channels for the fluid and the particles.Two different chip designs were used, one with a single outlet to demonstrate the basic manipulation, and one with three outlets, demonstrating applications in particle sorting, where the particle was guided to one of the outlets.The chip was illuminated by an LED light source and observed with a camera.The positions of the particles were tracked by a machine vision algorithm.
To compare the AMA controller to the ε-greedy controller from our previous work, [1,9] we performed a manipulation experiment where both controllers were tasked to guide a polystyrene particle (diameter: 70 μm, density: 1.05-1.06Â g cm À3 ) through a rectangular path, with the controller starting with no knowledge of the acoustic field shapes.Polystyrene microparticles have a positive acoustic contrast factor (Xue et al. calculated it as 0.58), [51] so they migrate toward the pressure nodes of the acoustic field.With the AMA controller, we repeated the rectangular path for five consecutive times.All experiments were repeated five times from the beginning.The results from these experiments are summarized in Figure 2a-d.Even when the AMA controller completed the task for the first time, it outperforms the εgreedy controller considerably, which is evident from the time it took to complete the manipulation (Figure 2a, %50 min for AMA, %108 min for ϵ-greedy) and can also be seen from the accuracy of the manipulation (how closely the controller follows the planned path, Figure 2b,d, see also Video S1 and Figure S2, Supporting Information).The ε-greedy has no long-term memory, and it is agnostic to the physics of the chamber, so repeating the task does not yield better results.Meanwhile, during consecutive cycles, the AMA controller learns to complete the task faster; the first manipulation cycle took 35-60 min, whereas the fifth manipulation cycle took only 6-20 min.
For each manipulation experiment, we calculate the manipulation accuracy as follows: 1) For each position of the particle, find the shortest distance of the position to the planned path of the particle.2) Calculate the median of these distances as a single number representing the typical accuracy of the manipulation.The manipulation accuracies for the data in Figure 2a are shown in Figure S2, Supporting Information.The manipulation accuracy improved during subsequent repetitions of a task.In the two examples showed in Figure 2b,c, on the first cycle the manipulation accuracy was 0.11 mm, whereas on the fifth cycle the manipulation accuracy was 0.04 mm. Figure S2, Supporting Information, shows that the manipulation accuracy for the AMA controller during the first cycle (0.11mm) was comparable to the manipulation accuracy of the ε-greedy controller (0.10 mm), but after five cycles, the manipulation accuracy of the AMA controller was significantly better (0.04 mm) than the manipulation accuracy of the ε-greedy controller.
To further compare the performance of the AMA controller to the ε-greedy controller, we redid the particle sorting experiments as in our previous article. [1]With a chip with three outlets, we sorted 30 particles, with 10 particles into each of these outlets.Examples of the sorting experiments are shown in Figure S3, Supporting Information.The average sorting time per particle was 7.7 min, whereas in our previous article, ε-greedy controller achieved sorting times in the range of 13-20 min.
In sum, all these experiments show that the AMA controller outperforms the ε-greedy controller, both in terms of manipulation time and accuracy, and can retain knowledge from past experiments.
To show that the method is not limited to manipulating specific objects, e.g., particles of specific material and diameter, we tested manipulating 50 and 100 μm diameter polystyrene particles in water, and %160 μm water droplets in hexadecane oil.The manipulation used the same pretrained models as in Figure 2e.The results are shown in Figure S4, Supporting Information.The manipulation times and accuracies slightly varied depending on the object being manipulated, but the manipulation was always successful.With the droplet in particular, the successful manipulation is notable, because the manipulation medium was hexadecane instead of water, so the pretrained models do not correspond well to the acoustic field shapes in hexadecane.
Note that in the current experimental setup, the hard limit in the size of the manipulated particles is the height of the chamber, 150 μm, which suggests that the droplet was already being squeezed by the chamber lid and bottom.On the other end, our imaging system and machine vision algorithm start losing the tracking of the particles when they are smaller than 25 μm.These practical engineering limitations suggest that closed-loop vision-based acoustofluidic manipulation setups need to be designed somewhat to target a specific size range of manipulated objects, but these kinds of limitations apply to any vision-based closed-loop acoustofluidic manipulation and are not particularly specific to the control algorithm proposed in this article.
Both the manipulation time and manipulation accuracy are affected by the actuation voltage (power).To show this, we tested different voltages (15, 25, and 40 V) and completed the same manipulation task as in Figure 2b.The results are shown in Figure S5, Supporting Information.Higher voltages resulted in faster manipulation (40 V: 40 min vs 15 V: 83 min), but poorer accuracy (40 V: 0.12 mm vs 15 V: 0.1 mm). 25 V was chosen as a decent tradeoff between these two, but for practical applications, voltage should be chosen based on the requirements of the application.
To show that the controller can generalize its knowledge to tasks that it has never done before, we first trained the controller Figure 2. a) Manipulation times for the proposed AMA controller, compared to the ε-greedy controller from our previous work. [1]The task is to manipulate a single particle through a rectangular path.Notice the logarithmic scale on the y-axis.During subsequent repetitions of the task, the AMA controller has built knowledge of acoustic field shapes and can better predict how the particle will move, resulting in shorter manipulation times.using a generic manipulation task and then tested it on a completely new manipulation task.The training task was designed so that the particle visited many different points inside the chamber, covering as much area as possible.To do this, we used a space-filling curve [52] as the training path of the particle (Figure S6, Supporting Information).After the training experiment was completed, we used the trained model to manipulate a particle along a path that the controller had never completed before.Figure 2e,f show the results.As it can be seen from the Figure 2e and Video S2, Supporting Information, the controller was able to generalize to the new task, with the particle following the path relatively well.Compare these results to the results shown in Figure 2f, where the manipulation task was completed without prior training experiments.With the trained model, the manipulation took %20 min to complete, whereas, without any pretrained model, the manipulation took %150 min to complete.Taking these results together, it shows that the controller has been able to generalize the knowledge it learned from the training experiment to a task that it has never done before.
Finally, to show that the knowledge accrued by the controller has a clear relation to the acoustic field shapes in the chamber, we compared the regression models U $ , learned during the training experiments with the space-filling curves, to experimentally obtained Chladni patterns.The Chladni patterns were obtained by placing many particles inside the chamber and then driving the actuator with a particular frequency for 2 s.Assuming the primary acoustic force is the dominant force, the particles move toward local minima of the Gor'kov potential, visualizing the acoustic fields inside the chamber.The results are shown in Figure 3, with more examples shown in Figure S7-S9, Supporting Information.The frequencies shown in Figure 3 are among the ones that have been used many times by the controller (Figure S7-S9, Supporting Information).Figure 3 shows a clear correspondence between the model U $ and the experimentally observed Chladni patterns.This is somewhat striking, considering that U $ was obtained only from single particle manipulation data, instead of a population of particles.These results show that the internal regression model U $ of our AMA controller has a physical interpretation: it approximates the Gor'kov potential for that frequency.
Overall, these manipulation experiments show that the proposed controller learns from the past experiments and develops physically meaningful knowledge about the acoustic fields inside the chip, but is also able to complete the manipulation tasks even on the first try, albeit more slowly.

Adaptiveness of the Controller
To show that the AMA controller can adapt even to radical changes in the system parameters, we designed an experiment where a controller was first trained on a particular chip, as in Figure 2e, and then we changed the manipulation system in different ways, without telling the controller explicitly about the change.We changed the system in two ways: 1) by suddenly increasing all manipulation frequencies by þ10%.The original frequencies were 100 frequencies linearly spaced between 65 and 700 kHz and the new frequencies were 100 frequencies linearly spaced between 71.5 and 770 kHz, but as an example, the controller still uses the model developed for 65 for the 71.5 kHz.The 10% frequency shift is enough to make all the acoustic field models meaningless [9] and mimics sudden changes in the chamber, such as a bubble entering the chamber or a large change in temperature.2) By physically changing the chip to a completely new one, this time with three outlets instead of one.This emulates a common use case, where a user disposes an old chip and replaces it with a new one, possibly with slightly different shapes and dimensions.We then completed two cycles of the square-shaped manipulation path in the new conditions.All experiments were repeated five times.
When the controller must relearn the models, the forgetting factor γ can become important, with γ ¼ 1, there is no forgetting, and controller adaptation is expected to be slower, whereas, with γ < 1, the controller is expected to adapt faster.Therefore, we also tested γ ¼ 1 and γ ¼ 0.99.The results are shown in Figure 4.
The data show that after the frequency shift (FREQ in Figure 4), the manipulation took far longer to complete.However, already in the second cycle, the controller performance has significantly improved, and the manipulation times are again comparable to what they were before the frequency shift.This shows that even trained controllers can readapt to new conditions in an online manner.
We hypothesized that a controller with a forgetting factor would adapt faster to changes, i.e., that the adaption would be faster with γ ¼ 0.99 compared to γ ¼ 1.However, the calculated P-values are above 0.05, so the data did not support the hypothesis.This also depends on how much data has been accumulated before the frequency shift.Nevertheless, the advantage of γ < 1 is that very old data with insignificant weights can be safely discarded, so in practice γ < 1 is still recommended because it guarantees that the memory requirements of the algorithm do not increase indefinitely.
Note that if we stopped updating the model (Figure 1b) and relied solely on the trained model, the controller always got stuck sooner or later, even without any frequency shift.For example, we tested manipulating the particle along a rectangular-shaped path with the trained model but without updating the model anymore.After 8 min, the particle had not even reached the first waypoint (Figure S10, Supporting Information).This is because of the greedy nature of the controller in exploitation: if the model of the greedily chosen frequency has errors and predicts significant motion toward the current waypoint, but results in no motion of the particles, then the controller will keep on choosing the same frequency over and over again.
When the chip was changed to a completely new one (CHIP in Figure 4), but with the controller trained for the old chip, the results were similar as with the frequency shift: the first cycle with the new chip was significantly slower than with the original chip.Nevertheless, already in the second cycle, the manipulation times are in the same range as they were before changing the chip.
In sum, these results highlight the importance of model adaptation during the experiments: the controller can adapt to changes in system parameters and can slowly correct its models during manipulation and regain its performance.

Manipulation of Multiple Particles
Finally, to show that the AMA controller is not limited to manipulating a single particle but can be also applied in the manipulation of multiple particles without retraining or changing the controller, we assigned the controller a task to simultaneously position four particles in the chamber to different locations within the chamber.The controller was initialized with the model obtained from the single particle training experiments.The results are shown in Figure 5a and Video S3, Supporting Information.We also performed an experiment where four particles were manipulated along complex paths simultaneously within the chamber.The results are shown in Figure 5b.The results in Figure 5a,b show that the AMA controller can position multiple particles relatively accurately within the chamber, although the path following accuracy is degraded compared to the single particle manipulation.Altogether, these results demonstrate that the AMA control algorithm is not limited to manipulating a single particle.

Conclusion
In this work, we proposed the AMA controller-a controller with an adaptive acoustic model-for manipulating particles inside microfluidic chips using bulk acoustic waves from a single transducer.Simultaneous manipulation of up to four particles was demonstrated.The controller uses similar approach as the ϵ-greedy algorithms [1] for balancing exploration and exploitation, but the novelty in our controller is that the exploitation is based on a sparse regression model of the acoustic fields, learned for each frequency in an online manner.The regression model enables the controller to retain knowledge from past experiments to better manipulate particles in the future.
We have shown that during subsequent repetitions of a task, the controller performance is increased, the manipulation times are shorter, and the precision of the manipulation is better, because the controller learns the acoustic field shapes.Furthermore, after training, we have shown that the performance of the controller improves even in tasks it has never done before, because the controller learns the acoustic field shapes of the chamber, not the specific actions required to complete a particular task.Despite this, the controller is still adaptive and can readjust its model when it becomes invalid due to sudden changes in the system parameters.
Compared to the ε-greedy control method we have reported previously, [1,9] the new controller shows far better path following precision and the manipulations are faster (e.g., %10 min for AMA and %108 min for ϵ-greedy).The poor precision and long manipulation times were the bottlenecks of applying the previous controller in practical applications of droplet microfluidics and particle sorting. [53]Therefore, before acoustic manipulation can become a routine practice, in applications such as lab on chips or particle/cell sorters, improving the speed and the precision is critical.However, even if we now cut down the manipulation times by one order of magnitude, whether this is yet sufficient depends on the application.In droplet-based labson-a-chips, one can envision devices with only a few droplets, where each droplet contains assay reagents or samples, and there are at most a few or perhaps tens of different chemical reactions taking place.In such a device, manipulation times measured in minutes might be sufficient, waiting tens of minutes for the results of a diagnostic test could be entirely reasonable.But in devices with thousands of particles or droplets, or even millions, acoustic manipulation which takes several minutes per particle is likely still too slow.However, in 1D, acoustic sorting with millions of cells into one of the outlets of a continuous flow device has been demonstrated; the core idea of our machine-learningbased algorithm can be adapted to optimize the performance of such a sorter also in an online manner.Also, even in 2D, it should be possible to parallelize the manipulations, i.e., have multiple chambers, each with their own transducer and machine vision system, to increase the throughput.
Compared to magnetic manipulation, the advantage of our acoustic manipulation method is that it can manipulate also nonmagnetic materials, including most biological materials.This makes it easier to apply acoustic manipulation in many different in vitro bio-applications.However, for in vivo applications, the nonspecificity of acoustic force can be also considered a disadvantage: magnetic fields affect only the magnetic objects inside the body, having minimal effect on the tissues, whereas highpower acoustic fields can cause heating of the tissues and soft tissues damp ultrasound.
Compared to (di)electric manipulation, our method does not require high voltages, which carry a risk of short circuits when used in liquid environments.We also do not need patterning of the electrodes inside the microfluidic chip, which simplifies the chip fabrication.However, with carefully designed electrodes, highly localized fields affecting only particles in the immediate vicinity of an electrode have been demonstrated, which has allowed sorting single particles or droplets at a high throughput, e.g., sorting droplets at a rate of 1.6 kHz. [53]Our closed-loop acoustic manipulation still needs to be sped up by five orders of magnitude to reach that level of throughput.
Compared to the control algorithm report by Zhou et al. [41] and other acoustic manipulation methods, [3,[41][42][43] our algorithm does not need simulated acoustic field shapes, large datasets to predict particle motions, or a large number of training episodes on how the particles move in response to different frequencies to be collected prior the manipulation.In our previous article, [1] we showed that the algorithm from Zhou et al. [41] was not able to complete any manipulation tasks successfully in this acoustofluidic chip, which was the main motivation for developing adaptive algorithms that learn online.The reason for not being to complete any manipulation was the same as in the current work; here, when we turned off the adaptation (Figure S10, Supporting Information), the controller will get stuck when it chooses a frequency that it thinks will move the particles in a particular way, but the frequency does not move the particles at all.In both algorithms, such a model discrepancy will lead the controllers to keep on choosing the same frequency repeatedly.Also, in the article of Zhou et al., [41] the data-driven models of the particle motion were obtained by distributing the particles throughout the plate manually using tweezers, which is not possible in a closed-channel acoustofluidic chip that we used here.Comparing the sizes of objects manipulated, Zhou et al. mostly manipulated 600 μm solder balls, while we manipulated spherical particles in the range of 50-100 μm, i.e., an order of magnitude smaller.
Finally, the advantage of the proposed AMA controller is that its internal model is developed based on physical insights; it models the Gor'kov potential for each frequency.This means that the model has clear interpretability and can be compared to experimentally observe acoustic field shapes, like we have done.This contrasts with black-box models, such as neural networks, [42,54,55] which can be difficult to interpret and therefore it is difficult to understand why the controller acts the way it does.However, some caution in interpreting the models is in order: any other phenomena transporting the particles, such as acoustic streaming [56][57][58] or fluid flows, [59] will affect the modeling results.In our case, our particles were relatively large, so the drag force from acoustic streaming can be assumed to be insignificant, and the pump was turned off to ensure that the fluid did not flow during manipulation.But if these other phenomena are significant, it may be necessary to fit models that include streaming and flow effects and not only the primary acoustic force.
In the future, the algorithm could be extended in several ways.For example, instead of using sinusoidal basis functions for the regression, one could use other types of functions, e.g., piecewise polynomials or mode shapes from numerical simulations.Such basis functions would likely work better with nonrectangular chambers.[58] To account for those effects, it would be easy to have two separate regression models for each frequency, one for the x-component and one for the y-component of the particle displacement, with the expense of doubling the number of parameters in the model.

Experimental Section
Design and Fabrication of Acoustofluidic Chips: The CAD diagrams of the two chips are shown in the Figure S1, Supporting Information.The chips were fabricated by Klearia, France and by wet etching of 1 mm thick fused silica glass.In the middle of the chip, there was a rectangular chamber (length: 7 mm, width: 6 mm, height: 0.15 mm), with one inlet and one outlet channel for the first chip and three outlets for the second (Figure S1, Supporting Information).A 1 mm thick fused silica glass lid was thermally bonded to the chips.Fluidic connectors (Nanoport, IDEX Health & Science, LLC) were adhesively bonded to the chips, and silicone tubes were attached to the connectors.
Particle Manipulation Experiments: Schematic of the particle manipulation setup is shown in Figure 1a.A piezoelectric transducer (NCE45, Noliac, Denmark, 15 mm Â 15 mm Â 2 mm) was adhesively bonded to the backside of the chip.The transducer was driven by signals from an arbitrary waveform generator (PCI-5412, National Instruments), amplified by a 400 W class AB RF amplifier (1400 L, Electronics & Innovation).The chamber and particles were imaged by a microscope camera (Basler acA2040-120uc, Germany, with 0.5Â magnification), and the 2D positions of the particles were recorded by a custom machine vision algorithm, written in MATLAB.To improve particle visibility for the machine vision to better track them, we illuminated the chamber with a 100 W LED panel.
For the manipulation experiments, we used polystyrene microparticles (DBP50K/DBP70K, LAB261, Palo Alto, CA, USA, density: 1.05 g cm À3 , diameters: 50/70 μm and 90 938-10 mL-F, Sigma-Aldrich, diameter: 100 μm).Polystyrene microparticles have a positive acoustic contrast factor. [60]We used blue-dyed particles so that the machine vision algorithm could better detect the particles.Before the experiments, we diluted 1 mL of particle stock solution with 100 mL of deionized water and mixed the solution thoroughly. 1 vol% Triton X-100 surfactant was added to prevent the particles from agglomerating due to their hydrophobic behavior.The particles were pumped into the chamber with a syringe pump (Aladdin, World Precision Instruments).For the manipulation experiments with droplets, we used the flow-focusing device from our previous work [9] with 2.5 vol% sorbitan monooleate (Span 80, Sigma-Aldrich) in hexadecane oil (Fisher Chemicals) as the continuous phase and blue-dyed deionized water as the dispersed phase (droplets).The flow rates were Q water = 0.5 μL min À1 and Q oil = 50 μL min À1 .
Unless otherwise noted, in our experiments, we used ε ¼ 0.1, γ ¼ 0.99, λ ¼ 2 Â 10 À4 , H ¼ 500, M ¼ f1, : : : , 6g Â f1, : : : , 6g, t ¼ 0.5 s, U piezo ¼ 25 V, and A was 100 frequencies linearly spaced in the range from 65 kHz to 700 kHz.In every control cycle, the piezoelectric transducer was driven by a sinusoidal signal with a duration of half a second.The duration of the signal was chosen to ensure that there was enough time for the standing waves to fully develop within the chamber, and so that it was significantly longer than the time it took for the camera and machine vision algorithm to detect particle locations (0.1 s).
When manipulating particles along paths, a waypoint along the path was assigned as the current target of each particle.When a particle reached within 100 μm of its current waypoint, a new waypoint was assigned.
Chladni Patterns: To obtain the Chladni patterns in Figure 3, the chamber was filled with many particles, and the transducer excited with a particular frequency at a voltage of 67 V for a duration of t ¼ 2 s.

Figure 1 .
Figure 1.a) Schematic diagram of the experimental setup.b) Flowchart of the control algorithm.
Figure2.a) Manipulation times for the proposed AMA controller, compared to the ε-greedy controller from our previous work.[1]The task is to manipulate a single particle through a rectangular path.Notice the logarithmic scale on the y-axis.During subsequent repetitions of the task, the AMA controller has built knowledge of acoustic field shapes and can better predict how the particle will move, resulting in shorter manipulation times.The images in panels b-d correspond to the data points indicated in the Figure b-d) Examples of the manipulation paths from different experiments.Notice that (b,c) exemplify the slowest manipulations with the AMA controller, while (d) exemplifies the fastest manipulation done with the ε-greedy controller; comparing the median results would show the AMA controller outperforming ε-greedy controller even more.The manipulation times and accuracies were 60 min and 0.13 mm (b), 15 min and 0.08 mm (c), 110 min and 0.1 mm (d).e) A fully trained AMA controller completing a manipulation task it has never done before (a complex path with four different sections).The manipulation accuracies were 0.05 mm (T), 0.06 mm (U), 0.05 mm (N), and 0.04 mm (I).f ) The AMA controller completes the same task as in e) without any training.The manipulation accuracies were 0.47 mm (T), 0.09 mm (U), 0.22 mm (N), and 0.09 mm (I).All scale bars: 1 mm.

Figure 3 .
Figure 3. Experimental acoustic patterns of the acoustofluidic chamber compared to the regression-based Gor'kov potential models for the same frequencies.Scale bar is 1 mm.

Figure 4 .
Figure 4. Adaptiveness of the controller.CHIP = a different chip was used, with a controller that was trained for the previous chip.FREQ = all manipulation frequencies were increased by 10% from the frequencies the controller was trained with.Cycle 1 = first manipulation cycle in the new conditions.Cycle 2 = second consecutive manipulation cycle in the new conditions.Note the logarithmic scale on the y-axis.The P-values were calculated using the Welch's unequal variances t-test.

Figure 5 .
Figure 5. Multiparticle manipulation.a) Four particles marked with blue symbols are guided to the waypoints marked with red.Each time the particles reach the defined waypoints new waypoints are assigned.b) Path following experiment with four particles.The manipulation accuracies were: 0.04 mm (T), 0.13 mm (U), 0.14 mm (N), and 0.06 mm (I).Scale bar is 1 mm.