Extremum seeking control of nonlinear dynamic systems using Lie bracket approximations

In this article, we consider extremum seeking problems for a general class of nonlinear dynamic control systems. The main result of the article is a broad family of control laws which optimize the steady‐state performance of the system. We prove practical asymptotic stability of the optimal steady‐state and, moreover, propose sufficient conditions for the asymptotic stability in the sense of Lyapunov. The results generalize and extend existing results which are based on Lie bracket approximations. In particular, our approach does not rely on singular perturbation theory, as commonly used in extremum seeking of nonlinear dynamic systems.


INTRODUCTION
Many problems in applications require the stabilization of a control system at some optimal operating point. An optimal operating point is often characterized as a state of the system which is an equilibrium point and where a given state-dependent cost (performance) function takes its minimal or maximal value. For complex systems such an optimal state cannot be determined easily because a model is often inaccurate or is not available at all. In addition, in applications the optimal state is often time-varying and state measurements are not available. Thus, the development of model-free real-time optimizing control laws, known as extremum seeking control laws, that stabilize an a priori unknown optimal operating point is both challenging in theory and highly relevant in practical applications.
Extremum seeking has been extensively studied in the control literature and can be traced back to the early 1920s. However, a solid theoretical foundation of extremum seeking control has been developed only over the last two decades (see, eg, Reference 1 for the literature overview). Today there exist a number of results, both on theoretical studies and practical applications (see, eg, . Classical extremum seeking control laws exploit time-periodic oscillating input perturbations in order to find and stabilize the optimal steady-state of the system without the knowledge of a system model. One such perturbation-based framework for the analysis and design of extremum seeking systems was proposed in Reference 9 and relies on Lie bracket approximations. The main idea therein is that trajectories of the extremum seeking system approximate trajectories of a so-called Lie bracket system which corresponds to a gradient-like dynamics which optimizes the cost function. Based on the Lie bracket system and its corresponding extremum seeking system, a whole analysis and design framework has been established, see, for example, References 16,17,[21][22][23][24][25][26][27][28][29][30][31][32]. In particular, extremum seeking for dynamic nonlinear systems using Lie bracket approximations has been addressed in Reference 25. In that article, a combination of Lie bracket approximations and singular perturbations techniques (time-scale separation) has been proposed. In general, for dynamic extremum seeking systems, a singular perturbation approach is quite common, see, for example, the articles. 2,4,11,17,33,34 In this article, however, we propose an alternative approach without singular perturbation theory using the techniques developed in Reference 16. In article 16, we extended the results of Reference 9 and gave a rather general description for a whole family of extremum seeking control laws. Using the Chen-Fliess series, it has been shown in Reference 16 that the proposed control laws achieve practical stability and, under additional assumptions, asymptotic stability in the sense of Lyapunov. However, the results in Reference 16 do not directly apply to nonlinear control systems with dynamic maps.
Therefore, the goal of this article is to extend the results of paper 16 to dynamic systems utilizing Chen-Fliess series techniques and to extend the results in Reference 25 in terms of a broader class of control laws and more general and stronger stability results. In particular, the main contributions of this article are as follows. First, we introduce a family of extremum seeking control laws for rather general nonlinear dynamic systems with a well-defined steady-state map. Second, we analyze the asymptotic stability properties of the extremum seeking system with the proposed control laws. Specifically, we prove practical asymptotic stability of the optimal steady-state, and in addition, we provide conditions to achieve asymptotic stability in the sense of Lyapunov. In contrast to References 2,4,11,17,25,33, our prove method solely relies on Chen-Fliess series techniques and does not depend on classical singular perturbation arguments. Hence our approach provides a general and monolithic way to the analysis and design of extremum seeking systems based on the Lie bracket approximation framework.
The remainder of this article is organized as follows. The problem statement is given in Section 2. Section 3 contains the main results of the article. The extremum seeking control laws are described in Subsection 3.1, while sufficient conditions for practical asymptotic stability and asymptotic stability in the sense of Lyapunov are stated in Subsection 3.2. In Section 4, we illustrate the obtained results by an academic example. The appendix contains some auxiliary statements and the proofs of the main results.

Notations and definitions.
Throughout the article, we use the following notations: k = 1, n-the integer number k varies from 1 to n ∈ N; ij -Kronecker delta: ii = 1 and ij = 0 whenever i ≠ j; R + -the set of all nonnegative real numbers; B (x * ) --neighborhood of an x * ∈ R n with > 0; M , M-the boundary and the closure of a set M ⊂ R n , respectively; M = M ∪ M; -the class of continuous strictly increasing function ∶ R + → R + such that (0) = 0; . Notice that the limit may exist at x ∈ R n even if f ,g are not differentiable at x.
If the above conditions hold for any ∈ (0, ], ∈ [ ( ), +∞) with some fixed > 0 and ( ) > 0, then a point x* is said to be uniformly asymptotically stable for the given system with respect to a set  ⊆ R n .
Let us emphasize that the parameter depends on parameter in Definition 1, which distinguishes it from the definition of practical asymptotic stability. Furthermore, the case  = R n means global attractivity (and asymptotic stability), while  ⊂ R n means that the above properties hold locally (in a set ).

PROBLEM STATEMENT
Consider the systeṁx and D x , D u are domains. If the output of system (1) depends only on u, that is, y(t) = h(u(t)), then we say (1) is a static system (or static map), otherwise, we say (1) is a dynamic system (or dynamic map).

Definition 2.
Consider system (1) and assume that there exists a map : Then the map u →h( (u),u) is called the steady-state map of system (1).
In this article, we consider the following problem setup. Problem 1. Consider system (1) and assume its steady-state map exists. Let J ∶ D y → R be a cost (performance) function which depends on the output of system (1) and assume that there exists a unique minimizer of the function in D u . We aim to construct a control law u(J(y(t)),t) which optimizes the steady-state performance in the following sense: for any x 0 ∈ D x , t 0 ∈ R, the solutions x(t) of system (1) with input u(J(y(t)),t) and x(t 0 ) = x 0 possess the property J(y(t)) → J * = min u∈D uĴ (u) as t → +∞.
Hence system (1) converges to the so-called optimal steady-state x = (u*) if the optimal steady-state input is applied to the system. Notice that we assume that the control laws only depend on performance output measurements. We do not assume that f , h, , J are known.
As it is shown in Reference 25, under suitable conditions, the dynamic control laẇ solves Problem 1 in the sense that it guarantees singular practical uniform asymptotic stability of the point q* = ( (u*), u*) for the composed system given by system (1) and the dynamic control law given above. Here, j ∈ N, e j denotes the jth unit vector in R n u , and , are some positive parameters. The main idea behind the approach of the article Reference 25 is to deduce the singular practical asymptotic stability of (1) with the above control law from the asymptotic stability properties of the so-called boundary layer model and the practical asymptotic stability properties of the so-called reduced systeṁũ The asymptotic practical stability for the reduced system is established by using the Lie bracket approximation approach, 21 while the stability properties of the actual closed-loop system are inferred by the stability properties of the boundary layer model and the reduced system using singular perturbation results. 35,36 In particular, the parameter > 0 guarantees a time-scale separation when chosen small enough, while > 0 guarantees a gradient-like dynamics forĴ(u) when chosen large enough (relatively to ). In other words, the role of parameter > 0 is to ensure that the trajectories of the reduced system tend asymptotically to a neighborhood of u*, while ensures that the system (1) is always in a quasi-steady-state (and thus is quasi static system with respect to the dynamics of dynamic control law).

Extremum seeking control laws
In this section, we will introduce a whole family of control laws for the solution of Problem 1 provided that system (1) satisfies certain assumptions.
Consider system (1) and let u* ∈ D u be the optimal steady-state input, that is, the unique minimizer of the function J(h( (u), u)). For stabilizing the optimal steady-state x = (u*), we propose the following family of dynamic output control laws:u where e j denotes the unit vector in R n u with nonzero jth entry if j ≤ n u or with nonzero (j − n u )-th entry if n u + 1 ≤ j ≤ 2n u . The time-varying dithers v , j (t) are given by v , with > 0, k j ∈ N, k j 1 ≠ k j 2 for all j 1 ≠ j 2 , and the so-called control functions g j , g j+n u ∶ R → R are such that for each ∈ R, In Reference 16, we have proposed the formula which fulfills the last condition, that is, that the Lie bracket is constant. In this article, we use a more convenient formula, that is, we parametrize the functions g j , g j+n u by g j ( ) = r j ( ) sin j ( ), where r j , j are such that r 2 j ( ) with r j ∈ C(R; R), j ∈ C 1 (R; R). Such a representation of the control functions g j , g j+n u has also been used in Reference 37. The proposed choice of g j , g j+n u ensures that the trajectories of the systeṁũ approximate trajectories of the so-called Lie bracket system, which in this case takes the form of a gradient descent dynamicsu see References 16,21 for more details. If the functions , then this property can be exploited to prove the singular practical asymptotic stability of q* = ( (u*),u*) for system (1) and (2) using the approach of Reference 25. However, many functions satisfying (3) fail to satisfy the C 2 assumption at q* (see Section 4), so that the results of Reference 25 are no longer applicable.

Stability results
In this subsection, we establish stability properties of the point q* = ( (u*),u*) for the closed-loop system (1) and (2). We make the following assumptions: u)), satisfies the following properties for all u ∈ D u : with some functions 11 , 12 , 21 , 22 ∈ , and a nondecreasing function 3 ∶ R + → R + . A3.1) For all j 1 ,j 2 ,j 3 ∈ {1, … ,2n u }, the functions g j 1 •J•h ∈ C(D x × D u ), g j 1 •Ĵ ∈ C 2 (D u ⧵ {u * }; R), and L e j 2 (g j 2 •Ĵ) (g j 1 •Ĵ), L e j 3 (g j 3 •Ĵ) L e j 2 (g j 2 •Ĵ) (g j 1 •Ĵ) ∈ C(D u ; R). Furthermore, for any compact subsets D ′ u ⊆ D u and D ′ x ⊆ D x , the functions g j 1 •J•h are Lipschitz continuous on D ′ x × D ′ u with respect to its first argument, and the functions g j 1 •Ĵ are Lipschitz continuous on D ′ u . *) To simplify the presentation, in the rest of the article we put
Note that Assumption A4.1) is common in extremum seeking literature, see, for example, References 17,25,38. Together with Assumption A1.1), this implies the existence of a Lyapunov function for system (1). 39 However, we expect that a similar result can also be obtained for systems which do not admit a Lyapunov function. One of such cases is considered in Reference 40.
Assumption A5.1) ensures that the distance between the trajectory x(t) and the quasi-steady-state (u(t)) cannot be infinitely large over a time interval of length , which means that subsystem (2) has to vary slow enough. In particular, for a static steady-state (u) ≡ const it means that x(t) cannot escape to infinity in finite time t = . In many cases, Assumption A5.1) directly follows from A4.1). One particular case is considered below.
The proofs of the obtained results represent constructive procedure for choosing small enough and large enough ( ). In particular, it can be seen that, under the conditions of Lemma 1, it suffices to take = √ with some > 0. This refines the estimate = 1 k with k > 2 proposed in Reference 25. Unlike the approaches of papers, 25,36 we do not apply singular perturbation theory to prove the obtained results. Instead, our proofs are similar to the techniques introduced in our article 16 and exploit the Chen-Fliess series expansion of the u-component of the solutions of closed-loop system (1)-(2) and a thorough analysis of the behavior of the functions J and V along the trajectories of (1)- (2). In the particular case of the static system with y = u, the Problem 1 and its solution coincide with Reference 16. Consequently, the asymptotic stability properties of (u*) for subsystem (1) can be established by the known results for systems with slowly varying parameters 39 or stability conditions for a family of sets. 27
Note that, although the proof of Theorem 2 relies on the techniques from Reference 16, the results of Reference 16 cannot be directly applied because the cost function J depends both on x and u. Thus, system (1) and the property of slow and fast dynamics in (1)-(2) have to be taken into account. This also yields that, unlike, 16 asymptotic decay of the function J(u) along the trajectories of system (1)-(2), in general, cannot be guaranteed.
Assumption A4.2) implies that, for each fixed value of u, the corresponding equilibrium point x = (u) is exponentially stable for system (1), while A2.2) ensures an exponential decay of the functionĴ along the trajectories of systeṁ As it will be demonstrated in Section 4, the property of asymptotic stability in the sense of Lyapunov may not be satisfied if subsystem (1) exhibits only asymptotic (but not exponential) stability properties. Nevertheless, we expect that it is possible to prove Lemma 1 and Theorem 2 under relaxed assumption on J and V. We leave this issue for the future studies. Let us underline that in A3.2) the optimal cost valueĴ(u * ) (but not the optimal state) is assumed to be known in order to guarantee that the vector fields of system (1)-(2) tend to 0 as the cost function approaches its optimum. Such situations naturally arise in many control problems when the optimal cost value is defined a priori (eg, in regression, synchronization or target tracking tasks), or when the deviations between the cost function and its optimal value can be measured. Another application arises from vibrational stabilization problems, as discussed, for example, in Reference 16.

EXAMPLE
In this section, the proposed extremum control laws are illustrated by the following academic example. Consider a spring-mass system oscillating in horizontal plane with friction and control, that is, described by the equation where l > 0 is length of the spring, k > 0 is spring constant, m > 0 is mass, (l) ≥ 0 is damping, and u is control. In the sequel, we assume m = 1. We aim to stabilize system (4) at the state l = l*,l = 0, assuming that only the measurements of y = l − l* are available for control design. Obviously, this problem can be solved by using time-invariant controls. However, we apply the dynamic control laws of type (2) and use the cost function J(y) = y 2 to illustrate the main results of the article.

Case 1: Linear damping
Assume first that damping is linear, that is, (l) =l with some > 0, and rewrite equation (4) in the variables x 1 = l, x 2 =l:̇x For each fixed value of u system (5) possesses the exponentially stable equilibrium , so that A4.2) holds. Thus, the quasi-steady-state is given by . Furthermore, as J(y) = y 2 = (x 1 − l*) 2 , the functionĴ(u) = 1 k 2 (u + kl * ) 2 satisfies A2.2). Following the approach of Section 3, one may puṫ where We consider four pairs of functions g 1 ,g 2 with different properties. Namely, functions exploited in Reference 25, uniformly bounded functions (see Reference 22) and functions vanishing at the optimal points (see Reference 29) In addition, we propose the function combining the properties of boundedness and vanishing at the optimal point (another example is given in Reference 16): For the numerical simulations we put k = 10, = 5, l* = 1, = 1 4 , = 25, = 10. The results of numerical simulations illustrate that the controls with the functions g 1 ,g 2 given by (7) and (8) lead to nonvanishing oscillations in a neighborhood of the optimal point, that is, only singular practical asymptotic stability is ensured (see Figure 1). However, the functions g 1 ,g 2 chosen according to (9) or (10) satisfy the additional property A3.2), so that the asymptotic stability in the sense of Lyapunov holds (see Figure 2) F I G U R E 1 Time plots of x 1 (t), x 2 (t), u(t), and ||u(t)|| for system (5)-(6) with functions g 1 , g 2 given by (7) (left), (8) (right). Initial , and ||u(t)|| for system (5)-(6) with functions g 1 , g 2 given by (9) (left), and (10) (right). Initial

Case 2: Nonlinear damping
Consider now the case of nonlinear damping (l) =l 3 , so that system (5) takes the forṁ System (11) has the same quasi-steady-state (u) = which, however, is not exponentially stable for constant u. Indeed, the matrix of linear approximation for (11) in a neighborhood of (u) always has purely imaginary eigenvalues ±i √ k.
Using Barbashin-Krasovskii theorem (or LaSalle invariance principle), one can show that, for each fixed u, (u) is asymptotically stable for (11) and the conditions of Theorem 1 are satisfied. This can be performed, for example, with the Lyapunov function Note that although system (11) does not admit a Lyapunov function satisfying A4.2), it is possible to construct a higher order polynomial Lyapunov function. 42,43 The results of numerical simulations for system (11) with controls (7) and (10) are shown on Figure 3. The values of system' and control parameters are taken the same as before. It is interesting to note that in this case even controls with vanishing amplitude do not yield asymptotic stability in the sense of Lyapunov.

CONCLUSIONS AND FUTURE WORK
In this article, we have addressed extremum seeking problems for a general class of nonlinear dynamic systems with a well-defined steady-state map. We proposed a broad family of dynamic control laws for extremum seeking problems based on Lie bracket approximation ideas, which generalizes the results in References 16,25 and which guarantees singular practical asymptotic stability of the optimal state-steady. Unlike the existing results in the literature, the control functions in the proposed control law are not required to be twice continuously differentiable at the optimal steady-state, which provides more flexibility for designing control laws. In particular, the relaxation of regularity requirement is crucial for generating control functions g j , g j+n u that vanish at optimal operating point and for establishing asymptotic stability results in the sense of Lyapunov. The proof techniques proposed in this article extend the approach for analyzing static systems as introduced in Reference 16 and do not rely on classical results from singular perturbations theory, but solely on Chen-Fliess series techniques.
The main results of the article are proved under the assumption that the system admits a well-defined steady-state map and a family of Lyapunov functions for all steady-states. Although this assumption is rather common in the extremum seeking literature, it imposes certain restrictions on the class of systems. In our future work, we aim to show that the proposed approach can be also applied to systems, for which the existence of a steady-state map and a family of Lyapunov functions is not known. Other possible research directions include, for example, extremum seeking problems with a time-varying optimal operating point or problems with constraints, as well as stabilization of underactuated control systems with fast and slow dynamics.

t), and let the functions h i be Lipschitz continuous in D with Lipschitz constant L. Then
The proof is based on the Grönwall-Bellman inequality. Another version of this lemma can be found, for example, in References 16,44,45.

then (t) can be represented by the Chen-Fliess series:
where is the remainder of the Chen-Fliess series expansion.
Recall that system (1)-(2) evolves in the space D x × D u , where D u = B Δ u (u * ), D x = B Δ x ( (u * )) with u* being the unique minimizer of the functionĴ(u) = J(h( (u), u)) in D u , and Δ x , Δ u ∈ (0, +∞]. Let us underline that the assumption D u = B Δ u (u * ), D x = B Δ x ( (u * )) is made only to simplify the technical details in the proofs, and our main results can be proved without that. Without loss of generality, assume t 0 = 0,Ĵ(u * ) = 0.
Namely, we will choose a small enough 1 > 0 and a large enough 1 ( ) in such a way that, for any ∈ (0, 1 ), > 1 ( ), and any u 0 ∈  c J , x 0 ∈ B x ( (u * )), the solutions (x(t),u(t)) of system (1)-(2) with initial condition From Lemma 4 and Assumptions A1.1) and A4.1), there exists a function V ∈ C 1 (D x ×D u ; R + ), functions 11 ,̃1 2 ,̃2 1 ∈  and a positive constant̃2 2 such that, for all x ∈D x , u ∈D u , Since (x(t), u(t)) ∈D x ×D u for all t ∈ [0, ], we have that Then estimating the time derivative of the function V along the trajectories of system (1)-(2), we geṫ Observe that, for any 0 > 0 and for all ≥ With such a choice of ,
Then integrating (B4) we get Under the above choice of , the right-hand side of the above inequality is negative at t = while the left-hand side is nonnegative for all t ≥ 0. The obtained contradiction proves that there exists a T 2 ∈ (0, ] such that Together with (B5), this yields ||x(t) − (u(t))|| ≤̃− 1 11 (̃1 2 ( 0 )) for all t ∈ [T 2 , ]. Putting and taking for any ∈ (0, 1 ] the values 0 =̃− 1 12 (̃1 1 ( )) and ∈ [ ( ) = max{ 0 ( ), 1 ( )}, +∞), we conclude that Recall that, by the property P1) obtained in Step 1, u( ) ∈  c J . Then by P2) and the obtained estimate ||x( ) − (u( ))|| ≤ ≤ xu , we may conclude that the solutions (x(t),u(t)) of system (1)-(2) with initial conditions u 0 ∈ B u (x * ), Then we repeat the previous argumentation of Step 2 for u( ) ∈  c J , x( ) ∈ B̃x u ( (u * )) (̃x u = min { xu , Δ x }) and the same choice of , ( ), and conclude that Note that the latter estimate holds and To sum up Steps 1 and 2, we have ensured that the solutions of system (1)-(2) with initial conditions u 0 ∈ B u (x * ), x 0 ∈ B x ( (u * )) are well-defined inD x ×D u for all t ∈ [0, 2 ]. Furthermore, after a time interval of length , the trajectories x(t) and (u(t)) became -close. In the next steps of the proof we switch to the analysis of the behavior of solutions on the time interval [ , 2 ].
Step 3. In this step we consider an auxiliary system corresponding to (2)  Namely, we will show that, if the initial condition of such auxiliary system at t 0 = is equal to u( ) ∈  c J , then for any ∈ (0, 1 ), > 1 ( ), and any u( ) ∈  c J , x( ) ∈ B̃x u ( (u * )), norm of the difference between u(t) and the corresponding trajectory of the auxiliary system is of order with the initial condition u( ) = u( ) ∈  c J . Note that as soon as u( ) ∈  c J , the solutions of system (B6) are well-defined inD u for all t ∈ [ , 2 ], for any ∈ (0, 0 ] and > 0 ( ), where 0 is defined Step 1 and 0 ( ) is defined by Assumption A5.1). Using the integral representations for the u-component of the solutions of system (1)-(2) and for the solutions of system (B6) with initial conditions u( ) = u( ), we may write the estimate The Grönwall-Bellman inequality implies As proved in Step 2, ||x(t) − (u(t))|| ≤ for t ∈ [ , 2 ]. Hence, where = c w L g e L J c w √ . Let us also observe that, as it follows from Lemma 3 and Assumption A3.1), the solutions of system (B6) can be represented as where In (B8), we exploited the fact that u( ) = u( ). Since the functions L e j 3 g j 3 •Ĵ L e j 2 g j 2 •Ĵ g j 1 •Ĵ are continuous inD u From Assumption A3.1), the remainder R( , ) of representation (B8) can be estimated as ||L e j 3 g j 3 L e j 2 g j 2 g j 1 (Ĵ(u))||. Note that the obtained estimate does not depend on and u( ).
Note that the vector fields of subsystem (2) depend on the functionJ(x, u), so that we cannot obtain the above-mentioned properties from the analysis of the behavior of functionĴ(u) along the trajectories of (2), as it was done in Reference 16. Instead, we consider the behavior of the functionĴ(u) along the trajectories of auxiliary system (B6) and exploit representation (B8) and estimates (B7), (B9).
Step 5. On this final step, we show that the solutions of system (1)-(2) enter a prescribed neighborhood of ( (u*),u*) after a finite time, and then remain there.

APPENDIX C. PROOF OF LEMMA 1
We keep the notations from the proof of Theorem 1, define the sets  c J andD u ⊆ D u in the same way, and put̃x u ∈ (0, √ 11 ∕ 12 Δ x ),D x = B xu ( (u * )). Assumption A4.2) obviously implies Assumption A4.1) of Theorem 1. Let us show that Assumption A5.1) is satisfied as well.