Optimal scheduling and control for constrained multi-agent networked control systems

In this paper, we study optimal control and communication schedule co-design for multi-agent networked control systems, with assuming shared parallel communication channels and uncertain constrained linear time-invariant discrete-time systems. To that end, we specify the communication demand for each system using an associated robust control invariant set and reachability analysis. We use these communication demands and invariant sets to formulate tube-based model predictive control and offline/online communication schedule co-design problems. Since the scheduling part includes an infinite dimension integer problem, we propose heuristics to find suboptimal solutions that guarantee robust constraints satisfaction and recursive feasibility. The effectiveness of our approach is illustrated through numerical simulations


INTRODUCTION
A networked control system (NCS) is a set of control loops closed over a communication network. While this entails advantages such as reduced wiring, lower cost, and more flexibility and maintainability of the system, it also causes some disadvantages, such as information losses and delays, which may degrade the performance or cause instability. 1 All these issues become even more challenging in a multi-agent setting, where the communication network is shared by multiple agents and some degree of coordination is required in order to ensure that each agent obtains sufficient access to the network. In the multi-agent setting, an important imperfection of the communication link in NCSs is therefore bandwidth limitation.
In order to allocate communication resources to the agents, a medium access protocol is necessary to determine which agents are allowed to communicate through the shared network at any given time. A medium access protocol can be deterministic, stochastic, or combination of the two. Deterministic scheduling is advantageous from the control design standpoint since it enables one to provide guarantees on the system performance. However, providing strong guarantees over a wide range of operating points may cause over-provisioning. On the other hand, random access schemes, typically developed for those applications where the communication is sporadic, reduce the issue of over-provisioning, that is, they potentially reduce the number of communication channels and the frequency of the communication by each system. Although in a random access scheme transmissions might be postponed due to unavailability of the communication medium, the average amount of transmissions may be significantly reduced, while some overall closed-loop system performance, for example, ℒ 2 stability, can be enforced. 2 However, the use of random access schemes in safety-critical applications makes it hard to guarantee satisfaction of state and input constraints at all times.
Previous work has focused on defining ways to relate the medium access protocol specifications to system theoretic properties. An important concept is the maximally allowable transmission interval (MATI), which establishes an upper bound between two consecutive transmissions such that stability is guaranteed while accounting for the maximum possible communication delays. 3 This approach has been further extended to incorporate communication constraints, time-varying transmission intervals, and time-varying delays. 4 Additionally, the introduction of minimal allowable transmission interval 5 and average allowable transmission interval 6 has been proposed to further improve the existing framework. In self-triggered transmission schemes the next communication time is decided after each transmission based on a trade-off between control performance and communication utilization, using concepts similar to MATI. This approach has been applied to constrained linear systems in order to reduce transmission while guaranteeing stability and constraint satisfaction. 7 This scheme has been further extended to deal with constrained linear systems with additive disturbance. 8 All previously cited schemes aim to reduce the amount of transmissions while preserving stability. However, an alternative approach, based on rollout control, aims at maximizing control performance while restricting the communication utilization according to a given traffic specification (TS). In window-based TS 9,10 the number of transmissions are limited during each window of time, and in token bucket TS [11][12][13] a dynamic restriction is introduced to limit the transmission rate. Token bucket and window-based TS schemes have been extended to constrained linear systems with additive disturbance, in which a tube-based model predictive control (MPC) is used to guarantee robust satisfaction of state and input constraints. 13 This method relies on a robust control invariant (RCI) set that is Htime-steps invariant for the system between consecutive transmissions. Therefore, the H-RCI property is a related concept to the MATI for constrained discrete-time systems with additive disturbance.
All the aforementioned studies focus on a single system and, therefore, can deal with communication medium access by assuming that the medium is available whenever the system requests it. In order to deal with multi-agent settings in which the communication medium is shared between agents, some form of coordination (i.e., a medium access protocol) needs to be introduced. A distributed dynamic schedule which gives priority to nodes which larger errors, called "maximum-error-first with try-once-discard,ε has been proposed as network access protocol, using bit-wise arbitration, for linear systems. 14 Conflict-free transmissions for self-triggered schemes have also been proposed. 15 While the first two approaches decide on communication online, offline medium access protocols have been derived by means of optimal control and schedule co-design. One possibility is to optimize the schedule and control to minimize an H ∞ cost, 16 where the schedule determines which sensor may transmit through the channel at any given time. This optimization problem is solved by formulating a linear matrix inequality (LMI) and a heuristic method for scheduling to avoid combinatorial complexity. In another contribution, the exponential convergence rate of systems is optimized in the presence of random packet losses using an LMI. 17 A mixed integer quadratic program over a finite time horizon is considered in Reference 18 for NCSs, where offline and online schedules are discussed using branch and bound algorithms. However, solving this optimization problem in real time might not be viable due to its combinatorial nature. This issue is addressed in Reference 19 where, given an offline schedule, a suboptimal online schedule called optimal pointer placement is introduced which is a shifted version of the offline schedule. While these studies directly consider the medium access for NCSs, they do not account for state and input constraints.
Scheduling and control for uncertain multi-agent constrained NCSs has been addressed using the concept of safe time interval to provide access to the communication medium to the agents. 20,21 The safe time interval plays a role similar to H-RCI and MATI, but is used in a scheduling framework to coordinate the access to the medium for all agents. This is done by mapping the scheduling problem into the Pinwheel Problem or the Windows Scheduling Problem (WSP), for which a feasible schedule can be found with polynomial time algorithms when a sufficient schedulability condition holds. This strategy guarantees satisfaction of the state and input constraints for all future times. This framework is able to deal with packet losses and to adapt the schedule online to improve performance. 21,22 Additionally, tube-based MPC can be used to maximize the safe time interval for systems with additive perturbations. 23 The main limitation of these approaches is that they do not directly optimize closed-loop performance.
In this paper, we consider optimal communication schedule and control co-design for multi-agent NCSs with uncertain constrained systems and bandwidth limitation by extending previous work, 20-23 that is, by jointly designing the control and the communication schedule through an optimization problem. This extension eliminates unnecessary performance loss due to sequential schedule and control designs in the previous work.
Because this optimization problem is prohibitively complex, we propose to approximate it by a heuristic in order to find a suboptimal solution in real time. Additionally, this framework can compensate packet loss while guaranteeing satisfaction of the constraints if the number of packet losses are bounded during a window of time.
The main contribution of this paper is a schedule and control co-design framework to optimize a given performance index for constrained uncertain multi-agent NCSs. Because the optimization problem is often intractable, we propose a heuristic with low computational burden that accounts for optimality in an approximate way while still providing strong constraint satisfaction guarantees. The proposed heuristic restricts the scheduling search space to a finite set of feasible schedules, which makes the co-design problem tractable.
The rest of this paper is organized as follows. Preliminary results are recalled in Section 2. Offline and online scheduling optimization problems are introduced and heuristics are proposed in order to find suboptimal solutions in Section 3.
Some numerical examples are given in Section 4 to illustrate the proposed method and finally the paper is concluded in Section 5.
Notation. We define the set of integer numbers between a and b as We define (t) ∶= ( 1 (t), 2 (t), … , n (t)) to be an ordered set of n integers at time t, and the sequence Based on the sequence , we define the set of time instants which indicates the times at which the integer i is a member of the set (t). We call members of the set T i ( ) as t i,0 , t i,1 , … which are ordered increasingly, that is, t i,j < t i,j+1 for all j ≥ 0. The sum of two polyhedra P and Q is defined as and their difference as Set  ∶= ( 1 , … ,  q ) is an ordered set of polytopes  i ⊂ R n i for all i ∈  q 1 . The lexicographic optimization lex max specifies the optimizer x ⋆ that satisfies and for all i = 2, … , n.

PRELIMINARIES
In this section, we consider an optimal communication schedule and control design problem for a class of multi-agent NCSs.
F I G U R E 1 Multi-agent networked control system configuration. At each time t only n of state measurements x i (t) for all i ∈  q 1 , specified by the vector (t), are transmitted through the communication medium. Each controller i ∈  q 1 uses its latest received state measurement to compute u i (t), which is then applied to the system i Consider an NCS, depicted in Figure 1, where q uncertain linear time invariant systems have decoupled dynamics where v i (t) is a bounded disturbance and x i (t), u i (t) are the system states and inputs, respectively. The controller is required to enforce the state and input constraints x i (t) ∈  i and u i (t) ∈  i . We assume that the sets  i ,  i , and  i are compact polytopes which contain the origin, and that the pair (A i , B i ) is stabilizable. We denote the communication schedule (t) as a vector of n integers, containing indices of the systems that receive their state measurements at time t. With slight abuse of notation, we write i ∈ (t) whenever j (t) = i for some j ∈  n 1 . We note that, as it can be seen in Figure 1, the communication scheduler has access to the current state measurements, that is, x i (t); however, the controller has only access to the state prediction with an initial conditionx i (0) = x i (0). At each time instant t ≥ 0, the central scheduler selects n sensors (through tuple (t)) which transmit their measurements through the communication medium. Each controller i uses the state predictionx i (t), updated as per (11), to compute the optimal control policy u ⋆ i (t), which is then applied to the system i by its actuator.
We aim at jointly designing the communication schedule and control for the described NCS by minimizing either the worst-case or the nominal prediction of a given cost function. Note that for perturbed systems the worst-case cost is in general unbounded. In order to address that issue, we define the worst-case cost as an average cost and the deviation from the optimal average cost. The minimization is then performed in a hierarchical, or lexicographic manner: minimizing the average cost is the most important objective and guarantees optimal operation over an infinite horizon; minimizing the deviation from this average cost further optimizes the transient behavior which would otherwise be neglected. These concepts are discussed in the literature on stochastic optimal control where further optimality criteria are also proposed. 24 For the sake of simplicity, we do not discuss them here, but remind the reader that popular cost functions include the total discounted cost and/or using the expected value or any other statistical measure instead of the worst-case cost. Note, however, that this last option requires knowledge on the distribution of the noise.
For each system i we define the cost functions where x i,0 andx i,0 are the initial state and state prediction, respectively, and Since the disturbance sequence V i is unknown, we introduce the variables x i andx i , which are the system state and the predicted state for the worst-case disturbance, respectively. Note that we optimize in a lexicographic order, that is, the optimization is performed first with respect to the first element in vector J i , then the next elements are optimized in a way which does not jeopardize optimality with respect to the preceding elements, with the vector defined as It ought to be stressed that, since we perform lexicographic optimization, the optimization of the second cost in J i (X i , U i ) includes an additional constraint enforcing that the first cost, that is,Ĵ i (X i , U i ), is not worse than its previously computed optimal value. Remark 1. Optimization problem (12) yields the worst-case cost of a given communication schedule, control policy, and initial conditions for the NCS. In case  i ∶=  i , the disturbance is not vanishing and the infinite horizon quadratic cost is unbounded. This is the reason why we consider lexicographic optimization. The most important objective is to minimize the average costĴ i , which is finite if the states and inputs remain bounded. This average cost is however independent of transients, for example, any bounded deviation from the optimum over a finite time horizon averages to 0. In order to also account for transient performance, we propose to optimize the second cost, which penalizes deviations from the optimal average cost. Note that in the nominal case, that is,  i ∶= {0}, we haveĴ i = 0 for any stabilizing policy, while the secondary cost selects the optimal policy among all stabilizing ones.
The communication schedule and control design should guarantee satisfaction of the input and state constraints for each system. To that end, we formulate the optimal communication schedule and control design using ideas from robust min-max 25 and tube-based MPC. 26,27 We define U ∶= (U 1 , … , U q ) and formulate the optimal communication schedule and control design problem as lex min U, is the reachable set for the state prediction error. Indeed, one can verify that the state prediction error is used to tighten the state admissible set  i in (15c) to account for the effect of the unknown disturbance on the state evolution in time. This is clarified in the following lemma. (15), defines a joint optimal communication schedule and control design that enforces the agents' constraints (10b). Unfortunately, this optimization problem is impossible to solve since x i (t) is unknown for any t > 0; furthermore, even if the states were known, the problem would be very difficult to solve since the schedule is an infinite sequence of vectors, each containing n integer variables. In order to resolve these issues, we use the following definitions.
Consider the feedback policy u i (t) = −K i x i (t) for the system (10). Then, one can define positively invariant sets for the system as follows.

Definition 1 (Robust positively invariant set). Set
Definition 2 (Maximal robust positively invariant set). Set  i,∞ is the maximal robust positively invariant set for system (10) if it is robust positively invariant and  i ⊆  i,∞ holds for all robust positively invariant sets  i .
By definition, when the system state is inside a robust positively invariant set initially, it is kept inside that set at all future times by feedback policy u i (t) = −K i x i (t). Additionally, invariance of  i ensures that the system constraints are satisfied at all future times.
In order to compute the maximal robust positively invariant set, define the admissible set  i as and the one-step controllable set 28,29  K i as One can compute the maximal robust positively invariant set  i,∞ using Algorithm 1. Now assume that no control policy is given for system (10) and define the one-step controllable set Then, one can compute control invariant sets by relying on the following definitions.
Maximal robust control invariant set computation (see Reference 28 or algorithm 11.5 in Reference 29) Definition 4 (Maximal RCI set). Set  i,∞ is the maximal RCI set for system (10) when  i,∞ is a RCI set for the system and  i ⊆  i,∞ for all RCI sets  i .
When the system state is inside its RCI set initially, there exists a feasible control sequence which guarantees that the state remains the invariant set for all future times and therefore, the system constraints are respected. One can compute the maximal RCI set  i,∞ using Algorithm 2.
Sufficient conditions for finite-time termination of Algorithms 1 and 2 can be found in References 30 and 31, respectively. For system (10), Algorithm 1 terminates in a finite time when the invariant set is nonempty, the closed-loop system is stable, and  i is compact. 29 Likewise, Algorithm 2 terminates in a finite time when the invariant set is nonempty and sets  i and  i are compact. 31 Note that these invariant sets may have complex shapes. In order to reduce complexity of the maximal invariant sets, one may instead use invariant sets that are approximations of the maximal ones, see Reference 32 for instance.

OPTIMAL COMMUNICATION SCHEDULE AND CONTROL CO-DESIGN
In this section, we reformulate and simplify optimization problem (15). To that end, we first use reachability analysis to derive necessary and sufficient conditions for existence of a feasible control policy. Such conditions depend on the system dynamics and constraints, and impose restrictions on how often the state measurements must be communicated to the system. Therefore, these necessary and sufficient conditions on the control design result in necessary and sufficient conditions on the communication schedule design. We note that, in general, finding any feasible schedule is very hard since the schedule is a countably infinite set of integer variables. In order to deal with this issue, we restrict the search space of the feasible schedules. This strategy simplifies the scheduling design at the expense of optimality. We provide offline and online schedules and prove recursive feasibility of the online one.

Necessary and sufficient conditions for feasibility of the communication schedule
In this subsection, we provide a necessary and a sufficient condition for feasibility of a communication schedule, which can be used to find an approximate solution for the optimization problem (15).
Any feasible solution of the optimization problem (15) consists of a communication schedule and a set of control policies that satisfy input and state constraints (15b) and (15c). For a given communication schedule , a feasible control policy for system i exists only if the set  i (t) can be kept small enough for all t ≥ 0, that is, the system receives state measurement updates frequently enough. On the other hand, when the control policy is given, each system has a certain communication demand to guarantee that its state and input constraints are respected. We next formalize these observations in order to provide mathematical conditions for the existence of a feasible communication schedule and control policy.
In case of no bandwidth limitation, each system can receive its state measurement at all times. Therefore, the current state is known exactly and  i (t) = {0}. Then, existence of a feasible control policy for the system i is equivalent to existence of a RCI set for the system which includes the initial state x i (0). In case the bandwidth is limited, that is,  i (t) ≠ {0} for some t ≥ 0, one can use reachability analysis to verify whether a feasible control policy exists. We formalize this concept by defining the safe time interval i as the longest period of time during which i ∉ (t) and a feasible control policy exists for system i: Equation (22) states that, if x i,0 ∈  i and i ∈ (0), then there exists an input sequence which guarantees Consequently, if  i is robustly control invariant and if i ∈ (t) at least once during every i consecutive time instants, then a feasible control policy exists such that x i (t) ∈  i for all t. In the rest of the paper, we use where  i,∞ is the maximal RCI set for the system since any RCI set  i is a subset of  i,∞ , that is, the maximal set includes a larger set of feasible states. We remark that inequality i ( i,∞ ) ≥ i ( i ) typically holds. This inequality implies that using the maximal control invariant set for the computation of i , that is, i ( i,∞ ), yields the longest allowable transmission interval for the system. 23 The set of safe time intervals { 1 , … , q } can be used to formalize a necessary condition which needs to be satisfied by any feasible communication schedule . Let us define the times when the system i receives state measurements as Then, is a necessary condition for any feasible schedule as pointed out by the following lemma.

Lemma 2.
Any schedule which is feasible for optimization problem (15) satisfies (24). (22) which is contradictory. ▪ We note that in (24) no constraint is imposed on t i,0 ( ) since its value depends on the initial values of the state x i (0) and the state predictionx i (0).
In the following lemma, we specify a sufficient condition for feasibility of a communication schedule.

Lemma 3.
Assume that all systems have access to their initial states, that is,x i (0) = x i (0) for all i ∈  q 1 . Then, any communication schedule which satisfies is a feasible schedule for the optimization problem (15).
Proof. For each system i, the initial state is known and there exists a control sequence such that x i (t) ∈  i,∞ for all t ≤ i by definition (22). Since t i,0 ( ) ≤ i , the measured state x i (t i,0 ( )) is inside the set  i,∞ . Given x i (t i,0 ( )) ∈  i,∞ , a control sequence starting from t i,0 ( ) exists such that x i (t) ∈  i,∞ for all t ≤ t i,0 ( ) + i . As a result, x i (t i,1 ( )) ∈  i,∞ since t i,1 ( ) ≤ t i,0 ( ) + i . Therefore, there exists a control sequence such that x i (t) ∈  i,∞ for all t ≥ 0 by induction. ▪ Lemma 3 specifies the sufficient condition (25) for a feasible schedule. However, as pointed out before, finding the optimal communication schedule subject to this sufficient condition is very difficult since the problem is combinatorial with a countably infinite set of integer variables. We note that not only finding the optimal schedule is very difficult, but it may also be difficult to find any feasible schedule.
While finding the optimal communication schedule which satisfies (25) is very difficult in general, one can use available scheduling algorithms to find feasible schedules and uses these schedules to find a suboptimal schedule. To that end, we search for a feasible schedule for the instance I ∶= { 1 , … , q } of the WSP, 33 which is equivalent to searching for a schedule which respects (25). 21 Necessary and sufficient conditions for existence of a feasible schedule for an instance of the WSP are based on the density (I), defined as Given an instance of the WSP, inequality (I) ≤ n is a necessary and (I) ≤ 0.5n is a sufficient condition for the existence of a feasible schedule for the WSP. 33 There are polynomial time algorithms which find feasible schedules for instances of the WSP, for example, when (I) ≤ 0.5n holds. 34,35 One can also find a feasible schedule for an instance of the WSP, if it exists, by formulating an optimization problem and using mixed-integer solvers such as CPLEX, Gurobi, etc. 21 Remark 2. If is a feasible schedule for the WSP, then all shifted versions of it, that is, are also feasible schedules for the WSP.
Remark 2 can be used to construct a set of schedules which respect (25). In the following section, we propose a heuristic which finds a suboptimal feasible communication schedule, based on a given nonempty set of feasible schedules Note that the optimality gap in each example depends on the set of feasible schedules S . If one increases cardinality of this set by adding more feasible schedules to it, the optimality gap might reduce, but the computation expense increases. While it is hard to evaluate the optimality gap exactly, a rather conservative lower bound for the cost can be obtained by assuming that the communication channel is unconstrained, as we will discuss with some examples in Section 4.

Offline communication schedule and control design
In this subsection, we assume that a feasible communication schedule is given and we formulate an optimal control design problem based on this schedule. This formulation is used in the following subsection to jointly design the communication schedule and the control policy. Assume that a schedule is given which satisfies (25); furthermore, assume that the initial state for the system i is inside the maximal RCI set for that system and this initial value is known to the controller, that is, Then, one can find the optimal control policy for the system i which guarantees the state constraint satisfaction by solving Furthermore, the recursive solution of (29) guarantees x i (t) ∈  i,∞ for all t ≥ 0, as pointed out in the following lemma.

Lemma 4. The optimization problem (29) is recursively feasible and its recursive solution guarantees x i (t) ∈  i,∞ for all t ≥ 0.
Proof. We first prove that the optimization problem (29) is initially feasible. Inclusionx i (0) ∈  i,∞ holds by construction and since the schedule satisfies (25), the inequality t i,0 ≤ i holds by assumption. Therefore, the control sequence u i (0), … , u i (t i,0 − 1) ∈  i exists that satisfies (29d) for all t ≤ t i,0 based on the definition (22). Satisfaction of (29f) and (29d) in this period and Lemma 1 imply that This reasoning implies that the problem is initially feasible by induction. The feasible solution of the optimization problem also guarantees x i (t) ∈  i,∞ for all t ≤ t i,0 based on definition of i and the assumption x i (0) ∶= x i (0) ∈  i,∞ . Assume that the input u ⋆ i (0), … , u ⋆ i (t i,0 − 1) has been applied to the system, orderly; since i ∈ (t i,0 ), the state x i (t i,0 ) is measured and available to the controller. One can define x i,0 ∶= x i (t i,j ) in (29) and solve the problem recursively at time instants t i,j for all j ≥ 0 which guarantees x i (t) ∈  i,∞ ⊆  i for all t ≥ 0. ▪ Using the cost function (29a), one can find an optimal feasible schedule by solving lex min where S , specified in (28), is a finite set of feasible schedules for the instance { 1 , … , q } of the WSP. Note that optimization problem (30) is not hard to solve since the optimal solution can be found by enumerating among members of the set S . One can use the optimal solution of (30) as the feasible schedule in the optimization problem (29) in order to calculate the optimal control policy. Note that the solution of (29) needs to get updated based on the state measurements in order to guarantee x i (t) ∈  i,∞ for all t ≥ 0. This scheme is summarized in Algorithm 3.
Remark 3. Note that in Algorithm 3 the schedule ⋆ is computed inside the communication scheduler, see Figure 1. At time k = 0, each controller i has access to x i (0) by assumption, and solves the optimization problem (29) to obtain the optimal control trajectory u ⋆ i (0), u ⋆ i (1), … . Each controller i applies u ⋆ i (k) to system i at time k. At each time k ≥ 1, if i ∈ ⋆ (k), then controller i receives a new state measurement, in which case it solves the optimization problem (29) with the new initial condition Find feasible schedules for the instance { 1 , … , q } of the WSP and define the set S as in (28) Solve the optimization problem (30) to find the offline schedule ⋆ Solve the optimization problem (29)  Assign k ← k + 1 end while Remark 4. The optimization problem (29) has several similarities and differences with the H-RCI approach for NCSs with TS. 13 Tube-based MPC is used to design controls for an uncertain linear time-invariant discrete-time system with state and input constraints in both (29) and Reference 13. Furthermore, both approaches use invariant sets to determine acceptable upper bounds on transmission intervals, that is, i in this paper and H in Reference 13. However, tube-based MPC is used in (29) only to guarantee that the system remains inside the admissible set, while Reference 13 aims at asymptotically stabilizing the system to a smaller set, for example, the minimal robust positively invariant set. This difference is reflected in slight but important differences in the tube-based MPC formulations, such that the two approaches are similar in philosophy but solve different problems.
The optimal solution of (30) is an offline schedule since it is designed initially and it does not get updated afterwards based on the available information on the state of the systems. The optimal solutions of (30) and (29) return an optimized communication schedule and control which is based on a specific disturbance sequence for the system; however, the disturbance sequence is neither controllable nor known a-priori. Therefore, at each time instant the state measurements can be used to update the communication schedule and control in order to reduce the cost in (30). Note that in the optimization problem (30), we assumed thatx i (0) = x i (0) for all i ∈  q 1 which cannot be assumed in the following time instants and therefore, one cannot solve the optimization problem recursively for the sake of finding an communication schedule. This issue is addressed next.

Online communication schedule and control co-design
The offline communication schedule is only optimal for the worst-case disturbance; however, the actual disturbance is in general different, such that the offline solution is suboptimal. In this subsection, in order to eliminate this deficiency, we detail how the offline communication schedule design can be used to construct an online schedule with improved performance.
In the previous subsection, the communication schedule is fixed and in order to update the control sequence, one only needs to solve (29) for n of the systems at each time instant t > 0. However, each system i might be allowed to wait for longer than i time instants to receive a new measurement update without violating the constraints, depending on the disturbances that are actually injected to the system during this time. As a result, the current state measurement for each system can be used to determine whether the state measurement update for that system can be postponed. This point is clarified by the following example.
Example 1 (Online scheduling constraints). Consider a scalar system described by (10) where A i = B i = 1,  i =  i = {x ∶ |x| ≤ 1}, and  = {x ∶ |x| ≤ 0.1}. In this system,  i,∞ =  i and, for example, u i (0) = −x i (0) and u i (t) = 0 for all t > 0 is a feasible control policy that preserves the invariance for a period of time. One can show that in this case, i = 10 and

F I G U R E 2
The open-loop evolution in Example 1. The system is simulated with the initial condition x(0) = 1 and the designed feasible control policy in the example. The disturbance sequence is assumed to be random with a uniform distribution. The system is simulated 10,000 times and the first figure shows the percentage of the number of times in which the state constraint is violated before time t. The second figure shows the state and the state constraints over time in these experiments a feasible offline schedule should update the system i no later than t = 10. Now assume that at t = 8, the system i has not received a state measurement update yet and the state measurement shows that x i (8) = 0.25. While according to the constraints at t = 0, the scheduler should transmit a state measurement update to the system i no later than two seconds, that is, t ≤ i = 10, the new measurement at t = 8 can be used to show that the system i remains in  i,∞ at least until t ≤ 15. We have simulated this system 10,000 times with different uniformly distributed random disturbances. The simulation results in Figure 2 indicate that while i = 10, in less than 0.2% of the experiments, the system needed a measurement update before t = 30. In this case, the average time for violating the state constraint is 317.8 which is much greater than the minimum time, that is, i + 1. Nevertheless, we note that this average time might be much closer to i + 1 when the system is unstable.
Example 1 illustrates that the current state measurement can be used to find less restrictive constraints on the communication schedule. Next, we formalize the state measurement update deadline for each system i based on the current state measurement. Assume that the last measurement update for the system i took place at t 0 ≤ 0 and define as the optimal solution to (29) solved at time t 0 for a feasible schedule . Furthermore, assume that the system is scheduled to receive the following state measurement update at t 1 ≥ 0. Note that the optimal control policy cannot change before t 1 ≥ 0 since the controller has received no additional information after time t 0 . Based on the current state measurement update x i (0), the measurement update deadline i (x i (0), U ⋆ i ), that is the time until which x i (t) is guaranteed to remain in  i,∞ when the previously calculated optimal control policy is applied, can be defined as Note that i ≥ t 1 since the control sequence U ⋆ i guarantees x i (t) ∈  i,∞ for all t 0 ≤ t ≤ t 1 . Therefore, one can use i to postpone the state measurement update for the system i and reduce the scheduling conservativeness.
We propose an online communication schedule and control co-design in which the schedule constraints are adjusted based on the current state measurements. Since the communication schedule changes, in general, at each time instant in an online communication schedule design, one needs to evaluate (29a) for all i ∈  q 1 and at each time instant in order to pick the optimal schedule. As a result, the optimization problem (29) should be solved at each time instants, rather than only at time instants at which the system i has received a state measurement update. Therefore, next we modify (29) to be able to evaluate its cost function at all time instants. Note that the optimal control policy U ⋆ i can not be modified during the open-loop evolution periods since the controller has received no new information. Therefore, in order to respect this limitation, we reformulate (29) as ∶= lex min where is the optimal control policy calculated at the previous time instant. U ⋆,prv i is used in (32g) to preserve the latest optimal control policy until a new state measurement update arrives. In (32d) the invariant set is tightened to account for the unknown disturbances starting from the time at which the system receives a new state measurement update. Variables x i,0 andx i,0 are the current state and current state prediction, respectively, which in general have different values, unlike (29). Next, we formulate (30) using the measurement update deadline i defined in (31) and the cost function (32) as lex min where U , is the optimal control policy at the previous time instant, x i,0 is the current measured state, andx i,0 is defined as A ixi (−1) + B i u ⋆,prv i (−1), that is the state prediction at the current time instant. Similarly to (30b), the constraint (33b) is considered to limit the scheduling search space and reduce the difficulty of the problem. Note that while according to the constraints (33b) and (33c) the system i is scheduled to receive a state measurement update in the min( i , i ) following time instants, this update can get postponed later on, since the following state measurements might indicate that the system does not need an update. In the optimization problem (33), the previous optimal inputs are needed, therefore, one could use (30) assuming thatx i (0) ∶= x i (0) at the first iteration to start the online co-design using (33) at the following time instants. This scheme is summarized in Algorithm 4.
Remark 5. Note that in Algorithm 4 the optimal schedule ⋆ is updated inside the communication scheduler at each time k ≥ 1, see Figure 1. At time k = 0, each controller i has access to x i (0) by assumption, and solves the optimization problem (29) to obtain the optimal control trajectory u ⋆ i (0), u ⋆ i (1), … . Each controller i applies u ⋆ i (k) to system i at time k. At each time k ≥ 1, problem (32) is solved inside by scheduler using the information available to the controller. This is necessary in order to define the cost to be optimized in problem (33) which yields the optimal schedule update ⋆ (0). If i ∈ ⋆ (0), then controller i also solves problem (32) with the new initial conditionx i,0 = x i,0 = x i (k) and updates Find feasible schedules for the instance { 1 , … , q } of the WSP and define the set S as in (28) Solve the optimization problem (30) Solve the optimization problem (33) which yields ⋆ (t) and u ⋆ i (t) (32), and update u ⋆ i (t) accordingly end for Pass u ⋆ i (k) to the actuator i, for all i ∈  Remark 9. The proposed joint communication schedule and control design in the optimization problems (30) and (33) can also cover packet loss under the assumption that at most n L i packets could be lost for system i over any time interval of length i . Consider, for example, the case of two systems with 1 = 3 and 2 = 5, and assume that at most one packet may be lost during each four consecutive transmissions. Then we have n L 1 = 1, since 1 < 4; and n L 2 = 2, since 2 < 2 ⋅ 4. Under this assumption for the packet loss distribution, schedulability of (30) and (33) is equivalent to existence of a schedule for instance I = {̃1, … ,̃q} of the WSP wherẽi ∶= i − n L i , for all i. 21 In (30), the strategy consists in updating the schedule such that the systems which lost a packet are immediately rescheduled for communication, that is, the schedule is shifted back by one time instant whenever a packet is lost. Since in the absence of packet losses each controller i is scheduled to receive a state measurement at least once during anỹi consecutive time instants, and this transmission may be postponed at most by n L i time instants in the shifted schedule, one has the guarantee that a measurement is received at least once every i consecutive time instants. In turn, this guarantees that the state remains inside the admissible set.
While the modified version (30) guarantees feasibility also in case of packet losses, it does not account for optimality. Similarly to the case of no packet losses, one can then use a modified version of (33), where is defined using̃i instead of i , and the schedule is updated by solving (33) is the number of packets that can be lost over the time interval [k, k + i − 1], given the packets lost in the past. In order to clarify this last aspect, consider the case of two systems having 1 = 3 and 2 = 5 at a given time instant k; assume that at most one packet may be lost during each four consecutive transmissions, and assume that only one packet was lost at time k − 1. Then we have m L 1 = 0, since having lost a packet at time k − 1 implies that no packet can be lost in the interval [k, k + 2] = [k, k + 1 − 1]. Similarly, one can verify that m L 2 = 1, since at most one packet can be lost in the interval [k + 3, k + 7], which entails that at most one packet can be lost in the interval [k, k + 4] = [k, k + 2 − 1].

NUMERICAL SIMULATION
In this section, we provide several numerical examples to show and compare the effectiveness of the proposed optimal scheduling and control design techniques. In Example 2, first we compare the communication schedule and control obtained from the optimization problems (30) and (33) when the control is computed by a static feedback gain. Then, we compare these designs when the control is optimized as per (32). In Example 3, we increase the disturbance levels and repeat the comparison between (30) and (33) for the optimized control case. In Example 4, we compare the solutions of the optimization problems (30) and (33) when the control is derived to minimize the cost for the worst-case disturbance.
Example 2 (Networked control vehicles). Consider five remotely controlled vehicles, each considered as a point mass with a double integrator model as with where x i (t) contains the position and the speed of the vehicle, u i (t) is the input acceleration, and v i (t) is a matched disturbance.
and the cost function matrices Also assume that only one vehicle can communicate at each time instant, that is, n = 1. We consider zero as the initial state and simulate the NCS for t = 0, … , 500.
First, consider u i (t) = −K ixi (t) as an additional constraint to the optimization problems (29) and (32) where K i is the optimal feedback gain from the corresponding Riccati equation, see Remark 7. In this case, the safe time intervals are and a feasible cyclic schedule for the WSP with instance {4, 5, 6, 9, 10}, is Since 1 has period 8, one can create a set of feasible schedules as  1 ∶= { 1 1 , 1 2 , … , 1 8 }, where we define 1 k ∶= 1 (k), 1 (k + 1), … as a shifted version of 1 for any k ∈  8 1 , such that 1 1 = 1 . In order to investigate sensitivity of the proposed approach to the set of the set of feasible schedules, we also consider 2 ∶= 1, 2, 3, 1, 4, 5, 2, 1, 3, 4, 1, 2, 5, 3, … , as another cyclic feasible schedule with period 14 for the problem and the set of the feasible schedules  2 ∶= { 2 1 , 2 2 , … , 2 14 }, that is the set of shifted versions of 2 . Next, we provide simulation results for five cases: The simulation results are shown in Figure 3. In these simulations, the disturbances are generated as uniformly distributed random values and kept the same in different cases for a fair comparison.
Since the cost functions in (29) and (32) are defined based on the predicted state, in the figure we represent a quadratic cost based on the actual states, that is, This figure indicates that the cost function for the offline schedules, that is, Cases 1 and 3, are much higher than the cost for the corresponding online schedules, that is, Cases 2 and 4. As expected, the cost for Case 5 is better than the other cases since the scheduling limitation is not present in this scenario. However, the performance of the online schedule is not dramatically worse, considering that communication is reduced by a factor 5. The distribution of the absolute value of the states of the vehicles and the inputs show that the constraints are respected; furthermore, it indicates that the performance of the offline and online schedules are not very sensitive to the schedule or the set of schedules that were used in the scheduling design. Now consider the same NCS without considering u i (t) = −K ixi (t) as an additional constraint in the optimization problems (29) and (32), which we call the MPC design. In this case, the safe time intervals are 9, 9, 11, 12}. ( Since the safe time intervals have increased in this case, 1 and 2 are feasible offline schedules for the network. The simulation results for the same cases are provided in Figure 4 where we have considered  i = {0} for all i ∈  5 1 . In this simulation, the disturbances are the same as in the previous one for sake of comparison.
Similar to the Figure 3, the cost function in Figure 4 shows that the cost function for the online schedules, that is, Cases 2 and 4, are improved in comparison to the offline schedules, that is, Cases 1 and 3. Note that the cost function in the MPC cases are rather similar to ones in the constant-gain state feedback cases.
We consider zero as the initial state and simulate the NCS for t = 0, … , 500. The disturbances are generated randomly with uniform distributions and reused in different cases for sake of comparison. Since these safe time intervals are the same as the cases with the constant feedback gain in the previous example, the same schedules are also feasible in this example. Note that in this case, the linear feedback results in lower or zero safe time intervals which results in an infeasible scheduling problem. The simulation results are presented in Figure 5. As in the previous example, this figure shows that the online schedules, that is, Cases 2 and 4, have better performances than the offline schedules, that is, Cases 1 and 3. The increase of the noise levels has increased the costs. One can also observe that the average communication ratio in Cases 2 and 4 for vehicle 1 have increased in comparison to the previous example since the noise level for this vehicle is increased more than other ones.
In the following, we provide an example to illustrate the results in case  i ≠ {0}. Since this optimization problem is inherently difficult to solve, we have selected an extremely simple case which allows us to easily interpret the results. We also note that, since min-max problems are very difficult to solve, for nontrivial examples the infinite horizon optimization problem is often replaced by a finite horizon one. In addition, even for finite horizons, it is customary to rely on formulations which only approximately solve the min-max problem. While a thorough discussion on these methods is out of the scope of this paper, we only remind that any such formulation can be used in combination with our approach. Example 4. Consider a network with three systems described by where x i (t) ∈  i = {x ∶ |x| ≤ 1}, u i (t) ∈  i = {u ∶ |u| ≤ 1}, and v i (t) ∈  i = {v ∶ |v| ≤ v i } with v 1 = 0.4, v 2 = 0.22, and v 3 = 0.18. We consider zero as the initial state and simulate the NCS for t = 0, … , 500. The disturbances are generated randomly with uniform distributions and reused in different cases for sake of comparison. In this example,  i,∞ = X i and 1 = 2, 2 = 4, and 3 = 5. One can verify that 3 ∶= 1, 2, 1, 3, … , is a feasible offline schedule for the optimization problem. Assume that  i =  i and consider  (41), and the last figure shows the distribution of the worst-case cost, as defined in (48). The means of the worst-case cost for Cases 1 and 2 are -6.2291 and −6.7520, respectively as the set of feasible schedules. For simplicity, consider Q i = 1 and R i = 0 for all i = 1, 2, 3; this implies that the optimal control law, using either MPC or LQR, is For any ∈  3 , one can verify thatJ • Case 2: solving (33) when  ∶=  3 ; Figure 6 shows the simulation results for the offline and the online schedules. The worst-case cost in the figure is defined as Note that the quadratic cost in Case 1 is less than the cost in Case 2. This is expected since in Case 2, the online scheduling aims to minimize the cost in the worst-case scenario. The worst-case cost shows that the online schedule is in fact better than the offline one. One can also observe that the online schedule communicates more often with system 1; this results in longer open-loop periods for systems 2 and 3.

CONCLUSIONS
In this paper, in order to jointly design communication schedule and control for constrained NCSs, we formulated an optimization problem with a quadratic cost function. The joint design of the schedule and control prevents unnecessary performance loss of disjoint design schemes. On the one hand, this scheme exploits the periodic side of the designed communication schedule to guarantee satisfaction of the input and state constraints and on the other hand, it adjusts the design based on the state measurements which improves the performance.
One can extend this work by considering a more general network in which both the sensor-controller and the controller-actuator links are constrained and independent. Furthermore, the proposed framework could be extended to the case of systems with coupled constraints/dynamics.