An LSTM‐based approach to precise landing of a UAV on a moving platform

A machine learning‐based method for the precise landing of an unmanned aerial vehicle on a moving mobile platform is proposed. The proposed approach attempts to predict the mobile platform's future trajectory based on the past states of the mobile platform. To that end, it combines a long short‐term memory‐based neural network with a Kalman filter. Hence, it aims at combining the advantages of a machine learning method with those of a state estimation method from established control theory. Based on the predicted trajectory, the unmanned aerial vehicle attempts to land precisely on the moving mobile platform. The experiment is conducted in the Gazebo simulation platform with a quadrotor and an omnidirectional mobile robot, and the proposed method is compared with the single‐method approaches of using only either the Kalman filter or the machine learning method alone.

platform must be gathered and time-series prediction techniques must be utilized. This study assumes that the mobile landing platform follows a certain intention (which is, however, completely unknown to the UAV). Thus, the platform is following a reasonable, although maybe complicated, path rather than moving aimlessly or randomly. This is meaningful for real-world applications.
Numerous strategies are investigated for forecasting the future states of an observed target, such as ensemble methods, 9 Gaussian processes, 10 and restricted Boltzmann machine-based neural networks. 11 Furthermore, according to Ref. 12, even a constant velocity model can quite accurately predict pedestrian movements compared to state-of-the-art approaches. Apart from these approaches, deep neural networks have demonstrated remarkable achievements in these types of applications. Recurrent neural networks (RNNs) show the ability to handle time-series problems, for example, for language translation and speech recognition. 13 However, when their structure is vast and complicated, RNNs suffer from vanishing gradients in back-propagation. To address this issue, the long short-term memory (LSTM) 14 and the gated recurrent unit 15 methods have been proposed, both of which have an information control mechanism to manage the flow of information from the past inputs. Even these deep neural network-based approaches have their advantage, in that they can learn the observed pattern efficiently based on collected data, but the results predicted directly by these approaches lack information about kinematics, which, however, are essential in real-world robotic applications.
The purpose of this study is to develop a method for guiding a UAV onto a moving platform. Because localizing of the landing platform can be impeded by obstacles in the environment or by the effective sensor range, the robust prediction of the mobile landing platform's trajectory is critical to ensure a safe and precise landing.
To make a more reasonable guess of the future trajectory of the moving platform, rather than solely training a neural network to obtain the prediction, the proposed method combines an LSTM network and a Kalman filter (KF) together, where the KF uses the LSTM network's prediction results as the future observation of the moving platform to make a more acceptable prediction. Furthermore, the performance of the proposed method is compared to that of two other approaches in three different scenarios. To the best of our knowledge, this is the first application of this method for guiding a UAV landing on a dynamic landing platform. Additionally, in contrast to our previous work, 16 which predicts a long-term trajectory for a platform based on its past route history and the guessed intended destination, this study focuses on the observed platform's motion pattern to forecast a short-term trajectory and achieve a precise landing maneuver.
The paper is organized as follows: Section 1 describes the methods utilized for the trajectory prediction. In Section 3, the experimental environment and the control scheme are introduced.
Subsequently, in Section 4, several scenarios and the experimental results are presented. Finally, conclusions are presented in Section 5.

| Target modeling and the KF
In this study, an omnidirectional mobile ground robot is utilized as the moving landing platform. Compared with differential-drive mobile robots, omnidirectional robots have more moving flexibility, and are widely used in industry and research. 17,18 Due to the fact that an omnidirectional mobile robot can move in any direction without turning, in the following, it is assumed that the robot maintains its orientation constant. Then, its orientation is irrelevant during landing, and its motion can be described using the state vector x x y x y = [ , ,˙,˙] r r r r r T without rotation, where x r and y r denote the positions of the robot in the xy-plane. The dynamical model of the landing platform can be written as where the time-invariant system and input matrices are defined by A and B, and the control input for the landing platform and Gaussian process noise are denoted as u k−1 and w k−1 , respectively. The system matrix is defined as where the time difference between two consecutive states is denoted as t Δ . Besides, in this study, since the quadrotor cannot directly affect or control the mobile landing platform, the input matrix B is set to be a zero matrix and the input of the landing platform u is neglected. The relationship between the state vector and the measurements from sensors can be described as where the measurement at time step k − 1 is denoted as y k r, −1 with corresponding Gaussian measurement noise v k−1 , and the output matrix C maps the state of the mobile landing platform to the observed output accordingly.
Because the landing platform is not influenced by the UAV, the landing platform's control input cannot be known in advance, and both process and measurement noise are also unknown to the UAV.
As a result, Equations (1) and (3) cannot be used by the UAV for simulating the behavior of the moving platform. It must be understood that the moving platform is moving due to its own intentions and control, whereas these quantities are not available to the UAV. Thus, the UAV can only estimate the future motion of the moving platform.
A KF is utilized in this study to estimate the state of the landing platform. 19 In the prediction phase, the predicted state  x k k r, −1 , the predicted measurement  ŷ k k r, −1 , and the a priori estimate covariance where the process noise covariance is denoted by Q.
where the measurement from sensors at time step k is denoted by y k , and the matrices L k and R are the Kalman gain and the covariance matrix of the measurement noise, respectively. Finally, the estimated is utilized as the state of the mobile landing platform at time step k for further computations.

| Prediction based on LSTM
To land precisely on the moving platform, one must be able to predict its future trajectory, allowing the UAV's approach trajectory to be

| Combination of the LSTM network and the KF
Although the LSTM algorithm can be used directly to predict the trajectory of the moving platform, the most straightforward prediction approach is to use the current state to make an integration for the future. Rather than setting up a fixed velocity model in Ref. 12, in this paper, we estimate the current state using a KF and to calculate estimated future states by integrating the system dynamics with control input set to zero. The KF with subsequent time integration (KF + TI) and the LSTM method perform very similarly in a straightline trajectory, as shown in Figure 3A. The LSTM method outperforms the KF + TI method during turning; see Figure 3B. It is worth noting that the prediction by KF + TI can only use the past measurements indicated by gray plus markers to obtain the KF results, which are marked by orange stars. By contrast, the tendency of the LSTM prediction marked with blue triangles corresponds to the target's actual movement, and it performs better than the prediction marked with brown circles using the KF + TI method.
However, because the LSTM approach does not explicitly consider any a priori knowledge about the dynamics of the robotic platform, it is more likely to generate nonphysical trajectories. For instance, it may introduce unphysical discontinuities, jumping from the original trajectory; see Figure 3. To enhance the prediction performance, this study uses the predicted trajectory from the LSTM model as the incoming measurement resource for the KF. Based on Equation (5), one can update each estimated state of the mobile robot by where y k LSTM, contains the predicted positions of the LSTM network. In Figure 3

| Scenario III
In the final scenario, an asymmetric eight curve is introduced to verify the performance of the three prediction methods. The trajectory is defined by The mobile landing platform starts from the crossing point in the center and follows the designed trajectory strictly as shown in  Table 2.
According to Equation (7) proposed method is evaluated and compared to conventional approaches in the simulation experiment conducted on the Gazebo platform. Three scenarios are set up, and the landing performance of each method is compared. The proposed method ensures that the UAV lands on the platform in all of the tested scenarios and shows relatively consistent performance. In all considered scenarios, the proposed combination LSTM + KF outperforms the other methods. In future work, one may apply moving horizon estimation, which will allow consideration of robotic platforms with nonlinear dynamics as well as constraints on the predicted trajectory. First hardware experiments have been prepared and are soon performed to get further insight how the described approaches work in practice.