Self‐Organized Stick Insect‐Like Locomotion under Decentralized Adaptive Neural Control: From Biological Investigation to Robot Simulation

Living animals and legged robots share similar challenges for movement control. In particular, the investigation of neural control mechanisms for the self‐organized locomotion of insects and hexapod robots can be informative for other fields. The Annam stick insect Medauroidea extradentata is used as a template to develop a biorobotic model to infer walking self‐organization with strongly heterogeneous leg lengths. Body dimensions and data on the walking dynamics of the actual stick insect are used for the development of a neural control mechanism, generating self‐organized gait patterns that correspond to the real insect observations. The combination of both investigations not only proposes solutions for distributed neural locomotion control but also enables insights into the neural equipment of the biological template. Decentralized neural central pattern generation is utilized with phase modulation based on foot contact feedback to generate adaptive periodic base patterns and a radial basis function premotor network in each leg based on the target trajectories of actual stick insect legs during walking for complex intralimb coordination and self‐organized interlimb coordination control. Furthermore, based on both study objects, a robot with heterogeneous leg lengths is constructed to preliminary validate the findings from the simulations and real insect observations.


Introduction
The control of locomotion in both insects and walking robot fields has received considerable attention. [1] Engineers and biologists ? From biological investigation to a biorobotic implementation From a biorobotic model to biological hypothesis testing F FC FC Figure 1. Bio-inspired development of a neural control mechanism for self-organized locomotion of a walking system with heterogeneous leg lengths. The Annam stick insect M. extradentata (left) is used as our biological template for creating our biorobotic model (right). The developed neural control mechanism can generate self-organized locomotion with various gaits. Among generated gaits, we also obtained stick insect-like gaits. Example of comparable (tripod) gaits of the real stick insect and our artificial insect (biorobotic model) driven by the neural control. The study not only proposes distributed adaptive neural locomotion control but also suggests possible key neural ingredients for self-organized locomotion of walking systems with heterogeneous leg lengths, like the one in stick insects. The developed biorobotic model and neural control mechanism can be further used as a tool for testing hypotheses of insect locomotion mechanisms. In the gait diagrams, black areas refer to stance phases (foot on the ground), whereas white areas denote swing phases (foot swinging in the air).
radiation since the Cretaceous period, realizing several different ecological niches. [24] As a result, many phasmids are well adapted to specific environments. This has given rise to different ecomorphotypes which are specialized in different aspects of their life history, [25,26] such as the oviposition of their eggs, [27,28] flight capability, [29][30][31] and the adaptation of their tarsal attachment systems to specific substrates. [32][33][34] These ecomorphotypes are subject to various trade-offs between mobility and other life-history traits, for example, morphological adaptation for escaping predators and hiding. One aspect of the trade-off between mobility and camouflage is the evolution of strongly heterogeneous leg lengths among the three walking legs. Depending on the habitat, body-leg-ratios can differ between the species. [35] The role of legs in different ecomorphs has been researched to some extent, most importantly in regard to the subfunctionalization of locomotor apparatus components. On the one hand, Theunissen et al., [35] compared the different walking dynamics resulting from the variations in leg and segment lengths between the three species with different body morphologies, while on the other hand, the motion ranges of both legs and antennae also differ. Some species with particularly elongated front legs have short antennae instead, and their front legs partially replace the sensory function of the antennae. [35] The active motion ranges of antennae and legs correlate with each other and the reduction of antennal length is compensated by front leg elongation and the increasing exploration effort of the legs. [36] The secondary functionalization of legs for sensing the environment is shown to have further implications, with the tactile exploration of the environment and walking performed simultaneously. Stick insects adapt their leg dynamics and single-step coordination of the six legs to various influences at once. [14,16,37] Besides adjusting the step coordination for tasks like gap crossing, [17,[38][39][40] overcoming obstacles, [41][42][43] dealing with substrate inclination, [44][45][46] and other perturbations, [47,48] it is noteworthy, that the dynamics of the stepping patterns differ according to the body morphometrics, [35,36] adjusting for the loss of legs. [49] Overall, the control of leg used in stick insects is shown to be highly adaptive with some effort spent on understanding the adaptive leg-joint coordination of single steps (i.e., intralimb coordination).
The control of locomotion is highly adaptive in stick insects [10,[50][51][52] and partially based on load-controlled proprioception. Studies on the neural control of motion in stick insects with thoracic connective lesions have shown that, in principle, cut thoracic connectives had an effort on the leg coordination of unrestrained walking stick insects. However, in general, mechanical coupling between the legs compensates for missing neural information. [53] This adaptive response of single legs for interlimb coordination is facilitated by the campaniform sensilla at the leg base, which provide proprioreceptive feedback for each leg. [53,54] This distributed (decentralized) control architecture enables adaptive control based on feedback from the legs themselves [55,56] and facilitates a smooth transition between the different walking gait patterns required in the respective situations. [50][51][52]57,58] Different approaches have made use of the knowledge obtained on the integration of stimuli for locomotion control from stick insects (reviewed in ref. [10]). The role of elongated forelimbs for near-range exploration has been highlighted in several studies. [35,36] However, the heterogeneous www.advancedsciencenews.com www.advtheorysimul.com length across different leg pairs is likely to have implications on the coordination of different legs and their adaptive distributed neural control.
To the best of our knowledge, although stick insect locomotion has been extensively investigated, an adaptive neural control mechanism for self-organized stick insect locomotion with heterogeneous leg lengths [59] has not been realized before (see Section 4 for more details on existing locomotion control mechanisms). To address this issue, we performed an integrative study combining biology and robotics. We employed videographic analyses of the walking dynamics of a real stick insect species with distinctly heterogeneous leg lengths to investigate the step relations of the different leg pairs and the intralimb coordination during walking as a source of information for our further biorobotic modeling and as the basis for developing adaptive neural control for self-organized locomotion in walking systems with heterogeneous leg lengths ( Figure 1).
The rest of this article is organized as follows. Section 2 is based on the study of stick insect locomotion, followed by the selforganized stick insect-like locomotion pattern under distributed adaptive neural control. Section 3 contains the experiments and results of stick insect-like robot simulation and preliminary results of a real stick insect-like robot. Section 4 provides discussion and conclusion.

Stick Insect Locomotion
The experimental use of stick insects did not require ethic approval nor was subject to any specific regulations. All applicable rules and regulations were followed. To analyze the walking dynamics of stick insects with heterogeneous leg lengths, the lengths of the body segments and the corresponding weights  (Figure 2) were recorded. The animals were placed on a flat vertical glass plate and filmed from the lateral view, while a mirror was placed in the background at an angle of 45 • to capture the dorsal aspect simultaneously. Walking was initiated voluntarily by the animals without external stimuli. Trials were randomized to minimize potential changes in the walking behavior caused by learning effects. A DSC-RX10M3 camera (Sony, Minato, Japan) was used at 250 fps to film five walking sequences per insect. Only undistorted sequences in which the insects walked without stopping or turning were analyzed later. Statistical analyses on both morphometrics (Tables 1-3) and the walking dynamics of the stick insects were carried out using SigmaPlot 12.0 (Systat Software Inc., San José, CA, USA). Prior to analysis, the data were tested for normal distribution and equal variance using Shapiro-Wilk's test and Levene's test, respectively. In the event of both normality and homoscedasticity, one-way analysis of variance (ANOVA) was used. Otherwise, the data was tested using Kruskal-Wallis ANOVA on ranks. A comparison of Figure 2. Locomotion of stick insects M. extradentata. a) Captured frame from the video footage showing the leg trajectories relative to the body of the front leg (gray), middle leg (green), and hind leg (red) for one step sequence in dorsal view. b) Leg trajectories for one step sequence in lateral view. c) Example of gait patterns observed. Black bars indicate stance phases and white bars indicate swing phases of the respective legs. R1, right front leg; R2, right middle leg; R3, right hind leg; L1, left front leg; L2, left middle leg; L3, left hind leg. d) Duration of stance (black) and swing (white) phases for the three leg pairs. FL, front legs; ML, middle legs; HL, hind legs. The front leg (FL) was the longest, followed by the hind leg (HL) with the middle (ML) being the shortest. The strongest influence on the total length of the legs was found in the tibia, which revealed the largest difference among the leg pairs (Table 3). Gait patterns and dynamics were analyzed using the tracking method described in ref. [60]. The pretarsi of every leg were manually tracked in the video footage using Adobe After Effects (Adobe Inc., San José, California) and extracted with a customized Java Script. [60] The obtained coordinates were used in R (Version 4.1.2; The R Foundation for Statistical Computing, Vienna, Austria) to visualize the walking gait pattern ( Figure 2) and analyze the duration of the stride and stance phases for the steps. The difference in the coordinates between two frames of the tracked footage per leg was used to calculate the foot velocity over time (see ref. [60]) to measure the duration of both stance and stride phases.
The unrestrained walking stick insects primarily showed tripod gait patterns, elevating three legs at once. A few transitions to tetrapod gaits, where just two legs moved at the same time (see ref. [49]), were observed within the sequences. The steps of the leg pairs were pooled per thorax segment. This resulted in ≈200 steps per leg pair for the analyses (N = 5, with each n = 5 sequences). The stance duration of ML (0.55 ± 0.04 s; mean ± SD) was observed to be the longest compared to HL (0.51 ± 0.03 s) and FL (0.50 ± 0.03 s). The stance duration of FL was the shortest. The average stance durations of all three leg parts were significantly different from each other (Kruskal-Wallis-ANOVA, n FL /n ML /n HL = 200, d.f. = 2, H = 116.428, p ≤ 0.001; Tukey's post hoc test, p < 0.05). Coincidentally, the stride durations were shortest for ML (0.16 ± 0.02), longest for FL (0.25 ± 0.04), and intermediate for HL (0.18 ± 0.02). All differences were statistically significant (Kruskal-Wallis-ANOVA, n FL = 202, n ML = 200, n HL = 199, d.f. = 2, H = 407.645, p ≤ 0.001; Dunn's post hoc test, p < 0.05).

Biorobotic Model of a Stick Insect M. extradentata
Our biorobotic model was a simulated stick insect robot [61] (Figure 3) based on the data from a real stick insect M. extradentata (Tables 1-3). Specifically, the robot shared the same morphometrics of leg and body dimensions, leg and body mass, and legbody relationships as the stick insect with a factor of five. This was to obtain a reasonable size to avoid simulation errors. To simplify the robot structure, the body was created in three segments (caput or head, thorax, and abdomen) which were fixed to each other, and each leg was also divided into three segments (coxa+trochanter, femur, and tibia+tarsus segments) with three joints (thoraco-coxal (TC-), coxo-trochanteral (CTr-), and femorotibial (FTi-) joints). For the sake of simplicity, each leg joint axis was arranged orthogonally. The tarsus and tibia were combined into one segment with the length equal to the total length of the tibia and tarsus. By doing so, computational effort could be reduced and ensured that the robot foot can reach all foot trajectory points obtained from the insect tracking data. All the leg segments were modeled as cylinders with a diameter of 15 mm, which was large enough to avoid a simulation error. Each joint was equipped with a joint angle sensor for monitoring the joint position and each leg had a foot contact sensor at the leg tip for detecting the ground reaction force. Here, a Gaussian-distributed noise was added with a standard deviation of 5% for all sensory information. The biorobotic model was used here to investigate and develop decentralized adaptive neural control for selforganized stick insect-like locomotion (described below). The update frequency of the simulation was 20 Hz.

Decentralized Adaptive Neural Control for Self-Organized Stick Insect-Like Locomotion
Here, a decentralized adaptive neural control that can generate self-organized locomotion of the aforementioned biorobotic model (i.e., simulated stick insect robot) was introduced. The underlying principle of this control approach was the use of neural dynamics, nonlinear neural transformation, neural plasticity, proprioceptive feedback (i.e., load sensing/foot contact sensor feedback), and robot body dynamics to adaptively coordinate robot limbs (known as adaptive interlimb coordination), resulting in insect-like locomotion. The proposed control system was based on a modular architecture and consists of four subneural modules ( Figure 3): 1) neural central pattern generator (CPG)-based control with neuromodulation and foot contact feedback for generating and adapting basic periodic patterns; 2) a premotor neural network for shaping the CPG periodic patterns to obtain stick insect-like leg movements (intralimb  Figure 3. The simulated stick insect robot with decentralized adaptive neural control. Each leg is independently driven by a modular neural control system consisting of four sub-neural modules ((I) a CPG-based control network, (II) a premotor network, (III) a forward model, and (IV) a dual-rate learner (or dual-rate learning mechanism)). The CPG is formed by two recurrent neurons (N 1,2 ) with a modulatory input neuron (MI). The premotor network is formed by multiple radial basis neurons (R 1,…,n ). The outputs of the premotor network are projected to three motor neurons (M 1,…,3 ) and one forward model neuron (FP). The motor neurons transmit joint angle commands to finally control the thoraco-coxal (TC-), coxo-trochanteral (CTr-), and femoro-tibial (FTi-) joints for position control. The output of FP predicts the foot contact signal which is further shaped by two postprocessing neurons (P 1,2 ) before comparing it with the actual foot contact signal at a linear comparator neuron (E). The foot contact sensory neuron (FC) receives a continuous foot contact signal. The signal is preprocessed at a sensory preprocessing neuron (SP) and compared with the predicted foot contact signal from the forward model. The difference in the comparison leads to an error which is then used in the dual-rate learner for sensory feedback strength adaptation. All neurons of the control network are modeled as discrete-time non-spiking neurons. The top view of the robot and a zoom image of a leg are shown on the left and right, respectively. coordination); 3) a neural forward model for predicting foot contact feedback; and 4) a dual-rate learning mechanism for continuously adapting foot contact feedback strength (called online sensory feedback adaptation). Each of which is described in detail below. In this setup, each leg was driven by an identical control system (i.e., one neural CPG-based control, one premotor neural network, one forward model, and one dual-rate learning mechanism). As a consequence, controlling the stick insect robot with six legs required six decentralized neural control systems. To provide the flexibility of adaptive interlimb coordination, any connection or coupling was not defined between the neural control systems (i.e., they are decoupled or have no direct neural communication). Instead, the coordination among them was mainly achieved by the interaction between the robot and the environment, resulting in self-organized locomotion where emerged tripod and non-tripod walking gaits could be obtained (see Section 3).
Previous studies have shown the success of self-organized locomotion control with load sensing feedback for different bio-inspired legged robots. [63][64][65][66] They demonstrated that it was possible to use foot contact feedback to adapt the phase between the legs of a walking robot to form stable interlimb coordination. The feedback-based phase adaptation eventually forms a stable gait, allowing the robot to walk. However, until now, the use of this approach had not been demonstrated for a legged robot with heterogeneous leg lengths (i.e., different lengths of front, middle, and high legs based on the morphometrics of a real stick insect). Having such different leg lengths required more complex signals for coordinating the leg joints to achieve stable intralimb coordination. The solution proposed here involves the nonlinear neural transformation of a premotor network that could translate the CPG signals into motor command signals to reproduce stick insect leg trajectories (i.e., intralimb coordination). For example, each robot leg here included three joints, allowing the movement of its tip with respect to the corresponding stick insect leg trajectory for self-organized locomotion (see below).

Neural CPG-Based Control with Neuromodulation and Foot Contact Feedback for Periodic Pattern Generation
The model of the CPG-based control circuit was realized using the discrete-time dynamics of a simple two-neuron recurrent network with neuromodulation (Figure 4a). It produced two periodic signals ( Figure 4B) which were further shaped by a premotor network (see below) to obtain final motor commands for driving the leg joints (TC-, CTr-, and FTi-joints). An extrinsic modulatory input MI (Figure 4a) was used as neuromodulation [67] to modulate the CPG frequency and project foot contact feedback FC ( Figure 3) to the CPG inputs S 1,2 ( Figure 3) to automatically and continuously adjust the CPG phase online for adaptive interlimb coordination. This technique could produce the appropriate CPG phase shifts between the legs, enabling the robot to achieve selforganized stable gaits. The neurons (N 1,2 ) of the circuit were modeled as non-spiking neurons. The activity of each neuron developed according to where w 11,22 are the self-connection weights of N 1,2 and w 12,21 are the connection weights between N 1,2 . S 1,2 are the CPG inputs, and o N i are the CPG outputs. is a plastic CPG input connection (synaptic plasticity). It is automatically adjusted by dual-rate learning (described in detail in the following section). Based on previous study [68] where FC is the negative continuous foot contact feedback at the leg. The sine and cosine functions of the neural activities a 1,2 were used to derive a proper correlation between the neural activity and feedback. The functions were related to the phases of the CPG outputs o N 1,2 which differ by ∕2 (see Figure 4b). The strength of the sensory feedback connection could be adapted to regulate the amount of sensory feedback to the CPG-based control. Through this connection, the foot contact sensory feedback could slow down the leg speed when highly loaded at the end of the stance phase, while increasing the speed of the leg trajectory when it was unloaded at the end of the swing phase. This allowed the robot to adaptively adjust its leg movement to form stable gaits with good body weight distribution. When implementing the neural control systems on different legs, the proper value of needed to be used to achieve stable locomotion and this value might have to be changed with respect to certain conditions (e.g., different walking speeds).
Predetermining an optimal value for all cases was timeconsuming and impractical. Thus, a dual-rate learning mechanism was applied as an automatic process for continuously and dynamically adjusting the value online (see below). Under this control scheme, there was no fixed and predefined interlimb coordination, but rather a flexible one, since the gaits obtained were derived from foot contact feedback, synaptic plasticity, neural activities, and body-environment interaction.

Premotor Network for Intralimb Coordination
For intralimb coordination (i.e., joint coordination) of each robot leg, the CPG outputs were projected to the TC-, CTr-, and FTijoints indirectly through the premotor neural network. The network shaped the periodic CPG patterns to finally generate robot foot trajectories following the stick insect foot trajectory data recorded during walking (see Section 2.1). A feedforward neural network with radial basis activation functions (i.e., radial basis function (RBF) network) was used as the premotor network. The RBF premotor network consisted of three layers (input, hidden, and output, Figure 3). The CPG network was integrated with the RBF premotor network (called the CPG-RBF network). In other words, the two CPG outputs represented the two RBF inputs which were further transmitted to the RBF hidden neurons with 2D radial basis or Gaussian activation functions (R 1,…,n , Figure 3). The neural activity of each RBF hidden neuron is given by , n ∈ 1, … , N (6) where n,0 and n,1 are two means of the RBF neuron. 2 is the standard variance for the two means, empirically set to 0.04 for all neurons. n is the activity of the RBF neuron driven by the CPG output signals o N 1,2 . The two means were empirically set in such a way as to equally distribute the RBF neurons along one period of the CPG output and target signals. N is the total number of RBF hidden neurons (here, N = 40, which is set to maintain sufficient signal complexity but not too large for computation [69] ). The output signals from the RBF neurons were linearly combined at four linear output neurons as w jn represents the weights used to shape and combine the transmitted signals between the RBF hidden and output neurons. The first three output neurons were used as three motor neurons (M 1,…,3 , Figure 3) for controlling the TC-, CTr-, and FTi-joints and the last output neuron as a forward model neuron (FP, Figure 3) for predicting foot contact feedback. This RBF premotor network structure was the same for all six neural control systems controlling the six legs of the robot. The output weights w jn were trained offline using a delta or error-based learning rule as follows w jn n (t)), j ∈ 1, … , 4, n ∈ 1, … , N The x-z plane trajectories of a stick insect (lateral view shown in Figure 2b), which expressed swing and stance durations, were the only ones used in this study for the sake of simplicity. The x-y plane trajectories (dorsal view shown in Figure 2a) were simplified as straight lines. The foot trajectories were first preprocessed, and the preprocessed trajectories (Figure 5a,b) were then applied to obtain the robot joint space. Here, the planned joint positions (T j ) served as the target signals (Figure 5c-e) for training the RBF network. is the learning rate, empirically set to 0.1. This training strategy followed our previous study. [73]

Forward Model and Dual Learning for Sensory Prediction and Adaptation
In order to achieve online sensory feedback adaptation (i.e., adapting the foot contact feedback strength , see Equations (1) and (2)) for stably forming interlimb coordination and adapting locomotion, a neural forward model and the dual-rate learning algorithm were implemented. [65] The forward model basically shaped the predicted foot contact feedback further (i.e., the output signal o FP of the FP output neuron of the RBF network) through its two postprocessing neurons (P 1,2 , Figure 3). The P 1 recurrent neuron acted as a low-pass filter neuron while the P 2 threshold neuron converted the filtered signal into the final discrete expected foot contact signal o P 2 for comparison with the real foot contact signal o FC . The simple forward model is given by o P 1 represents the output of the first low-pass filter neuron P 1 and o P 2 the output of the final threshold neuron P 2 (i.e., expected foot contact signal). The discrete expected foot contact signal should be zero when the leg is in the swing phase (in the air) and a high positive value when the leg is in the stance phase (on the ground).
In other words, a positive foot contact sensor value was expected when the leg touches the ground during the stance phase (downward position of the CTr-joint) and a zero value when the leg was in the air during the swing phase (upward position of the CTrjoint). is a factor in the range of [0, 1], shaping the output signal of the forward model. is the threshold for converting o P 1 into a simple discrete expected foot contact signal. In this study, and were empirically adjusted and set to 0.3 and 0.005, respectively. The forward model was applied to each leg. The difference between the forward model output and its corresponding discrete foot contact feedback o FC obtained from the foot sensor is calculated as: The difference e was further used to automatically and continuously tune the sensory feedback strength (Equations (1) and (2)) through dual learning (described below). Figure 6 shows the signals involved in the forward model before adaptation and during online sensory feedback strength adaptation. One can observe how the actual sensor value changes, providing a better match

R3
R2 R1 (a) Figure 5. Robot intralimb coordination. a) The simulated stick insect robot with its foot trajectories. b) The foot trajectories of the right hind, middle, and front legs. The left foot trajectories are the same as the right ones. The robot foot trajectories are generated from the filtered 2D foot trajectories in the x-z plane of a walking stick insect. It should be noted that the trajectory of the middle leg slightly differs from the original insect data to avoid collision between the middle and hind/front legs. c) The TC-joint movements of the right legs. d) The CTr-joint movements of the right legs. e) The FTi-joint movements of the right legs. Gray and white areas depict the stance phase and the swing phase, respectively.  with the predicted signal generated by the forward model. The changes were due to the robot-environment interaction and sensory adaptation. The sensory feedback strength transmitting foot contact feedback to the decoupled CPG circuits had a significant impact on the appropriate modulation of the CPG phase in order to generate self-organized stable locomotion and further adaptation. Setting the feedback strength too high might result in too strong inhibition, preventing the robot's movement, while setting it too low might result in an unnecessarily long adaptation period. [65] To automatically adjust the feedback strength online, a dual-rate learning algorithm was applied. [65] This learning mechanism comprised two parallel learning mechanisms (fast and slow learning) described as follows where f and s are the outputs of the fast and slow learners, respectively. The outputs were combined to automatically adapt in Equations (1) and (2) (sensory feedback strength adaptation). e is an error from Equation (11). B f and B s are the learning rates of the fast and slow learners. A f and A s are retention factors. The parameters were set as A s > A f and B s < B f . The fast-learning mechanism adapted more rapidly as indicated by a higher learning rate, but also forgets more rapidly as indicated by a lower retention factor. In contrast, the slow learning mechanism adapted slowly but had a longer memory. Combining the two mechanisms led to fast and stable adaptation of the sensory feedback strength. In this study, A s = 0.998, A f = 0.57, B s = 0.0004, and B f = 0.006. These values are used based on ref. [65]. The analysis of the learning mechanism can be seen at ref. [65].

Experiments and Results
This section presents the results of the experiments with the simulated robot (Figure 5a). The performance of the proposed neural control system was evaluated from two perspectives. The first experiment accessed the intralimb coordination where the foot trajectories and the duration of swing and stance phases under different CPG frequencies were observed (Figures 7 and 8). The second experiment was performed to evaluate the interlimb coordination and adaptation to different walking speeds. During the second experiment, we also monitored the robot gaits and measured the robot walking speed (Figures 9 and 10). The last experiment was carried out to compare the robot locomotion performance under the stick insect-like and simple (typically used) foot trajectories (Figure 11). Figure 7 shows that when the CPG frequency was raised by increasing the MI value, the RBF premotor network could temporally scale or adapt the TC-, CTr-, and FTi-joint patterns (i.e., adapting intralimb coordination) without necessitating a change in network output weights. [74] This results in a change in foot trajectories and swing and stance duration. As the CPG frequency (stepping frequency) increases, the swing and stance duration decreases (i.e., the foot trajectories become smaller). Figure 8 shows the swing and stance duration of different right legs at varying CPG frequencies. The control network generates intralimb coordination, with the hind and middle legs performing a much shorter swing and longer stance than the front leg. On average, the swing and stance duration of the middle leg is shorter and longer than that of the hind leg. This result for the robot is consistent with that of the stick insect (see Figure 2d) where the front leg has the longest swing duration (shortest stance duration) followed by the hind leg with the intermediate swing and stance duration and the middle leg with the shortest swing duration (longest stance duration). Figures 9 and 10 show the results for the robot walking on flat terrain at different CPG frequencies by varying the MI. As can be observed, the control approach could adapt the sensory feedback strength and generate self-organized locomotion within ≈15-20 s. Due to the self-organized interlimb coordination mechanism and robot dynamics, various gaits were observed during locomotion at different frequencies. The observed gaits included different types of gait patterns with simultaneous use of three legs, irregular gaits, a tetrapod gait, a tripod gait, and a direct-wave gait. In fact, the generated robot gait continuum can be described as a free gait. [75,76] Figure 11a shows a comparison of robot locomotion performance in terms of walking distance under a traditional simple (uniform or homogeneous) foot trajectory and our complex (heterogeneous) stick insect-like foot trajectories (Figure 11b). For the simple trajectory, we handcrafted the joint patterns (i.e., intralimb coordination) of all legs to obtain a semicircular foot trajectory where the stance phase follows a straight profile and the swing phase follows an arch profile. The two different foot trajectories were used as the basis for robot intralimb coordination while interlimb coordination was achieved by our self-organized locomotion process. We also tested this with different walking  frequencies (i.e., different MI values). As can be observed, our stick insect-like foot trajectories can lead to better locomotion performance than the simple trajectory that is, the robot could walk for longer distances under varying walking frequencies.
In addition to the robot simulation, we also performed a preliminary test on our real stick insect-like robot (named Black Mirror, Figure S1A, Supporting Information). The robot was built and scaled in accordance with the morphometrics of a real stick insect (described above). Rubber material was used for its feet. Figure 12 shows the result of the real robot walking on flat terrain at different CPG frequencies. Due to the hardware limitations, we tested it using very low and low CPG frequencies (i.e., MI = 0.05 and 0.10). As can be observed, the robot was able to autonomously form its gaits under the different frequencies within ≈ 30 s. During the adaptation or gait formation process, different random turns were observed. This is due to several factors, including I) the adaptation mechanism, which tries to inhibit a leg with an active foot contact signal during a swing phase (any leg can therefore become a pivot point); II) random robot body dynamics (or body oscillation); III) the imperfect robot structure; and IV) physical constraints of the experimental setup. After the adaptation, various gaits, including gait patterns found in the stick insect as well (tetrapod and tripod gaits) and irregular gaits, were observed during locomotion. However, the gaits differed slightly from those observed in the simulated robot (see Figures 9 and 10) due to the different system dynamics. It is worth noting that the rubber feet we used were designed to allow for slight slipping, which was advantageous for our case. This is because during the initial period of gait adaptation all legs move in the same phase. As a result, they cannot swing above the ground during the swing phase, and hence become resistance or pivot points contributing to non-straight walking. Thus, if the foot firmly attaches to the floor or has high friction, it may result in high mechanical stress, leading to damage, as well as significant non-straight walking ( Figure S1B, Supporting Information).

Discussion and Conclusion
Our real stick insect investigation shows that the length differences between the leg pairs of the actual stick insects enable the single legs to be used differently. Front legs can be used for tactile exploration in addition to walking and body weight support. [36] In the species used in our experiments, the sensing function of the antennae is strongly reduced compared to other species. [35,36] This enables the legs of the insects to receive immediate feedback on the structure of the walking path, without feedback from the antennae and to immediately adapt to the entire walking control. [55,56] However, both the trajectories of the legs and dynamics of the swing and stance phases may be subject to a more complex control compared to equally elongated legs to account for the heterogeneous leg lengths. Previous research has shown that tactile exploration is a major task for locomoting stick insects and the strongly heterogeneous leg lengths may be interpreted as a natural result of this front leg subfunctionalization. [35] Strongly elongated front legs are often found in species with comparably short antennae since they take over the function of the antennae. [36] We did not observe extensive tactile exploration of the nearer environment in our experiments. However, the stick insects used herein were not subjected to changes in the near-range environment, with no inclines, [44][45][46]49] obstacles, [41][42][43] or restricted walking paths. [77,78] However, in contrast, the animals were allowed to employ straightforward walking movements in unrestrained locomotion. The key difference to previous findings involving bimodal step length distribution [79,80] is that the extended exploration time of the front legs [16] or higher elevated front legs [14,37] is likely to result from the allowance of undisturbed forward walking. Specifically, short steps leading to a bimodal distribution of step lengths and repetitive steps of the same legs leading to increased step occurrence of short duration and repositioning of the same foot were almost absent.
The walking dynamics of insects might be influenced by the substrates on which the animals locomote. In stick Figure 9. Self-organized locomotion under low CPG frequencies. a) Sensory feedback strengths during simulation on the left and right legs. b) The change in the robot walking speed during the simulation. c) The swing and stance phases of each robot leg monitored from the foot tip. Black areas refer to stance phases (foot on the ground), whereas white areas denote swing phases (foot swinging in the air). Colored marks are used to visualize the main observed gaits including a direct-wave gait (the left and right legs swing together with the swing movements propagating from posterior to anterior) as observed in millipedes [64] and a tripod gait (the front and hind legs on the same side, as well as the middle leg on the opposite side, swing together, while the remaining three legs support the body) comparable to the gait observed in insects. Different colors reflect the various swing periods. Green references the first swing followed by yellow, red, and blue. Videos showing examples of self-organized locomotion can be viewed at www. manoonpong.com/SelforgLoco/VideoS1.mp4 for very low CPG frequency (MI = 0.05) (or see Video S1, Supporting Information), www.manoonpong. com/SelforgLoco/VideoS2.mp4 for low CPG frequency (MI = 0.10) (or see Video S2, Supporting Information).
insects, [18,81,82] the quality of the substrate influences the forces the animals produce to stay attached to it. This can correlate with the forces the insects are able to exert on the substrates for locomotion, such as the generation of propulsion. It has been shown that the walking gait patterns deviate from tripod gait according to the orientation to the substrate [83] and the foothold on the respective substrate. [20] When walking on the ceiling, various insects employ different walking gait patterns depending on their attachment performance. The safety factor (i.e., attachment force per body weight) also predefines how many legs can be detached from the substrate at the same time during walking. [84][85][86] Insects with lower attachment ability can lift comparatively fewer legs from the ceiling during walking and therefore switch from tripod (three legs in contact) to tetrapod (four legs in contact) or wave (five legs in contact) gaits to compensate for the reduction in attachment performance. It is also expected that the feedback received by the animals about the attachment ability during walking on certain substrates has an effect on the behavior. It would be interesting to examine the relationship between substrate quality and walking dynamics in regard to the attachment to the substrate in future studies.
Inspired by stick insect locomotion and employing our biological data, we successfully developed neural control for creating complex intralimb coordination and forming selforganized interlimb coordination in a stick insect-like robot with different front, middle, and hind leg lengths. The stick insect leg trajectory data in the x-z plane during walking were used to obtain our training data for the CPG-RBF-based individual leg control network to achieve intralimb coordination with swing and stance profiles and a duration comparable to the stick insect. To yield self-organized and flexible interlimb coordination, a decentralized neural control architecture with continuous CPG phase modulation was applied. Under this architecture, each leg is independently controlled by its own CPG-RBF control network and the interlimb coordination emerges from continuous dynamical embodied interactions between the body dynamics, neural con- The change in robot walking speed during the simulation. c) The swing and stance phases of each robot leg monitored from the foot tip. Black areas refer to stance phases (foot on the ground), whereas white areas denote swing phases (foot swinging in the air). Colored marks are used to visualize the main observed gaits, including different versions (V1-V3) of tripod-like gaits (three legs swing together while the remaining three support the body) and a tetrapod gait (at least four legs remain on the ground and diagonal pairs of legs swing roughly together). The tripod-like gait V2 is comparable to the tripod gait observed in insects. Different colors reflect various swing periods. Green references the first swing followed by yellow, red, and blue.
Videos showing examples of self-organized locomotion can be viewed at www.manoonpong.com/SelforgLoco/VideoS3.mp4 for high CPG frequency (MI = 0.15) (or see Video S3, Supporting Information), www.manoonpong.com/SelforgLoco/VideoS4.mp4 for very high CPG frequency (MI = 0.20) (or see Video S4, Supporting Information).  b) The snapshots and gait diagram illustrate robot walking behavior at a low CPG frequency (MI = 0.10). The swing and stance phases of each robot leg are monitored from the foot contact sensors. Black areas refer to stance phases (foot on the ground), whereas white areas denote swing phases (foot swinging in the air). Colored marks are used to visualize the main observed gaits, including tripod, tetrapod, and irregular. Different colors reflect various swing periods. Green references the first swing followed by yellow, red, and blue. It should be noted that the data of these walking experiments are extracted from long-period testing (see Video S5, Supporting Information, www.manoonpong.com/SelforgLoco/VideoS5.mp4). An example of the foot trajectories of the front, middle, and hind legs of the robot can be seen in Figure S2, Supporting Information. trol, and environment through minimal foot contact feedback information with sensory adaptation. Through the dynamical embodied interaction process, various emerged gaits, including tripod and non-tripod walking gaits, were observed (see Figures 9,10,and 12). In principle, the neural control architecture with foot contact sensory feedback and adaptation can also function for self-organized locomotion and adaptation on irregular terrain with a certain level of roughness (see ref. [87] for an example of self-organized quadruped locomotion on irregular terrain). One can also extend the control to achieve curve walking by regulating the magnitude of the motor neuron output of each TC-joint through an additional scaling factor. The scaling factor can be adjusted online to increase or decrease the TC-joint magnitude (o RBF 1 in Equation (7)). For example, to steer the robot for curve walking to the left, we can simply reduce the scaling factor values of all left TC-joint magnitudes and vice versa for curve walking to the right. The steering mechanism is not shown here but see ref. [87] for the implementation and results. While many others have studied self-organized locomotion control using foot contact feedback with continuous CPG phase modulation [65,88,89] or discrete CPG phase resetting, [64,90] the robotic models employed in such research typically have similar leg or limb architecture (i.e., equal leg length) and their intralimb coordination was predefined using a simple specific uniform leg trajectory with a circular or semicircular profile for all legs. Conversely, as shown here and in previous biological experiments, [91] insect walking data demonstrates that leg trajectories are more complex than the simple profile. Each leg has a unique profile. For example, the front leg has a long, asymmetric swing profile (slowly ascending and quickly descending) and a shorter stance profile, and the middle leg has a smaller asymmetric swingstance loop, slowly ascending and quickly descending in similarity to the front leg, while the hind leg has also a small loop but an opposite profile to the middle and front legs (slowly descending and quickly ascending) as shown in Figure 2. From this point of view, individual leg control requires a premotor network(s), acting as a nonlinear transformation tool, for complex intralimb coordination and leg trajectories. This can be achieved by employing different approaches.
For example, Cruse et al. [50,92,93] proposed WalkNet control with two separate recurrent (premotor) networks (one for swing control and the other for stance control). The swing and stance networks were optimized using a random search procedure with the Euclidean distance evaluation to produce the stick insect-like leg movement with respect to the biological experiments. [94] In the WalkNet control method, the interlimb coordination was realized by six predefined rules and a state selector network, switching between swing and stance control and forming gaits. The rules were obtained from behavioral experiments on stick insects, relying completely on proprioceptive feedback information (like joint angle and foot contact). Since the interlimb coordination is achieved by a predefined sensor-driven, rule-based, coordination method, [95] it is not entirely self-organized and a lack of proprioceptive input may result in system failure or unstable locomotion. However, on the other hand, this control method adapts immediately to local disturbances and different gaits. [51,52] Instead of employing two separate swing and stance networks with an extra selector network and behavior control rules as in WalkNet control, both complex swing and stance profiles can be encoded using a single premotor network. Different premotor network models for this purpose have been introduced, for instance, a recurrent premotor neural network model with long short-term memory (LSTM) activation functions, [96] a feedforward premotor neural network model with hyperbolic tangent activation functions (tanh), [68,97] radial basis or Gaussian activation functions (RBF) [70,73,98] (as also used here), or a deep premotor neural network model with tanh activation functions. [99] Under this strategy, the interlimb coordination is typically done by predefining the phase relationship between the legs for specific gaits, [68] applying machine learning techniques (e.g., proximal policy optimization (PPO), [96] probability-based black-box optimization (PIBB), [70] or an evolutionary algorithm [97] ). Predefined specific gaits may reduce gait flexibility and adaptability, whereas gait learning, which can result in adaptability, typically requires a long and complex learning process (longer than self-organized control methods with phase modulation or phase resetting, [64][65][66][88][89][90] and even up to many hours or days). This machine-learning-based approach thus becomes impractical.
In this study, we utilize the key ingredients of the different approaches, I) self-organized locomotion control with continuous CPG phase modulation, II) biological data-based intralimb coordination with a single premotor network for each leg, to create a framework for generating self-organized locomotion with complex intralimb coordination in a hetero legged robot. The RBF premotor network is trained by using supervised or error-based learning where the target trajectories are obtained from real stick insect leg trajectories during walking. Due to the overlapping radial basis (non-monotonic) function kernels of the RBF network, continuous signals can be generated, even if the CPG frequency varies (see Figure 7). This is an advantage over a feedforward neural network with a monotonic activation function, such as tanh, utilized in ref. [97]. The RBF network with an embedded temporal scaling feature can shape the CPG signals and further rescale the pattern online with respect to CPG frequency change. Furthermore, with the RBF network, different optimization algorithms, either simple or complex, can be applied to train the network. Here, however, we only employ a simple error-based learning rule with an imitation learning strategy, as opposed to the more complex objective function-based learning strategy. [70,97] In addition to the CPG-RBF network for each leg, a forward model is integrated into the control network for automatic sensory feedback strength adaptation. Basically, this forward model with a dual-rate learning algorithm fine-tunes the synaptic strength of sensory feedback as a slow time scale adaptation while a faster time scale adaptation facilitates the gait formation process through robotenvironment interaction. Further analysis of these different time scale adaptations and their interactions can be seen at ref. [65]. Taken together, this study demonstrates how to exploit biological data for complex intralimb coordination and use the embodied robot-environment interaction principle for gait continuum in robot locomotion under decentralized CPG-based control with minimal feedback (only foot contact feedback).
While this approach shows effective results, it still has limitations. So far, we have generated the robot intralimb coordination based on only 2D foot trajectories in the x-z plane of a stick insect, and leg posture control and adaptation have not been introduced into the control system. Therefore, the robot can become unstable if its legs are amputated since parts of the body may drop to the ground. Furthermore, climbing over an obstacle and walk-ing on uneven terrain with a high level of roughness, where the difference between the maximum and minimum heights of the terrain surface is higher than the robot swing amplitude, will require additional control mechanisms, like local leg extension and elevation control [68,100] or intraleg control of neuroWalknet [51,52] with additional sensory feedback.
Although changing the CPG frequency through MI can lead to various emerged tripod and non-tripod walking gaits, our approach has not been successful in achieving a pentapod (wave) gait. Additionally, it could not store the emerged gaits for gait recovery. This is because it lacks neural communication between the leg control units, which is an important component for wave gait control and gait memorization, as demonstrated in our previous studies [87,101] and in neuroWalknet. [51] Thus, in the future, we will include x-y plane foot trajectories of stick insect data into our control system and further investigate the integration of leg posture control as well as local leg extension and elevation control or neuroWalknet intraleg control. We will further enhance the control system by adding adaptive neural coupling mechanisms (i.e., neural communication) to store emerged gait patterns in the neural structure [87] and accomplish other stable gaits. [101] We will also implement muscle models [102] to achieve high adaptability for traveling over challenging terrain. The muscle models will be developed based on animal data. As the muscle architecture of animals with heterogeneous leg morphology should differ between the single leg pairs, the dissimilar muscular equipment in legs with different lengths should lead to different force outputs of the leg segments, depending on their size. [103] Therefore, activity patterns in insect legs should be timed with the different muscle anatomy and function [104] and adjusted to the respective leg morphology. It would be interesting, to obtain data on the actual muscle repertoire and add this information to the muscle models in a subsequent step.
Regarding the robot hardware design, the current joint orientations of all legs of the robot are simplified with an orthogonal arrangement, similar to standard hexapod robots. Therefore, the existing joint configuration is incapable of achieving detailed movements of individual parts (coxa, femur, and tibia) of a stick insect's legs. Thus, for future robot hardware improvement, we will use stick insect joint orientation data to redesign the robot joint orientations. This will be an important ingredient in achieving precise stick insect locomotion.

Supporting Information
Supporting Information is available from the Wiley Online Library or from the author.