Zhenglong Sun
email: z.sun@maastrichtuniversity.nl

Nico Roos

A Controller for Improving Lateral Stability in a Dynamically Stable Gait

In the previous work we presented a new gait for humanoid robots, such as the Nao developed by Aldebaran. This new gait implemented on a Nao, reduces the energy consumption by 41%. Then main feature of the new gait is the absence of an area of support. The foot can rotate freely around the ankle joint. This feature makes the gait suited for uneven terrains. Stability is an important aspect of walking on uneven terrains, especially the lateral stability. During the single support phase of a step the robot balances above the stance leg. If the robot steps on a bump or in a hole, the lateral stability may be disrupted. This paper presents a controller that guarantees the lateral stability in the present of such disruptions.

Introduction

Bipedal walking for humanoid robots is one of the most interesting challenges in robotics. In the papers [START_REF] Sun | An energy efficient dynamic gait for a nao robot[END_REF][START_REF] Sun | An energy efficient gait for a nao robot[END_REF][START_REF] Sun | Dynamic lateral stability for an energy efficient gait[END_REF], we have investigated the possibility of creating an dynamically stable and energy efficient gait without an area of support. Here, the absence of an area of support means the ankle joint can move freely while the foot is on the ground. In the sagittal direction the robot's Center of Mass (CoM) is falling forward till the foot of the swing leg touches the ground. In the lateral direction, the robot balances above the stance foot in the single support phase, and falls towards the new stance foot in the double support phase. The falling towards the new stance foot is stopped by putting a force on the new stance leg. The resulting gait 1 was subsequently evaluated on a real Nao robot. The stability of the gait is validated on flat ground but not on uneven terrain since there is no feedback on the controller, thus robot cannot adjust the gait parameters to compensate for the uneven floor. In this paper, we improve the gait's lateral stability on uneven terrain by introducing such a controller.

The remainder of this paper is organized as follows. In the next section, we will give a brief overview of existing research about kinematics models for humanoid robots, stability criteria and various approaches to obtain energy efficient bipedal walking. Section: 3 briefly describes the new gait that we developed and presented in [START_REF] Sun | An energy efficient dynamic gait for a nao robot[END_REF]. We used the Inverted Pendulum Model (IPM) to investigate the energy consumption in the sagittal plane. Subsequently, we extended the model to the lateral plane and describe a gait controller with multiple parameters for a 3D full-body humanoid robot. The controller can achieve a stable gait on a physical robot in the real world after we optimize the parameters through an Policy Gradient Reinforcement Learning (PGRL). Section: 4 introduces our work on the neural network controller to enhance the lateral stability. Section: 5 concludes this paper. We provide a brief summary of the results and outline the future research.

Related Work

Movement Models

Humanoid robots have complex bodies with irregular shape and mass distribution. Therefore, it is advantageous to obtain an elemental representation of the robot's dynamics. Ideal features of a model are simplicity, and both a conceptually and mathematically accurate representation of the dynamics of the real system. The main approaches employed to model the kinematics of humanoid robots are based on the Inverted Pendulum Model (IPM) ( [START_REF] Franklin | Feedback control of dynamics systems[END_REF]) which involves a simplification compared to the body of the robot. The IPM represents the whole body of the robot as a point mass located at the center of mass (CoM) of the actual robot. The point mass is linked to the base of the robot by a telescopic massless leg. Restraining the movements of the CoM to a horizontal plane allows to simplify the motion equation of the IPM. The resulting model is known as the Linear Inverted Pendulum Model (LIPM) which [START_REF] Kajita | The 3d linear inverted pendulum mode: A simple modeling for a biped walking pattern generation[END_REF] proposed to describe humanoid robot locomotion. The LIPM provides an efficient means to represent the kinematic behavior of the robot and it is therefore a popular tool to understand and manipulate the balance of a humanoid robot. With the LIPM and zero moment point (ZMP) stability criteria [START_REF] Vukobratovic | Biped locomotion: dynamics, stability, control and application[END_REF], institutes/companies have successfully built biped robots that can walk with various gaits adapting to different walking situations (e.g. [START_REF] Kajita | Biped walking pattern generation by using preview control of zero-moment point[END_REF][START_REF] Kajita | Cybernetic human hrp-4c: A humanoid robot with human-like proportions[END_REF][START_REF] Kanehiro | Openhrp: Open architecture humanoid robotics platform[END_REF][START_REF] Sakagami | The intelligent asimo: System overview and integration[END_REF]).

Energy Consumption

However, the movement model is not the only factor to be considered. The energy consumption of a gait is an important issue. Various approaches have been proposed to reduce the energy consumption of a gait. One of these approaches is passive-dynamic walking where the robot's dynamics are designed to enable a robot to walk down slight slopes without control input, except for the gravitational force. The paper of [START_REF] Mcgeer | Passive dynamic walking[END_REF] explained this well. [START_REF] Kuo | Choosing your steps carefully[END_REF] believed that there are three primary flaws of passive-dynamic walker: they can only walk down slopes, their gaits are restricted by their dynamics, and they are sensitive to perturbations. Realizing these limitations, researchers [START_REF] Collins | A bipedal walking robot with efficient and human-like gait[END_REF] have sought to improve passive-dynamic walker by adding actuators.

A second approach to obtain energy efficient bipedal walking is the application of mechanical compliance. In the work of [START_REF] Collins | A bipedal walking robot with efficient and human-like gait[END_REF] and [START_REF] Farrell | Energetic effects of adding springs at the passive ankles of a walking biped robot[END_REF], springs were added across the hip, thigh, knee and ankle simultaneously. [START_REF] Yang | Design and control of a planar bipedal robot ernie with parallel knee compliance[END_REF] exploited parallel knee compliance on the robot ERNIE and discussed how soft/stiff springs affect the energy efficiency at different walking speed. [START_REF] Pratt | Virtual model control: An intuitive approach for bipedal locomotion[END_REF] described the implementation of series-elastic actuation on Spring Flamingo (a MIT's planar bipedal walking robot) to enable the control of the ground reaction forces during walking.

A third approach to obtaining energetically-efficient bipedal walking is the design of gaits that minimize the energetic cost of walking. The most common means of design is to use parametric optimization to the parameters that specify the gait of the robot. For example, [START_REF] Chevallereau | Optimal reference trajectories for walking and running of a biped robot[END_REF] used parametric optimization to design fourth degree polynomial functions that give the joint motions over a step as functions of time. Unlike the previous example, in the work of [START_REF] Saidouni | Generating globally optimised sagittal gait cycles of a biped robot[END_REF] cubic splines connected at points uniformly distributed along the motion time are used to generate complete optimal steps, including a doublesupport phase.

Parametric optimization methods are also implemented to optimize the walking generator on humanoid Nao robots. In the work of [START_REF] Gökc ¸e | Parameter optimization of a signal-based omni-directional biped locomotion using evolutionary strategies[END_REF], the proposed method models the omni-directional motion as the combination of a set of periodic signals. The parameters controlling the characteristics of the signals are encoded into genes and evolutionary strategies is used to learn an optimal set of parameters. Nao humanoid robots are used as the test platform. [START_REF] Abdolmaleki | Omnidirectional walking with a compliant inverted pendulum model[END_REF] augmented the 3D inverted pendulum with a spring model and use policy search to optimize the parameters of the walking engines on Nao robots. [START_REF] Shahbazi | Biologically inspired layered learning in humanoid robots[END_REF] introduced a two-stage learning algorithm for Central Pattern Generator (CPG) of Nao robot's bipedal walking.

Our New Gait

This section briefly describes our new gait presented in [START_REF] Sun | An energy efficient dynamic gait for a nao robot[END_REF][START_REF] Sun | An energy efficient gait for a nao robot[END_REF][START_REF] Sun | Dynamic lateral stability for an energy efficient gait[END_REF]. We first analyzed the gait without an area of support using an IPM with telescopic legs. Then we designed a controller which implements the gait on a real Nao robot.

Kinematics Model in Sagittal Direction

The IPM with telescopic legs allows the length of the virtual support leg to vary during a step. We proposed the leg-length policy δ : [-π 2 , π 2 ] → [0, 1] that determines how much the virtual support leg will be shortened as function of the angle between stance leg with vertical axis. The shortening of the stance leg is realized by bending the knee joint, see the right side of Fig. 1.

To identify the leg-length policy that minimizes the energy consumption of a robot, we make use of the fact that the robot has to bend the knee in order to shorten the leg. The knee torque is the main factor determining the energy consumption [START_REF] Sun | An energy efficient dynamic gait for a nao robot[END_REF]. Fig. 3 shows the optimal leg-length policy δ(α) as a function of the angle α from the beginning till the end of the step that we identified and Fig. 4 shows the realization using the 5-link model. The detailed information can be found in our previous publication [START_REF] Sun | An energy efficient dynamic gait for a nao robot[END_REF].

Since we assumed the absence of an area of support and to further reduce the total energy cost, we set the stiffness on both ankle joints to almost zero. Thus, the stance leg of the robot can freely rotate around ankle joints, and the area of support reduces to a point.

𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑝𝑝𝑠𝑠 𝑑𝑠𝑠𝑑𝑠𝑠 𝑠𝑠𝑠𝑠𝑠𝑠𝑠 𝑠𝑝𝑝𝑠𝑠

Kinematics Model in Lateral Direction

For a simple forward step, it is insufficient to only consider 2D dynamics in sagittal direction. To address the lateral stability, we designed a lateral controller to regulate the CoM lateral movement during double support phase which is proposed in [START_REF] Sun | Dynamic lateral stability for an energy efficient gait[END_REF]. We use the upper body tilt to initiate the lateral movement of the CoM towards the swing foot. Next, we use a force generated by the swing leg to stop the movement when CoM is balanced above the swing foot, which then becomes the new stance foot. The force generated by the swing leg is described by a force policy. In order to smooth the CoM transition trajectory, we determines the shape of the force policy by means of Quadratic Bezier curve which introduces the quadratic bezier point θ 7 , one of the controller parameters in next subsection.

Controller Design

We designed a controller which implements this gait on a Nao robot. Because of the differences between the abstract model and the Nao, several parameters of the controller need to be fine-tuned. This subsection presents the parameters of a gait controller that realizes the leg-length policy described in Section: 3.1 and the parameters that control the lateral movement of the CoM in the DPS. We identified 9 parameters that are essential in controlling a dynamic gait:

- All parameters except θ 1 (the step length) will be optimized in the experiments. We do not consider the step length for optimization because we need step length to be variable when the velocity is changing. We manually set different walking velocity v in each experiment and determined the optimal Step Time θ 4 . The corresponding step length is given by: θ 1 = vθ 4 .

Algorithm for Learning Controller Parameters

We use a policy gradient reinforcement learning method [START_REF] Kohl | Policy gradient reinforcement learning for fast quadrupedal locomotion[END_REF] to automatically search the set of possible parameters with the goal of finding the stable and low energy cost walk. In order to generate a gait that is energy efficient and stable, we considered a fitness function based on the total energy cost and the stability over a certain distance of forward walk. The energy cost determines 30% of the fitness function value and the distance which the robot walks without falling determines 70% of the fitness function value [START_REF] Sun | An energy efficient dynamic gait for a nao robot[END_REF].

Learning Optimal Parameters in the Simulator To generate the optimal gait parameters and validate the gait's performance, we uploaded the controller of our proposed gait together with an implementation of the policy gradient algorithm into the Webots simulator. We used a relatively elementary hand-tune gait as a starting policy for the policy gradient algorithm. Each new policy was evaluated by letting a robot walk at a constant distance of 0.75 meters. During the walking, the energy consumption and stability were determined. The policy gradient algorithm converges to a parameters set P shown in Table 1. The algorithm presented here converges to a local optimum. In order to investigate whether the results could be a global optimum, we repeated the learning experiment 500 times, each time starting from a randomly generated parameter vector x π with the same velocity. The results of the experiments indicate that the local optimum we have in Table 1 is probably the global optimum. Therefore, the parameters set P most likely results in the most energy efficient gait.

The accompanying video material 2 shows the Nao robot walking on flat ground with our proposed gait controller at a speed of 6 cm/s. We also compare the new gait with the standard gait Aldebaran supplies with the Nao. The energy consumption of new gait is 41% less than the Aldebaran gait.

Walking on Uneven Floor

The gait without support areas we proposed is validated as a dynamically stable gait on flat ground. However, the controller cannot compensate for external disturbances. This means any disturbance such as a push or stepping on uneven terrain may jeopardize its balance, because the ankle stiffness is set to almost zero. To enhance the walking stability, the gait should adapt to unknown disturbances. For example, if the robot is standing on a slope or stepping on a bump in the floor, the feet are not on the same altitude in the lateral plane, this may cause the CoM undershoot/overshoot the balance position when switching the stance leg. In order to make the problem tractable, simplifying assumptions are made. Since bipedal robot's stepping on the uneven terrain makes the altitude of robot's two foot different, the robot's walking on uneven floor can be viewed as walking on the slope in the lateral direction. The gait controller should adapt the gait parameters to compensate for the slope in the lateral direction.

Controller design

As a first step in designing a controller that can handle disturbances influencing the lateral stability, we determined the optimal control parameters θ 2 to θ 9 when walking on a certain slope in the lateral direction. Since the left and right foot are at different height, the control parameters for the left and right leg might be different. Therefore, θ 3 to θ 9 are split into parameters for the left leg θ L i and the right leg θ R i . Next we addressed how to adapt the control parameters.

The robot does not know that it walks on a slope or about other disturbances. The only information it has available is (1) the angular speed β of the rotation of the CoM around the ankle of the stance leg. (2) the lateral acceleration measured by the Inertia Measurement Unit (IMU). This value approximates the angular acceleration β. (3) the angle β of the CoM w.r.t the swing foot. Fig. 2 shows the three parameters (β , β, β). We chose to use these three parameters as inputs for the controller that can adapt the gait parameters.

We designed a series of experiment in the simulator Webots to obtain the optimal control parameters and input vector (β , β, β) corresponding to certain slopes. In the experiments, the same policy reinforcement learning method in subsection 3.3 is used to find the proper control parameters that can generate the stable walking gait under different slopes. The experiments require the robot stands on various slopes where tilt angles varies from 0.00 to 0.139 (rad) in robot's lateral plane (see Fig. 6). We kept the robot walking on the slope and ran the policy search method to get the corresponding controller parameters which ensure robot's stability. Each new policy was evaluated by letting a Nao robot move for 5 seconds. The fitness function of each policy will get high score if the robot keeps stable. Otherwise, the function gets a penalized score. After the result of policy search converged while the robot's movement become stable, the control parameters are recorded. Table 2 shows the results fo those experiments. Next, from the beginning of DPS to its end, we sampled the data from IMU and joint sensors every 10 ms in order to determine (β , β, β) values during the DSP for each lateral slope. A controller that uses (β , β, β) as inputs cannot adapt θ 7 . At the end of the DSP, (β , β, β) must become equal to (0, 0, 0) for every θ 7 value. So, in the neighborhood of (0, 0, 0) the correct θ 7 value is not well defined. Therefore, instead of θ 7 , the stiffness determined by θ 7 will be used instead. The stiffness values are determined by sampling the the Bezier curve determined by θ 7 for different β values, see Fig 7.

From Table 2, we know that the parameters θ 3 , θ 5 and θ 8 are also influenced by the slope angle. The parameter θ 3 solely depends on the relative elevation between stance and swing leg, and its value can be determined at end of the DSP where β should become zero. Since θ 3 encodes the information about the slope angle, its value can be used to set the values of the other two parameters: θ 5 and θ 8 .

Controller Implementation and Evaluation

We implemented two neural networks to control and improve the robot's lateral stability by adjusting the controller parameters adaptive to unknown slope. The backpropagation 8: Architecture for the identification process of the lateral controller parameters in double support phase method has been applied for training multi-layer feedforward networks, see Fig. 5. With the trained network, we designed a simple neural network controller to maintain robot's walking stability on uneven terrain in the lateral direction. Fig. 8 depicts the general architecture of the lateral stability controller that was implemented in this paper. When the robots walking on uneven floor and a new DSP begins, with the data retrieved from joint sensors (β and β) and IMU ( β), the first neural network takes these three variables as the input vector and outputs the new stiffness values for swing leg during the DSP. The second network uses as input the value of θ 3 at the end of the DSP and outputs the values of θ 5 and θ 8 , which are used by the gait controller. Together with other fixed parameters, the gait controller generates updated joints command, to compensate for the uneven terrain. Fig. 9 shows the roll angles trajectories of left/right legs when robot is walking on slope in 0.00 rad, 0.07 rad (≈ 4 degrees) and 0.12 rad (≈ 7 degrees). From this figure, we can see that right foot is higher than the left one when the slope exists which makes the joints on right leg rotate in less angles to let CoM approach its balance point. Moreover, under the different slopes, the time for one step does not change which means our proposed gait can make a stable walk on different slopes without the loss of walking velocity. The accompanying video material 3 shows the Nao robot walking on uneven terrain with our proposed gait controller in the simulator Webots which proves our controller can handle the altitude difference of foot placement and adjust the control parameters to maintain balance. Fig. 9: The roll angle of both legs under different slope in 0 degree, 4 degrees, 7 degrees.

Conclusion

In previous work we have presented a new gait for humanoid robots. An implementation of the gait on a Nao robot reduces the energy consumption with 41% compared to the standard gait of the Nao. An important feature of new the gait is that it does not use an area of support. That is, the robot can rotate freely around the ankle joint while walking. This makes the new gait suited for uneven terrains because the feet can adapt to the slope of the terrain. The absence of an area of support implies that, in principle, the robot is unstable. In the sagittal plane, the robot falls forwards in each step, and in the lateral plane, the robot balances above the stance foot in the single support phase and falls towards the swing foot in the double support phase. Nevertheless, experiment with a Nao robot on an almost flat floor consisting of wooden planks showed that the gait is stable.

In this paper, we investigate how we can improve the lateral stability of the gait when walking on an uneven terrain. The most important aspect of walking on an uneven terrain is the lateral stability. Since the robot balances above the stance leg in the single support phase while it can turn freely around the ankle joint, a bump or a hole in the walking surface may disrupt the lateral stability. Therefore, during the double support phase the robot may over-or undershoot its stable end point, namely, balancing stable above the new stance foot. The paper presents a feedback controller based on a feedforward neural network, that adapts the gait parameters in order ensure the lateral stability while walking on an uneven terrain.

The feedback controller for the lateral stability also enables the robot to handle, to some degree, slopes in the sagittal plane. In future work, we will extend the feedback controller to specifically address the effects of walking uphill or downhill.

Fig. 2 :

 2 Fig. 1: The abstract Inverted Pendulum Model

Fig. 3 : 1 Fig. 4 :

 314 Fig.3: The optimal leg-length policy

  Step Length (θ 1 ): Defines the distance which Nao moves in a singe step (sagittal).-Step Height (θ 2 ): Defines the maximal altitude between ground and lifting feet.A high step height requires swing leg' faster move and may cause horizontal instability. A low step height increases the possibility of tripping and limits the step length.-Knee Bending (θ 3 ): Defines the maximal bending of the swing leg at the beginning of the double support phase which determines the value of δ(α b ), see Fig.3. This parameter determines the sagittal velocity and the energy cost. -Step Time (θ 4 ): Defines how long a single step lasts. This parameter determines the sagittal walking velocity. -Stretch Time (θ 5 ): Defines how long it takes for the stance leg to stretch from θ 3 (angle of bent knee) to its full length at the beginning of the single support phase, see Fig.3. -Torso Pitch Inclination (θ 6 ): Defines the maximum angle that torso leans in sagittal direction at the beginning of the first step. If positive, it will move the center of mass (CoM) in sagittal direction. If it is set not appropriate, a fall will occur. In our experiments, the inclination lasts for 200 ms. -Quadratic Bezier point (θ 7 ): Defines the magnitude of middle points in Quadratic Bezier Curves, which determines the force policy on swing leg (introduced in Section: 3). -Torso Roll Inclination (θ 8 ): Defines the maximum angle that torso leans in lateral direction. If positive, it will move the center of mass (CoM) towards the swing leg in lateral plane as discussed in Section: 3. -Ratio of single support duration (θ 9 ): Defines how long the single support phase lasts in one single step. The single support phase duration equals this parameter times step time θ 4 .

Fig. 6 :

 6 Fig. 5: Backpropagation Neural Network

Fig. 7 :

 7 Fig. 7: Sampled stiffness over angle β

Table 1 :

 1 Learned Parameters set P

	Parameter	Learned Value
	step length	0.1 3.9(cm)
	step height	0.02 3.24(cm)
	knee bending	0.1 14.2 (degree)
	step time	25 650(ms)
	stretch time	25 78(ms)
	torso pitch inclination 0.1 8.9 (degree)
	torso roll inclination 0.1 6.5 (degree)
	Quadratic Bezier point 0.1 (0.9*DSP time, 0.2)
	ratio	0.1 0.8
	velocity	6 (cm/s)

Table 2 :

 2 Learned control parameters adaptive to different slop angles

	p (slop angle) θ L 3	θ L 4 (ms) θ L 5 (ms) θ L 6	θ L 8 θ L 9	θ L 7
	0°14.2°650	78 8.9°6.5°0.8 (0.90,0.20)
	3°14.5°650	75 8.9°5.5°0.8 (0.86 0.16)
	5°15.3°650	76 8.9°3.5°0.8 (0.84 0.14)
	8°15.8°650	74 8.9°2.0°0.8 (0.80 0.12)
	10°16.4°650	75 8.9°0.5°0.8 (0.78 0.12)
	p (slop angle) θ R 3	θ R 4 (ms) θ R 5 (ms) θ R 6	θ R 8 θ R 9	θ R 7
	0°14.2°650	78 8.9°6.5°0.8 (0.90,0.20)
	3°13.5°650	74 8.9°6.5°0.8 (0.90 0.22)
	5°11.7°650	70 8.9°7.5°0.8 (0.91 0.26)
	8°9.6°650	68 8.9°8.5°0.8 (0.92 0.28)
	10°8.2°650	65 8.9°9.5°0.8 (0.94 0.29)

A video of the new gait at: https://project.dke.maastrichtuniversity.nl/ robotlab/?attachment_id=153

https://project.dke.maastrichtuniversity.nl/robotlab/ ?attachment_id=153

https://www.youtube.com/watch?v=7DxHVEd8hc8