Michel Aractingi

Pierre-Alexandre Léziart

Thomas Flayols

Julien Perez

Tomi Silander

Philippe Souères

A Hierarchical Scheme for Adapting Learned Quadruped Locomotion

The quadruped locomotion task is generally conditioned on a velocity command that is user-defined. However, certain features of the locomotion are not represented in this high-level definition of the task, e.g., swing feet height, step length and expended energy. Using reinforcement learning, many of these features can be determined by the reward function terms and scales that are often fixed. In this work, we propose a deep reinforcement learning (DRL) approach to learn control policies augmented with parameters that modify different aspects of the reward function and control setup which, in turn, result in variations of the locomotion. We can then define a hierarchical architecture where a high level policy infers the suitable parameters to complete a given task. We show that this setup makes it possible to learn more complex behaviours that can be adapted for different terrains and environments to ensure successful and efficient locomotion. We display our results by deploying the low-level parameterized policy on the MIT Mini-Cheetah quadruped.

I. INTRODUCTION

Quadruped locomotion has emerged as an interesting field of research in both model-based control [START_REF] Di Carlo | Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control[END_REF], [START_REF] Kim | Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control[END_REF], [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF] and modelfree learning [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Lee | Learning quadrupedal locomotion over challenging terrain[END_REF], [START_REF] Kumar | RMA: Rapid Motor Adaptation for Legged Robots[END_REF]. In both approaches, the objective is similar, i.e., produce robust locomotion when following a desired velocity command while minimizing the used energy. However, this is a very abstract definition of the locomotion task as, in reality, many low-level features of the motion are required such as how much to lift the feet, the length and frequency of footsteps, or the best nominal joint pose to center the movement around.

These different decisions intrinsically make the locomotion problem a highly multi-task one at different levels regarding the motion of the base and the joints [START_REF] Kim | Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control[END_REF]. At the high-level, the velocity command determines the trajectory of the center of mass (COM) of the robot, but in more complex environments that require precise intended movements, e.g., going up the stairs or reaching a given location, the velocity on its own is not enough. At the same time, the low-level joint control pattern might also need to be modified for such specific tasks. For example, on rough terrains the robot might need to lift its feet higher, and in steep slopes the robot would need to take smaller steps or use more torque [START_REF] Tonneau | An efficient acyclic contact planner for multiped robots[END_REF].

Therefore, there is a clear need to augment the task description and have the ability to convey the desired behavior to dictate the low-level motion. In this paper, we tackle the multi-task aspect of locomotion using hierarchical Fig. 1: Snapshots of the proposed low-level policy controlling the Mini-Cheetah through different challenging terrains. Full video can be found at: https://youtu.be/ B92HB964xq8. reinforcement learning (HRL). With HRL we can decompose a reinforcement learning (RL) problem into simpler sub-tasks that are easier to design and learn [START_REF] Pateria | Hierarchical reinforcement learning: A comprehensive survey[END_REF].

We propose learning a low-level policy with an elaborate reward function that governs many cost terms related to the desired joint-level control. Along with the state, this lowlevel policy is augmented with command parameters which are variables and weights that define the terms of the reward function and details of the state dynamics. We use the term command here because we view these parameters as an extension of the velocity commands that generally dictate the locomotion task.

The main contribution of this work is a hierarchical deep RL system that augments the baseline locomotion policy with a command parameters vector ω that manipulates different aspects of the locomotion (swing feet height, step length, initial pose and PD gains). This baseline policy can then be used in a hierarchical setup where a high-level policy determines suitable command parameters to adapt the locomotion to different situations. We show that this two-level design improves the performance of velocity tracking policies by reducing energy consumption and velocity tracking error in various situations while having better sample efficiency. We also show how these policies enable the real robot to cross different structured terrains that the baseline policy would regularly fail to be robust against in Figure 1. We illustrate our results on the MIT Mini-Cheetah quadruped [START_REF] Katz | Mini cheetah: A platform for pushing the limits of dynamic quadruped control[END_REF]. The paper is organized as follows. Section II discusses the related literature. Section III introduces some notions and definitions. Section IV presents in detail the approach and how the RL policies are designed and trained. Finally, the experimental results and conclusions are outlined in sections V and VII respectively.

II. RELATED WORK

Recent development of model-based controllers for quadruped robots have contributed to the growth of locomotion research [START_REF] Di Carlo | Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control[END_REF], [START_REF] Kim | Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control[END_REF], [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF]. An efficient modular architecture was proposed in [START_REF] Kim | Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control[END_REF]. It includes a contact planner, a model predictive controller (MPC) that plans the base motion and the feet contact reaction forces over a receding horizon, and a whole body controller that executes the plan by controlling the joints at a higher frequency. A similar scheme with a simplified inverse kinematics approach and state estimation was later developed in [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF]. While there is a hierarchy of optimization tasks in these model-based architectures, they are mainly built around base velocity tracking task and fixed values related to the gait pattern and feet trajectories. While they produce agile and efficient locomotion in nominal conditions, they do not allow for easy adaptation of that locomotion to new situations.

Alternative deep RL control paradigms were proposed to learn locomotion in an end-to-end manner. In [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Lee | Learning quadrupedal locomotion over challenging terrain[END_REF] authors detail different elements required to learn robust controllers that works on real robots, e.g., choice of action space, reward design and actuation modeling. In [START_REF] Ji | Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion[END_REF] state estimation is learned along with the policy which resulted in a rapid locomotion that broke the speed record of the Mini-Cheetah [START_REF] Katz | Mini cheetah: A platform for pushing the limits of dynamic quadruped control[END_REF]. A robust walking gait was learned by biasing the action space with central pattern generators (CPGs) in [START_REF] Miki | Learning robust perceptive locomotion for quadrupedal robots in the wild[END_REF]. Other authors suggested to learn an adaptation model along with the main policy to adapt its control to various environmental changes [START_REF] Kumar | RMA: Rapid Motor Adaptation for Legged Robots[END_REF].

While these deep RL techniques learn robust locomotion to track a velocity command, they produce monotonic motion that can be hardly generalized to new situations. In [START_REF] Choi | Fast adaptation of deep reinforcement learning-based navigation skills to human preference[END_REF] parameterized policies were learned for wheeled-robot navigation tasks. In order to adapt the behavior, as the dynamics of the underlying system changes, reward weightings were incorporated in the state. Recently, parameterized policies have also been introduced for versatile quadruped locomotion but without showing benefits of hierarchical RL [START_REF] Margolis | Walk these ways: Tuning robot control for generalization with multiplicity of behavior[END_REF].

Hierarchical reinforcement learning (HRL) methods have been proposed for learning locomotion strategies, for example by learning separate policies for recovery, standing and locomotion and then a high-level policy that learns to switch between them [START_REF] Lee | Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning[END_REF]. Hierarchical methods, in similar spirit, have also been developed in planning-based algorithms. The work in [START_REF] Tonneau | An efficient acyclic contact planner for multiped robots[END_REF] proposes a contact planner for locomotion in tasks where there are multiple stages of contact and decomposing the planning into sub-problems is necessary. A two-level approach was proposed in [START_REF] Peng | Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning[END_REF] to control a biped in simulation, where a low-level policy learns to follow desired footstep placements and a high-level policy learns to place the footsteps based on the terrain information. In our work, we propose embedding a policy with several aspects related to the control and reward function such that the behaviour of the low-level policy is very flexible and can be adapted in different dimensions via a high-level policy. We also show the effectiveness of the low-level policy on the real system in challenging terrains and not only in simulation.

The proposed approach to learn parameterized policies for quadruped locomotion follows an idea similar to the one developed in [START_REF] Choi | Fast adaptation of deep reinforcement learning-based navigation skills to human preference[END_REF]. However, we also use parameters like the gains of the low-level impedance controller that are not part of the reward, but can have a desired effect on the motion. In the literature of skill learning, some works propose to learn separate skills for different purposes [START_REF] Lee | Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning[END_REF], [START_REF] Li | Learning agile skills via adversarial imitation of rough partial demonstrations[END_REF], [START_REF] Fuchioka | Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors[END_REF], [START_REF] Li | Versatile skill control via self-supervised adversarial imitation of unlabeled mixed motions[END_REF] often via imitation learning of trajectories generated by an optimal controller. Here, we are more interested in learning a general locomotion policy that we can control to fulfill specific task. Though it is also possible to follow a hierarchical approach to learn locomotion policies from visual input [START_REF] Jain | From Pixels to Legs: Hierarchical Learning of Quadruped Locomotion[END_REF], the problem we consider in this work is to learn the best blind low-level locomotion policy so that we can control several aspects of it (foot height, stride length). Since our proposed policies are reactive, there is no planning involved.

III. PRELIMINARIES

A. Low-level parametric policies

We model the low level learning environment as a set of Markov decision processes (MDPs), indexed by parameters ω, with common, continuous state and action spaces [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]: M Ω = {(S, A, R ω , P, ρ 0 )|ω ∈ Ω}, where S is an infinite set of states, and A is an infinite set of actions. In each M ω ∈ M Ω , taking an action a in a state s yields a reward, defined by a function R ω : S ×A×S → R. The environment dynamics is described by a conditional transition probability distribution P : S ×A×S → R + , with the interpretation that P(s, a, s ′ ) = p(s t+1 = s ′ |s t = s, a t = a) is a probability (density) that the next state is s ′ given that the current state is s, and the action taken is a. ρ 0 is the initial state probability distribution. We assume parameters ω ∈ Ω ⊂ R d to be sampled from a static distribution ρ Ω , and define the learning task to be finding the common parameters θ for all stochastic policies π ω θ : S → P r(A|S) in order to attain a maximum expected discounted sum of rewards

J(θ) := E ρΩ E π ω θ ,Pω,ρ0 [ ∞ t=0 γ t R ω (s t , π ω θ (s t ), s t+1 )], (1)
where γ ∈ [0, 1] is a discount factor. In practice, π ω θ is implemented as a neural network thus θ corresponds to its weights.

B. High-level parameter controlling policy

Given a parameterized set of MDPs (as described above) and a parameterized set of low-level policies π Ω θ , we define the high-level control setting as an MDP M H = (S H , Ω, R H , P H , ρ H 0 ), where the superscript H stands for "high-level". The state space S H = S × S h may also contain additional information S h (e.g., vision data) and the action space consists of the set of low-level parameters Ω. The transition dynamics is supposed to respect the low-level dynamics so that it obeys

s ′h p H ((s ′ , s ′h )|(s, s h ), ω)ds ′h = P(s, π ω θ (s), s ′ ) (2)
for all s, s ′ ∈ S, s h ∈ S h , and ω ∈ Ω.

The high-level learning objective is to find the parameters ϕ for a high-level stochastic policy

π h ϕ : S H → P r(Ω|S H ) : π h ϕ (ω|s) = p ϕ (ω t = ω | s t = s) that optimizes the expected discounted cumulative reward J H (ϕ) := E π Ω θ ,π h ϕ ,P H ,ρ H 0 [ ∞ t=0 γ t H R H (s H t , ω t , s H t+1 )], (3)
where

γ H ∈ [0, 1] is a discount factor, s H t = (s t , s h t )
, and ω t is computed by the high level policy ω t = π h ϕ (s H t ). Notice also that s H t+1 = (π ωt θ (s t ), s h t+1 ).

IV. METHOD

In this section, we will first outline the main procedure for learning general end-to-end locomotion policies on the Mini-Cheetah quadruped. The low-level policy has to be robust while following a set of references determined by the command parameters. The learned policy also has to transfer to the real system. We will then describe the parameters that the low-level locomotion task is conditioned on. Finally, we introduce the high-level policy that adapts the parameterized locomotion.

A. Low-level policy

The goal of the low level policy is to produce joint angle control based on the state of the robot and the chosen command parameters. A general scheme of the low-level control is depicted in Figure 2.

Observation space. The observation of the robot mainly depends on the IMU readings and the proprioception of the joints. The observation at time t is:

o t = (θ body , ω body , q t , qt , a t-1 ), (4)
where θ body is the orientation of the base, ω body is the angular velocity of the base, q t and qt are the joint angles and velocities respectively. We also include the previous actions a t-1 to the observation. State space. Observations in the real system are noisy and the dynamics of the robot and terrain could vary depending on the situation. In order to learn policies that are robust and transferable, one needs to inject noise and randomization in the observation space and environment's dynamics [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Miki | Learning robust perceptive locomotion for quadrupedal robots in the wild[END_REF]. Therefore, relying on a single timestep observation as input to the control policies would often hinder both learning and transfer.

To further robustify the robot behaviour, we use the rapid motor adaptation (RMA) strategy [START_REF] Kumar | RMA: Rapid Motor Adaptation for Legged Robots[END_REF]. RMA is a distillationbased technique where an encoder network E is trained to encode useful privileged information (i.e., information only available in a simulator) to a latent vector z t that represents environment's dynamics that are randomized to mimic the model mismatches and noisiness that are likely to appear in the real world. At a second stage, an adaptation model A is trained to learn an estimate ẑt of z t from the history of observations and actions with a supervised learning loss. We then use A when deploying the policy on the real system. In the simulator, the state is s t = (o t , z t , ω t ), and in the real robot s t = (o t , ẑt , ω t ). In this work, we propose to augment the encoder input with privileged state information related to the robot's feet positions and contacts and the base linear velocity. Ablation studies (see Section V-B.2) indicate that this helps learning low-level policies that reach the commanded references better and with less training iterations. A full scheme of the RMA approach is depicted in Figure 3.

Reward function The objective of the reward function is to achieve a specified base velocity while minimizing costs associated with acquiring resilient and stable locomotion skills. A detailed breakdown of the reward function components and weightings is presented in Table I. V base x,y,yaw refers to the 3D vector consisting of the measured base linear velocity along x and y and angular yaw velocity. p f,z,i is the i th foot height in world coordinates while p max f,z refers to the desired foot height in world coordinates and V f,xy,i is the i th linear foot velocity along x and y. The foot clearance reward depends on how close the current foot height is to the desired one, scaled by the foot velocity, e.g., the faster the feet are moving the higher the robot should lift its feet. The total reward function is the weighted sum of these terms. The main positive term in the reward function is related to tracking the desired velocity, while negative costs are used for refining the movement in terms of action magnitude and smoothness, for raising the feet, and maintaining an appropriate posture and base stability. Locomotion command parameters. The reward design is similar to previous studies on end-to-end RL for locomotion [START_REF] Ji | Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion[END_REF], [START_REF] Lee | Learning quadrupedal locomotion over challenging terrain[END_REF], [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Miki | Learning robust perceptive locomotion for quadrupedal robots in the wild[END_REF], [START_REF] Aractingi | Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning[END_REF]. However, in these previous works, the velocity command was usually considered the main command parameter in the reward function. While training in simulation it is randomly sampled, and on the real robot, it is controlled by the user with a gamepad device. In this work, we consider more elaborate command parameters that are hard to be prescribed by the user. The main parameter vector ω given to the policy is:

ω = (V cmd , p max f,z , c jpos , Kp, Kd, q nominal ),
where V cmd is 3D the velocity command, linear along x, y and angular about z. p max f,z is the maximum foot height to reach in the swing phase, which is part of the foot clearance reward. c jpos is the coefficient that weights the joint angle deviation penalty, and Kp and Kd are the proportional and derivative gains for the PD controller, and q nominal is the initial pose around which the action space is centered.

Action space. The network π θ outputs the joint angle targets that are fed to a PD controller with zero joint velocity targets. We center the action space around the nominal joint angles at the initial position of the robot, so that the PD target at time t is q target t = q nominal + λπ θ (s t , ω t ). This centering is essential for the policy to learn and allows us to control the limits of the joint angles from the initial position. In our experiments we found λ = 0.3 to be the best choice. This action space was used in previous literature [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Lee | Learning quadrupedal locomotion over challenging terrain[END_REF].

B. High-level policy

The actions of the high-level policy are locomotion parameters that modulate the low-level policy. The main intuition is that the elements of ω can be varied to change the behaviour of the robot while still producing stable locomotion that is learned by the low-level policy.

The high-level policy π h ϕ can control V cmd when it is not tuned by a user, like for example, in an autonomous point goal navigation task. The maximum desired foot height, p max f,z , is an important parameter that has consequences on transfer [START_REF] Ji | Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion[END_REF], [START_REF] Aractingi | Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning[END_REF]. On complex terrain the robot might need to lift its feet higher than on flat terrain where the robot can safely save some energy by not lifting its feet much. c jpos determines the joint angle deviation penalty weight. In our experiments in Section V-B.1, we found that different values of this coefficient result in different stride length. Therefore, we use it as an indirect way of controlling the stride length. Controlling the PD gains adds variable impedance control aspects to learned policies [START_REF] Bogdanovic | Learning variable impedance control for contact sensitive tasks[END_REF]. It also adds robustness to variations in the environment [START_REF] Abu-Dakka | Variable impedance control and learning-a review[END_REF] and makes it possible to control the expended torques to perform tasks that require more power.

The choice of parameters the high-level policy controls depends on the task. In this work, we experiment with a highlevel task of traversing a variety of complex terrains while the high-level policy only optimizing for velocity tracking and energy consumption. The high-level policy takes a scan of the height measurements surrounding the robot as input Figure 5. The actions of this policy are deltas to the default command parameters vector ω excluding the velocity commands. We chose this task to show how we can easily learn to adapt the low-level policy behaviour that is trained on a flat terrain in order to be useful on more complex terrains. The high-level reward function r h is based on the velocity tracking term (in Table I) and the energy consumption terms:

r h t = r vel t -0.002|τ T t qt |. (5)
In the next section, an analysis of the resulting low-level policy with the elaborate locomotion parameters input is displayed. We then present some high-level tasks to showcase the benefits of the proposed hierarchical scheme. For each task, the high-level state, action and reward will be explained in detail along with the results.

V. EXPERIMENTAL RESULTS

This section includes full implementation details about the entire approach. We conduct our experiments to answer the following questions: (1) Which aspects of the locomotion vary when modifying the command parameters of the lowlevel policy? (2) Does our additions to the encoder network improve the learned performance? (3) How would a hierarchical policy improve on the baseline end-to-end approaches on flat terrains and complex structured terrains in the presence of exteroception?

A. Implementation details

As mentioned before, we conduct our experiments with the Mini-Cheetah quadruped, which has 12 degrees-of-freedom with 3 actuators per leg [START_REF] Katz | Mini cheetah: A platform for pushing the limits of dynamic quadruped control[END_REF]. The two levels are trained one after the other. First, the low-level policy is trained to follow different random values of the command parameters. After that, depending on the high-level task, one trains a high-level policy that controls the parameters of the already learned parametric low-level policy.

Both policies have the same architecture which is a multilayer perceptron (MLP) with three hidden layers of sizes 512, 256, 128. The RL algorithm used to train both levels is PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] with generalized advantage estimation [START_REF] Schulman | Highdimensional continuous control using generalized advantage estimation[END_REF]. We train our policy networks with actor-critic approach with critic having the same architecture as the actor but with a scalar value for the value estimation [START_REF] Konda | Actor-critic algorithms[END_REF]. The policies run at a frequency of 50Hz while the simulation frequency runs at 200 Hz. On the robot the same control frequency is maintained while the low-level PD control runs in a high frequency feedback loop of 40 KHz.

Low-level policy training. The observation space is 45dimensional while the actions are 12-dimensional, equal to the number of actuated joints. We use the exponential linear unit (ELU) activation function in the neural networks.The critic also receives directly the privileged state of the robot.

We use the IsaacGym simulator from Nvidia [START_REF] Makoviychuk | Isaac gym: High performance gpu-based physics simulation for robot learning[END_REF]. The environment code is based on the legged gym repository [START_REF] Rudin | Learning to walk in minutes using massively parallel deep reinforcement learning[END_REF]. In the current experiments, 4096 agents are run in parallel in an infinite horizon objective where the environment does not reset with each new training episode, but continues with the latest reached state. However, we found that introducing random resets also helps. We were able to learn the low-level policy with command parameters in 5000 iterations.

A linear curriculum was implemented on the penalty terms of the reward. A curriculum factor k c ∈ [0, 1] was introduced to scale the reward. This factor is increased as training progresses to give more weight to the penalties. At the start of training the agent is mostly concerned with learning a movement that follows the reference velocity in any possible way. As training progresses the movement is refined to optimize the penalty terms [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Aractingi | Controlling the Solo12 Quadruped Robot with Deep Reinforcement Learning[END_REF]. The reward function term weights in Table I are chosen to be: c vel = 1.0, c vz = -1.2, c q = c τ = -0.0003, c q = -0.00001, c θ = -3.0, c f cl = -5. and c smooth = -0.01.

Command parameters ω are sampled at the start of a new training episode and given as input to the low-level policy.

The parameters are resampled when the policy fails (e.g., robot falls) or at random times if the policy does not fail. The sampling range for each element in ω can be found in Table III 1 . Each parameter is sampled from a uniform distribution that samples deviations around the parameter's default value within the specified range. The sampling is also scaled by the curriculum factor k c , i.e., we use the curriculum on the command parameters as well as on the reward terms.

Random Observations:

Random Dynamics: θ body U 3 (-0.05, 0.05) Motor Strength U 12 (-0.9, 1.1) ω body U 3 (-0.20, 0.20) Ground Friction U (0.5, 1.0) q U 12 (-0.05, 0.05) Ground restitution U (0.0, 1.0) q U 12 (-1.00, 1.00) Domain randomization and RMA. For transferring the learned policy to the robot, it is essential to randomize various aspects of the observation and dynamics in order to mimic the perturbations and imperfections of the real system. The randomized values in the observations, robot dynamics, and terrain dynamics are shown in Table II. However, we found that, for the Mini-Cheetah, relying solely on domain randomization does not yield good transfer. To enhance transfer, we used rapid motor adaptation (RMA) [START_REF] Kumar | RMA: Rapid Motor Adaptation for Legged Robots[END_REF], [START_REF] Margolis | Rapid locomotion via reinforcement learning[END_REF], in which one first learns a latent embedding z t that encodes privileged information around the randomized dynamics, and then uses supervised learning to build an adaptation model that estimates this z t based on the sequence of state variables that are also available in the real robot.

Body Mass U (-1.0, 1.0) COM displacement U 3 (-0.2, 0.2)
In our setup, the embedding size for z t is 18 while the input size of the privileged information is 29. The privileged input is constructed from the vector of random dynamics values (ranges presented in Table II) and the privileged data regarding the feet height in world frame, terrain contact indicators and base linear velocities. The encoder E and adaptation model A are both MLPs with two hidden layers of sizes 256 and 128. The adaptation model takes as input the last N = 15 observations and outputs an estimate ẑ of the latent vector z. Further information on this method can be found in [START_REF] Kumar | RMA: Rapid Motor Adaptation for Legged Robots[END_REF], [START_REF] Margolis | Rapid locomotion via reinforcement learning[END_REF]. B. Low-level policy study 1) Understanding the parameterized low-level control: It is important to first quantitatively and qualitatively observe the effect that the parameters ω have on the final locomotion in order to be able to define useful tasks that could benefit from a policy that controls ω. In this section we outline the difference in aspects of the resulting locomotion when changing some command parameters individually. The joint angle deviation term in the reward penalizes the policy based on how far the joint positions are from the nominal pose. Intuitively, modifying this penalty coefficient in the reward should allow us to control the range of joint movement which affects the step length. We verify that the step length in the parameterized low-level policy is a function of the coefficient c jpos by examining the average foot step length of the forward left foot as a function of the forward velocity in Table IV. This table shows a clear relationship between the foot step length and coefficient value. For all values of c jpos the average step length increases with the increase in the commanded V x . However, the average step length for the same velocity command decreases when the absolute magnitude of c jpos is increased. In other words, the more weight given to the joint angle deviation term the smaller the average foot steps are and vice versa. We also notice the high variance of the average step length when c jpos = -0.1 which suggests the limit to which c jpos could be varied while the penalty still retains an effect on the overall behaviour.

The p max f,z parameter defines the target foot height when the foot is in swing mode. Figure 4 shows a plot of the The curves show two experimental setups: when additional information is added to the encoder network (yellow), and when it is not (purple). achieved height of the front left foot as a function of time when running the low-level policy with a forward velocity command of 1.0m/s for four seconds. The figure shows patterns when the foot is swinging and when it is resting on the ground. We can clearly see the increase in foot height for higher values of p max f,z . Another interesting behaviour that emerges from changing the foot height target is a change in the stepping period. The 5cm height target results in shorter stepping period than the 10cm target, which in turn results in a shorter period than the 15cm target. Note that the achieved foot height is lower than the target because the foot clearance reward does not only depend on p max f,z but is also scaled by the velocity of the feet, see Table I.

2) Ablation studies: In Section IV-A, we introduced additional information regarding the feet height, feet contacts and the base linear velocity as additional input to the encoder network E. We show in Figure 6 the average episodic reward achieved during training as a function of the training step with and without the additional data to the encoder input. Each curve in the plot is the average over five seeds of the same experiments. We notice the higher overall attained reward with the proposed additions. The additional input allows the learning to optimize rewards like foot clearance whose target value is varied from one episode to another. At the beginning of training, the average reward for both terms reaches higher values than at the end. This is the effect of the reward curriculum that scales down the negative reward penalties and the velocity curriculum that samples a very small range of the velocity command vector at the start of training.

C. High-level policy study

The strength of the proposed approach is the ability to reuse the low-level policy for different high-level tasks. In this section, we verify that we are able to learn a high-level policy that adapts the low-level policy, learned on a flat terrain, in order to successfully traverse complex terrain that includes flat and rough plains, stairs, slopes and obstacles as shown in Figure 5.

Baseline comparison. We compare the results of the proposed hierarchic approach with a baseline end-to-end approach that learns to traverse the terrain from scratch with the height map state as input along with the low-level policy observation and reward function. However, we add the energy consumption cost term to the reward function for fairness of the comparison between the baseline and the proposed hierarchical approach. The training approach using the elevation map is inspired by the work done in [START_REF] Rudin | Learning to walk in minutes using massively parallel deep reinforcement learning[END_REF]. We also use a similar curriculum on the terrain in order to gradually introduce the robot to terrains with increasing difficulty. It is important to note that the baseline training also uses the same RMA technique to enable the transfer of the learned policies on the real robot.

Figure 7 shows a scatter plot featuring the power consumption as a function of measured linear velocity. We collect the data points of the plot by running the two policies (hierarchic approach and baseline) on the velocity commands V x = {0.0, 0.5, 1.0, 1.5, 2.0}[m/s] for ten seconds at each velocity over a flat terrain. The figure shows that the proposed approach leads to lower overall power consumption with less than the baseline, especially at higher velocities. The hierarchic policy also tracks the commanded velocity better than the baseline policy both in terms of accuracy and variance.

Figure 8 shows the adaptation of the command parameters when the robot has to climb stairs. We see the change from their default value (dashed line). The high-level policy raises the foot height target (Fh) and the position gains which helps the robot use more power to climb the stairs. The angles of the hip and knee joints are lowered from their default values which could explain the overall improved energy efficiency of the robot (seen in Figure 7), since standing on straighter legs requires less power.

Sample Efficiency. Our hierarchical learning method also needs fewer samples for training since it splits complex problems into simpler sub-problems. In our experiments, the baseline method that learns to cross terrains from scratch required 20000 training iterations which is around two billion samples. However, with the hierarchical approach we are able to learn a low-level policy with 5000 iterations and a high-level policy with 3000 iterations. This means that our approach requires less than half the number of samples that the baseline needs because the complex problem of learning locomotion is done in the low-level policy on a simple terrain.

D. Real robot transfer

On its own, the low-level policy can be successfully transferred to the Mini-Cheetah robot without exteroception. We attach a video showing the policy in action with different command parameters. The video first shows the low-level policy run with different desired feet height and online Fig. 7: Average power vs. measured velocity for the hierarchic and baseline policies.

Fig. 8: Some command parameters' values adapted by the high-level policy while climbing stairs. changing of the nominal pose. We can see the feet lifted differently and the robot changing its pose as it is moving. We also show multiple videos of the low-level policy traversing difficult surfaces ( going up and down stairs, going over pavements and crossing steep slopes on muddy grass terrain) in a very robust manner when using parameters similar to the one learned by the high-level policy in the structured terrains. The video attachment also shows that a policy trained with fixed sets of parameters completely fails to be robust over structured terrains. The transfer of the highlevel policy to the robot requires more instrumentation and handling exteroception (vision) that is beyond the scope of the current paper and will be the aim of a future work.

VI. DISCUSSION AND LIMITATIONS

We presented a hierarchical scheme for adapting learned quadruped locomotion. The main strength of the work lies in the versatile low-level policy training that can be reused for different tasks and in different environments and terrains. The low-level policy embeds several behaviour through the command parameters. However, having a single low-level policy could also make training harder since we have to find one policy that is optimal for all combinations of the command vector. It is possible that training low-level policy on specialized subspaces of the command vectors would produce more specialized diverse skills. For example, we could train one policy that specializes in lifting its feet high and another one that specializes in taking long steps, and then combine these two policies in an ensemble manner [START_REF] Peng | Mcp: Learning composable hierarchical control with multiplicative compositional policies[END_REF]. This is an interesting question to explore in our future research.

We chose a specific set of commands that can directly and indirectly control aspects of the learned behaviour so we can simply show how they affect the policy when varied. For future work, an interesting objective could be to expand the command parameters to include all parameters related to the reward coefficients and learned pattern. For example, while we have assumed that PD gains are fixed for all joints, depending on the type of pattern we could propose learning a different gain for each leg or joint.

VII. CONCLUSION

In this work, we argued for expanding the notion of the command in learned locomotion policies by actively setting targets for locomotion features such as the swing feet height, stepping length, initial pose and gains. We show that we are able to learn a locomotion policy that reacts to the targets set by the command parameters. This makes it possible to adapt the low-level policy for different tasks. For this effect, we propose a hierarchical RL setup where a high-level policy learns to adapt the parameters of the low-level policy for versatile locomotion. We also showed that this hierarchical setup speeds up the learning of new tasks and that adapted policies can be better in quality than policies learnt from scratch. With the low-level parameterized policies we made our real robot successfully traverse complex terrains on which the baseline policy fails. In the future we will add exteroception to the Mini-Cheetah in order to be able to deploy and test high-level policies on the real robot in more complex tasks than those now demonstrated with a parameterized low-level policy.

Fig. 2 :

 2 Fig.2: Policy and control architecture of the low-level policy (right) and the high-level policy (left). The low-level policy's state and commands are fed to the policy network that outputs joint displacement offsets. A PD controller turns the network actions into torques. The high-level policy get task related information in order to actively control the low-level behaviour by actively adjusting the command parameters.

Fig. 3 :

 3 Fig.3: The environment encoder E encodes a representation of the privileged information into z. The adaptation module A learns to regress the output of the encoder from the history of the observations and actions to predict ẑ.

Fig. 4 :

 4 Fig. 4: Front left foot height for different p max f,z targets (5, 10 and 15cm), when running the robot controlled by the lowlevel policy at a 1.0m/s forward velocity command.

Fig. 5 :

 5 Fig. 5: Left: Terrain types in the HRL experiments. Right: Height-map measurements around the robot.

Fig. 6 :

 6 Fig. 6: The average episodic reward per training iteration.The curves show two experimental setups: when additional information is added to the encoder network (yellow), and when it is not (purple).

TABLE I :

 I Reward function terms and coefficients. The total reward is a weighted sum of all the terms above.

TABLE II :

 II Ranges and dimensions of uniform noise for randomizing the dynamics and observations.

	Element	Range	Default
	Foot height target [cm]	[3.0 , 15.0]	7.0
	Joint angle deviation	[-1.0, -0.2]	-0.5
	Position gain	[17.0, 30.0]	20.0
		X: [-2.0, 2.0]	0.0
	Velocity reference [m/s]	Y : [-1.0, 1.0]	0.0
		z yaw: [-1.0, 1.0]	0.0
		Shoulder: [0.0, 0.1]	0.05
	q nominal [rad]	Hip : [-2.0, -0.1]	-0.8
		Knee: [1.3, 1.9]	1.6

TABLE III :

 III Default values of the command parameters and ranges in which the commands are sampled.

TABLE IV :

 IV Average foot step length as a function of the forward velocity command (angular velocity is set to zero) and joint angle deviation penalty weight.

Note the value of the derivative gain is coupled with the position gain, i.e., K d is chosen to be a scaled value of Kp as in[START_REF] Bogdanovic | Learning variable impedance control for contact sensitive tasks[END_REF].


Anubhav Gupta
email: agupta@yseop.com

Frédéric Rayar
email: frederic.rayar@univ-tours.fr

FRAG at the NTCIR-17 MedNLP-SC Task

Keywords: NLP, Multilingual, Transformers, Clinical texts

. Our approach involved finetuning a multilingual transformer-based model on the train set. The team ranked 3 𝑟𝑑 for English (SM-ADE-EN), German (SM-ADE-DE) and Japanese (SM-ADE-JA) based on Exact accuracy and Binary scores.

INTRODUCTION

Prescribed by doctors, medicines are part of daily life for many people throughout the world. Yet, there is always a risk involved in taking medication. One of these risks is an Adverse Drug Event (ADE). It is "an injury resulting from the use of a drug. Under this definition, the term ADE includes harm caused by the drug (adverse drug reactions and overdoses) and harm from the use of the drug (including dose reductions and discontinuations of drug therapy)" [START_REF] Nebeker | Clarifying adverse drug events: a clinician's guide to terminology, documentation, and reporting[END_REF].

An example of ADE mention, taken from the ADE Corpus v2 [START_REF] Gurulingappa | Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports[END_REF]: "An 11-day-old infant became lethargic and apneic after a single drop of brimonidine". Since ADE can be dangerous to patients and a source of morbidity and mortality, hospitals and doctors need a system to support them in monitoring ADE occurrences in a fast and scalable way. To do so, Natural Language Processing (NLP) has been leveraged and has shown promising results [START_REF] Murphy | Adverse drug event detection using natural language processing: A scoping review of supervised learning methods[END_REF].

Task description

The MedNLP-SC Social Media subtask addresses ADE detection from social media texts in four languages: Japanese (SM-ADE-JA), English (SM-ADE-EN), German (SM-ADE-DE) and French (SM-ADE-FR). There are 7965 synthetically generated tweets and we have to:

(1) identify the texts mentioning ADE, (2) label the text with at least one of the 22 ADEs observed.

The training corpus is imbalanced: for each language 68.58% of the data did not have an ADE. Among the texts in the corpus that mention ADE, 53.19% had single label and among them 11 classes appeared in 80% of the messages with the 'diarrhea' label at 17.28%.

One can find thorough details on the dataset in the task overview paper [START_REF] Wakamiya | NTCIR-17 MedNLP-SC Social Media Adverse Drug Event Detection: Subtask Overview[END_REF].

We approached the task as a multi-label classification task rather than an entity detection one.

RELATED WORK

Rawat et al. [START_REF] Rawat | Drug Adverse Event Detection Using Text-Based Convolutional Neural Networks[END_REF] used CNN to do binary classification (ADE vs. no ADE) on text extracted from MEDLINE. Zhang et al. [START_REF] Zhang | Adverse drug reaction detection on social media with deep linguistic features[END_REF] worked on binary classification. They trained a SVM on data scraped from Dai-lyStrength and Twitter. Wunnava et al. [START_REF] Wunnava | A Dual-Attention Network for Joint Named Entity Recognition and Sentence Classification of Adverse Drug Events[END_REF] used a dual-attention network to perform joint task of ADE classification and NER. Wu et al. [START_REF] Wu | Chinese-Named Entity Recognition From Adverse Drug Event Records: Radical Embedding-Combined Dynamic Embedding-Based BERT in a Bidirectional Long Short-term Conditional Random Field (Bi-LSTM-CRF) Model[END_REF] created a tool composed of BERT, bi-LSTM-CRF [START_REF] Lample | Neural Architectures for Named Entity Recognition[END_REF] to identify ADE as named entities in the unstructured section of Chinese ADR reports from the ADR monitoring center of Jiangsu Province in 2010-2016. There are also works on ADR classification [START_REF] Alhuzali | Improving classification of Adverse Drug Reactions through Using Sentiment Analysis and Transfer Learning[END_REF][START_REF] Alimova | Entity-Level Classification of Adverse Drug Reaction: A Comparative Analysis of Neural Network Models[END_REF][START_REF] Foufi | Automatic Classification of Discharge Letters to Detect Adverse Drug Reactions[END_REF][START_REF] Sakhovskiy | Multimodal model with text and drug embeddings for adverse drug reaction classification[END_REF][START_REF] Wang | Adverse Drug Reaction Post Classification with Imbalanced Classification Techniques[END_REF]. COLING hosted a similar shared task, SMM4H Task 1a [START_REF] Weissenbacher | Overview of the Seventh Social Media Mining for Health Applications (#SMM4H) Shared Tasks at COL-ING 2022[END_REF], where the tweets in English containing ADE had to be identified. Most of the participants used BERT based models.

METHODS

First, we added a boolean column to the English dataset to state if there was at least 1 ADE or not. Then we separated 10% of the messages as validation set using scikit-learn's [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF] stratified train_test_split. We used the train-ids of English train and validation set to split the corpus for other languages. We combined train set of all the languages into a single dataset and we did the same with validation set. We fine-tuned a multi-lingual BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], bert-base-multilingual-cased1 , and a multi-lingual RoBERTa [START_REF] Conneau | Unsupervised Cross-lingual Representation Learning at Scale[END_REF], xlm-roberta-base2 using Huggingface [START_REF] Wolf | Transformers: State-of-the-Art Natural Language Processing[END_REF].

The parameters that have been used to fine-tune the two models are presented in Table 1. Except for the batch size, all the parameters are the huggingface defaults. At every epoch we computed the f1 for binary classification (ADE vs. no ADE). Around 4th epoch the eval loss for both models start rising and the eval f1 remains flat (See Figure 1 and Figure 2).

For the first submission, we fine-tuned the multi-lingual BERT with the default parameters on all four languages combined without any preprocessing. We fine-tuned xlm-roberta in the same manner for the second. We chose the number of epochs arbitrarily as 6 for both submissions. Other parameters are those mentioned in Table 1.

CONCLUSIONS

We presented a finetuned multi-lingual RoBERTa model to identify tweets mentioning ADE. These social media texts were generated using a T5 model. Our model was an exact replica of the baseline (XLM-R_all) except it was finetuned to 6 epochs instead of 10. This small variation in the training made a big difference in the final ranking. The fact the models (of all the participants) scored the highest for Japanese and the lowest for French on all metrics is worth exploring. Lastly, it would be interesting to see how these models trained on synthetic data compare against the models trained on human generated social media texts.

Figure 1 :

 1 Figure 1: BERT Fine-tuning

Figure 2 :

 2 Figure 2: XLM-RoBERTa Fine-tuning

Table 1 :

 1 Fine-tuning parameters.

		bert-base-multilingual-cased xlm-roberta-base
	max length 3	128	128
	learning rate	2e-5	2e-5
	weight decay	0.01	0.01
	epochs	10	10
	batch size	16	32

Table 2 :

 2 Binary Scores (ADE vs. no ADE) results.

	Team	Japanese English German French
	Srcb	0.881	0.872	0.873	0.869
	Frag (Submission 1)	0.83	0.82	0.82	0.82
	Frag (Submission 2)	0.868	0.855	0.846	0.845
	XLM-R_all	0.850	0.846	0.815	0.828

Table 3 :

 3 (Full) Per Label Scores results.

	Team	Japanese English German French
	Srcb	0.910	0.905	0.908	0.902
	Frag (Submission 1)	0.87	0.87	0.86	0.86
	Frag (Submission 2)	0.900	0.885	0.880	0.874
	XLM-R_all	0.885	0.876	0.852	0.862

Table 4 :

 4 Exact Match Accuracy results.

	Team	Japanese English German French
	Srcb	0.878	0.869	0.864	0.866
	Frag (Submission 1)	0.821	0.817	0.804	0.801
	Frag (Submission 2)	0.858	0.841	0.833	0.828
	XLM-R_all	0.837	0.828	0.803	0.806

https://huggingface.co/bert-base-multilingual-cased

https://huggingface.co/xlm-roberta-base

https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers. PreTrainedTokenizer.__call__.max_length