Michel Aractingi

Pierre-Alexandre Leziart

Thomas Flayols

Julien Perez

Tomi Silander

Philippe Souères

Learning to Adapt the Trotting Gait of the Solo Quadruped

Predefined gait patterns for quadruped locomotion can hardly be optimal in all situations with regard to stability, cost of transport and velocity tracking error. Hence, in this work, we tackle the challenge of adapting a predefined trotting gait, implemented in the model-based controller of Solo, to optimize both energy consumption and velocity tracking. To this end, we propose a model-free reinforcement learning method for adapting the timings of the contact/swing phases for each foot. The learned agent augments a control pipeline that was previously developed for the Solo robot. We also propose to use a self-attention mechanism over the history of states in order to extract useful information for adapting the gait. Through a comprehensive set of experiments, we demonstrate how, compared to the nominal gait, our method significantly reduces energy consumption, better tracks the desired velocity, and makes it possible to reach higher speeds. A video of the method is found at https://youtu.be/ykbDUyASXs4.

I. INTRODUCTION

Quadrupedal locomotion has become an increasingly popular topic of robotics and artificial intelligence research [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], [START_REF] Peng | Learning agile robotic locomotion skills by imitating animals[END_REF], [START_REF] Da | Learning a contact-adaptive controller for robust, efficient legged locomotion[END_REF], [START_REF] Tsounis | DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning[END_REF]. Unlike wheeled robots, legged robots can traverse challenging terrains [START_REF] Lee | Learning quadrupedal locomotion over challenging terrain[END_REF]. Recently, several robotic platforms have been developed for mastering quadruped locomotion [START_REF] Hutter | Anymal -a highly mobile and dynamic quadrupedal robot[END_REF], [START_REF] Grimminger | An open torque-controlled modular robot architecture for legged locomotion research[END_REF], [START_REF] Katz | Mini cheetah: A platform for pushing the limits of dynamic quadruped control[END_REF]. Robots like Spot, Mini Cheetah [START_REF] Katz | Mini cheetah: A platform for pushing the limits of dynamic quadruped control[END_REF], HyQ [START_REF] Semini | Design of hyq -a hydraulically and electrically actuated quadruped robot[END_REF], ANYmal [START_REF] Hutter | Anymal -a highly mobile and dynamic quadrupedal robot[END_REF] or Laikago [START_REF] Wang | Laikago Pro, Unitree Robotics[END_REF] provide reference test benches for designing control approaches. Solo [START_REF] Grimminger | An open torque-controlled modular robot architecture for legged locomotion research[END_REF] is a more recent alternative which provides a reliable, low-cost, open-access quadruped within the Open Dynamic Robot Initiative. Previous work on these platforms has managed to develop control schemes that feature dynamic and robust locomotion [START_REF] Di Carlo | Dynamic Locomotion in the MIT Cheetah 3 Through Convex Model-Predictive Control[END_REF], [START_REF] Kim | Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control[END_REF], [START_REF] Bellicoso | Dynamic Locomotion Through Online Nonlinear Motion Optimization for Quadrupedal Robots[END_REF].

Many classical methods propose complex control architectures composed of sequences of blocks with hand-tuned parameters. Each block outputs a solution to one part of the problem based on dynamic models and estimated states. Among them, the control scheme developed in [START_REF] Kim | Highly dynamic quadruped locomotion via whole-body impulse control and model predictive control[END_REF] for Mini Cheetah constitutes a reference in the domain. It combines Model Predictive Control (MPC) and Whole-Body Control (WBC). The MPC is tasked with long horizon planning based on the under-actuated part of the dynamics while the WBC handles low-level control at finer timesteps. This hybrid architecture proved to be stable while achieving a record running speed for this robot. A similar control pipeline, including simplifying solutions for the computation of the WBC, was used for Solo in [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF].

Model-free reinforcement learning (RL) makes it possible to learn controllers without hand-tuning such control 1 LAAS-CNRS, Université de Toulouse, CNRS, Toulouse, France 2 NAVER LABS Europe, 6 chemin de Maupertuis, Meylan, 38240, France * Corresponding author: michel.aractingi@naverlabs.com blocks. RL methods have shown some success in learning locomotion tasks [START_REF] Peng | Learning agile robotic locomotion skills by imitating animals[END_REF], [START_REF] Peng | Terrain-adaptive locomotion skills using deep reinforcement learning[END_REF], [START_REF] Peng | DeepLoco: Dynamic locomotion skills using hierarchical deep reinforcement learning[END_REF], particularly in adapting to new situations where classic controllers fail to work satisfactorily [START_REF] Song | Rapidly adaptable legged robots via evolutionary metalearning[END_REF]. A hierarchical RL approach was used in [START_REF] Peng | DeepLoco: Dynamic locomotion skills using hierarchical deep reinforcement learning[END_REF] to train policies at two levels of control: planning and execution. Then, in [START_REF] Peng | Terrain-adaptive locomotion skills using deep reinforcement learning[END_REF] a multiple actor-critic approach was introduced for training several policies that specialize in different maneuvers such as running, skipping and jumping. However these learning methods require the availability of privileged information about the terrain, which is only feasible when running experiments in simulation. Moreover, RL methods usually require lots of data, which limits training to simulated environments, hence creating a problem of transferring learned skills from simulation to hardware [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF]. Learning locomotion policies for real platforms has also been studied. For example, reference trajectories collected from motion capture recordings of a real dog were used to train a policy to mimic animal behaviours in [START_REF] Peng | Learning agile robotic locomotion skills by imitating animals[END_REF]. In [START_REF] Hwangbo | Learning agile and dynamic motor skills for legged robots[END_REF], a model of the actuation dynamics of the real ANYmal was used in simulation to produce policies that adapt to actuator noise. A meta learning approach was used in [START_REF] Song | Rapidly adaptable legged robots via evolutionary metalearning[END_REF] to adapt to the noise in the dynamics when deployed on the real system. In all cases, more work and data were required to bridge the sim2real gap.

Combining model-based control and RL has been proposed before. Such methods often replace or augment certain control blocks with function approximators such as neural networks that can be trained to optimize decisions that are difficult to model. Recently, a policy that learns to choose the next contact sequence from a set of predefined contacts was proposed in [START_REF] Da | Learning a contact-adaptive controller for robust, efficient legged locomotion[END_REF]. In [START_REF] Lee | Learning quadrupedal locomotion over challenging terrain[END_REF], the authors proposed learning a swing phase delta-variable, which decides the contact phase of the foot, along with the displacement of the step position. A method to learn gait transitions by learning the gait schedule as a function of the reference velocity was introduced in [START_REF] Yang | Fast and efficient locomotion via learned gait transitions[END_REF]. Modification of the guided policy search, where learning is guided by an MPC with the objective of optimizing the control Hamiltonian, was proposed in [START_REF] Carius | MPC-Net: A First Principles Guided Policy Search[END_REF]. Learning of high-level decision variables for a low-level MPC was also proposed to adapt to difficult situations [START_REF] Song | Learning High-Level Policies for Model Predictive Control[END_REF]. Model-based motion planning was used in [START_REF] Tsounis | DeepGait: Planning and Control of Quadrupedal Gaits Using Deep Reinforcement Learning[END_REF] to train a gait planner and gait executioner policies when moving on nonflat terrains. With learning, one can obtain context-dependent control parameters that are automatically adapted to better generalize the decisions. Moreover, limiting the use of RL to specific sub-problems of the control naturally requires less data than learning the full control from scratch.

In this paper, we study the problem of how to adapt the Fig. 1: Description of the controller. A user commands the reference velocity and gait. The planner outputs a gait matrix G t which is updated by the RL policy. Considering this updated value, the MPC plans contact forces for the ground feet while the swing feet trajectories are determined by a planner. The information from both blocks are sent to the whole-body control that outputs reference torques and reference joint positions and velocities. An impedance controller finally computes the joint torques based on encoders measurements.

gait of a quadruped in order to improve its spent energy and velocity tracking. Our motivation comes from the fact that humans and animals can lower their cost of transport by adapting their gait depending on their velocity [START_REF] Hoyt | Gait and the energetics of locomotion in horses[END_REF]. Our main contribution is a method that uses a model-free deep RL policy to adapt the timings of the contact/swing phases of each foot independently. We use this policy to augment a model-based controller that was previously developed to endow the Solo quadruped with trotting, walking and static gait capabilities [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF]. We show that the proposed method can adapt the nominal trotting in different situations between a walking trot pattern to a rapid trot. This improves the energy consumption and reference velocity tracking of the controller. Another contribution is the policy architecture. We argue that the prediction of events that would impact future decisions can be improved by considering a history of states. To take advantage of this history, we propose using self-attention mechanisms [START_REF] Vaswani | Attention is all you need[END_REF] that are commonly used in language tasks to handle sequences of inputs. The experiments demonstrate improvements in the convergence and performance of the learning.

The paper is organized as follows. Description of background information about the controller and RL is summarized in section II. The method and experimental results are outlined in section III and section IV respectively.

II. PRELIMINARIES

The robotic platform used in this paper is the Solo12 quadruped, a 12 degrees of freedom (DoFs) version of the Solo8 quadruped, introduced in [START_REF] Grimminger | An open torque-controlled modular robot architecture for legged locomotion research[END_REF], with three actuators per leg. The task is to follow a user-defined reference velocity. The RL approach in this paper is complementary to the model-based controller proposed by [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF], with minor modifications to the foot trajectory generator, so that the planned trajectory of the swing feet can be modified and adapted by the RL agent on the fly. In the following section we briefly describe the controller used and the RL formalism.

A. Nominal Controller Architecture

The nominal controller is centered around two main control blocks: a MPC and a WBC. The MPC computes a sequence of ground contact forces based on the centroidal dynamics model and the location of current and future footholds. Its objective is to have the body track the reference velocity. The WBC takes as inputs the desired contact forces, for the feet in stance phase, and the desired motion of feet in swing phase and outputs desired torques, positions and velocities for the 12 actuators. The final torque values sent to the actuators are the WBC torques values to which are added the feedback torques of a PD controller. While the WBC solves an instantaneous problem and provides lowlevel commands at high frequency (500 Hz), the MPC plans over a prediction horizon knowing the future footholds but at lower frequency (50 Hz) due to computational requirements.

The trajectory of a foot during swing phase is planned using polynomial functions to link its current position to a target position on the ground. The foot trajectory generator outputs a reference position, velocity and acceleration at each time step of the swing phase. These values are used by the WBC as references for the inverse kinematics and torques computation. A footstep planner outputs the target locations of footsteps using heuristics that rely on the gait, the current and desired body velocities to be tracked. The controller has shown successful trials on the real Solo12 platform. The controller works well for gaits where two or more feet are in contact with the ground, i.e., it supports various trotting, walking and static gaits. We refer to the controller's main paper for more details [START_REF] Léziart | Implementation of a Reactive Walking Controller for the New Open-Hardware Quadruped Solo-12[END_REF].

In general, a gait is defined as a periodic pattern of limb movements made during locomotion, both for robots and animals. Animals often have the ability to adapt their gait depending on the environmental conditions and desired characteristics such as speed, stability, manoeuvrability or energy efficiency. Formally, Solo's trotting sequence is determined at time t by a binary gait matrix G t ∈ {0, 1} M ×4 that Fig. 2: Multi-headed self-attention layers, with h = 8 heads, are used as the base of the policy. The actions modify the contact sequence given by the oscillations of each leg. The bottom plot shows an example of the nominal sequence of contact/swing in bold. Depending on the current contact state of foot j, the dotted line draws the new oscillation after the shifts in timings given by the policy are taken into account.

describes the planned feet contacts for the incoming M time steps. Column j of the matrix describes the future ground contact states for foot j. The i th row of G t describes the ground contact states of the four feet that the controller should consider for the i-th time step of the MPC prediction horizon, i.e, at time t + i × ∆t mpc , where ∆t mpc is the length of one MPC time step.

A trotting gait is defined by having two diagonally opposite feet in contact with the ground while the other two are in a swing phase. The policy can adapt the gait period and the duration of both stance and swing phases within one period, and therefore the potential overlap of those phases between the feet. These modifications can lead to variations of the hard-coded trot that have different properties in terms of speed, energy consumption, reactivity and robustness. The controller's gait matrix is pre-defined to trotting with a period of 0.32s. Our work offers a method to adapt the value of the period by modifying the timings of the contact/stance phases.

B. Reinforcement learning formalization

We model the RL learning environment as a Markov decision process (MDP) [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]. An MDP is defined by the tuple (S, A, R, T , P 0 ), where S is a set of states and A is a set of actions. Taking an action a in a state s yields a stochastic reward, expectation of which is defined by a function R : S × A → R. The environment dynamics is described by a conditional transition probability distribution T : S × A × S → R + , with the interpretation that T (s, a, s ) = p(s t+1 = s |s t = s, a t = a) is the probability (density) that the next state is s given that the current state is s and that action taken is a. P 0 is the initial state probability distribution. In RL setting, only spaces S and A of the MDP are known by the learning agent. By acting in the environment, the learning agent receives samples from the reward and state transition distributions.

We define a policy π θ (h, a) = p θ (a t = a | h t = h) parameterized by θ that gives the probability of taking an action a given a state-action history h = (s 0 , a 0 , s 1 , a 1 , ..., s t ). The learning objective is to find the parameters θ of the policy that maximize the value of the expected discounted sum of rewards J(θ) := E π θ ,P0,T [ H t=0 γ t R(s t , a t )], where H is the horizon of the episode and γ ∈ [0, 1] is the discount factor. Section III-B defines the core components of the MDP, i.e., state space, action space and reward function, that are used throughout this paper.

III. METHOD

We propose a method to learn how to adapt the contact patterns for each foot in order to improve the control performance. We choose to formalize the task as a sequential decision problem and solve it with RL techniques. One can write a trajectory optimization program, based on an optimal control solution, to adapt the contact phases of the gait [START_REF] Winkler | Gait and trajectory optimization for legged systems through phase-based endeffector parameterization[END_REF]. However, such a method would require intensive computations at each MPC cycle, whereas with a policy learned with deep RL the decision is made with a single forward pass through the network.

A. Controlling the gait timings

While we could in theory directly control the binary indicators in the gait matrices G t , this would create an intractable action space with 2 4M actions, most of which would not correspond to any relevant locomotion. Since the nature of quadrupedal locomotion is periodic, we propose to view the creation of the gait matrices through parameterized oscillation functions. We define a base oscillation f (t; τ 0 , τ 1 ) where τ 0 < τ 1 are the timings for which a change in value occurs:

f (t; τ 0 , τ 1 , T ) = 0, if τ 0 < t mod T < τ 1 , and 1, otherwise.
Given C(j), a binary indicator of the current contact state of foot j ∈ {1 . . . 4}, the oscillation function f j (t) : R + → {0, 1} describes the future contact states of foot j as a function of time t. The oscillation is parameterized by two switch timing parameters τ j s and τ j c , which indicate the beginning of the swing and stance phases of foot j respectively. f j is then defined as1 :

f j (t; τ j s , τ j c ) = δ 1,C(j) * f (t; τ j s , τ j c , T j ) + δ 0,C(j) * (1 -f (t; τ j c , τ j s , T j ))
, where the period length is defined as T j = max(τ j s , τ j c ). In order to control the four oscillation functions we introduce a 4 × 2-dimensional continuous action space, A = {(a 1 , a 2 , a 3 , a 4 )} with a j = (∆τ j s , ∆τ j c ) ∈ R 2 . For each foot j there are two actions that define the displacement with respect to the timings of the nominal trotting gait, τ j n = (τ j s,n , τ j c,n ), that are hardcoded in the controller. Controlling the deltas of the timing values rather than the values directly is the key for the method to work as it reduces the exploration space.

The gait matrix can be created by assigning G t [i, j] = f j (i * ∆t mpc ; τ j n + a j t ), where i ∈ {1, . . . , M } and j ∈ {1, 2, 3, 4}. We will next describe the nature of the state space, reward function and policy architecture that make the RL agent learn to effectively adapt contact timings.

B. MDP definition

The MDP is defined over fixed discrete timesteps. The RL policy runs at a frequency of 10Hz. We found that this frequency gives the policy enough time for executing an action and receiving a useful learning signal, while keeping its reactivity in adapting gait sequences quickly enough.

State definition: We define the observation O t ∈ R d=65 to contain proprioceptive information o t ∈ R 57 about the robot at time t along with the last eight-dimensional command a t-1 . The elements composing the observation are the base height and orientation q base ∈ R 4 , base velocity qbase ∈ R 6 , joint angles q ∈ R 12 and velocities q ∈ R 12 , feet positions relative to the body frame p f eet ∈ R 12 and the current and past gait contact sequences, both four-dimensional binary vectors. The velocity reference command qref ∈ R 3 is added to the observation so that the policy is aware of the command. The observation is thus constructed O t = (o t , a t-1 ).

The history of the last N = 16 observations are concatenated to construct the state s t ∈ R N ×d . We argue that having a history of observations, especially when the period between each RL action is very short, is necessary for decision making in order to detect changes in the environment dynamics.

Reward definition: We design a basic reward function based on three terms: (1) a positive constant c per timestep to encourage the policy not to commit any actions that would end the episode early, (2) the squared distance between the commanded velocity and the velocity of the robot and (3) an energy penalty term to encourage the policy to learn actions that save energy.

Our reward function is similar to the ones proposed by [START_REF] Da | Learning a contact-adaptive controller for robust, efficient legged locomotion[END_REF], [START_REF] Yang | Fast and efficient locomotion via learned gait transitions[END_REF]. However, we propose using the energy instead of the torques magnitudes. As the energy is a function of the torques and joint angles, optimizing the torques magnitudes, while important from a control perspective, does not equate to minimizing the energy under certain joint angles. The energy for joint l at t is the integration of power P l spent over the last RL timestep. The total energy at time t is summed over all joints:

E t = 12 l=1 t t =t-T RL P l,t dt ,
where T RL is the time period between each RL step. The reward at time t is then defined to be:

R t = c - t t =t-T RL qref,t -qbase,t 2 dt -λE t ,
where λ is the coefficient that balances the importance of energy conservation in the reward function. Throughout the experiments we used λ = 10 and c = 1.0.

Policy design: In order to take advantage of the sequence of observation-action pairs, we utilize self-attention layers in our policy model [START_REF] Vaswani | Attention is all you need[END_REF]. Self-attention has been very successfully used in language tasks where the input is sequential. We argue that with self-attention the model can focus on parts of the state that indicate changes in the dynamics. Moreover, it also facilitates coordination of the different legs by contextualizing inputs from one another. To our knowledge, this is the first time self-attention is used for learning locomotion. In Section IV-D, we show that using a self-attention model yields greater rewards with less samples than a standard stack of fully-connected layers.

IV. EXPERIMENTS

In this section, we present the experimental results of training a policy to adapt the trotting gait of the Solo12 quadruped with the proposed action space. The main questions we answer are: (1) can the policy learn to manipulate the trot so that the robot tracks the reference velocity while optimizing the energy consumption? and (2) What is the effect of using a self-attention mechanism in the policy network?

Environment:

The training process takes place in a synthetic environment. The simulation is based on PyBullet [START_REF] Coumans | Pybullet, a python module for physics simulation for games, robotics and machine learning[END_REF] that uses the Bullet physics engine for simulating rigid body dynamics and detecting collisions. The Pinocchio library [START_REF] Carpentier | The pinocchio c++ library -a fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives[END_REF] is used for low-level dynamics and kinematics, e.g., to get the positions of the feet in the body frame of the robot. Fig. 3: Top: achieved velocity of the robot when following a predefined velocity plan (dashed line). The policy is able to get the robot closer to the desired reference in most cases. Middle: the blue dashed line represents the nominal trotting frequency. The red line indicates the average frequency of all legs as adapted by the policy. The frequency of stepping is slowed down and sped-up to accommodate the reference velocity. Bottom: snapshots of the achieved gaits at different levels of the run.

Implementation details: As mentioned before, a selfattention mechanism is used in the architecture of our policy-network. We use the encoder layer of the Transformer architecture as our base model [START_REF] Vaswani | Attention is all you need[END_REF]. The output of the encoder is propagated through a multi-layer perceptron that outputs the actions. The actions are quantized into a Multi-discrete space. As the contact patterns are quantized over the timestep of the MPC, the shift in the timings are multiples of the MPC timestep. Therefore, the action space is implicitly discrete. We use the Proximal Policy Optimization algorithm (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] for learning the optimal policy. PPO is robust for a wide range of RL tasks [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. We found that pretraining the policy representation on a torque prediction task improves the overall performance of the RL (more details in Section IV-D).

A. Adapted gaits and velocity tracking

Our experiments demonstrate that we can learn a policy that adapts the nominal trotting gait for different reference velocities. At zero velocity, we obtain an optimal energy saving policy with a static gait where all feet are in contact with the ground. As the commanded velocity increases the gait evolves into a walking trot at low velocities. At high velocities the policy adapts the timings to output a fast rapid trotting which is more costly in terms of energy, but necessary to follow the velocity command with low error. Examples of the resulting gaits, due to the policy adaption of the nominal trotting, are shown in Figure 3 2 .

The learned policy shows higher fidelity in tracking the commanded velocity at each moment. In the middle figure, We observe a clear decrease in the frequency, making the stepping slower, for lower velocities. For high velocities the frequency is increased, thereby making the stepping faster which helps stabilize the base and follow the reference velocity.

B. Energy efficiency

We run the learned policy five times with different random seeds in a setup where the reference velocity is gradually increased starting from 0 m/s. Figures (4a,4b) show the trade-off between the average episodic energy consumption and velocity error for the learned policy vs. the nominal controller with fixed gait. We observe the improvement in energy consumption particularly at low velocities where the dynamic movement of the nominal trot is not necessary. At higher velocities, the velocity error is lower for the learned agent while still being comparable in energy efficiency. The lower velocity error is due to the policy learning a rapid trot that tracks higher velocities better.

At high velocities the energy consumption of the nominal controller and of the learned policy appear to be almost equal. However, it is important to note that Figure 4a is plotted as a function of the reference commanded velocity 4b, the model-based controller exhibits higher velocity error when the energy consumption is similar to the policy. Figure 3 (top) confirms that at high velocities the robot controlled by the model-based controller is slower than when controlled by the learned policy. Therefore, while the energy consumption is comparable for high reference velocities, for actual realized velocities the robot controlled by the learned policy consumes less energy than the one controlled by the nominal controller.

C. Maintaining stability at higher speeds

We found that the controller with the nominal gait can reach the forward velocity up to 1.1 m/s before failing and falling (see Figure 4b). The proposed learned policy was able to break that limit and achieve velocities up to 2.5m/s. The experiments were conducted by gradually increasing the reference velocity over 2.0m/s. This implements a simple linear curriculum over the difficulty of the task. The result is a very rapid trot with period around 0.12s-0.16s. The attached video shows the difference between the nominal gait that fails and the gait adapted by the policy that is successful in stabilizing the robot at high speeds.

D. Ablation studies

In this section, we present ablations of the policy architecture and of the method of pretraining the representation using torque prediction. We illustrate the advantages of using our proposed method over standard approaches. Throughout this section, we use a task where the agent is simply expected to learn how to stay still and balanced. The agent is rewarded for completing the task successfully without falling and it is penalized for its energy consumption and velocity error.

Policy architecture: We train a policy using three different setups: (1) The proposed self-attention based policy, (2) a feed-forward neural network policy where the input is a sequence of N + 1 last states, and (3) a feed-forward neural network policy without history of the last N states. Figure 4c shows the expected cumulative reward during the training process for each setup. By using self-attention we attain higher rewards with less training steps than using standard neural network. For the zero reference velocity, the proposed policy yields a static gait with all feet staying in contact with the ground thus conserving energy. The other policies are not able to reach such a solution in the given number of training iterations, but they settle for a dynamic gait. This explains their overall lower cumulative reward.

Pretraining for torque predictions: Pretraining neural networks to improve the overall sample efficiency and performance of the model is a common practice in computer vision and NLP. Therefore, we propose to pretrain our model to predict the torques using a fixed gait from the standard controller. We then use the pretrained model to initialize the policy network for learning the gait timing control task. Figure 4d illustrates the improvement in sample efficiency when using pretrained initialization. The overall asymptotic performance, in terms of average reward, is also higher. Without the pretrained representation many experiments with different seeds failed to converge. The pretrained representation stabilizes the training process and decreases the number of failed training runs.

V. CONCLUSION

We proposed a model-free RL method that adapts the nominal trot pattern of a model-based controller designed for the locomotion of the Solo quadruped. The proposed action space consists of displacements of the contact/swing timings for each leg independently. We propose using a self-attention mechanism and pretraining to improve overall performance and sample efficiency. Our experiments demonstrate that the learned policy is able to adapt the gait according to the reference body velocity. The resulting behaviour conserves energy better than the nominal controller at low velocities and tracks the commanded speed better at high velocities. With the learned policy our robot is able to reach higher speeds than with the original controller. In the future we plan to investigate controllers that implement other gaits like galloping and bounding. After these promising results with a model-based controller that is already proven to work on the real Solo robot, we are currently working on testing the method on the real platform.

  (a) Energy consumption. (b) Velocity error (c) Policy architecture. (d) Pre-training.

Fig. 4 :

 4 Fig. 4: Figures (a, b) are comparisons between the learned policy and the nominal controller at test time over an increasing forward velocity reference. (a) Plot of the average energy consumption per episode. (b) Plot of velocity error. The nominal trotting fails after 1.1 m/s. The Training curves in (c,d) are averaged over five random seeds. (c) Average reward for different policy architectures. (d) Effect of pre-training on a torque prediction task.

The mod refers to the modulus operation to indicate that after period T j the time resets to zero. δ i,j refers to the Kronecker delta, i.e., δ i,j = 1 if i = j else 0.

These views are snapshots of one of the policy test runs that are presented in the attached video in the supplementary material.