Evelyn D'elia

Jean-Baptiste Mouret

Jens Kober

Serena Ivaldi

Automatic Tuning and Selection of Whole-Body Controllers

Designing controllers for complex robots such as humanoids is not an easy task. Often, researchers hand-tune controllers, but this is a time-consuming approach that yields a single controller which cannot generalize well to varied tasks. This work presents a method which uses the NSGA-II multi-objective optimization algorithm with various training trajectories to output a diverse Pareto set of well-functioning controller weights and gains. The best of these are shown to also work well on the real Talos robot. The learned Pareto front is then used in a Bayesian optimization (BO) algorithm both as a search space and as a source of prior information in the initial mean estimate. This combined learning approach, leveraging the two optimization methods together, finds a suitable parameter set for a new trajectory within 20 trials and outperforms both BO in the continuous parameter search space and random search along the precomputed Pareto front. The few trials required for this formulation of BO suggest that it could feasibly be applied on the physical robot using a Pareto front generated in simulation.

I. INTRODUCTION

Despite recent advances in robotic control, researchers still encounter a myriad of difficulties when controlling humanoid robots. Ideally, a humanoid robot should have versatile capabilities and should be able to complete manual labor that is grueling or even unsafe for humans. However, since humanoid robots are extremely complex and are meant to be capable of performing vastly different trajectories such as locomotion or manipulation, it is difficult to design a stable, accurate controller, and it is also difficult to find controllers that are transferable. Our approach addresses both of these problems.

One way to solve this problem is by breaking the control down into a set of tasks of varying importance, each with a gain parameter to define its behavior. This is called a task priority-based whole-body control (WBC) approach, and can be implemented with either strict priorities [START_REF] Nakamura | Task-priority based redundancy control of robot manipulators[END_REF], [START_REF] Siciliano | A general framework for managing multiple tasks in highly redundant robotic systems[END_REF], [START_REF] Sentis | Control of free-floating humanoid robots through task prioritization[END_REF], which ensure more important tasks are completed at the expense of less important ones, soft priorities [START_REF] Salini | Synthesis of complex humanoid whole-body behavior: A focus on sequencing and tasks transitions[END_REF], which allow a continuum of priority level, or both [START_REF] Liu | Whole-body hierarchical motion and force control for humanoid robots[END_REF]. The priority-based formulation streamlines controller design, but still requires the user to choose task parameters that perform well. One typical way to choose these parameters is by hand-tuning them, but since this method is timeconsuming, it often yields only one controller to be used This work was supported by the European Union's Horizon 2020 Research and Innovation Programme under Grant Agreement No. 731540 (project AnDy) and No. 101070596 (project euROBIN) and partly funded by the CPER project "CyberEntreprises" and the CPER project SCIARAT. 1 Inria, University of Lorraine, CNRS, Loria, Nancy, France 2 Delft University of Technology, Delft, Netherlands 3 Artificial and Mechanical Intelligence Lab, Italian Institute of Technology, Genoa, Italy evelyn.delia@iit.it for many trajectories. The main drawback of this method is that a single "robust" controller compromises the quality of individual trajectories. Due to the inconvenience of hand-tuning WBC parameters, many recent approaches utilize learning algorithms to automatically tune them. Some such approaches leverage programming by demonstration to learn task parameters for manipulators [START_REF] Armesto | Efficient learning of constraints and generic null space policies[END_REF], [START_REF] Perico | Combining imitation learning with constraint-based task specification and control[END_REF], [START_REF] Mronga | Learning context-adaptive task constraints for robotic manipulation[END_REF]. The drawbacks of these methods are that they require time to carry out demonstrations. Conversely, an imitation learning approach proposed by Silvério et. al. [START_REF] Silverio | Learning task priorities from demonstrations[END_REF] is implemented on the humanoid COMAN, using simulation-generated demonstrations, but validation of this method is limited to manipulation trajectories.

Instead of using demonstrations to guide learning, optimization algorithms can be used to find suitable task parameters. In most cases of learning on a real robot, it is beneficial to first optimize controllers in simulation. One class of optimization algorithms, evolutionary algorithms, are useful in robotics when the search is in continuous space and gradient-based optimization is impossible. One well-known and often used evolutionary method is CMA-ES [START_REF] Hansen | Completely derandomized selfadaptation in evolution strategies[END_REF]. The authors of [START_REF] Dehio | Multiple task optimization with a mixture of controllers for motion generation[END_REF], [START_REF] Modugno | Safe trajectory optimization for whole-body motion of humanoids[END_REF] and [START_REF] Gomes | Humanoid whole-body movement optimization from retargeted human motions[END_REF] employ CMA-ES to search for optimal trajectory parameters in the form of basis function weights for simulated tasks on the child-sized humanoid iCub [START_REF] Metta | The iCub humanoid robot[END_REF]. The approach of [START_REF] Modugno | Safe trajectory optimization for whole-body motion of humanoids[END_REF] uses a two-step optimization, the first of which is unconstrained CMA-ES to find a feasible starting point for the second step. On the other hand, authors of [START_REF] Dehio | Multiple task optimization with a mixture of controllers for motion generation[END_REF] initialize with hand-tuned parameters, while those of [START_REF] Gomes | Humanoid whole-body movement optimization from retargeted human motions[END_REF] skip the bootstrapping step by using retargeted recorded human trajectories to initialize the optimization problem. Both of these studies highlight an important weakness of CMA-ES: it must be initialized within the feasible region to find a solution. Also, as a single-objective method, it cannot individually optimize multiple goals at once.

Another evolutionary algorithm, Non-dominated Sorting Genetic Algorithm II (NSGA-II) [START_REF] Deb | A fast and elitist multiobjective genetic algorithm: NSGA-II[END_REF], a multi-objective optimization (MOO) algorithm which solves the issues associated with CMA-ES, is employed by Penco et al. [START_REF] Penco | Learning robust task priorities and gains for control of redundant robots[END_REF] to create a set of simulation-trained Pareto-optimal controllers, which are then used to decrease the number of real-robot tests required. However, since this method trades off only two objectives, accuracy and stability, the results lack diversity. In this paper we aim to use NSGA-II with one objective per training trajectory, to allow the final set of controllers to be more varied.

Using a pre-generated set of possible solutions as the search space, a single-objective learning algorithm can be used to transfer Pareto solutions to new trajectories that are not part of the MOO training set, and choosing one that is data-efficient, as opposed to a big-data deep learning approach, leaves open the possibility of applying the approach on the real robot. One data-efficient class of learning algorithms, called policy search (PS) [START_REF] Deisenroth | A survey on policy search for robotics[END_REF], [START_REF] Chatzilygeroudis | A survey on policy search algorithms for learning robot controllers in a handful of trials[END_REF], is very promising in the context of real-world robot learning as a method that learns a successful controller with as little data as possible. In particular, the Bayesian optimization (BO) algorithm requires fewer trials than other PS methods [START_REF] Chatzilygeroudis | A survey on policy search algorithms for learning robot controllers in a handful of trials[END_REF].

Prior information has the potential to speed up learning drastically. Some recent work with BO encodes prior knowledge within the Gaussian process (GP) kernel function. For example, the Behavior-Based Kernel (BBK) introduced in [START_REF] Wilson | Using trajectory data to improve bayesian optimization for reinforcement learning[END_REF] encodes information about policy closeness by calculating the Kullback-Leibler divergence. Antonova et al. [START_REF] Antonova | Sample efficient optimization for learning controllers for bipedal locomotion[END_REF], [START_REF] Antonova | Deep kernels for optimizing locomotion controllers[END_REF] designed two kernels: one called Determinants of Gait (DoG), which has 16 hand-picked policy parameters to describe a gait, and uses short simulations to evaluate the success of a given parameter vector, and another called trajNN, which is learned with a neural network. Learning with these more informative kernels does improve the speed of learning, but they also make the algorithm much more complex.

Instead of incorporating them in the kernel, Cully et al. [START_REF] Cully | Robots that can adapt like animals[END_REF] encode priors into the GP mean function by first precomputing a behavior-performance map, which is a map in parameter space of how well the robot performs in simulation, and then using this map as the GP mean initial guess, which is shown to guide the real-world BO toward likely high-performing solutions.

We propose a two-pronged solution to the aforementioned challenges: a multi-objective optimization (MOO) step to generate a large set of control parameter options, and a Bayesian optimization step to narrow down these solutions to one which works for a new, untrained trajectory or a new robot model. The results of this approach surprisingly show that between different simulated robot models, the Pareto fronts generated by NSGA-II perform similarly, which indicates that there may be no need to use BO for closing the reality gap. This is why we focus instead on achieving transfer to novel trajectories.

II. BACKGROUND A. Task priority-based control

The control framework we employ is a generic wholebody control (WBC) formulation known as task prioritybased control [START_REF] Nakamura | Task-priority based redundancy control of robot manipulators[END_REF], with soft task priorities. Here a task is defined, depending on its type, as either a Cartesian position and/or orientation, or a set of joint angles, to track. Task priority-based control solves the following QP optimization problem at each time step to choose the next control input:

min u, q nt i=0 w i A i (q, q) q -b i (q, q) 2 + u 2 s.t.

    

A i,ineq (q, q) q ≤ b i,ineq (q, q) A i,eq (q, q) q = b i,eq (q, q) u = M (q) q + h(q, q) -J (q) f ,

where u is the control input torque, q, q, and q are the joint positions, velocities, and accelerations, n t is the number of tasks, w i is the soft priority weight (SPW), A i is the task's equivalent Jacobian, b i is the reference, is a regularization factor, and f is the external wrench. The constraints represent all control input, dynamics, environmental, and most importantly, model constraints on the robot. The reference b i , which depends on the desired trajectory, is defined for both Cartesian and postural tasks as:

b i (q, q) = pd i -Ȧi (q, q) q + λ P i e + λ D i ė, (2)
where superscript d refers to desired value, pd i is desired task acceleration, Ȧi is the time derivative of the task Jacobian, e = p i -p d i , ė = ṗiṗd i are the errors between desired and actual position p i and velocity ṗi vectors, where the goal position and/or orientation depends on the task i, and λ P i and λ D i = 2 λ P i are the proportional and derivative task convergence gains (CG), according to [START_REF] Rocchi | OpenSoT: A whole-body control library for the compliant humanoid robot COMAN[END_REF].

B. NSGA-II

The first part of the approach outlined here employs one of the most popular MOO algorithms, NSGA-II. NSGA-II is part of a class of evolutionary algorithms, whose strength lies in their ability to find a diverse set of Pareto optimal solutions, an important characteristic for finding a robot controller that is able to generalize to many types of tasks. NSGA-II outperforms other evolutionary MOO methods in terms of the diversity and accuracy of solutions found [START_REF] Deb | A fast and elitist multiobjective genetic algorithm: NSGA-II[END_REF].

MOO is a subset within the optimization field that allows optimization to be done over multiple objective functions, instead of a single one. The problem is formulated as:

min θ {f 1 (θ), f 2 (θ), . . . , f l (θ)} s.t. θ ∈ S, (3)
where f n (θ) is an objective function, l is the number of objectives and S is the feasible set to which the parameter vector θ must belong. Since the algorithm optimizes Step I

Step II offline online

Pareto front

Fig. 2. Outline of our two-part approach.

Step I uses a multi-objective optimization method, NSGA-II, to train a Pareto front of controller solutions on various whole-body trajectories. This step is performed offline, on the simulated robot.

Step II uses this Pareto front as a search space with Bayesian optimization and chooses the best-performing controller in the front for a new trajectory. This step is performed online, on the real robot.

many objective functions at once, without aggregating them together, there seldom exists a single solution (a point in parameter space within the feasible set S) that optimizes every objective function. Here, the concept of Pareto dominance [START_REF] Miettinen | Nonlinear Multiobjective Optimization[END_REF] is used: A solution θ 1 dominates another solution θ 2 , if and only if the following conditions are satisfied: (i) θ 1 is not worse than θ 2 with respect to all objectives, and (ii) θ 1 is strictly better than θ 2 with respect to at least one objective. A Pareto optimal solution is a solution for which no one objective function can be further optimized without compromising the performance of another objective, i.e., it is non-dominated. Solving a MOO problem results in a set of Pareto optimal solutions, called the Pareto front, which each perform differently with respect to each of the various objective functions.

C. Bayesian optimization

The second portion of this work aims to facilitate transfer of Pareto solutions to the real robot. For this, a surrogatebased PS method [START_REF] Chatzilygeroudis | A survey on policy search algorithms for learning robot controllers in a handful of trials[END_REF], Bayesian optimization, is chosen because this type of algorithm conforms to this particular problem's need for few rollouts.

BO is distinguished by the fact that it learns a surrogate model of the expected return Ĵ(θ), most often represented by a GP:

Ĵ(θ) ∼ GP(µ(θ), k(θ, θ )), (4)
where µ(θ) is the GP mean function, and k(θ, θ ) is the covariance, or kernel function. Using the set of recorded returns from past rollouts D 1:t , a distribution of the expected return can be estimated for a new parameter set θ * using the predictions of the GP:

µ(θ * ) = k K -1 D 1:t σ 2 (θ * ) = k(θ * , θ * ) -k K -1 k, (5)
where

k = k(D 1:t , θ * ) is a kernel vector, K is a kernel matrix with K i,j = k(θ i , θ j ), µ(θ * )
is the mean prediction of the GP and σ 2 (θ * ) is the GP variance prediction. The kernel function k(θ, θ ), used to quantify the difference in expected return Ĵ(θ) between two possible parameter sets, can be defined in various ways to shape the performance of the optimization.

In BO, the acquisition function, a measure of how much information will be provided by a parameter set, uses the surrogate model Ĵ(θ) to choose which policy parameters to test next, balancing exploration and exploitation. Upper Confidence Bound (UCB) [START_REF] Srinivas | Gaussian process optimization in the bandit setting: No regret and experimental design[END_REF], which is found to be the best of three common options [START_REF] Calandra | Bayesian optimization for learning gaits under uncertainty[END_REF], [START_REF] Brochu | A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning[END_REF], optimistically judges each prospective parameter set by the upper limit of the GP of its expected return:

α U CB (θ; D 1:t ) = µ(θ) + βσ(θ), (6)
where β is a hyperparameter that balances exploitation and exploration by specifying how high the upper confidence is.

III. METHODS

Our method has two overall steps, shown in Figure 2: optimize a Pareto front using the evolutionary MOO algorithm, NSGA-II, and then use that Pareto front as prior information to achieve solution transferability with BO.

A. Task formulation

In this work the control parameter set is composed of task priority-based control weights and gains as described in Section II-A. We optimize two parameters per task: the SPW w i , which measures the importance of the task compared to the other tasks, and the CG λ P i , which defines the responsiveness of the controller to errors in task position and velocity. We relate the derivative CG λ D i directly to λ P i and thus it is not independently optimized. The parameter set θ consists of these two parameter values, w i and λ P i , for each task.

B. Pareto front dimensions

To generate Pareto fronts with NSGA-II, we measure the quality of a given parameter set using objective functions. The main aspect of our NSGA-II implementation that sets it apart from that of [START_REF] Penco | Learning robust task priorities and gains for control of redundant robots[END_REF] is the objective function setup that is used. Penco et al. [START_REF] Penco | Learning robust task priorities and gains for control of redundant robots[END_REF] use two objective functions, where the performance of each of these is averaged out over the training trajectories. Since different movement types necessitate unique control parameters, we instead use a separate objective function for each training trajectory chosen in order to ensure diversity in the learned solutions. This way the resultant Pareto front is l-dimensional, where l is the number of training trajectories used.

C. Inclusion of prior information

Our implementation of Pareto-based BO uses the previously generated Pareto front as a search space, making the parameter space finite and easily searchable. This initialization strategy is similar to that of [START_REF] Cully | Robots that can adapt like animals[END_REF], where the initialization contained prior information in the form of a behaviorperformance map that was pre-computed in simulation. Our BO method differs from that approach in the way that the map (Pareto front in this case) is generated. Since in our case the parameter space is easily searchable, instead of using an optimizer, an exhaustive search algorithm similar to that used in [START_REF] Cully | Robots that can adapt like animals[END_REF] is employed. The mean function µ(θ) of the GP, which has a significant effect on the algorithm performance, is initialized using the average of the costs from the NSGA-II training trajectories. We use the Matérn 5 /2 kernel function to represent the GP covariance k(θ, θ ), and the UCB acquisition function to choose the next parameter set to test.

IV. EXPERIMENTS

In these experiments, we used the Task Space Inverse Dynamics (TSID) library [START_REF] Prete | Implementing torque control with high-ratio gear boxes and without joint-torque sensors[END_REF] on top of the Pinocchio framework [START_REF] Carpentier | The pinocchio c++ library : A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives[END_REF] in C++ to perform inverse dynamics task priority-based control (described in Section II-A). The robot used in this work is the 32-DoF Talos humanoid from PAL Robotics [START_REF] Stasse | TALOS: A new humanoid research platform targeted for industrial applications[END_REF], on which TSID runs at a frequency of 500 Hz.

A. NSGA-II: Experimental design

NSGA-II was parallelized on a 256-core computer using the Sferes v2 evolutionary algorithm framework [START_REF] Mouret | Sferes v2 : Evolvin' in the multi-core world[END_REF]. Parallelization drastically reduces the amount of time required for this computationally intensive algorithm, but nonetheless, to complete a single NSGA-II run of 100 generations with four objective functions and self-collision checking enabled requires roughly 24 hours to run.

1) Tasks used: As shown in Table I, there are five tasks used to guide the movements of the Talos. In total, the parameter set has 10 elements. Each of these was optimized in the range of [0, 2000], the same range used when handtuning the task parameters.

2) Objective function: For our implementation, in both the NSGA-II step and the BO step, each objective function is a measure of the Cartesian accuracy of 3 selected tasks: T {rh,lh} , T {rf,lf } , and T CoM . The form of each of these objective functions is: However, during trials on the real robot, due to the limitations of motion capture, it was not possible to accurately record the CoM position. Instead, a modified version of the objective function, which calculates the average tracking error of only the hands and feet, was used to compare each of the results:

f b =    T t=0 e CoM t +
f b,mod = 1 N t T t=0 e {rh,lh} t + e {rf,lf } t , (8)
Equation ( 8) was used to compute the cost of each real robot experiment from the motion capture measurements.

3) Training trajectories: The more diverse the Pareto front, the more likely it is to transfer well to new trajectories. Since the trajectories used by NSGA-II to learn the Pareto front have a significant effect on this diversity, it is essential to choose training trajectories that use different parts of the robot and have different goals.

In all, there are 8 training trajectories used, each of which has a duration of 20 s. These 8 trajectories are evenly balanced with 4 handmade trajectories (walk on spot, squat, clap, and touch ground), for which the target positions were chosen by hand, and 4 trajectories retargeted from human motions recorded with the XSens MVN motion tracking suit [START_REF] Roetenberg | Xsens mvn: full 6dof human motion tracking using miniature inertial sensors. xsens motion technologies bv[END_REF] (dance, lean and twist, right arm reach, and lift). The result of combining these trajectories is a suitably varied and diverse training set. The full training set was divided in half (Set 1: walk on spot, clap, lean and twist, right arm reach, Set 2: squat, touch ground, dance, lift) and NSGA-II was run with 4 training trajectories at a time, for 100 generations each. Five Pareto datasets were optimized each for Set 1 and 2. Each set contains 2 handmade and 2 recorded trajectories.

4) Modified robot models: In order to evaluate the transferability (i.e., the success in crossing the reality gap) of the Pareto solutions from NSGA-II, 5 different mass-modified models (URDF files) of the Talos robot were created: uniformly 10% heavier and 10% lighter, and with 5 kg added to the left shoulder, right hand, and as a backpack.

5) Experiments on the real Talos:

The learned parameters we chose to test on the real Talos were the best nonfailing solutions in the Gazebo simulator, which has higher fidelity than the simulator used for NSGA-II optimization. The values of the hand-tuned parameters used and the chosen NSGA-II-generated parameter sets are listed in Table II.

The 3-D hand and foot tracking data from the motion capture system was compared to the reference positions sent to the robot over the course of each rollout. Here, Umeyama's method [START_REF] Umeyama | Least-squares estimation of transformation parameters between two point patterns[END_REF] was used to estimate the proper transformation.

B. NSGA-II: Results

1) Learned parameter values:

The datasets resulting from our implementation of NSGA-II each contain roughly 100 Pareto solutions. For SPWs, Figure 3 shows that w CoM and w p are, respectively, very high and very low for almost every The CG boxplots in Figure 3 show that the λ P CoM values are universally very close to 2000, which is necessary to allow the robot to maintain stability. On the other hand, the torso and posture CGs are generally nearer to zero. It can be postulated that by making these robot parts less reactive, there are less jerky motions, lowering the risk of both selfcollision and falling.

The squat is the most stable and therefore "easy" trajectory to find control parameters for, so for this reason it has a wider range of successful weights and gains. Conversely, λ P f values for each trajectory have a wide range and a relatively high median, suggesting that the specific value of this parameter does not have as large an influence on the possibility of failure, but that it does play a role in stabilizing the robot.

Additionally, Figure 3 shows that the actual mean parameter values learned for a robust controller are very different from those in the hand-tuned set, further illustrating the usefulness of optimization via this method.

2) Performance on modified models: One sample Pareto front each for training Sets 1 and 2 was tested on the 5 modified robot models described in Section IV-A.4. We found that between different models, the best-performing region of the Pareto front is the same, which suggests transfer from simulation to reality is straightforward.

3) Experiments on the real robot: After evaluating the performance of the learned Pareto fronts in simulation, the squat and dance trajectories were validated on the real Talos robot to compare the performance with learned vs. handtuned parameters. A video comparison of the learned and hand-tuned performance of these trajectories on the Talos is attached 1 .

For the squat trajectory, Figure 1 shows that the hands move much less in the world frame with learned vs. handtuned parameters, which makes the whole movement more stable and precise. For the dance trajectory, it is more difficult to visualize the difference of the two solutions, because the robot is moving quite fast; however, the calculated cost of 8.96 cm for the learned parameter set is about 3 cm less than the 11.79 cm of the hand-tuned set, which is encouraging for a single trial.

In summary, both Pareto solutions that were tested worked well on the real robot. From these results we conclude that in addition to producing control parameters much more efficiently than hand-tuning, learning in simulation via NSGA-II also yields parameter sets that are transferable to reality. For this, each Set 1 trajectory was used as a "new" trajectory on the Set 2 Pareto front, and vice versa.

1) Algorithm settings: For Bayesian optimization, the many possible hyperparameter settings of the algorithm affect its performance differently. We chose to vary the initial GP mean function, the GP kernel smoothness l b , and the UCB exploration coefficient β. The choice of initial mean µ 0 (θ) determines whether the optimizer is optimistic or pessimistic, which influences its ability to find solutions.

Since Pareto front-based BO has access to the Pareto solutions, an optimistic initial mean is an average of the NSGA-II-trained fitness values over all training trajectories for each individual, ignoring failing costs. In contrast, the pessimistic initial mean for Pareto-based BO is the same average, but includes failing costs, which bring down the average significantly. Next, changing the smoothness hyperparameter l b affects how much the estimated expected returns Ĵ(θ) of parameter sets near the sampled one are changed. The last part that will be examined is the exploration coefficient β > 0 of the UCB acquisition function, which balances exploration and exploitation. In order to evaluate which of these settings work best, each of the combinations C1-4, listed in Table III, are tested.

D. BO: Results

Using BO allows us to train a Pareto front on a few trajectories, but later have a means of adding new trajectories to the robot's repertoire without having to repeat the MOO.

In addition to testing this approach on each of our training trajectories, we also perform trials with the hyperparameter combinations given in Table III. In order to gain a better understanding of how well this method performs, Figure 5 compares it both to direct BO, which searches in the continuous space, and random search of the Pareto front space, averaged out over 1000 runs. Figure 5 indicates that in most situations, the Pareto-based BO approach finds very good control solutions within the first few iterations, suggesting that this approach could also be applied on a real robot. Our approach outperforms both direct BO and Pareto-based random search in terms of convergence time and solution quality.

1) Pareto-based BO vs. benchmarks: One immediate takeaway from Figure 5 is that direct BO finds conspicuously worse solutions than either the random search along the Pareto front or Pareto-based BO. This proves that there is a clear benefit to using a Pareto front as a small, discrete search space. It should be noted that for the touch ground trajectory, none of the Pareto-based methods find a successful solution. In fact, no successful solution exists for this trajectory on the searched Pareto front. This points to either a lack of adequate diversity in the trajectory group which Set 1 is trained on, or an insufficient number of training generations with NSGA-II.

2) Comparison of different settings: C1-4 perform similarly, but uniformly better than either direct BO or random search. Notably, the filtered mean configurations outperform the unfiltered C1 for every trajectory except the clap. In general, the filtered mean describes the search space better than the unfiltered mean, which is full of large error estimates.

V. CONCLUSIONS AND FUTURE WORK

In this paper we present a solution for learning taskpriority-based control parameters for a humanoid robot in two steps: optimizing a Pareto front of parameter sets with NSGA-II, and searching in that Pareto front to learn suitable parameters for new trajectories with BO. This combination of methods saves time that would otherwise be spent handtuning a controller, improves tracking accuracy on individual trajectories, and is transferable to new, untrained trajectories.

One direction which deserves further investigation is to design and test other objective functions. The objective function used in this work assigns the same cost to all failing solutions and therefore provides little information to the optimizer about which direction in which to move in the parameter space to avoid failure. A more descriptive objective function, such as one that accounts for how much of the trajectory the robot completes before failing, would speed up the optimization. Other ways of initializing the GP mean function could also be useful to test, such as by measuring which NSGA-II training trajectory is most similar to the BO trajectory via some heuristic, and then using that training trajectory's Pareto front performance as the initial mean for learning the new trajectory. Another improvement consists of using whole-body retargeting to provide more complex behaviors, as was done in [START_REF] Penco | Learning robust task priorities and gains for control of redundant robots[END_REF] for the iCub robot.

To further investigate the transferability of the approach from simulation to reality, more tests on the real robot will help to statistically analyze the squat and dance performance as well as to validate performance on other trajectories.

Fig. 1 .

 1 Fig. 1. Comparison of squat trajectory performance between hand-tuned control parameters (top) and parameters learned using our method (bottom), on the Talos robot.

Fig. 3 .

 3 Fig. 3. Boxplot comparison of top 50 best-performing learned SPWs and CGs for each training trajectory. Data is from 5 separate Pareto fronts for each training set.

Fig. 4 .

 4 Fig. 4. Examples of the change in performance of a Pareto front between one robot model and another, for the walk on spot trajectory. Solutions with tracking error greater than 0.5 are labeled as failing solutions.

Fig. 5 .

 5 Fig. 5. Performance of our method, Pareto-based BO, on hyperparameter combinations C1-4 (deterministic, run once each) from TableIII, compared to direct BO and random search (along the Pareto front) results. A cost of 0.7 is used to represent all failing costs. The different combinations perform similarly, but uniformly better than either direct BO or random search.

  : Experimental design BO was tested along the Pareto fronts generated from a sample MOO dataset to allow the robot to learn to follow new trajectories, on which the Pareto front was not trained.

TABLE I SYMBOL

 I , DESCRIPTION, SPW NAME AND CG TYPE FOR EACH OPTIMIZED TASK.

	Task	Description	SPW	CG
	T {rh,lh}	hand pose (symmetric)	w h	λ P h = σ P h
	T {rf,lf }	foot pose (symmetric)	w f	λ P f = σ P f
	T CoM	CoM position	w CoM	λ P CoM
	Tto	torso orientation (roll, pitch)	wto	σ P to
	Tp	joint angles	wp	µ P p

  e is the training trajectory index, N t is the total number of time steps, T is the final time, t is the current time, and e is the 3-D Cartesian position (without orientation) error of the specific task at time t. The robot can fail either by falling over or by self-colliding. Using a separate objective function per training trajectory ensures that the learned Pareto front will contain a diverse set of parameters, likely to work well on a variety of test trajectories.

	{rh,lh} t Nt	{rf,lf } + e t	, if no fail
	1.0 × 10 10 ,		if fail,
			(7)
	where b

TABLE II HAND

 II -TUNED AND LEARNED PARAMETER SETS USED FOR REAL ROBOT TESTING.

	Trajectory	Type	w h	w f	w CoM	wto	wp	λ P h	λ P f	λ P CoM	σ P to	µ P p
	Squat	Hand-tuned	10	1000	1000	10	1.75	30	30	30	30	10
	Squat	Learned	1296.6	1241.3	1871.3	248.9	26.1	1591.1	916.9	1968.6	618.0	231.5
	Dance	Hand-tuned	100	100	2000	1000	100.75	1000	30	1000	30	60
	Dance	Learned	207.9	192.1	1885.1	852.3	45.9	898.6	1074.7	1935.8	26.0	215.9

TABLE III LIST

 III OF SETTING COMBINATIONS TESTED FOR BO.

Video also available at: https://youtu.be/X1iNwDKXUNQ