M Khamassi

G Chalvatzaki

T Tsitsimis

G Velentzas

C Tzafestas

A framework for robot learning during child-robot interaction with human engagement as reward signal

Keywords: HRI, Reinforcement Learning, Active Exploration, Autonomous Robotics, Engagement, Joint Action

A framework for robot learning during child-robot interaction with human engagement as reward signal

I. INTRODUCTION

In this short paper, we present recent progresses in developing robot learning abilities for the adaptation to humanspecific requirements during child-robot interaction. In particular, we aim at enabling the robot to vary the level of expressivity of its actions in order to increase the child's mutual engagement with the robot and thus contribute to further develop children's social interaction skills. Mutual engagement can be defined as "the process by which interactors start, maintain and end their perceived connection to each other during an interaction" [START_REF] Sidner | Explorations in engagement for humans and robots[END_REF].

Researches in the field of social robotics have recently shown a growing interest in monitoring human and robot gaze during social interaction [START_REF] Boucher | I reach faster when i see you look: gaze effects in human-human and human-robot face-to-face cooperation[END_REF], [START_REF] Bampatzia | Effects of gaze synchronization in human-robot interaction[END_REF]. Results show that gaze following improves intention readout, efficiency of joint action, and arouses on human partners the illusion of a social intelligence. Conversely, it has been proposed that monitoring the level of engagement of the human during the task, for instance through the monitoring of body posture and gaze, may provide the robot with crucial information to assess how it is perceived by the human, how this perception changes according to the behaviors shown by the social 1 All authors are with the School of Electrical and Computer Engineering, National Technical University of Athens, Greece.

2 Mehdi Khamassi is also with Sorbonne Université, CNRS, Institute of Intelligent Systems and Robotics, Paris, France.

3 Correspondence: ktzaf@cs.ntua.gr robot, and hence to improve the quality of human-robot interaction [START_REF] Anzalone | Evaluating the engagement with social robots[END_REF]. However, to our knowledge no one has yet proposed a way to make the robot learn on the fly in response to changes in human engagement. Previous researches having applied reinforcement learning to human-robot interaction have most of the time employed discrete action spaces (e.g. [START_REF] Khamassi | Robot cognitive control with a neurophysiologically inspired reinforcement learning model[END_REF]), hence preventing generalization to more complex tasks requiring continuous motor actions.

II. PILOT CHILD-ROBOT INTERACTION STUDY WITH

CHILDREN WITH AUTISM

The general experimental paradigm adopted here consists in having a small humanoid robot interact with children (one at a time), under the supervision of an observing human adult, and finding the appropriate robot behavior to maximize children's engagement in the task. This paradigm follows the objectives defined in the framework of the EU-funded project BabyRobot (H2020-ICT-24-2015-6878310), where a set of child-robot interaction use-cases have been designed and implemented to study the development of specific socioaffective, communication and collaborative skills in typically developing children as well as children with Autistic Spectrum Disorders (ASD). In this framework, we have set up a pilot experiment 1 where the NAO robot is interacting with a child (Fig. 1), and repeatedly points at an unreachable object while varying the level of expressivity of its pointing gesture (i.e., opening-and-closing hand for a certain duration, bending its torso with a certain angle in the direction of the object, gazing at the child for a certain duration) until the child understands the "intention" of the robot and engages himself/herself into joint action in order to help the robot grasp the object. The engagement estimation, in this pilot study, was provided in real-time by an expert who observed the child during the interaction with the robot, considering five discrete levels of engagement (0 to 4, with 0 meaning absence of engagement and 4 meaning full engagement and attempt to offer help).

We present here some preliminary results for this real HRI task for which we have yet performed the experiment only with a small number of children with mild and moderate ASD symptoms, plus a few children with severe symptoms (12 children in total so far). First, children with severe symptoms expressed no interest in the task, neither in the condition with the robot nor in a control condition where the child interacts with a human expert rather than with the robot. In contrast, children with mild symptoms displayed great enthusiasm and interest in playing with the robot as well as with the researcher and enjoyed the whole process. These children were able to respond quite well to the task and completed the experiment with success. Overall, we found that two out of eight children with mild symptoms successfully maximized their engagement in joint attention with the robot and gave the object to the robot spontaneously. The remaining six children successfully increased their engagement, although not optimally, ending up moving the object closer to the robot but not handing it in. The two children with moderate symptoms also increased engagement and ended up exploring the object pointed at by the robot. Finally, again children with severe symptoms did not respond to the task.

Figure 1 shows one child performing the task, looking at the NAO robot (moderate engagement) while the latter moved its arm down after pointing at the object on the small white table. The psychologist who can be seen near the red door is manually annotating the child's engagement so that the robot can adapt its behavior. These results are promising and stimulating in that eight children that we interviewed after the task said that they would like to play more often with the robot and that they found the tasks we proposed them relatively easy. But many more subjects for each level of severity of ASD symptoms are required before allowing some statistics on the results. Interestingly, studying how the robot's movements affected the child's engagement, we observed that when the robot opened and closed its grip or exchanged glances between the child and the object for a period of time while pointing at the object, it contributed to an increase in the child's engagement. This suggests that varying the level of expressivity in the robot's actions in time was key to increase child engagement. Nevertheless, different levels of expressivity appeared to be appropriate for different children. It is thus relevant to propose a way for the robot to autonomously learn the appropriate degree of expressivity appropriate for each child. Select discrete action a t (Eq. 2)

4:

Select action parameters θ a i,t (Eq. 3)

5:

Observe new state and reward (Eq. 6)

6:

Update Q t+1 (s t , a t ) (Eq. 1)

7:

Update V t+1 (s t ) and θ a i,t+1 (s t ) (Eq. 4-5)

8:

if meta-learning then 9:

Update reward running averages rt and rt 10:

Update β t+1 and σ t+1 11:

end if 12: end for

III. ROBOT LEARNING ALGORITHM

The proposed algorithm is summarised in Algorithm 1. It is based on reinforcement learning with parameterized action spaces [START_REF] Masson | Reinforcement learning with parameterized actions[END_REF], [START_REF] Hausknecht | Deep reinforcement learning in parameterized action space[END_REF]. It employs a set of discrete actions A d = {a 1 , a 2 , ..., a k }, where each action a ∈ A d features m a continuous parameters {θ a 1 , ..., θ a ma } ∈ R ma , which enables to benefit from the simplicity of task decomposition into a small set of discrete actions while at the same time being able to exploit the precision of continuous motor execution. Learning the value of discrete action a t ∈ A d selected at timestep t in state s t is done through Q-Learning [START_REF] Watkins | Q-learning[END_REF]:

∆Q t (s t , a t ) = α Q r t + γ max a (Q t (s t+1 , a)) -Q t (s t , a t ) (1)
where α Q is a learning rate and γ is a discount factor. The probability of executing discrete action a j at timestep t is given by a Boltzmann softmax equation:

P (a|s t , β t ) = exp (β t Q t (s t , a)) a exp (β t Q t (s t , a )) (2)
978-1-5386-7980-7/18/$31.00 ©2018 IEEE where β t is a dynamic inverse temperature meta-parameter which will be tuned through meta-learning (see below).

In parallel, continuous parameters θ a i,t with which action a is executed at timestep t are selected from a Gaussian exploration function centered at the current values θ a i,t (s t ) in state s t of the parameters of this action:

P ( θ a i,t |s t , a t , σ t ) = 1 √ 2πσ t exp -( θ a i,t -θ a i,t (s t )) 2 /(2σ 2 t ) (3)
where the width σ t of the Gaussian is tuned through meta-learning (see below) and continuous action parameters θ a i,t (s t ) are learned with the CACLA algorithm [START_REF] Van Hasselt | Reinforcement learning in continuous action spaces[END_REF]. A reward prediction error is computed from the critic: δ t = r t + γV t (s t+1 ) -V t (s t ) and is used to update the critic and the actor:

V t+1 (s t ) = V t (s t ) + α C δ t ( 4
)
θ a i,t+1 (s t ) = θ a i,t (s t ) + α A δ t ( θ a i,t -θ a i,t (s t )) (5)
where α C and α A are learning rates.

In order to perform active exploration, we apply a noiseless version of the meta-learning algorithm of [START_REF] Schweighofer | Meta-learning in reinforcement learning[END_REF], which tracks online variations of the agent's performance measured by short-term rt and long-term rt reward running averages. At each timestep, we use the difference between the two averages to simultaneously tune the inverse temperature β t used for selecting between discrete actions a, and the width σ t of the Gaussian distribution from which each continuous action parameter θ a i is sampled around its current value. The main idea is that when the performance is better than average, exploration should be decreased in order to reach optimality levels. In contrast, sudden drops in the performance should lead to increases in exploration in order to adapt to environmental non-stationarities.

Finally, we need to define a reward function for humanrobot interaction tasks. This is not an easy task since during interaction the actions performed by a robot may have delayed effects on the human's behavior and on his engagement. To mimic this, we chose a reward component to be given by a dynamical system which is based on the virtual engagement E of the human in the task. In our simulations, the quantified engagement arbitrarily starts at 5, increases up to a maximum E M = 10 when the robot performs the appropriate actions with the appropriate parameters, and decreases down to a minimum E m = 0 otherwise:

E t+1 = Et + η1(E M -Et)H(θ a t ), if at = a & H(θ a t ) ≥ 0 Et -η2(Em -Et)H(θ a t ), if at = a & H(θ a t ) < 0 Et + η2(Em -Et), otherwise (6)
where η 1 = 0.1 is the increasing rate, η 2 = 0.05 is the decreasing rate, and H(x) is the reengagement function given by H(x) = 2 exp -(x-µ ) 2 2σ 2 -0.5 where a , µ and σ are respectively the optimal action, action parameter and variance around a . The reward function is then computed as r t+1 = E t+1 + λ∆E t where λ = 0.7 is a weight and ∆E t = E t+1 -E t . This reward function ensures that the algorithm gets rewarded in cases where the engagement E t+1 is low but nevertheless has just been increased by the action n-tuple (a t , θ a 1,t , θ a 2,t , .., θ a ma,t ) performed by the robot.

IV. SIMULATIONS

We performed numerical simulations of the learning algorithm in a task identical to the child-robot interaction pilot task described above. We implemented the simulations in the virtual robot experimentation platform (V-REP). In the considered scenario, the NAO robot points at an object on a table with different degrees of action expressivity so as to catch the child's attention and thus increase mutual engagement (Fig. 2(b)). We parameterized the simulated pointing action of the robot with two parameters (t 1 , t 2 ) corresponding to the time in seconds the robot would spend iteratively opening-closing its hand during pointing, and the time spent exchanging glances with the child. Examples of different levels of expressivity defined by these parameters are shown in Table I. We initialized the algorithm based on the parameters obtained on average during previous interactions with simulated children. This way, the algorithm started from a meaningful average value of action parameters/durations ( t1 , t2 ), rather than being initialized randomly, and then adapted to each specific child. We defined a time range from 0 to 10 seconds. Fig. 2(b) shows the average performance over 10 simulations. The robot firstly interacted with an "average child", meaning that the child engaged optimally with parameters ( t1 , t2 ). Then, at timestep 40, the experiment involved another child (child 1) with different optimal parameters. The engagement of child 1 was initially low but progressively re-increased as the robot was finding the optimal continuous action parameters. The figure also illustrated the increased variance in executed action parameters during exploration followed by a re-focus around the learned parameters during exploitation. Similarly, at timestep 80 child 2 took the place of child 1 and the robot readjusted its parameters. Importantly, we observe that in less than 10 timesteps the robot found the optimal parameter values for the different children whose engagement reached 8 in just a few timesteps. This thus illustrates a sufficiently fast adaptation process to work online during real child-robot interactions.

Pointing gesture expressivity        point + open-close + glance (t 1 = 0, t 2 = 0) point + exchange glance (t 1 = 0, t 2 = 0) point + open-close hand (t 1 = 0, t 2 = 0) point (t 1 = 0, t 2 = 0)
978-1-5386-7980-7/18/$31.00 ©2018 IEEE

Fig. 1 .

 1 Fig. 1. Pilot child-robot interaction study with children with ASD. The figure shows a moment where a child with ASD showed moderate engagement while the robot moved its arm up and down to point at an object on a table.

Algorithm 1

 1 Active exploration with meta-learning 1: Initialize V 0 (s), θ a i,0 (s), Q 0 (s, a), β 0 and σ 0 2: for t = 0, 1, 2, ... do3:

This experiment has been approved by the ethical committee of Athena Research Center, Greece. The children's parents provided written consents.Proceedings of the

27th IEEE International Symposium on Robot and Human Interactive Communication, Nanjing, China, August 27-31, 2018 TuBT3.3 978-1-5386-7980-7/18/$31.00 ©2018 IEEE

ACKNOWLEDGEMENTS The authors would like to thank psychologists Christina Papaeliou and Asimenia Papoulidi, and the Special Elementary School for Children with ASD of Piraeus, Greece, for enabling us to make these experiments with children with ASD. This research work has been partially supported by the EU-funded Project BabyRobot (H2020-ICT-24-2015, grant agreement no. 687831), and by the Centre National de la Recherche Scientifique (MI ROBAUTISTE & PICS 279521).

Left: Before timestep 0 the robot executed the default parameters values, no adaptation was performed. After timestep 0, the robot adapted its action parameters (black) towards the optimal action parameters (red).

Right: Child's engagement reached 90% within less than 10 trials.

V. CONCLUSIONS AND FUTURE WORK

In this short paper, we presented recent progresses in developing robot learning abilities for the adaptation to human-specific requirements during child-robot interaction. In particular, we aimed at enabling the robot to vary the level of expressivity of its actions in order to increase the child's mutual engagement with the robot and thus contribute to further develop children's social interaction skills. We first showed some preliminary results in a pilot study involving a robot with a predetermined sequence of increased expressivity of action while pointing at an unreachable object until a child with ASD understands that the robot needs help and engages in joint action. The preliminary results suggest that the level of expressivity does play a role in engaging the child, but should nevertheless be adapted through online learning to each interacting child. We then presented a learning algorithm based on reinforcement learning in parameterized action spaces [START_REF] Masson | Reinforcement learning with parameterized actions[END_REF], [START_REF] Hausknecht | Deep reinforcement learning in parameterized action space[END_REF] -to benefit from the simplicity of task decomposition into a small set of discrete actions while at the same time being able to exploit the precision of continuous motor execution -to which we added active exploration so as to cope with the frequent nonstationarities that can occur during human-robot interaction. We presented simulation results showing that the algorithm can adapt in a sufficiently small number of trials to be applied to adaptation in real-time during interaction.

In future work, we plan to test the learning algorithm during real child-robot interaction. We moreover plan to study whether the average parameters over different interacting children is efficient or whether there exists distinct clusters of parameters -especially within the data obtained in the real experiments -that should be used as separate initialization points for the learning algorithm.


Frédéric Azevedo

Dalibor Klusá£ek
email: klusacek@cesnet.cz

Frédéric Suter

Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation

Monitoring and analyzing the execution of a workload is at the core of the operation of data centers. It allows operators to verify that the operational objectives are satised or detect and react to any unexpected and unwanted behavior. However, the scale and complexity of large workloads composed of millions of jobs executed each month on several thousands of cores, often limit the depth of such an analysis. This may lead to overlook some phenomena that, while not harmful at a global scale, can be detrimental to a specic class of users. In this paper, we illustrate such a situation by analyzing a large High Throughput Computing (HTC) workload trace coming from one of the largest academic computing centers in France. The Fair-Share algorithm at the core of the batch scheduler ensures that all user groups are fairly provided with an amount of computing resources commensurate to their expressed needs. However, a deeper analysis of the produced schedule, especially of the job waiting times, shows a certain degree of unfairness between user groups. We identify the conguration of the quotas and scheduling queues as the main root causes of this unfairness. We thus propose a drastic reconguration of the system that aims at being more suited to the characteristics of the workload and at better balancing the waiting time among user groups. We evaluate the impact of this reconguration through detailed simulations. The obtained results show that it still satises the main operational objectives while signicantly improving the quality of service experienced by formerly unfavored users.

Introduction

The analysis of workload traces is a common approach to understand and optimize the behavior of the system that manages the access to resources and the execution of jobs in a data center, i.e., the batch scheduling system. Most of the historical workload traces available in the Parallel Workload Archive (PWA) [START_REF] Feitelson | Experience with using the Parallel Workloads Archive[END_REF] were originally studied with such an objective in mind. More recently, a methodology to characterize HPC workloads and assess their heterogeneity based on the analysis of the workloads executed over a year on three systems deployed at NERSC was proposed in [START_REF] Rodrigo | Towards Understanding HPC Users and Systems: A NERSC Case Study[END_REF]. This study not only helps to understand the behavior of current HPC systems but can also be used to develop new scheduling strategies for the forthcoming exascale systems. Similar studies mixing the characterization, modeling, and prediction of HPC workloads have been proposed in [START_REF] Emeras | Evalix: Classication and Prediction of Job Resource Consumption on HPC Platforms[END_REF][START_REF] You | Comprehensive Workload Analysis and Modeling of a Petascale Supercomputer[END_REF]. In [START_REF] Klusá£ek | On Interactions among Scheduling Policies: Finding Ecient Queue Setup Using High-Resolution Simulations[END_REF], the authors proposed a complete reconguration of their batch scheduling systems, including the denition of the scheduling queues, based on a thorough analysis of the workload characteristics and detailed simulations. The eects of this reconguration have then been analyzed in [START_REF] Klusá£ek | Real-Life Experience with Major Reconguration of Job Scheduling System[END_REF].

The common point of all the aforementioned studies is to consider HPC workloads that are composed of parallel jobs spanning over multiple cores and multiple nodes. In this paper we study a large High Throughput Computing (HTC) workload trace coming from the Computing Center of the National Institute of Nuclear Physics and Particle Physics (CC-IN2P3) [14] which is one of the largest academic computing centers in France. One of the main characteristics of this workload trace is that it is composed of a vast majority of jobs running on only one core, i.e., Monte-Carlo simulations and analyzes made on experimental data. This trace is also much larger than HPC traces with nearly 3,000,000 jobs executed every month on 33,500 cores. Finally, this workload captures a mix of two dierent types of submissions. As one of the twelve Tier-1 centers engaged in the processing of the data produced by the Large Hadron Collider (LHC) at CERN, half of job submissions come from an international computing grid through a complex middleware stack. The other half is directly submitted to the batch scheduling system by users belonging to more than 70 scientic collaborations.

In a previous work, we described the process of simplication of the operation of this infrastructure by reducing the "human-in-the-loop" component in scheduling decisions [START_REF] Azevedo | Reducing the Human-in-the-Loop Component of the Scheduling of Large HTC Workloads[END_REF]. The main contribution of this paper is that we conduct a thorough analysis of the workload processed at a real large scale HTC system to show that the current system conguration tends to favor the jobs coming through the grid. The fair-share algorithm implemented by the scheduler is thus not as fair as it seems. Indeed, jobs submitted by local users suer from a significantly higher waiting time. Based on our observations, we propose and evaluate through simulation a drastic modication of the quotas and the conguration of the scheduling queues that aims at further improving the fairness of the produced schedule. The obtained results show that it still guarantees the satisfaction of the main operational objectives while signicantly improving the quality of service experienced by the formerly unfavored users. To ensure the reproduction and further investigation of the presented results, and thus favor Open Science, we made our large workload trace available to the scientic community [START_REF] Azevedo | Companion of the Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation article[END_REF].

This paper is organized as follows. Section 2 describes the computing infrastructure operated at CC-IN2P3 with details about the hardware, conguration of the batch scheduling systems, and operational constraints. In Sect. 3 we present and discuss the main characteristics of the workload executed on this infrastructure. Section 4 details the proposed modication of the system conguration and evaluates its impact on the quality of service and fairness experienced by the submitted jobs. Section 5 briey explains how we produce and make available the workload trace used in this work. Finally, we conclude this paper and detail future work directions in Sect. 6. These resources are managed by Univa Grid Engine (UGE v8.4.4) [START_REF]Grid Engine[END_REF] which implements the Fair Share Scheduler rst described in [START_REF] Kay | A Fair Share Scheduler[END_REF] and thus assigns priorities to all the unscheduled jobs to determine their order of execution. These priorities directly derive from the resources pledges expressed by the dierent user groups. As detailed in [START_REF] Azevedo | Reducing the Human-in-the-Loop Component of the Scheduling of Large HTC Workloads[END_REF], each group has to provide an estimation of its computing needs as an amount of work, expressed in Normalized HS06.hours [START_REF] Michelotto | A comparison of HEP code with SPEC 1 benchmarks on multi-core worker nodes[END_REF], to be done during each quarter of the following year. Once arbitration has been done with regard to the total available computing power, the respective share that has to be allocated to each group is converted into a consumption objective used by UGE to determine a fair-share schedule. This algorithm addresses one of the two main operational objectives of the CC-IN2P3: ensure that each user group is served according to its expressed resource request for the year.

In addition to this central scheduling algorithm, resources are organized in queues whose characteristics are given in Tab. 2. These queues are listed in the order in which they are considered by the job scheduler. They mainly dier by maximum allowed duration, both in terms of wallclock and CPU times, available memory and scratch disk space per job, and the type of jobs allowed to enter the queue, i.e., sequential or multi-core (denoted by the mc-prex).

The long queues can both access the entire infrastructure. As it will be shown in Sect. 3, these two queues have to absorb the bulk of the workload. However, jobs are not really distinguished by their execution time in this conguration, with a minimal limit on execution time set to 48 CPU hours (or 58 hours) for all jobs. The huge queues are intended to jobs that need more memory or disk space while the access to the longlasting queues is limited to certain user groups. We also see that all queues combined could virtually access more than three times the actual number of available cores. This conguration aims at achieving the highest possible utilization of the resources which is the second main operational objective of the center. Indeed, this ensures that load variations in the dierent scheduling queues cannot lead to leaving some cores idle.

Another important operational constraint of this system is related to the access to storage subsystems. High Energy Physics is a data-driven science, hence most of the jobs rely on locally stored data. Some user groups exhibit heavy I/O patterns that may become harmful to the storage subsystems in the worst case scenario where many jobs from these groups are executed concurrently. To prevent an overload of the storage subsystem, the number of concurrent jobs can be limited to a safe number for some groups by setting Resource Quota Sets (RQSs). The downside of this method is that it may increase the number of waiting jobs when this limit is reached. In such a case, operators can dynamically and temporarily modify the RQSs if the storage subsystems can cope with the extra load.

Workload Analysis

In this section we describe the main characteristics of the workload processed in November 2018 at CC-IN2P3. Over this month, 2,669,401 jobs were executed on the 33,456 available cores (or slots in the UGE vocabulary). We distinguish two sub-workloads depending on whether jobs are submitted to the batch system by Local users (1,174,078 jobs) or through a Grid middleware (1,495,323 jobs).

Figure 1 shows how many slots are simultaneously used. The dashed line indicates the total number of available slots while solid lines respectively show the overall utilization and which part of it comes from Grid and Local jobs.

This gure show that the main operational objective of the CC-IN2P3 is achieved with a global utilization well over 90%. We also see that this utilization is dominated by Grid jobs that use 3.45 times more slots than Local jobs while there are only 1.27 times more jobs coming through the grid than submitted by local users. About 28% of Grid jobs are multi-core, while 98% of Local jobs request only one core for their execution. There were less than 17,000 multi-core Local jobs submitted over the considered 1-month period which required at most 16 cores. Most of multi-core Grid jobs fall in two categories. First, production jobs submitted by two LHC experiments (ATLAS and CMS) always require 8 cores. Second, two-core short-lived probe jobs are periodically submitted to check for site availability. They represent about 5% of the multi-core Grid jobs. These two sub-workloads show several additional dierences. The rst one pertains to the expressed job duration. Figure 2 (left) shows a cumulative distribution function of the requested CPU time for Local and Grid jobs. There are only three values for the Grid jobs which correspond to the queue upper limits on CPU time (i.e., 48, 72, and 192 CPU hours). These predened requirements are automatically added at the level of the Computing Element, i.e., the interface to the resources of a Grid site. The rationale is that most of the user groups that submit their jobs through the Grid manage their own workloads with pilot jobs [START_REF] Turilli | A Comprehensive Perspective on Pilot-Job Systems[END_REF]. Local jobs show a larger diversity in their CPU time requirements, even though about 35% of the jobs either require the upper limits of the queues or do not express any requirements. We also observe that a large fraction of Local jobs expresses requirements that are much lower than the queues upper limits. Nearly 40% of the jobs explicitly announce that they run for less than 12 hours but fall in the same queue as jobs potentially running for two days. Figure 2 (right) shows the cumulative distribution functions of the actual run times of Local and Grid jobs. We can see that the distribution of job duration is very similar for both sub-workloads with a large proportion of jobs running for less than four hours. There is thus an important discrepancy between the characteristics of the jobs in terms of execution time and both the expression of requirements and the conguration of the scheduling queues.

Another important dierence between the two sub-workloads is about how fast jobs can start their execution, i.e., how much time jobs have to wait in queues before starting. Figure 3 shows the cumulative distribution function of this waiting time for Local and Grid Jobs. We can see that while 99% of the jobs wait for less than a day, Local jobs tend to wait much more than Grid jobs. Indeed, more than 75% of the Grid jobs but only 50% of the Local wait for less than one hour. To understand the origin of such a discrepancy, we analyze the evolution of the number of jobs waiting in the queues over the considered period. Figure 4 shows this evolution which conrms that there are much more Local jobs waiting than Grid Jobs. On average, there are about 1,650 waiting Grid jobs but nearly 5,600 waiting Local jobs. More importantly, while the maximum number of waiting Grid jobs is always less than 4,000, there can be more than 40,000 Local jobs waiting in queues. Several factors can explain this dierence. Local Grid q q q q q q q q q q q q q q q q q q q q q q q q q q q q 0 4000 8000 12000 First, they clearly dier by their submission patterns. Figure 5 (left) shows the daily arrival rates for Local and Grid jobs. The dashed lines depict the average number of submissions per hour which is quite similar for the two subworkloads (respectively 40,666 and 45,695 jobs per hour). However, we clearly observe two dierent submission patterns. Local jobs follow a traditional "working hours" pattern, while Grid jobs are submitted at an almost constant rate.

Figure 5 (right) shows the submission pattern of a specic user group that leverages the "array job" feature of UGE that enables the submission of up to 10,000 related jobs in a single command. Then we clearly observe burst submissions which are typical and correspond to "production" periods. They are usually followed by "result analysis" periods of lower activity. The combination of these general and specic submission patterns amplies the dierence in the number of waiting jobs with Grid submissions that are controlled upstream by monitoring and the use of pilot jobs.

The other factors that contribute to the observed discrepancy are that the groups submitting jobs through the Grid have the highest priorities in the fairshare computation, they are the main source of multi-core jobs that benet of higher priorities on the mc-* queues, and more stringent RQSs are set for Local users thus limiting the number of concurrent running jobs.

Revisiting the Conguration of the Batch System

The analysis conducted in the previous section conrmed that the main operational objectives, i.e., a maximal utilization of the computing resources and the respect of a fair sharing of resources derived from the pledges made by the dierent user groups, were achieved. However, we also observed that the overall fairness provided by the scheduling algorithm hides a more unfair behavior that has been overlooked by operators. We showed in Fig. 3 that jobs submitted by Local users wait much more than Grid jobs. Some of the root causes of these larger waiting times for Local jobs, i.e., higher submission rates during the day, and burst submissions, can not be solved by the scheduling system, but we believe that the current conguration of the system amplies the eects of these peaks in Local job submission.

The primary job discrimination factor currently used is whether a job is sequential or multi-core. Our analysis showed that 96% of the multi-core jobs are Grid jobs. Moreover, the queues dedicated to multi-core jobs have a higher priority and the mc-long has access to the entire set of cores. Then, an important fraction of Grid jobs (27%) is privileged with regard to the rest of the workload. A second important factor is that the user groups with the biggest shares are the main source of grid jobs, hence increasing the priority gap with Local jobs. Finally, while local jobs express an estimation of their execution time, the current conguration of scheduling queues does not leverage this information.

To better reect the respective characteristics of the two identied subworkloads and increase fairness by better balancing the waiting times across jobs, we propose a new conguration of the scheduling queues described in Tab. 3.

The rst three queues are dedicated to Local jobs and are considered in increasing order of expressed job duration. Moreover, the shorter the jobs are in a queue, the more resources the corresponding queue can access. Indeed, short jobs will quickly release resources, and thus can use more slots at a given time without harming the overall throughput. Then, all the Grid jobs are now placed in a single dedicated grid queue. By considering this queue after those for Local jobs, we give a higher priority to Local jobs and then reduce their waiting times. Finally, we merged the queues for jobs with special requirements to remove the sequential/multi-core distinction. The number of slots a queue can access has been empirically chosen based on the utilization shown in Fig. 1. We aim at having a good tradeo between preventing queue overload and ensuring a maximal utilization even under load variations within the sub-workloads.

To assess the impact of these modications on the overall utilization and the waiting times experienced by jobs, we implemented a simulator using the Alea job scheduling simulator [START_REF]Alea 4: Job scheduling simulator[END_REF][START_REF] Klusá£ek | Complex Job Scheduling Simulations with Alea 4[END_REF] which is based on the GridSim toolkit [START_REF] Sulistio | A toolkit for modelling and simulating data Grids: an extension to GridSim[END_REF]. Alea allows for detailed simulations, supports several mainstream scheduling algorithms (e.g., FCFS, variants of Backlling, Fair-Share) and commonly used system restrictions such as queue limits and quotas. We implemented two variants of the scheduling system. The rst one constitutes a baseline and aims at reproducing the current conguration of the system. It uses the same queue denitions, shares, and RQSs as those used by UGE. Moreover the simulation starts with a set of "dummy" jobs running to mimic the state of the system on Nov. 1, 2018 at 12 am. The modied variant only diers by the denitions of the scheduling queues to use those presented in Tab. 3. We compare in Tab. 4 the distribution of waiting times in the two sub-workloads in the baseline and modied simulated schedules. The proposed modication achieves its objectives. The average waiting time which was almost two times shorter for Grid jobs than for Local jobs is now more balanced, yet still shorter for Grid jobs. The median and third quartile values show that for a large fraction of Grid jobs, the impact of our modication is negligible, i.e., an increase of 6 minutes only, while the waiting times of Local jobs is reduced by half an hour. At the 90 th percentile, the waiting of Grid jobs almost doubles, but remains three times less than that of the Local jobs. Finally, our modication reduces the maximum waiting time for both sub-workloads with a noticeable reduction of one week for the most waiting Local job.

Then we verify in Fig. 6 that both variants of the simulator can reproduce the main trends of the original schedule for each sub-workload, even though it does not capture all the conguration parameters of the real system. We can see that the baseline version can be used to evaluate the impact of the proposed modication and that the proposed modication achieves a very similar overall utilization and thus does not compromise the main operational objective. We also propose to investigate the impact of a relaxation of the RQSs that are applied to local user groups and limit their access to resources. We make the assumption that the bigger is the original RQS, the bigger an acceptable increase can be. Indeed, a group constrained to use very few resources is more likely to cause I/O issues than a group allowed to use a large number of slots. We thus classify local user groups in three categories (i.e., 0-5%, 5-10%, and 10+%) according to the fraction of resources they can use. We propose two scenarios: a conservative one in which RQSs are respectively increased by 5%, 10% and 20%, and an extreme scenario in which increases are of 100%, 200%, and 300%. Table 5 shows a further reduction of the waiting time of Local jobs with a moderate impact on Grid Jobs. The extreme relaxation leads to the fairest schedule with similar waiting times up to the 90 th percentile. However, it may be too extreme to be accepted by the system administrators. From these experiments, we can conclude that the new scheduling queues in Tab. 3. combined with a conservative relaxation of the RQSs is a good candidate for production.

Workload Trace Production and Availability

The workload trace used in this work has been produced by extracting and combining information from dierent tables of the Accounting and Reporting Console (ARCo) of Univa Grid Engine. We rst curated and anonymized the data and then converted the trace into the Standard Workload Format (SWF) [START_REF] Chapin | Benchmarks and Standards for the Evaluation of Parallel Job Schedulers[END_REF] used by the PWA [START_REF] Feitelson | Experience with using the Parallel Workloads Archive[END_REF]. As this format does not allow us to log shares or RQSs, we accompany the trace with additional les. To ensure the reproduction and further investigation of the presented results, we made an experimental artifact that comprises the workload trace, its additional les, the code of the simulator, the simulation results and a companion document describing the data analyses [START_REF] Azevedo | Companion of the Improving Fairness in a Large Scale HTC System Through Workload Analysis and Simulation article[END_REF].

Conclusion and Future Work

The operation of a large data center is a complex task which is usually driven by a few selected operational objectives. In the case of the IN2P3 Computing Center, these objectives are to ensure a maximal utilization of the computing farm and the respect of a sharing of the resources proportional to the requests expressed for the year by the dierent user groups. To meet these objectives, operators rely on the implementation of the Fair-Share algorithm proposed by the Univa Grid Engine batch scheduling system and benet from the main characteristic of the processed workload to be composed of a vast majority of sequential jobs.

Thanks to the batch scheduling system, the objective of an overall fairness across user groups, in terms of resource utilization, is achieved. However, the deeper analysis of the workload and its processing that we proposed in this paper showed that the current conguration of the system leads to a signicant unfairness, in terms of job waiting time. We identied the root causes of this unfairness and proposed a pragmatic reconguration of the scheduling queues and quotas to address this issue. Detailed simulations of the original and modied congurations show that the proposed modications better balance the waiting times across jobs without hindering the overall utilization of the system. The main operational objective is thus still achieved and the quality of service is improved for local users. Jobs submitted through the grid, which were previously favored now experience waiting times on par with those of the local jobs.

We are aware that the presented analysis and proposed optimizations are tightly coupled to the very specic use case of the CC-IN2P3. The applicability of our ndings to other systems can thus be legitimately questioned. However, we believe that two important and more general lessons can be drawn from this study. First, the behavior and performance of a job and resource management system are not only driven by the sole scheduling algorithm. This central component is the most studied in the job scheduling literature, but we shown that other components such as the scheduling queues and resource quotas can be key factors too and have to be included in performance studies. Second, the conguration of a batch system is a slowly evolving process and decisions made at a given time may last beyond an important evolution of the workload and impede the operation. Regular analyses of the workload and revisits of the system conguration should thus be seen as good operation practices.

As future work, we would like to improve the simulator to better mimic the behavior of the system in production. A faithful replay of the original workload will allow us to better measure the impact of potential modications of the system, and thus ensure that these modications can be safely applied in production. We also plan to rene the congurations of the queues and investigate which parameters could be further tuned, e.g., handle array jobs more specically, to improve the quality of service experienced by CC-IN2P3's users.

Fig. 1 :

 1 Fig. 1: Utilization of the resources, in terms of slots, over the considered period. The dashed line indicates the number of available slots, the other lines show the global utilization and the respective contributions of Local and Grid jobs.

Fig. 2 :Fig. 3 :

 23 Fig. 2: Cumulative distribution functions of Local and Grid jobs requested (left) and consumed (right) run times.

Fig. 4 :

 4 Fig. 4: Evolution of the number of waiting Local and Grid jobs.

Fig. 5 :

 5 Fig. 5: Daily arrival rate for Local and Grid jobs on week days (left). Dashed lines depict the average number of submissions per hour. Submission pattern of a specic user group (right). Triangles correspond to submission bursts.

Fig. 6 :

 6 Fig. 6: Actual (UGE) and simulated (Alea) resource utilization, in slots, for Grid (top) and Local (bottom) jobs.

Table 1 :

 1 Characteristics of the nodes in the CC-IN2P3's HTC computing farm.

	Model	Nodes Cores/Node Memory/Node	Cores
	Xeon E5-2650 v4 2.20GHz	232	48	144 GB 11,136
	Xeon Silver 4114 2.20GHz	240	40	128 GB	9,600
	Xeon E5-2680 v2 2.80GHz	149	40	128 GB	5,960
	Xeon E5-2680 v3 2.50GHz	124	48	144 GB	5,952
	Xeon E5-2670 0 2.60GHz	24	32	96 GB	768
	Xeon Silver 4114 2.20GHz	1	40	1,512 GB	40
	Total	770			33,456
	2 System Description
	The IN2P3 Computing Center [14] is one of the largest academic computing
	centers in France. At the time of writing of this article, the CC-IN2P3 provides
	its users with about 33,500 virtual cores (i.e., hyper-threading is activated) on
	770 physical nodes, whose characteristics are given in Tab. 1. In addition to this
	HTC computing farm, the CC-IN2P3 also oers resources for parallel, GPU-
	based, large memory, and interactive jobs that we ignored in this study.

Table 2 :

 2 Names and upper limits (per job) of the queues. Queues are listed in the order in which they are considered by the job scheduler.

	Queue name	CPU Time Time Memory File Size Pool size (in cores)
	mc-long	48h	58h	3.6GB	30GB	33,456 (100%)
	mc-huge	72h	86h	8GB	30GB	9,040 (27%)
	mc-longlasting	202h 226h	3GB	30GB	19,800 (59%)
	long	48h	58h	4GB	30GB	33,456 (100%)
	huge	72h	86h	10GB	110GB	10,418 (31%)
	longlasting	168h 192h	4GB	30GB	3,931 (12%)

Table 3 :

 3 Names and upper limits (per job) of the queues in the new conguration. Queues are listed in the order in which they are considered by the job scheduler.

	Queue name CPU Time Time Memory File Size Pool size (in cores)
	local-short	6h	7h	4G	30G	20,000 (59.8%)
	local-medium	24h	28h	4G	30G	15,000 (44.8%)
	local-long	48h	58h	4G	30G	10,000 (29.9%)
	grid	48h	58h	3.6G	30G	25,000 (74.7%)
	huge	72h	86h	10G	110G	10,000 (29.9%)
	longlasting	202h	226h	3G	30G	5,000 (14.9%)

Table 4 :

 4 Distribution of job waiting times.

	Percentiles

Table 5 :

 5 Impact of RQS relaxation on job waiting times.

	Percentiles

Acknowledgements

We kindly acknowledge the support provided by MetaCentrum under the program LM2015042 and the project Reg. No. CZ.02.1.01/0.0/0.0/16_013/0001797 co-funded by the Ministry of Education, Youth and Sports of the Czech Republic. We also thank L. Gombert, N. Lajili, and O. Aidel for their kind help.


M Legrand

E De Montalivet

F Richer

N Jarrassé

G Morel

Reciprocal Kinematic Control: using human-robot dual adaptation to control upper limb assistive devices

INTRODUCTION

Upper limb (UL) assistive robots, such as exoskeletons, prostheses or supernumerary limbs, can rarely be fully autonomous devices. Indeed, it is generally not possible to use pre-defined patterns of motions because of the great diversity of tasks and the variety of UL movement strategies to achieve any of them. Control has to be provided to the users. For that purpose, the most widespread solutions to obtain user's motor intention use physiological signals (electromyograms or electroencephalograms e.g.) [START_REF] Oskoei | Myoelectric control systems -A survey[END_REF][START_REF] Velliste | Cortical control of a prosthetic arm for self-feeding[END_REF], distal functional joints (for instance, head or foot motions control the endeffector position and/or orientation) [START_REF] Resnik | Controlling a multi-degree of freedom upper limb prosthesis using foot controls: User experience[END_REF][START_REF] Jackowski | Head motion and head gesture based robot control: a usability study[END_REF] or inter-joint synergies models [START_REF] Merad | Intuitive prosthetic control using upper limb inter-joint coordinations and IMU-based shoulder angles measurement: a pilot study[END_REF]. Despite interesting results, they all still have important limitations: the first two are neither natural nor intuitive and suffer from robustness issues, the third one does not allow very versatile devices. To tackle these issues, we propose a new control approach, together with a new paradigm, that uses the motion strategies naturally developed by the Central Nervous System (CNS). When a limb mobility is reduced, or when an assistive device does not work properly, CNS compensates and takes advantage of motor redundancy of the body: it calls other joints to still perform the desired gesture. Typical compensatory joints for UL movements are the trunk and the scapula [START_REF] Cirstea | Compensatory strategies for reaching in stroke[END_REF]. Our concept is to servo the robot to these body compensations. The only task of the latter is to make its user come back to a comfortable posture, and this indirectly leads to the realisation of the intended motion. The reciprocal adaptation between human and robot allows both to reduce the body compensations and perform UL movements with the assistive device. We validated a proof of concept of this paradigm on ten healthy subjects who executed a path-tracking task with an elbow exoskeleton.

MATERIALS AND METHODS

Control law

The control law is built in such a way that the user only has to focus on the end-effector (EE) position, as he/she would have done with a healthy arm. The aim is that the EE position, first reached with body compensations, remains constant, while the device moves and makes the user's compensatory joints go back to a reference position. An integration step is then added to prevent any rigid position-position coupling that does not enhance the mobility. The approach has been first applied to control an elbow joint with the following law (see Fig. 1(a) for the definition of the anatomical parameters): 1. Compute δ the distance between the EE position and the reference position of the acromion (chosen to define a non-compensatory posture, as it reflects both trunk and scapula, the main joints involved in upper body compensations), 2. Compute βn the angle the elbow should have to allow the EE to be in its current position without any compensation, according to (Eq.1) with Lua and Lfa the lengths of the upper arm and the forearm respectively. 3. Compute the angular velocity command to be sent to the robotic elbow:

(Eq.2) with β the current elbow angle and λ the gain of the integrator set to 2. An activation threshold (Δβ>5 degrees) was set to avoid instabilities. Ten healthy subjects, aged 20-23, who all gave their written consent, wore an elbow exoskeleton prototype that guided their motions. The task consisted in following a moving target, carried by a WAM®Arm (Barrett Technology), that drew a rectangle in the sagittal plane of the subject. The dimensions of the rectangle were adapted to the subject's morphology (see Fig. 1(b)). There were five repetitions. The task was performed in three different ways (later called modes):

1. Natural (N): the subject did not wear the exoskeleton but moved freely, without any specific instructions nor constrains. This is to be used as the reference. 2. Elbow fixed (F): the robotic joint was locked at 90 degrees, it prevented any elbow motions. This worstcase scenario showed the body compensations that can be exhibited when the mobility is reduced. 3. Reciprocal Kinematic (RK) control: the subject wore the device which, commanded with the proposed control scheme, guided the movements of his/her elbow. The end-effector and the acromion positions were tracked in real time (f=100Hz) with the motion capture system Optitrack (NaturalPoint Inc.). The reference position was defined as the initial position of the subject. The subjects received no instructions nor explanation; they were only told to perform the task and that the assistive device would help them. A training on ten repetitions was allowed before recording five.

RESULTS

The first thing to notice is that the task was correctly performed with the three modes (no statistical difference between the precision errors). To evaluate the performance of RK control, several metrics were analysed; two are presented here: the range of motion (ROM) of the acromion, which represents the compensatory displacements, and the ROM of the elbow (see Fig. 2). Statistical analysis was performed: Lilliefors test was used to assess the normality of the data, then general linear models for normally distributed data and nonparametric Friedman test for the others. The ROM of the acromion is significantly higher for the F-mode than for N and RK ones (p<0.05) while no statistical difference exists between N-and RK-mode. This confirms that, due to reciprocal adaptation, RK control indeed reduces the user body compensations. Concerning the elbow ROM, F-mode is absent as there is no movement of the joint. With RK-mode, we see an over-extension compared to the natural ROM. More detailed analysis shows that this over-extension appeared in the third stage of the movement, when the target is the furthest.

DISCUSSION

To build a natural and intuitive control for UL assistive devices, we propose to servo the robot to the body compensations of its user. This new paradigm, in which the device only focuses on correcting its user's posture, was tested with an elbow exoskeleton. It makes it possible to perform a path-tracking task, while requiring only minimal compensatory movements to work. Without any explanation provided, the users mastered the use of the device with RK in few trials, which highlights the intuitiveness of the control. Some points of our experiment yet deserved more detailed comments. First, the motions performed with RK tend to show an over-extended elbow in the third stage of the task. This could be because the subjects leaned on one side to see better their end-effector, which led to unwanted elbow activations. This points out that the trunk is not always compensatory but can be functional (i.e. essential to do the task). We are working to detect the utilisation of the trunk (compensation or function) and adapt the answer of the device to avoid undesirable activations. Second, we worked with a motion capture system, which cannot be used in everyday life. However, the information it gave (positions of EE and acromion) can be obtained with other techniques, more suitable for home-use. For instance, it shall be possible to do the same with a simple set of two IMUs, one on the trunk and one on the arm, along with few anatomical data of the subject. The preliminary results we obtained attest the usability and the intuitiveness of RK control. They support future tests on disabled subjects (amputees or post-stroke patients e.g.) as well as further developments for several degrees of freedom assistive devices. Moreover, RK control is task-independent, as the error vector used in the control law depends only on the reference posture (neither tasks nor compensations are explicitly defined). The experimental test of this versatility with different tasks is scheduled in the very near future. The reference posture was fixed for this experiment but further studies will make possible to have an automatically adjusted one. RK control is thus promising for a natural and versatile control of UL assistive devices, from prostheses to exoskeletons or even supernumerary arms.

Validation

Fig. 1 (

 1 Fig.1(a)Anatomical parameters of the control law. (b)Experimental set-up. A WAM®Arm draws a rectangle in the sagittal plane of the subject; it pauses briefly at each corner. Numbers 1 to 4 indicate the movements steps. The dimensions are adapted to the subject's morphology. The subject is wearing a robotic joint acting like an elbow exoskeleton and a wrist splint with a rod attached to it.

Fig. 2

 2 Fig.2 Performance of RK controlled-motions compared to natural and fixed-elbow ones. (a) ROM of the acromion. (b) ROM of the elbow. Metrics are averaged over the five trials and last columns is the mean over the ten subjects.