Anis Najar

Olivier Sigaud

Mohamed Chetouani

Training a robot with evaluative feedback and unlabeled guidance signals

In this paper, we present a new method for training a robot by natural interaction using evaluative feedback and unlabeled guidance signals. Feedback signals are directly mapped to reward values and used for learning both the task and the meaning of the guidance signals. The learned guidance signals are used in return to bootstrap task learning. We propose to use unlabeled guidance signals as an alternative solution to preprogrammed guidance. We evaluate our method both in simulation and on a real robot.

I. INTRODUCTION

As more and more robots are intended to evolve in human environments either for domestic or industrial purposes, many efforts are made in designing new methods that would enable persons with no programming skills to teach them new tasks [START_REF] Argall | A survey of robot learning from demonstration[END_REF], [START_REF] Knox | Training a robot via human feedback: A case study[END_REF], [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF]. The main challenge is to design teaching methods that satisfy several criteria such as simplicity, efficiency and safety. In addition to these requirements, robots endowed with such learning capacities should be able to adapt to different user profiles having different preferences and teaching strategies.

The Reinforcement Learning (RL) framework provides a set of methods allowing an agent to autonomously learn a task [START_REF] Sutton | Reinforcement learning: An introduction[END_REF]. This is done through a predefined reward function that allows the robot to optimize its behaviour. While the main advantage of RL is the autonomy of the learning process, when applied to real-world problems it presents some limitations such as the difficulty to implement a reward function [START_REF] Ng | Algorithms for inverse reinforcement learning[END_REF], slow convergence and unsafe exploration [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF].

The Interactive Reinforcement Learning literature deals with these problems by allowing a human to guide the learning process of the robot by providing it with teaching signals such as instructions [START_REF] Grizou | Robot learning simultaneously a task and how to interpret human instructions[END_REF], advice [START_REF] Griffith | Policy shaping: Integrating human feedback with reinforcement learning[END_REF], demonstrations [START_REF] Argall | A survey of robot learning from demonstration[END_REF], guidance [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF], [START_REF] Thomaz | Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance[END_REF] and evaluative feedback [START_REF] Knox | Training a robot via human feedback: A case study[END_REF], [START_REF] Pilarski | Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning[END_REF].

Evaluative feedback represents an intuitive way for training a robot by evaluating its behaviour through positive and negative rewards. This way of providing rewards has some advantages over traditional RL reward functions such as being more directly informative about the optimal behaviour and easier to implement [START_REF] Knox | Training a robot via human feedback: A case study[END_REF]. Some examples of successful implementations can be found in [START_REF] Knox | Training a robot via human feedback: A case study[END_REF], [START_REF] León | Teaching a robot to perform task through imitation and on-line feedback[END_REF], [START_REF] Pilarski | Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning[END_REF]. However, feedback is still limited as it is only reactive to the robot's actions, so it does not allow to anticipate them: As the robot may perform many undesired and/or unsafe actions before trying the appropriate one, feedback does not allow to limit these unsuccessful trials nor to prevent the robot from *This work was supported by ROMEO2 project 1 Sorbonne Universités, UPMC Univ Paris 06, UMR 7222, ISIR, F-75005, Paris, France. anis.najar@isir.upmc.fr repeating them in every different state, which may result in problems in terms of efficiency and safety considerations [START_REF] García | A comprehensive survey on safe reinforcement learning[END_REF].

To deal with this problem, guidance can be used in addition to evaluative feedback to constraint the exploration towards a limited set of actions [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF], [START_REF] Thomaz | Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance[END_REF]. Classically, the meaning of guidance signals is supposed to be known by the robot before learning the task. However, as guidance signals could be task specific, it would be difficult to non expert users to program their meaning for every task. In addition, handcoded guidance would limit the possibility for different teachers to use their own preferred guidance signals.

One possible solution to overcome this limitation would be to use unlabeled guidance signals. Very few works have considered the question of learning from unlabeled teaching signals. This question is difficult since it requires to solve two problems simultaneously: learning the meaning of teaching signals and using these signals to learn the task. In [START_REF] Mohammad | Learning interaction protocols using augmented baysian networks applied to guided navigation[END_REF], the robot learns to associate human gestures to actions in the context of a navigation task, but does not use them to learn the task. In [START_REF] Grizou | Robot learning simultaneously a task and how to interpret human instructions[END_REF], the authors address both questions simultaneously by using unlabeled teaching signals to learn a pick and place robotic task. In a similar approach [START_REF] Najar | Social-Task learning for HRI[END_REF], we proposed a model that learns the meaning of teaching signals from task rewards while using them to accelerate task learning.

In this paper, we propose a modified version of the model in [START_REF] Najar | Social-Task learning for HRI[END_REF] that learns the meaning of guidance signals by using evaluative feedback instead of task rewards. In this model, unlabeled guidance signals indicate the optimal action while evaluative feedback is converted into reward values. This model also extends previous research [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF], [START_REF] Thomaz | Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance[END_REF] by relaxing the constraint of programming the meaning of guidance signals. We evaluate our model on the task proposed in [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF] and we compare its performance with respect to using only evaluative feedback.

In the next section, we provide a description of our model. Then we present our scenario in Section III. Sections IV and V report respectively the results obtained in simulation and on a Baxter Research Robot. Finally, we discuss the advantages and the limitations of our model in Section VI.

II. METHOD

The main idea of our model is to learn the meaning of guidance signals while using them for guiding the learning process. On the one hand, learning the meaning of guidance signals consists in learning the correspondence between each signal and the set of actions to which it refers. In this paper, we consider the special case where each guidance signal corresponds to a single action. So, here guidance can be considered as instructions or demonstrations that indicate the optimal action [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF]. In this case, interpreting guidance consists in finding the optimal action for each given signal. This problem is similar to finding the optimal action for each task state, but performed on an alternative state space of reduced complexity. On the other hand, using guidance for learning consists in transferring the information about the optimal action back to the original state space. In our model, these two processes are performed simultaneously (Fig. 1).

The motivation behind this idea is that in several applications, the state space is much more complex than the action space. While task states can be defined over complex and noisy sensory inputs, a limited number of actions can be sufficient to explore the entire state space. This is the case for example in industrial tasks where the robot has to perform a sequence of actions on different objects in different situations (cf. Section III). The set of actions can be the same for all objects, only their sequence may differ. So several states would share the same optimal action. Fig. 1: An example of task with 4 states and 2 actions: A teacher can provide a guidance signal g i ∈ G for each state s i ∈ S. When the robot performs action a, the teacher provides it with a reward r. This reward is used for learning the optimal action a * associated to each guidance signal g i ∈ G. This knowledge is then transferred to each task state s i ∈ S.

A. Model

We model our problem within the Reinforcement Learning framework as a Markov Decision Process (MDP) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] represented by a tuple < S, A, T, R, γ >. S is the state space of the task. A is a predefined set of actions that the robot can perform. T : S × A → S is a transition function between task states. R : S × A → R is a reward function and γ ∈ [0, 1] is a discount factor that determines how much future decisions should be taken into account. Within this formalism, a q-learning agent stores a state-action value function

Q : S × A → R computed by Q[s, a] ← Q[s, a] + α * (r + γ * max a Q[s , a ] -Q[s, a])
where r ∈ R(s, a), s ∈ T (s, a) and α a learning rate.

In our model that we call SVFB (for Social Value and Feedback, Fig. 2), we use two separate MDPs: a Task Model and a Teaching Model1 . The main difference between the two MDPs is in the state space domains. The state space S of the Task Model is defined over task states while the state space G of the Teaching Model is defined over the unlabeled guidance signals provided by the teacher. Both MDPs share the same action set A .

We use γ = 0 as discount factor for both MDPs. Many recent works point that this is more suitable for learning from evaluative feedback as it better reflects teachers expectations about how reward is processed by learning agents [START_REF] Ho | Teaching with rewards and punishments: Reinforcement or communication[END_REF], [START_REF] Knox | Reinforcement learning from human reward: Discounting in episodic tasks[END_REF], [START_REF] Thomaz | Teachable robots: Understanding human teaching behavior to build more effective robot learners[END_REF]. Consequently, the learning problem becomes a sequence of single-step problems and the update formula of q-learning can be rephrased in the form

Q[s, a] ← Q[s, a] + α * (r -Q[s, a]).
The reward functions of the two MDPs are defined by the probability of receiving a positive or a negative feedback for each state-action and guidance-action pair. In a given state, if the robot performs the optimal action corresponding to the guidance signal, the teacher provides it with a positive reward r = 1, and a negative reward r = -1 otherwise.

Finally, both MDPs communicate with each other through a third element, the Contingency Model that associates each task state to the corresponding guidance signal using:

ĝ(s) = arg max g∈G P (g|s); s ∈ S.

B. Algorithm

Algorithm 1 shows the different steps of our method. First, the robot evaluates the task state (line 1). If it receives a guidance signal from the human, it uses it with the task state to update the Contingency Model (lines 2 to 4). Then, the guidance signal is retrieved from the Contingency Model as the most likely guidance signal for the task state (line 5). This step allows to minimize the number of interactions for the teacher by recalling previously given guidance signals.

Second, if a guidance signal exists, the Task Model is updated towards the Q-values of the Teaching Model (lines 6 to 8). Then the robot performs an action (line 9). If a reward is given by the human (line 10), it is used for updating the Task Model (line 12). Finally, if a guidance signal exists, the reward is also used to update the Teaching Model (line 14). for all actions a in A do 8:

Q S [s, a] ← Q S [s, a] + α * (Q G [g, a] -Q S [s, a]
Q S [s, a] ← Q S [s, a] + α * (r -Q S [s, a]) 13: if g = null then 14: Q G [g, a] ← Q G [g, a] + α * (r -Q G [g, a])

III. SCENARIO

To evaluate our model, we consider the object sorting domain described in [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF] in which the robot has to learn to sort different objects according to their type.

A. Experimental setup

The experimental set-up (Fig. 3) is composed of a Baxter Research Robot facing a table on top of which we place three magnets. The magnets allow to place objects at three different positions on the table: left, middle and right. A webcam is placed between the robot and the table allowing to take pictures of the objects placed at the middle position. A Microsoft Kinect2 V2 sensor is placed on top of the robot's head and used for extracting guidance and feedback signals from the teacher. The robot is controlled using the ROS architecture presented in [START_REF] Najar | Social-Task learning for HRI[END_REF]. The screen on the robot's head is used as a transparency device for displaying some relevant information such as the task state, the associated guidance signal, the performed action and the perceived reward.

B. Task domain

The workspace is divided into three zones: z 1 , z 2 and z 3 . When the teacher places an object in z 2 , the robot must pick it up and then place it in the appropriate zone. Unicolor objects (P lain) must be placed in z 1 while objects with patterns (P attern) must be placed in z 3 (Fig. 4).

The task state space is defined by the tuple S = (L lh , L rh , L o , D). L lh = {z 2 , z 3 } and L rh = {z 1 , z 2 } represent the locations of the robot's hands. Each hand can be located in two different positions above z 1 , z 2 and z 3 . L o = {z 1 , z 2 , z 3 , lh, rh, U nknown} describes the position of the object which can be located in one of the three zones or in one of the robot's hands. The object position may also be unknown to the robot. Finally, D describes the features of the object 3 . For our experiments, we follow the same protocol as [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF] by considering two different conditions:

Small state space: In this condition, the object descriptor D ∈ {P lain, P attern, U nknown} contains a single variable based on the number of Speeded-Up Robust Features (SURF) [START_REF] Bay | Speeded-up robust features (SURF)[END_REF] descriptors of the object. This variable can have three possible values: P lain, P attern and U nknown. Theses values are obtained by thresholding the number of extracted SURF descriptors. If this number is less than 50, the object is considered as P lain. Otherwise it is considered as P attern. The number of different task states resulting from this representation is 72.

Large state space: In this condition, the object descriptor D = (SU RF, COLOR, SIZE) contains three variables: SU RF ∈ {P lain, P attern, U nknown} describes the number of SURF descriptors as in the previous condition, COLOR ∈ {Red, Green, Blue, U nknown} describes the dominant color of the object that can be red, green or blue, and SIZE ∈ {Large, Small, U nknown} describes the area of the bounding box of the object. This representation yields a total number of task states of 864.

Finally, the robot is able to perform nine elementary actions that are necessary for completing the task (11 in [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF]) A = {T akeP icture, xM oveLef t, xM oveRight, xP ick, xP lace}, where x ∈ {Lef tHand, RightHand}.

C. Teaching protocol

For training the robot, the teacher can use either only feedback or feedback plus guidance. These information are extracted from the Kinect sensor. Feedback is divided in two categories f b ∈ {head nod, head shake}. By convention, head nods are converted into positive reward while head shakes are converted into negative reward. Guidance signals are defined over the teacher hand gestures. For each hand, the system recognizes five different states h ∈ {pointing right, pointing lef t, pointing middle, raised open, raised closed} which allows 35 possible guidance signals using either one or both hands. In this paper, we use one signal per action, so we only use nine guidance signals.

IV. RESULTS IN SIMULATION

In order to evaluate our model (SVFB), we compare its performance in simulation with respect to the model using only feedback (FB) as in [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF]. In addition, we report the performance of Q-learning and the previous version of our model (referred to as SVRL). Both Q-learning and SVRL rely on a goal-based reward function that is defined as follows: the robot receives a positive reward r = 1 for placing the object in the correct zone, a negative reward r = -1 for placing it in the wrong zone and a reward r = 0 for all other transitions.

In order to compare FB and SVFB in the same conditions, we assume that the teacher provides feedback for every action. However, guidance in SVFB is provided only once for each different state. In the simulated task, we consider only the large state space condition without the T akeP icture action, which means that the descriptors of the object are always known to the robot. For FB and SVFB, we use γ = 0, α = 0.3 and a greedy action selection strategy as exploration can be controlled by human rewards. For Q-learning and SVRL we use α = 0.3, γ = 0.75 and an -greedy action selection strategy with = 0.1 and a decay parameter for after each step of δ = 0.001 ( reaches 0 after 100 steps). All Q-values are initialized to 0.5. The results for each model are averaged over 100 runs with 200 trials each. In order to evaluate the cost of our method with respect to the feedback-only method, we compare the maximum number of teaching signals (feedback + guidance) provided by the human until convergence. As the teacher provides a feedback in every step, the maximum number of teaching signals needed if using only feedback is equal to 391. However, by using unlabeled guidance in addition to feedback, the results show that a teacher needs to provide at most 126 teaching signals (102 feedback +24 guidance signals).

We conclude from these results that using unlabeled guidance signals in addition to evaluative feedback reduces the number of steps (74% less) as well as the number of required teaching signals (67% less).

V. EXPERIMENTS ON THE REAL ROBOT

We compared our model to the feedback-only model with the Baxter Research Robot in both the small and the large state space conditions. We conducted four experiments for each model in each condition which resulted in 16 experiments. All the experiments were performed by one of the authors. A video of one experiment can be found online 4 .

In each experiment, all of the six objects are presented one by one to the robot in a specific order. Four different orders were chosen randomly beforehand and the same orders were employed for both models in both conditions. Each experiment ends when each of the six object has been presented two times for the small state space condition and three times for the large state space condition, whether learning converged or not. In all experiments, the teacher provided a feedback for every step. In addition, for the model with guidance, the teacher provided a guidance signal only when the robot did not receive yet any guidance for the given state or when the recorded signal was erroneous.

Figure 6 reports the evolution of the number of provided guidance and negative feedback over time for each condition. The results are averaged over the four experiments. We can see that in the small state space condition, the feedback-only model converged after at most 36 minutes, while the model with guidance converged within 17 minutes. In the large state space however, the feedback-only model did not completely converge after an hour of training, while the model with guidance converged after at most 24 minutes.

Table I reports some standard statistics of the experiments such as the training time, the number of steps, number of teaching signals, number of discovered states, number of undesired states and the number of steps spent in undesired states. Undesired states define situations in which the robot is holding the object with the wrong hand, or is holding the object while its descriptors are unknown (this may happen if the robot takes a picture after picking the object). We also report the size of the Q-function measured as the number of state-action pairs for which the algorithm learned a value.

The experimental results are consistent with those obtained in simulation and with the results of [START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF]. They show that guidance reduces considerably the number of steps and training time (42% less steps for the small state condition and 64% in the large state condition). It is also more efficient by achieving better performance with less interaction (30% less signals for the small state condition and 53% in the large state condition). The robot also discovered less states (respectively 29% and 43% less in each condition), less undesired states (resp. 42% and 65% less) and spent less time in these states (resp. 65% and 85% less).

Finally, the ratio of the number of actions per state for which the robot has learned a value while using only feedback is of 3 actions per state for both conditions. However, by using unlabeled guidance signals the robot learned a value for 7 actions per state in both conditions. This means that our model allows to determine more effectively the Q-value function in less time.

VI. DISCUSSION

In this section we discuss some aspects of our model to provide some insights about its performance.

We first evaluate the complexity of our model measured as the number of q-values that the robot needs to compute in the worst case scenario. As we already mentioned, the main idea of our model is to learn the task in an alternative state space of reduced complexity and to transfer this knowledge back into the original state space. In our case, the alternative state space is defined over the guidance signals. As each guidance signal corresponds to an action, the size of the guidance state space is equal to the size of the action set (as we do not consider the case where different guidance signals relate to the same action, nor the case where the same guidance signal refers to different actions). If n is the size of the state space and a the size of the action set, learning the task on the original state space comes to compute the values of at most n × a state-action pairs. However, if we learn the value function on the guidance state space, the complexity of the learning process would be reduced to a 2 state-action values. For example, in the large state space condition of the sorting task, the robot needs to learn only 81 values instead of 7776.

We consider the cost of our method in terms of human load measured as the maximum number of teaching signals needed from the teacher during the learning process. If the robot learns from a predefined reward function, so without evaluative feedback as in SVRL, a teacher needs to provide only guidance for every state, so in total n signals for the Teaching Model to be fully observable. If the robot learns from evaluative feedback instead of task rewards, in the worst case scenario a teacher using guidance needs to provide n + a 2 teaching signals instead of n × a if only using feedback (945 instead of 7776 for the large state space condition).

Our model has some other advantages over using only feedback or using feedback with preprogrammed guidance. On the one hand, feedback is purely reactive as rewards are given after the action is performed. So, it does not allow to anticipate the robot's actions. In our model, however, previously provided rewards are also transferred from the Teaching Model to the Task Model before action is performed (line 8 of Algorithm 1). This explains why it is more efficient in preventing from undesired actions (cf. Table I).

On the other hand, preprogrammed guidance signals are classically used for restricting the choice of possible actions. By doing so, they only inform the robot about the allowed actions but do not inform about the other actions. If we look at the way humans provide guidance, we find that guidance signals are more informative. For example, when a child is about to put his fingers into a power plug and receives the warning from his parent to "not do that!", this signal informs him that this action may be dangerous for him even if he does not actually perform it. This is an important aspect from a safety point of view. In our model, when the robot encounters a new state and receives a guidance signal which action values are already known, all these values are transferred to the task state (line 8 of Algorithm 1). So, the robot is informed about the outcomes of all these actions even if they are never explored in that state. This explains the difference in the number of learned state-action values per state compared to the feedback-only model (cf. Table I).

Another potential advantage of our model is the facility to correct wrong feedback. When using only feedback, if the optimal action is rewarded negatively in a particular state, its value is decreased compared to the other actions.

Consequently, the teacher has to wait until all other actions are explored and rewarded negatively in that state in order to make the robot try again the optimal action and thus to be able to correct the wrong feedback. With our model, however, as several states may share the same action, a wrong feedback given in a particular state has less influence on the value of the associated guidance-action pair. In addition, the teacher will have more frequently the possibility to correct the feedback as the guidance signal may be encountered in many different states.

Finally, if the task changes and has to be relearned, using only feedback requires to restart from scratch by giving once again the new feedback for the entire state-action space (n × a signals). With our model, once the mapping between guidance and actions has been learned in the Teaching Model, relearning the task is reduced to giving the new guidance signals for each task state (n signals).

Despite all these advantages, our model still has some limitations. First of all, the model relies on discrete guidance signals, which assumes a preprogrammed segmentation of the human gestures. Even with an important category of gestures, the way guidance is provided is still constrained by the gestures the system is able to detect.

Another limitation is the autonomy of the learning process with respect to the human. Using γ = 0 means that the teacher needs to provide guidance and evaluative feedback in every state, which can be a burden for the teacher in large and complex domains. Using γ ∈ (0, 1] may reduce the dependency on the teacher but may result in unsuccessful learning when learning from evaluative feedback [START_REF] Knox | Reinforcement learning from human reward: Discounting in episodic tasks[END_REF], [START_REF] Ho | Teaching with rewards and punishments: Reinforcement or communication[END_REF]. This is however more suitable when learning from a predefined reward function. Some promising solutions have been proposed for combining both types of reward [START_REF] Knox | Combining manual feedback with subsequent mdp reward signals for reinforcement learning[END_REF]. But the challenge is still to find an efficient way to design a reward function for real robotic tasks [START_REF] Ng | Algorithms for inverse reinforcement learning[END_REF].

VII. CONCLUSION AND FUTURE WORK

We presented a model for training a robot using unlabeled guidance signals and evaluative feedback. We demonstrated the advantage of using unlabeled guidance signals over using only evaluative feedback both in simulation and in a real robotic task. We have shown that our model reduces the training time and the number of interactions with respect to using only evaluative feedback.

Although we can consider that our proposed model generalizes action values over different task states by the means of guidance signals, in this paper we did not consider the question of generalization over state features. This is a challenging question as it requires to find the correct generalizations online within a minimum number of steps. Furthermore, we did not verify the robustness of our model against the sparsity and the inconsistency of teaching signals. Also, we considered only the particular case of guidance where there is a direct mapping between guidance signals and intended actions.

In the future, we propose to tackle these questions and to generalize our work to the case where different signals are used for the same action or where the same signal relates to different actions. We also propose to remove the constraint about the segmentation of guidance signals by taking into account continuous signals. Finally, we intend to verify the results obtained in this paper in a broader study by evaluating our model with naive users.

Fig. 2 :

 2 Fig. 2: The three elements of the SVFB model. The Contingency Model stores the contingency between task states s and guidance signals g. The Teaching Model learns the meaning of guidance signals using evaluative feedback r. The Task Model learns the task using both evaluative feedback r and the values learned by the Teaching Model.

Algorithm 1

 1 SVFB 1: s ← task state 2: g ← guidance signal 3: if g = null then 4: update contingency(s,g) 5: g ← get contingency(s) 6: if g = null then 7:

  )

  9: a ← arg max a∈A Q S [s, a] 10: r = teacher reward 11: if r = null then 12:

Fig. 3 :

 3 Fig. 3: Experimental setup. The table is divided into three regions: z 1 , z 2 and z 3 . A webcam allows to extract the features of an object placed in z 2 . A Kinect V2 sensor extracts feedback and guidance signals from the teacher.

Fig. 4 :

 4 Fig.4: Objects used in the experiments. We consider two types of objects: P lain (bottom) and P attern (top). In addition, we consider two different sizes and three colors.

Fig. 5 :

 5 Fig. 5: Probability to converge before n steps. For each experiment, we cumulate the number of steps over the trials until convergence. For each model, we compute the density histogram of this measure over all experiments then convert it into a cumulative distribution function.

Fig. 6 :

 6 Fig. 6: Number of guidance (blue) and negative feedback (red) over time.

Figure 5

 5 Figure 5 reports the probability for each model to converge before n steps. The results show that Q-learning converges in at most 2359 steps, SVRL converges in at most 1052 steps, FB converges in at most 391 steps and finally our model converges in at most 102 steps.In order to evaluate the cost of our method with respect to the feedback-only method, we compare the maximum number of teaching signals (feedback + guidance) provided by the human until convergence. As the teacher provides a feedback in every step, the maximum number of teaching signals needed if using only feedback is equal to 391. However, by using unlabeled guidance in addition to feedback, the results show that a teacher needs to provide at most 126 teaching signals (102 feedback +24 guidance signals).We conclude from these results that using unlabeled guidance signals in addition to evaluative feedback reduces the number of steps (74% less) as well as the number of required teaching signals (67% less).

TABLE I :

 I Experiment statistics. The results are averaged over four experiments. Training time, number of steps, number of provided feedback, negative feedback and guidance signals, number of discovered states, number of undesired states, number of steps spent in undesired states and number of learned Q-values.

We called it Social Model in[START_REF] Najar | Social-Task learning for HRI[END_REF], but Teaching Model is more appropriate.

https://dev.windows.com/en-us/kinect, accessed 20-

We rely on a more compact representation than[START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF] that removes redundancies between some state descriptors.

http://www.isir.upmc.fr/vid/intRL.mp4


Anis Najar
email: najar@isir.upmc.fr

Olivier Sigaud
email: sigaud@isir.upmc.fr

Mohamed Chetouani
email: chetouani@isir.upmc.fr

Socially Guided XCS: Using Teaching Signals to Boost Learning

Keywords: CCS Concepts, Computing methodologies → Reinforcement learning, •Computer systems organization → External interfaces for robotics, Human-Robot Interaction, Interactive Reinforcement Learning, Learning Classifier Systems

In this paper, we show how we can improve task learning by using social interaction to guide the learning process of a robot, in a Human-Robot Interaction scenario. We introduce a novel method that simultaneously learns a social reward function on the teaching signals provided by a human and uses it to bootstrap task learning. We propose a model we call the Socially Guided XCS, based on the XCS framework, and we evaluate it in simulation with respect to the standard XCS algorithm. We show that our model improves the learning speed of XCS.

INTRODUCTION

In addition to the autonomous capacity of humans to adapt to their physical environment, social interaction has been demonstrated to play a major role in enhancing human learning [START_REF] Meltzoff | Foundations for a new science of learning[END_REF]. These two different aspects of learning, autonomous and social, have been for a while a source of inspiration for many works in robotics [START_REF] Asada | Cognitive developmental robotics: a survey. Autonomous Mental Development[END_REF]. Our work aims at studying the relationship between these two aspects in the context of Human-Robot interactions. We consider a context in which a robot is endowed with the capacity to learn autonomously to accomplish a certain task by collecting rewards from its environment. In addition, the robot can get some help from a human that would provide it with teaching Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. signals. Initially, the meaning of these signals would be unknown to the robot, so it should learn to interpret them in order to use them afterwards to learn better and faster. Our idea is to learn a model of the human (the Social Model) that would assign reward values for each teaching signal, by using the rewards coming from the environment (the Task Model). Our research question is motivated by the fact that some human feedbacks like warning, encouragement, disapproval or consent could carry some affective values. We address the question of how these affective values could be grounded in the robot's experience and how they could contribute to the robot's learning process.

To illustrate our idea, we draw an example inspired from the child development literature. We consider an infant exploring its room. When alone, the infant is able to learn to interact with its environment only by autonomous exploration, by trying different actions in different situations and by privileging actions that yield more desirable outcomes. In some situations, a parent could intervene in order to guide him by providing advice or warnings about desirable or dangerous actions. For example, if he stands close to a power plug, his mother would warn him about this situation to prevent him from putting his fingers into the plug. Suppose in that moment, the child walks away and does not get any harmful outcome. On the one hand, he will be able to update the model of his environment (Task Model) by the information that, when he is near the power plug and decides to walk away, he does not receive any bad outcomes. On the other hand, he will also be able to incorporate the information that, when his mother warns him and he does the action of walking away from the power plug, he will not get any bad outcome (Social Model). Now suppose that by the next day, the child is facing the same situation, but this time he is alone. If this time he decides to touch the power plug, the negative outcome of this action will allow him not only to update his knowledge about his environment (Task Model) but also to get a better understanding of the meaning of his mother's warnings (Social Model), that will be then associated to danger, even if she is not present, only by remembering that she warned him the last day, when he was in the same situation. So in forthcoming situations, when the child gets warned by his mother, he will be able to exploit the meaning of his mother's warnings (Social Model) in a better way in order to interact with his environment (Task Model).

Starting from this intuition, we designed a system we call the Socially Guided XCS (SGXCS), based on two distinct models: one related to the task (Task Model) and another related to social interaction (Social Model). These two models influence each other in the sense that the learning of the one contributes to the learning of the other. This influence is carried out through a third model (Contingency Model), that serves as a bridge, by representing the contingency between task states and teaching signals. The Social and Task models are based on XCS, an RL system endowed with a generalization capability [START_REF] Sigaud | Learning classifier systems: a survey[END_REF]. The choice for XCS was motivated by its interesting features: the generalization property, the possibility to interpret the learned rules which could be of big interest for social interaction analysis, and the possibility to use the classifier's fitness as a confidence criterion on the reliability of the learned rules.

In the next section, we give an overview of related work. Then, in Section 3, we present our model. In Section 4, we show some experimental results about the performance of our system in simulation. In Section 5, we discuss the mechanisms of our model. Finally, we conclude this paper by providing some ideas for future work.

RELATED WORK

Reinforcement Learning (RL) [START_REF] Sutton | Reinforcement learning: An introduction[END_REF] offers a variety of techniques that allow a computational agent to autonomously learn to achieve a task by interacting with its environment. Compared with supervised learning, an RL agent does not require the presence of a supervisor to provide it with the right answers, but tries to optimize its performance by trialand-error while exploring its environment. However, when applied to a real world problem, the autonomous exploration aspect of RL raises some issues, such as slow convergence rate and unsafe exploration [START_REF] Kober | Reinforcement learning in robotics: A survey[END_REF].

In order to deal with these challenges, Interactive Learning has been proposed as a new paradigm for guiding the learning process of an agent by employing human teaching signals such as instructions [START_REF] Grizou | Interactive learning from unlabeled instructions[END_REF], demonstrations [START_REF] Argall | A survey of robot learning from demonstration[END_REF], advice [START_REF] Griffith | Policy shaping: Integrating human feedback with reinforcement learning[END_REF] or feedbacks [START_REF] Knox | Combining manual feedback with subsequent mdp reward signals for reinforcement learning[END_REF][START_REF] Thomaz | Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance[END_REF]. In [START_REF] Thomaz | Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance[END_REF], an RL-based software agent is guided by reward signals delivered by a human through a clicking interface. The values of these rewards are fixed in advance and added to the task rewards. [START_REF] Tenorio-Gonzalez | Dynamic reward shaping: training a robot by voice[END_REF] takes a similar approach by associating verbal feedbacks to predetermined reward values and by adding them to task rewards. Other approaches [START_REF] Isbell | A social reinforcement learning agent[END_REF][START_REF] Pilarski | Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning[END_REF][START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF], however, consider only the human rewards and not task rewards. In [START_REF] Knox | Combining manual feedback with subsequent mdp reward signals for reinforcement learning[END_REF], human generated rewards are used to learn a model of the the task reward function.

All these works have in common that they consider predetermined scalar values for human rewards, for example 1 and -1, and do not consider the question of learning these values. In contrast, in our work, only task rewards are predetermined while human rewards are learned. Furthermore, excepting [START_REF] Tenorio-Gonzalez | Dynamic reward shaping: training a robot by voice[END_REF] which considers verbally delivered feedbacks, all these works rely on an artificial reward interface such as a keyboard or mouse clicks and do not consider real teaching signals. In our work, we focus on the social interaction aspects by considering raw, nonverbal teaching signals. We find a similar approach in [START_REF] Grizou | Interactive learning from unlabeled instructions[END_REF], where the system learns to solve the task by using unlabeled teaching signals. However,in [START_REF] Grizou | Interactive learning from unlabeled instructions[END_REF], teaching signals are not considered as rewards but are rather used to infer the task reward function by Inverse Reinforcement Learning.

An important feature of these Interactive Learning systems is the autonomy of the learning agent with respect to the human. In [START_REF] Grizou | Interactive learning from unlabeled instructions[END_REF], the learning agent is entirely dependent upon the human teaching signals so it is able to learn the task only when the human is present. It is also the case in [START_REF] Isbell | A social reinforcement learning agent[END_REF][START_REF] Pilarski | Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning[END_REF][START_REF] Suay | Effect of human guidance and state space size on interactive reinforcement learning[END_REF], where the agent has no access to any rewards but those provided by the human. In contrast, in [START_REF] Thomaz | Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance[END_REF] and [START_REF] Tenorio-Gonzalez | Dynamic reward shaping: training a robot by voice[END_REF], human reward signals are added to task rewards, so the robot is able to learn autonomously beyond the human additional rewards.

METHOD

Our idea is to learn a model of the human rewards as a social reward function, that we distinguish from the task reward function. Formally, the distinction between the two functions lies only in the input space description. While the task reward function R T (s, a) is defined on task states s ∈ S T , the social reward function R S (s, a) is defined on human teaching signals s ∈ S S .

For this purpose, we propose a model we call Socially Guided XCS (SGXCS), based on three main components: a Task Model, a Social Model, and a Contingency Model. The Task and Social models are represented by two different Markov Decision Processes (MDP) and implemented by two different XCSs. The Task MDP serves to learn the task, while the Social MDP is used to learn the social reward function. The Contingency Model constitutes a mapping function between task states and social states (teaching signals), that allows to transfer the reward values from one model to another. The social reward function is learned by associating the rewards coming from the Task MDP to the corresponding teaching signals. It is then used for bootstrapping the learning process of the Task Model.

In the remainder of this section, we present our scenario, we give a detailed description of the three components of our model. Then, we present our algorithm.

Scenario

We consider a scenario (Figure 1) where a robot has to learn to press buttons. The experimental set-up is composed of a humanoid robot facing a table on top of which there is a set of buttons with different shapes and colors. At each moment, a screen displays the color or the shape of the button that the robot has to press. The robot is able to perform two kinds of action: gazing to one of the different buttons or pressing the one it is facing. So the task is a multi-step problem, meaning that in order to press the right button, the robot has to look for it first, and then to perform the action of pressing. While the robot is learning to perform the task, a human can sit in front of it in order to help it, as an adult would do with a child who is trying to solve a puzzle, by giving him some indications or feedbacks about his actions.

Model

In this section, we present the different components of SGXCS: the Task Model, the Social Model and the Contingency Model.

Task Model

The Task Model relies on two different principles, the use of the gaze as an intention device, and the use of affordances in action selection. Gazing as an intention device: Before taking an action, the robot looks at the object upon which it intends to act. We think that this could be a solution for the problem of random exploration by avoiding worthless and unsafe exploration.

Affordances in action selection: Instead of considering all the actions of the robot at the same level, we adopt a hierarchical model for actions by using the principle of affordances. The robot acts only on the objects it sees, and the set of possible actions is determined by this object. This could also contribute to limiting the drawbacks of exploration by reducing the space of possible actions in each state.

We represent the task as a Markov Decision Process M DP T =< S T , A T , T T , R T > where • A T ∈ {G1, ..., GN , P } is the set of possible actions in each state where Gi is the action to gaze the button i and P is the action of pressing the button that the robot is gazing.

• S T = C × S × G
• T T : S T ×A T → P r[S T ] is the task transition function.

• R T : S T × A T → is the task reward function.

In this preliminary work, we only consider a deterministic environment, meaning that T T and R T are deterministic. The action of gazing does not provide any reward. However, the robot receives a positive reward if it presses the right button and a negative reward if it presses a wrong button. The problem ends when the robot presses any button and receives a non null reward.

Social Model

In each state of the task problem, the human can provide to the robot an indication about the action to perform. We choose to use pointing and head movements as state features for the teaching signals. The human points to the button that the robot must press. If the robot is gazing to the right object, the human provides it with a head nod and if the robot is not gazing to the right object, the human provides it with a head shake. We model this problem as single-step, meaning that the problem ends after the robot performs any action, regardless of the value of the received reward.

The Social Model is a Markov Decision Process M DP S =< S S , A S , T S , R S > where • S S = H × P is the set of possible human states which are represented by two features H and P , where H ∈ {nod, shake} is the information about the human head movements, and P ∈ {1...N } is the information about the human pointing.

• A S ∈ {G1, ..., GN , P } is the set of possible actions in each state where Gi is the action to gaze at the button i and P is the action of pressing the button that the robot is gazing. Note that A T and A S are the same, meaning that we consider the same action set for both Task and Social models. So, in the remainder of this paper, we note it A.

• T S : S S × A S → P r[S S ] is the social transition function. As the problem here is single-step, we do not consider transitions between human states.

• R S : S S × A S → is the social reward function that we aim to learn.

Contingency Model

The contingency model represents the probability of observing a teaching signal in a certain task state. The model continuously receives states from both Task and Social models and updates their joint probabilities. When the robot receives a teaching signal s while being in a task state s , the Contingency Model updates the joint probability P (s ∩ s ). So, even if the human is not present, the robot can still use the Social Model by getting the most likely teaching signal from the Contingency Model: ŝ = arg max s∈S S P (s|s ); s ∈ S T

Algorithm

Algorithm 1 shows the different steps of our method. First, the robot evaluates the task state (line 1). If it receives teaching signals from the human, it uses them with the task state to update the Contingency Model (lines 2 to 4). Then, the social state is retrieved from the Contingency Model as the most likely teaching signal for the task state (line 5). Second (lines 7 to 11), if the social state is not empty, the Task Model is updated using the predictions of the Social Model, which are considered as social rewards. In this step, we rely on a minor modification of XCS, where the prediction array is used also to store the prediction of the most reliable classifier (with highest fitness). We need to do this because the prediction, computed as a weighted average of the predictions of all the classifiers belonging to an action set, can be distorted by newly created classifiers with low fitness but with high and wrong predictions; and so it can mislead the update of the Task Model. Then (lines 13 to 15), the robot performs an action, receives a reward from the task and uses it for updating the Task Model, like in the standard XCS algorithm. Finally, if the social state returned by the Contingency Model is not empty, the task reward is also used to update the Social Model (lines 17 and 18).

EXPERIMENTS

In order to evaluate the contribution of the Social Model on task performance, we conducted experiments in simulation by using a modified version of XCSLib [START_REF] Lanzi | Xcslib: The xcs classifier system library[END_REF] that allows employing two different instances of XCS, one for the Task Model and another for the Social Model. For these experiments, the human behavior was implemented in a simple manner, through a function in the Environment class of the Social Model that takes as input a task state and returns the corresponding teaching signal deterministically. The task was instantiated with 5 buttons.

In this section, we describe the settings of the experiments and then, we discuss the results.

Settings

In order to investigate how the Social Model contributes to the task performance, we performed the experiments in two different settings. In the first setting, we compare SGXCS with the standard XCS. As a second setting, we modified the Social Model in SGXCS so that, instead of returning a teaching signal for a task state, it returns the task state itself. So, here, the social MDP and the task MDP become equivalent, aside the fact that in the first one, we consider a single-step problem, while in the second the problem is multi-step. This setting would be equivalent to a modelbased XCS that learns the reward function on the task states and uses it to update its model as in Algorithm 1. We refer to this model as RBXCS.

For both settings, we run 10 experiments of 20000 problems, by alternating learning and test trials. We use the same XCS parameters for all models (see Table 1).

For performance evaluation, we use the following criteria: the evolution over the test trials of: the number of steps, the accumulated rewards and the population size at the end of the trial. For each criterion, we report the median, the minimal and the maximal values over the 10 experiments. The performance curves are smoothed by computing a moving average over a window of 500 trials.

Results

Figures 3 and4 report respectively the evolution of the number of steps and the evolution of accumulated rewards for XCS, RBXCS and SGXCS. We can observe that SGXCS outperforms both XCS and RBXCS in the number of steps and the accumulated rewards. Figure 5 reports the evolution of the population size for the three compared models. We can see that SGXCS and RBXCS equally outperform XCS, finding much more compact generalizations.

Figure 2 provides statistics about the performance of the three models over the 10 runs, in terms of number of trials until convergence. The convergence of the algorithm is considered when it reaches the optimal values in both the number of steps and the accumulated rewards criteria. A Mann-Whitney U test shows that the difference in performance between XCS and RBXCS is significant for a confidence level of 95%, while the difference between SGXCS and the two other models is significant for a confidence level of 99, 9%. Table 2 shows the learned rules in the Social Model. We can see that the model found correct generalizations on the teaching signals. Lines 1 to 5 predict with maximum accuracy a null reward on all the gazing actions, whatever the teaching signal. Lines 6 and 9 predict with maximum accuracy a reward of -1000 for pressing a button while the teaching signal contains a head shake (01 in the first two bits), whatever the pointing information (the last five bits). In the same way, lines 7 and 8 predict with maximum accuracy a reward of 1000 for pressing a button while the teaching signal contains a head nod (10 in the first two bits), whatever the pointing information.

DISCUSSION

In this section, we provide some insights about the mechanisms of our model and their implications for task learning. First, we evaluate the cost of our model in terms of interaction load for the user. Then we discuss its complexity. Finally, we review the rules learned by the Social Model.

Interaction load

First of all, we consider the cost of our method in terms of human load measured as the number of interactions needed from the human in order to reach optimal performance. In the experiments that we performed in simulation, the Environment class that models the human behavior in the Social Model is designed to interact systematically with the algorithm by providing teaching signals in each task state. So it assumes that the human is able to interact with the robot thousands of times, which is not reasonable. However, the Contingency Model allows to relax this assumption by memorizing the corresponding teaching signal to each task state. Consequently, to achieve optimal performance, the human needs to interact with the robot only the number of possible task states, by giving teaching signals only in newly encountered situations. This may be done for example through a hard-coded behavior that asks for help whenever the Contingency Model provides an empty response for an unknown situation. So, the system does not need to receive a teaching signal from the human in order to use it for updating the Task Model. The Contingency Model allows to update the Task Model with the rewards coming from the Social Model only through the teaching signals that have been previously provided by the human. So, even if the human is absent or not engaged with the robot, the Social Model still continues to boost the learning process of the Task Model (lines 7 to 11 of Algorithm 1). This remark is also valid in the opposite way, meaning that the Social Model can always be updated with an effective task reward, without the need for an effective teaching signal as it can get one from the Contingency Model (lines 5, 17 and 18 of Algorithm 1).

Furthermore, if the human does not interact with the robot, it will not worsen its performance with respect to the standard XCS algorithm. Indeed, if neither the human nor the Contingency Model provide a teaching signal in a given situation, the algorithm will behave normally as the standard XCS, by updating the Task Model only through effective task rewards (lines 13 to 15 of Algorithm 1). However, the more the human provides teaching signals, the more he improves the robot's performance. So the robot's learning process will be more or less improved depending on the level of the human engagement in teaching.

The genetic generalization mechanism provided by XCS allows to go one step further. In fact, a teaching signal can contribute to updating a task state even if they have never been contingent, and even to updating states that have never received a teaching signal. Consider two different task states s T 1 and s T 2 that have in common a set of matching classifiers M T (cf. Figure 6). Assume that s T 1 has a corresponding teaching signal s S 1 in the Contingency Model and s T 2 has no corresponding teaching signal. During the update of s T 1 by s S 1 , the classifiers in M T are updated by the social reward given by s S 1 . So, s T 2 is indirectly affected by s S 1 even if it has no associated teaching signal in the Contingency Model. This property is also reciprocal, meaning that the influence of a task state on a teaching signal through a task reward (line 18 of Algorithm 1) can be carried out indirectly through the generalization mechanism in the Social Model, even if the task state and the teaching signal have never been contingent. So, we can say that genetic generalization provides an extended influence to both Task and Social models, so that two states from both sides can influence each other even if they have never been encountered at the same moment. It also gives another dimension to the Contingency Model that acts beyond the state level, by reaching states at features level, even if it is not explicitly designed in this way. Consequently, the minimal number of interactions needed by the system to reach optimal performance could be less than the number of task states, depending on the extent to which the Task Model and the Social Model are generalizable.

Complexity analysis

The extended effect of rewards allowed by genetic generalization provides a better understanding of the contribution of the teaching signals in SGXCS with respect to task states in RBXCS. One could think that the same thing happens in RBXCS as in SGXCS, meaning that there is an extended propagation of the rewards in both directions because of generalization. So, let us consider the same example for RBXCS as previously, with two states s T 1 and s T 2 from the Task Model that share a common matching classifiers M T and a teaching signal s S 1 from the Social Model that is associated with s T 1 through the Contingency Model. The state s T 2 is indirectly influenced by the social rewards of s S

1 through M T , as in the previous example. But knowing that in RBXCS, s S 1 = s T 1 , the influence of s T 1 on s T 2 is already carried out by genetic generalization within the Task Model. So, with RBXCS, there is no additional information that could be provided by the extended influence of rewards, apart from the fact that, compared to XCS, the Task Model is updated several times in one step using the previously encountered rewards, whereas in the standard XCS the update is performed only once in each step for the current situation and the effective reward. So, we can say that teaching signals provide more information than those provided by the extended influence of rewards.

Actually, beyond the effect of genetic generalization, the Social Model in a generalization over the state-space of the Task Model. There are two main factors to that: the state-space complexity of each model and the mapping function between task states and teaching signals.

On the one hand, state-space complexity in the Task Model is in o(n 3 ), where n is the number of buttons. However, the state-space complexity of the Social Model in SGXCS is in o(n). So, the state-space of the Social Model is smaller than the state-space of the Task Model, which means that learning in the former is faster than in the latter.

On the other hand, the mapping function between task states and teaching signals contributes further to the reduction of the complexity of the Task Model. In fact, one single teaching signal can correspond to many different task states. This is due to the fact that teaching signals represent a global evaluation of the robot's state and provide an indication on the action to perform, beyond the details of the robot's state. For example, when the robot gazes at the wrong button, the human will always provide it with the same information not to press the button (through a head shake) and the same information about the button to press (through pointing), whatever the button that the robot is gazing at.

So, we can say that the state-space of the Social Model in SGXCS constitutes a certain generalization of the statespace of the Task Model, that contributes, in addition to the mechanisms that we mentioned above, to the acceleration of the learning process.

Social rules

In this paragraph, we review the rules learned by the Social Model and their contribution to the learning process.

In Table 2, we can see that the pointing features do not provide any useful information for the robot's decision mak-ing. Indeed, the "social rules" tell that, whatever the pointing, the robot has to press the button if there is a head nod and gaze to any button if there is a head shake, without specifying to which one. This is due to the fact that the Social Model learns to predict the direct rewards of the task, and so, it has a myopic vision of action outcomes. In other words, the Social Model does not learn, when the robot is gazing to the wrong button, that the pointing information could get it in a situation in which it can get a positive reward. It only tells not to press the button it is looking at, but not which button it has to look at. This is because in this model, head movements inform about the action of pressing or not, and the pointing informs about the action of gazing. As the action of pressing yields a direct reward and the action of gazing yields only long-term outcomes, the Social Model as it is designed (to predict direct rewards) has a myopic vision that does not exploit the long term information of pointing. One solution to that would be to use the Social Model to learn the state-action values instead of the reward function, so the information driven by the pointing would be more informative.

CONCLUSION AND FUTURE WORK

In this paper, we presented a method using the XCS framework, for guiding an RL agent with unknown teaching signals. Our model, the Socially Guided XCS (SGXCS), is based on three main components: a Task Model that is responsible for learning the task, a Social Model that learns a social reward function and uses it to bootstrap the learning process of the Task Model, and a Contingency Model that allows to associate task states with their corresponding teaching signals. We showed that our model improves the task learning speed with respect to the standard XCS algorithm.

In the experiments we performed in simulation, we considered only a deterministic environment. So, in future work, we propose to extend our model to the stochastic case. We also intend to test our model on other domains, specially domains with a sparse reward function.

Using unknown teaching signals and learning their values online provides to our model a certain flexibility for the human in choosing its own teaching strategy. For example, inverting head nods and head shakes would not affect the model performance. However, we have to study to what extent our model resists to inconsistent teaching signals. This question will also be addressed in future work.

As discussed in Section 5.1, the performance of our system could vary depending on the number of interactions with the human. So, we propose to study the evolution of the performance according to the level of engagement of the human in the teaching task. Furthermore, if we take into account the extended effect of rewards allowed by genetic generalization, another non-trivial question would be to find the minimal set of interactions required by the system to reach optimal performance.

Finally, in order to benefit from the long-term information provided by the teaching signals, we propose to investigate the possibility to extend our model to learn the state-action values instead of the reward function.

GECCO' 15

 15 Companion, July 11 -15, 2015, Madrid, Spain c 2015 ACM. ISBN 978-1-4503-3488-4/15/07. . . $15.00 DOI: http://dx.doi.org/10.1145/2739482.2768452

Figure 1 :

 1 Figure 1: Scenario: The robot must press the button corresponding to the the information displayed on the screen. A human can help it by providing it with teaching signals.

  is the set of possible task states which are represented by three features C, S and G, where C ∈ {1...N } and S ∈ {1...N } are respectively the color and the shape of the button that the robot has to press and G ∈ {1...N } is the button the robot is looking at.

Figure 2 :Figure 3 :Figure 4 :Figure 5 :

 2345 Figure 2: Number of trials to convergence for 10 runs. Mann-Whitney U test : p < 0.05( * ), p < 0.001( * * * )

Table 2 :

 2 Learned classifiers in the Social Model: the first two bits represent head movement information. The remaining bits represent the pointing information.

Figure 6 :

 6 Figure 6: Extended influence: The classifiers M T of the task state s T 2 are updated with the social reward coming from the teaching signal s S 1 , through the contingency of s S 1 and s T 1 . Similarly, the classifiers M S of the teaching signal s S 2 are updated with the task reward coming from the task state s S 1 , through the contingency of s S 1 and s T 1 .

Table 1 :

 1 Parameters used for XCS, Task Model and Social Model within SGXCS and RBXCS

	Parameter	Notation Value
	Population size	N	2000
	Learning rate	β	0.2
	Discount factor	γ	0.7
	GA threshold	θGA	25
	Crossover probability	χ	0.8
	Mutation probability	µ	0.04
	Accuracy criterion	0	1
	Accuracy falloff rate	α	0.1
	Accuracy exponent	v	5
	Prediction initial	p 0	10
	Error initial	0	0
	Fitness initial	f 0	0.01
	Deletion threshold	θ Del	20
	GA subsumption threshold	θGA sub	20
	Hash probability	P #	0.3
	Maximum steps	Ns	10

ACKNOWLEDGMENTS

This work is funded by the Romeo2 project.


Serena Ivaldi
email: serena.ivaldi@isir.upmc.fr

Salvatore M Anzalone
email: anzalone@isir.upmc.fr

Woody Rousseau
email: woody.rousseau@ensta-paristech.fr

Olivier Sigaud
email: sigaud@isir.upmc.fr

Mohamed Chetouani
email: chetouani@isir.upmc.fr

Robot initiative increases the rhythm of interaction in a team learning task

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Robot initiative increases the rhythm of interaction in a team learning task

[Extended Abstract]

INTRODUCTION

Verbal and non-verbal behaviors, interplays and interpersonal synchrony are critical to determine the functional role of humans during social tasks [START_REF] Dong | Modeling functional roles dynamics in small group interactions[END_REF]. Non-verbal communication is critical for achieving natural interaction between the human and the robot, especially to achieve engagement and synchronization in tasks where the two are teamed. Implicit non-verbal communication positively impacts human-robot tasks performance [START_REF] Breazeal | Effects of nonverbal communication on efficiency and robustness in human-robot teamwork[END_REF]. The inter-personal synchrony and the timing of the attention response during HRI can be used as an indirect measure of the human engagement toward the robot and the task [START_REF] Delaherche | Interpersonal synchrony : A survey of evaluation methods across disciplines[END_REF][START_REF] Rich | Recognizing engagement in human-robot interaction[END_REF]. Particularly in turn-taking scenarios where the partners collaborate in a systematic way, time is critical [START_REF] Steinfeld | Common metrics for human-robot interaction[END_REF]: synchrony, delays in replies, and rhythm of interaction can impact on the perception of the robot by the human, amplifying or decreasing the perceived engagement and influencing impression and responsive behaviors. This is true for robots [START_REF] Thomaz | Turn-taking based on information flow for fluent human-robot interaction[END_REF] and agents in general [START_REF] Mark Ter Maat | How turn-taking strategies influence users' impressions of an agent[END_REF]. Therefore, there is an apparent link between the timing of interaction, the role of the partners in the interaction and their impressions and responsive behaviors: while this has been shown for virtual conversational agents [START_REF] Mark Ter Maat | How agents' turn-taking strategies influence impressions and response behaviors[END_REF], there are still open questions about robots in teaching, collaborative and conversational scenarios. Here, we hypothesize that the reaction time of a human in response to a robot utterance can depend on the roles of the human and robotic partners during the interaction. We also hypothesize that the different roles of the partners can influence the rhythm of interaction and the perceived engagement [START_REF] Ivaldi | Robot initiative in a team learning task increases the rhythm of interaction but not the perceived engagement[END_REF]. We study the effects of a simple joint attention system during a learning task, in terms of induced joint attention (reaction time) and engagement perceived on the human side. We designed a two-step learning experiment, where people had to teach the iCub the color of objects. The learning situation was simple, as shown in Fig. 1, and interaction between robot and human was naturalistic. Remarkably, participants were not required to do any calibration nor wear eye-tracking devices, such as in [START_REF] Yu | Investigating multimodal real-time patterns of joint attention in an hri word learning task[END_REF]. Objects were selected by simply gazing at them by both pairs, and the information transfer about their color property was based on verbal communication (speech). We compare the effect of the initiator of the task in the teaching phase (human or robot) on the reaction time of the human gaze when responding to attention cues of the robot and the rhythm of interaction.

Figure 1: The object learning experimental scenario.

MATERIALS & METHODS

Experimental protocol. The experimental scenario is shown in Fig. 1: the human is standing in front of the robot, between the two there is a table with several colored objects.

The experiment consists of a supervised learning process, across two phases: a teaching phase and a verification phase.

In the first phase, the robot is taught the labels of the objects by the human partner. The object of interest is selected through joint gaze. In the second phase, the human gazes to one of the objects, and the robot responds with the learnt label. The teaching phase can be performed in two different conditions: Human Initiative (HI) and Robot Initiative (RI). HI and RI conditions are used to establish which partner initiates the action, that is the first who gazes at the object of interest.

Experimental procedure. We recruited 13 adult volunteers within the local campus population, mostly from the ISIR laboratory, who had no prior experience of interactions with iCub. Volunteers were divided into two groups, and associated to condition HI or RI. The only requirement for the participants was the lack of prior experience with the robot; that is we selected volunteers that had never interacted with the robot before. Age and sex were not control variables for our study, and interpersonal differences were not relevant. Participants were equally informed about the purpose of the study and the technological limitations of the robot. They were equipped with a lavalier microphone, and then could enter the robot's room. They were simply instructed to stay in front of the robot (their position was not fixed a priori) and do the teaching task. No calibration was required. iCub was not moving except for the head, which was a choice made to avoid proactive gaze effects. Three colored balls were placed on a table in between the human and the robot, on the left, on the center and on the right. During the two phases, participants were free to speak and interact with the robot in the way they were feeling more comfortable with. The teaching session was composed in both HI and RI conditions of only 3 trials, corresponding to the 3 objects to teach. Once the 3 colors were learnt, the verification phase would begin: the humans needed to verify that the robot had learnt correctly the 3 objects.

Robotic framework. The robot behavior was controlled by a pool of software modules developed in YARP and ROS [START_REF] Ivaldi | Object learning through active exploration[END_REF], particularly for robot gaze, 3D people tracking, head pose estimation, object recognition, verbal communication. Technical details about these modules can be found in [START_REF] Rousseau | Learning object names through shared attention[END_REF].

Measurements. The course of the experiment was controlled by a finite state machine. The timing of the events generated by the computer, the robot actions and the participants' responses were imported from the log of each experiment. The human gaze was continuously estimated from the RGB-D sensor through a gaze tracking module. Overall, gaze strategies for the HI and RI groups were diverse, but while the timing of the reactions was critical, the interindividual differences about gaze were not relevant for our study. We retrieved two important measurements. The first is the reaction time of the human in response to the attention stimulus of the robot, i.e., the request to select an object. We measure in this case the time elapsed between the onset of robot speech and the time when the human gaze, stabilized on the object of interest, is correctly identified by the robot.

The second is the interval between two successive requests from the robot, marking the amount of time dedicated by the partners to exchange information about the object of interest. This measurement is inversely proportional to the pace of interaction as it has been defined by [START_REF] Rich | Recognizing engagement in human-robot interaction[END_REF]. The shorter the interval, the higher the pace or the faster the rhythm.

RESULTS

Table 1 and 2 report the reaction time and the indirect pace measurement for the participants of the two groups. The time distributions were compared with Wilcoxon's test. The test shows that there is a difference in the timing between the two groups (p ≤ 0.005). People in the RI group react faster than the ones of the HI group, and the interaction with the robot has a higher rhythm (see Figure 2). Figure 3 shows the normalized gaze heat-maps of the two groups. Each map is a plot in the head's pitch-yaw space, thus each point represents the gaze direction of the human during the interaction with the robot. The range of pitch and yaw is [-90, 90] degrees. For the head pitch, 90 is on top of the head, 0 is in front of the head, -90 is below the head. For the yaw, 0 is in front of the head, while -90 and 90 represent left and right. We identified the four clusters associated to the robot head and the three objects by applying K-means on the points, indicated in the left upper corner of each plot. We compared the density of each cluster in the two conditions with Wilcoxon's test. The test showed that there is no significant statistical difference in the clusters for both conditions (p > 0.1). It is however interesting to observe the amount of time spent by the participants in looking at the different salient topics, which is proportional to the density of the clusters. Overall humans spent 66% of their time looking at the robot. For the three objects, the amount of time is unequal: while the left and right objects get almost the same amount of time (7% and 6%), the object in the center was the focus of attention for almost twice the time spent for the others (21%). This has a double explanation: on one side, it is more difficult for the robot to detect that the human has moved the head to gaze at the object of interest if the movement is exclusively on the head pitch; on the other side, sometimes participants spontaneously looked downward to match the robot's gaze (this behavior is in fact "normal" for humans and rather a

DISCUSSION

The response times reported in Table 1 show that humans respond faster to robot's utterances in the verification phase when in the previous phase of the task the robot was leading the interaction (RI condition). The measurements verify our initial hypothesis, that is the difference in the initiator/leader of the learning task in the first phase is reflected in different reaction times in the second phase of the task. In the HI teaching phase, the robot asks the human to choose an object, leaving the choice to the human, and making him the main actor of the interaction. Once the human has gazed to the object, and its gaze is correctly estimated, the robot looks at the object of interest. The rhythm of the interaction is essentially determined by the human response to the robot's utterance: in terms of time, the human can move more or less quickly his head, and make the movement more or less "readable" by the robot, thus influencing the time needed by the robot to estimate the head direction correctly.

Once the direction is estimated, the robot moves its eyes and head with a practically constant movement, determined by the gaze controller -the same used in [START_REF] Boucher | I reach faster when I see you look: gaze effects in human-human and human-robot face-to-face cooperation[END_REF]. In the RI teaching phase, the robot randomly picks an object on the table and asks the human to tell the color of the object. The choice in this case is made by the robot, which initiates the interaction. The rhythm of the interaction as well as its success is determined by the readability of the robot, its capability to induce in the human a prompt response to the robot attention request, and of course by the readability of the human that needs to have the same referential focus as the robot to make the interaction advance1 . Again, the duration of the robot's movements is fixed, so the human is the main actor responsible for setting the pace of the interaction through his behavior. Why do these two conditions reflect in different reaction times in the verification phase? There could be several reasons. One possible reason is that in the RI condition, participants learned how to "read" the robot behavior to advance in the teaching phase, and reply to its questions contingently. Therefore, in the verification phase they could be facilitated in responding promptly to the robot attention request. Another possibility is that, in the RI condition, the robot is interacting in a more "active" way, because it asks questions about the objects. As observed by [START_REF]Toward social robots[END_REF], this pro-active behavior regulates the interaction and provides a feedback signal to the human about the internal state of the robot. This behavior is also likely to induce in humans a social parenting effect: humans could have the impression that they are teaching the objects properties to a curious child. Conversely, in the HI case the robot acts "passively": it asks the human to provide the attention stimulus. So not only the learning process is led by the human, but the human could also be more hesitating in front of such request. The active/passive attitude could be responsible for making the robot more transparent to the human, in a way that the human would have or not a clear intuition about the robot's internal state. This claim is partially supported by some negative evaluations provided by the participants in the post-experiment questionnaire (see [START_REF] Ivaldi | Robot initiative in a team learning task increases the rhythm of interaction but not the perceived engagement[END_REF]). To summarize, the prior experience of an "active" robot leading the learning task makes the human react faster to the robot's attention utterances. Among the possible reasons, the robot active attitude improves its readability and the intuition of the human about the robot's state, hence the human reacts faster when he is interrogated by the robot. Our observations can be put in relation with the ones of [START_REF] Huang | Effects of responding to, initiating and ensuring joint attention in human-robot interaction[END_REF], where they showed that "a robot responding to joint attention is more transparent, such that interactive task performance is faster and more efficient". It is clear that the variable proactive element in the two conditions was decisive for setting the rhythm of the interaction in the two phases, especially in the verification phase. Though preliminary, our study suggests that the leading role of the robot can influence the pace of the interaction in a social task. Further, in this work we investigated also how social interaction can be made not only effective for the accomplishment of the task (i.e., teaching something to the robot) but also natural as if the human was teaching to another human. A remarkable result that suggest a natural interaction is the average amount of time that the participants in our study spent looking at the robot.

According to [START_REF] Argyle | Gaze and mutual gaze[END_REF], during human-human interaction people look at each other about 60% of the time, and look more while listening than while talking (during which they give frequent short glances). Interestingly the participants in our study looked at the robot about 65 -67% of the time, which is comparable with the human-human case. This could be a positive indicator of natural interaction 2 . Though preliminary, our results provide insights for improving the engagement system of the robot and make interaction with the human more natural and effective.

Figure 2 :

 2 Figure 2: Reaction time to robot attention stimuli and time interval between consecutive attention requests in the verification phase.

Figure 3 :

 3 Figure 3: Normalized gaze heat map of the human partners in the HI and RI groups. The plots show the points in the pitch-yaw space representing the gaze direction of the human partners during the interaction with the robot. Note that the range of pitch and yaw is [-90, 90]degrees. For the head pitch, -90 is on top of the head, 0 is in front of the head, 90 is below the head. For the yaw, 0 is in front of the head, while -90 and 90 represent left and right.

Table 1 :

 1 Reaction time (seconds) in response to robot attention stimuli (utterances) during verifica-

	tion phase
	Group	mean	std	median Wilcoxon's test
	HI	1.932 0.711	1.917	W=418,
	RI	1.296 1.145	1.106	p-value=0.005
	Table 2: Time interval (seconds) between consecu-
	tive robot attention stimuli (utterances) during ver-
	ification phase
	Group	mean	std	median Wilcoxon's test
	HI	9.524 1.515	8.588	W=447;
	RI	7.287 1.653	7.257	p-value=1.6e-5

positive sign of natural, engaged interaction with the robot).

In the experiment, the robot was programmed in a way that it was waiting for the human to match its referential focus. So an erroneous situation -i.e., the human looking at a different object than the one chosen by the robot-would have been caught. However, during the experiments this situation never occurred: the human always looked at the correct object pointed by the robot.

The work was partially funded by the European Commission, through the Projects CoDyCo (FP7-ICT-2011-9, No. 600716) and Michelangelo (FP7-ICT No. 288241), and by French National Agency of Research, through the Projects MACSi (ANR-10-BLAN-0216) and EDHHI/SMART (ANR-11-IDEX-0004-02).