Ashesh Vasalya

Gowrishankar Ganesh

Abderrahmane Kheddar

Distinct motor contagions during and after observation of actions by a humanoid co-worker

Keywords: Motor Contagions, Humanoid Robot Coworker, Human-Robot Interaction

Multiple studies have shown that the mere observation of movements by a robot can affect an observing human's movement; effects referred to as motor contagions. However, previous studies have either analyzed motor contagions induced during (which we call on-line contagions), or induced after (off-line contagions) observation of the robot, but never both together. It thus remains unclear whether and how these two contagions differ from each other. Here, in an empirical industrial co-worker setting, we examine the differences in the off-line and on-line contagions induced in participants by the observation of the same movements performed by a human, or a humanoid robot co-worker. We observed that while the off-line contagions predominantly affect the participant's movement velocity, the on-line contagions affect their movement frequency. Furthermore, the off-line contagions were prominent after observing another human, while the on-line contagions were equally strong with either a human or a humanoid coworker. These results suggest that actions by a humanoid robot can induce distinct effects on human behaviors, during and after observation.

I. INTRODUCTION

Motor contagions are implicit effects that cause certain features of an individual's action (like kinematics, goal, or outcome) to become similar to that of the observed action. Studies over the past two decades have reported various motor contagions in human behaviors caused by the observation of other humans as well as robots [START_REF] Blakemore | The role of motor contagion in the prediction of action[END_REF], [START_REF] Fadiga | Motor facilitation during action observation: a magnetic stimulation study[END_REF], [START_REF] Ganesh | Beyond watching: Action understanding by humans and implications for motion planning by interacting robots[END_REF], [START_REF] Sciutti | Measuring human-robot interaction through motor resonance[END_REF], [START_REF] Prinz | Perception and Action Planning[END_REF]. Understanding the effects of robots has recently developed pace due to the increased use of robots in co-worker scenarios with humans. In these scenarios, understanding how the behavior of robots affect humans can be beneficial for developing robot behaviors, both to ensure that they are perceived well and do not disturb humans, as well as for modulating human behaviors for the benefit of the task and humans.

Motor contagions may be divided into two categories depending on when, relative to the action observation, they are induced. On-line contagions are induced during the observation of actions performed by another human or robot [START_REF] Kupferberg | Video observation of humanoid robot movements elicits motor interference[END_REF], [START_REF] Oztop | Human-humanoid interaction: is a humanoid robot perceived as a human?[END_REF], [START_REF] Chaminade | Social cognitive neuroscience and humanoid robotics[END_REF], [START_REF] Kupferberg | Moving just like you: Motor interference depends on similar motility of agent and observer[END_REF], [START_REF] Brass | Movement observation affects movement execution in a simple response task[END_REF], [START_REF] Press | Robotic movement elicits automatic imitation[END_REF]. For example, [START_REF] Kilner | An interference effect of observed biological movement on action[END_REF] analyzed the variance in movements of a human participant when s/he observed spatially congruent and in-congruent movements made by either another human or a robot. Their experiment *This work is partially supported by the EU H2020 COMANOID www.comanoid.eu 1 CNRS-AIST JRL (Joint Robotics Laboratory) UMI3218/RL, Japan 2 CNRS-University of Montpellier LIRMM, Interactive Digital Human, France Fig. 1. Experimental setup: The participants in our experiment worked in three conditions; (i) with a robot co-worker performing biological movements (R biol ), (ii) a human co-worker (H), and (iii) a robot coworker performing non-biological movements (R nonbiol ). The coordinate axis defining the movement setup is indicated in white and fixed on the participant's table .   thus focused on on-line contagions, and showed that on-line contagions (in terms of a change in movement variance) are induced while observing a human but not while observing a robot making non-biological movements.

Off-line contagions on the other hand, are effects induced after the observation of actions by another human or robot [START_REF] Noy | Simple movement imitation: Are kinematic features sufficient to map perceptions into actions[END_REF], [START_REF] Kilner | Interference effect of observed human movement on action is due to velocity profile of biological motion[END_REF], [START_REF] Bisio | Automatic versus voluntary motor imitation: Effect of visual context and stimulus velocity[END_REF], [START_REF] Heyes | Automatic imitation[END_REF], [START_REF] Ikegami | Watching novice action degrades expert motor performance: Causation between action production and outcome prediction of observed actions by humans[END_REF], [START_REF] Ikegami | Prediction error induced motor contagions in human behaviors[END_REF]. For example, [START_REF] Bisio | Motor contagion during human-human and human-robot interaction[END_REF] measured changes in a participant's hand velocity, with and without an object, after observing the same movement being performed by a human or a humanoid robot. Their result shows that the observation of movement can subsequently affect a participant's hand velocity, both when the observed movements are by a human, or a robot, but again, only when the humanoid robot followed biological laws of motion.

Though both on-line and off-line contagions have been extensively investigated, all the previous studies have concentrated on either type of contagion and never analyzed the two together. Therefore, it remains unclear whether and how the on-line and off-line contagions are different in terms of the movement features they affect, and the magnitude of these effects. Here we address this question by comparing the on-line and off-line contagions induced in participants by the observation of the same actions, performed either by a human, or a humanoid robot.

We examined an empirical repetitive industrial task in which a human participant and a co-worker (either a humanoid robot or another human) work near each other. We systematically varied the behavior, specifically movement frequency, of the co-worker task and examined the on-line and off-line contagions that are induced. The induced contagions were examined when the robot made biological (or Fig. 2. A) Trial protocol: The participants worked in repeated trials with either a robot or human co-worker (the figure shows the trial with a human co-worker). Each trial consisted of a period when the participant worked alone and co-worker relaxed (participant-alone period), both worked together (together period), and the co-worker worked alone (co-worker-alone period). The notation of the kinematic and time variables (represented in general by η) in each period are shown in the figure . B) The trajectories made by the robot co-worker in the R biol and R nonbiol conditions. C) The time trajectories followed by the robot co-worker in the R nonbiol condition in the Y and Z dimension, and the via-points (blue circles) used to generate the trajectory. human type), and when it made non-biological (or industrial) movements. We specifically examined three questions:

1) Can on-line and off-line contagions from the observation of a same movement affect different movement features of the human participant? 2) how do the strengths of the on-line and off-line contagions vary with the nature of the co-worker (ie. if human or robot), and the behavior of the co-worker? 3) Consequently, are the on-line and off-line contagions different, or do they constitute the same effect observed at different instances?

II. MATERIALS AND METHODS

A. Participants

42 participants (20 males, 22 females of 11 nationalities, aged 20-39, mean±SD, 25.9±4.35) took part in our study. All participants had normal or corrected to normal vision. Two of them were left-handed according to the Edinburgh Handedness Inventory. The experiment was approved by the local ethics committee at the National Institute of Advanced Industrial Science and Technology (AIST) in Tsukuba, Japan. Before the experiment, the participants gave informed consent to participate in the study. All participants were naïve to the motive of the experiment.

B. Setup

Our experiment setup is shown in Fig. 1. The participants sat comfortably on a chair in front of a large table. A 'coworker', either a humanoid robot or a human experimenter, sat on the other side of the table. The participants were presented with two red circles of diameter 5cm at a distance of 50cm from each other, on a horizontally placed touchscreen (DELL HD Touch Monitor P2314T) on the table in front of them. The co-worker was similarly presented with two red circles of 9cm at a distance of 50cm. The whole setup was enclosed by movable panels.

Ten passive reflective markers were placed on the arms and hands of the participant and co-worker. These were tracked using six kestrel infra-red cameras (Motion Analysis Co.,) at 200Hz.

We used an HRP-2Kai (154cm tall, 58kg, 32DOF) humanoid robot [START_REF] Kaneko | Humanoid robot HRP-2Kai -Improvement of HRP-2 towards disaster response tasks[END_REF] as the robot co-worker. A trained experimenter (M, 37) acted as the human co-worker. Both coworkers used their right hand in the experiment.

1) Experimental task and conditions: Our task was motivated by the hand movements during an industrial pick-nplace or part-assembly task. Participants were required to repeatedly touch the red circles on the touch-screen with a stylus in the right hand. A co-worker (human or the HRP-2Kai robot), worked on the same task in front of them. The participants worked in a series of 50 second trials with the co-worker. In each trial, they initially performed alone for 10 seconds (participant-alone period), performed with the coworker for the next 20 seconds (together period), and then relaxed while watching the co-worker performs its/his task for the last 20 seconds (co-worker-alone period) (Fig. 2A).

The participants were instructed to "always hold the stylus like a stamp and touch alternatively inside each red circle on the touch-screen with continuous and smooth hand movements at a comfortable speed". No other instructions were given regarding the speed or movement trajectories. All participants wore headphones (through which we sent white noise) and had no audio feedback of the noise from the moving robot (confirmed in the post experiment questionnaire). They were specifically told to "focus on your own task and ignore the co-worker when he/it starts after them".

A total of six experimental conditions were studied. Some aspect of the co-worker's physical appearance or behavior was changed in each condition. We report results from three conditions relevant for distinguishing the on-line and offline contagions. The other conditions, in which we checked how the physical features of the co-worker effect participant behavior, are not considered in this study. 1First, in the Human (H) condition, the participants worked with a human co-worker. In the Robot biological (R biol ) condition, the HRP-2Kai robot played the role of the coworker and played back (biological) hand movements of a human volunteer (blue plot in Fig. 2B), that were recorded in a preliminary experiment (also see section II-B.2). Finally, in the Robot non-biological (R nonbiol ) condition, the participants worked again with the HRP-2Kai robot as the co-worker, but the robot now performed a non-biological movement profile that was roughly trapezoidal in shape and velocity profile (magenta plot in Fig. 2B).

The participants in our experiment were divided into six condition combination groups, with each participant in a combination group working in the R biol condition, and two of the remaining 5 conditions. This allowed us to compare the behavior of participants in any condition with his behavior in the R biol condition. Note that our each conditions lasted over 20 minutes, resulting in over 1 hour of total experiment time, and to avoid participants being tired we couldn't allow them to experience all the conditions. The order of the conditions was random across participants. Here we report results from participants in R nonbiol and/or H conditions, in addition to the behavior of the same participant's R biol condition.

In each condition, the participant worked in 10 trials. The co-worker performed at a constant, unique, pseudo-randomly selected frequency (in the range of 0.16 to 1.1Hz) in each trial. The pseudo-random nature of the co-worker performance was critical to avoid contamination by behavioral drifts across trials. The human co-worker was provided with a metronome using earphones (like in [START_REF] Bisio | Motor contagion during human-human and human-robot interaction[END_REF]), to cue him of the required movement frequency, and help maintaining the particular movement frequency.

2) HRP-2Kai movement trajectories: The arm movements played on HRP-2Kai in the R biol condition were a playback of the human hand movements recorded in a preliminary experiment with three volunteers (2 males and a female). The recording was done using the same (Motion Analysis Co.,) motion tracking system, while the human movements were cued by an audio metronome. Movements were collected at several frequencies between 0.16 to 1.1Hz. We found the movements of the three volunteers to be statistically similar in the x, y and z velocity profiles (p > 0.05), and showing similar trend in movement height with movement frequency -trajectory height consistently decreased with increase of movement frequency. We therefore chose to use the movements recorded from one volunteer (a male) in this experiment. We deemed this to be better than taking an average trajectory by the three volunteers so as to maintain not only the trajectory shape but also the variance characteristic of human trajectories.

The human movements in our task were characterized by smooth velocity changes and did not exhibit any via points (with direction changes). Hence, next for the R nonbiol condition, we utilized a robot trajectory with via-points. Inspired by the constant velocity and trapezoidal shape trajectories of industrial manipulators during pick-n-place task, we designed a trapezoidal trajectory for this condition. As the human movements were largely restricted in the YZ plane, we designed the robot movements in the YZ plane. We used two temporal via-points [START_REF] Biagiotti | Trajectory planning for automatic machines and robots[END_REF], and developed a piecewise polynomial in position-time profile using third order segments which restricted the slope (velocity) to zero at the start, the end and the via-points (see Fig. 2C). The initial and final Y positions (y 0 and y f respectively) were set to zero and 50cm, corresponding to the movements required by the participants. The maximum Z elevation (z max ) for the robot in the R nonbiol condition was set to 13cm one way and 8cm the other, again to give it a non-biological behavior.

III. DATA ANALYSIS

A. Variables

Our analysis is based on the position data of the markers placed on the participant's and co-worker's stylus. In order to tease out possible behavioral differences between the movements towards and back between the touch points, we analyzed behavioral variables across each movement between the red circles on the touch-screen, which we call as iterations (such that two consecutive iterations constituted a movement cycle). The participants and co-workers made continuous movements without stopping at the touches, and hence we could extract individual iterations by a participant or co-worker by examine the directional changes of their yvelocity in the recorded motion capture data. We concentrate on kinematic variables along the Y and Z axes inside each iteration and analyzed the maximum movement length (y max ), maximum movement height (z max ), maximum absolute velocities (max | ẏ|, max | ż|), mean absolute velocities (| ẏ|, | ż|), maximum accelerations (max(ÿ), max(z)), and minimum accelerations (min(ÿ), min(z)), by the participants and co-workers to understand whether and how the coworker behavior were affected by the on-line and off-line contagions. In addition to the kinematic variables, we also analyzed the time between the touches in each iteration, which we will refer to as the half-time period or (htp).

B. Participant sample size

Our initial sample size was 35 participants such that all of them participated in the R biol condition and 14 participants in each of the five other conditions (consisting of five participant groups, where participants performed in R biol condition along with one of the five condition). The number '14' corresponds to participant numbers in similar previous studies [START_REF] Bisio | Automatic versus voluntary motor imitation: Effect of visual context and stimulus velocity[END_REF], [START_REF] Bisio | Motor contagion during human-human and human-robot interaction[END_REF] and this participant number '14' also corresponds to the G* power analysis [START_REF] Erdfelder | GPOWER: A general power analysis program[END_REF] using two-way one sample T-test (α = 0.05, β = 0.85, d = 0.9) [START_REF] Verma | Determination of Sample Size and Power Analysis with G*Power Software: Step-wise Illustrated Manual for Research Scholars[END_REF] for the biological experiments. We observed substantial motor contagions in the htps in the R biol condition (median = 0.014, Z(31) = 3.14, p = 0.0016, 3 participants with slopes beyond the 95% confidence interval were removed as outliers). We thus assumed that a positive htp slope in R biol condition as true, and checked the htp values in the R biol conditions in each participant group. But with these participants numbers, the htp slopes in the R biol condition were not significant across the participant groups (p<0.05, one-way ANOVA). The htp slope during R biol condition were observed significant with two participant groups (p = 0.022, p = 0.038), marginally significant with other two participant groups (p = 0.07, p = 0.08) and not significant in one participant group (p = 0.36). We therefore decided to add 7 participants (50%) across these groups, making a total of 42 participants. This addition ensured that the htp slopes across the participant groups become similar (P = 0.99; one-way Kruskal-Wallis H-test). After removal of three outliers, this gave us participants numbers of 13 in the H condition, 18 in the R nonbiol , and 39 in total for the R biol condition.

C. Quantifying the off-line contagions

We quantified a participant's change in behavior after observing the co-worker, by analyzing how the average value of a given kinematic or time variable η p by a participant during the participant-alone period in trial i (η a p (i)), compared with that of the co-worker in the co-worker-alone period of the previous trial (η c (i -1)). We used the Akaike Information Criteria, or AIC [START_REF] Akaike | Information theory and an extensión of the maximum likelihood principle[END_REF] to choose either a first order or second order regression model to explain the data and performed the regression using MATLAB's fitlm function. Some representative fittings are shown in Fig. 3A. We then collected the slope at the minimum co-worker variable value (min[η c (i)]) across participants. The collected slope data for each variable and condition was checked for normality using the Shapiro-Wilk test and analyzed for a difference from zero either using a one-sample T-test or a Signed Rank test depending on whether the distribution was normal or not, respectively. The data plots of | ẏ| from the three conditions we report here are shown in Fig. 4A.

D. Quantifying the on-line contagions

For quantifying effects due to the on-line contagions, we looked again at the average value of each of the analyzed kinematic or time variable η p in the together period. However, in order to remove any persistent off-line contagions in this period, we regressed the change in the participant's behavior, between the together period and alone-period in a trial (η t p (i)-η a p (i)), and the corresponding value of the same variable in the co-worker behavior in the same trial η c (i). A first order, or second order regression model was chosen again using AIC for each participant, and like with the offline contagion analysis, the tangent slope at the minimum co-worker variable value (min[η c (i)]) was collected across participants, checked for normality using the Shapiro-Wilk, and then analyzed for difference from zero using a onesample T-test or a Signed Rank test. The fitting of htp in representative participants in the three reported conditions are shown in Fig. 3B and the collection of slopes are in shown in Fig. 5B.

E. Statistical correction

As reported earlier, every participant in our study participated in three conditions: the R biol condition, and two of the remaining five conditions. We therefore make two comparisons for each participant, between R biol and the two other conditions. Correspondingly, in our comparisons in Fig. 5, we use a Bonferroni correction of (3 conditions -1) 2, and all p values below 0.05 were multiplied by 2.

F. Movement congruency analysis

Finally, in the case of the on-line contagions, we examined if movement congruency between the participant and coworker influenced the on-line contagions in | ẏ| and htp of participants. We compared the velocity of the participant in every iteration to the velocity of the co-worker, and categorized it as a congruent iteration if the co-worker moved in the same direction as the participant for more than 50% of the iteration time, and as an incongruent iteration otherwise. We then performed the same regression analysis as described above to obtain two slopes for each participant, taking either their congruent, or incongruent iterations. We then averaged the difference of the two slopes across the participants to analyze whether congruency affected the on-line contagions. The plots of the difference of | ẏ| and htp between the congruent and incongruent iterations are shown in Fig. 6.

IV. RESULTS

A. Off-line contagions affect mean velocities but not htps

Recent studies [START_REF] Noy | Simple movement imitation: Are kinematic features sufficient to map perceptions into actions[END_REF], [START_REF] Kilner | Interference effect of observed human movement on action is due to velocity profile of biological motion[END_REF] have shown that off-line motor contagions affect the hand movement velocity of participants. Agreeing with these results, we observed (Fig. 4A) a significant positive slope between the mean absolute yvelocity (| ẏ|) of participants and the human co-worker in the H condition (median = 0.040, p = 0.017, orange plot in Fig. 4A). In the R biol condition, in which the robot coworker made biological movements, the slope tended to significance for the | ẏ| velocity (median = 0.017, Z(38) = 1.86, p = 0.063, blue plot in Fig. 4A). Finally in the R nonbiol condition, when the robot movement was not biological, the results again agreed with previous findings and the slope between the | ẏ| of participants relative to the | ẏ| of the robot was zero (p = 0.47, magenta plot in Fig. 4A). We also observed a positive slope between the maximum absolute yvelocity (max | ẏ|) of the participants and that of the human co-worker in the H condition (median = 0.54, p = 0.017), but this was absent in the robot co-worker conditions (R biol : p = 0.18; R nonbiol : p = 0.29). Overall, these observation support previous results which showed that the mean velocity of human participants are affected by off-line contagions after seeing a human or robot co-worker, but only when the robot co-worker performs biological movements.

On the other hand, we did not observe a significant effect on the participant's htps due to off-line contagions. The htp slopes were observed to be insignificant with human coworker in the H condition (median = 0.006, p = 0.06), as well as the robot co-workers R biol : (median = 0.007, Z(38) = 1.89, p = 0.06); R nonbiol : (median = -0.002, Z(17) = -0.18, p = 0.25). As can be seen however in Fig. 4B, the p values were marginally insignificant.

Note that it is not strange to observe a strong positive slope in the | ẏ|, but not in the corresponding movement times, or htps in our task. This is because the participant movements in our task were in the YZ plane, and therefore the htp, which is determined by when the participants touches on the touch-screen, depends not only on the y-velocity, but also the z-velocities of the participant. On the other hand, due to the same reason, any effect induced in the | ẏ| would partly show up in the htps, and this was probably the reason behind the marginal insignificance observed in the participant htps.

Finally, we did not find an effect (p > 0.1) on any of the other analyzed kinematic variables (maximum movement length (y max ), maximum movement height (z max ), maximum absolute velocities (max | ẏ|, max | ż|), mean absolute velocity (| ż|), maximum accelerations (max(ÿ), max(z)), and minimum accelerations (min(ÿ), min(z))) in all three conditions H, R biol and R nonbiol due to the off-line contagions.

B. On-line contagions affect htps and not mean velocities

We observed that the on-line contagions are distinct from off-line contagions. Primarily, unlike off-line contagions, we observed a significant effect on the htp of participants when they worked in parallel to the co-worker. The htp slope was strongly significant both, in the H condition, when they worked with a human co-worker (median = 0.014, p = 0.0017, orange plot in Fig. 5B), as well as in the R biol condition when they worked with a robot co-worker who made biological movements (median = 0.017, Z(38) = 3.70, p = 0.0002, blue plot in Fig. 5B). Again, no effects were observed when the robot co-worker's movement were nonbiological in R nonbiol (p = 0.777, magenta plot in Fig. 5B). On the other hand, while we did find an effect on the mean absolute y-velocity (| ẏ|) of the human participants in the H condition (median = 0.034, p = 0.022, orange plot in Fig. 5A), this effect was completely absent in the R biol condition (median = 0.013, Z(38) = 0.13, p = 0.90, blue plot in Fig. 5A), and, not surprisingly, in the R nonbiol condition (p = 0.39, magenta plot in Fig. 5A). We also found effects in the max(ÿ) and max(z), but only with the human co-worker (and not in any of the robot co-worker conditions). Hence we concentrate on htp changes in this manuscript.

The results in the R biol condition suggest that off-line contagions affect a participant's htp but not a participant's | ẏ|. However, in the H condition, we observed effects on both the htp and | ẏ|. To resolve this conflict, we next examined whether the results in the H condition were coupled; that is, whether the | ẏ| was indeed affected in the H condition, or whether it was a consequence of the effect on the htp. We separated movement iterations in the H conditions depending on whether a participant's movement was predominantly congruent (cong iterations), or incongruent (in-cong iterations) with an observed (co-worker's) movement (see section III-F for details), and compared the on-line contagions in these two types of iterations separately. Corresponding to previous studies [START_REF] Noy | Simple movement imitation: Are kinematic features sufficient to map perceptions into actions[END_REF], [START_REF] Kilner | Interference effect of observed human movement on action is due to velocity profile of biological motion[END_REF], [START_REF] Bisio | Automatic versus voluntary motor imitation: Effect of visual context and stimulus velocity[END_REF], we hypothesized that if the on-line contagions affect the | ẏ|, then the contagion strength (the Fig. 6.

Effect of congruency on on-line contagions: the difference in slopes, between the velocity congruent and incongruent iterations across participants, was zero for both the | ẏ| and htps of participants during the observation of the human (H, orange plot) condition and robot co-worker (R biol , blue plot) condition. The lack of effect difference suggests that the on-line contagion does not affect the movement velocities in our study.

signed slope) to be significantly different between the cong iterations, when the co-worker's movement corresponds to that of the participant, and the in-cong iterations, when the movements do not correspond. On the other hand, if the on-line contagions are in the htp, which is a time unit, the congruency of the observed movement (relative to the participant's own movement) should not change the strength of the contagions.

Our results indicate no difference in the | ẏ| and htp slopes in the cong and in-cong iterations of the H condition (p = 0.26 and p = 0.73, orange plots in Fig. 6 respectively). Similarly, no difference was observed between the slopes of | ẏ| and htp in the cong and in-cong iterations of the R biol condition (p = 0.76 and p = 0.59, blue plots in Fig. 6 respectively). These results strongly suggest that the on-line contagions predominantly affect the participant's htps and not velocity.

V. DISCUSSION AND CONCLUSION

We started with three specific questions in regard to affects on one's behavior during and after observation of a human or humanoid co-worker. Our findings on off-line motor contagions agree with the previous studies and showed that the mean absolute velocity (in the Y direction, as this was the predominant movement direction in our study) of human participants are implicitly affected after observing a human or robot co-worker, but only when the robot co-worker makes biological movements (Fig. 2B, blue plot). On the other hand, we found the effects on the participant's htps due to off-line contagions to be minimal (Fig. 4B). In contrast, the effects of the on-line contagions were observed predominantly in the participant htps during both, working with a human, as well as a robot co-worker, again only when the robot co-worker makes biological movements (Fig. 5B). An affect on the mean absolute y-velocity of participants was also observed while working with human co-workers (Fig. 5A), but our congruency analysis (section III-F) strongly suggest that this effect was in fact a spill-over of the effect on the htp. Overall these results suggest that on-line and off-line contagions from the observation of a same movement affect different movement features of the human participant. While the off-line contagions predominantly affect velocity, the online contagions are more prevalent in the movement rhythm or frequency (quantified by htp) of the movements.

The on-line contagions in our study are quantified as the relation (slope) between the subtraction between the human participant's movement feature when working with the coworker compared to working alone, and the co-workers feature. The subtraction removes the off-line effects (which are in fact due to the observation of a previous and different co-worker movement) in the participant behavior. Therefore, it should be noted that the lack of a particular effect in our on-line contagions analysis, does not mean that the effect is not present during observation of the co-worker. The on-line contagions in this study represents specifically effects that changed when working in parallel to a co-worker, compared to working alone.

We found the nature of the co-worker, that is whether he was a human or a robot, tended to affect the off-line contagions more than the on-line contagions. Strong offline contagions were observed in the participant's | ẏ| with the human co-worker (p = 0.017, Fig. 4A), but this effect seemed weaker when the co-worker was a robot (p = 0.063, Fig. 4A). The difference between the effects were however, not significant between the two conditions (p = 0.34), so it is difficult to conclude this definitely. However, in case of on-line contagions, the effect on the participant's htps was clearly visible, both with the human co-worker (p = 0.0017, Fig. 5B) and robot co-worker (p = 0.0002, Fig. 5B), and these affects were not different from each other (p = 0.62). These results suggest that the off-line contagion may be more sensitive to the nature of the co-worker than the on-line contagions. The two contagions are also probably affected by the age, and physical and behavioral charateristics of the partner, but these variables were not manipulated sufficiently in the current study and are left for future research. Finally, both the off-line and on-line contagions were observed only with the human co-worker and when the movement's of the robot co-worker was biological, and hence both were observed to be sensitive to the behavior of the co-worker.

Overall, our observations suggest that distinct motor contagions are induced in human participant's during the observation of a co-worker (on-line contagions) and after (offline contagions). The distinctions that were observed in the movement feature being affected, and the sensitivity of this effect to the nature of the co-worker, provide a better understanding of how human movements may be affected by robots working near them. This understanding will be critical to the physical and behavioral design of robots working near humans.

Fig. 3 .

 3 Fig. 3. Examples of linear regression fits obtained in the H (orange), R biol (blue), R nonbiol (magenta) conditions: A) Off-line contagions in the participant's | ẏ| (y-axis) as a function of co-worker's | ẏ| (x-axis); B) Online contagions in the participant's htps (y-axis) as a function of co-worker's htps (x-axis). We used the AIC to choose either a first or second order model to fit the data for each participant. The lines represent the tangent slopes at the minimal co-worker feature value.

Fig. 4 .

 4 Fig. 4. The off-line contagions: Observed changes in the participant's | ẏ| and htp in the H (orange plots), R biol (blue plots), R nonbiol (magenta plots) conditions. All p values are Bonferroni corrected.

Fig. 5 .

 5 Fig. 5. The on-line contagions: Observed changes in the participant's | ẏ| and htp in the H (orange plots), R biol (blue plots), R nonbiol (magenta plots) conditions. All p values are Bonferroni corrected.

They are submitted elsewhere that we cannot reference to meet the double blind submission requirements.

ACKNOWLEDGMENT

We much thank Dr Gergondet and Dr Audren for their support to operate HRP-2Kai.


François De

La Bourdonnaye

Céline Teulière

Jochen Triesch

Thierry Chateau

Learning to touch objects through stage-wise deep reinforcement learning

Learning complex behaviors through reinforcement learning is particularly challenging when reward is only available upon successful completion of the full behavior. In manipulation robotics, so-called shaping rewards are often used to overcome this problem. However, these usually require human engineering or (partial) world models describing, e.g., the kinematics of the robot or high-level modules for perception.

Here we propose an alternative method to learn an object palm-touching task through a weakly-supervised and stagewise learning of simpler tasks. First, the robot learns to fixate the object with its cameras. Second, the robot learns eye-hand coordination by learning to fixate its end effector. Third, using the previously acquired skills an informative shaping reward can be computed which facilitates efficient learning of the object palm-touching task. We demonstrate in simulation that learning the full task with this shaping reward is comparable to learning with an informative supervised reward.

I. INTRODUCTION

In recent years, deep reinforcement learning (RL) algorithms have allowed the learning of complex behaviors directly mapping pixels to actions without using hand-crafted features [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF], [START_REF] Levine | End-to-end Training of Deep Visuomotor Policies[END_REF]. However, learning can be prohibitively slow when reward is only available when the full task has been completed successfully. This problem of sparse rewards is frequently addressed by specifying more informative reward functions that rely on additional forms of supervision. Indeed, informative shaping rewards have long been applied *This work is sponsored by the French government research program "Investissements d'avenir" through the IMobS3 Laboratory of Excellence (ANR-10-LABX-16-01), by the European Union through the program Regional competitiveness and employment (ERDF Auvergne region), and by the Auvergne region. J.Triesch acknowledges support from the Quandt foundation and the European Union's Horizon 2020 Research and Innovation Programme under grant agreement no. 713010 (GOAL-Robots Goalbased Open-ended Autonomous Learning Robots).

1 F. de La Bourdonnaye, C.Teulière and T. Chateau are with the university of Clermont Auvergne, the Pascal Institute CNRS, UMR6602, Aubière, France 2 J.Triesch is with the Frankfurt Institute for Advanced Studies, Frankfurt am Main, Germany in manipulation robotics [START_REF] Ghadirzadeh | Deep predictive policy training using reinforcement learning[END_REF], [START_REF] Chebotar | Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning[END_REF], because they help to discriminate values of close states. For instance, [START_REF] Ghadirzadeh | Deep predictive policy training using reinforcement learning[END_REF], [START_REF] Chebotar | Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning[END_REF], [START_REF] Deisenroth | Learning to Control a Low-Cost Manipulator using Data-Efficient Reinforcement Learning[END_REF], [START_REF] Gu | Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates[END_REF] and [START_REF] Tsurumine | Deep dynamic policy programming for robot control with raw images[END_REF] use a distance measure between the current pose and a target pose to design an informative reward in manipulation tasks such as block stacking, reaching and door pushing or pulling. These rewards require knowledge of robot kinematics and target pose. Similarly, for tasks such as placing wooden rings or screwing bottle caps onto bottles [START_REF] Levine | End-to-end Training of Deep Visuomotor Policies[END_REF] and [START_REF] Levine | Learning contact-rich manipulation skills with guided policy search[END_REF] use an informative reward based on the distance between actual positions of end-effector or target object and their target positions. Tracking these points requires knowledge of the kinematics or non-trivial vision modules. For the same kind of task, [START_REF] Finn | Deep spatial autoencoders for visuomotor learning[END_REF] designs a reward based on a distance between visual features produced by an autoencoder and target features. This requires to place the robot in a target position and extract the target visual features each time the target location changes.

In contrast to these previous works, the aim of this paper is a robot that learns an object palm-touching task with only weak human intervention or supervision i.e. without kinematic models, hand-crafted features, or calibration parameters. More precisely, the task consists of reaching with the end-effector palm an object put on a table. This behavior can be seen as a natural predecessor to grasping. To learn this task with little supervision, we propose a stage-wise learning strategy inspired by infant development [START_REF] Hoffmann | Learning visuomotor transformations for gaze-control and grasping[END_REF], [START_REF] Carey | Magnetic Misreaching[END_REF], [START_REF] Fischer | A Theory of Cognitive Development: The Control and Construction of Hierarchies of Skills[END_REF]. The central idea is that an informative reward is constructed from knowledge that has been acquired during the learning of simpler predecessor tasks. Specifically, our robot first learns to fixate an object or its own end-effector using deep RL. Based on the knowledge acquired during these tasks an informative reward function is constructed that allows the robot to efficiently learn the object palm-touching task. Our results demonstrate that learning this task based on the learned informative reward signal works about as well as learning with a fully supervised reward, while naively learning from only a sparse reward signal fails completely in the same setting.

II. RELATED WORK

Our approach resembles developmental robotics methods [START_REF] Hoffmann | Learning visuomotor transformations for gaze-control and grasping[END_REF], [START_REF] Nori | Autonomous learning of 3D reaching in a humanoid robot[END_REF], [START_REF] Law | From Saccades to Grasping: A Model of Coordinated Reaching Through Simulated Development on a Humanoid Robot[END_REF], [START_REF] Chinellato | Implicit Sensorimotor Mapping of the Peripersonal Space by Gazing and Reaching[END_REF] which learn to reach using a hand-eye coordination function and object fixation. However, they are usually paired with supervision for the object and/or endeffector fixations [START_REF] Nori | Autonomous learning of 3D reaching in a humanoid robot[END_REF], [START_REF] Law | From Saccades to Grasping: A Model of Coordinated Reaching Through Simulated Development on a Humanoid Robot[END_REF], [START_REF] Chinellato | Implicit Sensorimotor Mapping of the Peripersonal Space by Gazing and Reaching[END_REF] or computation of action primitives [START_REF] Hoffmann | Learning visuomotor transformations for gaze-control and grasping[END_REF].

Our work is also related to methods which use sparseonly rewards in the sense that our sparse term fully defines the task. Though hardly using any supervision, sparse-only rewards generally require some strategies to increase the probability of getting a sparse reward. This can be done by increasing the number of agents learning the task [START_REF] Levine | Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection[END_REF] or by simplifying the action space using hand-crafted preprocessing [START_REF] Boularias | Learning to Manipulate Unknown Objects in Clutter by Reinforcement[END_REF]. Besides, if the agent has access to one of its goal states, a mechanism of learning from easy missions [START_REF] Asada | Purposive Behavior Acquisition for a Real Robot by Vision-Based Reinforcement Learning[END_REF] can be implemented to learn very precise robotic manipulation tasks [START_REF] Florensa | Reverse Curriculum Generation for Reinforcement Learning[END_REF]. We propose to increase the touching probabilities by building a weakly-supervised reward shaping term from learned skills without using action space simplification, several agents or the indication of a goal state.

III. METHODS

A. Background

An RL algorithm is usually based on Markov decision processes < S, A, R, T > where S is the set of states, A the set of actions, T the transition model (T : S × A → S) and R the reward function (R :

S × A × S → R).
The goal of the agent is to maximize the sum of discounted future rewards: J = ∞ k=0 r k γ k , where γ ∈ [0, 1] is a discount factor and r k the reward value at step k.

To optimize the criterion, we train a deterministic policy π : S → A. In the paper, we also estimate the state-action value function:

Q π (s, a) = E π ∞ k=0 r k γ k s, a , (s, a) ∈ S × A. (1)
Both the policy and the Q function are approximated by a neural network. Specifically, we use the DDPG algorithm [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] which combines the deterministic policy gradient algorithm [START_REF] Silver | Deterministic Policy Gradient Algorithms[END_REF] and the DQN [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. DDPG is an "actor-critic" algorithm updating the critic Q φ with parameters φ and the deterministic policy π θ with parameters θ as follows. At each time-step, we choose N b transitions from a large memory buffer using a uniform distribution:

< s i , a i , r i , s i > i∈{1,...,N b } ∈ S × A × R × S.
The targets of the Q φ neural network are computed using a TD(0) update (with a learning rate equal to 1):

∀i ∈ {1, ..., N b }, y i = r i + γQ φ s i , π θ (s i ) . (2)
φ and θ are the parameters of the target networks updated using a rate parameter τ (t denotes a time-step):

φ t+1 = τ φ t + (1 -τ )φ t , θ t+1 = τ θ t + (1 -τ )θ t , (3)
The Q φ network updates its weights by minimizing the squared error

1 2N b N b i=1 (y i -Q φ (s i , a i )) 2 .
Using the Q φ network and the fact that the policy is deterministic, the following policy gradient is derived:

∂Q φ ∂θ 1 N b N b i=1 ∂Q φ (s i , π θ (s i )) ∂a ∂π θ (s i ) ∂θ . (4)
This update makes the policy select the actions that maximize the Q function at the batch states. In addition to this algorithm, we use the inverting gradient procedure of [START_REF] Hausknecht | Deep Reinforcement Learning in Parameterized Action Space[END_REF] to bound the actions. This method downscales the gradient when the action computed by the policy approaches its limit. When it exceeds its limit, the gradient is inverted. This mechanism prevents the actions from becoming too large.

B. Overview

We describe here the stage-wise learning process (see Figures 1 and2 for a schematic view). The task consists of touching the object with the palm. In the following, we use the notations:

• I = (I left , I right ) represents the images from the left and right cameras. • q = (q camera , q robot ) represents the 3 camera joint angles (one common tilt angle and two independent pan angles) and 7 joint arm joint angles. Our stage-wise learning framework involves three successive tasks:

First, the robot learns from raw pixels to fixate the object with a two-camera system. For this, we use [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF] to learn to fixate an object with weak supervision. At the end of the fixation, the camera system coordinates q camera fix implicitly encode the object position in 3D space.

Second, the robot learns a hand-eye coordination function f η which maps robot joint coordinates to virtual camera coordinates:

q camera virt = f η (q robot ). (5)
These virtual camera coordinates correspond to the camera coordinates which would make the camera system look at the end-effector. Finally, a reward signal using q camera fix and q camera virt to make the end-effector close to the object is computed and combined with a sparse reward, indicating if there is palm-touching or not. In the following, we describe each of the three steps.

C. Learning binocular object fixations

Fixating an object means bringing the object at the center of I left and I right by moving the cameras. To learn it, we build on our prior work [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF].

The task is learned with the DDPG algorithm [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] using I and q camera as states, and ∆q camera as actions. The reward function is the sum of left and right camera components: r obj = r left obj + r right obj . For each camera c = left or right, the reward function is an affine decreasing function of the distance between the image center P i and the estimated object position P c o (computed below):

r c obj = 2 1 2 d max -||P i -P c o || 2 d max ∈ [-1, 1], (6)
with d max being the maximal distance between the image center and the object pixellic position. An episodic set-up is used. For each episode, a random object is put at a random location above the table. The episode ends when a given number of transitions has been reached.

The object detection mechanism involves a convolutional autoencoder training step in which images of the environment without object are encoded. Then, when an object appears in the environment, it is detected as an anomaly and localized in the images I left and I right . More precisely, we apply a kernel density estimator on the autoencoder reconstruction error images respectively |I left -Îleft | and |I right -Îright |. After that, the estimated object pixellic positions, respectively P left o and P right o are at the maximal probabilities as described in [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF]. This object detection principle only requires an autoencoder pre-training step without object and the assumption that there is an object in the scene subsequently.

D. Learning a hand-eye coordination function f η

To build f η , we need to have a database D of input-output pairs (q robot , q camera ) where q camera makes the camera look at the end-effector. To produce such samples, we learn how to fixate the end-effector the same way [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF] learns to fixate an object. We use the DDPG algorithm and a reward requiring weak supervision. The Markov Decision Process is the same as for the object fixation with the exception of the reward function. The latter involves the end-effector detection P c e (computed below) instead of P c o :

r c eff = 2 1 2 d max -||P i -P c e || 2 d max ∈ [-1, 1]. (7)
The set-up is also episodic. For each episode, random robot joint coordinates are generated using uniform distribution with fixed limits. They are empirically set to provide a large variety of reachable arm configurations.

During learning, training pairs (q robot , q camera ) are added to D when the reward is above a fixed threshold. When the number of samples in D is higher than the batch size N bf , we train f η on random batches of D as soon as a new sample is added to D.

We describe here how we detect the end-effector in the image. Unlike [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF] which uses an autoencoder to detect the object as an anomaly, the end-effector image position is computed using the difference in the image before and after pre-defined end-effector finger moves. This end-effector detection method inspired by [START_REF] Metta | Early Integration of Vision and Manipulation[END_REF] only requires to specify finger moves. Figure 3 presents the different steps of the end-effector detection:

• The image before the end-effector I c b is saved. • The end-effector fingers make pre-defined moves and the image I c a is saved • The difference of images is calculated and the endeffector position P c E is computed using a kernel density estimator the same way P c O is calculated from the autoencoder reconstruction error image.

E. Learning to touch

The goal of the robot is to touch with the end-effector palm the object above the table from a large set of initial arm positions.

It is defined by a sparse reward term r sparse which indicates if there is palm-touching or not:

r sparse = 1, if palm-touching, 0, otherwise. (8)
As the object is kept at the same position, the state space S is composed of the arm and camera coordinates, q. Indeed, for this task, we do not need to add images in the state space since the right arm movements can be decided from the robot joint angles only. The actions consist of the variations of the robot joint angles: a = ∆q robot which are seven real-valued scalars.

The object binocular fixation task and the hand-eye coordination function are used to compute the touching reward function.

After the execution of an object fixation step (using the object fixation policy π ψ ), we get the fixation camera angles q camera fix which localize the object. After that, using equation 5 at each time-step, the hand-eye coordination function f η gives us q camera virt which localizes the end-effector. Then, a reward shaping term r shaping can be computed:

r shaping = c cam ||q camera fix -q camera virt || 2 , (9)
with c cam < 0. r shaping represents an informative term which depends on the distance between the virtual camera coordinates and the camera coordinates which make the camera system fixate the object. It can be seen as an openloop term which constrains the 3 translational degrees of freedom of the end-effector. It takes negative values to ensure quick trajectories are learned. Furthermore, the slope c cam is chosen to ensure shaping rewards are small compared with the non-zero sparse reward. Finally, the reward function is the sum of the two terms:

r cam = r shaping + r sparse . (10)
IV. EXPERIMENTS

In this section, we describe the experimental protocol. The experimental objective is two-fold. First, we want to evaluate performances of our weakly-supervised reward in a palmtouching task. Second, we wish to compare our reward with an informative reward requiring external information.

A. Experiment environment

The experiments are carried out using the Gazebo simulator jointly with the ROS [START_REF] Quigley | ROS: an open-source Robot Operating System[END_REF] middleware. A two-camera system is mounted on a bi-arm robotic platform (see Figure 1). Only one arm is used for the experiments and a Barrett hand is attached to this arm. A table from the Gazebo object database is placed below the cameras and in front of the biarm platform. Note that this table is not present when we learn the hand-eye coordination function. Our method does not depend on a specific object model because the robot learned to look at any object in [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF]. In our experiments, we use a blue ball.

B. Different reward functions

In this paragraph, we present the reward functions which will be used in our experiments:

• r cam = r shaping + r sparse . This is the proposed reward function (described in Section III-E).

• r sparsePen = 1, if success, -0.0125, otherwise. This rewards the robot only when the palm touches the object. Besides, it penalizes each unsuccessful movement to encourage the robot to quickly touch the object.

• r cart = c cart ||X -X target || 2 +r sparse , with c cart < 0. To build this reward, we give a 3-dimensional end-effector target pose X target for the shaping part and we add a sparse reward. This reward is the closest to the proposed r cam but its shaping term requires forward kinematics and 3D object pose information. Finally, the slope c cart is chosen to make the shaping term take about the same values as r shaping .

We choose not to compare our reward function with one which would constrain both the end-effector position and orientation. Indeed, for such a reward function, a success would be to touch with the palm from a specific orientation and position. In our case, a success can be to touch with the palm from any position. The tasks are then too different to be compared in terms of touching improvement.

C. Experimental protocol

We describe how we compare the reward functions for the touching task. The protocol contains a training and a test phase.

1) Training phase:

For training, we use the DDPG algorithm [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] and the previously defined reward functions. Learning happens on N tot fixed-length episodes of size N eps . Each episode has an initial arm position and an object position. The object position is kept fixed. For the initial arm position, we use two different settings. The first (P 1) one uses a single initial arm position for learning as in [START_REF] Tsurumine | Deep dynamic policy programming for robot control with raw images[END_REF]. This position is made well oriented for a palm-touching purpose. This means that the robot has to move its end-effector mainly in translation to touch the object. The second (P 2) uses random robot joint angles generated from uniform distributions. The goal of this setting is to check if we can learn to touch the object with our reward signal including initial positions with a badly oriented end-effector. In such a position, the robot has to substantially modify its end-effector orientation to touch the object. Furthermore, it is known that learning from different initial states allows to learn policies that are more robust to perturbations [START_REF] Rajeswaran | Towards Generalization and Simplicity in Continuous Control[END_REF].

We use constant zero-mean Gaussian noises trans and rot for the exploration. The variances of the noise are higher for joints which strongly influence the end-effector orientation ( rot ). Indeed, more exploration is required for wrist angles because "touching" orientations are only rewarded by the sparse reward.

To accelerate learning, we handle the times when the robot reaches "unsuccessful" contact areas where its movement is blocked. Indeed, in the training phase, the robot often touches the object without a palm contact though it is close to being successful. To make exploration easier in these areas, we record contact-less positions through a short-sized circular buffer. In case of contact, we pick the oldest position in the buffer, compute the action to go to this position, and apply it. This allows to more correctly discriminate actions in the contact areas.

Through the training experiments, we wish to compare our reward requiring weak supervision with other ones. Consequently, for all the reward signals, in order to monitor the learning progress, we specifically plot the touching frequency f type t over the episodes, type referring to a specific reward function. We average three experiments per setting with the exception of the sparse reward setting. The latter is done once.

2) Test phase: The goal of test experiments is to evaluate final reaching performances. We test the learned policies without any exploration noise on N tot random episodes. At the end, we compute the touching frequency f type test . Note that we do not apply the trick used in the training phase to deal with blocked situations. Instead, when the robot is blocked, it just computes its next action with the policy. Like in the training phase, the results are averaged over three experiments per setting.

D. Implementation details

For all the neural network algorithms, we use the caffe library [START_REF] Jia | Caffe: Convolutional Architecture for Fast Feature Embedding[END_REF]. A GPU (nvidia GeForce GTX Titan X) is used for the experiments.

We use the same neural network structures as in [START_REF] De La Bourdonnaye | Learning of binocular fixations using anomaly detection with deep reinforcement learning[END_REF] for the end-effector and object fixation tasks. The hyperparameter values are also the same with the exception of the number of iterations: 200000. The hand-eye coordination function is a neural network with 2 fully connected hidden layers of 10 and 5 neurons and a batch size N bf of 32 is used to learn it. For the episode initialization of the end-effector fixation task, the seven arm joint angle distribution amplitudes are 11, 46, 69, 92, 92, 86 and 0 • if we consider the ascending order in the kinematic chain i.e. from the base link to the end-effector.

For the touching task, the Q network has 3 fully connected layers with 250, 200 and 1 neural units. The policy network involves 3 fully connected layers with 200, 150 and 7 neural units. The weights are updated using the Adam [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] solver. Table I and II provide values for the parameters used in the experiments. For the Q update, the discount factor γ is equal to 0.99. For the episode initialization of the P 2 problem, the distribution limits are the same as in the handeye coordination learning with the exception of the last joint (end-effector spin angle) for which the amplitude is 11

V. RESULTS AND DISCUSSION

We present here the results in the two previously defined settings which are used in the experiments.

A. P 1

Figure 4 shows the touching rates. First, we observe that f sparsePen t is zero almost everywhere. In the experiment, it touches one time and is never able to further improve. This is because transitions with high TD values are not trained enough through DDPG. Second, we have similarly good performance of over 90% for the proposed reward r cam and the reward function r cart , which requires forward kinematics and 3D object pose information. For this task, this shows that a shaping term which requires forward kinematics and precise 3D target information can be replaced without loss of performance by a weakly-supervised shaping reward resulting from learning simpler predecessor tasks. The reason is that the objective of the task is defined by the terminal sparse reward. The shaping term is used only to get closer to good touching postures. When the terminal rewards are back-propagated through the Q function to all the trajectory states, the shaping term becomes small compared to the Q values and therefore less important. The results from policy exploitation show that we reach the object with 100 % of successes for both policies for 300 test experiments. This shows that the residual errors at the end of learning were only due to the exploration noise applied to the policy.

B. P 2

Figure 5 shows the results of training experiments for the P 2 problem. We still observe that, with our setting, sparse rewards do not allow to efficiently learn a touching policy. Furthermore, we observe again that our weaklysupervised reward and its supervised counterpart give similar performances. It means that with very different initial positions including those with difficult end-effector orientations, the robot can learn to correctly orientate its end-effector given a long-delayed sparse reward. In the test phase, we Fig. 5. Learning curves for the P 2 problem observe 94,9 % and 95.4 % of success for the policy learned with, respectively, r cam and r cart , which represent good and comparable results. This shows that our reward signal with very little supervision allows to learn a complex touching task from different initial positions.

VI. CONCLUSION

For a complex palm-touching task, we have designed a stage-wise learning procedure which does not require any kinematic models, hand-crafted features, calibration parameters and needs minimal human supervision. Inspired by developmental robotics, our approach involves three steps. First, a binocular vision system learns to fixate an object. This implicitly encodes the position of the object in the 3D camera angle space, while requiring only an autoencoder pretraining step. Second, by learning to fixate the end-effector from its finger moves, the robot learns a hand-eye coordination function which maps robot joint angles to camera angles. In a nutshell, from a given arm configuration, this function allows to localize the end-effector in the 3D camera angle space. Third, we compute the reward signal for the palmtouching task by summing an informative shaping reward computed with the previous steps and a sparse terminal reward indicating the true objective of the task. This shaping term encourages the end-effector to be close to the object in the 3D camera angle space.

The experiments show that our reward signal allows to learn complex touching policies, notably from various initial positions. In addition, policies learned with our reward and with a supervised counterpart reward give similar performances. This shows that for this complex task, our reward based on weak supervision can replace a reward based on forward kinematics and object tracking.

One advantage of our framework is that it is suitable for other types of manipulation tasks with different terminal rewards such as object grasping or pushing. Indeed, our shaping term is used to bring the robot closer to terminal rewards. Thus, switching to another task just requires to change the terminal sparse reward.

Even though our reward computation framework is based on weak supervision, it still has a limitation regarding autonomy. Indeed, even if the object fixation method hardly requires external information, it cannot cope with environment changes in an online way. If the environment varies, the autoencoder has to be trained again. In future work, we wish to implement an object detection method which would adapt to environment variations. Finally, we wish to learn the same task in a real world setting.

Fig. 1 .

 1 Fig. 1. Palm-touching learning process: (a) Object fixation, (b) End-effector fixation and hand-eye coordination, (c) Palm-touching

Fig. 2 .

 2 Fig. 2. Overall scheme of the touching task learning procedure

Fig. 3 .

 3 Fig. 3. End-effector computation scheme

Fig. 4 .

 4 Fig. 4. Learning curves for the P 1 problem