Don Joven Agravante

Andrea Cherubini

Antoine Bussy

Pierre Gergondet

Abderrahmane Kheddar

Collaborative Human-Humanoid Carrying Using Vision and Haptic Sensing

Keywords:

We propose a framework for combining vision and haptic information in human-robot joint actions. It consists of a hybrid controller that uses both visual servoing and impedance controllers. This can be applied to tasks that cannot be done with vision or haptic information alone. In this framework, the state of the task can be obtained from visual information while haptic information is crucial for safe physical interaction with the human partner. The approach is validated on the task of jointly carrying a flat surface (e.g. a table) and then preventing an object (e.g. a ball) on top from falling off. The results show that this task can be successfully achieved. Furthermore, the framework presented allows for a more collaborative setup, by imparting task knowledge to the robot as opposed to a passive follower.

I. INTRODUCTION

Humanoid robots provide many advantages when working together with humans to perform various tasks. This is because humans have an extensive experience in physically collaborating with each other. Hence, humanoids can interact with humans because of their human-like range of motion and sensing capabilities. This reduces the need to learn how to interact with the robot. However, many challenges are still present in the various research areas that study physical human-robot collaboration. Here, the area of interest is using vision and force information together to enable human-robot joint actions, which are collaborative tasks requiring both parties to physically interact with each other (e.g. carrying a large object together). In such tasks, the robot:

1) must move safely and regulate interaction forces, 2) shares control with a human-in-the-loop, 3) can only use its on-board sensors. The first two items are the main aspect of all physical humanrobot collaborative tasks. The last constraint is important for true autonomy. For example if vision has a limited field-ofview, external room cameras should not be used.

Physical human-robot collaboration has largely relied on the use of haptic data (force/torque) for control. This is because the main priority is the regulation of the interaction forces between the human and robot. For example, previous works [START_REF] Yokoyama | Cooperative works by a human and a humanoid robot[END_REF]- [START_REF] Bussy | Proactive behavior of a humanoid robot in a haptic transportation task with a human partner[END_REF] have demonstrated that using only haptic 1

CNRS-UM2

LIRMM UMR 5506, Interactive Digital Human group, 161 Rue Ada, 34392 Montpellier, France {firstnames.lastname}@lirmm.fr.

2 CNRS-AIST, JRL (Joint Robotics Laboratory), UMI 3218/CRT, Intelligent Systems Research Institute, AIST Central 2, Umezono 1-1-1, Tsukuba, 305-8568, Japan. information, a humanoid robot can help a human carry large objects (e.g. a table, a beam or a panel). A possible future application of this is in construction sites [START_REF] Yokoyama | Cooperative works by a human and a humanoid robot[END_REF]. The same scenario can also be applied to the household, such as moving furniture (e.g. table ). While doing this task, one can imagine the need to prevent an object on top from falling off. For example in moving a table a short distance, it might be necessary to move it carefully with the objects on top rather than removing the objects, transporting the table and then placing the objects back on top. In this scenario, haptic information alone is not rich enough to give the robot knowledge about the state of the objects on top of the table. But vision can provide such information, being largely complementary to haptics (analogous to human sight and touch). Using both information sources may enable a humanoid to perform more complicated tasks, similar to a human. Although the benefits are great, there are not many established methods integrating vision and force control.

In [START_REF] Nelson | Improved force control through visual servoing[END_REF], three general categories for combining vision and force control are identified: traded, hybrid and shared. Traded control is the simplest, and switches between a pure visual servoing controller and a pure force control method given a certain threshold of the task error. In hybrid methods, a prior specification of a "task-frame" [START_REF] Mason | Compliance and force control for computer controlled manipulators[END_REF], [START_REF] Baeten | Integrated vision/force robotic servoing in the task frame formalism[END_REF] is required to decouple vision and force into orthogonal spaces. With this, the controllers can be designed separately. Finally, shared control methods aim at utilizing both vision and force information together in the same space, such that all available information is used [START_REF] Nelson | Improved force control through visual servoing[END_REF]. For example, in [START_REF] Prats | Vision force control in task-oriented grasping and manipulation[END_REF], a force feedback is used to correct the visual servo control trajectory.

In this paper, the impedance control framework [START_REF] Hogan | Impedance control -An approach to manipulation. I -Theory. II -Implementation. III -Applications[END_REF], is used. This allows a manipulator to be compliant by defining a virtual impedance. In this framework, vision can be used to provide a reference trajectory that is tracked in the absence of external forces [START_REF] Santis | Human-robot interaction control using force and vision[END_REF]- [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF]. When contact does occur, it has the properties of shared control methods where vision and force determine the control of the same degree of freedom (DOF) simultaneously. This approach is preferred over the others since it can allow for compliance in all DOF. Previous works using this methodology [START_REF] Santis | Human-robot interaction control using force and vision[END_REF], [START_REF] Morel | Impedance based combination of visual and force control[END_REF] presented experiments of a robot interacting with objects. Here and in our previous work [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF], we use this approach for physical human-humanoid collaboration experiments. Having a human as a physical collaborator means that some issues of this framework need to be revisited: the implication of impedance parameters and how vision and haptics are combined in the context of physical collaboration.

Our work is partly based on the joint object transportation framework which was introduced in [START_REF] Bussy | Human-humanoid haptic joint object transportation case study[END_REF] and utilized in [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF]. The overall system is hybrid -using the 3 DOF that are controlled with only haptic information in [START_REF] Bussy | Human-humanoid haptic joint object transportation case study[END_REF] and designing shared controllers for the remaining DOF. These new controllers use both vision and force to affect the DOF at the same time. This complicates the semantics to describe the whole system. In [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF], vision was used together with haptic data to stabilize the height of the table, providing the reference trajectory of the impedance controller in 1DOF. As a continuation to this work, we use the same general framework of building a visual servoing controller for providing a reference trajectory to the impedance controller. However, we relax the constraint on having a static object in [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF]: the object on the beam is free to move so control in 2 DOF is needed. This is a more difficult task and vision is indispensable for acquiring the state of the task. Another contribution of this work is the analysis of collaboration in joint actions when each agent has his/her own notion of how to do the joint task, as opposed to the common leaderfollower strategy.

To continue, the general framework is described in Section II. The example task of human-humanoid table carrying while keeping an object on top from falling is then described in Section III. The details on implementing the general framework to this specific task are given in Section IV. Results from experiments are then presented in Section V. The novelty of equal collaboration and the challenges it poses are discussed in Section VI. Finally, Section VII concludes and outlines some future works to be done.

II. GENERAL CONTROL FRAMEWORK

Our general approach to combining vision and haptic cues is coupling a visual servoing controller to an impedance controller. This simplifies the design by decoupling the vision and force controllers in a systematic way. An overview of the complete control framework is shown in Fig. 1. The following subsections explain this general framework in a bottom-up approach starting from the lower level controllers and abstracting it higher to the cognitive level. The lowest level of control is the inner joint-level control. This is represented by q in Fig. 1. To abstract from the joint level to the "task level", the Stack-of-Tasks framework is used [START_REF] Mansard | A versatile generalized inverted kinematics implementation for collaborative working humanoid robots: The stack of tasks[END_REF]. It is a generalized inverse kinematics abstraction layer that creates a hierarchical organization of different tasks to be executed giving higher priority to critical tasks [START_REF] Mansard | A versatile generalized inverted kinematics implementation for collaborative working humanoid robots: The stack of tasks[END_REF]. It allows for easier integration with sub-tasks. For example, our experiments make use of the walking algorithm in [START_REF] Herdt | Walking without thinking about it[END_REF] as a sub-task.

A. Impedance Control

The other sub-task concerns the grippers. In Fig. 1 the humanoid uses its grippers to co-manipulate an object with a human. To do this, it needs to be safe and intuitive to use. Here, impedance control [START_REF] Hogan | Impedance control -An approach to manipulation. I -Theory. II -Implementation. III -Applications[END_REF] is used to regulate the contact interaction (for safety) between the robot and its environment. It is based on a simple physical analogy to the control -a virtual mass-spring-damper system [START_REF] Hogan | Impedance control -An approach to manipulation. I -Theory. II -Implementation. III -Applications[END_REF]. This system is governed by the general equation:

f = M(ẍ d -ẍ) + B( ẋd -ẋ) + K(x d -x).
(

) 1
The contact interaction is measured by the force-torque sensors in the robot grippers and is represented as f . The vectors x d , ẋd and ẍd are a desired pose and its first and second derivative. Correspondingly, vectors x, ẋ and ẍ represent an actual pose and its first and second derivative. Finally, matrices M, B and K are the inertia, damping and stiffness parameters that define the desired virtual massspring-damper system [START_REF] Hogan | Impedance control -An approach to manipulation. I -Theory. II -Implementation. III -Applications[END_REF]. Strictly following the terminology and causality from [START_REF] Hogan | Impedance control -An approach to manipulation. I -Theory. II -Implementation. III -Applications[END_REF], our implementation on the HRP-2 humanoid, is an "admittance controller" since the robot is position-controlled by the Stack-of-Tasks, which uses the output of x, ẋ and ẍ from the impedance controller. These are obtained by solving the differential equation of Eq. ( 1) given the other variables. The parameters M, B, and K are determined empirically to provide comfort for the human collaborator. Finally, x d , ẋd and ẍd are the desired pose and trajectory of the mass-spring-damper's reference position. These are detailed in the next subsection.

B. Proactive Behavior and Visual Servoing

For the general impedance controller of Eq. (1) a "passive" behavior is defined by setting the desired pose x d as constant. This case is illustrated in Fig. 2(a) where only the human knows about the task to be done. This is the "classical" case in human-robot collaboration. In such a case (and considering constant impedance parameters M, B, K), the robot's motion (x, ẋ, ẍ) can only be initiated by an external force f due to Eq. ( 1). Recent research aims to make the robot a proactive follower to make the system more comfortable for the human. A way to achieve this is by creating a suitable desired pose and trajectory (x d , ẋd , ẍd ) such that the human effort is minimized [START_REF] Bussy | Human-humanoid haptic joint object transportation case study[END_REF], [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF], [START_REF] Maeda | Human-robot cooperative manipulation with motion estimation[END_REF]. These works differ in the approach taken to produce the desired pose and trajectory. In [START_REF] Maeda | Human-robot cooperative manipulation with motion estimation[END_REF], human motion is predicted by a minimum jerk model to give the desired pose. In [START_REF] Bussy | Human-humanoid haptic joint object transportation case study[END_REF], a human pair doing a joint transportation task was studied and it was observed from the data that the pair moves in constant velocity phases during this task. A finite state machine (FSM) is then designed by using the constant velocity assumption, giving the desired pose and trajectory. Haptic cues are used to determine the switching of states in the FSM [START_REF] Bussy | Human-humanoid haptic joint object transportation case study[END_REF]. Our latest work [START_REF] Agravante | Humanhumanoid joint haptic table carrying task with height stabilization using vision[END_REF] takes the same approach as the one in this paper and is illustrated by Fig. 2(b). Here, the humanoid is given knowledge of the task. This is done by designing a visual servoing controller specific to the task and using the output as the desired trajectory (x d , ẋd , ẍd ) of the impedance controller. This also means that the robot has some autonomy in doing the task driven by its own knowledge of the state of the task. With the reasonable assumption that during the collaborative task human motion is task-driven, the source (human intention to do the task) is taken into account rather than the result (human motion). This differentiates our approach from those that aim to model/predict human motion such as [START_REF] Maeda | Human-robot cooperative manipulation with motion estimation[END_REF]. Visual servoing consists in controlling robots using visual information [START_REF] Chaumette | Visual servo control. i. basic approaches[END_REF]. To create the visual servoing portion of the framework, two important components are needed: visual feature tracking and a controller based on this feature [START_REF] Chaumette | Visual servo control. i. basic approaches[END_REF]. However, in the current state-of-the-art for both modules there is no "best" approach that fits all tasks and problems. Existing methods have important tradeoffs to consider for the whole system [START_REF] Chaumette | Visual servo control. i. basic approaches[END_REF]. In our works, we take an analytical approach to building the visual servoing portion.

III. TASK DESCRIPTION

As a test for the general framework described, the task of jointly transporting a surface while keeping a mobile object on top from falling off is used. Fig. 3 illustrates the task with the reference frames and naming convention used in the rest of this paper. The vectors composing the Cartesian frames are color coded: Red-Green-Blue correspond to ( x, y, z) respectively.

Fig. 3 shows that the robot can control the table through its hands {rh} and {lh}. The control design consists in driving a reference "control frame" {cf }, rigidly linked to the table, to a desired pose with respect to a local frame {l}, rigidly linked to the robot torso. This pose is represented by the homogeneous transformation matrix l T cf . To achieve this, the hand poses {rh} and {lh} are controlled in the local frame according to:

l T h = l T cf cf T h h = {rh, lh}.
Assuming a rigid grasp of the table, the homogeneous transformation matrices cf T rh and cf T lh are constant and known once {cf } has been defined. For the implementation of the impedance controller, l T cf is converted into the 6dimensional vector x = l [x, y, z, φ x , φ y , φ z ] ⊤ cf made up of the Cartesian coordinates and Euler angles (the ZYX convention is used which conveniently places the singularity at φ y = ±90 • , an impossible case of the joint transportation task).

An intuitive description of the task is to "keep the object on the table from falling off". The control design can be defined to attract the object (o) towards an appropriate desired goal point (d) (refer to Fig. 3). To realize this task, direct force application on the object is not possible, since the priority is table transportation. Hence, only an indirect action can be applied by tilting the table to contrast or exploit gravity. To do this, a decoupled approach can be used, where the control of t y o is done through φ x , and that of t x o through z. It is chosen to regulate t x o with z and not with φ y because the latter option would make the task uncomfortable for the human. In fact, controlling φ y forces the human to actively move his/her z position. Instead, by controlling z and leaving φ y compliant, the human at the other end just needs to be compliant in his/her φ y , which is more comfortable.

To integrate the whole system, the important part is defining the admittance controller's desired trajectory (x d , ẋd , ẍd ) for all 6 DOF. The vision-based control takes care of two DOF (z, φ x ). Three DOF (x, y, φ z ) of the pose are defined from the FSM of our group's earlier work [START_REF] Bussy | Human-humanoid haptic joint object transportation case study[END_REF], [START_REF] Bussy | Proactive behavior of a humanoid robot in a haptic transportation task with a human partner[END_REF] (described briefly in Section II-B). Finally, the remaining DOF (φ y ) is made compliant by setting φ y,d = 0.

IV. IMPLEMENTATION DETAILS

As explained in Section II-B, the approach is to design a visual servoing controller to make the robot proactive throughout the task. Two main components are needed: visual tracking and the control design.

A. Vision Algorithm

In the HRP-2 robot, RGB-D (color + depth) data is obtained from an embedded ASUS Xtion device located in the head. Fig. 4 shows typical data of the task. The aim of the vision algorithm is to process this raw data into visual features that can be used for control. An error signal can be defined by t x o -t x d and t y o -t y d . For the example task here, z is irrelevant, since t z d ≡ t z o . Since the desired location t (x, y) d is arbitrarily defined, the vision algorithm only needs to obtain t (x, y) o . A variety of vision algorithms that can do this may be used, with speed as another consideration. For example, given the object model and the table model, it is possible to use a model based tracker. Designing a novel vision algorithm is not the focus of this work, so we use well-known methods [START_REF] Gonzalez | Digital image processing using MATLAB[END_REF]- [START_REF] Bradski | Learning OpenCV: Computer vision with the OpenCV library[END_REF]. Nevertheless, the methods used here are briefly described for completeness. The features used here are the centroids of the object and of the table. The first step is to segment these from the image. Color segmentation is used in our system. For example the pink object in Fig. 4 and yellow object in Fig. 3 can be easily characterized and thresholded by a specific hue range and a high saturation (from the HSV color space). To add robustness, morphological operations (opening and closing) are used to remove outliers. After this, sliding window detection (sped up using the image pyramids concept) finds the most probable location. The centroid of the detected blob is (u, v) in pixel coordinates. This is then converted into c x o and c y o by using the intrinsic camera calibration parameters (f x , f y , c x , c y ) and the depth c z o in the following equations:

c x o = c zo(u-cx) fx , c y o = c zo(v-cy) fy . ( 2
)
The next step is to segment the table in the image. A flood fill algorithm [START_REF] Bradski | Learning OpenCV: Computer vision with the OpenCV library[END_REF] is run in saturation-value-depth space. This algorithm starts with a "seed" point and grows the region based on a connectivity criterion between neighboring pixels.

Here, the seed point is the bottom pixel of the ball. A low saturation and high value characterize well the "white" color of the table. The addition of depth ensures connectivity in Cartesian space, simplifying for example the segmentation between table and floor pixels. Finally, some morphological operations (opening and closing) are done to remove outliers. From these segmented points, the Cartesian centroid is used as c t t (a translation vector). The Cartesian coordinates of the object in the table frame are then obtained by:

t t o = c T -1 t c t o . (3)
The homogeneous transformation matrix c T t is composed of the table centroid position c t t and the rotation matrix c R t . A simple approximation consists in setting c R t equal to c R cf , which is obtained from proprioception.

B. Vision-Based Control

The control design needs to drive t t o to t t d . Several existing methods can be used. Here, a simple PD controller is used such that:

C i (s) = K p,i + K d,i s i = {x, y} . (4)
This choice is justified by analyzing the task using a simple sliding model (i.e., neglecting friction and angular momentum). Fig. 5 illustrates the necessary variables for this analysis. Since a control with z rather than φ y is desired, the trigonometric identity z r = l t sin φ y is used, where l t is the length of the table and z r is the differential height. z r can be converted to z by a trivial change of frame. The Lagrangian equation of motion along t x is:

mẍ = mg sin φ y = mgz r /l t . (5)
Along y, linearization of the Lagrangian equation about φ x = 0 leads to:

mÿ = -mgφ x . (6)
Taking the Laplace transforms of these two equations yields:

s 2 X(s) = gZ r (s) /l t s 2 Y (s) = -gΦ (s) . (7)
Rearranging, the transfer functions describing the dynamics on the 2 DOF can be derived:

P x (s) = X(s) Zr(s) = g lts 2 P y (s) = Y(s) Φ(s) = -g s 2 . (8)
It should be noted that both are double integrators. As such, they are only marginally stable when feedback controlled with a Proportional gain. But the Proportional Derivative controller (PD) chosen can be used. The denominator of the closed loop system transfer function in the two cases is:

D x (s) = l t s 2 + gK d,x s + gK p,x D y (s) = s 2 -gK d,y s -gK p,y . (9)
The two systems are asymptotically stable if all the roots of these two polynomials have non-multiple negative real parts. This condition is verified, for a second order polynomial, if all the coefficients are strictly positive. In the case of the characteristic polynomials in [START_REF] Hogan | Impedance control -An approach to manipulation. I -Theory. II -Implementation. III -Applications[END_REF], this is equivalent to:

K p,x > 0 K d,x > 0 K p,y < 0 K d,y < 0. (10)
Finally, the applied controllers are:

z = K p,x (x d -x) -K d,x ẋ φ x = K p,y (y d -y) -K d,y ẏ. ( 11
)
By numerical differentiation ẋ (and ẏ) is obtained as:

ẋ (t) = x (t) -x (t -∆t) ∆t ,
with ∆t the sampling step. Tuning the gains in [START_REF] Morel | Impedance based combination of visual and force control[END_REF] according to [START_REF] Santis | Human-robot interaction control using force and vision[END_REF] guarantees stability of the closed loop system, as long as the linear approximation is valid. This implies that t t o will converge to t t d , as desired. The outputs of ( 11) are fed to the admittance controller (1) as desired values z d and φ x,d . Numerical differentiation is used to obtain ż, φx in ẋd . However, for ẍd a piece-wise constant velocity is assumed such that z = φx = 0. This also prevents too much noise introduced by a second numerical differentiation.

V. RESULTS

For the experiments, we chose a ball to be the moving object. This makes it similar to a well-studied problem/example in control theory: the "ball-and-plate" system, which is a 2-DOF generalization of the "textbook example" ball-onbeam system (used to study advanced control methods [START_REF] Hauser | Nonlinear control via approximate input-output linearization: The ball and beam example[END_REF]). Although similar, significant differences exist -notably that collaboration is the main issue here.

Several experiments were performed and with 2 different balls -a yellow tennis ball which tends to move slower and a pink ball which moves quite fast. A few different users also tested this early system, but as the described experience was similar this is not discussed here. Some experiments are shown in the accompanying video and in Fig. 6. The video also shows some results of the vision algorithm detecting the ball and the table. In the initial experiments, both human and humanoid stand stationary and balance the ball on the table. Some disturbance is then introduced (e.g. the ball is pushed by another person) and the gains of the PD controller are tuned according to [START_REF] Santis | Human-robot interaction control using force and vision[END_REF] in order to be able to handle such a disturbance. After "light" gain tuning of the visionbased controller with such experiments, we test the complete system where the human-humanoid dyad transport the table with the ball on top. Here, walking introduces a significant disturbance that can move the ball. The experiments show that although the ball moves a lot, it doesn't fall off the table during this transportation task.

From the recorded data of the force/torque sensors in the HRP-2 wrists, we found that during this task τ x (the total torque about the x-axis of {cf}) averages to about 0 N m, which means that this interaction torque with the human is regulated well. Furthermore, f z (again with {cf} as the reference) averages to about 12 N . This means that the robot carries part of the weight of the table and thus lightens the burden on the human. Finally, we notice that in both signals a noticeable oscillation occurs which correlates to the frequency of the walking gait and the disturbance that it causes.

VI. DISCUSION ON EQUAL COLLABORATION

The results show that the complete system (Fig. 2b) can do the job well: the vision-based controller tries to keep the ball on the table while the impedance controller regulates interaction forces. A simple analysis of Fig. 2 shows that a disadvantage of the pure follower (Fig. 2a) is that the success/failure of the vision task depends solely on the human partner. Specifically, the human needs to use his/her vision to observe the state of the task and then apply a sufficient force to haptically communicate to the robot what s/he wants to do. Instead, in Fig. 2b the cognitive load of the task is shared in some capacity -both human and robot are able to observe the state of the task and act accordingly. However, this sharing can become a disadvantage when the human and robot disagree on the state of the task and the action to take [START_REF] Kheddar | Human-robot haptic joint actions is an equal controlsharing approach possible?[END_REF]. Experimentally, this is handled in our system by making the robot more compliant and less stiff (impedance parameter tuning). This ensures that the human can always safely impose his/her intention through the haptic channel. This also shows a possible extension of the system which is to dynamically change the impedance parameters: making it more stiff when the robot is more certain of his observations and more compliant when there is more uncertainty. In effect, this makes the impedance parameters a method to weigh the importance between visual (task knowledge) and haptic (human intention) information channels. But, it is important to note that this disadvantage of equal collaboration also applies to human-human pairs and more generally in teams -"teamwork" (or the lack of it). Some preliminary experiments have been made with both the passive follower and the approach of equal collaboration and the advantages/disadvantages briefly described here can be observed by the human collaborator. One difficulty in presenting these results is in the use of proper evaluation methods since the most important aspect -the comfort of the human collaborator -is very subjective. Another difficulty is to separate the contribution of the human and robot. Although in the results presented here the human is told to be more "passive" (does not try "too much" to keep the ball on the table) he also does not try to make the ball fall off, since teamwork is a factor in the overall result. The resolution of these issues is left for future work.

VII. CONCLUSION AND FUTURE WORK

In this paper, a general framework for human-robot joint collaborative tasks was presented. It uses a visual servoing controller to realize the task and a haptic channel to recognize human intention. Both vision and force control are combined in the impedance control framework. This is implemented and tested on a joint transportation task where a human and humanoid robot carry a table with a freely moving ball on top. The objective is to transport the table while keeping the ball from falling off. This task is used to explore some important issues in robotics: the combination of vision and force information and the issues concerning collaborationsafety and effective human-robot collaboration strategies.

To continue the work here, it is planned to further investigate the combination of vision and force information. Another major area for continued study is in collaboration, such as the idea of dynamically changing the impedance parameters described in Section VI. Further works are to utilize good statistical methodology and experiments with different users (usability study) to better analyze the qualitative results.

VIII. ACKNOWLEDGEMENT

Fig. 1 .

 1 Fig. 1. The general control framework applied to the task of balancing an object on the table

Fig. 1

 1 Fig. 1 also shows the task example used in this paperbalancing an object on the table. The following subsections

  Fig. 2. Human-humanoid collaboration. (a) shows the passive case with the robot as a pure follower guided only by haptic information. (b) illustrates an equal collaboration approach where both human and robot have a complete knowledge of the task (represented by the blue arrows). Furthermore, each uses both vision (green) and haptic (red) information to achieve this task.

Fig. 3 .

 3 Fig. 3. Human-humanoid table carrying task with reference frames.

Fig. 4 .

 4 Fig. 4. Typical raw data (RGB + Depth images) during the task. Left: RGB image. Right: Depth image, where dark red→bright red corresponds to "far"→"near" and black pixels are regions without data.

Fig. 5 .

 5 Fig. 5. A simplified "thin beam" model used to control the table height

Fig. 6 .

 6 Fig. 6. Snapshots of two experiments where the human-humanoid dyad transports a table with a ball (fast in the top sequence, slow in the bottom one).

This work is supported in part by the FP7 IP Robo-How.Cog project (www.robohow.eu). FP7-ICT-2011-7 Contract No 288533. The authors would like to thank Franc ¸ois Keith, Damien Petit, Hervé Audren, Kenji Kaneko and Eiichi Yoshida for their help with the experiments.