Don Joven Agravante

Hervé Audren

Stanislas Brossette

Stéphane Caron

Benjamin Chrétien

Giovanni De Magistris

Pierre Gergondet

Adrien Pajon

Antonio Paolillo

Damien Petit

Joris Vaillant

Daniel Jartoux

Joan Massot

Keywords: force sensing from vision, motion capture, humanoid robotics. Résumé capture de force par vision, capture de mouvement, robotique humanoïde

The human sense of touch is of fundamental importance in the way we perceive our environment, move ourselves, and purposefully interact with other objects or beings. Thus, contact forces are informative on both the realized task and the underlying intent. However, monitoring them with force transducers is a costly, cumbersome and intrusive process. In this thesis, we investigate the capture of haptic information from motion tracking. This is a challenging problem, as a given motion can generally be caused by an infinity of possible force distributions in multi-contact. In such scenarios, physics-based optimization alone may only capture force distributions that are physically compatible with a given motion, rather than those really applied. In contrast, machine learning techniques for the black-box modelling of kinematically and dynamically complex structures are often prone to generalization issues. We propose a formulation of the force distribution problem utilizing both approaches jointly rather than separately. We thus capture the variability in the way humans instinctively regulate contact forces while also ensuring their compatibility with the observed motion. We present our approach on both manipulation and whole-body interaction with the environment. We consistently back our findings with ground-truth measurements and provide extensive datasets to encourage and serve as benchmarks for future research on this new topic.

List of figures

(c) However, the occluded finger poses are physically impossible as none hold the object. The accumulation of tracking errors can lead to (d) implausible and even (e) impossible poses. . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Reference grasps from left to right: large diameter, precision sphere and tripod. 2.10 Each column represents the optimal solution yielded by our algorithm for increasing values of parameter α. The two first rows show the grasp candidate at the beginning of the experiment (front and back views). The third row corresponds to the same instant as the frame depicted in Fig. 2.8a. We can thus reconstruct various physically plausible grasps, that become closer to the initial observations as we increase α. . . . . . . . . . . . . . . . . . .

3.1

We collect the manipulation kinodynamics dataset using dedicated instrumented devices of adjustable shape, friction, mass distribution and contact configuration (a-c). Additionally, we construct devices based on everyday objects, instrumented so as to allow intuitive interactions (d-f). . . . . . . . contact with the environment. The equations of motion dictate that the net contact force should be constant (top row), which is not apparent on the force sensor measuments (red line) due to sensing uncertainties. Forces compatible with the observed kinematics can be computed using an SOCP (green and blue lines). The minimization of the L 2 norm alone yields forces that are physically plausible but differ significantly from the measurements. Instead, minimizing the discrepancy to the uncertain measurements yields forces that are realistic both physically and compared to actual distributions. . . . . . Hand or object pose q = (q x , q y , q z , q w ) Quaternion of imaginary part (q x , q y , q z ) and real part q w ω ω ω, α α α Rotational velocity and acceleration v, a Linear velocity and acceleration of the center of mass Operators d (N) • dt (N) N-th order time differentiation operator

M -1 Inverse of M M -T Transpose of M u • v Dot product

Introduction

The recent years have let us witness impressive advances in the fields of robotics and artificial intelligence. In October 2015, Google DeepMind's AlphaGo became the first computer program to defeat world-class champion Lee Sedol at go [SHM + 16], a game long considered extremely difficult for computers to win due to its large branching factor. New techniques for large-scale machine learning are enabling robots to develop grasping skills without human supervision [START_REF] Levine | Learning hand-eye coordination for robotic grasping with large-scale data collection[END_REF], and progress in actuation and balancing systems are allowing biped robots to withstand human aggression with a stick. In parallel, augmented and virtual reality technologies are getting more and more immersive and affordable to customers. Extrapolating from here, one could imagine a future where self-driving cars take over the world by running over humans too absorbed in mixed reality activities to look around when crossing the street. Fortunately, this scenario remains (rather bad) science fiction. The winning AlphaGo system garnered hundreds of CPUs and GPUs in parallel for a very specialized purpose, illustrating the fact that it remains a major challenge to construct a truly general artificial intelligence. Robots still lack sensing abilities for both themselves and the environment, e.g., tactile sensors enabling the perception of subtle haptic clues for dexterous manipulation, or robust object identification and tracking from vision without relying on fiducial markers.

Stimulating the human sense of touch also remains a challenge for virtual reality systems to achieve further immersion and embodiment. Overall, a central theme towards action understanding and control is that of haptic perception. Indeed, a privileged way humans interact with their environment is through touch, i.e., the application of contact forces to move objects, themselves and others. Being able to capture these by means of a simple and affordable setup would open a wide range of possibilities for multiple fields of research and engineering. For instance, in robotics, this could enable intuitive interfaces for learning from demonstration and human-robot interaction. In neuroscience, being able to estimate the forces applied during manipulation and reproducing them by means of haptic feedback could benefit the sensation of virtual or robotic embodiment for virtual reality and teleoperation. In rehabilitation, monitoring forces applied during manipulation and locomotion could help detect musculoskeletal conditions and guide movement training. In effect, force models are already being introduced in computer vision, for action understanding and robust motion capture based on physics. Overall, haptic perception has been shown to be crucial in the human ability to grasp and manipulate objects [START_REF] Johansson | Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects[END_REF]. However, traditional force sensing technologies are often costly, cumbersome, as well as of limited accuracy under repeated use. Importantly, they are also intrusive, in that mounting them onto objects can noticeably impact their physical properties. Alternatively, using wearable sensors (e.g., placed at the fingertips) hinders the natural perception of crucial properties (e.g., friction) by the human sense of touch. In both cases, instrumenting either is a time-consuming process that induces further limitations due to both the whole equipment needed (e.g., power supply, cabling) and its effect on the manipulation task (e.g., limitations on the natural range of motion). Computer vision research has resulted in multiple successful methods for monitoring motion information from markerless observations. Thus, a challenging question is: can we also estimate contact forces from vision? If successful, such a method would enable applications such as those discussed before, while also tremendously benefiting their usability. The completely non-intrusive estimation of contact forces would be particularly useful for monitoring and learning of daily activities in the context of home and service robotics.

However, this is an extremely challenging problem, for multiple reasons. First, before even considering forces, the markerless tracking of human subjects (whole-body or restricted to the hands) is itself an active topic in computer vision research. Indeed, such problems typically involve a large number of degrees of freedom, while visual observations may be subject to strong occlusions. In addition, the tracking method must account for, or be robust to, sensing limitations (e.g., number of cameras, resolution) and uncertainties (e.g., depth sensing noise). At this stage, it is unclear whether the state of the art in markerless visual tracking permits the accurate estimation of kinematic information (e.g., velocities and accelerations) on unconstrained interactions (e.g., rapid motions). Second, estimating when and where contact occurs between the human subject and the environment is not trivial due to mutual occlusions inherent to contact situations. Third, even when the motion and its characteristics are perfectly known, the force distribution problem is indeterminate in multi-contact. Indeed, while the knowledge of a force distribution completely characterizes the resulting kinematics, the converse is generally not true. Instead, in multi-contact, a desired net force can generally be distributed in an infinity of different configurations on a given set of contact points.

Still, spatiotemporal relationship between actor and objects can help enhance the joint understanding of both. In this thesis, we leverage the state of the art in computer vision, robotics and machine learning to investigate the problem of force sensing from vision (FSV). Typically, biomechanics approaches aim at solving the force distribution problem through inverse optimization, i.e., by searching for the criteria supposedly optimized by the central nervous system. While theoretically sound, such approaches have so far mainly produced models limited to very specific scenarios and grasping conditions (e.g., holding an object still between two fingers), due to the great complexity of the human body and the limited observability of physiological parameters without invasive surgery. Conversely, force models for physics simulation have recently been employed as optimization priors in hand-object markerless tracking. While helpful for this purpose, these approaches only aim at computing force distributions that are physically plausible, rather than the actual forces being applied.

In contrast, throughout this thesis, it is a primary concern for us to demonstrate the validity of the FSV framework we propose by consistently comparing the forces distributions estimated by our method with ground-truth measurements acquired experimentally on real tasks. We consider the force distribution problem both for hand-object manipulation and whole-body interaction with the environment. The core of our approach lies in the utilization of physics-based optimization and machine learning jointly, rather than separately. On its own, the former may be limited to physical plausibility, rather than fidelity to real force distribution patterns. The latter, by itself, is often subject to generalization issues, e.g., when Introduction it is applied to examples that differ significantly from those present in the training dataset. By using them together, we capture the variability in the way humans naturally apply contact forces when interacting with objects and their environment, while ensuring that the resulting force distributions are compatible with the observed motion. The organization of this thesis is as follows:

• In Chapter 1, we review the state of the art in motion and force sensing technologies as well as visual tracking techniques and contact dynamics modeling.

• In Chapter 2, we introduce the topic of force sensing from vision in the context of manipulation. First, we evaluate the performance of a state-of-the-art hand-object tracker regarding motion and kinematics estimation. We then formulate an optimization problem estimating the minimal forces required to achieve the observed motion, and complement it with artificial neural networks that predict the additional forces humans naturally apply to secure the object in the grasp.

• In Chapter 3, we extend our work to 3D (normal and tangential) manipulation forces while accounting for time continuity and variability across object and grasp configurations with a new pipeline combining physics-based optimization and recurrent neural networks in mutual interaction. To assess the performance and extensibility of our approach, we also construct and release the first large-scale dataset on human manipulation kinodynamics with high-precision motion and force sensors.

• In Chapter 4, we challenge the estimation of whole-body contact forces in interaction with the environment from motion capture. We show that in such situations, forces estimated from optimization only differ significantly from the forces applied in reality, while conversely, force sensor measurements can be rather unreliable. We collect a new dataset on whole-body kinodynamics using an inertial motion capture system and external force sensors, and show that our approach can be successfully extended to challenging multi-contact configurations.

Chapter 1

Literature Review

The observation of humans in interaction with their environment is of great interest for multiple fields of research, such as robotics, computer vision, graphics and rehabilitation. Such interactions can be observed and encoded in terms of motion, i.e., the relative poses through time between one or multiple human actors and surrounding objects. From a lowerlevel perspective, humans interact with their environment in a privileged way through touch, i.e., the application of contact forces and torques onto surrounding objects, permitted by the remarkable dexterity of the human hand. As such, the monitoring of interaction forces is informative of both the resulting motions and the human intent. We first review the motion and force sensing technologies and their applications (Section 1.1). We discuss the state in the art in visual tracking (Section 1.2). We review existing kinematic and dynamic models for human motion analysis (Section 1.4). Finally, we discuss numerical techniques used for kinodynamic modeling (Section 1.5).

Monitoring Human Interactions With The Environment

In this section, we discuss the types of sensors used for motion and force monitoring and illustrate their combined utilization in past works.

Motion Sensors

Motion capture aims at tracking the movement of target objects or subjects. It is today commonly used to animate virtual characters or avatars in computer graphics and virtual reality. The tracking target can be rigid (e.g., a mug), articulated (e.g., whole-body tracking), or deformable (e.g., facial motion capture). The motion can be monitored in terms of positions (linear, angular) or the subsequent derivatives (e.g., velocity, acceleration).

Motion capture techniques based on vision can be placed into two categories: markerbased or markerless. In the former, markers placed at prespecified landmarks on the tracking target are located in space from the visual observations (e.g., by thresholding), and mapped to the target kinematic structure [START_REF] Moeslund | A survey of computer visionbased human motion capture[END_REF][START_REF] Moeslund | A survey of advances in vision-based human motion capture and analysis[END_REF]. While marker-based methods generally yield very precise tracking, the required instrumentation is cumbersome. Markerless approaches aim at alleviating this issue by relying on visual observations only and were reviewed in [START_REF] Poppe | Vision-based human motion analysis: An overview[END_REF] in the context of human motion analysis. A problem inherent to vision-based systems is the possibility of mutual occlusions between multiple tracking targets (e.g., an object hiding the hand manipulating it, and conversely) or self-occlusions between different parts of the same target (e.g., the palm hiding some fingers). This issue can be alleviated by using multi-camera systems to limit the amount of occlusions. Such systems also allow the obtention of depth information by 3D reconstruction from multiple views [START_REF] German | Shape-fromsilhouette of articulated objects and its use for human body kinematics estimation and motion capture[END_REF].

However, multi-camera systems are not very portable and require extensive calibration, making them difficult to use in uncontrolled environments. Inspired by human binocular vision, stereo vision systems use two calibrated cameras with parallel optical axes to seek corresponding points between view pairs and extract depth information [START_REF] Lucas | An iterative image registration technique with an application to stereo vision[END_REF]. Alternative methods used in consumer-grade RGB-D (color and depth) sensors include structured light [START_REF] Scharstein | High-accuracy stereo depth maps using structured light[END_REF], which consists in projecting a known pattern onto the scene and analyzing the corresponding deformations (e.g., Microsoft Kinect, Asus Xtion), and time-offlight [START_REF] Gokturk | A time-of-flight depth sensor-system description, issues and solutions[END_REF], based on the delay between the emittance of a light pulse and its reflection by the objects of the scene (e.g., Microsoft Kinect v2, SoftKinetic DepthSense DS325).

Finally, strong occlusions can occur even when using multiple cameras, for example when multiple tracking targets interract with each other; in tight spaces; or in uncontrolled environments (e.g., outdoor). In such scenarios, body joint angles can be tracked using non optical systems worn on the subject's body. Inertial motion capture systems rely on inertial measurement units (IMUs) placed at specified landmarks on the subject's body (e.g., Perception Neuron, Xsens MVN Awinda). The joint angles are computed by sensor fusion and integration of the IMUs' accelerometer and gyroscope measurements [START_REF] Roetenberg | Xsens mvn: full 6dof human motion tracking using miniature inertial sensors[END_REF]. As such, inertial motion capture systems are prone to positional drift. External camera systems can be used in combination with inertial motion capture to provide absolute positioning.

Force Sensors

Force sensing is a key objective in understanding physical interactions between humans and their environment. Not only can tactile feedback provide valuable insight when setting up haptic interfaces, it is also of vital importance when monitoring manipulation tasks performed by robots, e.g., frail object grasping. Force sensors now come up in a variety of types and specifications (e.g., capacitive, piezoresistive, single or multi-axis) [START_REF] Cutkosky | Springer Handbook of Robotics, chapter Force and Tactile Sensors[END_REF], with particular requirements and applications in robotics [START_REF] Ravinder | Tactile sensing -from humans to humanoids[END_REF][START_REF] Kappassov | Tactile sensing in dexterous robot hands -review[END_REF]. Common drawbacks of mechatronic force sensing devices reside in their extensive need for calibration, accuracy and repeatability limitations (e.g., hysteresis) and cost. Besides their sensing capabilities, another difficulty lies in their intrusiveness. For example, consider a human subject manipulating an object of given physical properties. In order to measure manipulation forces, sensors must be placed at the contact locations between the object and the hand. If mounted onto the object:

• the contact locations must be chosen in advance

• the object must be modified to fit force transducers at the specified locations as well as any additional instrumentation required (e.g., wires)

• such modifications can be cumbersome, time-consuming, and importantly affect the physical properties of the object (e.g., shape, mass distribution)

If mounted onto the hand:

• contacts can be placed arbitrarily but additional hand tracking is required to know their position throughout the experiment

• the force sensors can significantly impair the human haptic sense (e.g., friction perception) and limit the natural range of motion (e.g., joint angles, contact orientations)

We depict such instrumentation examples in Fig. 1.1. A less intrusive approach could consist in covering the surface of the object with lightweight, flexible tactile sensing surfaces, reviewed in [START_REF] Stassi | Flexible tactile sensing based on piezoresistive composites: A review[END_REF]. Doing so would allow the monitoring of manipulation forces on arbitrary contact points with minimal impact on the object and the hand. However, such technologies are still limited in terms of captured dimensions (e.g., normal forces only) and precision (i.e., providing accurate measurements of the applied forces). During our preliminary experiments, we found that commercially available force-sensing resistors (FSR, Interlink Electronics), are better suited for contact detection than accurate force sensing, despite extensive calibration.

An alternative to pressure sensors proposed in [START_REF] Mascaro | Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction[END_REF] consists in instrumenting fingers with miniature LEDs and photodetectors to measure changes in fingernails coloration, that are then correlated to the touch force applied at fingertips. Later, this technology evolved to predict normal and shear forces, and even changes in posture, that appear to have different blood volume patterns [START_REF] Mascaro | Measurement of finger posture and three-axis fingertip touch force using fingernail sensors[END_REF]. Fingernail color and surrounding skin changes were also monitored and processed using an external camera system to estimate contact fingertip forces in [SHM08, SHM09, GHM13, UBO + 13]. Conversely, computer graphics models were developed to simulate fingertip appearance changes based on simulated forces [START_REF] Andrews | Data-driven fingertip appearance for interactive hand simulation[END_REF]. This approach is, however, limited to fingertip contacts and requires extensive calibration for each individual user, since nail appearances can vary between subjects and through time. It is also limited by the necessity of having fingernails visible at all time and at high resolution, requiring appropriately mounted miniature cameras. Still, this result illustrates that the problem of estimating contact forces during manipulation can indeed be tackled by computer vision.

Applications of Motion and Force Monitoring

Observing a scene with a single RGB-D sensor in [START_REF] Kyriazis | Physically plausible 3d scene tracking: The single actor hypothesis[END_REF], Kyriazis et al. showed that the computation of interaction forces explaining the motion of visible objects could help infer the motion of objects hidden from the camera. Hand-object grasps were classified in terms of shape and contact configurations in multiple taxonomies [Cut89, LFNP14, FRS + 16]. These taxonomies were augmented with contact force estimates in [START_REF] Rogez | Understanding everyday hands in action from rgb-d images[END_REF] to construct a dataset on functional grasp understanding. Also in action and scene understanding, Zhu et al. also used forces as a physical concept to model tool use [START_REF] Zhu | Understanding tools: Taskoriented object modeling, learning and recognition[END_REF] and learn human utilities [ZJZ + 16], e.g., by quantifying comfort intervals while sitting on a chair in terms of forces exerted on body parts. Kry et al. proposed an acquisition setup combining markerbased motion capture and force transducers to estimate hand joint compliance and synthesize interaction animations [START_REF] Kry | Interaction capture and synthesis[END_REF].

Besides computer vision, grasp taxonomies were used in robot learning from demonstration (LfD) for the planning of in-hand manipulation actions [START_REF] Prieur | Modeling and planning high-level in-hand manipulation actions from human knowledge and active learning from demonstration[END_REF]. The monitoring of contact forces is of critical importance for such dexterous actions. Towards this purpose, Liu et al. proposed a robotic fingertip equipped with a 6-axis force-torque sensor and a 1.2 Markerless Visual Tracking rubber skin allowing the accurate and high-speed sensing of contact wrench and even contact location [LNP + 15]. In [START_REF] Rozo | A robot learning from demonstration framework to perform force-based manipulation tasks[END_REF], Rozo et al. introduced an LfD framework to teach a robotic manipulator to place a ball in a box and pour a drink relying solely on force perception and haptic feedback. In the context of whole-body motion, [START_REF] Englsberger | Biologically inspired dead-beat controller for bipedal running in 3d[END_REF] modelled ground reaction forces (GRF) measured during human running experiments as polynomial splines to construct a general controller for bipedal running, independent of specific hardware constraints for the considered robot. Similarly, in graphics, the whole-body motion of a human subject was reconstructed from GRFs measured with consumer-grade pressure sensing platforms and a hand-tracking device in [START_REF] Ha | Human motion reconstruction from force sensors[END_REF].

Still in computer graphics, ensuring that a desired motion is physically plausible through the application of compatible contact forces allows the generation of realistic-looking animations. In [START_REF] Liu | Synthesis of interactive hand manipulation[END_REF] and [START_REF] James | Dextrous manipulation from a grasping pose[END_REF], Liu introduced an optimization framework allowing the production of such physically plausible manipulation animations with only inputs a starting grasp pose and a partial trajectory for the object. This work was later on extended in [START_REF] Ye | Synthesis of detailed hand manipulations using contact sampling[END_REF] to animate a hand manipulating objects from the motion capture of the objects and the subject's wrist, without tracking the motion of the individual fingers. Hand control strategies were also constructed in [START_REF] Andrews | Goal directed multi-finger manipulation: Control policies and analysis[END_REF][START_REF] Bai | Dexterous manipulation using both palm and fingers[END_REF] to accomplish general actions such as reorienting a ball in a chosen direction rather than specifying its exact trajectory. Mordatch et al. introduced the contact-invariant optimization (CIO) method in [START_REF] Mordatch | Discovery of complex behaviors through contact-invariant optimization[END_REF], enabling the synthesis of complex animations from high-level goals only by simultaneous optimization of contact and motion. This method was applied to dexterous manipulation in [START_REF] Mordatch | Contact-invariant optimization for hand manipulation[END_REF]. In the context of visual servoing, i.e., the incorporation of visual information for robot control [START_REF] Espiau | A new approach to visual servoing in robotics[END_REF][START_REF] Chaumette | Visual servo control. i. basic approaches[END_REF], Agravante et al. combined haptic information and vision for the human-robot collaborative carrying of a table while preventing a ball on top from falling off [ACB + 14].

Markerless Visual Tracking

Towards the non-intrusive monitoring of contact forces, our work capitalizes on the recent advances of markerless visual tracking for pose estimation and motion perception. Public libraries enabling the robust and efficient tracking of one or multiple objects are now routinely deployed in production or robotics research systems, e.g., ARToolKit [START_REF] Kato | Marker tracking and hmd calibration for a video-based augmented reality conferencing system[END_REF], ViSP [START_REF] Marchand | Visp for visual servoing: a generic software platform with a wide class of robot control skills[END_REF], BLORT [MPR + 10]. Such methods typically rely on feature tracking [START_REF] Marchand | Feature tracking for visual servoing purposes[END_REF] (e.g., dots, contours, SIFT keypoints [START_REF] David | Distinctive image features from scale-invariant keypoints[END_REF]) and can even scale to hundreds of objects in real time [START_REF] Pauwels | Real-time pose detection and tracking of hundreds of objects[END_REF] or articulated systems [START_REF] Comport | Kinematic sets for real-time robust articulated object tracking[END_REF][START_REF] Pauwels | Real-time model-based articulated object pose detection and tracking with variable rigidity constraints[END_REF], provided accurate visual and kinematic models of the latter. In contrast, tracking human subjects is subject to different constraints, e.g., body shape and appearance variety across individuals and populations, kinematic complexity and high dimensionality, self-occlusions between body parts. In this section, we review the state of the art in the markerless visual tracking of human subjects, either whole-body or restricted to the hand(s). Pose estimation methods can be classified into mostly bottom-up, mostly top-down, or hybrid. In bottom-up approaches, given visual observations of the subject, the first step is to identify the location of body parts in the image. The detected body parts are then assembled into a complete pose based on considerations such as body part proximity and temporal coherence. Conversely, in top-down approaches, a generative process creates subject pose hypotheses that are evaluated against actual observations. Final pose estimates are obtained by solving a multi-parameter optimization problem, e.g., the minimization of the discrepancy between the pose hypotheses rendered graphically and the visual observations. We depict representative examples of bottom-up and top-down pose estimation approaches in Fig. 1.2.

Bottom-Up Methods

Marker-based motion capture is a case of bottom-up pose identification, as body parts are first localized in space, then matched to a target kinematic structure. In the first work on markerless hand tracking [START_REF] Rehg | Visual tracking of high dof articulated structures: an application to human hand tracking[END_REF], Rehg and Kanade extracted fingertip locations and finger bone central axes from the observed silhouette of the hand. Recent bottom-up approaches are typically data-driven and rely on discriminative models to learn a mapping between 1.2 Markerless Visual Tracking visual inputs and poses from labeled examples. Wang and Popović used a colored glove to identify hand parts and track the hand in real time by nearest neighbor search in a hand-pose appearance database [START_REF] Wang | Real-time hand-tracking with a color glove[END_REF]. Romero et al. generated a database of synthetic hand images to track the hand without a colored glove [START_REF] Romero | Monocular real-time 3d articulated hand pose estimation[END_REF] and later extended this idea to synthetic hand-object poses [START_REF] Romero | Hands in action: real-time 3d reconstruction of hands in interaction with objects[END_REF]. Random decision forests (RDF) [START_REF] Breiman | Random forests[END_REF] were trained on synthetic data for hand tracking [START_REF] Keskin | Real time hand pose estimation using depth sensors[END_REF][START_REF] Keskin | Hand pose estimation and hand shape classification using multi-layered randomized decision forests[END_REF] Bottom-up approaches are typically computationally cheap as they avoid the computation of graphic renderings and discrepancies to the visual observations. [SFB + 11] thus performed whole-body tracking at 200 frames per second, enabling its use as a side process for realtime applications (e.g., human-computer interaction, video gaming). Another advantage is that bottom-up approaches can operate on a per-frame basis, without relying on temporal coherence, while top-down approaches require manual initialization (at least) for the first frame of the sequence. However, the quality of the pose estimation is directly contingent on the training data and may not generalize well to previously unseen inputs, e.g., different hand poses, occlusion cases, or hand-object interactions.

Top-Down Methods

Rather than discriminative, top-down methods are generative, or model-based. Using an explicit model of the subject (e.g., 3D geometry, inertial parameters), pose hypotheses are optimized by evaluation against the visual observations. Global optimization is generally not possible due to the large dimensionality of the search space and the computational cost of rendering and evaluating pose hypotheses. Instead, local search is performed in the vicinity of an initial pose estimate, which can be initialized manually for the first frame of the sequence, or taken as the optimal pose found at the previous frame. In an early work on model-based whole-body tracking [START_REF] Dariu | 3-d model-based tracking of humans in action: a multi-view approach[END_REF], Gavrila et al. decomposed the search space in a hierarchical manner, following the kinematic tree, by searching first for the head and torso, then the upper arms and thighs, then the forearms and lower legs. At each step, the pose of the corresponding limbs was found by discretizing the reduced search space and maximizing a similarity measure between visual observations given by a calibrated multi-camera setup and the synthesized appearance of the pose hypotheses in each camera view.

Literature Review

In [START_REF] De | Model-based 3d hand pose estimation from monocular video[END_REF], de La Gorce et al. proposed an objective function differentiable with respect to the hand pose and its lighting, allowing its efficient optimization using a quasi-Newton method. Still, the 3D color rendering of a hand under different conditions of lighting remained particularly expensive, leading to a total computation time of approximately 40 s per frame (i.e., 0.025 fps). In addition to color, depth information has been increasingly used in recent approaches. The iterative closest point (ICP) algorithm [START_REF] Besl | A method for registration of 3-d shapes[END_REF], used to minimized the discrepancy between two point clouds, was adapted to whole-body tracking in [START_REF] Ganapathi | Real-time human pose tracking from range data[END_REF] and ran at 125 fps on a single-threaded CPU implementation. However, ICP alone is easily trapped in local minima. Tagliasacchi [START_REF] Kennedy | Particle swarm optimization[END_REF] using either a multi-camera setup [START_REF] Oikonomidis | Markerless and efficient 26-DOF hand pose recovery[END_REF] or a single RGB-D sensor [START_REF] Oikonomidis | Efficient model-based 3d tracking of hand articulations using kinect[END_REF]. The use of stochastic optimization alone allows the incorporation of arbitrary priors in the optimization process regardless of differentiability constraints. The fact that two different objects cannot share the same physical space was implemented by penalizing interpenetrations between 3D shapes in the cost function, allowing the same framework to also track a hand in interaction with an object [START_REF] Oikonomidis | Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF] or two strongly interacting hands [START_REF] Oikonomidis | Tracking the articulated motion of two strongly interacting hands[END_REF]. Complete occlusions were also treated through physicsbased simulation [START_REF] Kyriazis | Physically plausible 3d scene tracking: The single actor hypothesis[END_REF]. Wang et. al proposed a contact-based sampling approach allowing the monitoring of subtle hand-object interactions during dexterous manipulation using a multi-camera setup [WMZ + 13] and physics-based simulation.

In effect, top-down methods allow the treatment of occlusions not as a distractor, but rather as a source of information. A major advantage resides in their ability to tackle virtually any situation, provided the models of all the objects in the scene. However, their computational cost is generally considerable, although modern implementations (e.g., GPGPU) now enable real-time tracking. Still, reinitializing the pose search during the sequence remains problematic (e.g., as tracking errors accumulate through time or when the subject exits the field of view).

Hybrid Methods

Hybrid methods aim at combining the advantages of top-down and bottom-up approaches. Discriminative methods can provide a fast and rough pose estimate to efficiently initialize a slower but more refined search through local optimization.

In [START_REF] Sridhar | Interactive markerless articulated hand motion tracking using rgb and depth data[END_REF], Sridhar et al. combined a generative method based on a Sum of Gaussians (SoG) model, with a linear SVM classifier detecting fingertips from depth maps. The SoG model was introduced earlier for whole-body tracking [SHG + 11] to provide a differentiable cost function for fast local optimization. The SoG model was also used in combination with RDFs for the tracking of a hand in [START_REF] Sridhar | Fast and robust hand tracking using detection-guided optimization[END_REF] and that of a hand manipulating an object [SMZ + 16], in real time. Qian et al. [QSW + 14] combined a fingertip detector with a generative model making use of both ICP for fast local optimization and PSO to explore the search space more thoroughly. In the approach of Sharp et al. [SKR + 15], the discriminative model does not produce just a single good pose estimate but rather a distribution over possible hand poses, which are then fitted to the depth observations using a variant of the PSO algorithm. The resulting hand tracker was highly flexible with respect to camera placement, demonstrated great robustness to tracking failure, and ran in real time on a GPU implementation. This approach was extended in [TBC + 16] with a smooth hand model and a differentiable cost function, enabling the use of gradient-based optimization techniques in the generative process for real-time tracking on CPU only.

Overall, hybrid methods have demonstrated promising results for the tracking of a hand in isolation. However, the tracking of multiple hands or hand-object interactions has been comparatively less studied. For these situations, current approaches suffer from the same drawbacks as bottom-up methods, in particular in the presence of strong occlusions.

Model-Based Hand-Object Tracking

While computationally expensive, top-down, model-based tracking methods allow the tracking of arbitrary subjects and objects in a unified framework. In this section, we review the FORTH hand-object tracking method [OKA10, OKA11a, OKA11b, OKA12, KA13], upon which we built our framework for force sensing from vision. First, we describe the requirements of the method (Section 1.3.1). We then summarize the hypothesize-and-test pose estimation strategy (Section 1.3.2). Finally, we discuss the incorporation of tracking priors in the optimization process (Section 1.3.3).

Observations and Models

We consider a set of N C cameras calibrated intrinsically and extrinsically. At a given time step i, we denote by F i, j the frame acquired by camera j. We call multiframe M i the set of images captured by all cameras at time step i, i.e., M i = F i,1 , . . . , F i,N C . We denote by S the sequence of N S multiframes acquired through time, i.e., S = {M 1 , . . . , M N S } . In [START_REF] Oikonomidis | Markerless and efficient 26-DOF hand pose recovery[END_REF][START_REF] Oikonomidis | Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF], the multiframes are acquired by 8 color cameras observing the scene at 30 fps and 1280 × 960 resolution. In [OKA11a, OKA12, KA13], the scene is observed with a single RGB-D sensor providing synchronized color and depth maps at 30 fps and 640 × 480 resolution. Fig. 1.3a illustrates sample color and depth frames captured by a SoftKinetic RGB-D sensor.

Hands and objects are tracked based on their 3D shape. In the case of rigid objects, a dedicated 3D model must be provided (e.g., made available by the manufacturer or by CAD). The pose of a rigid object is then characterized by 6 DoF, i.e., its 3D orientation and the position p = (p x , p y , p z ) T of a reference point of its geometry. For numerical reasons, 3D orientations are commonly represented by unit quaternions q = (q x , q y , q z , q w ) T among other representations and conventions (e.g., Euler angles, orthogonal matrices, etc.). The 6-DoF pose of a rigid object can thus be encoded by a 7-element vector: θ θ θ rigid = (q x , q y , q z , q w , p x , p y , p z ) T , with q 2 = 1.

(1.1)

The hand is modeled as an articulated system of rigid bodies following [START_REF] Albrecht | Construction and animation of anatomically based human hand models[END_REF]: the palm and three segments for each of the five fingers. We denote the finger set by F : F = {thumb, index, middle, ring, pinky} .

(1.2)

The palm is chosen as root of the kinematic tree and its global pose is encoded by 7 parameters as in Eq. (1.1). The finger poses are then characterized by 4 parameters each, i.e., 2 DoF for the finger base and 1 DoF for each of the two remaining joints. In total, the 26-DoF hand model is encoded by a 27-element vector:

θ θ θ hand = θ θ θ palm , θ θ θ k k∈F , (1.3)
with θ θ θ palm the 7-parameter pose of the palm and θ θ θ k the 4 joint angles of finger k ∈ F . Provided an instance of θ θ θ hand , the 3D pose of each segment is computed by forward kinematics. To each segment is associated a visual representation using a combination of the following geometric primitives: cones, cylinders, ellipsoids and spheres, Autocollisions between different segments of the hand and interpenetration between hands and objects

Pose Estimation Strategy

In this section, we describe a representative pipeline for the tracking of a hand in isolation using a single RDB-D sensor, as performed in [START_REF] Oikonomidis | Efficient model-based 3d tracking of hand articulations using kinect[END_REF]. More complex cases are treated with the incorporation of tracking priors in the objective function, discussed in Section 1.3.3. We consider a color image of the hand and the corresponding depth map as acquired by an RGB-D sensor at a given time step i. First, the area of interest is obtained by segmenting the hand from the rest of the color image by skin color detection [START_REF] Argyros | Real-time tracking of multiple skin-colored objects with a possibly moving camera[END_REF]. We denote by o s the 2D map of the segmented skin color. The depth of the skin-colored pixels is conserved from the raw depth map, while the rest is set to zero. We denote by o d the resulting depth map. The segmented color and depth observations are O = (o s , o d ).

The objective is then to find the hand pose estimate θ θ θ * that minimizes a cost function E , or energy, quantifying the discrepancy between observations O and pose hypotheses θ θ θ :

θ θ θ * = argmin θ θ θ {E (θ θ θ , O)}.
(1.4) The energy E is comprised of two terms: a cost E 3D measuring the discrepancy between the visual observations and the rendered 3D hand model, and a cost E kin that quantifies the likelihood of the hand pose hypothesis by itself, regardless of the observations:

E (θ θ θ , O) = λ 3D • E 3D (θ θ θ , O) + λ kin • E kin (θ θ θ ), (1.5)
with λ 3D and λ kin normalization coefficients adjusted manually for the optimization process.

Given the camera intrinsic and extrinsic parameters, a hand pose hypothesis θ θ θ can be rendered graphically to generate a synthetic depth map r d . The cost function E 3D thus implements a discrepancy metric between synthetic and measured depth maps, e.g., the pixel-wise absolute difference between r d and o d . Regardless of the observations, E kin penalizes kinematically implausible hand poses, e.g., those resulting in auto-collisions between finger segments. We depict the pose estimation process in Fig. 1.4a.

Incorporating Tracking Priors

Provided perfect observations, the optimization of the visual discrepancy cost function E 3D alone could theoretically lead to the real hand pose. In reality, observations are often partially missing (e.g., occlusions) or subject to measurement uncertainties (e.g., motion blur). To guide the pose estimation, tracking priors are incorporated through additional terms in the definition of the objective function E in Eq. (1.5). As reviewed in Section 1.2.2, the energy E is minimized by PSO. This choice is motivated by the limited number of hyperparameters to adjust manually (e.g., number of particles) and its efficiency in exploring the search space beyond local optima. Additionally, it does not impose any constraint on the objective function (e.g., differentiability). Thus, arbitrary priors can be incorporated in the optimization process in a unified computational framework.

In [START_REF] Oikonomidis | Efficient model-based 3d tracking of hand articulations using kinect[END_REF], the hand kinematic plausibility cost E kin penalized overlapping adjacent fingers by comparing their abduction-adduction angles (i.e., how fingers are spread apart). This approach was also used for the tracking of two hands in interaction [START_REF] Oikonomidis | Tracking the articulated motion of two strongly interacting hands[END_REF]. However, this analytical formulation did not account for every possible collision (e.g., between thumb and pinky fingertips) and penalized certain valid poses (e.g., crossed index and middle fingers). Instead, [START_REF] Oikonomidis | Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF] computed hand-hand and hand-object penetration volumes using a third-party physics engine [S + 05], exploiting the flexibility of PSO with respect to the objective function. In [START_REF] Kyriazis | Physically plausible 3d scene tracking: The single actor hypothesis[END_REF], the cost function was augmented with a term comparing the observations with physics-based simulation outputs [C + 13], allowing the pose recovery of fully occluded objects based solely on the tracking of the hand and the simulation of its effect on the objects of the scene, visible or not.

Another advantage of PSO is its parallel nature, enabling real-time implementations on the GPU [START_REF] Oikonomidis | Efficient model-based 3d tracking of hand articulations using kinect[END_REF][START_REF] Oikonomidis | Tracking the articulated motion of two strongly interacting hands[END_REF]. However, the computational complexity grows geometrically with the number of tracked subjects and objects, when accounting for them simultaneously. On the other hand, using multiple independent trackers scales better with the number of objects but does not account for occlusions between them. The scalability of generative methods was addressed in [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] through the concept of Ensemble of Collaborative Trackers (ECT), in which each individual trackers per object broadcast their results with each other. Fig. 1.4b depicts the tracking of two hands in interaction with multiple objects. In this thesis, we tracked hand-object interactions using a variant of ECT.

Modeling Contact Dynamics

In this section, we first review existing techniques for the modeling of the human body (Section 1.4.1). We then discuss the estimation of contact dynamics for the cases of whole-body contacts with the environment (Section 1.4.2) and hand-object interactions (Section 1.4.3).

Human Dynamic Model

The identification of the objective function optimized by the central nervous system in daily activities (e.g., locomotion, manipulation) is a long-standing problem in kinesiology research [START_REF] Vladimir | Kinetics of human motion[END_REF]. In [START_REF] Prilutsky | Optimization-based models of muscle coordination[END_REF], Prilutsky and Zatsiorsky reviewed the state of the art in the prediction of muscle activation patterns (MAP) from optimization, i.e., the understanding of how human efforts are regulated at the musculoskeletal level by the central nervous system (CNS). In doing so, they noted that a major difficulty lies in the high dimensionality of the human body, which allows 244 kinematic DoFs with approximately 630 muscles. As such, it is a highly redundant system, making it difficult to consider biological parameters in isolation. In addition, the observability of such parameters (e.g., muscle forces, joint torques) may be limited without invasive surgery, which further hinders optimization-based techniques requiring ground-truth data.

Whole-body motion and forces are also linked through the equations of motion and the Body Segment Inertial Parameters (BSIP), i.e., the mass, position of the center of gravity and inertia tensor of each body segment. In an early work by Dempster [START_REF] Dempster | Space Requirements of the Seated Operator: Geometric, Kinematic, and Mechanical Aspects of the Body with Special Reference to Limbs[END_REF], eight cadavers were dismembered to provide data on mass, center of gravity, density and moments of inertia. Further measurements on cadavers were performed in [CCM + 75] and compared to body segments models based on geometric primitives such as cylinders and ellipsoids, as done in [START_REF] Hanavan | A mathematical model of the human body[END_REF] While anthropomorphic tables allow the fast computation of BSIP estimates from body segment measurements only, an important caveat resides in the specificities of the studied subjects, e.g., college-aged Caucasian males and females in [START_REF] Zatsiorsky | Methods of determining mass-inertial characteristics of human body segments[END_REF], whose results may not directly extend to other populations. Furthermore, the resulting BSIPs generally assume symmetry between left and right halves of the human body, which may be a rather inaccurate assumption, e.g., when particular muscles are atrophied due to neuromuscular diseases. Towards these issues, Jovic et al. proposed a hierarchical optimization framework for the online estimation of both robot and human BSIPs from direct motion and force-torque measurements [JEA + 16], without further assumption. In [START_REF] Bonnet | Fast determination of the planar body segment inertial parameters using affordable sensors[END_REF], Bonnet and Venture used a Microsoft Kinect RGB-D sensor for visual tracking and a Nintendo Wii Balance Board for force-torque sensing, enabling the online estimation of BSIPs with consumer-grade sensors.

Whole-Body Dynamics

Whole-body motion and ground reaction forces can typically be measured accurately using commercially available solutions, e.g. marker-based motion capture and force plates, as reviewed in Section 1.1. Measuring muscle forces in vivo is difficult without invasive surgery. Instead, a common approach in biomechanics research is to first use the equations of motion to estimate joint torques through inverse dynamics [START_REF] Davy | A dynamic optimization technique for predicting muscle forces in the swing phase of gait[END_REF], and compute muscle forces that minimize criteria such as metabolic energy expenditure [START_REF] Anderson | Dynamic optimization of human walking[END_REF]. However, inverse dynamics solutions are contingent on the underlying BSIP model. The influence of BSIP estimation on inverse dynamics was studied in [PC99, RABF06, RHCF07, WDGJ14] in the context of gait analysis. In [START_REF] Muller | A comparative study of 3 body segment inertial parameters scaling rules[END_REF], Muller et al. estimated joint torques during overhead throwing and reported relative distances up to 70% between BSIP obtained with three different models of the literature. Overall, such approaches are limited by the difficulty to obtain ground-truth data, often limited to muscle excitation patterns measured with electromyography (EMG).

Alternatively, the human motion can be studied from the perspective of sensorimotor control optimality, reviewed in [START_REF] Todorov | Optimality principles in sensorimotor control[END_REF]. A given task, e.g., taking a step foward, can generally be executed in multiple ways due to the redundancy of the human body. Still, it is commonly assumed that humans naturally execute movements that are optimal with respect to certain criteria, shaped by evolution and experience [START_REF] Mcn | The gaits of bipedal and quadrupedal animals[END_REF]. Prilutsky and Zatsiorsky suggested that such criteria might be based on three major physiological cost functions: metabolic energy expenditure, muscle fatigue, and sense of perceived effort [START_REF] Prilutsky | Optimization-based models of muscle coordination[END_REF]. In [START_REF] Mombaur | Using optimization to create self-stable human-like running[END_REF], Mombaur formulated the generation of running motions as an optimal control problem, i.e., the computation of trajectories that respect a given set of constraint and are optimal with respect to a chosen cost function. Conversely, in inverse optimal control, the goal is to find the cost function that is optimized during the observed motion. In [START_REF] Liu | Learning physics-based motion style with nonlinear inverse optimization[END_REF], Liu et al. captured motion styles (e.g., sad / happy) in terms of preferences for applying torques at some joints rather that others to synthesize new walking and running animations. In [START_REF] Mombaur | From human to humanoid locomotion -an inverse optimal control approach[END_REF], Mombaur et al. identified cost functions optimized during human locomotion from motion capture and applied the resulting models to generate natural trajectories on a humanoid robot.

Very much related to our work, the estimation of contact dynamics was also addressed from a computer vision perspective. In [NYFS05], Nakamura et al. performed inverse dynamics on a detailed model of the human body to compute sensomatory information from motion capture, e.g., stimuli perceived at the level of the organs, muscles, tendons and ligaments. Our work was also inspired by Brubaker et al., who estimated joint torques from inverse dynamics and motion capture by parameterizing GRFs with a spring-based model [START_REF] Brubaker | Estimating Contact Dynamics[END_REF]. General contact configurations between the whole body and objects of the environment were also computed by considering the human body elastic [ZJZ + 16]. Zhu et al. used the resulting force estimates to learn human utilities when interacting with their environment, e.g., quantifying preferred poses when sitting in terms of contact forces between the body and the chair. While a major limitation to these works lies in the difficulty of validating the force models with ground-truth measurements, they also illustrate the interest of capturing contact dynamics for action understanding in computer vision, as illustrated in Fig. 1.5.

Prehension and Manipulation Dynamics

Besides whole-body dynamics estimation, the case of prehension and manipulation is another active topic of interest for kinesiology research. This interest stems from the remarkable dexterity of the human hand. It particular, it is a complex and redundant system, such that a given task can generally be executed using multiple force distributions. Similarly to the case of whole-body locomotion, it is difficult to identify clear criteria supposedly optimized by the CNS during multi-finger prehension. From experiments on thumb-index pinching [WJ84], Westling and Johansson showed that grip control is mostly influenced by the object's surface condition and its mass through the safety margin ratio, defined as the proportion of applied forces that are unnecessary to achieve the object's observed kinematics, with respect to the total forces applied. Cadoret and Smith precised that the influence of the surface condition is most important regarding its friction coefficient rather that its texture [START_REF] Cadoret | Friction, not texture, dictates grip forces used during object manipulation[END_REF].

The notion of safety margin is largely related to that of nominal and internal forces. Humans do not manipulate objects using nominal closures (i.e., minimal grasp forces). They tend to "over-grasp" and produce workless internal forces, i.e. firmer grasps than mechanically required through the equations of motion. This grasping property is described by considering finger forces as two sets formalized in [KR86, YN91]: nominal forces responsible for the object's motion and internal forces that cancel each other out [MS85, MSZ94] and thus do not affect the object's kinematics. For instance, when holding a cup statically, nominal forces directly compensate gravity, while internal forces secure the object in place. Humans typically apply internal forces to prevent slip [START_REF] Johansson | Roles of glabrous skin receptors and sensorimotor memory in automatic control of precision grip when lifting rougher or more slippery objects[END_REF][START_REF] Flanagan | Hand movements. Encyclopedia of the[END_REF] and control their magnitude to avoid muscle fatigue or damaging fragile objects [START_REF] Stacey | Manipulation of a fragile object[END_REF][START_REF] Park | Optimality versus variability: effect of fatigue in multi-finger redundant tasks[END_REF].

When manipulating objects, excessive forces applied by a finger are naturally compensated by others, making the problem of force sharing between fingers particularly challenging.

Thus, inverse optimization approaches for manipulation have mostly resulted in models that rely on rather strong simplifying assumptions. The most common restriction is on the motion's dimensionality. e.g., static prehension [START_REF] Niu | Reconstruction of the unknown optimization cost functions from experimental recordings during static multi-finger prehension[END_REF]. Other approaches allow limited motion, but using a simplified grasp model in which individual fingers and hand surfaces are grouped into functional units named virtual fingers [START_REF] Arbib | Coordinated control programs for movements of the hand. Hand function and the neocortex[END_REF][START_REF] Iberall | Opposition space as a structuring concept for the analysis of skilled hand movements[END_REF]. For instance, a hand holding a cup is seen as the thumb on one side and a virtual finger on the opposite side, that realizes the total wrench due to the four antagonist fingers. Under this formalism, a five-finger grasp is effectively seen as two-finger. The thumb-virtual finger model was used in conjunction with nominal-internal force decompositions on 1D horizontal, vertical and transversal cycles in [START_REF] Gao | Internal forces during object manipulation[END_REF], and to predict normal forces on 2D circular trajectories in [START_REF] Slota | Grip forces during object manipulation: experiment, mathematical model, and validation[END_REF]. In this simplified model, given the object's kinematics, the knowledge of one force fully determines the other through the equations of motion. This greatly conceals the issue of force sharing indeterminacy. since in reality, full-hand forces can compensate each other in an infinity of different distributions that all cause the same motion.

In computer vision and haptics, Mohammadi et al. computed forces between the hand and deformable objects by finite element analysis [START_REF] Mohammadi | Fingertip force estimation via inertial and magnetic sensors in deformable object manipulation[END_REF]. Rogez et al. showed that manipulation forces play a crucial role in hand-object interaction understanding [START_REF] Rogez | Understanding everyday hands in action from rgb-d images[END_REF], and noted the challenge of obtaining the ground-truth contact points and forces humans use instinctively, which we address in our work.

Numerical Techniques

In this section, we discuss the techniques used throughout this thesis for numerical differentiation (Section 1.5.1), physics-based optimization (Section 1.5.2), and time-series modeling (Section 1.5.3).

Numerical Differentiation

Through the Newton-Euler equations, contact forces applied during manipulation determine the resulting kinematics in terms of linear and rotational velocity and acceleration. These quantities are not directly provided by common positional tracking techniques, such as the markerless visual tracking approach of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF], that instead capture linear and rotational positions. Mathematically, velocities and accelerations can directly be computed from the first and second-order derivatives, respectively, of the tracked positions. In practice, this is a delicate process due to tracking noise and sensing uncertainties. In such situations, straightforward finite difference leads to exploding velocities and accelerations. On the other hand, smoothing techniques may attenuate acceleration spikes occuring during manipulation.

A common approach for the numerical differentiation of noisy signals consists in finding a smooth approximation of the original signal. e.g., spline interpolation with a least-square criterion [START_REF] Ibrir | A numerical procedure for filtering and efficient high-order signal differentiation[END_REF]. However, such approximations are constructed based on the observation of the whole signal, or at least a large portion of it. Instead, the well-known Gaussian smoothing in image processing can be implemented as a finite impulse response filter, making it suitable for real-time applications. Still, in the context of motion capture, while the visual acquisitions themselves (e.g., depth sensing) may be subject to Gaussian noise, the errors do not necessarily follow the same statistical properties (e.g., pose estimation errors can accumulate through time, rather than be uncorrelated). From the field of control engineering, Fliess and Sira-Ramírez introduced an algebraic framework for parameter identification in linear systems [START_REF] Fliess | An algebraic framework for linear identification[END_REF]. This framework laid the foundation for the recent theory of model-free control [START_REF] Fliess | Model-free control and intelligent PID controllers: towards a possible trivialization of nonlinear control[END_REF] as well as a new class of derivative estimators for noisy signals, independent of the noise statistical properties, termed algebraic numerical differentiators [START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF].

In the original work of Mboup et al. [START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF], the goal is to identify the n-th order derivative of a noisy signal x. With N ≥ n, we consider its N-th order truncated Taylor expansion:

x N (t) = N ∑ i=0 x (i) (0) t i i! , (1.6)
which is such that d N+1 dt N+1 x N (t) = 0. Moving into the frequency domain, Eq (1.6) becomes:

s N+1 xN (s) = s N+1 N ∑ i=0 x (i) (0) 1 s i+1 = N ∑ i=0 s N-i x (i) (0), = s N x(0) + • • • + s N-n x (n) (0) + • • • + x (N) (0).
(1.7)

The target derivative

x (n) (0) is estimated by considering the terms s N-i x (i) (0) i =n
as undesired perturbations that are to be "annihilated" using a linear differential operator Π and a complex-valued function ρ such that: [START_REF] Riachy | Numerical differentiation on irregular grids[END_REF]. We produced an open-source implementation of these works through the course of this thesis1 .

Π xN = ρ(s)x (n) (0). (1.8)

Physics-Based Optimization

Optimization methods play a key part in many physics-inspired problems. In the context of rigid body dynamics, the Newton-Euler equations dictate that the net force F F F and torque τ τ τ exerted on a body of mass m and inertia tensor J q are linked to its linear acceleration a, rotational velocity ω ω ω, and rotational acceleration α α α by:

F F F = ma τ τ τ = J q • α α α + ω ω ω × (J q • ω ω ω) .
(1.9) Each contact force F c i can be decomposed along normal n i and tangential t i vectors as:

F c i = f i n i + g i t i .
(1.10)

Orienting n i and t i such that f i ≥ 0 and g i ≥ 0 and denoting by µ the friction coefficient at contact i, the Coulomb friction model takes the following simple form:

g i ≤ µ f i . (1.11)
In the case of dynamic friction, i.e., when the relative tangential velocity v i between the two surfaces in contact is non-zero, Eq. (1.11) becomes an equality and t i is parallel to v i .

In the context of dynamics simulation (see Fig. 1.6b), Baraff formulated Eq. (1.9) as equality constraints of a linear program (LP) to compute contact forces preventing interpenetration between rigid bodies [START_REF] Baraff | Analytical methods for dynamic simulation of non-penetrating rigid bodies[END_REF]. In [START_REF] Baraff | Coping with friction for non-penetrating rigid body simulation[END_REF], Eq. (1.11) was incorporated as a linear inequality in the LP to model dynamic friction in conjunction with physics-based simulation. The case of static friction is more complex, since the direction of t i is not known a priori in the 2D tangential plane. Instead, it is described by two orthogonal vectors t x i and t y i , which we depict in Fig. 1.6a :

F c i = f i n i + g i t x i + h i t y i .
(1.12) In that case, the Coulomb friction constraint becomes:

g k t x k + h k t y k 2 ≤ µ k f k , (1.13)
which Baraff addressed using quadratic programming (QP) with a discretized friction cone. Cone constraints such as that of Eq. (1.13) were later addressed without linear approximations by means of second-order cone programming (SOCP) [START_REF] Sousa Lobo | Applications of second-order cone programming[END_REF][START_REF] Boyd | Fast computation of optimal contact forces[END_REF].

Neural Networks for Time Series Modeling

The recent successes of deep learning applications for whole-body control, manipulation and monitoring of human activities [MLA + 15, LLS15, ZZCZ15, KS16], suggest that datadriven approaches can successfully account for model or perception uncertainties while avoiding the need for arbitrary constraints and hand-engineering [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF]. At the heart of these approaches lies the notion of artificial neural networks (ANNs), that take a scalarvalued vector as input and typically passes it through linear transformations and nonlinear activation functions (e.g., hyperbolic tangent). In the context of supervised learning, i.e., when both inputs and expected outputs are available, interconnection weights are updated by backpropagation, i.e., gradient descent with respect to a chosen cost function (e.g., squared prediction error).

In the recent review of [LBH15], LeCun et al. report that the recent advances in parallel programming on GPUs made it easier and faster for researchers to train large neural network models. Deep learning methods thus consist in the multi-layered stacking of simple modules. A popular architecture in image processing is that of the convolutional neural network (CNN) [START_REF] Lecun | Gradientbased learning applied to document recognition[END_REF], designed to process multidimensional data (e.g., 2D images, 3D videos). In this thesis, we capture the sequential nature of contact dynamics using recurrent neural networks (RNN) [START_REF] Elman | Finding structure in time[END_REF]. In essence, RNNs are networks of neurons that maintain an internal state, updated with each input. Graphically, RNNs can be visualized as networks with loops that allow information to persist through time, as illustrated in Fig. 1.7a. Long short-term memory (LSTM) networks [START_REF] Hochreiter | Long short-term memory[END_REF] implement an explicit memory mechanism and have demonstrated better performance at handling long-term dependencies than regular RNNs on multiple applications, such as machine translation [CVMG + 14] or even in conjunction with CNNs for image captioning [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF]. We subsequently use LSTMs to model contact dynamics as a particular type of time series.

Chapter 2

Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces

Introduction

Reliably capturing and reproducing human haptic interaction with surrounding objects by means of a cheap and simple set-up (e.g., a single RGB-D camera) would open considerable possibilities in computer vision, robotics, graphics, and rehabilitation. Computer vision research has resulted in several successful methods for capturing motion information. A challenging question is: to what extent can vision also capture haptic interaction? The latter is key for learning and understanding tasks, such as holding an object, pushing a chair or table, as well as enabling its reproduction from either virtual characters or physical (e.g., robotic) embodiments. Contact forces are usually measured by means of haptic technologies such as force transducers. The main drawback of such technologies is that they are obtrusive. Computer vision techniques would therefore be an ideal alternative to circumvent this issue. Yet, is it possible to estimate forces from visual observation? There is evidence that haptic perception can be induced through illusion and substitution dominated by vision, e.g. [LCK + 00]. We aim at exploring computer vision to infer the forces exerted by humans on surrounding objects. In particular, we consider hand-object grasping and manipulation. The problem is extremely complex. Indeed, establishing that a hand-object contact has occurred is difficult because of occlusions and tracking inaccuracies. Nevertheless, the detection of events like an object being lifted or discontinuities in body motion may provide useful hints towards Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces Figure 2.1: Using a single RGB-D camera, we track markerless hand-object manipulation tasks and estimate with high accuracy contact forces that are applied by human grasping throughout the motion.

disambiguating discrete events. Additionally, even if contact positions can be determined efficiently, the estimation of the applied forces is still challenging because of the inherent multiplicity of solutions. We demonstrate that, by solely using computer vision, it is possible to compute interaction forces occurring in hand-object manipulation scenarios where object properties such as shape, contact friction, mass and inertia are known, along with the geometry of the human hand. First, we monitor both the hand and the object motions by using model-based 3D tracking (other visual tracking techniques can also be used if they meet performance requirements). From the tracking data, we estimate hand-object contact points through proximity detection. Algebraic filtering computes the object's kinematics, i.e. velocity and acceleration. Contact force distributions explaining the kinematic observations are then resolved using conic optimization. When manipulating objects, humans typically apply more (internal) forces than the (nominal) forces required from the Newton-Euler dynamics. Thus, we improve our estimation method by using neural networks to learn the amount and distribution of these internal forces among the fingers in contact. The experimental results obtained on datasets annotated with ground-truth measurements show the potential of the proposed method to infer hand-object contact forces that are both physically realistic and in agreement with the actual forces exerted by humans during grasping. To the best of our knowledge, this is the first time that this problem is addressed and solved based solely on markerless visual observations.

Force Sensing From Vision

We consider a rigid body of mass m and inertia matrix J relative to the center of mass G. For a given element e of the environment (e.g. human hand, table), let µ e denote the corresponding Coulomb friction coefficient between the object and e. In this work, we assume these quantities known, e.g., obtained from the object's CAD model or existing identification techniques [START_REF] Schedlinski | A survey of current inertia parameter identification methods[END_REF]. Interestingly, it has been shown that aspects of such information (e.g., mass distribution) can also be estimated by visual means [START_REF] Mkhitaryan | Visual estimation of object density distribution through observation of its impulse response[END_REF]. We then consider a scenario where the object is grasped and manipulated by a human hand, with possible contacts with the environment. We observe the scene with a single RGB-D camera that we suppose calibrated intrinsically and extrinsically so that the direction of the gravity vector is known. Our goal is to estimate the interaction forces between the object and the user's hand, and between the object and the environment when such contacts occur. We address the problem of force sensing from vision (FSV) in four steps, as follows:

1. We track the object and the hand and perform, for each time step, vision-based proximity or collision detection to identify contacting fingers and corresponding contact points (Section 2.2.1).

2. Let θ θ θ i = (p G , q) be the estimated 6-DoF object pose at instant i, with p G the 3D position of the center of mass and q the object's orientation, encoded by a quaternion. Based on the sequence of pose estimates (θ θ θ i ) i∈[0,N] , we estimate for each frame the body's first and second-order kinematics, i.e. translational (resp. rotational) velocity v i (resp. ω ω ω i ) and acceleration a i (resp. α α α i ) (Section 2.2.2).

3. We compute a (nominal) force distribution explaining the object's state computed at step 2 following the Newton-Euler's laws of motion and Coulomb's friction model, using the contact points identified at step 1 (Section 2.2.4).

4. We learn and reproduce how humans naturally distribute among the fingers in contact (Section 2.2.6).

Each of these subproblems presents a number of challenges. First, the observation of manipulation tasks may be subject to mutual occlusions between the hand and the object. To overcome this issue, we address step 1 by means of model-based tracking as inspired Manipulation Forces by [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF]. Second, the limited camera acquisition frequency along with tracking errors can make the differentiation process of step 2 unstable. We tackle this issue by estimating derivatives using algebraic filtering derived from [START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF]. Algebraic filtering was chosen for the sake of robustness, as it relies on no statistical assumption on the signal's noise. We then address step 3 by computing minimal force closure distributions as solutions of a second-order cone program (SOCP). Finally, step 4 stems from the fact that in contrast with [START_REF] Sun | Predicting fingertip forces by imaging coloration changes in the fingernail and surrounding skin[END_REF] where multiple photodetectors monitor each fingernail's blood flow individually, such microscopic features cannot be observed by a single RGB-D camera observing the whole scene. The object may indeed be grasped with more or less intensity without this being visible at a macroscopic scale. We tackle this statical indeterminacy with machine learning on usual human grasping practices.

Hand-Object Tracking

Our approach requires a good 3D pose estimate of the manipulated object together with that of the user's hand. To achieve this, we rely on a variant of the method proposed in [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] that is tailored to our needs. In [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF], the model-based hand-object 3D tracking is formulated as an optimization problem, which seeks out the 3D object(s) pose and hand configuration that minimizes the discrepancy between hypotheses and actual observations. The optimization problem is solved based on PSO [START_REF] Eberhart | Swarm Intelligence (The Morgan Kaufmann Series in Evolutionary Computation[END_REF].

Since this method estimates the generalized pose of a hand interacting with an object, it is straightforward to compute the 3D positions of the estimated fingertips in relation to the object's surface (i.e., contact points). Still, in our implementation of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF], we have incorporated one important modification. The original 3D hand-object tracking framework provides solutions that are compatible with visual observations and are physically plausible in the sense that the hand and the object do not share the same physical space (i.e., the hand does not penetrate the modeled volume of the object). However, occluded fingers may have different poses that respect the above constraints, making the estimation of contact points an under-constrained problem. To overcome this issue, we assume that contact points do not change significantly when they cannot be observed. Time and space coherency is thus enforced by penalizing solutions in which hidden contact points are far from their last observed position.

Numerical Differentiation for Kinematics

In theory, velocity and acceleration can be estimated by numerical differentiation of poses obtained from tracking. However, this process is highly dependent on two factors: (a) the acquisition frequency of the RGB-D frames, and (b) the quality of the motion tracking. First, even a perfect tracking would result in poor velocity and acceleration estimates if performed over time steps far apart from each other, also depending on the way the hand moves. However, this is not a freely controllable parameter, as most commercial RGB-D cameras offer acquisition frame-rates capped between 30 and 60 fps. We present our results on a 30 fps SoftKinetic DepthSense 325 camera. Second, acceleration profiles occurring in manipulation tasks are naturally spiky (see for example Fig. 2.5). Therefore, numerical differentiation is challenging in that while the number of samples used for each derivative estimate must be sufficient to alleviate tracking errors, it must also be kept minimal to discern the sudden variations that acceleration profiles are subject to.

As an alternative to existing numerical differentiation methods, algebraic parameter estimation approaches [START_REF] Fliess | An algebraic framework for linear identification[END_REF] led to a new class of derivative estimators called algebraic numerical differentiators [START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF]. The tracking errors resulting from the employed modelbased tracking framework seem to follow a Gaussian distribution, yet they are not independent of one another, which rules out the white noise formalism. Subsequently, and in order to keep the kinematics estimation process unbiased by the use of a particular tracking method, we implement the so-called minimal (κ, µ) algebraic numerical differentiators, which do not assume prior knowledge of the signal errors' statistical properties.

From Kinematics to Dynamics

We suppose the manipulated object subject to n d non-contact forces (F d k ) k∈[1,n d ] applied at points (P d k ) k∈{1,...,n d } (e.g., gravitation, electromagnetism). We consider them fully known based on the object's properties. We seek to estimate n c contact forces (F c k ) k∈[1,n c ] applied at contact points with the hand or the environment (P c k ) k∈[1,n c ] that are obtained from tracking (Section 2.2.1). Using the object's kinematics as estimated in Section 2.2.2, its motion is governed by Newton-Euler equations. Therefore, the resulting net force F F F c and torque τ τ τ c due to the contact forces are such that:

F F F c = ma -F F F d τ τ τ c = J q • α α α + ω ω ω × (J q • ω ω ω) -τ τ τ d , (2.1)
with F F F d and τ τ τ d the net force and torque due to non-contact forces, and J q the inertia matrix at orientation q.

The contact forces are subject to friction, which we model using Coulomb's law. Let n k be the unit contact normal oriented inwards the object at contact point P c k . Let then t x k and t y k be two unit vectors orthogonal to each other and to the normal n k , thus defining the tangent Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces plane. Each contact force F c k is decomposed as follows:

F c k = f k n k + g k t x k + h k t y k , (2.2)
With µ k the friction coefficient at P c k , Coulomb's law reads:

g k t x k + h k t y k 2 ≤ µ k f k , (2.3)
which is a strict equality in the case of dynamic friction.

Nominal Forces From Cone Programming

We address the estimation of the minimal contact forces responsible for the observed motion (i.e., nominal forces) as a second-order cone program (SOCP) [LVBL98, BV04, BW07]:

min C (x) = 1 2 x T Px + r T x s.t.        A j x + b j 2 ≤ c T j x + d j , j = 1, . . . , m Ex ≤ f Gx = h.
(2.4) As we track the object and the user's hand, we can determine, at each timeframe, newly established and broken contacts, and also those that remain still and those that slide. Therefore, we are explicitly considering static and kinetic (i.e. dynamic) friction in the constraints formulation. With n c,s and n c,k the respective numbers of friction forces and n c their sum, we construct the optimization vector as follows:

x = ( f 1 , g 1 , h 1 , . . . , f n c,s , g n c,s , h n c,s , f n c,s +1 , f n c,s +2 , . . . , f n c,s +n c,k ) T (2.5)
x is a vector of 3n c,s + n c,k elements. The SOCP formulation in Eq. (2.4) then allows the direct handling of Coulomb static friction as cone inequality constraints by defining, for each contact point j = 1, . . . , n c,s , matrices A j , b j , c j and d j such that:

A j x + b j = g j h j and c T j x + d j = µ j f j . (2.6)
Moreover, having each normal vector n k oriented inwards the object, we formulate n c linear inequality constraints such that f k ≥ 0. This is done by defining E as a n c × (3n c,s + n c,k ) matrix and f as a n c -element vector such that, for each contact point j = 1, . . . , n c,s + n c,k :

E j x = -f j and f j = 0 , (2.7)
with E j and f j the j-th rows of E and f, respectively. Equality constraints ensuring that the resulting contact force distribution explains the observed kinematics stem from Newton-Euler's equations, as combining Eq. (2.1) with contact force expressions from Eq. (2.2) directly yields six linear equations in x. G and h are thus of respective size 6 × (3n c,s + n c,k ) and 6 × 1, with rows 1 to 3 accounting for net force constraints, and rows 4 to 6 for net moment constraints.

As stated earlier in Section 2.2, there exists an infinity of possible force distributions for a given kinematics and set of contact points. We use the (squared) L 2 norm of the contact force distribution (i.e., the sum of squares of the individual components) as an indicator of the intensity of the grasp. We thus complete the SOCP with the following cost function:

C L 2 (x) = ∑ k∈F f 2 k + g 2 k + h 2 k = ∑ k∈F F c k 2 2 , (2.8)
where F is the set of contacting fingers. The objective function C L 2 allows to search for the optimal grasp in the L 2 sense, although other cost functions can be tested. Numerically, we formulate and solve the SOCP using the CVXOPT library for convex optimization [START_REF] Andersen | Cvxopt: A python package for convex optimization[END_REF].

Reproducing Human Grasping Forces

Humans do not manipulate objects using nominal closures (i.e. minimal grasp forces). They tend to "over-grasp" and produce workless internal forces, i.e. firmer grasps than mechanically required. This human grasp property is described by considering finger forces as two sets [KR86, YN91]: nominal forces responsible for the object's motion, and internal forces that secure the object through a firm grip but do not affect the object's kinematics as they cancel each other out [START_REF] Mason | Robot Hands and the Mechanics of Manipulation[END_REF][START_REF] Murray | A Mathematical Introduction to Robotic Manipulation[END_REF]. Studies showed that humans apply internal forces to prevent slip [JW84, FJ02] and control their magnitude to avoid muscle fatigue or damaging fragile objects [START_REF] Stacey | Manipulation of a fragile object[END_REF][START_REF] Park | Optimality versus variability: effect of fatigue in multi-finger redundant tasks[END_REF]. We extend the formulation of the SOCP to address such decompositions and construct a dataset on how humans apply internal forces when manipulating objects, extracted from tactile sensor measurements during real experiments.

Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces Each finger force

F k is decomposed into a nominal component F (n)
k and an internal component

F (i) k : F k = F (n) k + F (i) k with    F (n) k = f (n) k n k + g (n) k t x k + h (n) k t y k F (i) k = f (i) k n k + g (i) k t x k + h (i) k t y k .
(2.9)

Although both forces are decomposed along the same contact frame (n k , t x k , t y k ) as in Eq. (2.2), note that nothing constraints

F (n) k and F (i)
k to be colinear. We subsequently redefine the optimization vector x by considering the nominal and internal components individually rather than their sum as in Eq. (2.5):

x = ( f (n) 1 , g (n) 1 , h (n) 1 , f (i) 1 , g (i) 1 , h (i) 1 , . . . , f (n) n c,s , g (n) n c,s , h (n) n c,s , f (i) n c,s , g (i) n c,s , h (i) n c,s , f n c,s +1 , f n c,s +2 , . . . , f n c,s +n c,k ) T (2.10)
By definition, nominal forces are responsible for the object's motion through the Newton-Euler equations while internal forces are neutral regarding its state of equilibrium:

       ∑ k∈F F (n) k = F F F c , ∑ k∈F --→ GP k × F (n) k = τ τ τ c ∑ k∈F F (i) k = 0, ∑ k∈F --→ GP k × F (i) k = 0.
(2.11) Equation (2.11) provides a new set of constraints that we integrate into the SOCP of Section 2.2.4. Ensuring that the resulting distribution still obeys Coulomb's law of friction, we finally compute the distribution of nominal and internal forces that best match the tactile sensor measurements ( f k ) k∈F , using a new objective function:

C d, f k (x) = ∑ k∈F F (n) k 2 2 + f (n) k + f (i) k -f k 2 .
(2.12)

The reason why we do not directly identify internal forces as the differences between the measurements f k and the minimal forces resulting from the initial SOCP of Section 2.2.4 is that possible sensor measurement errors may lead them not to compensate each other. By integrating their computation into the SOCP, we ensure that the resulting internal forces f

(i) k bridge the gap between f (n)
k and measurements f k without perturbing the object's observed kinematics. We illustrate the decomposition process in Fig. 2.2(a).

Vision

Kinematics, contact points

Tactile Sensors Finger forces

( f k ) k∈F Decompose f k into nominal f (n) k and internal f (i) k components: SOCP: min x C d, f k (x) f (n) k , f (i) k k∈F
Tracking Measurements (a) Vision Kinematics, contact points ANN L 2 , ANN contrib.

Internal forces ( f

(i) k ) k∈F
Reconstruct physically plausible forces: SOCP: min

x C r, f (i) k (x) f (n) k , f (i) k k∈F Tracking Predictions (b)

Learning Internal Force Distributions

Recent studies attempted to build mathematical models correlating grasp forces to kinematic data, yet limited to cyclic movement patterns and two-finger grasps [GLZ05, SLZ11], hence concealing the issue of static indeterminacy, i.e., the fact that in multi-contact, the knowledge of the motion does not suffice to completely characterize the underlying force distribution. In contrast, our approach learns how humans apply internal forces using artificial neural networks (ANN). We first construct an experimental dataset by having human operators manipulate an instrumented box (see Section 2.3) over tasks such as pick-and-place, lift and release, rotations, and unguided compositions of these. Experiments were conducted over a pool of six participants: three female (two right-handed, one left-handed), and three male (all right-handed) operators using their preferred hand on different contact and object mass configurations. Executing 160 manipulation experiments of approximately 10 s duration, we perform motion tracking and record the tactile sensor measurements to compute the best-matching decompositions f

(n) k , f (i) k
following the SOCP of Section 2.2.5.

The next step is to learn the variations of internal forces f

(i)
k with motion and grasping features. We select the learning parameters as those that directly impact the force distri-Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces butions through the Newton-Euler equations. Contact forces vary with the object's mass and acceleration, or more accurately including the contribution of gravity. We thus consider the target net contact force, which can be computed from the object's kinematics and the gravity vector g alone:

F F F c = m • (a -g).
We can consider this dependence as twofold: on the magnitude of F F F c itself, and on the relative orientation of F F F c with the contact normals, as in [START_REF] Gao | Internal forces during object manipulation[END_REF]:

p 1 = F F F c 2 (2.13) p 2,k = n k • u F F F c , with u F F F c = F F F c F F F c 2 .
(2.14)

Similarly, we consider the case of rotational kinematics through the magnitude of the net contact torque τ τ τ c of Eq. (2.1) and the individual torques each finger is able to generate:

p 3 = τ τ τ c 2
(2.15)

p 4,k = --→ GP k × n k • u τ τ τ c , with u τ τ τ c = τ τ τ c τ τ τ c 2 . (2.16)
Finally, we learn internal forces as a function of kinematics and grasp parameters p 1 , (p 2,k ) k∈F , p 3 , (p 4,k ) k∈F using two ANNs: a first network, ANN L 2 , estimates the amount of internal forces applied, quantified as the L 2 norm of their distribution, while a second network, ANN contrib. , jointly estimates the relative contribution of each finger in the grasp's intensity. The outputs of ANN contrib. are percentages constructed as the individual forces normalized with the overall L 2 norm. Note that, as that similar motions can stem from different force distributions, using a single ANN would mean linking similar inputs to highly varying individual forces. Yet, we observed that different grasp intensities still tend to be similarly shared among fingers, hence two ANNs to account for natural intensity variance but consistent decompositions. From the entire collected measurements, we only use as learning data those where the net forces computed from the observed kinematics and by summing up the tactile sensor measurements are within a specified threshold from each other. This allows us to avoid samples where visual tracking or tactile sensor measurements are unreliable, i.e., not compatible with each other with respect to the equations of motion. In our experiments, setting this threshold to 1.5N yields a final dataset of 8200 samples, which we partition into training and validation datasets to construct and assess different ANN configurations by cross-validation. Performing numerical resolution with the neuralnet package for statistical analysis software R [FGS12, R C14], we choose ANN L 2 and ANN contrib. with logistic neurons trained with resilient backpropagation and two hidden layers, with respectively 6 and 8 neurons in the first hidden layer, and 7 and 13 neurons in the second.

Experiments

In order to assess our approach, we perform manipulation experiments on a rectangular cuboid of dimensions 171mm × 111mm × 60mm. The simplified shape of this ground-truth object is chosen to meet sensing instrumentation constraints and offer several grasping possibilities. We instrument the box with two types of sensors. The first is an Xsens MTi-300 attitude and heading reference system (AHRS) motion sensor measuring reference rotational velocities and translational accelerations. Its purpose is to validate the numerical differentiation of tracking data by algebraic filtering, see Section 2.2.2. The second consists of five Honeywell FSG020WNPB piezoresistive one-axis force sensors that can be positioned at different predefined grasp spots on the box. We depict the instrumented device in Fig. 2.3. We evaluate the contact forces estimated from the SOCP in Section 2.2.4 with the force sensor measurements in terms of: (i) normal forces per finger, (ii) resulting net force, and (iii) sum of squares. We summarize the validation protocol in Fig. 2.4. Manipulation Forces Figure 2.4: Validation protocol.

Kinematics From Vision vs AHRS

We assess the validity of our approach by executing motions emphasizing each of the three coordinates of both translation accelerations and rotational velocities, and comparing the kinematics estimated from vision to measurements from the Xsens MTi-300 AHRS. Statistical analysis of the estimation errors shows that algebraic numerical differentiation is well suited for kinematics estimation (see Table 2.1). Though on translational acceleration, its average error is slightly higher than with Gaussian filtering, its variance is also considerably lower. Its performance on rotational kinematics is also the best of all three tested approaches. We illustrate the results of the six-axis experiments in Fig. 2.5.

Nominal Forces From Vision-Based Kinematics

We now validate our vision-based force estimation framework using normal force sensors placed at pre-specified positions over the instrumented box. As a first validation step, contact points obtained from vision were compared to the expected contact points based on the sensors' locations and resulted in estimation errors of mean -1.55mm and standard deviation 6.13mm. Furthermore, we assessed the sensitivity of FSV to these uncertainties by comparing the force distributions obtained using either the contact points from vision or the tactile sensor positions. We found that FSV is relatively robust to such estimation errors, resulting in force uncertainties of mean 0.216N and standard deviation 1.548N. Therefore, we rely solely on vision-based kinematics and contact points for the rest of this work. When performing experiments, we also observed that the force applied by the pinky finger was consistently below the sensitivity threshold of our force sensors, hence we present our results on four-finger experiments. Still, as the force distribution problem introduced in Section 2.2 becomes statically indeterminate from three fingers, using four fingers maintains some high indeterminacy in the force distribution problem, and thus preserves the generality of our results. We represent the force sensor measurements along with FSV's outputs in Fig. 2.6. As mentioned in Section 2.2.4, the comparison of the normal components from vision and from tactile sensors shows that the latter's measurements are overall greater. This illustrates the fact that humans seize objects harder than the required force closure, in contrast with the L 2 -optimal grasp estimated from vision, which is visible in the sum of squares plot. Still, the resulting net forces are matching well, which demonstrates that FSV can successfully capture the object's motion characteristics and compute a force distribution that physically explains the observed kinematics.

Reconstructing Full Contact Force Distributions

By recording new manipulation experiments, we extract the kinematics and grasping parameters described in Section 2.2.6 over time p 1 , (p 2,k ) k∈F , p 3 , (p 4,k ) k∈F and use the trained ANNs to predict the internal forces the human operator most likely apply throughout the experiment, f

(i) k k∈F
. We finally construct the full contact force distributions using a variant of the SOCP described in Section 2.2.5. For this new purpose, we formulate an objective function that aims at reconstructing full contact forces by minimization of, on one hand, the nominal components in isolation, and on the other hand, the discrepancy between optimal internal components and ANN predictions f

(i) k k∈F : C r, f (i) k (x) = ∑ k∈F F (n) k 2 2 + f (i) k -f (i) k 2 .
(2.17)

We illustrate the final estimation process in Fig. 2.2(b). By feeding the ANN internal force predictions into the SOCP, we ensure that the final internal forces

F (i) k k∈F
are not only consistent with natural grasping patterns but also physically correct and do not impact the object's observed kinematics through the resulting net force, as shown in Fig. 2.7.

Robustness Analysis

We investigate the robustness of our approach to features that do not appear in the training dataset. To this end, we train another instance of the ANNs described in Section 2.2.6, not over the entire dataset but on a partial subset relative to a single operator, on a single grasp pose, and a single mass configuration. We then evaluate the resulting ANNs on datasets obtained with another user, another grasp, and/or a 10% mass increase. We report the relative errors with respect to the tactile sensor measurements in Table 2.2 along with reference results from fully-trained ANNs.

First, it appears that ANNs trained over a single operator may be generalized to other users with no significant performance decrease, which suggests that humans tend to apply internal forces following similar patterns. Second, reasonable changes in mass do not seem to significantly impact the estimation accuracy either. This is allowed by the fact that in our problem formulation, mass is not a training variable by itself but is implicitly taken into account through the product F F F c = m • (ag). Under this formalism, manipulating a heavy object with a given kinematics is analogous to manipulating a lighter object with a higher acceleration. Therefore, the ANNs may accomodate mass changes provided that they were trained over a sufficient variety of kinematics. In the end, ANNs seem most sensitive to grasp pose changes. This may be explained by the fact that placing fingers differently may substantially change their synergies. Still, the performance decrease remains reasonable while force distributions, by construction, still explain the observed motion. Eventually, the main sensitivity to grasp poses is comforted by the fact that also changing user and mass does not decrease the estimation accuracy further.

Grasp Recovery by Force Optimization

As an application example, we now show that FSV can be used, along with grasp taxonomies, as an implicit force model to reconstruct physically realistic manipulation sequences from possibly incomplete visual observation and inaccurate visual tracking pose estimates.

Initializing Reference Grasps

Manipulating an object with static contact points, we monitor the scene using a single RGB-D sensor and track the object and the hand jointly. Two common issues commonly arose during our experiments. First, tracking a manipulation scene sometimes led to mutual confusion, i.e., the object was mistaken for the hand, or conversely. Second, in some situations, selfocclusions produced physical incoherent hand-object poses despite visual consistency from the camera's point-of-view. We illustrate such mis-tracking examples in Fig. 2.8. In order to identify the real grasp being applied, we propose to optimize the force distribution in the vicinity of the hand pose estimate from tracking, taken as an initial guess rather than an absolute reference.

As described in Section 1.3.1, we represent the hand pose as a 27-parameter vector H = θ θ θ palm , θ θ θ F , with θ θ θ palm a 7-parameter vector encoding the palm pose (3D position and quaternion) and θ θ θ F a 20-parameter vector encoding the finger poses (4 joint angles for each of the five fingers). With no prior knowledge on the instant the tracker loses the target, we initialize the pose search when contact occurs. This instant may be accurately obtained from the tracking of the object alone, i.e., without relying on the accuracy of the hand tracking and collision detection. Indeed, from the perspective of physics-based optimization, it can be implicitly defined as the moment at which the observed object kinematics cannot be explained by contacts with the environment alone, i.e., when the SOCP has no solution. We denote by H = θ θ θ palm , θ θ θ F such an initial hand pose from tracking.

The grasp taxonomy of [START_REF] Feix | A comprehensive grasp taxonomy[END_REF] describes a set of 17 grasp poses categorized by function and geometry (e.g., power grasp, large diameter). From it, we select a subset P of N P grasp poses suitable to manipulate the object. In our experiments, we chose three grasps compatible with the instrumented device, e.g., firmly securing it from opposite sides or dexterously manipulating it from the corner, as illustrated in Fig. 2.9. We characterize each reference pose of the taxonomy P ∈ P by their finger poses θ θ θ F ,P . We finally initialise our search space with reference hand poses H P specified as the union of the palm pose from tracking with finger configurations from the grasp taxonomy:

∀P ∈ P, H P = θ θ θ palm , θ θ θ F ,P (2.18)

We thus define an initial set of primitive hand poses H P P∈P .

Generating New Grasp Poses

With σ σ σ a 27-element vector of standard deviations for the hand pose parameters, we now construct new grasp candidates H P through Gaussian random sampling N (•, σ σ σ ) in the vicinity of each primitive hand pose H P :

∀P ∈ P, H P = N (H P , σ σ σ ) (2.19)
For each sampled grasp candidate H P , we compute the 3D hand pose by forward kinematics.

We then check for interpenetrations between the hand and the object using the SWIFT++ library for collision detection [START_REF] Ehmann | Accurate and fast proximity queries between polyhedra using convex surface decomposition[END_REF]. For each primitive composing the hand model, we consider that a contact occurs when it is within a chosen threshold to the object (in our Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces Figure 2.9: Reference grasps from left to right: large diameter, precision sphere and tripod.

experiments: 5 mm). If, for any primitive, penetration exceeds this threshold, the current grasp pose candidate H P is discarded from future consideration and a new one is sampled following Eq. (2.19).

Next, we ensure that the resulting contact points allow the manipulation of the object following the tracked object trajectory. We do so by formulating the SOCP of Section 2.2.4 with the contact locations estimated from collision detection and ensuring that, for each time step, with contacts fixed at the estimated locations, there exists a nominal force distribution

F (n) k k∈F
explaining the observed object kinematics through the equations of motion. If the grasp is not physically able to cause the desired motion, the grasp pose candidate is discarded. If it is, we predict the internal forces F (i) k k∈F humans are most likely to apply throughout the motion with the neural networks of Section 2.2.6. Finally, we denote by (F k ) k∈F the complete force distribution obtained by combining nominal and internal components in the FSV framework. Finally, we quantify the grasp intensity throughout the manipulation sequence by computing the L 2 norm of the total distribution, averaged over the duration of the experiment (N S samples):

w H P = 1 N S N S ∑ j=1 ∑ k∈F F k 2 2 .
(2.20)

In addition, we introduce an indicator d H on the discrepancy between the initial hand pose from tracking H and the grasp candidate H P , e.g., the L 2 distance :

d H H P = H P -H 2 .
(2.21)

Finally, we consider a weighted cost function C H combining w and d H :

C H H P = w H P + α • d H H P . (2.22)
The weight α can be tuned to favor either the grasping energy along the observed trajectory or the fidelity with respect to the pose estimate from tracking. We subsequently present results for different values of α.

Results

In order to assess the validity of our approach, we consider a manipulation experiment involving a box being rapidly moved in arbitrary directions, both in translation and in rotation. Due to mutual occlusions between the hand and the object, at the beginning of the sequence, the pose estimates from model-based tracking match the observable features but are not physically plausible (see Fig. 2.8(a-c)). Throughout the rapid motion, the accumulation of tracking uncertainties results in completely wrong pose estimates (see Fig. 2.8(d-e)).

Using the numerical differentiation framework introduced in Section 2.2.2, we rely on the good estimation of the box pose to compute its kinematics over time. As described in Section 2.4.1, we combine grasp taxonomies and tracking data to initialize the search space. Following Section 2.4.2, we generate pose hypotheses by Gaussian sampling in the vicinity of the reference hand poses and verify their feasibility regarding interpenetrations and compatibility with the motion. In our experiments, we sampled 792 grasp candidates that were valid with respect to collision constraints in 10 min of computation time (Intel i7-4700MQ processor, single-threaded implementation). Rather than sampling the configuration space and checking for collisions, a possibly more efficient approach could be to sample the contact space, as explored in [WMZ + 13]. We finally fed each grasp candidate into the FSV framework, yielding 493 grasp poses that were able to achieve the observed motion.

For each valid pose, we computed the grasp intensity w and discrepancy d H costs of Eqs. (2.20) and (2.21). Finally, we combined these values into the weighted cost C H of Eq. (2.22) for different values of the normalization factor α. We depict the resulting optimal grasps in Fig. 2.10. As expected, minimizing the grasp intensity in priority (i.e., α small) yields grasps that are possibly far from the initial guess. Convertly, favorizing poses that are Figure 2.10: Each column represents the optimal solution yielded by our algorithm for increasing values of parameter α. The two first rows show the grasp candidate at the beginning of the experiment (front and back views). The third row corresponds to the same instant as the frame depicted in Fig. 2.8a. We can thus reconstruct various physically plausible grasps, that become closer to the initial observations as we increase α.

close to the tracking hypothesis (i.e., α big) allows the recovery of physically realistic grasps that match the actual observations despite inaccurate tracking data.

Summary and Discussion

Our work establishes that a single RGB-D camera can be used to capture interaction forces occurring in rigid object manipulation by a human hand without the need for visual markers or dedicated force sensing devices. Force sensing from vision is a novel and important contribution since it circumvents the intrusive instrumentation of object, environment and hands. Its exploitation can expand to the robotics field for daily on-line human activities monitoring, serving various purposes such as imitation learning.

Our method is validated with several experiments based on ground truth data and is able to estimate fairly accurately the force distributions applied during actual manipulation experiments. Although we confirmed that tracking noise is well mitigated by algebraic filtering, which produces truthful pose derivative estimates, guessing the hand-object contact points under strong occlusions remains a challenging, open problem in computer vision. We achieved this by using a state-of-the-art model-based tracking method under the somewhat practical assumption that occluded fingers remain at their last observed position until they are visible again. While this assumption is fairly valid in numerous interesting cases, it is not true when considering tasks such as dexterous manipulation with finger repositioning or sliding. Still, this limitation does not call into question the force estimation framework per se, and could be alleviated by extending the markerless tracking method to multi-camera inputs, which would remain non-intrusive and keep an edge over tactile sensors regarding usability and cost.

With respect to computational performance, SOCP and internal force predictions are performed in real-time, and only hand-object tracking is computationally expensive. Given the recent developments on GPGPU implementations of hand-object tracking [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF], our framework could be employed in real-time applications. This, combined with our reliance on a single camera, makes FSV suitable for daily observation and learning. Still, our approach is generic enough to accommodate any advance to the topic of 3D hand tracking and could be seamlessly extended to other methods, for instance when non real-time performance and a heavier setup are possible. Conversely, our framework could also be used as an implicit force model for physics-based tracking and motion editing, as human-like forces could augment the pose search with biomechanical considerations such as muscle fatigue or energy expenditure. We demonstrated such a use case to reconstruct physically plausible grasps in the presence of strong occlusions, which could also be incorporated in the tracking itself for interactive, physics-based correction.

Towards estimating contact forces from vision, we tackled the issue of static indeterminacy by applying machine learning techniques to internal forces. Rather than predicting new force distributions based on past observations, an alternative approach would be to formulate the evolution of the full contact forces following various objects and grasp taxonomies as an inverse optimal control problem. If invariants are found, they could be used to refine the cost function, which could result in more reliable contact forces than the nominal distributions computed by minimization of the grasp's L 2 -norm. Extending the ground truth force measurement setup with embedded three-axis or force-torque miniature sensors would also benefit both learning and optimal control approaches. Further work could also address the case of surface contact models in place of point contacts (as the fingertip is deforming), namely for dexterous manipulations, or make use of synergy properties of the hand for bimanual tasks. Finally, combining our approach with visual SLAM or automated camera calibration methods would allow it to be deployed in unknown, varying environments, e.g. on mobile robots.

Chapter 3

Hand-Object Contact Force Estimation From Markerless Visual Tracking

Introduction

Touch (i.e. physical contact) is of fundamental importance in the way we naturally interact with objects and in our perception of their physical and functional properties. Human manipulation remains little understood at the level of the underlying interaction forces, which are traditionally measured using force transducers. The latter are costly, cumbersome, and intrusive on both the object and the human haptic sense. Moreover, if mounted onto the hand, they often hinder or reduce the range of possible motions. Recent work has showed how the latter could be inferred from vision [GKD09, ZZCZ15, YYFA16]. Moreover advances in markerless visual tracking opened up the possibility for monitoring hand-object motions in a non-intrusive fashion. Computer vision techniques would therefore be an ideal substitute for current force sensing technologies. This is an extremely challenging perspective. Indeed, tracking a hand interacting with an object is difficult due to strong mutual occlusions. Moreover, even when a manipulation trajectory is fully known, the force estimation problem is ill-posed or indeterminate in multi-contact. Indeed, given the physical properties of the object, there generally exists an infinity of force distributions resulting in the same motion (e.g. using different grip strengthsi.e. internal workless forces). While it is possible to compute physically plausible force distributions, capturing the real forces being applied is an open problem explored in multiple fields (see Section 1.4). In particular, kinesiology research has resulted in successful attempts at modeling grip forces by inverse optimization, e.g., during static prehension [START_REF] Niu | Reconstruction of the unknown optimization cost functions from experimental recordings during static multi-finger prehension[END_REF] or two-finger circular motion [START_REF] Slota | Grip forces during object manipulation: experiment, mathematical model, and validation[END_REF]. Although these scenarios are of limited scope, this suggests that it may be possible to construct a general model on human grasping, provided a rich dataset on manipulation kinodynamics (motion and forces).

In our work, we show that physics-based optimization can be used in conjunction with learning to capture manipulation forces from non-intrusive visual observation, on a setup as simple as a single RGB-D camera.

• We construct the first large-scale dataset on human manipulation kinodynamics, containing 3.2 hours of high-frequency measurements for 193 different object-grasp configurations (Section 3.2).

• We propose a force estimation framework that relies simultaneously on a recurrent neural network to predict forces that are consistent with the way humans naturally manipulate objects, and on a second-order cone program guaranteeing the physical correctness of the final force distribution (Section 3.3).

• We thoroughly validate our approach on ground-truth measurements (Section 3.4) and show that it can seamlessly be extended to visual tracking (Section 3.5).

Due to instrumentation constraints, our dataset is dedicated to constant contacts on prismatic grasps, i.e., with the thumb in direct opposition to the antagonist fingers. We discuss these limitations and show that the dual optimization-learning framework can still address scenarios beyond the focus of our study (Section 3.6). Finally, we discuss thoroughly the current limitations, possible extensions and applications of our work (Section 3.7). A preliminary version of this research, focusing on estimating normal forces from vision, was presented in Chapter 2 and appeared in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]. Our current study extends the latter idea and includes: an improved formulation of the optimization and learning models accounting for individual normal and tangential components, time-coherent manipulation forces, as well as algorithmic descriptions and extensive validation experiments that have not been presented before. To foster the research in this new topic, we make the manipulation kinodynamics dataset publicly available1 .

Manipulation Kinodynamics Dataset

Over the last years, the release of public datasets has massively benefitted the research in fields related to this work, such as object recognition and scene understanding [KAJS11, LBF14], whole-body and hand tracking [SPSS12, TSLP14], and robotic grasping [SDN08, ÇWS + 15]. In contrast, datasets viewing human manipulation not only from the angle of vision but also Additionally, we construct devices based on everyday objects, instrumented so as to allow intuitive interactions (d-f).

of touch have been more scarce so far. A notable example is the interaction capture technique of [START_REF] Kry | Interaction capture and synthesis[END_REF] for joint compliance estimation in graphics and synthesis of interaction animations. In this section, we introduce a new, extensive dataset dedicated to the kinodynamics of human manipulation.

Experimental Setup

Our objective is to construct a general force model capable of capturing the whole range of manipulation forces that are commonly applied during daily activities. The manipulation kinodynamics dataset was thus collected for diversity and genericity, regarding both the objects being manipulated and the way they are grasped. While using real objects may initially seem ideal, instrumenting them with force and motion sensors is impractical and makes it difficult and lengthy to collect a diverse dataset. Additionally, physical properties of arbitrary objects (e.g., inertia matrices) are seldom publicly available and must therefore be manually identified [START_REF] Schedlinski | A survey of current inertia parameter identification methods[END_REF][START_REF] Kiran | Computing the physical parameters of rigid-body motion from video[END_REF]. Finally, the instrumentation may result in measured forces that substantially differ from those that would have been applied on the original objects.

We address these caveats with dedicated instrumented devices, pictured in Fig. 3.1, composed of two symmetric parts for the thumb and the antagonist fingers. Each half consists of a base serving as support for an attitude and heading reference system (AHRS, Xsens MTi-300), and a sensor plate on which 3D precision force transducers (Tec Gihan USL06-H5-50N) can be positioned by 8 mm steps on the surface. Thickness layers can be inserted in between to increase the grasp width by 5 mm increments, bringing the total grasp width range between 46 mm and 86 mm. The force transducers are fitted with support caps of different surface textures: PET, sand paper of grit 40 (coarse), 150 (fine) and 320 (extra fine). The mass distribution can be adjusted with balancing weights inside and on the surface of the instrumented device. We 3D-print four sets of instrumented modules, with sensor plates of dimensions 80×152, 56×152, 80×96 and 56×96 mm 2 . This setup allows the efficient collection of force and kinematics measurements under diverse grasp poses, friction conditions and mass distributions, obtained from the CAD models of the individual components.

Still, instrumentation constraints make it difficult to collect ground-truth measurements for arbitrary object shapes and grasps [FRS + 16], which we consider essential to also prove the validity of any force prediction approach. Indeed, it would require a significantly heavier experimental setup to allow the individual adjustment of degrees of freedom such as local curvatures and finger repositioning. Note that these limits only apply to the dataset and not to the force estimation framework itself, which can still produce physically correct force distributions for such scenarios, although possibly different from the real forces being applied. We discuss these limitations and apply our algorithm to manipulation scenarios beyond the explicit scope of our study in Section 3.6.

The Dataset

Eleven right-handed volunteers, three females and eight males, took part as subjects in our experiments. Each subject was instructed to perform series of up to eight manipulation sequences as follows. For each series, the subject is given an instrumented box of randomly picked shape, thickness and surface texture as described in Section 3.2.1. The initial object configuration is completed by mounting the AHRS either at the top or at the bottom of the instrumented device, and at random with an additional 400 g mass inside. The subject is then instructed to perform manipulation tasks on eight variations of the initial configuration. Before each trial, the force transducers are placed on the box according to the subject's preferred grasp pose and their signals are adjusted following the manufacturer's recommended acquisition and calibration procedure. Each trial consists in the subject grasping the object and manipulating it for approximately 60 s. Every 10 s, in order to ensure the diversity of the kinematics and forces present in the final dataset, the subject is given randomly picked instructions on speed, direction and task (e.g., slow forward pouring motion, fast left and right oscillations). After each trial, a 50 g balancing weight is attached to a randomly picked side, excluding sensor plates. Throughout the eight trials, we measure the effect of mass variations between 0 g and 350 g or 400 g and 750 g with the additional internal mass, arranged differently across series. Finally, the subject can interrupt the series whenever the object becomes uncomfortable to manipulate.

Overall, we collect motion and force measurements for 3.2 hours of manipulation experiments under 193 conditions of motion, friction, mass distribution and grasp. For each experiment, we provide: the global orientation q, rotational velocity ω ω ω and translational acceleration a measured by the AHRS at 400 Hz; 3D force measurements expressed in the reference frame of the object R obj. , subsampled from 500 Hz to 400 Hz to match the AHRS; the physical properties of the object: mass m, inertia matrix J about the center of mass G; and the grasp parameters: for each finger k ∈ F , the friction coefficient µ k at contact point P c k , and R k = n k , t x k , t y k a local right-handed reference frame with n k the normal to the surface oriented from the finger to the object. Friction coefficients are estimated by instructing the subjects to press and pull the force transducers until slipping and computing the maximum ratio between tangential and normal forces through the Coulomb model:

g k t x k + h k t y k 2 ≤ µ k f k , (3.1)
with ( f k , g k , h k ) the local decomposition of contact force F k :

F k = f k n k + g k t x k + h k t y k . (3.2)

Equations of Motion and Synchronization

Let F F F c and τ τ τ c be the net force and torque due to individual contact forces, and F F F d and τ τ τ d be the net force and torque due to non-contact forces (e.g., gravitation); the Newton-Euler equations of motion at the center of mass are:

F F F c = ma -F F F d τ τ τ c = J q • α α α + ω ω ω × (J q • ω ω ω) -τ τ τ d , (3.3)
with J q the inertia matrix at orientation q and α α α the rotational acceleration of the object, obtained by numerical differentiation of the AHRS rotational velocity measurements ω ω ω. The left hand side elements correspond to the contributions of the force transducer measurements while the right hand side elements can be computed from the object properties and AHRS kinematics measurements. This allows us to synchronize the kinematic and dynamic measurements temporally while also accounting for sensor uncertainties. First, the two signals can be synchronized temporally by computing the cross-correlation between the sequences of net forces obtained either from the AHRS or from the force transducers. Second, both the AHRS and the force transducers are subject to measurement errors, resulting in discrepancies in the resulting net force and torque. The specified AHRS maximum acceleration measurement error is of ±0.3 m • s -2 . For an object of mass 500 g, this amounts to net force errors up to ±0.15 N. In contrast, non-linearity and hysteresis can cause measurement errors up to ±1 N per force transducer, i.e. ±5 N at most on the net force. In practice, the average net force discrepancy between AHRS and force transducers throughout the whole dataset is 0.33 N. For each experiment, we compute the average net force ∆F F F c and torque ∆τ τ τ c discrepancies between AHRS and force transducers. We align their values by computing the minimal offsets (∆F k ) k∈F that result in ∆F F F c and ∆τ τ τ c :

min C F F F c + C τ τ τ c + C var , (3.4)
with force-torque discrepancy and variation cost functions:

                     C F F F c ((∆F k ) k ) = ∆F F F c -∑ k∈F [∆F k ] 2 2 C τ τ τ c ((∆F k ) k ) = ∆τ τ τ c -∑ k∈F --→ GP k × ∆F k 2 2 C var ((∆F k ) k ) = ∑ k∈F ∆F k 2 2 (3.5)
In practice, it is preferrable to normalize C F F F c and C τ τ τ c , e.g., with the initial discrepancies ∆F F F c and ∆τ τ τ c respectively. We solve the optimization problem using sequential least squares programming and correct the force transducer measurements with the resulting offsets.

Force Model

Based on the Newton-Euler equations, the net contact force F F F c and torque τ τ τ c are completely determined by the object's motion and physical properties. However, given F F F c and τ τ τ c can generally be achieved by an infinity of different force distributions. Our force model addresses these two aspects by combining physics-based optimization and learning to reconstruct force distributions that are both physically plausible and similar to actual human grasping.

Physics-Based Optimization for Manipulation

In this section, we formulate the Newton-Euler equations and Coulomb model as constraints of an optimization problem allowing the extraction of force distributions compatible with a given motion. We integrate these constraints in a second-order cone program (SOCP) [LVBL98, BV04, BW07] of the form:

min C (x) = 1 2 x T Px + r T x s.t.        A j x + b j 2 ≤ c T j x + d j , j = 1, . . . , m Ex ≤ f Gx = h. (3.6)
We express conditions of physical plausibility using the local decompositions of Eq. (3.2) as 15 optimization parameters:

x = ( f 1 , g 1 , h 1 , . . . , f 5 , g 5 , h 5 ) T (3.7)
Positivity. Recall that for each finger k, we choose the contact normal n k oriented inwards the object. With this convention, the normal components f k are non-negative:

f k ≥ 0, k = 1, . . . , 5. (3.8)
This can be rewritten in Eq. (3.6) with linear inequality matrices E and f of respective sizes 5×15 and 5×1, with:

E(i, j) = -1 if j = 3(i -1) + 1 0 else f(i, 1) = 0.
(3.9)

Friction. The Coulomb model of Eq. (3.1) can be written as five cone constraints, i.e., one per finger. For each finger k, the cone constraint matrices A k , b k , c k , d k , are of respective sizes 2×15, 2×1, 15×1 and 1×1, such that:

A k x + b k = g k h k and c T k x + d k = (µ k f k ) .
(3.10)

Hand-Object Contact Force Estimation From Markerless Visual Tracking

Their elements are defined as follows:

A k (i, j) = 1 if j = 3(k -1) + 1 + i 0 otherwise b k (i, 1) = 0 c k (i, 1) = µ k if i = 3(k -1) + 1 0 otherwise d k (1, 1) = 0.
(3.11)

Equations of motion. Recall from Eq. (3.3) that the net contact force F F F c and torque τ τ τ c can be determined from kinematic quantities only. The individual finger forces are such that:

       F F F c = ∑ k∈F F k τ τ τ c = ∑ k∈F --→ GP k × F k .
(3.12)

We express the Newton-Euler equations in the global reference frame

R global = (v 1 , v 2 , v 3 ).
The equality constraint matrices G and h are of respective sizes 6×15 and 6×1 with: ∀i = 1, . . . , 3; ∀ j = 1, . . . , 15; ∀k = 1, . . . , 5;

G(i, j) =              n k • v i if j = 3(k -1) + 1 t x k • v i if j = 3(k -1) + 2 t y k • v i if j = 3(k -1) + 3 0 otherwise h(i, 1) = F F F c • v i G(i + 3, j) =                  --→ GP k × n k • v i if j = 3(k-1)+1 --→ GP k × t x k • v i if j = 3(k-1)+2 --→ GP k × t y k • v i if j = 3(k-1)+3 0 otherwise h(i + 3, 1) = τ τ τ c • v i (3.13)

Force Model

61

Cost function. Physically plausible force distributions can be computed with a cost function depending only on the optimization variables, e.g. minimal (squared) L 2 norm [PKQA15b]:

C L 2 (x) = ∑ k∈F f k 2 +g k 2 +h k 2 = ∑ k∈F F k 2 2 . (3.14)
Yet, the resulting forces can significantly differ from those humans really apply (see Fig. 3.2). Instead, we consider a cost minimizing the discrepancy with given target forces F k :

C F k (x) = ∑ k∈F F k -F k 2 2 (3.15)
In the following, we use C F k to correct force transducer measurements and neural network prediction uncertainties.

Learning Features

The criteria that are optimized by the central nervous system in hand-object manipulation are still unknown (see Section 1.4.3). A major obstacle to their identification is a dependency on musculoskeletal parameters that can be difficult to measure precisely [START_REF] Erdemir | Model-based estimation of muscle forces exerted during movements[END_REF]. Rather than explicitly considering such low-level parameters, the force model we propose in this work relies on an artificial neural network that predicts manipulation forces from high-level kinematic features. Based on the dataset presented in Section 3.2, we group the available parameters into three categories:

• Object and grasp parameters: location of the center of mass G in R obj. , mass m, inertia matrix J, contact point locations P k and friction coefficients µ k .

• Kinematic parameters: appearing in Eq. (3.3) are the object's orientation q in R global , rotational velocity ω ω ω, rotational acceleration α α α and translational acceleration a. The quantities q, ω ω ω, a are directly measured by the AHRS. α α α is obtained by simple numerical differentiation of ω ω ω. Alternatively to the AHRS, these kinematic parameters can also be estimated from visual tracking, through double differentiation of the object's pose through time (see Section 3.5.2).

• Force transducer measurements F k .

To alleviate sensing uncertainties, we extract physically plausible force distributions F k in the vicinity of the possibly inaccurate measurements F k , as depicted in Fig. 3.3. The objective is then to learn the extracted force distributions F k based on input parameters that depend only on the grasp, the object and its kinematics. We select these input features based on their contribution to the Newton-Euler equations of motion. A first approach could be to take the raw parameters listed above. However, their influence is often determined not individually but rather in interaction with other parameters. From Eq. (3.12), the positions of the center of mass G and contact points P k are meaningful not on their own but in relation to each other as --→ GP k . Similarly, from Eq. (3.3), we summarize the contributions of m, a, J, q, ω ω ω, α α α into the target net contact force F F F c and torque τ τ τ c .

Recall that F F F c and τ τ τ c are expressed in R global . Since the dataset focuses on static grasps, for each experiment, the contact points are constant in any frame attached to the object. We account for translational and rotational invariances by projecting F F F c , τ τ τ c and --→ GP k on R obj. . Thus, the input features stemming from the Newton-Euler equations are:

∀(k, v) ∈ F × R obj. ,        p F F F c v = F F F c • v p τ τ τ c v = τ τ τ c • v p P k v = --→ GP k • v . (3.16)
In addition, we consider the average friction coefficient: We regroup these parameters, derived from the grasp-object properties and kinematics, into a 22-element vector K:

p µ = µ k k∈F (3.17) Hand-Object Contact Force Estimation From Markerless Visual Tracking KDN-FH-F Recurrent Neural Network Linear Layer K i D i-1 D i (a) KDN-FH-F: full hand forces. KDN-VF-F KDN-VF-F (th.-VF) RNN LL K (th.-VF) i D (th.-VF) i-1 D (th.) i KDN-VF-F (ant.) RNN LL K (ant.) i D (ant.) i-1 D (ant.) i K i D i-1 D i (b) KDN-VF-F: two-stage thumb-virtual finger network.
K = p F F F c v , p τ τ τ c v , p P k v , p µ (k,v)∈F ×R obj.
(3.18)

Similarly, we denote by D the 15-element vector of the force distribution expressed in the local frame:

D = (F k • v) (k,v)∈F ×R obj. (3.19)
Note that attaching the frame to a chosen finger also helps preserve invariances througout objects and experiments. Using the thumb contact space R th. = t x 0 , t y 0 , n 0 with t y 0 towards the palm, all four antagonist fingers share the same coordinate along n 0 , hence reducing K to 19 elements.

Neural Network Modelling

Given an object-grasp configuration, the goal of our work is to obtain an accurate estimate of the force distribution applied to achieve an observed motion, e.g. by reconstructing a force distribution function F such that:

D = F(K) (3.20)
In [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF], we approximated such a function with an MLP learning internal forces. Yet, our previous formulation has two important limitations:

• Similar tasks can be achieved with different force distributions, i.e., multiple values of D can be associated to the same value of K. As such, different distributions would tend to be averaged albeit equally valid.

• In Eq. (3.20), consecutive force distributions are independent through time. Instead, since contact is never broken, we should expect that the force distribution D i at timestamp i depends not only on the corresponding task parameters K i but also on the past.

Therefore, we adopt the following alternative formulation:

D i = F K i , D i-1 , K j , D j-1 j=1,i-1 (3.21)
Through the dependency on past kinodynamics, the first limitation is also mitigated since forces are distinguished based on K i trajectories rather than single samples.

We capture the sequential nature of manipulation kinodynamics using recurrent neural networks (RNN) [START_REF] Elman | Finding structure in time[END_REF], with long short term memory (LSTM) neurons [START_REF] Hochreiter | Long short-term memory[END_REF] that allow for better learning of long-term dependencies. In this work, we investigate four kinodynamics network (KDN) architectures. The first model we propose, KDN-FH-F, directly predicts full hand forces D i from the current kinematics K i and previous distribution D i-1 using a single RNN:

D i = KDN-FH-F(K i , D i-1 ). (3.22)
Alternatively, we propose a two-stage network inspired by the virtual finger model, KDN-VF-F. A first RNN estimates thumb forces D (th.) i based on parameters reducing the full grasp to a thumb and virtual finger:

D (th.) i = KDN-VF-F (th.-VF) K (th.-VF) i , D (th.) i-1 .
(3.23) 66

Hand-Object Contact Force Estimation From Markerless Visual Tracking

We associate the virtual finger with the centroid of the antagonist fingers F ant. and their average friction coefficient:

K (th.-VF) i = p F F F c v , p τ τ τ c v , p P th.
v , p µ th. , p P ant. v , p µ ant.

v∈R th.

with

   p P ant. v = p P k v k∈F ant.
p µ ant. = µ k k∈F ant.

(3.24)

We compute the total wrench due to the antagonist fingers based on the contribution of the estimated thumb force F th. :

∀v ∈ R th. ,    p F F F ant. v = (F F F c -F th. ) • v p τ τ τ ant. v = τ τ τ c - ---→ GP th. × F th. • v (3.25)
The second stage of the network learns the resulting distribution D (ant.) i over the antagonist fingers:

D (ant.) i = KDN-VF-F (ant.) K (ant.) i , D (ant.) i-1 with K (ant.) i = p F F F ant. v , p τ τ τ ant. v , p P k v , p µ ant.
(k,v)∈F ant. ×R th.

(3.26)

We depict KDN-FH-F and KDN-VF-F in Fig. 3.4. In order to further address the fact that the same motion can be due to different yet equally valid force distributions, we introduce alternative versions of KDN-FH-F and KDN-VF-F that associate current kinematics K i and past forces D i-1 to force variations ∆D i . In doing so, we explicitly associate the same output to two sequences that differ by a constant internal force distribution. We denote these alternative architectures by KDN-FH-∆ and KDN-VF-∆. Full manipulation forces are then reconstructed by sequentially adding predicted force variations. As such, these architectures are prone to drift and may require additional control.

Experiments

We train the four architectures KDN-FH-F, KDN-FH-∆, KDN-VF-F, KDN-VF-∆ on the manipulation kinodynamics dataset of Section 3.2. Note that its sampling rate (400 Hz) far exceeds the frame rate of off-the-shelf RGB-D sensors such as Microsoft Kinect (30 fps) and Asus Xtion (60 fps). In order to be compatible with vision-based kinematics (Section 3.5), we down-sample the dataset to 60 Hz and split it for training (60 %), validation (20 %) and Object-grasp param., kinematics (i) Object-grasp param., kinematics (i)

KDN K i D (raw) i-1 (i -1) ) (i + 1) D (raw) i K i+1 KDN Raw force predictions
KDN K i D (CL) i-1 SOCP (i -1) ) (i + 1) SOCP D (raw) i (CL) i K i+1 KDN (

Force Reconstruction Model

From Eq. (3.21), each force distribution D i is computed from the corresponding kinematics K i and the distribution at the previous time step D i-1 . Due to this sequential process, the predicted forces may drift away from the transducer measurements throughout the experiment. We assess the influence of the experiment duration in Section 3.4.2. Similarly, the predicted sequence also depends on the choice of the initial force distribution D 0 , which we address in Section 3.4.3. In this section, we discuss the reconstruction of physically plausible manipulation forces from KDN predictions and present our results on full-length experiments with ground-truth initialization. Manipulation forces are obtained by projecting the components of D i onto the local reference frame following Eq. (3.19). Since the Newton-Euler and Coulomb laws are not explicitly enforced by the RNNs, the raw predictions are not guaranteed to result in the observed motion. We depict the open-loop prediction process in Fig. 3.5a. Using the SOCP described in Fig. 3.3 with the KDN outputs instead of the force transducer measurements, the sequence of raw predictions can be post-processed to yield physically plausible force distributions in their vicinity. Another important point is that the training sequences are physically coherent. Thus, repeatedly feeding incompatible kinematics and forces into the KDN may result in growing prediction errors. We tackle this issue by integrating the SOCP in closed-loop with the KDN such that force predictions are consistently corrected between time steps. We depict the closed-loop prediction process in Fig. 3.5b. We compute the estimation errors (average and standard deviation) for the four network architectures using open-loop prediction, offline post-processing or closed-loop prediction and report the results in Table 3.1. In general, post-processing and closed-loop prediction perform better than open-loop prediction. This is especially the case for the networks estimating force variations ∆D i , as these tend to be rather unstable and prone to drift. For instance, in Fig. 3.6, the open-loop predictions rapidly drift away from the net force and torque producing the target kinematics. Additionally, the individual normal forces become negative, which would mean that fingertips pull rather than press on the contact surface. Offline postprocessing looks for physically valid forces in the vicinity of negative raw predictions, finally yielding distributions of minimal norm. In contrast, closed-loop prediction can help the network recover from incorrect predictions and maintain human-like grasping forces. Overall, the networks predicting force distributions generally perform better than those estimating force variations. For those, post-processing does not appear to significantly improve the open-loop estimations, which shows that these RNNs are rather successful at capturing the relationship between kinematics and underlying forces. Finally, the better accuracy of KDN-VF-F indicates that the virtual finger model can be a useful tool to decouple the static indeterminacy stemming from the thumb and antagonist fingers. Still, the two-stage architecture makes KDN-VF-∆ more prone to drift since thumb force predictions cannot be corrected alone before computing the antagonist forces.

Force Drift Over Time

Due to the infinity of force distributions compatible with a given motion, the force predictions are likely to deviate from the transducer measurements over time. We quantify this effect by splitting the experiments into sub-sequences of maximum duration 1, 2, 4, 8, 16, 32 s (resp. 60, 120, 240, 480, 960, 1920 samples) and computing the resulting estimation errors for the four architectures with ground-truth initialization and offline post-processing or closed-loop prediction. For completeness, we reproduce the estimation errors over the full length sequences (average duration 60.1 s, standard deviation 3.8 s). We report the results in Table 3.2.

In line with the observations made on the full-length experiments, KDN-VF-∆ is the worst-performing network for every sequence duration, whereas KDN-VF-F is consistently best-performing or closely behind. This indicates again that decoupling thumb and antagonist redundancies is a viable strategy, yet more unstable in the presence of force variation uncertainties. We also observed that KDN-FH-∆ yields better results than its full force counterpart KDN-FH-F on the 1 s sequence duration and 2 s to a lesser extent. Recall that the ∆D i networks were introduced to accommodate the possibility of having the same motion caused by an infinity of force distributions. It appears here that KDN-FH-∆ is better at matching the real force variations on short sequences. Still, the applicability of this result on real manipulation tasks is limited due to the two following aspects. First, for sequence lengths greater than 2 s, the accumulation of ∆D i prediction errors becomes predominant. Second, the accuracy of the predicted force sequence is contingent on its initialization on the real forces being applied as measured by force transducers, which, ultimately, the force estimation framework aims at completely circumventing.

Force Sequence Initialization

Manipulation forces are sequentially computed based on an initial distribution that can be adjusted freely. We assess the force variability following non ground-truth initialization for sequences of maximum duration 4.0, 8.0, 16.0 and 32.0 s. Each sequence is initialized as follows. Using the average and standard deviation µ µ µ, σ σ σ of each finger force throughout the manipulation kinodynamics dataset, we pick a random sample D 0 following the normal distribution N (µ µ µ, σ σ σ ). We then correct D 0 using the SOCP of Section 3.3.1. Thus, we ensure that the resulting distribution D 0 is compatible with the initial kinematics K 0 . We report the force estimation errors for random and ground-truth initialization in Table 3.3.

Expectedly, ground-truth initialization yields better force estimates overall. Still, for each architecture, the performance difference decreases with the sequence duration. Indeed, even when starting from the same distribution, the predicted sequence is likely to deviate from the transducer measurements due to the infinity of force variations producing the same motion. This mitigates the importance of the force initialization over time. In the case of the best-performing network, KDN-VF-F (closed-loop), the difference is actually minor even starting from 8.0 s sequences. Finally, note that for any initial force distribution, the resulting sequence is constructed to be physically plausible given the observed motion and compatible with the forces a human could likely apply, based on the manipulation kinodynamics dataset. This allows the generation of force sequences following different profiles for the same motion (e.g., light or strong starting grasp). This method can also be used to reinitialize the prediction model when the resulting distributions are unreliable, as it may happen in the presence of motion tracking discontinuities.

Force Sensing From Vision

In the previous sections, we showed that the finger forces applied during manipulation can be inferred based on the kinematics of the object, as measured by a high-performance AHRS. Now, we propose to estimate the object's kinematics from markerless visual tracking, thus circumventing the need for any instrumentation whatsoever.

Model-Based Tracking

Along with the physical properties of the manipulated object, the force estimation framework requires its kinematics and the location of the contact points over which forces are distributed.

Object kinematics and contact points can be attained by means of tracking the hand and the manipulated object in 3D. Given such a successful 3D tracking, the kinematics can readily be computed from the motion of the object, and the contact points by reasoning about the proximity of the object and the fingers of the hand. Achieving hand-object tracking at the level of accuracy and robustness that is required for visual force estimation is a challenging task. We recorded experiments for quantitative evaluation using a SoftKinetic DepthSense 325 sensor. In the recorded sequences, the motion of the hand-object compound was such that a wide range of linear and angular velocities was explored. In practice, such motions frequently induce high levels of motion blur and strong (in some cases, complete) occlusions.

There is also considerable noise in the depth measurements provided by the sensor which, in some cases, is systematic (e.g. slanted surface artifacts). We used the 3D hand-object tracking method of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF]. This choice was derived from our experience in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF] which showed the efficacy and flexibility of the Ensemble of Collaborative Trackers (ECT) when dealing with more than a single object or hand. Through extensive quantitative experiments, we found that ECT yields accurate object kinematics estimates, as we discuss in Section 3.5.2. The accuracy of the force estimates depends mostly on that of the contact points. Indicatively, simulating a Gaussian noise of standard deviation 5 mm (resp. 10 mm) on the true contact points yields force reconstruction errors of zero mean (same net forces) and 0.87 N (resp. 1.54 N) standard deviation. In our preliminary experiments, the average contact point estimation error was greater than 20 mm. It should be noted that tracking the object alone fails due to the object occlusions by the manipulating hand not being accounted for. To deal with this problem, we capitalize on the observation that in the scenarios we are interested in, the hand achieves a firm grasp that changes only slightly when moving the object around. Under this assumption, as soon as the hand grasps the object, the hand and the object can be viewed as a single rigid compound. Thus, in a first step, we track hand-object interaction with [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF]. We then select a frame where the mutual hand-object occlusions are minimal. For that particular frame, we execute anew the optimization step by incorporating an extra term in the objective function that favors a hand pose where the fingertips touch the object at the known contact points. This leads to a hand-object configuration that is most compatible to observations, while respecting the contact point soft constraints. To arrive at this configuration, both the configuration of the hand and the object are revised. This configuration is then considered as a rigid compound which is used to track the whole sequence anew. The first tracking pass involves the optimization of 34 parameters per frame, 27 for the hand and 7 for the object. The second pass corresponds to 7 parameters only: the rigid transform of the compound.

Kinematics Estimation From Visual Tracking

With the camera calibrated intrinsically and extrinsically such that the gravity vector is known, we record and process 12 tracking experiments using the following objects. First, the instrumented device used in Section 3.2, in a configuration that does not appear in the manipulation kinodynamics dataset (mass 279 g). Second, three objects used in daily activities, 3D-printed and equipped with AHRS and force transducers for ground truth: a cuboid box (856 g), a small bottle (453 g), and a mug (174 g). We use the latter as an application of the force model on non-prismatic grasps in Section 3.6.2. We depict sample tracking results in Fig. 3.7.

Given the pose of the object throughout the experiment, we estimate its first and secondorder kinematics by numerical differentiation. This process is rather delicate as noise in the estimated trajectory generates spikes in its derivatives, i.e. velocity and acceleration, therefore forces. The effects of noise can usually be mitigated by smoothing the original signal over several samples or using appropriate filters, e.g. Gaussian. However, force profiles occurring in manipulation tasks are naturally spiky (see Fig. 3.6), as simply moving and stopping an object yields successive acceleration vectors in opposite directions. Therefore, smoothing the trajectory of the object comes at the expense of the ability to discern sudden variations in acceleration profiles, which is crucial.

As an alternative to classical numerical differentiation methods, we investigate the use of algebraic numerical differentiators [START_REF] Fliess | An algebraic framework for linear identification[END_REF][START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF] which do not assume any statistical properties on the signal's noise. We compare the kinematics estimates to the AHRS measurements on translational acceleration and rotational velocity. In order to quantify the effect on force estimation, we also compute the decomposition of the force transducer measurements on AHRS and vision-based kinematics. Denoting by T s = 1/60 s the time period between frames, we find an optimal Gaussian kernel of standard deviation σ = 3T s truncated at ±4σ . Similarly, the (κ, µ) algebraic numerical differentiator performs best as a filter of half width 4T s with parameters κ = µ = 0.5. We report the resulting kinematics estimation errors in Table 3.4.

On typical tracking sequences, smoothing techniques appear necessary to compute reliable kinematics estimates. Both the Gaussian and algebraic filters yield reasonable force discrepancies despite possible tracking uncertainties and discontinuities. Overall, while the Gaussian filter seems to perform slightly better than the algebraic filter, the latter also requires significantly less samples per estimate. This allows for a shorter lag for real time applications while also better capturing high frequency force variations, at the cost of a slightly larger sensitivity to tracking noise.

Force Prediction From Vision-Based Kinematics

Using a single camera, we track manipulation experiments and estimate the object's kinematics with algebraic filtering. In Section 3.4, although the four network architectures are trained on AHRS data, the object's kinematics is used as an input without consideration of the way it is measured. Thus, the trained networks can seamlessly generate force sequences from vision-based kinematics. In order to be completely independent of ground-truth sensing, we use the random initialization process described in Section 3.4.3. We compute the resulting estimation errors with respect to ground-truth force transducer measurements, along with, for reference, force predictions derived from the AHRS kinematics, none of these being used in the vision-based estimation process. We report our results in Table 3.5.

Under the same initialization conditions, forces computed from vision are comparable to forces computed from AHRS measurements. The decrease in accuracy is most noticeable on networks estimating force variations ∆D i due to a higher tendency to drift, as discussed in Section 3.4, but also additional uncertainties from visual tracking. We depict an example of forces estimated from vision in Fig. 3.8. Tracking discontinuities (e.g., lost hand-object pose), following second-order differentiation, are perceived by the force estimation framework as acceleration spikes and result in sudden fingertip force variations. These errors accumulate in the case of ∆D i networks since each prediction is directly relative to the preceding sample. When erroneous kinematics can be identified, their impact can be mitigated by reinitializing the prediction process based on the last reliable sample. However, while doing so is straightforward when AHRS measurements are available, it is difficult from the tracked kinematics alone, since acceleration spikes are not necessarily due to discontinuities but can also stem from actual sudden motions. Overall, KDN-VF-F appears the most resilient architecture to visual tracking uncertainties.

Discussion

Visual Tracking Assumptions

In Section 3.5.1, we suppose the contact points known and use them to compute a static grasp throughout the motion. Note that our force estimation framework itself is independent of the tracking method employed as long as reliable motion and contact information can be provided. The difficulty for us was to collect ground-truth measurements to validate our approach. Therefore, we forced the positioning of the fingertips at desired locations for both the real objects and the visual tracking system. Indeed, to allow arbitrary finger placement, the experimental apparatus should be covered with an array of high-precision 3D force transducers (that are not available in the required dimensions), or alternatively with Our force estimation framework can readily challenge in-hand manipulation scenarios with more sophisticated tracking systems (e.g., multi-camera). Again, assessing such tasks is limited by the difficulty of measuring the actual forces without obstructing the subject's haptic sense, which we consider essential in our demonstration. In effect, the tracking method we describe does not introduce any constraint besides those relative to the ground-truth instrumentation, while making it possible to monitor manipulation forces using a single off-the-shelf depth sensor.

Beyond Prismatic Grasps

For the sake of completeness, we evaluate the force estimation framework on a non-prismatic grasp. We construct a mug-shaped instrumented device, pictured in Fig. 3.7d, and arrange the force transducers on a circle, with the contact normals pointing towards the center. We then compute force distributions from visual tracking and AHRS measurements using the model trained on prismatic grasps. We depict the resulting predictions in Fig. 3.9. We observe the following. First, by considering the hand and the object as a single rigid compound, we are able to track the mug fairly accurately using a single depth sensor, despite it being essentially rotationally symmetric, except for a handle that is easily occluded. Second, in general, the RNN predictions do not follow the subtle force variations along the normal n k and tangential directions t x k as closely as the tangential directions t y k . Indeed, recall from Section 3.3.2 that the individual t y k per finger are defined, uniformly, as oriented towards the palm. This property is preserved in the case of the mug. However, while for prismatic grasps the n k are collinear with each other and perpendicular to the t x k , couplings appear between and among each set in the case of the mug. Still, although RNN predictions and force transducer measurements can quite differ, the SOCP ensures that the final distributions are physically plausible based solely on the observed kinematics and the object-grasp properties, regardless of the RNN training dataset.

While we could imagine extending the force estimation framework further by training new network architectures on arbitrary grasps, this is difficult in practice. The groundtruth instrumentation used in the manipulation kinodynamics dataset captures 11 degrees of freedom for the contact space (grasp width and 2D tangential position of each finger on the tangential space). In contrast, for general grasps, the instrumentation should allow 25 degrees of freedom (5 per finger, ignoring the transducer orientations about the normal axes). Due to a greater contact space dimensionality, it would require significantly more experiments to obtain a dataset that is both diverse and extensive, as well as a much heavier experimental setup to be able to fine-tune the position and roll-pitch of each transducer independently.

Computational Performance

On a computer equiped with an Intel i7-4700MQ CPU (quad-core 2.40GHz) and an NVIDIA GTX 780M GPU, we apply the KDN-VF-F closed-loop architecture on the testing dataset (39 experiments, total duration 2470 s, 60 samples per second). We report the computation time in Table 3.6. While at first the computation time appears greater than the dataset duration, the decomposition per process shows that the current implementation is actually rather sub-optimal. In fact, the three core components of our approach take only 5.29 ms per sample. First, algebraic differentiators implemented as finite impulse response filters are of minor impact on the computation time. Second, RNN predictions are parallelized on the GPU using the Torch7 framework [START_REF] Collobert | Torch7: A matlab-like environment for machine learning[END_REF]. Third, SOCP solving is done with the CVXOPT library [START_REF] Andersen | Cvxopt: A python package for convex optimization[END_REF].

In the current implementation, we construct the RNN input vectors and SOCP constraint matrices within their respective frameworks. A typical iteration is as follows:

1. Given the current kinematics and the SOCP corrected forces F i-1 at the previous step, we construct the RNN input vector (K i , D i-1 ). 4. We solve the SOCP and get the corrected forces F i . Steps 1 and 2 are executed in Lua for Torch7, while steps 3 and 4 are executed in Python for CVXOPT. Both being interpreted languages explains part of the overhead in preparing the data for each process. However, the majority of the computation time is actually spent on managing the two interpreters in succession, as represented by the Lua/Python bridge value in Table 3.6, which measures the time elapsed between steps 2 and 3, and between steps 4 and 1 (next iteration). Note that no calculation is performed during that time, only spent on switching between Lua and Python contexts. For this reason, simply implementing our method within a unified computational framework would certainly yield a tremendous increase in performance enabling real-time use. Other possible improvements at the numerical level include refactoring data structures to reduce redundancies and update constraint matrices only when needed, initializing the SOCP search at the RNN predictions, and rewrite the physical plausibility problem as a quadratic program (QP) using a discretized friction cone.

Conclusion and Future Work

Our work establishes that monitoring hand-object interaction forces at the fingertip level, a problem that is traditionally addressed with costly, cumbersome and intrusive force transducers, can be addressed in a cheap, reliable and transparent way using vision. Based on the first large-scale dataset on manipulation kinodynamics, the approach we present estimates force distributions that are compatible with both physics and real human grasping patterns. While the case of static prismatic grasps may appear restrictive, this limitation is only relative to the instrumentation required to collect ground-truth measurements, essential to prove the validity of the approach. Provided such an experimental setup, we expect that our method can be seamlessly extended to arbitrary grasps. Note that, even without, the current SOCP formulation is independent of the dataset used to train the networks and always produces distributions that result in the observed motion. Finally, even limited to prismatic grasps, the estimation of 3D forces for all five fingers on arbitrary motions greatly extends the state of the art in interaction capture. Using our approach, it is achieved with a setup as simple as a single RGB-D camera, enabling its use for monitoring of human activities and robot learning from demonstration in daily settings.

Our approach is readily compatible with any method providing accurate object kinematics. For qualitative evaluation, we applied our technique to the alternative object trackers and datasets of [KMB + 14, IWGC + 16] with the contact points handpicked from the visual observations. We depict snapshots of these experiments in Fig. 3.10 and invite the reader to refer to the supplementary material of [START_REF] Pham | Hand-object contact force estimation from markerless visual tracking[END_REF] for video results2 . When the situation allows a richer setup, a multi-camera system can also be used to track the hand and the object separately. Our future work involves alleviating the limitations induced by the ground-truth instrumentation. In order to monitor non rigid grasps, we aim to apply the force estimation framework in conjunction with tracking to guide the pose search as an implicit model for grasp plausibility and realism [START_REF] Pham | Capturing and reproducing hand-object interactions through visionbased force sensing[END_REF]. Additionally, the generalization to arbitrary grasps could be addressed by considering the variability of manipulation forces with grasp and object properties as an inverse optimal control problem. The manipulation kinodynamics dataset could thus be used to refine the force optimization problem with physiological criteria, e.g., grasp efficiency [START_REF] Zheng | Evaluation of grasp force efficiency considering hand configuration and using novel generalized penetration distance algorithm[END_REF]. In the long term, we plan to extend the force estimation framework to general articulated bodies for bi-manual grasping. In Chapter 4, we extend our approach to whole-body interactions with the environment.

Chapter 4

Whole-Body Contact Force Sensing From Motion Capture

Introduction

Humans purposefully interact with their environment through physical contact to manipulate and move themselves or objects. The contact forces that are applied during a given task are informative on both the resulting motion and the underlying intent. Thus, force sensing has direct applications in research fields such as robot learning from demonstration and control [START_REF] Rozo | A robot learning from demonstration framework to perform force-based manipulation tasks[END_REF][START_REF] Englsberger | Biologically inspired dead-beat controller for bipedal running in 3d[END_REF], physics-based animation [HBL11, ZSZ + 14] and visual tracking [START_REF] Kyriazis | Physically plausible 3d scene tracking: The single actor hypothesis[END_REF][START_REF] Pham | Capturing and reproducing hand-object interactions through visionbased force sensing[END_REF]. Contact forces are typically measured using force transducers that are costly, cumbersome and of limited, varying accuracy under repeated stress [START_REF] Ravinder | Tactile sensing -from humans to humanoids[END_REF]. In this work, we propose a method to infer human whole-body contact forces from motion capture alone. If combined with markerless visual tracking technologies [START_REF] Michel | Tracking the articulated motion of the human body with two rgbd cameras[END_REF], this would enable the non-intrusive monitoring of contact forces in daily activities. However, the problem is very challenging.

By means of the equations of motion for articulated rigid bodies, the knowledge of external and internal forces uniquely determines the resulting kinematics. In contrast, the reverse problem is generally indeterminate in multi-contact with the environment and the knowledge of a given motion may not suffice to fully characterize the underlying force distribution. For instance, one can stand still while applying foot forces of varying magnitude in opposite, lateral directions. The force distribution problem in whole-body locomotion is an active research topic in multiple fields (Section 1.4.2). In Chapters 2 and 3 we proposed a combined optimization and learning approach for force sensing from vision in the context of manipulation. However, these approaches do not directly extend to the case of whole-body Whole-Body Contact Force Sensing From Motion Capture multi-contact with the environment due to, in particular, the higher dimensionality of the human body and the variety of possible contact configurations. We address it as follows:

• Akin to our data-driven approach for manipulation, we collect real measurements on whole-body kinodynamics, in the form of 100 min of motion and force measurements for diverse contact configurations (Section 4.2).

• We propose a force estimation framework relying jointly on a recurrent neural network that learns how humans instinctively distribute contact forces while accounting for varying multi-contact configurations, and a second-order cone program that guarantees the physical plausibility of the resulting distributions with respect to the whole-body equations of motion (Section 4.3).

• We consistently validate our approach with ground-truth measurements throughout our work and demonstrate its accuracy on challenging scenarios (Section 4.4).

Finally, we discuss the current limitations of our work as well as possible applications and extensions (Section 4.5). To accelerate the research on this new topic and encourage alternative implementations, we make our datasets and algorithms publicly available1 .

Whole-Body Kinodynamics Dataset

Experimental Setup

We collect kinodynamic measurements (motion and forces) on human activities using two types of sensors in parallel. First, the human whole-body motion is tracked using a motion capture system (Xsens MVN Awinda) consisting of 17 wireless inertial measurement units (IMU) and batteries strapped at specified body landmarks. The choice of this motion capture technology is motivated by our intention to collect human kinodynamic measurements in confined and eventually outdoor environments. Vision-based systems (e.g., Vicon) are limited by strong occlusions occurring in whole-body interactions with the environment, and difficult to apply in uncontrolled environments on the fly (e.g., outdoor). The motion of the subject's body, modeled as a 23-segment skeleton, is recorded at 100 Hz. For each sample, the system provides the 6-DoF pose of each segment as well as the corresponding linear and rotational velocity and acceleration. Contact forces at the subject's feet are monitored with instrumented shoes (Xsens ForceShoe), equipped with 6-DoF forcetorque sensors at the heel and toes and IMUs recording the sensor orientations. We measure types of sensors can be corrected in isolation, based on physical considerations described in Section 4.2.2. Second, physical consistency between whole-body kinematics and contact force measurements can be enforced through the equations of motion for articulated systems of rigid bodies (see Section 4.3.1).

Preparing Measurements for Dynamics Analysis

The Xsens MVN Awinda system captures the motion of the subject using a 23-segment skeleton. At each time step, the whole-body pose is encoded by 161 parameters, i.e., 7 parameters per segment (3D position and orientation as quaternion). To facilitate the dynamics analysis, we transform the motion capture output into a kinematic tree rooted at the subject's pelvis and link the 23 segments with 22 spherical joints allowing 3 rotational degrees of freedom. In practice, we represent spherical joints with 3 chained revolute joints for compatibility with the Unified Robot Description Format (URDF)2 . Thus, the whole-body pose is summarized, without loss, as a 73-element vector q containing the quaternion and position of the base link in the global frame and 66 joint angles. We augment the kinematic tree with body segment inertial parameters (i.e., mass, center of gravity, inertia tensor) computed from the subject's weight and measurements with the anthropomorphic tables of Dumas et al. [START_REF] Dumas | Adjustments to mcconville et al. and young et al. body segment inertial parameters[END_REF]. The resulting dynamic model is implemented using the RBDyn library for rigid body dynamics [V + 13]. Corrections based on the equations of motion are discussed in Section 4.3.1. Prior to that, motion capture and force sensing errors can be mitigated based on the following considerations.

Recall that the whole-body tracking system employed in this work is based on inertial measurement units. Positions and orientations are computed by sensor fusion and successive integrations of acceleration and velocity measurements, making the pose estimation sensitive to drift over time. As such, contact configurations cannot be directly identified based solely on the reported body segment positions. For instance, we found that contacts between the feet and the ground during walking could not be consistently segmented by simple thresholding on their vertical positions. Instead, we thresholded the force sensor measurements to know when a contact occurs. Noticeable pose estimation errors also appeared when considering contacts others than with the ground, e.g., between the hand and the environment, as depicted in Figs. 4.2a and 4.2b. In such cases, we manually constrained the palm to be either horizontal or vertical. We also observed body segments in unrealistic orientations, illustrated in Figs. 4.2c and 4.2d, due to the motion capture system not enforcing joint limits in the skeletal tracking. We implemented them manually in our kinematic model and limited the range of the revolute joints. Note that this material limitation does not affect the generality of our approach and can be fully cirvumvented with additional visual observations, at the cost of portability and flexibility for the experimental setup. Finally, the force measurements are subject to noise, either from the sensors themselves or due to interferences in the wireless transmission. We attenuate it by smoothing all signals with a Gaussian filter of kernel σ = 0.05 s. Second, a slow-varying bias can appear in the force-torque measurements due to repeated stress and battery drain. We compute the bias through time by averaging the signals that persist when a sensor is not in contact with the environment, which should only be caused by the inertia of the moving parts attached to the sensing surface (e.g., force shoe external sole).

Experiments and Data Collection

In a preliminary study, four male volunteers took part as subjects in our experiments. Their weights (between 69.6 kg and 79.8 kg, plus the 3.5 kg acquisition system), heights (between 1.79 m and 1.94 m), and individual body segment lengths were measured to initialize the motion capture skeletal tracking model and BSIPs following [START_REF] Dumas | Adjustments to mcconville et al. and young et al. body segment inertial parameters[END_REF]. All sensors (motion and force-torque) were calibrated and reset between experiments following the manufacturers' recommended acquisition procedure to reduce the effects of measurement drift and hysteresis. The subjects were instructed to perform the following tasks: We illustrate these experiments in Fig. 4.3. The goal of the first task is to allow neural networks to capture the centroidal dynamics relationship between motion and forces in bipedal contact. The second task follows the same principle and also provides examples of static indeterminacy, i.e., how humans apply forces that cancel each other out and do not affect their state of equilibrium. The third and fourth tasks go further and are typical scenarios where the straightforward minimization of multi-contact forces leads to distributions that are physically plausible but not representative of those humans really apply, as discussed in Section 4.3.

For each experiment, we record the 6-DoF pose of the 23 segments through time as estimated by the motion capture suit and convert it to an equivalent 73-element vector of generalized coordinates q. The inertial motion capture system readily provides the angular and linear velocities and accelerations of each segment in the global frame. We construct the whole-body velocity (resp. acceleration) vector q (resp. q) from the angular and linear velocities (resp. accelerations) of the base link in the global frame and from the joint velocities (resp. accelerations) of the children segments, computed by projecting the segment angular velocities (resp. accelerations) onto the individual parent frames. We then perform a preliminary correction of both the motion capture and force-torque sensor measurements as described in Section 4.2.2. Overall, we construct a dataset of total duration 100 min comprising synchronized motion and force-torque measurements on 51 experiments.

Force Sensing From Whole-Body Motion

Whole-Body Force Optimization

We consider an articulated system of rigid bodies subject to N τ τ τ internal joint torques τ τ τ (i) :

τ τ τ (i) = τ (i) 1 , . . . , τ (i) N τ τ τ T (4.1)
and N F external wrenches F k = (τ τ τ k , f k ), with τ τ τ k and f k the respective external torque and force at contact k, expressed in the global frame. Considering the (free) position and orientation of the base link, the number of degrees of freedom is N DoF = N τ τ τ + 6. We denote by q, q, q the respective generalized coordinates, velocity and acceleration of the articulated system. The whole-body equations of motion can be expressed as:

H(q) q + C(q, q) = 0 6 τ τ τ (i) + N F ∑ k=1 J T k F k , (4.2)
with:

• H(q) the N DoF × N DoF mass matrix,

• C(q, q) the N DoF × 1 bias vector of the Coriolis, centrifugal forces and gravity terms,

• J k the N DoF × 6 Jacobian matrix at contact k,

• 0 6 the 6 × 1 internal wrench directly applied at the base link (zero for floating base).

For each contact k, we denote by z k the (uniquely defined) normal vector oriented from the environment to the body and choose two orthogonal vectors x k and y k in the tangential plane. We can thus express the external wrench F k in the contact space C k :

C k F k = τ x k , τ y k , τ z k , f x k , f y k , f z k T , with τ τ τ k = τ x k x k + τ y k y k + τ z k z k f k = f x k x k + f y k y k + f z k z k . (4.3)
Whole-Body Contact Force Sensing From Motion Capture

Having chosen z k oriented towards the body, the normal force component is such that:

f z k ≥ 0. (4.4)
The Coulomb model with friction coefficient µ k requires that:

f x k x + f y k y 2 ≤ µ k f z k . (4.5)
Eq. (4.2), (4.4) and (4.5) can be respectively incorporated as equality, linear inequality and cone constraints of a second-order cone program (SOCP) of optimization parameters:

x = τ τ τ (i) , C k F k ) k=1,N F , = τ (i) 1 , . . . , τ (i) N τ τ τ , τ x 1 , τ y 1 , τ z 1 , . . . , f x N F , f y N F , f z N F T . (4.6)
x is a vector of size N τ τ τ + 6N F .

Force Correction and Reconstruction

From this formulation, it is directly possible to construct physically plausible force distributions by minimizing a cost function depending only on the optimization parameters, e.g., the (squared) L 2 norm of the internal and external wrenches:

C L 2 (x) = x 2 2 , = τ τ τ (i) 2 2 + N F ∑ k=1 F k 2 2 .
(4.7)

The resulting forces, by construction, are necessarily compatible with the observed motion. However, in multi-contact, when there exists more than a single distribution for a given task, there is no guarantee that the L 2 -optimal distribution coincides with the actual forces being applied. As discussed in Section 1.4.2, the identification of the cost function supposedly optimized by the human central nervous system is still an active research topic. In our work, rather than trying to reconstruct an explicit formulation of the force distribution cost function, we instead propose to build an implicit model relying on machine learning techniques to capture how humans naturally distribute interaction forces in multi-contact.

Recall that due to measurement uncertainties, the preliminary correction process described in Section 4.2.1 does not fully ensure that the force-torque measurements are physically consistent with the motion. For this purpose, we formulate a cost function for the SOCP that quantifies the distance between the external wrenches to optimize F k and target wrenches F k :

C disc (x) = ε τ τ τ (i) 2 2 + N F ∑ k=1 F k -F k 2 , (4.8)
with ε an optimization weight for the internal joint torques. With ε big, the SOCP searches for external wrench distributions that minimize the magnitude of the joint torques. Conversely, with ε small, we extract the force distributions that are the closest to the reference F k while being physically compatible with the observed motion, regardless of the joint torques. Experimentally, we set ε = 10 -6 , non-zero for numerical resolution of the SOCP with the CVXOPT library for convex optimization [START_REF] Andersen | Cvxopt: A python package for convex optimization[END_REF]. By taking for F k the force-torque sensor measurements acquired experimentally, the resulting cost function can be used to mitigate sensing uncertainties by extracting physically correct force distributions in the vicinity of the uncertain measurements. Alternatively, in Section 4.3.4, we take for F k force-torque predictions estimated by a neural network based on whole-body kinematic observations and correct them with the same SOCP formulation. Fig. 4.4 illustrates how the accumulation of individual measurement errors over all force sensors can result in a measured net force that is in considerable disagreement with physics. In the following, we denote by ground truth the physically realistic distributions obtained by correcting the sensor measurements with the SOCP (relative to the dynamic model).

Learning Features

Our goal is to construct a mapping F between a set of input features K representing the whole-body kinematics and contact configuration, and output features D representing the underlying dynamics, i.e., external wrenches:

D = F (K).
(4.9)

For the sake of generality, we aim at modelling human force distribution patterns based on an optimal selection of high-level features rather than a large set of hand-engineered parameters. Akin to the case of manipulation, we select those based on their contributions on the equations of motion. However, while the Newton-Euler equations for rigid bodies allow the extraction of relevant parameters from a limited set of well-identified physical quantities (see Section 3.3.2), the complete equations of motion for articulated bodies are significantly more complex (see Eq. In this sequence, the subject stays still while applying varying forces in triple contact with the environment. The equations of motion dictate that the net contact force should be constant (top row), which is not apparent on the force sensor measuments (red line) due to sensing uncertainties. Forces compatible with the observed kinematics can be computed using an SOCP (green and blue lines). The minimization of the L 2 norm alone yields forces that are physically plausible but differ significantly from the measurements. Instead, minimizing the discrepancy to the uncertain measurements yields forces that are realistic both physically and compared to actual distributions.

body segments S , we denote by m s its mass and G s its CoM. In the global frame, we denote by v s the linear velocity of G s and R s its orientation matrix. In the segment frame, we denote by ω ω ω s and I s its local angular velocity and inertia tensor, respectively. With m the total mass of the articulated system and G its centroid, the linear momentum P P P and angular momentum L L L G at G are defined by:

       P P P = ∑ s∈S m s v s , L L L G = ∑ s∈S m s --→ GG s × v s + R s I s ω ω ω s .
(4.10)

With L L L G and Ṗ P P the time derivatives of the angular and linear momenta, respectively, g the gravity vector and G F k the contact wrench at contact k transformed to G, the Newton-Euler equations for centroidal dynamics state that:

L L L G Ṗ P P = 0 mg + N F ∑ k=1 G F k . (4.11)
We gather gravity, linear and angular momenta as a gravito-inertial wrench w (gi) [CPN16]:

w (gi) = L L L G Ṗ P P -mg (4.12)
Denoting by P k the location of contact k, Eq. (4.11) can thus be rewritten as:

w (gi) = N F ∑ k=1 τ τ τ k + --→ GP k × f k f k . (4.13)
The left-hand side of Eq. (4.13) is a purely kinematic term that can be directly computed from the whole-body pose and its derivatives, while the right-hand side summarizes the contributions of each contact wrench. This representation makes it a good candidate for the selection of optimal learning features extracting the gist of locomotory dynamics.

Recall that the whole-body motion is expressed in the world frame. To account for translational and rotational invariances, we extract and express learning features in a reference frame G of origin the centroid G and of orientation fixed with respect to a chosen body frame (e.g., that of the pelvis). Thus, walking straight to the North is perceptually equivalent to walking straight to the East. Based on Eq. (4.13), we extract as kinematics-based features:

• G w (gi) the 6-element gravito-inertial wrench expressed in G ,

• G P k the 3D position of each contact k expressed in G . This results in 6 + 3N F features for a motion involving N F contacts. In practice, the mapping F of Eq. (4.9) may require a fixed-size input vector. Thus, the formalism adopted should Whole-Body Contact Force Sensing From Motion Capture account for varying contacts. Instead of only considering the location of the contacts G P k,i that are active at timestamp i, we monitor a set of N c possible contact locations through time and encode contact configurations using parameters δ k,i such that:

δ k,i = 1 if contact k is active at time step i, 0 otherwise. (4.14)
In our experiments, we continuously monitor the position of the subject's feet and hands. As discussed in Section 4.2.2, we extract the parameters δ k by thresholding on the force sensor measurements, although this could be achieved by body and object segmentation when visual observations are available. Overall, the complete input features at timestamp i are:

K i = G w (gi) i , G P k,i , δ k,i k=1,N c T (4.15)
K i is a vector of size 6 + 7N c . Similarly, we define as output features the contact wrenches (i.e., forces and torques) applied at each monitored potential contact point k, expressed in G :

D i = G F k,i k=1,N c T (4.16)
For each monitored point k, the wrench G F k,i is zero if k is not in contact with the environment. D i is a vector of size 6N c .

Neural Network Model

In Eq. (4.9), the mapping F does not account for temporal continuity. As such, consecutive force distributions are independent of each other. Instead, we introduce a dependency on both the current motion and the past trajectory using the following formulation:

D i = F K j j=1,i . (4.17)
We model this time series structure using recurrent neural networks (RNN) [START_REF] Elman | Finding structure in time[END_REF] with long short-term memory neurons (LSTM) [START_REF] Hochreiter | Long short-term memory[END_REF]. A simple network architecture, which we denote by WBN-D (whole-body network, direct), thus consists in a simple RNN directly mapping K i to D i while keeping track of long-term dependencies to the past: For the purpose of training, validation and testing, we construct a random partitioning of the whole-body kinodynamics dataset into three subsets of respective size 70 %, 15 % and 15 %. We implement the WBN-D and WBD-F neural network architectures within the Torch7 framework [START_REF] Collobert | Torch7: A matlab-like environment for machine learning[END_REF]. Both architectures take the kinematics features K i as input, as well as D i-1 for WBD-F, pass them into two LSTM hidden layers of size 256, and compose the results with a linear layer returning the dynamics features D i . We train the networks using mini-batch stochastic gradient descent with a standard regression criterion (mean square error) and dropout to avoid overfitting [SHK + 14]. The SOCP correction is implemented using the CVXOPT library for convex optimization [START_REF] Andersen | Cvxopt: A python package for convex optimization[END_REF].

D i = WBN-D (K i ) .
In Table 4.1, we summarize the average error and standard deviation (between parentheses) between ground truth and the following force data:

• raw force sensor measurements • forces obtained from direct L 2 norm minimization • WBN-D outputs: raw and corrected offline • WBN-F outputs: raw and corrected between consecutive iterations We observe the following:

• Force-torque sensors are rather imprecise by themselves without physics-based correction (Table 4.1, first row, first column), in terms of bias (average error) and repeatibility (standard deviation).

• On the other hand, forces computed with a direct L 2 criterion also greatly differ from actual measurements (see Fig. 4.4). Thus, physics-based optimization, by itself, is not enough to capture the real forces being applied.

• Finally, the accuracy of all methods relying on learning and optimization is at least comparable (MBN-F) or significantly better (MBN-D) than that of the force-torque sensors.

Our main outcome is thus that, provided a rich dataset on human kinodynamics, the method we propose can outperform physical force sensing both in terms of accuracy and usability.

Comparing the force prediction architectures in detail, we observe that, in the absence of SOCP correction (first column), the WBN-D architectures performs better than WBN-F. This is expected, since feeding past raw predictions does not bring new information to the system. Additionally, all networks are trained on physically coherent data, i.e., input kinematics K i and output dynamics D i are compatible with respect to the equations of motion. However, raw neural network predictions D (raw) i are not guaranteed to be compatible with K i . This directly impacts the WBN-F architecture, since possibly inaccurate force predictions are repeatedly fed back into the network (see Eq. (4.18)), which does not happen for the direct architecture WBN-D (see Eq. (4.19)). With SOCP correction (second column), we observe significant improvement in the accuracy of WBN-D but not WBN-F. A possibility is that the embedded SOCP correction interferes with the recurrent neural network prediction and internal state update processes. As future work, we aim at combining convex optimization and learning into a unified computational framework [START_REF] Yan | A collective neurodynamic approach to constrained global optimization[END_REF] or guiding the neural network training with physics-based constraints [START_REF] Stewart | Label-free supervision of neural networks with physics and domain knowledge[END_REF].

Results on Restricted Training

During walking, most of the time is spent with only one foot on the ground. In single contact, the equations of centroidal dynamics, see Eq. (4.11), dictate that the contact wrench can be uniquely identified from the body kinematics. Therefore, it may not be necessary to extensively train neural networks on such examples. Instead, the prediction accuracy may suffer if multi-contact examples (where the difficulty resides) represent a minority of the dataset. We assess this effect by training the previous neural network architectures not on the whole dataset, but on two sets containing only either walking or multi We apply each network type on both its own testing set and that of the other type. We illustrate the application of WBN-D-W and WBN-D-M on a triple contact example (leaning on a table) in Fig. 4.6 and on walking in Fig. 4.7. In both cases, the raw predictions are corrected with the SOCP to ensure their physical compatibility with the observed motion. As it can be expected, the architecture trained only on walking fails at capturing the actual force distributions applied by humans in multi-contact (see Fig 4 .6). In contrast, the architecture that was not extensively trained on walking accurately reconstructs contact forces even on such scenarios (see Fig. 4.7). This confirms that physics-based optimization is a valuable complement to recurrent neural networks for the latter to focus on multi-contact indeterminacy.

Discussion and Future Work

Our work introduces a novel method for the inference of contact forces applied by human subjects from their motion only. Our system estimates forces that are both physically plausible and in agreement with ground-truth measurements, even in challenging contact configurations where the force distribution problem is highly indeterminate. Trained on our (public) dataset, the neural network architectures can be applied to any centroidal representation, while the SOCP can be formulated for any articulated body. As such, our approach can be seamlessly generalized to any whole-body tracking system. Applying our method to markerless visual tracking would thus enable fully non-intrusive monitoring of whole-body forces in daily activities. Still, our approach would certainly benefit from the further collection of ground-truth force measurements on even more rich motion and contact configurations. Another possibility could be to integrate convex optimization and learning into a unified computational framework [START_REF] Yan | A collective neurodynamic approach to constrained global optimization[END_REF]. In the long term, we also plan to apply our framework to force-based robot learning from demonstration, on-line multi-contact motion retargeting and knowledge-based multi-contact planning and control.

Conclusion

In this thesis, we demonstrated that the estimation of interaction contact forces, a problem that pertains to the human sense of touch, could be addressed through the lens of vision.

In doing so, we formulated the novel problem of force sensing from motion capture, in which the goal is to estimate the real, rather than just physically plausible, contact forces that humans exert when interacting with their environment, based only on the observation of their motion. This, in turn, makes it possible to circumvent the traditional approach that consists in instrumenting objects or actors with costly, cumbersome, intrusive force transducers in monitoring motion together with force data. Towards force sensing from vision, we first challenged the estimation of normal forces during manipulation. We constructed a first ground-truth acquisition prototype measuring real normal forces applied by human subjects. Performing model-based visual tracking using a single RGB-D sensor, we estimated the object's kinematics by differentiating the captured pose through time. From the Newton-Euler equations, we computed the net force sequence explaining the observed motion and compared it to the net forces obtained by summing up the force transducer measurements. We thus confirmed that the state of the art in markerless hand-object tracking could indeed bridge the first gap between motion and (net) forces. However, we observed that the actual forces measured during real experiments were significantly greater than those obtained by straightforward physics-based optimization, a caveat neglected by force models used only for simulation. We addressed this indeterminacy with artificial neural networks capturing how humans naturally distribute internal forces in addition to the nominal forces required to achieve a given motion.

We then made our approach more robust and general, in particular by suppressing the reliance of an arbitrary cost function decomposing full force measurements into nominal and internal components. Instead, we harnessed recent advances in machine learning and trained recurrent neural networks on a novel, large-scale dataset on human manipulation kinodynamics. We developed a new ground-truth acquisition setup measuring 3D contact forces with five precision force transducers, repositionable on the object's surface, and the object's kinematics with AHRS-embedded accelerometers and gyroscopes. Together with Conclusion reconfigurable properties such as shape, friction and mass distribution, this allowed us to gather a dataset comprising 3.2 h of motion and force measurements under 193 different object-grasp configurations. We then showed that recurrent neural networks trained on this dataset could successfully be applied to kinematics estimated from vision. We thus formulated a generic force estimation framework in which recurrent neural networks trained on the manipulation kinodynamics dataset were applied to kinematics estimated from vision, and their predictions interactively corrected between consecutive time steps by a second-order cone program ensuring their physical consistency.

Finally, we proposed an extension of our force estimation framework from the case of manipulation to that of whole-body contacts with the environment. Similarly, we observed that in multi-contact, forces obtained from optimization only significantly differed from those really applied. We collected a new database on whole-body kinodynamics using an inertial motion capture suit in combination with force-torque sensors under the subject's shoes and held in-hand. In essence, the generality of our previous formulation allowed its extension to whole-body contacts simply by transposing the SOCP from the Newton-Euler equations for a 6-DoF rigid object to the equations of motion for a 72-DoF articulated system, while training the RNN on the whole-body centroidal dynamics. Performing experiments both on walking and challenging multi-contact configurations, we noted that our approach could actually outperform physical force sensing, in particular due to the large amount of noise and uncertainties high-capacity force-torque sensors are subject to, under repeated stress.

Although we obtained significant results towards the estimation of contact forces from motion capture, we believe it is still a new path of research for which we are only laying the foundations. In the case of manipulation, the quality of the force estimation is still contingent on the accuracy of the hand-object tracking. In particular, while we were consistently able to capture the object's kinematics, locating the contact points with sufficient accuracy has proven to be particularly challenging in the current PSO-based tracking framework. Instead of sampling pose hypotheses in the configuration space, an alternative could be to sample the contact space and optimize hand poses together with the resulting manipulation forces. Removing the current (artificial) constraint on grasp staticity, would thus enable the monitoring of contact forces during in-hand manipulation. Still, the acquisition of groundtruth measurements, which we consider crucial to validate any force model, is extremely difficult for truly arbitrary grasps and especially contact orientations. In other words, by trying to get rid of force transducers (the goal of this thesis), we fully experience how cumbersome they are! Still, our approach was purposely made generic to accomodate further advances in both visual tracking and force sensing technologies. In effect, we already show its applicability to alternative object tracking datasets. In the case of whole-body interaction, our approach would also benefit from the collection of a broader dataset involving even more complex contact configurations and motions. Alternatively, we would also like to combine our whole-body force model with kinematics estimated from markerless visual tracking.

We also identified other mid-term possible developments. First, we would like to investigate the incorporation of physics-based constraints directly in the training of the neural networks, allowing the reconstruction of physically correct and human-like force distributions in a unified computational framework. Conversely, the force distribution problem could be considered again from the perspective of inverse optimal control. Using our large-scale datasets on both manipulation and whole-body kinodynamics, we could look for invariants in the way humans distribute forces on their environment while also regulating internal forces and joint torques. Alternatively, our force estimation framework could also be used for robot learning from demonstration, by providing a natural way to teach tasks using only vision instead of heavy physical haptic interfaces. We would also like to investigate the use of FSV not alone, but in complement of low-cost force transducers. Such a combination could be mutually beneficial. On one hand, cheap sensors providing for instance only low-accuracy, normal force measurements, could be corrected to be compatible with the observed motion and also augmented with predicted tangential components. On the other hand, uncertain force measurements could still serve as a reference for FSV. This would thus alleviate the current biggest limitation of the current system, namely that it only predicts the force distributions that are most likely given past observations and can be 'deceived' by applying forces that significantly deviate from the norm (e.g., by gripping stronger than natural).

In the long term, we would like to consider the estimation of contact forces for manipulation and whole-body interactions together, rather than separately. Instead of considering only one rigid body or one articulated system, subject to external forces, we could instead consider a set of multiple agents, either active (i.e., that ) or passive (i.e., objects) with respect to the application of contact forces, each with their own dynamic model. This would enable the consideration of interaction scenarios involving synergies between actors, e.g., for bimanual prehension or multiple persons carrying large objects in cooperation. Alternatively, we would also like to investigate the use of additional types of sensors, as long as these remain minimally intrusive. In particular, physiological sensors such as electroencephalography (EEG) headsets or surface electromyography (EMG) electrodes may provide valuable information in discerning otherwise dynamically equivalent force distributions. Finally, in our work, we made the assumption that the physical properties of the manipulated object were readily available (e.g., through inertial parameter identification or CAD). Conversely, we could explore the reverse problem, i.e., retrieve the physical properties of manipulated objects using only visual observations on how people naturally interact with them.

1 A few recent

  advances in artificial intelligence, robotics and virtual reality. . 1.1 Instrumentation examples. (a): on object, (b): on hand, (c): haptic interface. 1.2 Bottom-up and top-down pose estimation methods. . . . . . . . . . . . . . 1.3 Tracking inputs and models. (a): RGB-D sensor observations, (b): hand model. 1.4 Tracking: (a) hand in isolation [OKA11a], (b) two hands and multiple objects [KA14]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Dynamics estimation from motion capture for human and action understanding. 1.6 Coulomb friction model and dynamics simulation with quadratic programming. 1.7 RNN and LSTM graphic visualization [Ola15]. . . . . . . . . . . . . . . . 2.1 Using a single RGB-D camera, we track markerless hand-object manipulation tasks and estimate with high accuracy contact forces that are applied by human grasping throughout the motion. . . . . . . . . . . . . . . . . . . . 2.2 (a) Measurements from tactile sensors are used to estimate nominal and internal force decompositions from vision. (b) Full contact forces are reconstructed by combining ANN internal force predictions with an SOCP ensuring physical plausibility. . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Instrumented device for quantitative and qualitative evaluation. . . . . . . . 2.4 Validation protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Comparison between vision-based kinematics and AHRS-embedded accelerometer and gyroscope. . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 Contact forces from vision based on L 2 criterion are individually lower than tactile sensor measurements but result in the same net force. . . . . . . . . 2.7 Artificial neural networks used in conjunction with cone programming successfully predict force distributions that both explain the observed motion and follow natural human force distribution patterns. . . . . . . . . . . . . xii List of figures 2.8 (a) Visible by the camera, (b) palm and thumb are successfully recognized.

3. 2

 2 Force distributions computed only by physics-based optimization are guaranteed to result in the observed motion (net force and torque) but can significantly differ from the real distributions at the finger level. . . . . . . . . . 3.3 For each experiment, we extract force distributions compatible with the observed motion in the vicinity of the transducer measurements. . . . . . . 3.4 Two RNN architectures learning the manipulation forces at each fingertip based on the current kinematics and past forces. . . . . . . . . . . . . . . . 3.5 Open-loop and closed-loop force generation processes. . . . . . . . . . . . 3.6 Open-loop, post-processed and closed-loop force predictions for KDN-VF-∆ (normal components). In this example, the open-loop estimation drifts away from physically plausible solutions (negative normal forces). Compatibility with the observed motion is enforced through offline post-processing or closed-loop control at each time step. . . . . . . . . . . . . . . . . . . . . 3.7 The hand and the object are tracked as a rigid compound. . . . . . . . . . . 3.8 Force estimates from AHRS measurements and visual tracking with closedloop KDN-VF-F and random initialization. . . . . . . . . . . . . . . . . . 3.9 Force estimates with non-prismatic grasp (mug). . . . . . . . . . . . . . . 3.10 Qualitative force predictions (red) with manually picked contact points (yellow) on alternative object tracking datasets: (a) [KMB + 14], (b) [IWGC + 16]. 4.1 Acquisition system for whole-body kinematics and contact forces. . . . . . List of figures xiii 4.2 Erroneous tracking examples. (a): against a table, the right hand should be horizontal with the contact normal pointing upwards. (b): against a wall, the hand should be vertical with the contact normal in the horizontal plane. (c): right foot flipped backwards when raised on a foot stand. (d): foot orientation drift with subject standing still. . . . . . . . . . . . . . . . . . . . . . . . 89 4.3 Sample poses from the whole-body kinodynamics dataset. . . . . . . . . . 90 4.4 In this sequence, the subject stays still while applying varying forces in triple

kGFF

  94 4.5 Direct and feedback whole-body network architectures. . . . . . . . . . . . 98 4.6 Triple contact example. Trained on similar examples, WBN-D-M successfully estimates the actual forces being applied. In contrast, WBN-D-W predicts physically valid but significantly different force distributions. . . . 101 4.7 Walking example. Despite not having been extensively trained on such examples, the performance of WBN-D-M used in conjunction with physicsbased optimization is comparable to that of WBN-D-W. . . . . . . . . . . 102 List of tables 2.1 Kinematics estimation errors (average and standard deviation) for central finite difference, Gaussian filtering, and algebraic filtering. . . . . . . . . . 2.2 Relative force estimation errors based on the exhaustivity of the training dataset. and × indicate features that respectively appear or not in the partial training dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1 Force Estimation Errors on Full-Length Manipulation Sequences . . . . . . 3.2 Force Estimation Drift Through Time . . . . . . . . . . . . . . . . . . . . 3.3 Influence of Force Prediction Initialization . . . . . . . . . . . . . . . . . . 3.4 Kinematics Estimation Errors from Tracking . . . . . . . . . . . . . . . . . 3.5 Force Estimation Errors From Visual Tracking . . . . . . . . . . . . . . . . 3.6 Computation Time Decomposition by Process . . . . . . . . . . . . . . . . 4.1 Force Estimation Errors [N] on Testing Set (16 min) . . . . . . . . . . . . . Local contact space (normal-tangential) at contact k ( f k , g k , h k ) Local force decomposition at contact k along n k , t x k , t y k Center of mass of the manipulated object F k Contact force applied at contact k Nominal and internal components of contact force F k P k 3D position of contact point k Finger set: thumb, index, middle, ring, pinky J Inertia matrix in object frame, taken at the center of mass m Object mass F F F , τ τ τ Net force and torque exerted on the manipulated object θ θ θ

  between u and v u × v Cross product between u and v -→ AB Vector from A to B x (N) N-th order time derivative of signal x Whole-Body Interactions R s Orientation matrix of body segment s in the global frame ω ω ω s Angular velocity of body segment s in the body frame L L L G Angular momentum about G C(q, q) Bias vector including Coriolis, centrifugal forces and gravity terms G Centroid of the articulated system G s Center of mass of body segment s G F k Contact wrench at contact k transformed to GGReference frame attached to the centroid and fixed with respect to a chosen body segment (e.g., pelvis)G P k 3D position of contact point k expressed in G G w (gi) gravito-inertial wrench w (gi) expressed in G P k 3D position of contact point k C k = (x k , y k , z k ) Local coordinate system at contact k δ k,iActivity flag for contact k at time step i

Introduction ( a )

 a AlphaGo playing against Lee Sedol [SHM + 16]. (b) Robot arms learning grasping [LPKQ16]. (c) Boston Dynamics Atlas robot pushed with a stick. (d) Oculus Touch marker-based tracking system.

Figure 1 :

 1 Figure 1: A few recent advances in artificial intelligence, robotics and virtual reality.

  Force transducer mounted on fingertip for interaction capture and synthesis [KP06]. (c) Haptic interface to demonstrate a robotic arm how to pour drinks based on force measurements during teleoperation [RJC13].

Figure 1

 1 Figure 1.1: Instrumentation examples. (a): on object, (b): on hand, (c): haptic interface.

  θ θ θ * Generate new pose hypothesis (b) Top-down approach for the tracking of two hands in interaction [OKA12].

Figure 1

 1 Figure 1.2: Bottom-up and top-down pose estimation methods.

  and whole-body pose estimation [SFB + 11, SSK + 13]. Tang et al. explored alternative RDF models and addressed the discrepancy between synthetic data and real observations [TYK13, TCTK14]. In [TSLP14], Tompson et al. extracted hand pose features using RDFs in combination with convolutional neural networks (CNN) [LBBH98]. Rogez et al. recovered arm and hand poses using support vector machines (SVM) in the context of egocentric vision with a chest-mounted camera [RSR15a].

( a )

 a Raw color (top left) and depth (top right) are captured separately by two cameras. From the intrinsic parameters and relative positions, depth is registered to color (bottom left) and conversely (bottom right). (b) Articulated 3D hand model (top) and sphere-based collision model (bottom) [OKA11b].

Figure 1 . 3 :

 13 Figure 1.3: Tracking inputs and models. (a): RGB-D sensor observations, (b): hand model.

( a )

 a Left to right: raw RGB, aligned depth, segmented hand, pose estimate. (b) Hand-object tracking.

Figure 1 . 4 :

 14 Figure 1.4: Tracking: (a) hand in isolation [OKA11a], (b) two hands and multiple objects [KA14].

  Contact forces when sitting from finite element analysis with deformable body model [ZJZ + 16].

Figure 1

 1 Figure 1.5: Dynamics estimation from motion capture for human and action understanding.

( a )

 a 3D normal-tangential force decomposition. (b) Rigid body dynamics simulation [Bar91].

Figure 1 . 6 :

 16 Figure 1.6: Coulomb friction model and dynamics simulation with quadratic programming.

( a )

 a Unrolled recurrent neural network. (b) Long short-term memory cell state.

Figure 1

 1 Figure 1.7: RNN and LSTM graphic visualization [Ola15].

Figure 2

 2 Figure 2.2: (a) Measurements from tactile sensors are used to estimate nominal and internal force decompositions from vision. (b) Full contact forces are reconstructed by combining ANN internal force predictions with an SOCP ensuring physical plausibility.

( a )

 a CAD view: AHRS (upper plate) and five force sensors (cuboids) repositionable to assess different contact configurations (cylinders). (b) Sensor configuration example (green) and other locations (holes).

Figure 2

 2 Figure 2.3: Instrumented device for quantitative and qualitative evaluation. Central Gaussian Algebraic Trans. acc. Avg. -0.029 -0.022 -0.024 [m • s -2 ] St.d. 1.686 1.627 0.904 Rot. vel. Avg. 0.084 0.070 0.052 [rad • s -1 ] St.d. 1.559 1.294 1.241

Figure 2 . 5 :

 25 Figure 2.5: Comparison between vision-based kinematics and AHRS-embedded accelerometer and gyroscope.

Figure 2

 2 Figure 2.7: Artificial neural networks used in conjunction with cone programming successfully predict force distributions that both explain the observed motion and follow natural human force distribution patterns.

( a )

 a Input color image. (b) Hand-object pose estimates. (c) Synthesized back and side views. (d) Real thumb mistaken for thumb and index.(e) Lost tracking: hand-object interpenetration.

Figure 2

 2 Figure 2.8: (a) Visible by the camera, (b) palm and thumb are successfully recognized. (c) However, the occluded finger poses are physically impossible as none hold the object. The accumulation of tracking errors can lead to (d) implausible and even (e) impossible poses.

( a )

 a AHRS base, thickness layers, sensor plates for repositionable transducers (four sizes). (b) 3D force transducers, support caps of various frictional characteristics, AHRS. (c) Assembled instrumented device. The cables are tied to the subject's elbow to minimize force perturbations. (d) 3D-printed box-shaped device with extra mass. (e) Bottle-shaped device. (f) Mug-shaped device for non-prismatic, spherical grasp.

Figure 3 . 1 :

 31 Figure3.1: We collect the manipulation kinodynamics dataset using dedicated instrumented devices of adjustable shape, friction, mass distribution and contact configuration (a-c). Additionally, we construct devices based on everyday objects, instrumented so as to allow intuitive interactions (d-f).

Figure 3 . 2 :F k -F k 2 2 Figure 3

 3223 Figure 3.2: Force distributions computed only by physics-based optimization are guaranteed to result in the observed motion (net force and torque) but can significantly differ from the real distributions at the finger level.

Figure 3

 3 Figure 3.4: Two RNN architectures learning the manipulation forces at each fingertip based on the current kinematics and past forces.

( a )

 a Open-loop: force predictions are not checked for physical plausibility.

  b) Closed-loop: force predictions are corrected between time steps.

Figure 3

 3 Figure 3.5: Open-loop and closed-loop force generation processes.

Figure 3

 3 Figure 3.6: Open-loop, post-processed and closed-loop force predictions for KDN-VF-∆ (normal components). In this example, the open-loop estimation drifts away from physically plausible solutions (negative normal forces). Compatibility with the observed motion is enforced through offline post-processing or closed-loop control at each time step.

Figure 3 . 7 :

 37 Figure 3.7: The hand and the object are tracked as a rigid compound.

Figure 3 . 8 :

 38 Figure 3.8: Force estimates from AHRS measurements and visual tracking with closed-loop KDN-VF-F and random initialization.

2..Figure 3 . 9 :

 39 Figure 3.9: Force estimates with non-prismatic grasp (mug).

Figure 3 .

 3 Figure 3.10: Qualitative force predictions (red) with manually picked contact points (yellow) on alternative object tracking datasets: (a) [KMB + 14], (b) [IWGC + 16].

( a )

 a Contact with table. (b) Contact with wall. (c) Raised right foot. (d) Static stance.

Figure 4 . 2 :

 42 Figure 4.2: Erroneous tracking examples. (a): against a table, the right hand should be horizontal with the contact normal pointing upwards. (b): against a wall, the hand should be vertical with the contact normal in the horizontal plane. (c): right foot flipped backwards when raised on a foot stand. (d): foot orientation drift with subject standing still.

  (a) Walking. (b) Upper body balancing. (c) Wall support. (d) Leaning on a table.

Figure 4 . 3 :

 43 Figure 4.3: Sample poses from the whole-body kinodynamics dataset.

  Figure4.4: In this sequence, the subject stays still while applying varying forces in triple contact with the environment. The equations of motion dictate that the net contact force should be constant (top row), which is not apparent on the force sensor measuments (red line) due to sensing uncertainties. Forces compatible with the observed kinematics can be computed using an SOCP (green and blue lines). The minimization of the L 2 norm alone yields forces that are physically plausible but differ significantly from the measurements. Instead, minimizing the discrepancy to the uncertain measurements yields forces that are realistic both physically and compared to actual distributions.

  (4.18) Whole-Body Contact Force Sensing From Motion Capture Body kinematics, contact config. (i) are direcly computed from the kinematics and contact configuration.Body kinematics, contact config. (i) Force predictions are corrected between consecutive time steps.

Figure 4 . 5 :

 45 Figure 4.5: Direct and feedback whole-body network architectures.

  -contact examples. Both are again randomly partitioned into training (70 %), validation (15 %) and testing (15 %) subsets. We denote by WBN-D-W and WBN-F-W the respective direct and feedback architectures trained on walking examples only, and by WBN-D-M and WBN-F-M the networks trained on multi-contact examples.

Figure 4 . 6 :Figure 4 . 7 :

 4647 Figure 4.6: Triple contact example. Trained on similar examples, WBN-D-M successfully estimates the actual forces being applied. In contrast, WBN-D-W predicts physically valid but significantly different force distributions.

  et al. [TST + 15] augmented the ICP with priors built from a hand pose database. Both approaches achieved the tracking of a hand from depth in real time. The signed distance function (SDF) [CL96] used previously in conjunction with ICP for surface mapping and tracking [NIH + 11] was extended by Schmidt et al. to general articulated models [SNF14] such as the hand or the whole body. Ballan et al. tracked two hands in interaction with an object [BTG + 12] by searching for salient points (e.g., fingernails and fingertips) using a discriminative model, in combination with the local optimization of a differentiable cost function accounting for edges, optical flow and collisions. Tzionas et al. extended this approach with physical simulation to improve the realism of the hand-object pose estimates [TBS + 15]. Oikonomidis et al. tracked a hand with particle swarm optimization (PSO)

  . McConville et al. combined anthropomorphic measurements from 31 living male subjects with densities measured on cadavers to construct tables linking body segment measurements and inertial parameters [MCC + 80]. Young et al. extended this approach on 46 living female subjects [YCS + 83]. BSIP estimation on living subjects was also performed by Zatsiorsky et al. using gamma-ray in [ZS83, ZSC90b, ZSC90a]. Deleva et al. adjusted these results to alternative anthropomorphic conventions in [DL96]. Similarly, the tables of McConville et. al and Young et al. were adjusted by Dumas et. al in [DCV07], which we use in this thesis.

  Mboup et al. proposed candidate Π and ρ operators that, moving back Eq. (1.8) into the time-domain, can be implemented as finite impulse response filters. This work was further extended to multidimensional signals, over possibly irregular sampling grids[START_REF] Riachy | Multivariate numerical differentiation[END_REF]

Table 2

 2

.1: Kinematics estimation errors (average and standard deviation) for central finite difference, Gaussian filtering, and algebraic filtering.

  Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer Manipulation Forces Figure 2.6: Contact forces from vision based on L 2 criterion are individually lower than tactile sensor measurements but result in the same net force.

  Table 2.2: Relative force estimation errors based on the exhaustivity of the training dataset. and × indicate features that respectively appear or not in the partial training dataset.

	Towards Force Sensing From Vision: Observing Hand-Object Interactions to Infer
					Manipulation Forces
				Part. training Full training
	User	Mass	Grasp	Avg. St.d. Avg. St.d. [%] [%] [%] [%]
				10.7 12.4	9.71 12.0
	×			10.9 12.3	10.3 11.8
		×		10.8 11.3	10.4 12.4
			× 14.6 14.5	10.9 11.3
	× × × 14.9 14.8	9.94 12.6

Table 3

 3

	.1: Force Estimation Errors on Full-Length Manipulation Sequences
		Open-loop	Post-processed Closed-loop
	KDN-FH-F	0.49 (4.14)	0.44 (4.07)	0.16 (3.54)
	KDN-FH-∆ -43.67 (156.72)	0.60 (4.74)	0.50 (11.03)
	KDN-VF-F	0.29 (3.19)	0.29 (3.13)	0.12 (2.60)
	KDN-VF-∆ 1145.06 (3984.86)	3.54 (11.80)	2.32 (6.60)

Table 3

 3

			.2: Force Estimation Drift Through Time
		1.0 s	2.0 s	4.0 s	8.0 s	16.0 s	32.0 s	Full length
		80)	0.15 (2.42)	0.36 (3.22)	0.56 (3.89)	0.68 (4.34) 0.56 (4.62)	0.60 (4.74)
	KDN-FH-∆, closed-loop	0.02 (1.87)	0.11 (2.48)	0.27 (3.44)	0.45 (5.14)	0.58 (7.39) 0.57 (9.32) 0.50 (11.03)
	KDN-VF-F, post-processed 0.07 (2.09)	0.13 (2.51)	0.20 (2.82)	0.25 (2.99)	0.27 (3.07) 0.28 (3.11)	0.29 (3.13)
	KDN-VF-F, closed-loop	0.02 (1.86)	0.04 (2.16)	0.07 (2.38)	0.10 (2.50)	0.11 (2.56) 0.12 (2.58)	0.12 (2.60)
	KDN-VF-∆, post-processed 0.43 (2.93)	0.87 (4.47)	1.64 (7.11)	2.37 (9.33) 2.90 (10.61) 2.94 (11.13) 3.54 (11.80)
	KDN-VF-∆, closed-loop	0.41 (2.47)	0.76 (3.45)	1.24 (4.74)	1.69 (5.69)	1.99 (6.17) 2.15 (6.43)	2.32 (6.60)

KDN-FH-F, post-processed -0.21 (2.06) -0.21 (2.43) -0.13 (2.86) -0.04 (3.22) 0.07 (3.54) 0.19 (3.76) 0.44 (4.07) KDN-FH-F, closed-loop -0.13 (2.20) -0.12 (2.47) -0.07 (2.80) 0.00 (3.07) 0.06 (3.24) 0.08 (3.33) 0.16 (3.54) KDN-FH-∆, post-processed 0.00 (1.

Table 3

 3

		.4: Kinematics Estimation Errors from Tracking
		Central	Gaussian	Algebraic
	Trans. acc. [m • s -2 ] 0.31(25.36) -0.02(2.92) -0.05(3.03)
	Rot. vel. [rad • s -1 ] 0.14(446.45) -0.05(30.94) 0.01(31.76)
	Force [N]	1.18(8.94)	0.01(0.72)	0.01(0.75)

Table 3

 3

	.5: Force Estimation Errors From Visual Tracking
	Kinematics	AHRS	AHRS	Vision
	Initialization	ground truth	random	random
	KDN-FH-F, PP -1.10 (2.95) -1.12 (2.95) -1.18 (3.11)
	KDN-FH-F, CL -1.37 (3.12) -1.37 (3.13) -1.25 (3.61)
	KDN-FH-∆, PP	0.72 (3.38)	0.85 (3.42)	0.94 (3.39)
	KDN-FH-∆, CL 1.21 (5.80)	2.27 (11.86) 3.50 (17.28)
	KDN-VF-F, PP	0.18 (2.64)	0.14 (2.68)	0.15 (2.69)
	KDN-VF-F, CL -0.01 (2.20) 0.02 (2.27) -0.04 (2.30)
	KDN-VF-∆, PP 5.40 (27.61) 5.16 (23.06) 5.94 (24.54)
	KDN-VF-∆, CL 2.20 (16.31) 3.87 (19.99) 7.37 (25.15)
	dedicated force sensing surfaces [SCCP14], generally limited in accuracy and range (e.g.,
	normal forces only).

Table 3

 3

	.6: Computation Time Decomposition by Process
		Total	Per sample Per timestep
	Experiment duration 2470.0 s 16.67 ms	100.00 %
	Computation time	3521.4 s 23.76 ms	142.57 %
	Algebraic diff.	22.3 s	0.15 ms	0.90 %
	RNN prediction	120.4 s	0.81 ms	4.87 %
	֒→ Data formatting	86.2 s	0.58 ms	3.49 %
	SOCP correction	641.8 s	4.33 ms	25.98 %
	֒→ Initialization	659.0 s	4.45 ms	26.68 %
	Lua/Python bridge	1991.7 s 13.44 ms	80.64 %

Table 4 .

 4

	Min. L 2	N/A	2.19 (46.0)
	WBN-D	0.75 (38.4)	0.89 (29.4)
	WBN-F	1.26 (48.1)	0.77 (47.3)
	4.4 Experiments
	4.4.1 Results on Complete Dataset

1: Force Estimation Errors [N] on Testing Set (16 min) Raw Corrected Force sensors -4.58 (46.1) ground truth

https://github.com/ph4m/eand

(a) α = 0, w = 188.46 (b) α = 1 • 10 3 , w = 236.86 (c) α = 2 • 10 3 , w = 311.80 (d) α = 3 • 10 3 , w = 349.19

https://github.com/jrl-umi3218/ManipulationKinodynamics.

https://www.youtube.com/watch?v=NhNV3tCcbd0

The dataset and algorithms will be released at https://github.com/jrl-umi3218/ WholeBodyKinodynamics.

http://wiki.ros.org/urdf

http://www.comanoid.eu

Acknowledgements Acknowledgements

The work presented in this chapter was partially supported by the FP7 EU RoboHow.Cog project. It resulted in a conference paper [PKQA15b] and a workshop paper [PKQA15a], in collaboration with Ammar Qammaz from FORTH-ICS and Antonis A. Argyros from FORTH-ICS and the University of Crete.

Acknowledgments

The work presented in this chapter was partially supported by the FP7 EU RoboHow.Cog project and the Japan Society for the Promotion of Science (JSPS): Kakenhi B No. 25280096. It was submitted as a journal paper [PKAK16], in collaboration with Nikolaos Kyriazis from FORTH-ICS and Antonis A. Argyros from FORTH-ICS and the University of Crete.

Acknowledgments

The work presented in this chapter was partially supported by the H2020 RIA COMANOID project 3 and the Japan Society for the Promotion of Science (JSPS): Kakenhi B No. 25280096. It resulted in a conference paper [PBCK16], in collaboration with Adrien Bufort and Stéphane Caron from CNRS-UM LIRMM.

other interaction forces with the environment using an additional 6-DoF force-torque sensor (ATI Mini-45) held in the subject's hand. All force-torque measurements are recorded at 100 Hz. We depict our acquisition setup in Fig. 4.1.

Being based on inertial measurements, the motion capture system is prone to drift compared to marker-based tracking methods (e.g., Vicon). We are working on a solution to attenuate this problem. Similarly, wearable force sensors can be of lower accuracy than force plates due to repeated pressure and deformations. Still, a major benefit of our lightweight setup is the efficient and continuous acquisition of kinematics and contact forces on highlydynamic motions through time, which is generally not possible with static force plates. Additionally, the simultaneous monitoring of the whole-body motion and forces allows their correction in two steps. First, low-frequency sensing inaccuracies (e.g., drift) for both A typical iteration at timestamp i is as follows:

1. from the whole-body motion, compute the kinematics-based input features K i 2. feed K i into WBN-D, get raw predicted dynamic features D (raw) i 3. project D (raw) i from G to the global frame to extract contact wrench predictions F k,i

We illustrate the WBN-D architecture in Fig. 4.5a. Although the RNN is expected to implicitly capture the relationship between kinematics and forces, the raw predicted forces are not guaranteed to fully comply with the whole-body equations of motion and friction constraints. Therefore, we compute physically plausible solutions F k in the vicinity of the raw wrench predictions F k using the SOCP of section 4.3.1, with the complete equations of motion of Eq. (4.2) and the discrepancy cost function of Eq. (4.8). This step can be done offline, after the prediction of the complete raw wrench sequence from kinematics alone.

Alternatively, we propose an architecture that implements a feedback loop, WBN-F, allowing the correction of raw wrenches between consecutive predictions:

For prediction, the force distribution sequence is initialized with the distribution D 0 of minimal L 2 norm, as described in Eq. (4.7). Such a distribution is computed from the kinematics alone. Subsequent iterations i are then as follows:

1. fetch the previous dynamic features D i-1

2. from the current whole-body motion, compute the current kinematic features

Publications

Accepted / Published: