Tu-Hoa Pham

Nikolaos Kyriazis

Antonis A Argyros

Senior Member, IEEE Abderrahmane Kheddar

T.-H Pham

• N Kyriazis

Hand-Object Contact Force Estimation From Markerless Visual Tracking

Keywords: Force sensing from vision, hand-object tracking, manipulation, pattern analysis, sensors, tracking. !

We consider the problem of estimating realistic contact forces during manipulation, backed with ground-truth measurements, using vision alone. Interaction forces are usually measured by mounting force transducers onto the manipulated objects or the hands. Those are costly, cumbersome, and alter the objects' physical properties and their perception by the human sense of touch. Our work establishes that interaction forces can be estimated in a cost-effective, reliable, non-intrusive way using vision. This is a complex and challenging problem. Indeed, in multi-contact, a given motion can generally be caused by an infinity of possible force distributions. To alleviate the limitations of traditional models based on inverse optimization, we collect and release the first large-scale dataset on manipulation kinodynamics as 3.2 hours of synchronized force and motion measurements under 193 object-grasp configurations. We learn a mapping between high-level kinematic features based on the equations of motion and the underlying manipulation forces using recurrent neural networks (RNN). The RNN predictions are consistently refined using physics-based optimization through second-order cone programming (SOCP). We show that our method can successfully capture interaction forces compatible with both the observations and the way humans intuitively manipulate objects, using a single RGB-D camera.

INTRODUCTION

H APTICS is of fundamental importance to interact with objects and perceive their physical and functional properties. Recent work has showed how the latter could be inferred from vision [START_REF] Gupta | Observing human-object interactions: Using spatial and functional compatibility for recognition[END_REF], [START_REF] Zhu | Understanding tools: Taskoriented object modeling, learning and recognition[END_REF], [START_REF] Ye | What can i do around here? deep functional scene understanding for cognitive robots[END_REF]. In contrast, human manipulation remains little understood at the level of the underlying interaction forces, which are traditionally measured using force transducers. The latter are costly, cumbersome, and intrusive on both the object and the human haptic sense. Recent advances in markerless visual tracking enabled the non-intrusive monitoring of hand-object motions. Computer vision techniques would thus be an ideal substitute for current force sensing technologies. This is an extremely challenging perspective, as tracking hand-object interactions is difficult due to strong mutual occlusions. Even when object and motion are fully known, the force estimation problem is ill-posed or indeterminate in multi-contact, i.e., there exists an infinity of compatible force distributions (e.g., of varying grip strengths). While it is possible to compute physically plausible forces, capturing the real forces applied is an open problem in multiple fields (Section 2). Kinesiology research produced successful attempts at modeling grip forces by inverse optimization, e.g., during static prehension [START_REF] Niu | Reconstruction of the unknown optimization cost functions from experimental recordings during static multi-finger prehension[END_REF] or two-finger circular motion [START_REF] Slota | Grip forces during object manipulation: experiment, mathematical model, and validation[END_REF]. Albeit of limited scope, these suggest that it may be possible to construct a general model on human grasping, provided a rich dataset on manipulation kinodynamics (motion and forces). In our work, we show that physics-based optimization can be used together with learning to capture manipulation forces with a single RGB-D camera.

• We construct the first large-scale dataset on human manipulation kinodynamics, containing 3.2 hours of highfrequency measurements under 193 different objectgrasp configurations (Section 3). • We propose a force estimation framework that relies simultaneously on a recurrent neural network (RNN) to predict forces that are consistent with the way humans naturally manipulate objects, and on a secondorder cone program (SOCP) guaranteeing the physical correctness of the final force distribution (Section 4). • We thoroughly validate our approach on ground-truth measurements (Section 5) and show that it can seamlessly be extended to visual tracking (Section 6).

Our dataset is dedicated to static prismatic grasps, i.e., with the thumb in direct opposition to antagonist fingers. We discuss instrumentation limitations and show that the optimization-learning framework can still address scenarios beyond the focus of our study (Section 7). Finally, we thoroughly discuss the current limitations, extensions and applications of our work (Section 8). A preliminary version of this research, estimating normal forces from vision, appeared in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]. Our current study extends the latter idea and includes an improved formulation of the optimization and learning models accounting for 3D time-coherent forces, as well as algorithmic descriptions and extensive validation experiments that have not been presented before. To foster the research in this new topic, we make the manipulation kinodynamics dataset publicly available 1 .

RELATED WORK

Monitoring Hand-Object Interactions. Current force transduction and sensing technologies are costly and require frequent calibration. Mounting them onto objects biases physical properties such as shape, mass distribution and friction, while mounting them onto hands obstructs the human haptic sense, limiting the natural range of motion. In contrast, there is evidence that fingertip forces can be correlated to changes in the coloration of fingernails and surrounding skin [START_REF] Mascaro | Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction[END_REF], [START_REF] Sun | Estimation of fingertip force direction with computer vision[END_REF], [START_REF] Urban | Computing grip force and torque from finger nail images using gaussian processes[END_REF]. The latter setups already suggest that computer vision can measure touch forces.

In conjunction with force transducers, marker-based motion capture was used in [START_REF] Kry | Interaction capture and synthesis[END_REF] to estimate hand joint compliance and synthesize interaction animations. Motion capture markers being arguably invasive and difficult to deploy in daily activities, the topic of markerless hand tracking was introduced in [START_REF] Rehg | Visual tracking of high dof articulated structures: an application to human hand tracking[END_REF] and lately received renewed attention in [START_REF] Oikonomidis | Efficient modelbased 3d tracking of hand articulations using kinect[END_REF], [START_REF] De La Gorce | Model-based 3d hand pose estimation from monocular video[END_REF], [START_REF] Keskin | Hand pose estimation and hand shape classification using multi-layered randomized decision forests[END_REF], [START_REF] Qian | Realtime and robust hand tracking from depth[END_REF], [START_REF] Tang | Latent regression forest: Structured estimation of 3d articulated hand posture[END_REF], [START_REF] Tompson | Real-time continuous pose recovery of human hands using convolutional networks[END_REF], [START_REF] Krejov | Combining discriminative and model based approaches for hand pose estimation[END_REF], [START_REF] Rogez | Understanding everyday hands in action from rgb-d images[END_REF], [START_REF] Cai | A scalable approach for understanding the visual structures of hand grasps[END_REF], [START_REF] Huang | How do we use our hands? discovering a diverse set of common grasps[END_REF], [START_REF] Sharp | Accurate, robust, and flexible real-time hand tracking[END_REF], [START_REF] Taylor | Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences[END_REF], [START_REF] Supancic | Depth-based hand pose estimation: data, methods, and challenges[END_REF], [START_REF] Koller | Deep hand: How to train a cnn on 1 million hand images when your data is continuous and weakly labelled[END_REF], [START_REF] Ge | Robust 3d hand pose estimation in single depth images: from single-view cnn to multi-view cnns[END_REF], [START_REF] Sinha | Deephand: Robust hand pose estimation by completing a matrix imputed with deep features[END_REF], [START_REF] Wan | Crossing nets: Combining gans and vaes with a shared latent space for hand pose estimation[END_REF], [START_REF] Ge | 3d convolutional neural networks for efficient and robust hand pose estimation from single depth images[END_REF], [START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF]. During manipulation, hand-object interactions cause mutual occlusions that generative approaches can employ to enforce priors in the optimization process [START_REF] Oikonomidis | Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF], [START_REF]Tracking the articulated motion of two strongly interacting hands[END_REF], [START_REF] Tzionas | Capturing hands in action using discriminative salient points and physics simulation[END_REF], [START_REF] Sridhar | Real-time joint tracking of a hand manipulating an object from rgb-d input[END_REF]. In particular, force models were used to select hand poses that are compatible with the observations through physical simulation [START_REF] Kyriazis | Physically plausible 3d scene tracking: The single actor hypothesis[END_REF], [START_REF] Zhao | Robust realtime physicsbased motion control for human grasping[END_REF], [START_REF] Wang | Videobased hand manipulation capture through composite motion control[END_REF]. In contrast with our approach, these models only need to capture physically plausible distributions rather than the actual forces being applied, which may substantially differ. Biomechanical Models for Human Prehension. Prehension is an active research topic in the kinesiology field; an interest that stems from the remarkable dexterity and complexity of the human hand. Thus, inverse optimization approaches for manipulation have mostly resulted in models that, albeit sophisticated, rely on rather strong simplifications such as on the motion's dimensionality, e.g., static prehension [START_REF] Niu | Reconstruction of the unknown optimization cost functions from experimental recordings during static multi-finger prehension[END_REF]. Other approaches allow limited motion, e.g. circular [START_REF] Slota | Grip forces during object manipulation: experiment, mathematical model, and validation[END_REF], using a simplified grasp model in which individual fingers and hand surfaces are grouped into functional units named virtual fingers [START_REF] Arbib | Coordinated control programs for movements of the hand[END_REF]. A hand holding a cup is thus seen as the thumb on one side and a virtual finger on the opposite side realizing the total wrench due to the four antagonist fingers. Under this formalism, the five-finger grasp is effectively seen as two-finger, and the knowledge of one force fully determines the other. In reality, the force distribution problem is generally indeterminate as five-finger forces can compensate each other to cause the same motion.

The virtual finger model was also applied on nominalinternal forces during 1D translational motions [START_REF] Gao | Internal forces during object manipulation[END_REF]. Internal forces represent the set of forces that humans apply in excess to the nominal forces that are required to create a given motion [START_REF] Kerr | Analysis of multifingered hands[END_REF], [START_REF] Yoshikawa | Manipulating and grasping forces in manipulation by multifingered robot hands[END_REF]. For instance, when holding a cup statically, nominal forces directly compensate gravity, while internal forces secure the object through a firm grip but cancel each other out [START_REF] Mason | Robot Hands and the Mechanics of Manipulation[END_REF], [START_REF] Murray | A Mathematical Introduction to Robotic Manipulation[END_REF]. Past studies showed that humans control internal forces to prevent slip, muscle fatigue or damaging fragile objects [START_REF] Flanagan | Hand movements[END_REF], [START_REF] Gorniak | Manipulation of a fragile object[END_REF], [START_REF] Park | Optimality versus variability: effect of fatigue in multi-finger redundant tasks[END_REF]. Overall, in reviewing several optimization-based models attempting to predict muscle activation patterns, [START_REF] Prilutsky | Optimization-based models of muscle coordination[END_REF] showed that the high redundancy of the human body makes it particularly difficult to identify clear optimization criteria in the way the central nervous system regulates human efforts at the musculoskeletal level.

Force Sensing From Vision. The force sensing from vision (FSV) framework we present is a continuation of our earlier work in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF], that was limited to 1D normal force measurements, four-finger grasps and relatively limited experimental conditions. Normal forces on four-finger grasps were also estimated in the recent work of [START_REF] Ferm Üller | Prediction of manipulation actions[END_REF]. In contrast, our present work is grounded in a new dataset of 3D force measurements on five-finger, diverse manipulation experiments. Our past work used shallow multilayer perceptrons (MLP) to learn internal forces. Such an approach is difficult to generalize as the decomposition into nominal and internal components is not intrinsic, but rather depends on the objective function chosen to minimize nominal forces. While the extended approach we present here still builds upon the formulation of the force distribution problem as a second-order cone program (SOCP) [START_REF] Lobo | Applications of second-order cone programming[END_REF], [START_REF] Boyd | Fast computation of optimal contact forces[END_REF], we also capitalize on the recent success of deep learning applications to manipulation and monitoring of human activities [START_REF] Zhu | Understanding tools: Taskoriented object modeling, learning and recognition[END_REF], [START_REF] Lenz | Deep learning for detecting robotic grasps[END_REF], [START_REF] Koppula | Anticipating human activities using object affordances for reactive robotic response[END_REF] to construct a network that directly learns full 3D manipulation forces, avoiding the need for arbitrary constraints and hand-engineering [START_REF] Bengio | Representation learning: A review and new perspectives[END_REF]. Our work is also inspired by [START_REF] Brubaker | Estimating Contact Dynamics[END_REF], which estimated ground reaction forces from motion capture using a damped spring model. Recently, contact forces were computed for deformable objects [START_REF] Mohammadi | Fingertip force estimation via inertial and magnetic sensors in deformable object manipulation[END_REF] and conversely by considering the human body elastic [START_REF] Zhu | Inferring forces and learning human utilities from videos[END_REF]. [START_REF] Rogez | Understanding everyday hands in action from rgb-d images[END_REF] showed that forces play a crucial role to understand hand-object interactions from vision and noted the challenge of obtaining ground-truth contact points and forces humans use instinctively, which we address in our work.

MANIPULATION KINODYNAMICS DATASET

Public datasets have benefitted multiple research topics such as scene understanding [START_REF] Koppula | Semantic labeling of 3d point clouds for indoor scenes[END_REF], [START_REF] Lai | Unsupervised feature learning for 3d scene labeling[END_REF], visual tracking [START_REF] Tompson | Real-time continuous pose recovery of human hands using convolutional networks[END_REF], [START_REF] Sung | Unstructured human activity detection from RGBD images[END_REF] and robotic grasping [START_REF] Saxena | Robotic grasping of novel objects using vision[END_REF], [START_REF] Walsman | Benchmarking in manipulation research: The YCB object and model set and benchmarking protocols[END_REF]. In contrast, datasets viewing manipulation not only from the angle of vision but also of touch have been more scarce so far [START_REF] Kry | Interaction capture and synthesis[END_REF]. We introduce a new, extensive dataset on human manipulation kinodynamics.

Experimental Setup

While using real objects may initially seem ideal, instrumenting them with force and motion sensors is impractical, making data collection difficult and lengthy. Additionally, physical properties of arbitrary objects (e.g., inertia matrices) are seldom publicly available and must therefore be manually identified [START_REF] Schedlinski | A survey of current inertia parameter identification methods[END_REF], [START_REF] Bhat | Computing the physical parameters of rigid-body motion from video[END_REF]. Finally, the instrumentation may result in measured forces that substantially differ from those that would have been applied on the original objects.

Instead, we construct dedicated instrumented devices, pictured in Fig. 1. Two symmetric parts (for thumb and antagonist fingers) form a base holding an attitude and heading reference system (AHRS, Xsens MTi-300). Sensor plates are mounted on both sides, on which 3D precision force transducers (Tec Gihan USL06-H5-50N) can be positioned by 8 mm steps on their surface. Thickness layers can be inserted to increase the grasp width by 5 mm increments, bringing its range between 46 mm and 86 mm. The force transducers are fitted with support caps of different surface textures: PET, sand paper of grit 40 (coarse), 150 (medium) and 320 (fine). The mass distribution can be adjusted with balancing weights inside and on the surface of the instrumented device. We 3D-print four sets of instrumented modules, with sensor plates of dimensions 80×152, 56×152, 80×96 and 56×96 mm 2 . This setup allows collecting force and kinematics measurements under diverse grasp poses, friction conditions and mass distributions, obtained from the CAD models of the individual components.

The Dataset

Eleven right-handed volunteers (three females, eight males), took part in our experiments. Each subject performed series of eight manipulation experiments as follows. For each series, the subject was given an instrumented device of random shape, thickness and surface texture, with the AHRS either at the top or the bottom, and at random an additional 400 g mass inside. Before each trial, force transducers were placed at the subject's grasp preference. Each trial consisted in the subject grasping the object and manipulating it for approximately 60 s. Every 10 s, to ensure the diversity of motion and forces in the final dataset, the subject was given randomly chosen instructions on speed, direction and task (e.g., slow forward pouring motion, fast left and right oscillations). After each trial, a 50 g balancing weight was attached to a randomly chosen side, excluding sensor plates. Trials could be interrupted when the object became uncomfortable to manipulate. Throughout eight trials, we thus measured the effect of mass variations between 0 g and 350 g, or 400 g and 750 g with the additional internal mass, arranged differently across series. Overall, we collect motion and force measurements for 3.2 hours of manipulation experiments under 193 conditions of motion, friction, mass distribution and grasp. Precisely, we counted 71 unique grasp shapes in terms of fingertip poses relative to each other and 193 unique grasps relative to the object's center of mass. For each experiment, we provide: the global orientation q, rotational velocity ω and translational acceleration a measured by the AHRS at 400 Hz; 3D force measurements expressed in the reference frame of the object R obj. , subsampled from 500 Hz to 400 Hz to match the AHRS; the physical properties of the object: mass m, inertia matrix J about the center of mass C; and the grasp parameters: for each finger k ∈ F, the friction coefficient µ k at contact point P c k , and R k = (n k , t x k , t y k ) a local right-handed reference frame with n k the normal to the surface oriented from the finger to the object and (t x k , t y k ) two orthogonal vectors forming a basis of the contact tangential plane. Friction coefficients are estimated by instructing the subjects to press and pull the force transducers until slipping and computing the maximum ratio between tangential and normal forces through the Coulomb friction model:

g k t x k + h k t y k 2 ≤ µ k f k , (1)
with (f k , g k , h k ) the local decomposition of contact force F k :

F k = f k n k + g k t x k + h k t y k . (2)
µ k can also be measured geometrically as the tangent of the maximum angle an object can be tilted before sliding, enabling its characterization even without force transducers. Models dedicated to finger contacts [START_REF] Ciocarlie | Soft finger model with adaptive contact geometry for grasping and manipulation tasks[END_REF] can also be considered for dextrous tasks, e.g., in-hand manipulation.

Equations of Motion and Synchronization

Let F c and τ c be the net force and torque due to individual contact forces, respectively, and F d and τ d the net force and torque due to non-contact forces. The Newton-Euler equations of rigid body motion at the center of mass are:

F c = ma -F d τ c = J q • α + ω × (J q • ω) -τ d , (3)
with J q the inertia matrix at orientation q and α the rotational acceleration of the object, obtained by numerical differentiation of the AHRS rotational velocity measurements ω. Typically, F d = mg and τ d = 0, with g the gravity vector (g ≈ -9.81z, z the vertical vector pointing up in the world frame). Intuitively, Eq. ( 3) means that F c and τ c can be computed and as a function of purely kinematic terms, as well as from individual finger forces (see Eq. ( 10)). By computing in R obj. net contact forces F kin c from AHRS kinematics and F dyn c from force transducers, both are synchronized temporally by computing, by cross-correlation, the best delay ∆T such that F kin c (t) = F dyn c (t + ∆T ). Second, both AHRS and force transducers are subject to measurement errors (manufacturer specifications: ±0.3 m • s -2 maximum AHRS translational acceleration error, ±1 N maximum force error per transducer). Acceleration errors ∆a result in net force errors m∆a, e.g., ±0.15 N for a 0.5 kg object. In contrast, individual force transducer errors can potentially add up to ±5 N over five fingers. In practice, we measured an average net force discrepancy of 0.33 N between AHRS and force transducers across the dataset. For each experiment, we compute the average net force ∆F c and torque ∆τ c discrepancies between AHRS and force transducers signals. We align the latter (noisier) onto the former by computing offsets (∆F k ) k∈F that are minimal and best result in ∆F c and ∆τ c , through three costs:

                       C var ((∆F k ) k ) = k∈F ∆F k 2 2 C F c ((∆F k ) k ) = ∆F c - k∈F [∆F k ] 2 2 C τ c ((∆F k ) k ) = ∆τ c - k∈F --→ CP k × ∆F k 2 2 (4)
We compute minimal force transducer offsets by solving:

min ∆F k {C var + C F c + C τ c } (5)
In practice, we normalize C F c and C τ c with the initial discrepancies ∆F c and ∆τ c , respectively. We solve Eq. ( 5) by sequential least squares programming [START_REF] Kraft | Algorithm 733: Tomp-fortran modules for optimal control calculations[END_REF] and correct the force transducer measurements with the resulting offsets.

FORCE MODEL

From Eq. ( 3), only net forces and torques are determined by the object's motion. We reconstruct individual finger forces by combining physics-based optimization and learning.

Physics-Based Optimization for Manipulation

We construct forces causing a given motion by solving a second-order cone program (SOCP) [START_REF] Lobo | Applications of second-order cone programming[END_REF], [START_REF] Boyd | Fast computation of optimal contact forces[END_REF] of the form:

min C(x) = 1 2 x T Px + r T x (6) s.t. A j x + b j 2 ≤ c T j x + d j , j = 1, . . . , m (7) Ex ≤ f (8) Gx = h, (9)
with x = (f 1 , g 1 , h 1 , . . . , f 5 , g 5 , h 5 )
T the force components of Eq. ( 2), taken as 15 optimization parameters. We now define constraint matrices

P, r, A j , b j , c j , d j , E, f , G, h.
Positivity. With the convention that each contact normal n k is oriented inwards the object, normal forces f k are nonnegative: ∀k ∈ F, f k ≥ 0. In Eq. ( 8), E is a selection matrix (with only 0 and 1) such that Ex = ((f k ) k∈F )

T and f = 0.

Friction. We define one friction constraint (see Eq. ( 1)) per finger, i.e., m = 5 in Eq. [START_REF] Mascaro | Photoplethysmograph fingernail sensors for measuring finger forces without haptic obstruction[END_REF].

∀k ∈ F, A k is a selection matrix such that A k x = (g k , h k ) T , c k is a selection vector (with only 0 and µ k ) such that c T k x = (µ k f k ),
F c = k∈F F k and τ c = k∈F --→ CP k × F k . ( 10
)
We formulate 6 equality constraints by projecting Eq. ( 10) in the world frame

R W = (v 1 , v 2 , v 3
). In Eq. ( 9), G and h are of respective sizes 6×15 and 6×1, with: ∀i = 1, . . . , 3; ∀j = 1, . . . , 15; ∀k = 1, . . . , 5;

G(i, j) =          n k • v i if j = 3(k -1) + 1 t x k • v i if j = 3(k -1) + 2 t y k • v i if j = 3(k -1) + 3 0 otherwise G(i + 3, j) =                --→ CP k × n k • v i if j = 3(k-1)+1 --→ CP k × t x k • v i if j = 3(k-1)+2 --→ CP k × t y k • v i if j = 3(k-1)+3 0 otherwise h(i, 1) = F c • v i and h(i + 3, 1) = τ c • v i . (11)
Cost. Physically plausible forces can be computed with a cost depending only on x, e.g., minimal L 2 norm [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]:

C L 2 (x) = k∈F f k 2 +g k 2 +h k 2 = k∈F F k 2 2 . (12)
Yet, the resulting forces can significantly differ from those humans actually apply (see Fig. 2). Instead, we consider a cost minimizing the discrepancy with given target forces F k :

C F k (x) = k∈F F k -F k 2 2 (13)
In Section 4.2, we take F k as force transducer measurements to correct sensing uncertainties. From Section 5.1 onwards, target forces F k are neural network force predictions. We depict the SOCP force correction architecture in Fig. 3.

Learning Features

The dataset parameters of Section 3 fall into three categories:

• Object and grasp parameters: location of the center of mass C in R obj. , mass m, inertia matrix J, contact point locations P k and friction coefficients µ k . • Kinematic parameters: appearing in Eq. ( 3) are the object's orientation q in R W , rotational velocity ω, rotational acceleration α and translational acceleration a. q, ω, a are directly measured by the AHRS. α is obtained by simple numerical differentiation of ω. Alternatively, the relevant kinematic parameters can be

min (Fk) k k∈F F k -F k 2 2
Fig. 3. By SOCP, we extract force distributions compatible with the observed motion in the vicinity of target forces (measured or predicted).

obtained from visual tracking, through double differentiation of the object's pose and orientation. • Force transducer measurements F k . To alleviate sensing uncertainties, we extract physically plausible force distributions F k in the vicinity of the possibly inaccurate measurements F k , as depicted in Fig. 3. The objective is then to learn F k based on input parameters that depend only on the grasp, the object and its kinematics. We select these input features based on their contribution to the equations of motion. A first approach could be to take the raw parameters listed above. However, their influence is often determined not individually but rather in interaction with other parameters. From Eq. ( 10), the positions of the center of mass C and contact points P k are meaningful not on their own but in relation to each other as from Eq. ( 3), we summarize the contributions of m, a, J, q, ω, α into the target net contact force F c and torque τ c .

--→ CP k . Similarly, KDN-FH-F Recurrent Neural Network Linear Layer Ki Di-1 Di (a) KDN-FH-F: full hand forces. KDN-VF-F KDN-VF-F (th.-VF) RNN LL K (th.-VF) i D (th.-VF) i-1 D (th.) i KDN-VF-F (ant.) RNN LL K (ant.) i D (ant.) i-1 D (ant.) i K i D i-1 D i (b) KDN-VF-F: two-stage thumb-virtual finger network.
Recall that F c and τ c are expressed in R W . Since the dataset focuses on static grasps, for each experiment, the contact points are constant in any frame attached to the object. We account for translational and rotational invariances by projecting F c , τ c and --→ CP k on R obj. . Thus, the input features stemming from the Newton-Euler equations are:

∀(k, v) ∈ F × R obj. ,      p F c v = F c • v p τ c v = τ c • v p P k v = --→ CP k • v . (14)
With p µ = µ k k∈F the average friction coefficient, we regroup these parameters, derived from the grasp-object properties and kinematics, into a 22-element vector K:

K = p F c v , p τ c v , p P k v , p µ (k,v)∈F ×R obj. ( 15
)
In particular, attaching R obj. to the thumb frame R th. = (t x 0 , t y 0 , n 0 ), with t y 0 towards the palm, also helps preserve grasp invariances as the four antagonist fingers now share the same coordinate along n 0 , reducing K to 19 elements. Also expressing forces in R th. yields a 15-element vector D:

D = (F k • v) (k,v)∈F ×R obj. ( 16
)
Convertly, finger forces F k are directly obtained from D by projecting R obj. coordinates into their local reference frames.

Neural Network Modelling

Given an object-grasp configuration, we aim at constructing a mapping F between motion and forces such that D = F (K). In [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF], F was modeled with an MLP learning internal forces. Yet, as contact is maintained, forces D i at timestep i should depend on the current motion K i and the past:

D i = F (K i , D i-1 ), (K j , D j-1 ) j=1,i-1 (17)
This formulation accounts for temporal continuity and helps learn force-motion trajectories rather than single sample associations. We model this sequential structure using recurrent neural networks (RNN) [START_REF] Elman | Finding structure in time[END_REF] with long short term memory (LSTM) neurons [START_REF] Hochreiter | Long short-term memory[END_REF], that allow for better learning of long-term dependencies. We investigate four kinodynamics network (KDN) architectures. The first model, KDN-FH-F, predicts full hand forces D i from the current kinematics K i and previous distribution D i-1 using a single RNN:

D i = KDN-FH-F(K i , D i-1 ). (18)
Second, we propose a two-stage network, KDN-VF-F, inspired by the virtual finger model. We reduce the four antagonist fingers F ant. to a virtual finger associated to their centroid p Pant. v = p P k v k∈Fant. and average friction coefficient p µant. = µ k k∈Fant. , thus yielding reduced task parameters:

K (th.-VF) i = p F c v , p τ c v , p P th.
v , p µ th. , p Pant. v , p µant. v∈R th.

.

A first RNN then estimates thumb forces D (th.) i separately:

D (th.) i = KDN-VF-F (th.-VF) K (th.-VF) i , D (th.) i-1 . (20)
Given thumb force estimates F th. , force-torque parameters due to antagonist fingers are p

F ant. v = (F c -F th. ) • v and p τ ant. v = τ c - ---→ CP th. × F th.
•v, yielding task parameters:

K (ant.) i = p F ant. v , p τ ant. v , p P k v , p µant. (k,v)∈Fant.×R th. (21)
A second RNN then learns antagonist finger forces D (ant.) i :

D (ant.) i = KDN-VF-F (ant.) K (ant.) i , D (ant.) i-1 (22)
We depict KDN-FH-F and KDN-VF-F in Fig. 4. Finally, we introduce alternative versions of KDN-FH-F and KDN-VF-F that associate current kinematics K i and past forces D i-1 to force variations ∆D i . In doing so, we explicitly associate the same output to two sequences that differ by constant internal forces. We denote these alternative architectures by KDN-FH-∆ and KDN-VF-∆. Full forces are then computed by summing up consecutive predicted force variations.

EXPERIMENTS

We train the four KDN architectures on the manipulation kinodynamics dataset, down-sampled from 400 to 60 Hz for compatibility with vision-based kinematics (Section 6), and split for training (60 %), validation (20 %) and testing (20 %). In KDN-FH-F and KDN-FH-∆, the RNN contains two hidden-layers of size 256. In KDN-VF-F and KDN-VF-∆, both RNNs contain a single hidden-layer of size 256. We implement and train the networks within the Torch7 framework [START_REF] Collobert | Torch7: A matlablike environment for machine learning[END_REF] using stochastic gradient descent with a mean square error criterion and dropout [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF] to avoid overfitting.

Force Reconstruction Model

As physics is not explicitly enforced by the RNNs, raw force predictions can be inconsistent with the observed motion.

We depict such open-loop predictions in Fig. 5a. Using these as target forces F k in the SOCP of Fig. 3 allows the extraction of physically plausible forces in their vicinity. This can be done after collecting a complete sequence of openloop predictions. We depict this offline correction process in Fig. 5b. However, as the RNNs are trained on physically

Object-grasp param., kinematics (i) Object-grasp param., kinematics (i)

KDN K i D (raw) i-1 (i -1) (i + 1) D (raw) i K i+1 KDN Raw
KDN K i D (CL) i-1 SOCP (i -1) (i + 1) SOCP D (raw) i D (CL) i K i+1 KDN
(c) Closed-loop: correct force predictions and feed back to the next step. consistent data, repeatedly feeding inconsistent motionforce sequences may lead to growing errors. Force predictions can be corrected between time steps by integrating the SOCP in closed-loop with the KDN, depicted in Fig. 5c. We compute the estimation errors (average and standard deviation) for the four network architectures using openloop prediction, offline correction or closed-loop prediction, on complete manipulation sequences with D 0 initialized from ground-truth forces in Eq. [START_REF] Tompson | Real-time continuous pose recovery of human hands using convolutional networks[END_REF]. We report the results in Table 1, highlighting for each neural network architecture the lowest mean and standard deviation errors across correction processes. Humans being able to apply infinitely many force distributions to produce the same motion results in some intrinsic variability even in performing the same task repeatedly. Indicatively, standard deviations values between 0 and 2 N were reported on static finger pressing tasks in [START_REF] Scholz | Understanding finger coordination through analysis of the structure of force variability[END_REF]. We argue that this intrinsic variability is likely greater in the case of unconstrained, dynamic motions, though difficult to measure confidently.

We observe from Table 1 that offline correction and closed-loop prediction outperform open-loop prediction on all architectures. This is especially the case for the networks estimating force variations ∆D i , as these tend to be rather unstable and prone to drift. For instance, in Fig. 6, the openloop predictions rapidly drift away from the net force and torque producing the target kinematics. Additionally, the individual normal forces become negative, which would mean that fingertips pull rather than press on the contact surface. Offline correction searches for physically valid forces in the vicinity of negative raw predictions, finally yielding distributions of minimal norm. In contrast, closed-loop prediction can help the network recover from incorrect predictions and maintain human-like grasping forces. Overall, the networks predicting force distributions generally perform better than those estimating force variations. For the former, offline correction does not appear to significantly improve the open-loop estimations, which shows that these RNNs are rather successful at capturing the relationship between kinematics and underlying forces. Finally, the better accuracy of KDN-VF-F indicates that the virtual finger model can be a useful tool to decouple the static indeterminacy stemming from the thumb and antagonist fingers. Still, the two-stage architecture makes KDN-VF-∆ more prone to drift since thumb force predictions cannot be corrected alone before computing the antagonist forces.

Force Drift over Time

Due to the infinity of force distributions compatible with a given motion, the force predictions are likely to deviate from the transducer measurements over time. We quantify this effect by splitting the experiments into sub-sequences of maximum duration 1, 2, 4, 8, 16, 32 s and computing the resulting estimation errors for the four architectures with ground-truth initialization and either offline correction or closed-loop prediction. For completeness, we reproduce the force estimation errors over the full length sequences (aver-age duration 60.1 s, standard deviation 3.8 s). We report the results in Table 2, highlighting for each duration the lowest errors across the proposed architectures.

In line with the observations made on the full-length experiments, KDN-VF-∆ is the worst-performing network for every sequence duration, whereas KDN-VF-F is consistently best-performing in terms of standard deviation. This indicates again that decoupling thumb and antagonist redundancies is a viable strategy, yet more unstable in the presence of force variation uncertainties. We also observe that KDN-FH-F outperforms KDN-VF-F on sequence durations 4, 8, 16 and 32 s in terms of average error. These results are representative of two partially competing goals in force estimation. On one hand, estimating the full hand forces together, the KDN-FH-F architecture tends to predict an average plausible distribution based on past motion-forces associations. On the other hand, the separate estimation of the thumb force in the first stage of the virtual finger architecture explicitly decouples the major indeterminacy between thumb and antagonist fingers, enabling KDN-VF-F to better track the individual variability across subjects. We also observe that KDN-FH-∆ yields better results than its full force counterpart KDN-FH-F on the 1 s sequence duration and 2 s to a lesser extent. Recall that the ∆D i networks were introduced to accommodate the possibility of having the same motion caused by an infinity of force distributions. It appears here that KDN-FH-∆ is better at matching the real force variations on short sequences. Still, the applicability of this result on real manipulation tasks is hindered by, first, the accumulation of ∆D i prediction errors over time, and second, the accuracy of the predicted force sequence being contingent on its initialization from real force measurements.

Force Sequence Initialization

Manipulation forces are sequentially computed based on an initial distribution that can be adjusted freely. We assess the force variability following non ground-truth initialization for sequences of maximum duration 4.0, 8.0, 16.0 and 32.0 s. Each sequence is initialized as follows. Using the average and standard deviation µ, σ of each finger force throughout the manipulation kinodynamics dataset, we pick a random sample D 0 following the normal distribution N (µ, σ). We then correct D 0 using the SOCP of Section 4.1. Thus, we ensure that the resulting distribution D 0 is compatible with the initial kinematics K 0 . We report the force estimation errors for random and ground-truth initialization in Table 3, highlighting, for each duration and architecture, the lowest errors between the two initialization methods.

Expectedly, ground-truth initialization yields better force estimates overall. Still, for each architecture, the performance difference decreases with the sequence duration. Indeed, even when starting from the same distribution, the predicted sequence is likely to deviate from the transducer measurements due to the infinity of force variations producing the same motion. This mitigates the importance of the force initialization over time. In the case of the bestperforming network, KDN-VF-F (closed-loop), the difference is actually minor even starting from 8.0 s sequences. Finally, note that for any initial force distribution, the re- sulting sequence is constructed to be physically plausible given the observed motion and compatible with the forces a human could likely apply, based on the manipulation kinodynamics dataset. This allows the generation of force sequences following different profiles for the same motion (e.g., light or strong starting grasp). This method can also be used to reinitialize the prediction model when the resulting distributions are unreliable, as it may happen in the presence of motion tracking discontinuities.

FORCE SENSING FROM VISION

We propose to estimate finger forces from markerless visual tracking, circumventing any instrumentation whatsoever.

Model-Based Tracking

Force estimation requires the kinematics of the object and contact point locations. Due to the intensity of the interaction (firm grasps, high velocities) and its observability (occlusions, motion blur), the 3D state of the object cannot be computed independently of that of the hand. The requirement for joint tracking in such situations is discussed in [START_REF] Oikonomidis | Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF], [START_REF]Tracking the articulated motion of two strongly interacting hands[END_REF], [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF]. We thus tackled hand-object tracking using the approach of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF]. A method that automatically computes contact points for force estimation was presented in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]. This method did not apply well in our case due to the larger variety of motions (high velocities) and objects (easily occluded). In our preliminary experiments, tracking the hand together with the object produced both inaccurate object poses and contact locations (in the order of 20 mm), see Fig. 7a. To assess the effect of contact point errors, we decomposed force transducer measurements on AHRS kinematics (see Fig. 3), using either ground-truth contact points or perturbed by a Gaussian noise of standard deviation 0 to 20 mm. We depict the resulting force reconstruction errors in Fig. 7b. Net forces are only determined by the motion, hence zero mean errors. However, different contact point enable different ways of distributing the same net forces, hence growing force variability. Since the best performing KDN architecture achieves a lowest standard deviation error of about 2.60 N, we consider it as an upper bound on acceptable force reconstruction errors due to kinematics. Beyond that (possibly even before, due to uncertainties adding up), contact point errors are guaranteed to counterbalance the accuracy of the force predictions. 20 mm contact point errors yield about 3 N force reconstruction error (std. dev.), and thus cannot be used for accurate force prediction.

In our work, all recorded motions and forces are annotated with ground-truth contact locations (i.e., that of the force transducers), which we use to circumvent their estimation. We implement the hypothesis that the grasp pose does not change noticeably during manipulation and thus consider the object and the hand as a single, rigid compound. This reduces the problem of tracking an articulated hand and a rigid object (34D) to only a rigid object (7D), significantly increasing efficiency and robustness under mutual occlusions. The origin of the rigid compound in object space is set to the origin of the object alone in object space. This makes the rigid compound's trajectory that of the object itself. To implement this hypothesis in 3D tracking, we 1) track both hand and object during the entire sequence using [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF], 2) select a frame to form the rigid compound and 3) track the rigid compound only using [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF].

Step 1) is fully automatic. We use [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] to generate an initial estimate of the trajectories for the hand and the object. Despite noise, fast motion and the lack of any problemspecific prior, the result is visually plausible but the truth that fingers touch the object at the known contact points is not reconstructed accurately enough. To have the method generate results of higher fidelity, a large computational budget is given to the optimizer (particle swarm optimization, PSO [START_REF] Eberhart | Swarm Intelligence[END_REF]), namely 100 particles and 100 generations per entity (hand, object). The GPU-parallelized implementation of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] yields a processing rate of 0.5fps.

Step 2) is partially automatic. Its goal is to yield a handobject compound that best matches the actual grasp realized by the subject. To this end, from the entire sequence, a single frame is sought whose back-projection maximizes a) the image area corresponding to the hand, b) the viewable surface percentage of the hand's 3D mesh, c) the image area corresponding to the object, d) the viewable surface percentage of the object's 3D mesh and e) in which the hand grasps the object. Though we performed this selection empirically, we believe it can clearly be automated by quantifying each of the aforementioned criteria and formulating an objective function to be optimized over frame indices. Currently, due to manual intervention, this step takes the most time. For the selected frame, a single tracking step of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] is executed anew, but with an additional term that favors hand-object hypotheses where the fingertips touch the known contact points. More specifically, let E(h, o) be the error function used in [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF], for a hand pose h and an object pose o. Then, a new objective function E (h, o) is defined as:

E (h, o) = E(h, o) + λ k∈F S k (h) -T k (o) , (23)
with F the set of fingers, S k (h) the world space position of fingertip k under hand pose hypothesis h, and T k (o) the world space position of the known contact point for finger k under object pose hypothesis o. This new objective is minimized using the optimization scheme of [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] to yield a new hand-object configuration which is a balance between i) matching the observations, as in [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF] and ii) making sure the finger tips touch the contact points. Search for the handobject poses is initialized from the tracking results in the preliminary tracking of Step 1). After optimal hand and object poses are computed, a new geometry is introduced, that is the union of the hand and object geometries, under those new poses. This new geometry models the object to be further tracked. This step is computationally similar to Step 2) and only slightly increased in order to compute the additional term. In our experiments we used λ = 0.0002.

Step 3) is fully automatic. It makes use of the tracking result of Step 1) and the geometry of Step 2) to initialize and track a single rigid compound across the same sequence, again employing [START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF]. Its result is the trajectory of the compound, which also constitutes the trajectory of the object alone. Computationally, this is the lightest tracking step and can be performed at processing rates higher than 30fps.

Kinematics Estimation From Visual Tracking

With the camera calibrated intrinsically and extrinsically such that the vertical direction in the world frame is known (to account for gravitation g, see Section 3.3), we record and process 12 tracking experiments using the following objects: First, the instrumented device of Section 3, in a configuration that does not appear in the training dataset (mass 279 g). Second, three objects used in daily activities, 3D-printed and equipped with AHRS and force transducers for ground truth: a cuboid box (856 g), a small bottle (453 g), and a mug (174 g). We use the latter as an application of the force model on non-prismatic grasps in Section 7.2. We depict sample tracking results in Fig. 8.

Given the pose of the object throughout the experiment, we estimate its first and second-order kinematics (i.e., velocity and acceleration) by numerical differentiation. Due to noise in the estimated trajectory, direct differentiation of the tracked pose signal (e.g., by central difference) results in large spikes in the derivatives, and thus the estimated forces. This effect can be mitigated by smoothing the original signal before differentiation. However, force profiles occurring in manipulation tasks are in fact spiky (see Fig. 6). Thus, smoothing a trajectory comes at the risk of suppressing not only noise, but actual acceleration spikes.

Besides direct differentiation by central difference, we evaluate two smoothing techniques on visual tracking trajectories: Gaussian filtering and algebraic numerical differentiation [START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF]. In the former, the initial signal is smoothed by convolution with a Gaussian function

G(x) = 1 √ 2πσ 2 exp(-x 2
2σ 2 ), with standard deviation σ. Conveniently, differentiating the smoothed signal also amounts to convolving the original signal with the derivatives of G. Denoting by T s = 1/60 s the sampling period, we experimentally choose σ = 3T s and truncate G at ±4σ. Alternatively, we investigate the use of algebraic numerical differentiators [START_REF] Mboup | Numerical differentiation with annihilators in noisy environment[END_REF], which do not depend on the noise statistical properties and previously produced good kinematics estimates in [START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF]. Such filters are characterized by parameters (κ, µ), akin to the Gaussian σ, and a temporal window of half-width T . We empirically choose T = 4T s and κ = µ = 0.5. For each differentiation method, we compare the resulting translational accelerations and rotational velocities to those measured by the AHRS. Indicatively, we quantify the effect of kinematics estimation errors on force distributions by computing distributions that are closest to force transducer measurements as in Fig. 3, but using the kinematics estimates provided by each method. We report the resulting errors in Table 4, highlighting, for each motion and force quantity, the lowest error across differentiation methods. On typical tracking sequences, smoothing techniques appear necessary to compute reliable kinematics estimates. Both Gaussian and algebraic filters yield reasonable force discrepancies despite possible tracking uncertainties and discontinuities. Overall, while the Gaussian filter seems to perform slightly better than the algebraic filter, the latter also requires significantly less samples per estimate. This allows for a shorter lag for real-time applications while also better capturing high frequency force variations, at the cost of a slightly larger sensitivity to tracking noise. To further quantify the effects of tracking errors on contact points, we also compute force reconstruction errors for kinematics estimates perturbed by between 0 and 200 % of the errors reported in Table 4 for Gaussian filtering, depicted in Fig. 7b. In contrast with contact points, kinematics errors directly influence the net force, hence non-zero mean errors, that also grow in standard deviation. Interestingly, adding contact errors has no additional effect on the mean force error as explained in Section 6.1, but having both kinematics and contact errors leads to larger error standard deviations than either individually. This graph is also informative of the level of accuracy that is needed to reach a given precision. For example, the upper bound in admissible force estimation errors of 2.60 N discussed in Section 6.1 is reached at 200 % of the current error baseline for rigid compound tracking (kinematics errors only, red plot). Tracking errors beyond that should systematically be rejected for the purpose of force estimation. Also, the same graph suggests that to reach the 1 N precision that the rigid compound tracking currently achieves (red plot at 100 % baseline errors), yet without enforcing a static grasp (contact and kinematics errors, green plot), contact errors must be below 5 mm and tracking must become twice as accurate (50 % baseline errors).

Force Prediction From Vision-Based Kinematics

Using a single camera, we track manipulation experiments and estimate the object's kinematics with algebraic filtering. In Section 5, although the four network architectures are trained on AHRS data, the object's kinematics is used as an input without consideration of the way it is measured. Thus, the trained networks can seamlessly generate force sequences from vision-based kinematics. In order to be completely independent of ground-truth sensing, we use the random initialization process described in Section 5.3. We compute the resulting estimation errors with respect to ground-truth force transducer measurements, along with, for reference, force predictions derived from the AHRS kinematics; none of these being used in the vision-based estimation process. We report our results in Table 5, highlighting, for each motion type and initialization method, the lowest errors across prediction-correction architectures.

Under the same initialization conditions, forces computed from vision are comparable to forces computed from AHRS measurements. The decrease in accuracy is most noticeable on networks estimating force variations ∆D i due to a higher tendency to drift, as discussed in Section 5, but also additional uncertainties from visual tracking. We depict an example of forces estimated from vision in Fig. 9. Tracking discontinuities (e.g., lost hand-object pose), following second-order differentiation, are perceived by the force estimation framework as acceleration spikes and result in sudden fingertip force variations. These errors accumulate in the case of ∆D i networks since each prediction is directly relative to the preceding sample. When erroneous kinematics can be identified, their impact can be mitigated by reinitializing the prediction process based on the last reliable sample. However, while doing so is straightforward when AHRS measurements are available, it is difficult from the tracked kinematics alone, since acceleration spikes are not necessarily due to discontinuities but can also stem from actual sudden motions. Overall, KDN-VF-F appears the most resilient architecture to visual tracking uncertainties. We depict sample force prediction results in Fig. 10.

DISCUSSION

Visual Tracking Assumptions

In Section 6.1, we suppose the contact points known and use them to compute a static grasp throughout the motion. Note that our force estimation framework itself is independent of the tracking method employed as long as reliable motion and contact information can be provided. The difficulty for us was to collect ground-truth measurements to validate our approach. Therefore, we forced the positioning of the fingertips at desired locations for both the real objects and the visual tracking system. Indeed, to allow arbitrary finger placement, the experimental apparatus should be covered with an array of high-precision 3D force transducers (that are not available in the required dimensions), or alternatively with dedicated force sensing surfaces [START_REF] Stassi | Flexible tactile sensing based on piezoresistive composites: A review[END_REF], generally limited in accuracy and range (e.g., normal forces only). Our force estimation framework can readily challenge in-hand manipulation scenarios with more sophisticated tracking systems (e.g., multi-camera). Again, assessing such tasks is limited by the difficulty of measuring the actual forces without obstructing the subject's haptic sense, which we consider essential in our demonstration. In effect, the tracking method we describe does not introduce any constraint besides those relative to the ground-truth instrumentation, while making it possible to monitor manipulation forces using a single off-the-shelf depth sensor.

Beyond Prismatic Grasps

We evaluate the force estimation framework on a nonprismatic grasp using a mug-shaped device (see Fig. 1f). Force transducers are arranged on a circle, with the contact normals pointing towards the center. We compute force distributions using either visual tracking or AHRS and depict the resulting predictions in Fig. 11. First, we observe that by considering the hand and the object as a single rigid compound, we are able to track the mug fairly accurately using a single depth sensor, despite it being essentially rotationally symmetric, except for a handle that is easily occluded. Second, RNN predictions do not follow the subtle force variations along the normal n k and tangential directions t x k as closely as the tangential direction t y k . Recall from Section 4.2 that the individual t y k per finger are defined, uniformly, as oriented towards the palm. This property is preserved in the case of the mug. However, while for prismatic grasps the n k are collinear with each other and perpendicular to the t x k , couplings appear between and among each set in the case of the mug. Still, the SOCP ensures that the final distributions are physically plausible based solely on the observed kinematics and the object-grasp properties, regardless of the RNN training dataset. We also observe that predicted and measured thumb forces remain rather comparable. This suggests that knowledge acquired from prismatic grasps on the 3-dimensional indeterminacy between thumb and antagonist fingers can generalize and serve as a prior during physics-based optimization (e.g., to compute the thumb force separately and redistribute the complementary antagonist force with a weighted L 2 criterion). Such a prior can also be used to predict forces on grasps involving less than five fingers, with additional SOCP equality constraints ensuring that non-contacting fingers do not exert forces.

While we could imagine extending the force estimation framework further by training new network architectures on arbitrary grasps, this is difficult in practice. The current ground-truth instrumentation captures 11 contact space degrees of freedom (grasp width and 2D tangential finger positions). In contrast, for general grasps, the instrumentation should allow 25 degrees of freedom (5 per finger, ignoring transducer orientations about the normal axes). Due to the greater dimensionality, it would require significantly more experiments to obtain a diverse and extensive dataset, as well as a much heavier experimental setup to fine-tune the position and roll-pitch of each transducer independently.

Computational Performance

On a computer equipped with an Intel i7-4700MQ CPU (quad-core 2.40GHz) and an NVIDIA GTX 780M GPU, we apply the KDN-VF-F closed-loop architecture on the testing dataset (39 experiments, total duration 2470 s, 60 Hz). We report computation times in Table 6. While the total computation time appears greater than the dataset duration, the decomposition per process shows that the current implementation is actually rather sub-optimal and the three core components take only 5.29 ms per sample. First, algebraic differentiators implemented as finite impulse response filters are of minor impact on computation times. Second, RNN predictions are parallelized on the GPU using the Torch7 framework [START_REF] Collobert | Torch7: A matlablike environment for machine learning[END_REF]. Third, SOCP solving is done with the CVXOPT library [START_REF] Andersen | Cvxopt: A python package for convex optimization[END_REF]. A typical iteration is as follows: 1) Given current kinematics and previous corrected forces F i-1 , we construct the RNN input vector (K i , D i-1 ).

2) The network produces a raw force prediction D (raw) i .

3) We assemble SOCP constraint matrices from the target kinematics, and the cost function from D (raw) i . 4) We solve the SOCP and get the corrected forces F i . Steps 1 and 2 are executed in Lua for Torch7, while steps 3 and 4 are executed in Python for CVXOPT. Both being interpreted languages explains part of the overhead in preparing the data for each process. However, the majority of the processing time is actually spent on managing the two interpreters in succession without actually performing calculation (Lua/Python bridge value in Table 6). Thus, simply implementing our method within a unified computational framework would certainly yield a tremendous increase in performance enabling real-time use. Other possible computational improvements include refactoring data structures to reduce redundancies and update constraint matrices only when needed, initializing the SOCP search at the RNN predictions, and rewriting the physical plausibility problem as a quadratic program (QP) with a discretized friction cone.

CONCLUSION AND FUTURE WORK

Our work establishes that monitoring hand-object interaction forces at the fingertip level can be addressed in a cheap, reliable and transparent way using vision. Based on the first large-scale dataset on manipulation kinodynamics, our approach estimates force distributions that are compatible with both physics and real human grasping patterns. While the case of static prismatic grasps may appear restrictive, this limitation is only relative to the instrumentation required to collect ground-truth measurements. Provided an extended experimental setup, we expect that our method can seamlessly extend to arbitrary grasps. Besides, the current SOCP formulation is independent of the training dataset and always produces forces causing the observed motion. Finally, even limited to prismatic grasps, the estimation of 3D forces for all five fingers on arbitrary motions greatly extends the state of the art in interaction capture. Our approach is achieved with a single RGB-D camera, which enables its use for monitoring of human activities and robot learning from demonstration in daily settings.

Our approach is readily compatible with any tracking method providing accurate object kinematics. We present such qualitative results [START_REF] Krull | 6-dof model based tracking via object coordinate regression[END_REF], [START_REF] Issac | Depth-based object tracking using a robust gaussian filter[END_REF] in the supplementary material 2 , using hand-picked contact locations. To monitor non rigid grasps, we aim to apply the force estimation framework in conjunction with tracking to guide the pose search as an implicit model for grasp plausibility and realism [START_REF] Pham | Capturing and reproducing hand-object interactions through visionbased force sensing[END_REF]. Our work would also benefit from future advances in hand tracking and force transducing technologies. The former would facilitate data acquisition on diverse objects for learning on visual data only. Individual finger movements (e.g., contact sliding or fingertip deformations) could yield subtle clues on the subject's perceived effort and be informative of even slight changes in manipulation forces. Our future work involves alleviating current instrumentation limitations and implementing soft finger contact models [START_REF] Ciocarlie | Soft finger model with adaptive contact geometry for grasping and manipulation tasks[END_REF] for dextrous manipulation. The generalization to arbitrary grasps could be addressed by considering the variability of manipulation forces as an inverse optimal control problem involving physiological criteria, e.g., grasp efficiency [START_REF] Zheng | Evaluation of grasp force efficiency considering hand configuration and using novel generalized penetration distance algorithm[END_REF]. It would also be valuable to find a systematic way to quantify visual tracking accuracy requirements to achieve a target force estimation precision. In the long term, we plan to extend the force estimation framework to general articulated bodies for bi-manual grasping and whole-body interactions [START_REF] Pham | Multi-contact interaction force sensing from whole-body motion capture[END_REF].

Fig. 1 .

 1 Fig. 1. Instrumented devices of adjustable physical and grasping properties (a-c), or based on everyday objects to allow intuitive interactions (d-f).

  and b k = d k = 0. Equations of motion. From Eq. (3), we compute net contact force F c and torque τ c from kinematics only. By construction, F c and τ c are linked to individual finger forces by:

Fig. 2 .

 2 Fig.2. Force distributions computed only by physics-based optimization are guaranteed to result in the observed motion (net force and torque) but can significantly differ from the real distributions at the finger level.

Fig. 4 .

 4 Fig. 4. Current forces are predicted from current motion and past forces.

  force predictions (a) Open-loop: force predictions are not checked for physical plausibility.Object-grasp param., kinematics (i) Offline correction: fix open-loop predictions without propagating.

Fig. 5 .

 5 Fig. 5. Open-loop, offline correction and closed-loop force prediction.

Fig. 6 .

 6 Fig.6. Open-loop, offline-corrected and closed-loop force predictions for KDN-VF-∆. Open-loop forces drift away from physically plausible forces, becoming negative. Compatibility with the observed motion is enforced through offline correction or closed-loop control at each time step.

Force

  error std. dev. [N] (b) Propagation of contact point and kinematics estimation errors to force reconstruction.

Fig. 7 .

 7 Fig. 7. Tracking the articulated hand pose together with the object rigid motion produces (a) inaccurate object and contact positions that (b) propagate to forces. Baseline errors (std. dev.): contact: 10 mm (pos.), kinematics: 2.9 m • s -2 (trans. acc.), 30.9 rad • s -1 (rot. vel.) .

Fig. 8 .

 8 Fig. 8. The hand and the object are tracked as a rigid compound.

Fig. 9 .

 9 Fig. 9. Forces from closed-loop KDN-VF-F and random initialization.

Fig. 10 .

 10 Fig. 10. Forces predicted from vision (red) vs. ground-truth (grey).

Fig. 11 .

 11 Fig. 11. Force estimates with non-prismatic grasp (mug).

TABLE 1 Force

 1 Estimation Errors on Complete Sequences -Mean (Std. Dev.) [N]

		Open-loop	Offline corr.	Closed-loop
	KDN-FH-F	0.49 (4.14)	0.44 (4.07)	0.16 (3.54)
	KDN-FH-∆	-43.67 (156.72)	0.60 (4.74)	0.50 (11.03)
	KDN-VF-F	0.29 (3.19)	0.29 (3.13)	0.12 (2.60)
	KDN-VF-∆	1145.06 (3984.86)	3.54 (11.80)	2.32 (6.60)

TABLE 2 Force

 2 Estimation Errors for Proposed Prediction-Correction Architectures over Increasing Sequence Durations -Mean (Std. Dev.) [N]

	Sequence duration	1.0 s	2.0 s	4.0 s	8.0 s	16.0 s	32.0 s	Full length
	KDN-FH-F, offline correction	-0.21 (2.06) -0.21 (2.43) -0.13 (2.86) -0.04 (3.22)	0.07 (3.54)	0.19 (3.76)	0.44 (4.07)
	KDN-FH-F, closed-loop	-0.13 (2.20) -0.12 (2.47) -0.07 (2.80)	0.00 (3.07)	0.06 (3.24)	0.08 (3.33)	0.16 (3.54)
	KDN-FH-∆, offline correction	0.00 (1.80)	0.15 (2.42)	0.36 (3.22)	0.56 (3.89)	0.68 (4.34)	0.56 (4.62)	0.60 (4.74)
	KDN-FH-∆, closed-loop	0.02 (1.87)	0.11 (2.48)	0.27 (3.44)	0.45 (5.14)	0.58 (7.39)	0.57 (9.32)	0.50 (11.03)
	KDN-VF-F, offline correction	0.07 (2.09)	0.13 (2.51)	0.20 (2.82)	0.25 (2.99)	0.27 (3.07)	0.28 (3.11)	0.29 (3.13)
	KDN-VF-F, closed-loop	0.02 (1.86)	0.04 (2.16)	0.07 (2.38)	0.10 (2.50)	0.11 (2.56)	0.12 (2.58)	0.12 (2.60)
	KDN-VF-∆, offline correction	0.43 (2.93)	0.87 (4.47)	1.64 (7.11)	2.37 (9.33)	2.90 (10.61) 2.94 (11.13) 3.54 (11.80)
	KDN-VF-∆, closed-loop	0.41 (2.47)	0.76 (3.45)	1.24 (4.74)	1.69 (5.69)	1.99 (6.17)	2.15 (6.43)	2.32 (6.60)

TABLE 3 Force

 3 Estimation Errors for Ground-Truth vs. Random Force Initialization over Increasing Sequence Durations -Mean (Std. Dev.) [N]

	Duration	4.0 s	8.0 s		16.0 s	32.0 s
	Initialization	Reference	Random	Reference	Random	Reference	Random	Reference	Random
	KDN-FH-F, OC	-0.13 (2.86)	-0.00 (3.42)	-0.04 (3.22)	0.12 (3.60)	0.07 (3.54)	0.21 (3.76)	0.19 (3.76)	0.19 (3.80)
	KDN-FH-F, CL	-0.07 (2.80)	0.09 (3.36)	0.00 (3.07)	0.10 (3.43)	0.06 (3.24)	0.09 (3.42)	0.08 (3.33)	0.06 (3.36)
	KDN-FH-∆, OC	0.36 (3.22)	0.34 (3.72)	0.56 (3.89)	0.52 (4.25)	0.68 (4.34)	0.64 (4.49)	0.56 (4.62)	0.52 (4.73)
	KDN-FH-∆, CL	0.27 (3.44)	0.37 (4.08)	0.45 (5.14)	0.53 (5.75)	0.58 (7.39)	0.63 (7.35)	0.57 (9.32)	0.56 (9.59)
	KDN-VF-F, OC	0.20 (2.82)	0.22 (3.01)	0.25 (2.99)	0.27 (3.08)	0.27 (3.07)	0.28 (3.13)	0.28 (3.11)	0.29 (3.14)
	KDN-VF-F, CL	0.07 (2.38)	0.12 (2.61)	0.10 (2.50)	0.12 (2.63)	0.11 (2.56)	0.13 (2.63)	0.12 (2.58)	0.13 (2.63)
	KDN-VF-∆, OC	1.64 (7.11)	1.79 (7.55)	2.37 (9.33)	2.37 (9.50)	2.90 (10.61) 2.70 (10.32)	2.94 (11.13) 2.99 (11.10)
	KDN-VF-∆, CL	1.24 (4.74)	1.27 (5.11)	1.69 (5.69)	1.75 (5.86)	1.99 (6.17)	2.06 (6.29)	2.15 (6.43)	2.18 (6.47)

TABLE 4

 4 Kinematics and Force from Vision -Mean (Std. Dev.) Errors From Central Difference, Gaussian Filtering and Algebraic Differentiation

		Central	Gaussian	Algebraic
	Trans. acc. [m • s -2 ]	0.31(25.36)	-0.02(2.92)	-0.05(3.03)
	Rot. vel. [rad • s -1 ]	0.14(446.45) -0.05(30.94)	0.01(31.76)
	Force [N]	1.18(8.94)	0.01(0.72)	0.01(0.75)

TABLE 5 Force

 5 Estimation Errors From AHRS vs. Vision -Mean (Std. Dev.)[N]

	Kinematics	AHRS	AHRS	Vision
	Initialization	ground truth	random	random
	KDN-FH-F, OC	-1.10 (2.95)	-1.12 (2.95)	-1.18 (3.11)
	KDN-FH-F, CL	-1.37 (3.12)	-1.37 (3.13)	-1.25 (3.61)
	KDN-FH-∆, OC	0.72 (3.38)	0.85 (3.42)	0.94 (3.39)
	KDN-FH-∆, CL	1.21 (5.80)	2.27 (11.86)	3.50 (17.28)
	KDN-VF-F, OC	0.18 (2.64)	0.14 (2.68)	0.15 (2.69)
	KDN-VF-F, CL	-0.01 (2.20)	0.02 (2.27)	-0.04 (2.30)
	KDN-VF-∆, OC	5.40 (27.61)	5.16 (23.06)	5.94 (24.54)
	KDN-VF-∆, CL	2.20 (16.31)	3.87 (19.99)	7.37 (25.15)

TABLE 6 Computation

 6 Time Decomposition by Process

		Total	Per sample Per timestep
	Experiment duration 2470.0 s	16.67 ms	100.00 %
	Computation time	3521.4 s	23.76 ms	142.57 %
	Algebraic diff.	22.3 s	0.15 ms	0.90 %
	RNN prediction	120.4 s	0.81 ms	4.87 %
	→ Data formatting	86.2 s	0.58 ms	3.49 %
	SOCP correction	641.8 s	4.33 ms	25.98 %
	→ Initialization	659.0 s	4.45 ms	26.68 %
	Lua/Python bridge	1991.7 s	13.44 ms	80.64 %

ACKNOWLEDGMENTS

This work was partially supported by the FP7 EU Robo-How.Cog project (www.robohow.eu), and the Japan Society for Promotion of Science (JSPS) Kakenhi B No. 16H02886.