Antoine Petit

Stéphane Cotin
email: stephane.cotin@inria.fr

Vincenzo Lippiello

Bruno Siciliano
email: bruno.siciliano@unina.it.

Capturing Deformations of Interacting Non-rigid Objects Using RGB-D Data

This paper presents a method for tracking multiple interacting deformable objects undergoing rigid motions, elastic deformations and contacts, using image and point cloud data provided by an RGB-D sensor. A joint registration framework is proposed, based on physical Finite Element Method (FEM) elastic and interaction models. It first relies on a visual segmentation of the considered objects in the RGB images. The different segmented point clouds are then processed to estimate rigid transformations with on an ICP algorithm, and to determine geometrical point-to-point correspondences with the meshes. External forces resulting from these correspondences and between the current and the rigidly transformed mesh can then be derived. It provides both non-rigid and rigid data cues. A classical collision detection and response model is also integrated, giving contact forces between the objects. The deformations of the objects are estimated by solving a dynamic system balancing these external and contact forces with the internal or regularization forces computed through the FEM elastic model. This approach has been here tested on different scenarios involving two or three interacting deformable objects of various shapes, with promising results.

I. INTRODUCTION

The problem of capturing the deformations of non-rigid objects is an active research field in computer vision that finds prospective applications in robotic manipulation, human/machine and tangible interactions, or motion capture. It becomes a challenging issue when considering interacting multiple objects. Accurately and reliabily registering the interactions can be crucial in applications such as bin picking tasks of soft objects, grasping non-rigid objects with a soft manipulator, to handle food for instance, or performing robotic surgical operations in complex deformable environments with interacting organs or tissues.

Although the case of rigid or articulated objects in close interaction has shown an increasing interest in the research community in the recent years, considering deformable objects remains an open problem. The parameter space becoming much larger than 3D rigid poses or articulation joint states, several challenges must be overcome to jointly estimate the deformations of the interacting non-rigid objects. A main one regards inter occlusions between objects and deformations due to contacts. Tracking independently the different objects would indeed result, due to inaccuracies and ambiguities in the visual or point cloud information, in inter-penetrations and physically unrealistic behaviours between objects. Instead, a simultaneous registration method relying on modelling the contacts between the objects would inherently handle these issues and reach a reliable behavior.

Following this idea, we introduce a solution to integrate collision detection and response models into a rigid and non-rigid registration framework based on successive RGB-D data. This work extends the approach proposed in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF][START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] which is designed for single objects with FEM elastic deformation model. Collisions are handled by hierarchical bounding volume to contact point detection methods and on penalty forces to compute the contact response.

The method described here has been tested on scenarios involving two or three deformable objects, the scenes being observed by commodity RGB-D sensors.

The paper is organized as follows: after a review of the related literature, along with the motivations for the proposed system in section II, descriptions of the elastic and interaction models are provided in section III and IV. Section V introduces the multiple objects registration framework and some experimental results are reported in section VI to illustrate and validate the approach.

II. RELATED WORKS AND MOTIVATIONS

The goal is here to continuously estimate, based on successive RGB-D frames, the deformations undergone by multiple objects, given their known meshes. In the literature, various approaches have been proposed to register deformable objects, using vision and/or range data, and they can be classified according to the underlying model of the considered object, the resolution method, and according to the extracted visual and geometric features.

A. Single object non-rigid registration

Based on implicit physical models, approaches in [START_REF] Kass | Snakes: Active contour models[END_REF][START_REF] Bartoli | Direct estimation of non-rigid registrations[END_REF][START_REF] Pilet | Fast non-rigid surface detection, registration and realistic augmentation[END_REF] use a 1D parametric curve or 2D splines models (Bsplines, Radial Basis Functions) to track single deformable objects in monocular images. This class of methods relies on the minimization of an energy function involving an external energy term related to some image features, and an internal energy term regularizing curvature, bending or twisting, compelling the model to vary smoothly. Adapting these techniques to register 3D shapes or surfaces in monocular images is much complex, since 3D deformations can imply ambiguous 2D transformations, resulting in an underconstrained problem. A first attempt by Terzopoulos et al. [START_REF] Terzopoulos | Constraints on deformable models: Recovering 3d shape and nonrigid motion[END_REF], relying on 3D splines and inspired by Kass et al. [START_REF] Kass | Snakes: Active contour models[END_REF], densely processes gradient features, to compute the data energy term. Less ambiguous feature-based approaches [START_REF] Salzmann | Surface deformation models for nonrigid 3d shape recovery[END_REF] have been preferred and additional constraints are often added to solve ambiguities. With point cloud data, methods in [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF][START_REF] Zollhöfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF] employ an RGB-D sensor to register the acquired point cloud to a surface mesh by minimizing an error function accounting for geometric or direct depth and color errors, and a stretching penalty function for the mesh. By means of a NURBS parametrization [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF] or an optimized GPU implementation [START_REF] Zollhöfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF], real-time performance can be achieved. Although these two systems have shown promising and impressive results, they are still limited to isometric or small elastic deformations, by means of regularization functions proportional to squared distances between nodes of the mesh. Here, we wish to model elasticity in a more physically realistic manner, in order to handle volumetric effects and larger strains.

Indeed, another formulation of the problem relies on physics-based deformable models to perform registration. Other sorts (such as non-linear elasticity) and magnitudes of deformations can then be handled, inferring more consistently shape and/or volumetric regularization. The solution can be determined by balancing internal and external forces or, equivalently, minimizing energy functions. Physics-based methods include discrete mass-spring-damper systems [START_REF] Elbrechter | Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling[END_REF][START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], or more explicit approaches relying on the Finite Element Method (FEM), based on continuum mechanics. In [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], based on a mass-spring-damper system, 3D-3D correspondences, determined through a probabilistic inference, enable the computation of the external forces applied to the mesh. However mass-spring systems are limited in terms of magnitude of elastic deformations, and do not guarantee certain mechanical properties such as volume preservation, failing to handle volumetric effects. First attempts for registration employing the FEM for 3D surfaces in [START_REF] Cohen | Deformable models for 3-d medical images using finite elements and balloons[END_REF][START_REF] Mcinerney | A finite element model for 3d shape reconstruction and nonrigid motion tracking[END_REF] used linear models. More recently in [START_REF] Malti | Monocular templatebased 3d reconstruction of extensible surfaces with local linear elasticity[END_REF], registration in monocular images is addressed by designing a stretching/shrinking energy using continuous mechanical constraints on 2D elements assuming linear elasticity, and some 3D boundary conditions. Haouchine et al. [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF] use a linear tetrahedral co-rotational FEM model, coping with larger elastic deformations, external forces being related to correspondences between tracked 3D feature points mapped to the 3D mesh by means of a stereo camera system. This system has been then extended in [START_REF] Haouchine | Impact of soft tissue heterogeneity on augmented reality for liver surgery[END_REF], involving a nonlinear St-Venant Kirchoff FEM model to efficiently address hyperelastic behavior, registration being addressed through monocular keypoint based external forces.

B. Multiple objects registration

Estimating simultaneously the 3D transformations undergone by multiple objects with vision data is already a well studied problem for rigid or articulating objects. Consisting in retrieving the 6D pose of all rigid objects or the various joint angles in the case of articulated objects, common approaches address the task by independently processing the different objects [START_REF] Tejani | Latent-class hough forests for 3d object detection and pose estimation[END_REF][START_REF] Badami | Depth-enhanced hough forests for object-class detection and continuous pose estimation[END_REF]. Other works propose to register objects and capture interactions, and to cope with inter occlusions. In the case of articulated hand-hand, handobject and object-object interactions, methods are numerous [START_REF] Oikonomidis | Full dof tracking of a hand interacting with an object by modeling occlusions and physical constraints[END_REF][START_REF] Hamer | Tracking a hand manipulating an object[END_REF][START_REF] Oikonomidis | Tracking the articulated motion of two strongly interacting hands[END_REF][START_REF] Pham | Towards force sensing from vision: Observing hand-object interactions to infer manipulation forces[END_REF][START_REF] Kyriazis | Scalable 3d tracking of multiple interacting objects[END_REF][START_REF] Sridhar | Real-time joint tracking of a hand manipulating an object from rgb-d input[END_REF][START_REF] Kim | Keyframe-based modeling and tracking of multiple 3d objects[END_REF][START_REF] Salzmann | Physically-based motion models for 3d tracking: A convex formulation[END_REF].In these works the interactions are modelled through physics-based constraints, enabling to properly deal with collisions and occlusions, using single or multiple RGB or RGB-D cameras. However, to our knowledge, handling multiple deformable objects in interaction is a quite left over problem. We can mention the recent system designed in [START_REF] Guo | Real-time geometry, albedo, and motion reconstruction using a single rgb-d camera[END_REF], which introduces a collision detection process in its non-rigid reconstruction pipeline, to deal with auto-collisions. However this technique falls in the class of non-rigid surface reconstruction of the whole observed scene. Our method, as a template tracking technique, would instead explicitly consider different known deformable interacting entities, and in a volumetric manner.

C. Contributions

In addition to the generation of occlusions, interactions and contacts also result in deformations on the considered objects. Therefore, we advocate for the use of a physics based approach to model both the deformations and the interactions, by relying on volumetric FEM-based models, as proposed in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF][START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] for single elastic objects, and on collision detection models. Based on successive RGB-D data inputs, the main contribution of this paper is thus the introduction of a collision detection and response model.Another improvement lies in an estimation of rigid transformations into the FEM resolution, by the integration of rigid ICPbased elastic forces.

D. Overview of the system

As illustrated in Figure 1, our frame-by-frame tracking system can be outlined as follows: Input: the known 3D volumetric meshes of N O objects, a given RGB-D frame, and assuming correct registrations at the previous frame.

1) Visual segmentation of the considered objects, with a parallel graph cut-based approach on each object. 2) Using the resulting segmented point clouds, perform rigid Iterative Closest Point (ICP) algorithms to estimate the rigid transformations from the point clouds to the meshes. 3) Derive linear external elastic forces between the current meshes and both the rigidly transformed meshes and the segmented point clouds, using closest point correspondences. 4) Numerical resolution of mechanical equations to compute the deformations on the objects, based on tetrahedral linear co-rotational FEM models and on collision detection and response models.

III. ELASTIC DEFORMATION MODEL

We assume the considered objects to be elastic and isotropic, and we employ for each one a co-rotational Finite Element Method elastic model, as in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF][START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF]. It provides physical realism, by relying on continuum mechanics, while being computationally efficient.

Let us note that the modularity of the implemented framework would allow us to use other physical deformation models (hyperelastic, viscous, isotropic or anisoptropic, fractures...).

For an exhaustive description of the FEM, the reader can refer to [START_REF] Cook | Finite element modeling for stress analysis[END_REF]. As a concise description, the method consists in tessellating the deformable object into a mesh made of elements connecting a set X i = {x i,j } n X i j=1 of 3D vertices. The deformation fields over the elements are approximated as continuous interpolations of the displacements of the vertices. We rely here on a volumetric linear FEM approach with tetrahedral elements. For elasticity modelling, we then resort to the linear elasticity theory, with Hooke's law, and to the infinitesimal strain theory [START_REF] Cook | Finite element modeling for stress analysis[END_REF], modified by adopting a co-rotational approach [START_REF] Etzmuß | A fast finite element solution for cloth modelling[END_REF] so as to accommodate to rotation transformations.

If the deformations ûe can be written as ûe = x e -x e,0 , we define here ûr e = R -1 e x e -x e,0 , with R -1 e x e the back rotated deformed coordinates of the four vertices of e, stacked into the 12 × 1 vector x e . R e is a 12 × 12 block diagonal matrix containing four copies of the 3 × 3 rotation matrix corresponding the rotational component of the deformations of the element.

The internal elastic forces f e exerted on the vertices of e can then be related to ûr e through:

f e = R e K e ûr e ( 1
)
with K e the stiffness matrix of the element. K e depends on two elastic parameters of the material, the Young modulus E i and the Poisson ratio ν i , which are specific to each involved object. With this technique, the overall forces on the whole mesh can be summed to zero, while computational efficiency is ensured since K e can be computed in advance, in contrast to non-linear FEM approaches.

IV. INTERACTION MODEL

Modelling the interactions between the considered objects constitutes a central issue to allow a joint registration of the deformations. We propose here a state-of-the-art model consisting in classical collision detection and collision response strategies. Collision being performed on surface meshes, we rigidly map a collision triangular surface mesh of each object on their corresponding volumetric tetrahedral mesh.

A. Collision detection

Collisions can be handled by employing different approaches to determine the intersections and the contacts between the interacting objects. Reviews of collision detection methods can be found in [START_REF] Jiménez | 3d collision detection: a survey[END_REF][START_REF] Teschner | Collision detection for deformable objects[END_REF]. Intersections are computed by performing successive distance tests, based on a hierarchical set of bounding volumes around the polyhedral triangular collision meshes.

Here, we employ hierachies of axis aligned bounding boxes (AABB) around the considered objects. First, as a broad phase, intersections between the coarsest AABBs are found. Then, as a narrow phase, we descend in the AABB hierarchies of the objects in a coarse-to-fine manner. Within the finest AABBs, a search for intersections between pairs of geometric primitives is performed. The primitives are triangles in our case. The computation of intersections is based on a threshold on the distances between the primitives (triangleto-triangle), and returns corresponding pairs of collision (or contact) points on the colliding primitives.

B. Collision response

In order to model the contact forces between two colliding deformable objects i and j, we use a simple penalty-based method. For two detected collision points p and q on the collision surfaces of the interacting objects, barycentric mappings on the tetrahedral meshes are carried out to determine two corresponding points x i (p) and x j (q), in both volumetric models. Then a penetration distance δ can be defined as δ = d -D, with d = x i (p) -x j (q) and D a proximity threshold. As a collision normal n i,j we choose n i,j = p-q p-q . A spring-like repulsive penalty force f c j/i (x i ), exerted by x j on x i , can then be computed as:

f c j/i (x i ) = -kδn (2)
with k a stiffness factor.

V. MULTIPLE OBJECTS REGISTRATION

The frame-by-frame registration problem that we address consists in fitting the point cloud data, provided by an RGB-D sensor, with the tetrahedral meshes, in terms of both rigid and non-rigid transformations. The method proposed in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF] for single objects is here extended to multiple objects. Consequently, point cloud segmentation, matching, rigid and non-rigid registration techniques are adapted, as described below.

Preliminary parallel visual segmentation.

The visual segmentation step presented in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF], based on a graph cut minimization over foreground/background color models, is first performed. The objective is to restrict the acquired point clouds to the considered objects, so as to avoid ambiguities in the matching process, with the background or with occluding shapes. Dealing with multiple objects, the process is carried out in a parallel manner on each object O i , giving 3D target point clouds {Y i }. We limit the size of each Y i by sampling it on a regular grid in the image plane.

Parallel rigid pose estimation

Each mesh X i , initially in its state computed for the previous frame, is then rigidly registered to each observed segmented point cloud Y i . In contrast to the approach proposed in [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF][START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF], for which the rigidly transformed meshes are used to initialize the non-rigid phase, we derive elastic forces attracting towards them. The idea is to integrate these rigid transformations in a smooth and simultaneous manner into the FEM simulation framework, to properly handle collisions. We actually do not directly consider X i but its vertices which appears visible from the previous estimated state in the camera frame. We define X V,i as the visible vertices of X i , lying on the surface of the object. X V,i is determined by a visibility test on the rendered mesh, with respect to its previous state. It is performed on the whole rendered scene to consider occlusions between the interacting object.

As in [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF][START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF] a classical rigid ICP algorithm [START_REF] Chen | Object modelling by registration of multiple range images[END_REF] is then carried out for each object, between Y i and X V,i . We can compute the rigidly transformed mesh

X t i = {x t i,j } n X t i j=1
through:

x t i,j = T i x i,j (3)
with T i the rigid transformation estimated with the ICP. Elastic forces f r ext,i between both sets X t i and X i can then be written as:

f r ext,i (x i,j ) = k r i,ext (x t i,j -x i,j ) (4)
with k r ext,i a stiffness parameter.

Parallel point cloud matching for non-rigid registration

With the aim of fitting the meshes to the point clouds in a non-rigid manner, still based on [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF], we compute external forces that are exerted by the observed segmented point clouds Y i on the different corresponding sets X V,i . We use sets of external forces f d ext,i related to geometrical information, as introduced in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF]. The method consists first in determining nearest neighbors correspondences, both from the segmented point clouds to the meshes and from the meshes to the segmented point clouds. For each object, based on the two sets of mesh-to-point cloud and point cloud-to-mesh correspondences, we can compute an external elastic force f d ext,i exerted on each x i,j in X V,i as follows:

f d ext,i (x i,j ) = k d ext,i (x i,j -y f i,j ) (5)
where y f i,j is a linear combination of points in the point clouds which are matched to x i,j , either from mesh-to-point cloud and from point cloud-to-mesh correspondence sets. k d ext,i is the stiffness of these external elastic forces, which is here common to each object.

Resolution of the deformations

The objective of the resolution process is to balance the sets of external forces capturing rigid, non-rigid transformations of the objects and the interactions between themselves, with the internal forces based on the different deformation FEM models presented in Section III. The estimation consists in solving for each object a dynamic system of linear ordinary differential equations involving the internal and the different external forces, based on Lagrangian dynamics:

M i ẍi + C i ẋi + f i = f d ext,i + f r ext,i + f c i (6)
with

f i = K i x i + f 0,i
where x i is a 3n Xi × 1 vector containing the positions to estimate of the vertices of each object O i . M i and C i are the 3n Xi × 3n Xi mass and damping matrices, K i the 3n Xi × 3n Xi N o global stiffness matrix which sums the 3n Xi × 3n Xi element-wise rotated stiffness matrices K r e = R e K e R -1 e , written with respect to whole set of vertices, and f 0,i the corresponding global offset summing the elementwise ones R e K e x e,0 . f d ext,i , f r ext,i and f c i are 3n Xi × 1 vectors containing the external forces and resulting from the interaction model and the collision responses between object O i and the other objects.

An Euler implicit integration scheme is used to solve the system with respect to each x i , along with a conjugate gradient method. We could consider the static case, with the static equilibrium of the deformations assumed to be reached, so the transient and the dynamic terms of equation ( 6) would be neglected, leading to simply solving the equality between internal elastic forces and external forces with the conjugate gradient method.

VI. RESULTS

The results presented here report different scenarios involving objects of different shapes, potentially textureless, and with different sort of interactions.

The fist sequence indeed shows a stuffed toy and elastic rubber toy brain colliding, the second a silicon elastic disk wrapping a silicon crocodile puppet filled with with plastic balls. In the third sequence, the disk, a silicon object and cube of foam are stacked and squeezed together.

The point clouds of the investigated scene are acquired using different structured light commodity RGB-D cameras, an Asus Xtion: 320 × 240 RGB and depth images being processed, for the first sequence, and an Intel RealSense SR300 (960 × 540 images) for the second and third test sequence.

In order to create the geometric and deformation models of some of the considered objects (the squirrel, the crocodile, the toy brain, the light red silicon object), surface meshes of the undeformed objects were first reconstructed offline. We have used the RGB-D based dense 3D reconstruction techniques KinectFusion [START_REF] Newcombe | Kinectfusion: Real-time dense surface mapping and tracking[END_REF] and InfiniTAM [START_REF] Kähler | Real-time large-scale dense 3d reconstruction with loop closure[END_REF] which is implemented for the RealSense SR300. The large resolution and short range of the RealSense SR300 with respect to the Asus Xtion enables to capture and model finer grain geometries, such as the one of the crocodile. Finally, some remeshing and smoothing tasks have been carried out with a modelling engine, to get fair, closed and clean surface meshes of the objects. The corresponding volumetric tetrahedral meshes have been generated by achieving a 3D Delaunay triangulations within the surface meshes, thanks to the CGAL library 2 . As an approximation, we assume the isotropy of the materials, in order to apply the deformation model described in Sect. III. In Figure 2 are featured some the considered tetraherdral meshes, as well as the visual and collision surface models for the crocodile. For computational efficiency, the collision mesh is made coarser than the visual mesh. The software implementation relies on the Simulation Open Framework Architecture (SOFA) [START_REF] Faure | Sofa: A multi-model framework for interactive physical simulation[END_REF], which provides the FEM elastic and the collision models.

Qualitative results are shown in Figure 3 for the different scenarios, as in the provided video. Let us note that initialization is performed manually, rigidly, close to the actual initial poses and we assume that the objects are not initially in contact.

The case of the silicon disk wrapping the crocodile toy is to be noticed. Indeed, the collision between both objects enables to constraint sufficiently the silicon disk so that it gets registered coherently, even with severe bending deformations, with occlusions by the crocodile, and with errors in the segmentation.

In Figure 4 we can visualize the vertices of the visible surface of the meshes (green squares), which are processed for registration with the segmented point clouds (brown squares). Different samplings for the point clouds are used, accordingly to the resolutions of the meshes. The detected collisions between the interacting objects can be seen on the third row with segments between the found contact points.

The focus on the collision regions in Figure 5 with wireframe views illustrates that contacts and occlusions between the objects can be properly handled, when objects are squeezed, wrapped or stacked together. Independently tracking them leads to inter-penetrations.

In Figure 6 are shown some failure situations when forces resulting from the rigid ICP registration are not integrated in the resolution.

As for computational issues, segmentation being addressed in a parallel manner, and the integration of collision requiring extra computations when object are getting into proximity, the method does not achieved fast performance. 5 -10f ps framerates are reached, depending on the number of objects and on the resolutions of the collision surface meshes.

VII. DISCUSSION

The experiments presented above show some promising results for the method. However a few limitations of the method shall be discussed. The system requires some tuning for both the FEM models of the objects and the collision detection and response models to obtain stability and a proper behavior. Some instabilities or ripples can indeed appear when the involved object undergo fast rigid motions or when to many interactions are involved, for instance when two many objects of complex shapes and collision models are involved. In these cases the addition of numerous and potentially large and opposite external forces coming from the point cloud correspondences, the rigid motion estimation and the collisions responses can make the Lagrangian dynamics equations hardly to be solved using the classical implemented Conjugate Gadient solver. Thus in these cases some tuning can be required to achieve a successful resolution of the whole system, and contact detection distances need to be set quite high in some cases (see Figure 4, third row).

The collision detection and response models are also very classical and a more elegant approach, guaranteeing a more physical response, could be undertaken, by resorting to the Signorini's model and by solving the contacts as a Linear Complementary Constraint problem [START_REF] Allard | Volume contact constraints at arbitrary resolution[END_REF]. Including friction through Coulomb's law could also provide some improvements.

Although the system has proven to be fast when considering a single object, with multiple objects computations linearly increase with the number of objects, and even more when contacts occur. One major burden in the computations lies actually in the segmentation process which is handled independently for each considered object. A simultaneous segmentation method, in a pyramidal manner, could be designed to obtain a better efficiency.

VIII. CONCLUSION

In this paper we propose the first system able to simultaneously register multiple interacting deformable elastic objects. It relies on a co-rotational FEM elastic models to accurately and efficiently model elasticity and on an interaction model consisting in collision detection and response models to cope with contacts between object. Based on RGB-D input data, point cloud matching and registrations processes are achieved on segmented data in a parallel manner, enabling to deal with occlusions. Rigid motions, deformations and contacts of all the considered object are then estimated in a common solver, providing a realistic behaviour and coherently constraining the registration on the involved entities.

The method has been tested on simple scenarios with elastic objects but could be extended to other physical models, to articulated objects, for instance for robotic manipulation applications.

Future works would also aim at proposing a more efficient segmentation process for the different objects, integrating more distinctive point cloud features. Another interesting direction would be to infer realistic contact forces from the registered interaction.

Fig. 1 :

 1 Fig. 1: Proposed pipeline for multiple deformable objects tracking.

Fig. 2 :

 2 Fig. 2: Tetrahedral meshes for the brain, squirrel and crocodile, plus the visual and collision models for the crocodile toy.

Fig. 3 :

 3 Fig.3: Results of the proposed registration method for two (first six rows) and three interacting objects (the last three), with the input images, the segmented images (second, fifth and eigths rows), the registered mesh reprojected in the input image. See the provided video for comparisons with rigid registration and collision free approaches.

Fig. 4 :

 4 Fig. 4: Visualization of the registered point clouds (brown squares) for the meshes (greens squares as vertices on the visible surfaces) of examples with two and three interacting objects. The displayed segments on the third row show the detected collisions.

Fig. 5 :

 5 Fig. 5: For the different examples, comparison between registration based on a collision detection model (left column), and without such a model, resulting in large inter-penetrations (right column).

Fig. 6 :

 6 Fig.6: For the different examples, failure cases when the proposed rigid registration approach is not included in the resolution.

http://www.cgal.org

This research has been partially funded by the EC Seventh Framework Programme (FP7) within RoDyMan project 320992.


Fabio Ruggiero

Antoine Petit

Diana Serra

Aykut C Satici

Jonathan Cacace

Alejandro Donaire

Fanny Ficuciello

Luca R Buonocore

G Andrea Fontanelli

Vincenzo Lippiello

Luigi Villani

Bruno Siciliano

Nonprehensile manipulation of deformable objects: Achievements and perspectives from the RoDyMan project

The goal of this work is to disseminate the results achieved so far within the RODYMAN project related to planning and control strategies for robotic nonprehensile manipulation.

The project aims at advancing the state of the art of nonprehensile dynamic manipulation of rigid and deformable objects to future enhance the possibility of employing robots in anthropic environments. The final demonstrator of the RODYMAN project will be an autonomous pizza maker. This article is a milestone to highlight the lessons learned so far and pave the way towards future research directions and critical discussions.

I. OVERVIEW ABOUT NONPREHENSILE MANIPULATION IN

ROBOTICS

M ANIPULATING an object roughly entails changing its currentstate to a desired one. Suppose to have an object on a table that has to be moved from configuration A to configuration B on the same table. How would you move it? Maybe, the two most rated answers would be: (i) grab the object, lift it up and place it back on the table in B; (ii) push the object on the table from configuration A to B. In case (i), the object is completely grabbed between the fingertips and/or the palm, and the hand is theoretically able to resist any external disturbance wrench applied to the object. Hence form closure occurs even when infinitesimal motions of the object are prevented by the hand, otherwise force closure occurs [START_REF] Prattichizzo | Grasping[END_REF]. Bilateral constraints are exhibited by the grasp in both closures. In case (ii), the object is instead pushed by one or more fingertips and these last are able to resist only to forces counteracting the direction of the pushing. Hence, nonprehensile manipulation, or graspless manipulation, occurs since only unilateral constraints are involved. These tasks can also be endorsed as dynamic when the dynamics of both the object and the robot are essential to successful task execution.

Therefore, a nonprehensile dynamic manipulation task can be generally described as a task in which the object is subject only to unilateral constraints and the dynamics of both the object and the manipulating hand, as well as the F. Ruggiero, J. Cacace, F. Ficuciello, G.A. Fontanelli, V. Lippiello related kinematics and the (quasi-)static forces play a crucial role. Pushing objects, folding clothes, carrying items on a tray, cooking in pan, performing some surgery operations are examples of nonprehensile manipulation tasks. From a robotic point of view, most of nonprehensile manipulation systems are underactuated, raising controllability challenges. However, dynamic nonprehensile manipulation benefits of several advantages such as the increase of available robot actions, bigger operative workspaces, enhanced dexterity in dynamic tasks.

On one hand, the robotics literature is well-established regarding grasping techniques [START_REF] Prattichizzo | Grasping[END_REF] and control methods for manipulation tasks with grasp [START_REF] Murray | A mathematical introduction to robotic manipulation[END_REF]. Within industrial applications, where simplicity and costs are relevant above all, grippers or special-purpose devices are widely used. Nevertheless, the necessity for robots working in anthropic environments is growing fast as shown by the European Strategic Resarch Agenda1 (eSRA), where it is outlined that robots will pervade a portion of the market domain in domestic appliances, assisting living, entertainment and education. Therefore, robots should not need specific tools for each actions, but they should exploit multi-purpose devices, like multi-fingered robotic hands, and rely upon the dexterity conferred by the designed control algorithms. Manipulation dexterity is one of the main research challenge currently addressed by the robotic community. As explained above, a nonprehensile manipulation task is a dexterous task par excellence. As a matter of fact, some tasks are intrinsically prehensile like, for instance, (un)screwing a bottle cap. Other tasks can be instead tackled both in a prehensile or a nonprehensile manner, like the aforementioned example of moving an object on a table. Other tasks are inherently nonprehensile, like carrying a glass full of water on a plate. Other tasks are hybrid, in the sense that to reach the goal both prehensile and nonprehensile actions are required, like a juggler that has to repetitively catch-and-throw balls in a cascade juggling pattern.

On the other hand, the robotics literature is not fully developed for nonprehensile manipulation tasks. The classic way to cope with them is to split a task in simpler subtasks, referred to as nonprehensile manipulation primitives [START_REF] Lynch | Control of nonprehensile manipulation[END_REF], such as throwing [START_REF] Satici | A coordinatefree framework for robotic pizza tossing and catching[END_REF], dynamic catching [START_REF] Bätz | Dynamic manipulation: Nonprehensile ball catching[END_REF], batting [START_REF] Serra | An optimal trajectory planner for a robotic batting task: the table tennis example[END_REF], juggling [START_REF] Reist | Design and analysis of a blind juggling robot[END_REF], dribbling [START_REF] Bätz | Ball dribbling with an underactuated continuous-time control phase: Theory & experiments[END_REF], pushing [START_REF] Lynch | Stable pushing: Mechanics, controllability, and planning[END_REF], sliding [START_REF] Vose | Friction-induced velocity fields for point parts sliding on a rigid oscillated plate[END_REF], rolling [START_REF] Donaire | Passivity-based control for a rolling-balancing system: The nonprehensile disk-on-disk[END_REF] and so on. Each primitive, equipped with its own motion planner and controller, is then turned on and off during a complex manipulation task by a high-level supervisor [START_REF] Woodruff | Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks[END_REF]. Among the mentioned nonprehensile manipulation primitives, only rolling and batting boast a considerable amount of work in the literature. There is also a lack of a general unified theoretical framework in the field, causing the continuous investigation of ad-hoc motion planners and controllers to solve the specific tasks one by one. The main reason may be found in the possible change of the contact status during a nonprehensile manipulation task, leading to non-smooth dynamics of the whole system, which complicates the control design. For this reason, dynamic nonprehensile manipulation may be considered as the most complex manipulation action, deserving attention as requested by the eSRA, and posing many research challenges to be solved.

In the described context, the RODYMAN project aims at the development of a service robot able to manipulate elastic and soft objects, as well as to manipulate both rigid and non-rigid objects in a nonprehensile way, with the ambitious goal of bridging the gap between robotic and human task execution capability. In order to reach the planned goals, three main research challenges have been identified within the RODYMAN project. Namely:

• Mechatronic development and assembly. A mobile robotic platform equipped with two commercial arms and multi-fingered hands is necessary to perform the dynamic manipulation tasks planned for the project. • Modelling and perception. Real-time requirements posed by robot interaction with deformable objects during dynamic nonprehensile manipulation actions are essential to fulfil the required tasks. • Control techniques for nonprehensile dynamic manipulation. The goal of the project is to advance the state of the art in controlling rigid objects in a nonprehensile way, and starting investigating the problems relative to the prehensile and nonprehensile manipulation control of deformable objects as well. The final demonstrator of the project will be an autonomous pizza maker since preparing a pizza involves an extraordinary level of manual dexterity.

Some other projects have tried to address nonprehensile manipulation problems using different approaches. The RIBA robot2 is able to lift up and set down patients from/to their beds and/or wheelchairs. The soft body of the robot is designed to make safe the interaction with the humans. The performed transporting task is indeed nonprehensile, but the manipulation task is not dynamic since the patient's body is considered as a rigid object, and only motion planning techniques for lifting up the body are investigated. The task is very similar to a pick-and-place operation where the transporting motion is addressed in a nonprehensile fashion. Also the ERC SHRINE project3 goals are focused to enhance robot manipulation capabilities, so as to overcome barriers preventing robots from safe and smooth operations within anthropic environments. The sought robot dynamic manipulation, in some cases also performed in a nonprehensile way, is addressed to cooperate with humans. In the following, the results achieved so far within the RODYMAN project for the above three outlined research challenges are described. Videos of the related experiments can be found in the related YouTube channel 4 .

II. ROBOT DESIGN AND ARCHITECTURE

A. Mechatronic design

The built mechatronic set-up, referred to as RODYMAN like the project name, is a 21-degree-of-freedom (DoF) humanoidlike robot (see Fig. 1). An omnidirectional mobile platform allows the robot to move in the space. An actuated mechanism gives the ability to enlarge the support polygon during the execution of dynamic and rapid movements of the upper body. The battery pack and UPS unit used to provide power to all the devices are housed within the mobile platform, and provide at the same time the weight to stabilize the platform. Two standard PCs are also located in the base. One is a QNX-based PC used for real-time and low-level of the motors and the implementation of safety procedures. The second one is a Linux-based PC used for perception and high-level planning and control algorithms. The upper-body limbs of the robot are two SCHUNK LWA 4P arms with 6 DoFs each. The seventh joint of each arm, required to add human-like redundancy, is provided by a SCHUNK PRL-100 integrated into the shoulder. To the best of the authors' knowledge, the SCHUNK were the only arms on the market to have both dimensions similar to the human arms and the control directly on the CAN bus without an external controller box. Nevertheless, experimental results show that the high friction and the low joint velocities exhibited by these arms represent a limitation on the execution of particular and complex tasks like tossing. It is worth pointing out that this solution represents only a first prototype, and the design of new arms, with advanced dynamical characteristics, is within the RODYMAN project plan.

Moreover, the dynamic model of the whole structure has been derived in a symbolic form. The LMI method in [START_REF] Sousa | Physically feasibility of robot base inertial parameters identification: A linear matrix inequality approach[END_REF] has been employed to obtain the identification of the dynamic parameters by absorbing the physical constraints within the optimization procedure. Experimental results have indeed shown that friction, mostly the static part, is the dominant component in the measured torque. Therefore, a friction identification has been firstly performed separately, and then the friction parameters have been used as constraints within the LMI optimization procedure.

The RODYMAN platform is completed by two motors to actuate the torso and one for the pan-tilt neck. In order to provide enhanced dexterous manipulation skills, two anthropomorphic SCHUNK Servo-electric 5-Finger SVH hands can be applied at the end-effector tip of the two arms. However, these hands are very delicate and they are replaced with suitable 3Dprinted tools for those tasks requiring non-trivial weights in action, like the pizza-peel task described afterwards. From the perception point of view, the platform is equipped with two laser scanners in the base for odometry operation; two force sensors can be mounted on the wrist to measure the interaction forces between the end effector and the environment, while the interaction forces exerted on the robot structure can also be obtained using proper estimators [START_REF] De Luca | Collision detection and safe reaction with the DLR-III lightweight manipulator arm[END_REF]. Finally, the head is equipped with a stereo camera system, an RGB-D sensor and a time-of-flight camera to obtain a precise depth estimation.

B. High-level software architecture

In order to carry out the expected activities involving complex manipulation actions, a control architecture is designed to handle high-level planning tasks. A sketch of the control architecture is shown in Fig. 2, and it is described below.

The Human Robot Interaction (HRI) Interface module is used to specify high-level tasks as inputs for the system (e.g. the pizza tossing). The Supervisor module is responsible of the task decomposition process, splitting the high-level actions received by the HRI Interface in lower-level actions considering both the state of the robot and the information generated by the Perception module. After the decomposition process, each lower level action can be executed. Examples of high-level tasks are Grasp(Object), Search(Object) or Toss(Object), and sequences of nonprehensile manipulation primitives. In order to suitably perform the task decomposition process, the supervisor module is provided with a library of hierarchical tasks, analogous to Hierarchical Task Networks (HTNs), which can be composed by the system to achieve the desired goal [START_REF] Cacace | Aerial service vehicles for industrial inspection: Task decomposition and plan execution[END_REF]. In this context, if a nonprehensile manipulation action is required, the related low-level controller is invoked from the dynamic manipulation task list while the supervisor waits its termination. Otherwise, the Executors module is responsible for the action by implementing both the path and the motion planning functionalities to find an obstacle-free way for the end effectors of the robot and its base. This module relies on MoveIT! [START_REF] Chitta | MoveIt! [ROS topics[END_REF], a framework that integrates Universal Robot Description File, Open Motion Planning Library and other toolkits. The generated trajectories for the joints and the base of the robot are streamed to the robot actuators from the Controller modules. Finally, information about the robot environment is extracted from the Perception module via image elaboration algorithms.

The proposed high-level control architecture perfectly matches the requirements of the RODYMAN robotic platform, statically allocating the best low-level controller to accomplish desired actions. This improves the current literature since very few ways of decomposing a high-level task for nonprehensile manipulation have been developed [START_REF] Woodruff | Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks[END_REF]. Future directions are inclined to increase the level of collaboration of the robot with human operators allowing shared task planning and execution.

III. PERCEPTION OF DEFORMABLE OBJECTS

Using the point cloud data provided by the RGB-D sensor, a method to cope with real-time (35 fps) tracking of a deformable object is presented. Several contributions are proposed such as handling various large elastic deformations while ensuring physical consistency, coping with fractures, rigid motions and occlusions [START_REF] Petit | Tracking elastic deformable objects with an RGB-D sensor for a pizza chef robot[END_REF]. The case of a pizza dough being stretched and tossed is taken as example.

Since the considered system attempts to deal with large deformations and elastic volumetric strains, a realistic mechanical model, based on continuum mechanics and on a volumetric tetrahedral Finite Element Method (FEM), is employed. This model can be suitably used for real-time applications through the SOFA simulator. Besides, an explicit physical modelling would enable a reliable prediction of internal forces undergone by the object.

In order to model elastic deformations, the infinitesimal strain theory and Hooke's law are taken into account, providing a linear relation between the displacement of the tetrahedral elements of the mesh and the internal forces exerted on their nodes. The co-rotational approach is used as a good compromise between the ability to model large deformations of the elements and computational efficiency.

Based on the FEM co-rotational model, fractures in the mesh are detected by decomposing the internal forces on the nodes into tensile and compressive forces to measure pure tensile forces acting on each node, through a so-called separation tensor. The fracture is propagated by simply removing attached elements intersected by the fracture plane. The model is illustrated in Fig. 3. The frame-by-frame tracking framework in Fig. 4 relies on a prior visual segmentation of the object in the image, based on a graph-cut based segmentation technique using color cues. The corresponding segmented point cloud is first registered through a classical Iterative Closest Point (ICP) method, and then by fitting the known mesh of the object on the point cloud. The basic idea is to derive external forces exerted by the point cloud on the mesh and to integrate them with the internal forces computed using the physical model into Lagrangian mechanical equations: M ẍ + C ẋ + K ′ x + f 0 = f ext , where x ∈ R n contains the positions of the n vertices, M ∈ R n×n , C ∈ R n×n and K ′ ∈ R n×n are the mass, damping and stiffness matrices, f ext ∈ R n is the external forces vector, and f 0 ∈ R n is an offset on the internal forces due to rotational effects. An Euler implicit integration scheme and a conjugate gradient method are used to solve the system with respect x. The elastic forces f ext based on geometrical correspondences between the point cloud and the mesh [START_REF] Petit | Tracking elastic deformable objects with an RGB-D sensor for a pizza chef robot[END_REF].

To validate the method, some results have been obtained with various soft objects, deformations due to bending, stretching or compression actions, and fractures, and under challenging conditions, like occlusions or fast motions (see Fig. 5). Some preliminary experiments integrating the method into a robotic manipulation task of a silicon pizza dough, have also been carried out (see Fig. 5c and Section IV-B).

IV. NONPREHENSILE OBJECTS MOTION PLANNING AND CONTROL

Four manipulation primitives have been considered so far within the RODYMAN project. Namely, nonprehensile rolling, sliding, tossing and batting/juggling. Sliding and tossing take into account deformable objects, while rolling and batting/juggling only rigid ones. It is indeed difficult to find relevant applications involving deformable objects in pure rolling and juggling tasks. In the following, the controller and/or the motion planner designed for the aforementioned primitives are described.

A. Nonprehensile rolling

An actuated manipulator of a given shape, referred to as hand, manipulates an object only through purely rotations, without grasping or caging it. Therefore, the object can only roll upon the shape of the hand. Case studies like the manipulation of a ball on a plate, the ball on a beam, and so on, are deeply examined in the literature. In this article, only planar rolling is described.

Since highly-geared harmonic drives are present within the RODYMAN mechatronic platform, it is suitable to suppose the acceleration of the hand θh ∈ R as input a h ∈ R for the system. The dynamic model for a nonprehensile planar rolling manipulation system, in which the hand can only rotate around its center of mass, is described by Coriolis matrix of the mechanical system, and g 2 ∈ R is the second element of the (2 × 1) gravity-force vector. Detailed expressions of each term are provided in [START_REF] Lippiello | The effects of shapes in input-state linearization for stabilization of nonprehensile planar rolling dynamic manipulation[END_REF], where it is also noticed that if the Coriolis terms are zero, a nonprehensile planar rolling manipulation system is differentially flat with the output b12 b22 θ h + s h . Among the class of systems for which the aforementioned assumption is true, it is worth recalling the ball-on-disk (BoD) system, which is mathematically equivalent to the disk-on-disk (DoD) system in the transversal plane [START_REF] Donaire | Passivity-based control for a rolling-balancing system: The nonprehensile disk-on-disk[END_REF].

θh = a h , (1a)
The BoD consists of a ball rolling on a disk and arranged one on top of the other as shown in Fig 6 . The disk is the actuated hand and the ball rolling on the hand is the object. The control problem for the BoD is to balance the object at the Fig. 6: The RODYMAN platform actuating the BoD system. The disk is actuated by the movement of the RODYMAN joints. The displayed structure, made by three connected bars, is employed to make possible to start the experiments with the ball in a position which is different from the desired equilibrium on the top of the disk. The world frame is depicted in red, the one attached to the rotating wheel is in green, θ h represents the angle between the two, while s h measures the contact position of the ball on the wheel. upright position while driving the hand to a desired angular set-point. This problem is solved using passivity-based control (PBC) for port-Hamiltonian (pH) systems. In its standard form, this approach applied to nonprehensile rolling aims at finding a control law for system (1a)-(1b) such that the closed-loop dynamics can be written as follows

q ṗ = 0 B -1 B d -B d B -1 J 2 (q, p) -R d (q, p) ∇H d (q, p), (2)
where q ∈ R 2 and p ∈ R 2 are the generalized coordinate and moment vectors,

H d = 1 2 p ⊤ B -1 d (q)p + V d (q) ∈
R is the desired total energy of the closed-loop system, B d ∈ R 2×2 and V d ∈ R are the desired mass matrix and potential energy, respectively, and J 2 ∈ R 2×2 and R d ∈ R 2×2 represent the gyroscopic forces and damping injection of the closed loop, respectively. The objective is to shape the desired energy of the closed-loop dynamics to get a minimum of the potential energy at the desired equilibrium. The asymptotically stability of the closed loop is ensured by using the desired energy as Lyapunov function and the detectability of the passive output. The full development of the control design is reported in [START_REF] Donaire | Passivity-based control for a rolling-balancing system: The nonprehensile disk-on-disk[END_REF] for the DoD example.

Within the RODYMAN project, control laws related to the nonprehensile rolling primitive have been developed also for the 3D case, like the stabilization of a ball on a flat plate and the control of a robotic hula-hoop [START_REF] Gutirrez-Giles | Modeling and control of a robotic hula-hoop system without velocity measurements[END_REF]. As a milestone, it is possible to affirm that nonprehensile rolling can be successfully modelled through the pH formalism and consequently controlled with PBC approaches. This is relevant because it means that there exists a unified framework at least for such class of nonprehensile manipulation primitive.

B. Friction-induced manipulation primitive

As a case study, in order to uniformly cook a pizza, the dough must be rotated through a peel inside a wooden oven, in which the heat source is present only in a side of the structure. Similar actions are performed by chefs when some food must be brown or rotated in a pan.

From a dynamic point of view, friction plays a key role because of the sliding manipulation primitive between the tool and the part to be manipulated. In the literature, frictioninduced manipulation has been extensively studied to create virtual velocity fields on a vibrating plate actuated by a mechanical system equal or similar to a Stewart platform [START_REF] Vose | Friction-induced velocity fields for point parts sliding on a rigid oscillated plate[END_REF]. A similar concept has been suitably modified for the pizza case [START_REF] Higashimori | Dynamic manipulation inspired by the handling of a pizza peel[END_REF].

The RODYMAN platform has successfully achieved a bimanual nonprehensile manipulation task through sliding by handling a peel to rotate the pizza placed on it. With reference to Fig. 1, the peel is chosen to be only translated (and rotated) along (and around) its longitudinal direction. A suitable combination of these two movements creates the desired motion of the object on the peel: an acceleration along the longitudinal direction moves the object back and forth on the peel once static friction is overcome; while, an angular acceleration around the same axis creates a non-uniform pressure distribution on the object. This, together with the linear acceleration, creates a rotation of the object. It is worth noticing that the object rotation is not decoupled from a linear displacement on the peel. Adaptations from [START_REF] Higashimori | Dynamic manipulation inspired by the handling of a pizza peel[END_REF] have been necessary to apply the concepts on the RODYMAN platform. Namely, two suitable smooth sinusoidal accelerations, with the same tunable frequency and different tunable amplitudes and phases, are planned for the linear and angular accelerations of the peel. The motion of the RODYMAN joints are then retrieved by means of a standard closed-loop inverse kinematic algorithm. The above-described tracking of deformable objects has been employed to control the center of mass of the pizza towards the center of the peel through a simple PI-controller, while a complete rotation of the circular shape is requested.

Friction estimation is crucial within this task and several tests have been performed to suitably tune all the parameters of the control model to fit the real set-up. Current work aims at finding structural properties for the controller, like the design of orbital stabilization for the object on the peel (i.e., reach a desired rotational velocity).

C. Tossing task

The problem of tossing and catching a deformable object, like a pizza dough, is a procedure that is frequently dexterously performed by human pizza chefs. There are at least three reasons why tossing the dough during the preparation of the pizza is attractive: (i) the dough is stretched to a desired size, (ii) the dough naturally assumes a configuration that is thicker at the ends and thinner in the middle, and (iii) as the spinning dough freely falls, the outside of the dough dries, making it crunchy in the outside but light in the middle. The pizza chef is trained to perform a streamlined hand motion to toss and catch the dough, while a similar feat is a desired achievement for the RODYMAN robot.

The combined model of the dough grasped with robotic fingers through unilateral constraints, and the kinematics and dynamics of the robot manipulator has been derived in [START_REF] Satici | A coordinatefree framework for robotic pizza tossing and catching[END_REF]. Upon that, a control law achieving the desired tossing motion can be designed. Furthermore, with a perfect knowledge of the motion of the dough, optimal trajectories can be generated in SE(3) for the catching phase. The optimal trajectory generation is repeated as new sensor information is available. The trajectories are generated in such a way that the initial position, velocity, acceleration and final velocity and accelerations are matched, therefore, it is at least thrice continuously differentiable. An optimal trajectory, whose initial and final accelerations are prescribed, has to satisfy a sixthorder Boundary Value Problem (BVP). Such BVP is generated by using the necessary conditions for a path to minimize a convex combination of the jerk and acceleration functionals. While minimizing the jerk functional reduces the vibrations in the structure of the robotic manipulator, minimizing the acceleration functional reduces the total amount of energy expended during the catching motion. Only the case where the final position is left free and is part of the minimization problem is considered. More details can be found in [START_REF] Satici | A coordinatefree framework for robotic pizza tossing and catching[END_REF].

Experimental validations are in progress. Nevertheless, preliminary results show that such kind of task requires high peak currents in the motors to toss the dough for more than 10 cm. As anticipated above, the motors of RODYMAN do not have such skills. As a matter of fact, analogies between tossing and walking gaits can be found within mathematical models. Similarly to robotic legs, hydraulic actuators seem to be more performing and the same might hold for tossing tasks. The stretching-the-dough task can be also performed in an alternative form which will be explored in the future.

D. Batting/Juggling skills

A very challenging primitive from the control view point is the one involving impacts. Inside batting, for instance, an object (a ball) is intercepted by the end-effector (a paddle) without grasping it, and it is thrown towards a precise goal. This motion primitive is typically used by athletes, such as baseball or table tennis players. Also jugglers use such primitive when their hands control the continuous motion of one or more objects through intermittent contacts. These dynamic motions require high velocity and precision. The design of planning and control methods to deal with them would strongly enhance capabilities of robot manipulators, extending the workspace size and enhancing dexterity.

The batting task dynamics is typically defined as hybrid, since it consists of the continuous aerodynamics of the manipulated ball (a differential equation), and the discontinuous reset of the velocity at impact time (two difference equations), given by

pb = -g -k d || ṗb || ṗb + k l S(ω b ) ṗb , (3a)
ṗ+ b = v p + Γ a ( ṗ- b -v p ) + Γ b ω - b , (3b)
ω + b = Γ c ( ṗ- b -v p ) + Γ d ω - b , (3c)
where p b ∈ R 3 and ω b ∈ R 3 are the position and the spin of the ball, respectively; v p ∈ R 3 is the paddle velocity; Γ j (R p ) ∈ R 3×3 , j = {a, b, c, d}, are transformation matrices dependent on the rebound parameters and on the orientation of the paddle R p ∈ SO(3) at the impact time; k d ( ṗb , ω b ) ∈ R and k l ( ṗb , ω b ) ∈ R are, respectively, drag and lift parameters; g ∈ R 3 is gravity acceleration vector; ||•|| denotes the Euclidean norm; S(•) ∈ R 3×3 is the skewsymmetric matrix, while superscriptsand + represent the state before and after the impact, respectively. The matrices Γ j can be detailed on the basis of the addressed rebound (ball impacting the table and/or a rubber paddle). Their expression may become complicated in non-trivial situations (like nonspherical objects): this leads to the use of some (string) assumptions and model reductions. Five different phases have been considered to solve the batting problem by using the RODYMAN platform simulator. Firstly, a vision system is assumed to measure the trajectory of the ball. By assigning the impact time, the prediction of the impact position and pre-impact velocity of the ball are obtained numerically solving its aerodynamic model (3a). Then, the post-impact velocity of the ball, such that it goes towards a desired goal in a predefined time, is computed solving equation (3a) backward in time. The configuration of the paddle to generate such velocity of the ball results from the analytic solution of the discontinuous part of the ball-paddle model, given by the difference equations (3b) and (3c). Thereafter, the motion of the paddle to reach the desired configuration is a result of the minimization of its linear and angular acceleration with a coordinate-free approach, assuming that the path is generated on an arbitrary Riemannian manifold, similarly to the tossing primitive. Finally, the motion of the RODYMAN joints is derived from a classical second-order closed-loop kinematic inversion. More details can be found in [START_REF] Serra | An optimal trajectory planner for a robotic batting task: the table tennis example[END_REF].

A similar algorithm can be thought to accomplish different juggling patterns. While doing this, the lesson learned is that these techniques may also be applied to other dynamic tasks which share the same hybrid nature with impacting manipulations, such as walking or running tasks.

V. FINAL DISCUSSION Despite several progresses have been addressed within the RODYMAN project so far, like real-time tracking of deformable objects employed in tossing and sliding tasks, several issues have to be tackled yet. The mechatronic platform should be revised to cope with issues given by the high-velocity of some nonprehensile manipulation primitives. In general, experiments involving nonprehensile actions are not easy to solve due to the uncertain dynamics mainly due to friction: parameters estimation and/or robust controllers are thus essential. Moreover, physics terms causing non-smooth behaviour are often neglected when deriving the mathematical model of a given nonprehensile task: this makes the nonprehensile system look like a prehensile one. This happens for instance within rolling, sliding and pushing nonprehensile manipulation primitives. The proof that the designed controller does not violate the given assumptions is usually performed a-posteriori. A method to directly control the contact forces should be indeed addressed and this might be a future research direction, leading to the design of non-smooth and hybrid controllers which are also a new frontier for the research community.

Another approach might instead be the observation of pizza chef activities to learn task simplification and synthesize human-inspired control strategies. For instance, an integrated robotic platform able to acquire and transfer human body motion to a robotic system is obtained by interfacing RODYMAN with a low-cost motion capture system (see Fig. 7). Once the teleoperation algorithm for real-time replication of human motion on RODYMAN is developed, a comprehensive taxonomy of dynamic prehensile and non-prehensile tasks, ordered for different levels of hand-arm and dual-arm coordination, can be built from scratch. To this aim, taking inspiration from the researches conducted on anthropomorphic hands [START_REF] Ficuciello | Postural synergies of the UB Hand IV for human-like grasping[END_REF] or a single hand-arm system [START_REF] Ficuciello | Synergy-based policy improvement with path integrals for anthropomorphic hands[END_REF], a study on postural synergies for dual-arm robotic manipulation can be conducted to develop a framework where to simplify learning strategies from human imitation. Such approach will also take advantage of dimensionality reduction strategy to successfully apply supervised reinforcement learning algorithms using synergistic motion. Today, from these observations, it has been learned that the motion planner is crucial for nonprehensile tasks since the repetitive actions seem well imprinted in the pizza chef's mind, while the corrections made by the hands are very small despite the difference between various doughs. Therefore, it is reasonable to state that a good motion planner is the essential instrument within nonprehensile manipulation.

One further question that may arise is: why has the pizzamaking procedure been taken as example? Is there the need of having a robot making pizza? In truth, the pizza-making process is only a media expedient with scientific purposes. It is indeed clear that if a robot is able to manipulate a pizza dough, it might be able to perform similar difficult manipulation tasks. For instance, in 1997 the RoboCup started certainly not to replace the real soccer players, but rather to advance the state of the art while facing both gaming and difficult problems for robots. With the same aim, RODYMAN is trying to mimic the artistic ability of a pizza chef. While facing this big challenge, many sub-problems have to be addressed in parallel which could have an impact in other domains. The perception of elastic objects is currently being applied in the medical context to shape variations of muscles and organs. The manipulation performed while tossing the deformable dough is currently under investigation to improve the automation of gluing the shoes' lower surfaces. The batting process has similar dynamics to the walking gaits, and as such it could be used to improve autonomy of humanoids, or employed for actuated prostheses.

Fig. 1 :

 1 Fig. 1: The RODYMAN platform handling the peel with two arms and two proper grippers at the end-effectors. The displayed tool is a real pizza peel employed by chefs to cook the dough in the oven. A blue silicon disk, usually employed by acrobatic pizza chefs for training, is employed in the experiments.

Fig. 2 :Fig. 3 :

 23 Fig. 2: RODYMAN high-level control architecture.

Fig. 4 :

 4 Fig.4: Overview of the developed approach for deformable object tracking.

Fig. 5 :

 5 Fig. 5: Results of the tracking process with the input images (4 different objects in the first row), and the corresponding registered re-projected mesh (second row, in red for object (b), in blue for objects (b) and (d), in white for object (c)).

  sh = -b -1 22 b 12 a h + c 21 θh + c 22 ṡh + g 2 , (1b) where s h ∈ R is the contact position of the object on the hand, whose shape is parametrized through arclength, b 12 ∈ R and b 22 ∈ R are entries of the inertia matrix B ∈ R 2×2 , while c 21 ∈ R and c 22 ∈ R are entries of the (2 × 2)

Fig. 7 :

 7 Fig. 7: The renowned pizza chef Enzo Coccia wearing the Xsens MVN suite, with the RODYMAN avatar in the background acquiring and repeating the movements of the chef.

  , L. Villani and B. Siciliano are with CREATE Consortium and with Department of Electrical Engineering and Information Technology, University of Naples Federico II, via Claudio 21, 80125, Naples, Italy. A. Petit is with MIMESIS team, INRIA, 1, place de l'Hopital, 67000 Strasbourg, France. D. Serra is with Rete Ferroviaria Italiana SpA, Via De'Cattani 1, 50145, Osmannoro. A.C. Satici is with Boise State University, 1910 W University Dr, Boise, United States. A. Donaire is with Queensland University of Technology, 2 George st, 4000, Brisbane, QLD, Australia. L.R. Buonocore is with CERN, Route de Meyrin 385, Geneve 23, Switzerland.

https://www.eu-robotics.net/cms/upload/topic˙groups/SRA2020˙SPARC.pdf

http://rtc.nagoya.riken.jp/RIBA/index-e.html

http://www.shrine-project.eu

https://www.youtube.com/user/ThePRISMAlab

ACKNOWLEDGMENT

The research leading to these results has been supported by the RoDyMan project, which has received funding from the European Research Council FP7 Ideas under Advanced Grant agreement number 320992. The authors are solely responsible for the content of this manuscript.


Quentin Brabant

Miguel Couceiro
email: miguel.couceiro@loria.fr

Didier Dubois

Henri Prade
email: prade@irit.fr

Agnès Rico
email: agnes.rico@univ-lyon1.fr

Sugeno Integral for Rule-Based Ordinal Classification

We present a method for modeling empirical data by a rule set in ordinal classification problems. This method is nonparametric and uses an intermediary model based on Sugeno integral. The accuracy of rule sets thus obtained is competitive with other rule-based classifiers. Special attention is given to the length of rules, i.e., number of conditions.

Introduction

Let X = X 1 × • • • × X n , where each X i is a totally ordered set called attribute domain, and let L be a totally ordered set, whose elements are called classes. The minimal and maximal element of any totally ordered set X are denoted by 0 X and 1 X , respectively. An instance is a pair (x, y) ∈ X × L. A dataset is a collection of instances (in which the same instance can appear several times). A model of a dataset is a function f : X → L, and its accuracy is the proportion of instances for which f (x) = y in agreement with the dataset.

We consider datasets that can be accurately modeled by a nondecreasing function. Such datasets are typically found in Multi-Criteria Decision Aid, where evaluation of alternatives depends on several criteria, but also in some medical diagnosis problems. The task of finding an accurate nondecreasing model of a dataset has been addressed in several ways (see, e.g., [START_REF] Gutiérrez | [END_REF]). In this short paper, we focus on rule-based models, since rules provide an explicit justification for each class prediction they make. We consider sets of (selection) rules of the form if x 1 ≥ α 1 and . . . and x n ≥ α n then y ≥ δ

(1)

where (α 1 , . . . , α n ) ∈ X. The VC-DomLEM algorithm [Blaszczyński et al., 2011] allows us to learn such a set of rules, which yields a good accuracy compared to other interpretable models.

In [Brabant et al., 2018], we proposed an alternative method for learning rule sets, which relies on Sugeno integrals. This method does not require the tuning of any hyperparameter and is competitive with VC-DomLEM in terms of accuracy. Moreover, this method raised new questions about the relevance of capacities (i.e., monotonically increasing set functions) in data-modeling.

Rule-based and capacity based models

Let R be a set of rules of the form (1). There may be several functions that are compatible with R. We denote by f R the smallest function compatible with R, defined by f R (x) = max r∈R f r such that for each rule r:

f r (x) = δ r , if ∀i ∈ C, x i ≥ α r
i , and 0 otherwise. We will say that a function

f is equivalent to R if f = f R .
In what follows, we use the notation [n] = {1, . . . , n}. Let µ : 2 [n] → L be a capacity, i.e., a set function 2

[n] → L such that µ(∅) = 0 L , µ([n]) = 1 L , and µ(I) ≤ µ(J) for all I ⊆ J ⊆ [n]. The Sugeno integral w.r.t. µ is the aggregation function S µ : L n → L defined by S µ (x 1 , . . . , x n ) = max I⊆[n]
(min(µ(I), min

i∈I x i )).
Note that the Sugeno integral can be a model for ordinal classification only if

X 1 = • • • = X n = L. A Sugeno util- ity functional (SUF) is a more expressive model which can merge values from different scales. Let ϕ = (ϕ 1 , . . . , ϕ n ), where each ϕ i : X i → L is a nondecreasing function such that ϕ i (0 Xi ) = 0 L and ϕ i (1 Xi ) = 1 L . The SUF S µ,ϕ is the function defined by S µ,ϕ (x 1 , . . . , x n ) = S µ (ϕ 1 (x 1 ), . . . , ϕ n (x n )).
It was shown in [Brabant et al., 2018] that:

1. Any SUF is equivalent to a rule set.

2. Any single rule is equivalent to a SUF. 3. Some rule sets are not equivalent to a single SUF. In other words, some combinations of rules cannot be expressed by one SUF. However, the second assertion allows to say that any rule set is equivalent to some function M S : X → L defined by M S (x) = max{S µ,ϕ (x) | S µ,ϕ ∈ S}, where S is a set of SUFs. We call such function a max-SUF.

There is no reason to think that a max-SUF provides a better interpretability than its equivalent rule set. However, an interesting question is whether it can serve as an intermediary model that helps guiding the learning process of a rule based model. Indeed, in [Brabant et al., 2018], it is shown that a non-parametric learning algorithm based on a max-SUF is competitive with VC-DomLEM.

3 From SUFs to rule sets and vice-versa Any SUF S ϕ,µ is equivalent to the rule set

I⊆[n] δ≤µ(I) {∀i ∈ I, x i ≥ α i ⇒ y ≥ δ}, (2)
where

α i = min{a ∈ X i | ϕ i (a) ≥ δ}.
Note that this set is likely to contain redundant rules. Now let us show a method of translation of a rule set R into a SUF.

1. Initialize µ and ϕ = (ϕ 1 , . . . , ϕ n ) with minimal values.

For each rule x

1 ≥ α 1 , . . . , x n ≥ α n ⇒ y ≥ δ in R: (a) let A = {i ∈ [n] | α i > 0}, (b) increase µ(A) up to δ, (c) for each i ∈ A, increase ϕ i (α i ) up to δ
After these steps we always have

S µ,ϕ ≥ f R . When S µ,ϕ > f R , no SUF is equivalent to R.
In some cases, it is not problematic that S µ,ϕ > f R . For example, if f R is a model of a dataset D, we may want to find an SUF that best fits D. Obtaining S µ,ϕ = f R is not always possible since SUFs are not expressive enough. However, equality can be always achieved using a max-SUF [Brabant et al., 2018]. The method presented in the next section relies on this fact.

Learning rules from empirical data

Let D be a dataset. The following three steps provide a method for modeling D by a max-SUF.

1. Selection of an order-preserving subset of data. Two instances (x, y) and (x', y ) can be anti-monotonic together, i.e, x ≤ x' and y ≤ y. We iteratively remove instances from D, starting from those that are anti-monotonic with the highest number of other instances, until no anti-monotonic pair remains. We denote by D -the dataset obtained in this way.

2. Modeling D -by a rule set R. Initialize R to ∅. For each instance ((a 1 , . . . , a n ), y) in D -, search for A ⊆ [n] with minimal cardinality, such that the rule

∀i ∈ A, x i ≥ a i ⇒ y ≥ δ, (3)
is not contradicted by any instance in D -. Add the rule (3) to R. At the end of this step, the class of each instance in D -is exactly predicted by f R .

3. Translation of R into a max-SUF. See Algorithm 1. The obtained max-SUF is not necessarily equivalent to R, but it fits D -precisely.

Note that the max-SUF given by this method can be translated back into a rule set, which constitutes an equivalent model and is easier to interpret.

Algorithm 1: Makes a partition P of R such that the max-SUF M S verifies M S (x) = y for each instance (x, y).

Empirical study

We compared our method to VC-DomLEM on the 12 datasets in [Blaszczyński et al., 2011]. In order to show the importance of Step 3 in our method, we separately evaluated the rule set given by steps 1 and 2 alone, and the max-SUF given by steps 1,2, and 3. We see that Step 3 do not increases the accuracy on average. Therefore, the good results of this method are not due to the use of SUFs, but to the 2 first steps.

The length of a rule is the number of attributes X i where α i > 0 Xi (since the condition α i ≥ 0 Xi is trivial). Shorter rules are easier to interpret and constitute more concise models. Table 2 shows the rule length distribution obtained after steps 1, 2, and 3. The dual of max-SUFs are the min-SUFs that correspond to sets of (rejection) rules of the form if x 1 ≤ α 1 and . . . and x n ≤ α n then y ≤ δ.

When learning min-SUFs by a dual method, the rule-length distribution differs from that obtained by learning max-SUFs. Long rules of one type sometimes go along with short rules of the other type.

  a SUF S µ,ϕ 6 if S µ,ϕ (x) ≤ y for all instance (x, y) in D -

Table 1 :

 1 to P Accuracy obtained with each method on each dataset. Datasets are numbered as in[Blaszczyński et al., 2011]

		1	2	3	4	5	6	7	8	9 10 11 12 avg.
	Steps 1,2.	74.4 95.8 97.7 93.6 91.7 65.6 83.2 27 67 63.6 58.2 51.4 72.4
	Steps 1,2,3.	76 95.3 97.2 89.3 92.4 65.2 84.5 26.4 69.4 63 56.7 53.2 72.4
	VC-DomLEM 76.7 96.3 97.1 91.7 95.4 67.5 87.7 26.9 66.7 55.6 56.4 54.6 72.7

Table 2 :

 2 [Blaszczyński et al., 2011] J. Blaszczyński, R. Slowiński, and M. Szelag. Sequential covering rule induction algo-Percentage of rules with a given length, for each data set.

									Rule length
			1	2	3	4	5	6	7	8	9 10 11 12 13 14 15 16
		1		1	64	23	11	1
		2	13	72	13	1	1	1	1	1
		3			3	24	41	32
		4	38	47	10	3	1	1
	dataset	5 6 7 8	3 2 6	25 20 20 62	31 37 21 16	13 22 21 16	5 6 12	2 2 9	3 1 15	2 1 3	1 1	1 9	3	5	3	1	1
		9	6	26	36	32
		10	5	26	42	27
		11	6	2	25	45	9	5	4	1		3
		12	3	24	44	21	5	1				2