Antoine Petit
email: antoine.petit@unina.it

Vincenzo Lippiello
email: vincenzo.lippiello@unina.it

Bruno Siciliano Dieti

Tracking fractures of deformable objects in real-time with an RGB-D sensor

This paper introduces a method able to track in real-time a 3D elastic deformable objects which undergo fractures, using the point cloud data provided by an RGB-D sensor. Our framework relies on a prior visual segmentation of the object in the image. The segmented point cloud is registered by non-rigidly fitting the mesh, based on the Finite Element Method to physically model elasticity, and on geometrical point-to-point correspondences to compute external forces exerted on the mesh. Fractures are handled by processing the stress tensors computed on the mesh of the FEM model, in order to detect fracturable nodes. Local remeshing around fracturable nodes is then performed to propagate the fracture. The real-time performance of the system is demonstrated on real data involving various deformations and fractures.

Introduction

Tracking non-rigid objects has aroused much interest in recent years in the computer vision, computer graphics and robotics communities, addressing various applications in fields such as augmented reality, medical imaging, robotic manipulation, by handling a huge variety of objects: tissues, paper, rubber, viscous fluids, cables, food, organs, etc.

This study comes within the scope of the RoDyMan project 2 , consisting in a unified framework for robotic dynamic manipulation of deformable objects, as shown by the demonstration scenario in Fig. 1.

A particularly novel challenge lies in dealing with fractures of deformable objects, and in designing a model of both deformations and fractures, and fitting it with vision and range data. This registration problem also involves critical real-time concerns, which are especially required for instance for robotic dynamic manipulation. Although some recent studies have proposed efficient techniques to handle 3D objects which undergo isometric or elastic deformations, to our knowledge there exists no real-time 3D vision system which considers topological changes undergone by the objects, such as fractures. The aim of this paper is thus to propose, based on the method suggested in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF], a realtime tracking system able to handle elastic objects which may get fractured, using data provided by an RGB-D sensor. To cope with elastic deformations and fractures, our approach involves a physical modeling of the considered object, by relying on a Finite Element Method (FEM) model, with an implementation running in real-time. Our whole system is indeed able to run at around 30 frames per seconds. The remainder of the paper is organized as follows. Some works related to ours are presented in Sect. 2 and the system is outlined in Sect. 2.5. In Sect. 3 the physical deformation and fracture model of the object is introduced, Sect. 4 explains how the point cloud data is processed and matched with this model to perform registration. Finally, some experimental results are presented in Sect. [START_REF] Elbrechter | Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling[END_REF].

Related works and motivations

In the literature, the various approaches proposed to register deformable objects, using range and vision data, could be classified according to the underlying model of the considered object and its physical realism.

Registration using implicit physical modeling

Based on implicit physical models, approaches in [START_REF] Kass | Snakes: Active contour models[END_REF][START_REF] Bartoli | Direct estimation of nonrigid registrations[END_REF][START_REF] Pilet | Fast non-rigid surface detection, registration and realistic augmentation[END_REF] use a 1D parametric curve or 2D splines models (Bsplines, Radial Basis Functions) to track deformable objects in monocular images. This class of methods relies on the minimization of an energy function involving an external energy term related to some image features, and an internal energy term regularizing curvature, bending or twisting, compelling the model to vary smoothly. With point cloud data, methods in [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF][START_REF] Zollhöfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF] employ an RGB-D sensor to register the acquired point cloud to a surface mesh by minimizing an error function accounting for geometric or direct depth and color errors, and a stretching penalty function for the mesh. By means of a NURBS parametrization [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF] or an optimized GPU implementation [START_REF] Zollhöfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF], real-time performance can be achieved.

Registration using explicit physical modeling

Instead, another formulation of the problem relies on physics-based deformable models to perform registration, by modeling more explicitly elasticity. With respect to implicit methods, other sorts (such as non-linear elasticity) and magnitudes of deformations can be treated, inferring more consistently shape and/or volumetric regularization. Statistically, the solution can be determined, by setting internal and external forces equal or, equivalently, minimizing energy functions. Physics-based methods include discrete mass-spring-damper systems [START_REF] Lippiello | Floating visual grasp of unknown objects using an elastic reconstruction surface[END_REF][START_REF] Elbrechter | Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling[END_REF][START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], or more explicit approaches relying on the Finite Element Method (FEM), based on continuum mechanics. In [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], based on mass-spring-damper systems, 3D-3D correspondences, determined through a probabilistic inference, enable the computation of the external forces applied to the mesh. First attempts for registration employing the FEM for 3D surfaces in [START_REF] Cohen | Deformable models for 3-d medical images using finite elements and balloons[END_REF][START_REF] Mcinerney | A finite element model for 3d shape reconstruction and nonrigid motion tracking[END_REF] used linear elasticity FEM models. Haouchine et al. [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF] uses a linear tetrahedral co-rotational FEM model, coping with larger elastic deformations, external forces being related to correspondences between tracked 3D feature points mapped to the 3D mesh by means of a stereo camera system.

Handling topological changes

As methods dealing with topological changes of a 3D deformable object, we can mention 3D reconstruction methods [START_REF] Varanasi | Temporal surface tracking using mesh evolution[END_REF][START_REF] Zaharescu | Topology-adaptive mesh deformation for surface evolution, morphing, and multiview reconstruction[END_REF], based on the TransforMesh system to address mesh evolution and topological changes. However these are model-free methods and reconstruct a single mesh at each time step [START_REF] Zaharescu | Topology-adaptive mesh deformation for surface evolution, morphing, and multiview reconstruction[END_REF] or are not suited for real-time applications [START_REF] Varanasi | Temporal surface tracking using mesh evolution[END_REF]. Instead, the goal is here to continuously estimate the rigid transformations, the deformations and splitting topological change (or fractures) undergone by the object, modeled by a known mesh, in real-time.

Motivations and contributions

Since our system would attempt to handle elastic volumetric strains and fractures, a realistic mechanical model, based on the FEM, has been adopted. The recent suitabil-ity of these physical models for real-time applications, as demonstrated by promising approaches [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF][START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF], has confirmed our choice. In contrast to [START_REF] Varanasi | Temporal surface tracking using mesh evolution[END_REF][START_REF] Zaharescu | Topology-adaptive mesh deformation for surface evolution, morphing, and multiview reconstruction[END_REF], fractures could this way be modeled physically, using linear elastic fracture mechanics, through the internal forces acting on the object. We assume the prior knowledge of a consistent mesh (which could be automatically reconstructed offline) and of the material properties (through the Young modulus, the Poisson ratio, and a fracture threshold), which could be estimated offline. With respect to related approaches [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF][START_REF] Schulman | Tracking deformable objects with point clouds[END_REF][START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF][START_REF] Zollhöfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF], our main contribution has been to integrate the fracture model, along with being able to track large deformations, and handling these tasks in real time (30 fps), making our solution the first real-time vision tracking system able to deal with deformations and fractures of a non-rigid objects, using a RGB-D sensor.

Overview of the system

Our tracking system can be outlined as follows: Input : the 3D volumetric mesh of the object, a given RGB-D data, and assuming a fair registration at the previous time step. [START_REF] Anderson | Fracture mechanics: fundamentals and applications[END_REF]. Visual segmentation of the considered object, with a graph cut-based approach ensuring temporal coherence.

2. Using the resulting segmented point cloud, compute external linear elastic forces exerted on the vertices of the mesh.

3. Compute elastic stresses and internal forces, based on a tetrahedral linear co-rotational FEM model.

4. Numerical resolution of mechanical equations to compute the deformations.

5. Detect fracturable nodes in the mesh.

6. Split the fracturable nodes and perform local remeshing around them according to the fracture plane.

Modeling deformations and fracture: an FEM approach

Since we deal with objects which may undergo elastic deformations, a major issue lies in the definition of a relevant physical model. The Finite Element Method (FEM) [START_REF] Cook | Finite element modeling for stress analysis[END_REF] provides a realistic physical model, by relying on continuum mechanics, instead of finite differences for massspring systems for instance. It consists in tessellating the deformable object into a mesh made of elements. We rely here on a volumetric linear FEM approach with tetrahedral elements. The deformation field u e over an element e is then approximated as a continuous interpolation of the displacements ûe of its four vertices, through a polynomial basis function contained in the 3 × 12 matrix N e (x):

u e (x) = N e (x)û e , (1)
with being ûe = x e -x e,0 , x e and x e,0 respectively the deformed and initial world coordinates of the vertices of e.

Modeling elastic deformations

In order to model deformations and elasticity, we resort to the infinitesimal strain theory to compute the Cauchy's linear strain tensor e within the tetrahedron. It can be linearly expressed with respect to ûe , using Voigt notations since e is symmetric:

e = L e ûe , (2)
with L e a constant 6 × 12 matrix. We then rely on Hooke's law linear elasticity theory for a continuous isotropic material, which lead us to write the infinitesimal stress tensor σ e in the element as:

σ e = C e e = C e L e ûe , (3)
where C e is a 6 × 6 symmetric matrix depending on two elastic parameters of the material, the Young modulus E and the Poisson ratio ν. By deriving the strain energy in e, the internal elastic forces f e exerted on the four vertices of e of the mesh can be linearly related to their displacements ûe :

f e = K e ûe = V e L T e σ e . (4)
Although it is insensitive to translation transformations, the model, by using an infinitesimal approximation of the strain tensor, is however inaccurate when modeling large rotations of the elements, leading for instance to non-zero summations of the forces. A work-around consists in the co-rotational approach [START_REF] Etzmuß | A fast finite element solution for cloth modelling[END_REF], used for registration purposes in [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF], which is a good compromise between the ability to model large linear elastic deformations and computational efficiency. Since the displacement of an element can be decomposed into a rigid transformation and a pure deformation, the idea is to extract the rotation matrix R e related to the rigid transformation. Then the stiffness matrix can be warped with respect to this rotation, so as to accommodate rotation transformations, giving:

f e = R e K e ûr e = R e K e (R -1 e x e -x e,0 ), (5)
with being ûr e = R -1 e x e -x e,0 , with R -1 e x e the deformed coordinates of the vertices of e back rotated to the unrotated frame, the forces K e ûr e being then re-rotated to the current deformed element through the multiplication by R e . In this way, the overall forces on the whole mesh can be summed to zero, while computational efficiency is ensured since K e can be computed in advance, in contrast to non-linear FEM approaches.

Modeling fractures

In the computer graphics community, various systems modeling fractures have been proposed. The detection of fractures or cracks in 3D object simulations has been initially achieved using mass-spring systems [START_REF] Norton | Animation of fracture by physical modeling[END_REF], by removing springs whose lengths exceed a certain threshold. Based on the FEM, [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF] proposed a physically consistent method using the linear elastic fracture theory [START_REF] Anderson | Fracture mechanics: fundamentals and applications[END_REF], through the computation, for given strains, of tensile forces undergone by the nodes. A fracture is then propagated from each detected fracturable node (or crack tip) by subdividing the neighboring tetrahedra according to a fracture plane. In a similar manner, [START_REF] Müller | Realtime simulation of deformation and fracture of stiff materials[END_REF][START_REF] Müller | Interactive virtual materials[END_REF] have suggested a more computationally efficient solution by resorting to element wise stresses, the fracture being propagated along element boundaries. Although improvements have been achieved from these widespread approaches, for both fracture detection and propagation [START_REF] Glondu | Real-time simulation of brittle fracture using modal analysis[END_REF], we propose here an approach based on [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF]. One reason, as justified hereafter in section 3.2.2, is to be able to continuously cope properly with the provided noisy point cloud data. Another reason is to preserve real-time performances.

Detecting fractures

Detecting fractures in the mesh is performed in a similar way to [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF], by decomposing the internal forces exerted on the nodes into tensile and compressive forces, with the difference that [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF] relies on a standard linear FEM method.

For this purpose, the stress tensor σ e undergone by a tetrahedra is separated into a tensile and a compressive components. Here, using a co-rotational approach and based on (5) and (3), we compute σ e as:

σ e = C e L e ûr e = C e L e R -1 e x e -x e,0 . (6)
Then, as in [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF], the three eigenvalues v i (σ e ) and eigenvectors n i (σ e ) of σ e are computed, the positive eigenvalues ones corresponding to tensile stresses whereas negative ones to compressive stresses. Tensile and compressive tensors can be then defined as:

σ + e = i max(0, v i (σ e ))m(n i (σ e )) (7)
σ

- e = i min(0, v i (σ e )m(n i (σ e )), (8)
with, for any vector

a ∈ R 3 m(a) = aa T a if a > 0 0 if a = 0. (9)
Based on these tensile and compressive stresses, internal forces exerted on the vertices of e can be decomposed as well, using (4):

f + e = V e L T e σ + e (10) f - e = V e L T e σ - e . (11)
Let us first define X = {x j } n X j=1 the set of vertices of the mesh. Then for each node x j in the mesh, we can compute, as in [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF], the Separation Tensor ζ j , using the sets {f + e,j |j ∈ e} e and {f - e,j |j ∈ e} e respectively of tensile and compressive forces exerted by the elements {e|j ∈ e} attached to the node:

ζ j = 1 2 (-m(f + j ) + fe,j ∈{f + e,j |j∈e}e m(f e,j ) +m(f - j ) - fe,j ∈{f - e,j |j∈e}e m(f e,j )) (12)
with

f + j = fe,j ∈{f + j |j∈e}e f e,j and f - j = fe,j ∈{f - j |j∈e}e f e,j (13)
ζ j is a tensor which enables to evaluate deformation imbalance between tensile and compressive forces, while being invariant to imbalance resulting in rigid motions. A fracture at node x j is then detected if the largest positive eigenvalue v + j of ζ j is above a certain threshold corresponding to the toughness of the material. The eigenvector n j corresponding to v + j defines the normal to the fracture plane at node x j .

Fracture propagation and remeshing

In order to propagate the fracture from the detected fracturable nodes, in the direction of the fracture plane, the mesh around the considered node shall be modified. [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF] proposes to subdivide the elements which are attached to the node and intersected by the fracture plane, according to the orientation of the plane. Other neighboring elements also need to be modified to maintain the consistency of the mesh. However, such a remeshing process, which is proper and suitable for simulations purposes since it manages to maintain the orientation of the fracture, is computationally costly and tends to generate an always growing number of elements, what is not desirable in our case to keep real-time performances. Also, for registration concerns, by changing locally and drastically the resolution of the mesh, matching the remeshed regions with the acquired point cloud, which is of constant and homogeneous resolution, while being noisy, may lead to heterogeneous or unstable behaviors. Another approach, suggested in [START_REF] Smith | Fast and controllable simulation of the shattering of brittle objects[END_REF][START_REF] Müller | Realtime simulation of deformation and fracture of stiff materials[END_REF][START_REF] Müller | Interactive virtual materials[END_REF], consists in propagating the fracture along the boundaries of the neighboring elements. This method, although it enables to keep the resolution constant, tends to produce jumbled fracture patterns and artifacts, which is also not desirable in our case when coping with the noisy point cloud data.

We have instead opted here for a solution preserving the mesh homogeneity and performing a neat fracture propagation. It is shown in Fig. 2. As in [START_REF] O'brien | Graphical modeling and animation of brittle fracture[END_REF], the fracturable node is first replicated, with the same positions. Then, within the set of elements attached to the original fracturable node, the ones which are not intersected by the given fracture plane are determined and are reassigned to the original fracturable node or to the replicated one, given on which side of the plane they are located. This procedure appears coarse, affects the volume of the model and tends to degenerate the mesh. But it remains acceptable since we consider, in the experimental scenarios proposed in Sect. 5, objects undergoing simple fracture events, so that element losses in the mesh can be neglected.

x j n j x j x j f + + - - ∩ ∩ Figure 2:
Remeshing procedure, represented here in the 2D case with triangles for simplicity. The fracturable node xj is replicated, initially with the same position, providing a new vertex x f j for the mesh (it is here translated from xj for clarity). The intersected elements are removed. The elements on the positive side of the fracture plane (determined by its normal nj) assigned to x f j and the others remain attached to xj.

Registration with the point cloud data

The deformable registration problem consists in fitting the point cloud data provided by an RGB-D sensor, with the tetrahedral mesh. The basic idea is to derive external forces exerted by the point cloud on the mesh and to integrate these forces, along with the internal forces computed using the physical model presented in Sect. 3, into a numerical solver solving the resulting mechanical equations.

Segmentation and deformable iterative closest point

The approach presented here uses the same methods as the ones described in the work [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF] to segment the point cloud, relying on the Grabcut visual segmentation technique, in order to restrict the acquired point cloud to the considered object, so as to avoid ambiguities, in the matching process with the background or with occluding shapes.

The deformable iterative procedure worked out in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF] is also employed, first to determine the nearest neighbors correspondences, both from the segmented point cloud to the mesh and from the mesh to the segmented point cloud.

Based on the two sets of mesh-to-point cloud and point cloud-to-mesh correspondences, external elastic forces exerted on the vertices of the mesh can be computed. When computing these forces, a trade-off has to be found between these two sets of correspondences, whether the application deals with stretching or compression actions on the object, and whether occlusions or segmentation errors are to be dealt with. Let us note that the contour weighting function proposed in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF] is not used here and that occlusions is not an issue we want to cope with in this case.

Estimating the deformations of the mesh consists in solving a dynamic system of non-linear ordinary differential equations involving the internal forces, determined with (5), and the external forces, based on Lagrangian dynamics.

Processing fractures

Once deformations are computed, topological changes due to fractures, as modeled in Sect. 3.2, are handled. For each vertex x j in X, we detect if it is a fracture node, based on Sect. 3.2.1, and for fracturable nodes the remeshing procedure presented in Sect. 3.2.2. Let us not that for remeshing, an element attached to different fracturable nodes is treated only once, by the first investigated node. At the end of the procedure a new mesh X f is obtained and will be directly employed for the next frame.

Experimental results

In order to evaluate the performance of our method and contributions, some experimental results are shown in this section, in a qualitative manner on some real data.

Implementation and experimental set-up

For the non-rigid registration phase, we have employed the Simulation Open Framework Architecture (SOFA) simulator [START_REF] Faure | Sofa: A multi-model framework for interactive physical simulation[END_REF], which enables to deal with various physical models and to evolve simulations in real-time.

In order to carry out experiments on real data, the point cloud of the investigated scene is acquired from a calibrated RGB-D camera Asus Xtion, 320 × 240 RGB and depth images being processed. A standard laptop with an NVIDIA GeForce 720M graphic card has been used, along with a 2.4GHz Intel Core i7 CPU. The segmentation process relies on a CUDA implementation. The results presented here deal with elastic bar and plate objects, made of modeling clay.

For the cylindrical bar object, the involved mesh, depicted in Fig. 3, has a circumferential/radial/height resolution of 10 × 20 × 2, resulting in 220 vertices and 570 tetrahedral elements. The material is here poorly elastic, the Young Modulus being empirically set to E = 2.0M P a and the Poisson ratio to ν = 0.01. The plate object (see Fig. 3, right) instead has a length/width/height resolution of 18 × 9 × 2, resulting in 324 vertices and 816 elements. For material properties, we set E = 2.5M P a and ν = 0.01 in this case.

Three cases of fracture are here demonstrated: fracture due to opening by excessive bending for the bar object and fractures due to shearing and to opening by stretching for the plate object, tearing the object. Let us note that in the first case the parameter λ (0 ≤ λ ≤ 1) which tunes the balance between mesh-to-point cloud and point cloud-tomesh forces (see [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF]) is set to λ = 0.5. In the second case λ = 0.1 so that vertices of the mesh lying on the fracture crack of the object will be driven towards the nearest observed point on the border of the crack.

Let us note that in [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF], the registration process is iteratively repeated (3 iterations were performed in the experiments). Here only one iteration is executed to maintain realtime performances, since fracture detection remains costly (see Fig. 1).

Results

As seen in Fig. 4, opening though bending deformations can be tracked quite properly for the bar object, so tensile internal forces computed on the nodes remain physically consistent, making the fracture model valid, as shown on the third column. For the plate object, tearing it through out-ofplane shearing (see Fig. 5) can also be tracked, despite some artifacts can be noticed on some unfractured regions.

However, when opening the plate object by stretching it, as shown in Fig. 6, only vertices lying on the occluding con-tour of the mesh are attracted to the expanded areas in the point cloud, since correspondences are established based on 3D geometry, without any discriminative descriptors. As a consequence, forces attracting the contours and stretching the mesh are weak. Therefore, fracture in this case appears with a certain delay, when both fractured parts split away. However, the fracture crack is quite coherent and both parts are finally consistently recovered. Computational costs Regarding computational aspects, in Tab. 1 are shown the mean computation times of the various phases of the algorithm, for the sequence presented in Fig. 5. Ext. forces is the step involving the determination of the closest points between the mesh and the point cloud, and the computation the subsequent external forces exerted on the mesh. Resolution corresponds to the computation of the internal forces and in the resolution of the Lagrangian mechanical equations, to compute the deformations, based on the computed external and internal forces. Fracture detection consists in computing the separation tensors to detect fractures on the nodes of the mesh. Finally Remeshing shows the time dedicated to remesh the mesh around the fracturable nodes. As noticed, the suggested method runs on this sequence at around 27 fps.

Phases

Mean execution times (ms)

Discussion

The proposed approach is a first simple attempt to deal with fractures of deformable objects. Although the results presented above are promising and show the relevance of employing a physically rigorous method to detect fracture, several issues shall be discussed.

Stiffness of the materials

Experiments are shown on two different simple objects with particular material properties (high stiffness), the proposed method being for the moment mostly suitable for such materials. With more elastic materials, the estimated deformations and internal forces of the mesh would be rougher, resulting in a more variable sepa-ration tensor and so potentially spurious detections of fractures.

Tuning material parameters A cumbersome process can lie in the determination of coherent material properties of the object, which consist in the Young modulus, the Poisson ratio and the fracture threshold. There are set empirically in this paper and having a fair physical realism may require some efforts.

Tracking stretching deformations As observed in the experiments, our method faces some problems when tracking opening tearing fractures due to the expansion of the area covered by the object, since only the vertices lying on the contour of the mesh are attracted. A method proposed in the work [START_REF] Petit | Real-time tracking of 3d elastic objects with an rgb-d sensor[END_REF] suggests to weight the vertices of the visible surface of the mesh, given their distance to the occluding contour of the mesh. However, this technique tends to alter the physical homogeneity of the material, making in the case of a fracture model fractures to likely occur around the contour of the mesh.

Sensitivity to occlusions Since in the segmentation no distinction is made between a fracture crack and unobserved areas of the object due to occlusions or segmentation errors, our method remains sensitive to these latter events.

Remeshing Handling the propagation of a fracture by removing the elements attached to a fracturable node and intersected by the fracture plane leads to a progressive degeneration of the mesh if several fracture events occur.

Conclusion

In this paper we propose a method to track fractures of deformable objects, as a first attempt to deal with such topological changes. The main idea has been to integrate in a physical FEM based deformation model a model of fracture events, by first detecting fracturable nodes in the considered mesh, by decomposing the internal forces into tensile and compressive components. A second step consists in a simple local remeshing of the elements around the fracture nodes. The registration process with the RGB-D data involves performing a deformable iterative closest point procedure to compute the deformations and finally investigating fracture events over the mesh and eventually carrying out remeshing. Experimental results on two different objects and three sorts of fractures show the relevance of this simple physical based method, which can however be improved by handling more properly the remeshing phase, by dealing with larger elastic, due to stretching actions for instance, and plastic deformations, and by handling occlusions by distinguishing them with fracture cracks.

Figure 1 :

 1 Figure 1: Artistic views of the RoDyMan robotic platform and the pizza making process.

Figure 3 :

 3 Figure 3: Meshes of both the cylindrical bar (left) and plate (right) objects.

Figure 4 :

 4 Figure 4: Results of the tracking process for the cylindrical bar object, with the input images (first row), the segmented frames (second row), and the registered mesh reprojected in the input image.

Figure 5 :

 5 Figure 5: Results of the tracking process for the plate object, torn through out-of-plane shearing, with the input images (first row), the segmented frames (second row), and the registered mesh reprojected in the input image.

Figure 6 :

 6 Figure 6: Results of the tracking process for the plate object, torn through opening, with the input images (first row), the segmented frames (second row), and the registered mesh reprojected in the input image.

Table 1 :

 1 Mean execution times, in milliseconds, for the different phases of the approach.

	Segmentation	10.1
	Ext. forces	3.9
	Resolution	2.3
	Int. forces & fracture detection	20.6
	Remeshing	0.6
	Total	37.5

http://www.rodyman.eu/ The research leading to these results has been supported by the RoDyMan project, which has received funding from the European Research Council (FP7 IDEAS) under Advanced Grant agreement

number 320992. The authors are solely responsible for its content. It does not represent the opinion of the European Community and the Community is not responsible for any use that might be made of the information contained therein.


Antoine Petit

Vincenzo Lippiello

Bruno Siciliano

Real-time tracking of 3D elastic objects with an RGB-D sensor

This paper presents a method to track in real-time a 3D textureless object which undergoes large deformations such as elastic ones, and rigid motions, using the point cloud data provided by an RGB-D sensor. This solution is expected to be useful for enhanced manipulation of humanoid robotic systems. Our framework relies on a prior visual segmentation of the object in the image. The segmented point cloud is registered first in a rigid manner and then by non-rigidly fitting the mesh, based on the Finite Element Method to model elasticity, and on geometrical point-to-point correspondences to compute external forces exerted on the mesh. The real-time performance of the system is demonstrated on synthetic and real data involving challenging deformations and motions.

I. INTRODUCTION

Unlike vision-based tracking problems with rigid objects, for which a certain maturity has been reached, perception for non-rigid objects is still a challenging problem. It has aroused much interest in recent years in the computer vision, computer graphics and robotics communities. A lot of potential applications would indeed be targeted, in fields such as augmented reality, medical imaging, robotic manipulation, by handling a huge variety of objects: tissues, paper, rubber, viscous fluids, cables, food, organs, etc.

This study comes within the scope of the RoDyMan project 2 , consisting in a unified framework for robotic dynamic manipulation of deformable objects. As seen in Fig. 1, a demonstration scenario is the humanoid dual-arm/hand manipulation of the pizza dough, in an authentic manner, showing a humanoid robot involved in culinary traditions and rituals.

With respect to rigid objects, the problem of dealing with deformations poses several additional challenges such as modeling the properties of the considered material, and fitting this model with the vision and/or range data. This registration problem also involves critical real-time concerns, which are especially required for robotic dynamic manipulation. Although numerous studies have proposed efficient real-time techniques to handle 3D surfaces (paper, clothes) which undergo isometric or slightly elastic deformations, a large open field remains when considering larger elastic deformations. The aim of this paper is thus to propose a real-time tracking system able to handle elastic objects, potentially textureless, by tracking large deformations and fast rigid motions, using visual and range data provided by an RGB-D sensor.

To cope with such deformations, our approach involves a physical modeling of the considered object, by relying on a Finite Element Method (FEM). The considerable progresses recently made within the computer graphics and medical simulation domains have enabled real-time performance for processing such models. As demonstrated in this paper, our whole system is able to run fastly at around 35 frame per seconds.

The remainder of the paper is organized as follows. Some works related to ours are presented in Sect. II. Our system requires the visual segmentation of the object, what is addressed in Sect. III. Then in Sect. IV the mechanical model of the object considered here is introduced, Sect. V explains how the point cloud data is processed and matched with the model to perform registration. Finally, some experimental results are presented in Sect. VI.

II. RELATED WORKS AND MOTIVATIONS

In the literature, the various approaches proposed to register deformable objects, using vision and/or range data, could be classified according to the underlying model of the considered object and its physical realism. Let us first clarify our scope and distinguish it from non-rigid reconstruction methods for which at each frame provided by the vision/range sensor, a single mesh is reconstructed, as in [START_REF] Bartoli | Coarse-to-fine low-rank structure-from-motion[END_REF][START_REF] Weiss | Home 3d body scans from noisy image and range data[END_REF][START_REF] Richard A Newcombe | Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time[END_REF]. Instead, the goal is here to continuously estimate the rigid transformations and the deformations undergone by a specific object, modeled by a known mesh.

A. Registration using implicit physical modeling

Based on implicit physical models, approaches in [START_REF] Kass | Snakes: Active contour models[END_REF][START_REF] Bartoli | Direct estimation of nonrigid registrations[END_REF][START_REF] Pilet | Fast non-rigid surface detection, registration and realistic augmentation[END_REF] use a 1D parametric curve or 2D splines models (Bsplines, Radial Basis Functions) to track deformable objects in monocular images. This class of methods relies on the minimization of an energy function involving an external energy term related to some image features, and an internal energy term regularizing curvature, bending or twisting, compelling the model to vary smoothly. Adapting these techniques to register with 3D shapes or surfaces in monocular images is much more complex, since 3D deformations can imply ambiguous 2D transformations, resulting in an underconstrained problem. A first attempt by Terzopoulos et al. [START_REF] Terzopoulos | Constraints on deformable models: Recovering 3d shape and nonrigid motion[END_REF], relying on 3D splines and inspired by [START_REF] Kass | Snakes: Active contour models[END_REF], densely processes gradient features, to compute the data energy term. Less ambiguous feature-based approaches such as [START_REF] Mathieu Salzmann | Surface deformation models for nonrigid 3d shape recovery[END_REF] have been preferred and additional constraints are often added to solve ambiguities. With point cloud data, methods in [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF][START_REF] Zollhöfer | Real-time nonrigid reconstruction using an rgb-d camera[END_REF] employ an RGB-D sensor to register the acquired point cloud to a surface mesh by minimizing an error function accounting for geometric or direct depth and color errors, and a stretching penalty function for the mesh. By means of a NURBS parametrization [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF] or an optimized GPU implementation [START_REF] Zollhöfer | Real-time nonrigid reconstruction using an rgb-d camera[END_REF], real-time performance can be achieved. Although these two systems have shown promising and impressive results, they are still bounded to isometric or slightly elastic deformations, by means of regularization functions proportional to squared distances between nodes of the mesh, whereas we wish to model elastic in more physically realistic manner, to handle larger strains. Another limitation of these methods is that they process mesh to input point cloud correspondences in their data error functions, and are thus sensitive to missing data, or unobserved areas of the considered object due to occlusions. We consider in this paper also correspondences from the input point cloud to the mesh, through the use of a segmentation method to restrict the input point cloud to the observed areas of the object, and based on these correspondences, the occluded or unobserved areas would not affect registration.

B. Registration using explicit physical modeling

Instead, another formulation of the problem relies on physics-based deformable models to perform registration, by modeling more explicitly elasticity. With respect to implicit methods, other sorts (such as non-linear elasticity) and magnitudes of deformations can be treated, inferring more consistently shape and/or volumetric regularization. Statistically, the solution can be determined, by setting internal and external forces equal or, equivalently, minimizing energy functions. Physics-based methods include discrete a mass-spring-damper system [START_REF] Lippiello | Floating visual grasp of unknown objects using an elastic reconstruction surface[END_REF][START_REF] Elbrechter | Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling[END_REF][START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], or more explicit approaches relying on the Finite Element Method (FEM), based on continuum mechanics. In [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], based on massspring-damper systems, 3D-3D correspondences, determined through a probabilistic inference, enable the computation of the external forces applied to the mesh. First attempts for registration employing the FEM for 3D surfaces in [START_REF] Laurent | Deformable models for 3-d medical images using finite elements and balloons[END_REF][START_REF] Mcinerney | A finite element model for 3d shape reconstruction and nonrigid motion tracking[END_REF] used linear elasticity FEM models. More recently, in [START_REF] Malti | Monocular template-based 3d reconstruction of extensible surfaces with local linear elasticity[END_REF], registration in monocular images is addressed by designing a stretching/shrinking energy using continuous mechanical constraints on 2D elements assuming linear elasticity, and some 3D boundary conditions. Haouchine et al. [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF] uses a linear tetrahedral co-rotational FEM model, coping with larger elastic deformations, external forces being related to correspondences between tracked 3D feature points mapped to the 3D mesh by means of a stereo camera system. To the best of our knowledge, this latter method proposes the most realistic physical elastic model within a real-time visionbased tracking system, and we propose a similar model in this paper.

C. Contributions

Since our system would attempt to handle large deformations and elastic volumetric strains, a realistic mechanical model, based on the FEM, has been adopted. Besides, for potential robotic dynamic manipulation applications, an explicit physical modeling would enable the reliable computation and prediction of internal forces undergone by the object and thus to perform proper force control tasks. The recent suitability of these models for real-time applications, as demonstrated by promising approaches [START_REF] Elbrechter | Bi-manual robotic paper manipulation based on real-time marker tracking and physical modelling[END_REF][START_REF] Schulman | Tracking deformable objects with point clouds[END_REF][START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF], has confirmed our choice. We assume the prior knowledge of a consistent mesh (which could be automatically reconstructed offline) and of the material properties (through the Young modulus and the Poisson ratio), which could be estimated offline. Robustness concerns, regarding for instance textureless objects, have lead us to rely on an RGB-D sensor.

Among the methods having the closest goals, motivations and constraints to ours, we can mention [START_REF] Jordt | Direct model-based tracking of 3d object deformations in depth and color video[END_REF][START_REF] Schulman | Tracking deformable objects with point clouds[END_REF][START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF][START_REF] Zollhöfer | Real-time nonrigid reconstruction using an rgb-d camera[END_REF]. With respect to them, several contributions are proposed, such as handling various large deformations like elastic ones, handling rigid motions, handling occlusions, and addressing all these tasks in real-time (35 fps).

D. Overview of the system

As also represented in Fig. 2, our tracking system can be outlined as follows: Input : the known 3D volumetric mesh of the object, a given RGB-D data , and assuming a fair registration at the previous time step.

1) Visual segmentation of the considered object, with a graph cut-based approach ensuring temporal coherence. 2) Using the resulted segmented point cloud, perform a rigid Iterative Closest Point (ICP) to estimate a rigid transformation from the point cloud to the mesh. 3) Using the resulting segmented point cloud, compute external linear elastic forces exerted on the vertices of the mesh from the point cloud to the mesh and conversely, based on closest point correspondences. 4) Compute elastic internal forces, based on a tetrahedral linear co-rotational FEM model. 5) Numerical resolution of mechanical equations.

III. SEGMENTATION

In this work we advocate the use of a prior segmentation step in order to restrict the acquired point cloud to the object of interest (see section V for a more detailed justification).

A. Grabcut segmentation

We rely here on the efficient and widespread GrabCut method [START_REF] Rother | Grabcut: Interactive foreground extraction using iterated graph cuts[END_REF], based on graph cuts. In its original formulation, the Grabcut algorithm addresses the visual bilayer segmentation task as an energy minimization problem, based on statistical models of the foreground (the object) and the background.

For an input image I, we denote by α = {α i } N i=1 the set of the unknown binary labels of the set of pixels (α i = 0 for the background pixels, α i = 1 for the foreground). Estimating the values α of the labels can be formulated as the minimization of an energy-based Markov Random Field objective function E(α), with respect to α:

E(α) = E data (α) + γE smooth (α) (1) with E data (α) = i U i (α i ) (2)
E data is the data energy term, with U i (α i ) accounting for the observation probability for a pixel to belong to the foreground or to the background, based on some image "data" (intensity, color, location...) observed on the pixel, using the statistical models built for the background and the foreground.

In order to compute the optimal solution of this energy minimization problem and determine α, a graph cuts minimization algorithm [START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF] is employed, providing us with a segmented frame I s .

Statistical models for the data energy function are Gaussian Mixture Models (GMM) based on color distributions, learned for both the foreground and background layers, which are initially determined by the user through a bounding box around the foreground on the initial image. Besides, pixels outside this bounding box are definitely assigned to the background layer (U i (α i = 0) = inf ), whereas inside their label is unknown, so that energy minimization only has effects inside the bounding box.

B. Temporal coherence and real-time issues

Once the initial image is segmented through user interaction, the following frames are treated by updating the area to effectively segment. As shown in Fig. 3, the silhouette contour of the previous segmented foreground is extracted, and the distance transform is computed over it, providing a signed distance map to these contours. According to a fixed threshold on this distance map, we define a narrow strip around the contour, in which labels of the pixels are unknown (grey area on 3d), whereas they are definitely assigned to the foreground on the inner side of the strip (U i (α i = 1) = inf , white area on 3d), and to the background otherwise (U i (α i = 0) = inf ). In this manner, temporal consistency is ensured, since energy minimization is only effective within this strip, in the vicinity of the previous segmentation boundary, avoiding some outliers outside or inside, and reducing significantly computations.

IV. DEFORMABLE OBJECT MODELING WITH THE FINITE ELEMENT METHOD

Since we deal with objects which may undergo large elastic deformations, a major issue lies in the definition of a relevant physical model. The Finite Element Method (FEM) [START_REF] Robert | Finite element modeling for stress analysis[END_REF] provides a realistic physical model, by relying on continuum mechanics, instead of finite differences for massspring systems for instance. It consists in tessellating the deformable object into a mesh made of elements, usually tetrahedrons. The deformation field u e over an element e is then approximated as a continuous interpolation of the displacements ûe of its vertices. We rely here on a volumetric linear FEM approach with tetrahedral elements.

By resorting to the infinitesimal strain theory and linear elasticity through Hooke's law, the internal elastic forces f e exerted on the four vertices of e of the mesh can be linearly related to their displacements:

f e = K e ûe (3)
with being K e the 12 × 12 stiffness matrix of the element e, depending on two elastic parameters of the material, the

Young modulus E, which measures the ratio between the tensile stress and the extensional strain and the Poisson ratio ν, which measures, under compression efforts on the object, the amount of expansion it undergoes in the two perpendicular directions [START_REF] Robert | Finite element modeling for stress analysis[END_REF].

Although it is insensitive to translation transformations, the model, by using an infinitesimal approximation of the strain tensor, giving a constant K e linearizing the elastic forces, is however inaccurate when modeling large rotations of the elements, the non-linear effects leading to non-zero summations of the forces and causing for instance unexpected growth of volume. A work-around consists in the co-rotational approach [START_REF] Etzmuß | A fast finite element solution for cloth modelling[END_REF], used for registration purposes in [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF], which is a good compromise between the ability to model large elastic deformations, cope and computational efficiency. Since the displacement of an element can be decomposed into a rigid transformation and a pure deformation, the idea is to extract the rotation matrix R e related to the rigid transformation. Then the stiffness matrix can be warped with respect to this rotation, so as to accommodate rotation transformations, giving:

f e = R e K e ûr e = R e K e (R -1 e x e -x e,0 ), (4)
with being ûr e = R -1 e x e -x e,0 , with R -1 e x e the back rotated deformed coordinates of the vertices of e, to an unrotated frame, the forces K e ûr e being then rotated to the current deformed element through the multiplication by R e . In this way, the overall forces on the whole mesh can be summed to zero, while computational efficiency is ensured since K e can be computed in advance, in contrast to non-linear FEM approaches.

V. REGISTRATION WITH POINT CLOUD DATA

Our deformable registration problem consists in fitting the point cloud data provided by an RGB-D sensor with the tetrahedral mesh. The basic idea is to derive external forces exerted by the point cloud on the mesh and to integrate these forces, along with the internal forces computed using the physical model presented in Sect. IV, into a numerical solver solving the resulting mechanical equations.

In this work, these external forces are computed based on geometrical point-to-point correspondences between the point cloud and the mesh, relaxing the assumption of having a textured object [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF] with a rough surface, for which 2D/3D keypoints can be extracted and matched. We assume that the mesh is available (manually designed here) and correctly initialized. Let us however note that off-line automatic reconstruction and meshing techniques could be considered to build the mesh and initialization could be addressed through some learning and recognition of spin images [START_REF] Johnson | Using spin images for efficient object recognition in cluttered 3d scenes[END_REF] or local 3D features. Besides, the Young modulus and Poisson ratio of the considered material are assumed to be known.

A. Segmented and sampled point cloud

As introduced in Sect. III, we use the acquired RGB image sequence to visually segment the object of interest from its background and occlusions. Since we do not rely on some distinctive visual features, the point cloud provided by the depth sensor is indeed restricted to the considered object so as to avoid ambiguities in the matching process with the background or with occluding shapes, and to be able to process correspondences from the input point cloud to the mesh. Using the segmented image I s , a segmented depth map D s is obtained by aligning and intersecting the original input depth map D with the segmented area in I s . Then by backprojecting D s in the sensor frame, the desired segmented point cloud Y = {y j } n Y j=1 , with y j a 3D point in the sensor frame, is determined. For computational concerns, we limit the size of Y by first sampling D s on a regular grid in the image plane.

B. Rigid iterative closest point

A first step in our method is to register the observed segmented point cloud Y in terms of rigid translation and rotation transformations, initially considering the mesh of the object as rigid. Let us first define X = {x j } n X j=1 the set of vertices of the mesh, in its previous computed state. We suggest a classical rigid ICP algorithm between Y and the vertices of the visible surface X V of the mesh transformed with respect to the previous RGB-D data. X V is determined by performing a visibility test on the rendered depth map of the projected 3D mesh of the object. Through this procedure, which converges rapidly, fast rigid motions can be tracked and a fair initialization for the non-rigid process can be obtained.

C. Deformable iterative closest point

In order to register the segmented point cloud with the mesh in a non-rigid manner, we suggest an ICP-like procedure.

1) Nearest neighbor correspondences: By means of K-d tree searches, nearest neighbor correspondences are determined, both from the segmented point cloud to the visible surface of the mesh and from the visible surface of the mesh to the segmented point cloud. This step provides us with the sets of nearest neighbors

N X V = {NN Y (x j ) | x j ∈ X V } and N Y = {NN X (y j )} n Y j=1 respectively in Y for X V , with the 1-NN function NN Y , and in X V for Y , with the 1-NN function NN X V .
Both sets of correspondences are processed since relying on the sole geometrical proximity may lead to inconsistent matches using single point-to-point matches.

Indeed from the segmented point cloud to the mesh, correspondences enable to track for instance expansion deformations under stretching forces, for which the observed segmented point cloud Y would spread over the visible surface of the mesh X V . The extended areas of Y with respect to X V can be matched with the outer areas of X V . These correspondences also enable to deal with occlusions and segmentation errors since the corresponding unobserved areas of the object would not affect the underlying areas of X V . Conversely, from X V to Y , correspondences are instead more suited to track shrinking deformations under compression actions, the outer areas of X V being coherently matched with the outer areas of the observed point cloud Y of the compressed object. As a drawback, unobserved areas (occlusions, segmentation errors) would affect the underlying areas X V which would match with the closest areas of Y .

As described hereafter, a trade-off has to be found between these two sets of correspondences, whether the application deals with stretching or compression actions on the object, and whether occlusions or segmentation errors are to be dealt with.

2) Computation of external forces: Based on the two sets of mesh-to-point cloud and point cloud-to-mesh correspondences, given by N X V and N Y , an external elastic force f ext exerted on each x j in X V , can be computed as follows:

f ext (x j ) = k(x j -y f j ) (5)
with

y f j =                λNN Y (x j ) + (1 -λ) 1 n K j yj ∈Kj y j if K j > 0 λNN Y (x j ) + (1 -λ)x j if K j = 0 (6)
where

K j = {y i ∈ Y |NN X V (y i ) = x j }
is the set of points in Y whose nearest neighbors are x j ; k is the stiffness of these external elastic forces. As in [START_REF] Haouchine | Single view augmentation of 3d elastic objects[END_REF], it is set to the same order of magnitude of the Young modulus, to be physically consistent. The fixed scalar λ tunes the balance between the mesh-to-point cloud and point cloud-to-mesh correspondences, as a trade-off suggested above (section V-C.1) between the stretching or compression actions to be tracked. If K j is empty, the missing point cloud-to-mesh correspondences are replaced by a self-contribution for the vertice x j , compelling it to remain at its current position. In Fig. 4, the vectors x j -y f j are displayed, from each x j . Some outliers in the point cloud may result in aberrant correspondences and thus aberrant forces exerted on some vertices. A simple solution has been to discard points in the point cloud whose point-to-point distances with their nearest neighbors in the mesh are above a certain threshold with respect to the mean value and the standard deviation of the whole set of point-to-point distances. In this case, for the considered vertices x j , we have f ext (x j ) = 0.

Finally, regarding points x j in X which are not visible, we also set f ext (x j ) = 0.

The whole set of forces is finally concatenated in a vector f ext of size n X .

3) Weighting forces using contours: A limitation of this method lies in tracking large elastic deformations due to stretching efforts for instance. In this case, since correspondences are established based on 3D geometry, only vertices lying on the outer contour of the mesh are attracted to the extended area in the point cloud. As a consequence, forces attracting the contours are weak. We propose to emphasize them by weighting the vertices of the visible surface of the mesh, given their distance to the occluding contour of the mesh in the image plane. Based on the depth map d M of the projected mesh, we compute the distance map of the occluding contour of the mesh, determined by rendering the projection of the mesh in the image plane. Then, the weight w j for the vertex x j is computed as follows:

w j ∝ e - d M j σ (7)
where d M j is the distance from x j to the nearest contour of the projected mesh, σ is a parameter which is empirically set; w j is normalized so that we get an observation probability. Finally, forces are computed this way:

f ext (x j ) = w j k(x j -y f j ) (8)
4) Numerical solver to compute the deformations: Estimating the deformations of the mesh consists in solving a dynamic system of non-linear ordinary differential equations involving the internal and external forces, based on Lagrangian dynamics:

M ü + C u + f = f ext ( 9
)
where M is the mass matrix, and C is the damping matrix, and f contains the element-wise forces f e . An Euler implicit integration scheme is used to solve the system, along with a conjugate gradient method. Using the estimated displacements û of the vertices of the mesh, X can be updated. In case of severe deformations, correspondences initially established may not be very consistent, and thus the procedure is iteratively repeated, up to a fixed number K of iterations (K = 3 in the experiments presented in section VI).

VI. EXPERIMENTAL RESULTS

In order to evaluate the performance of our method and contributions, some experimental results are shown in this section, in a quantitative manner on some computergenerated data, and in a qualitative manner on some real data. Different objects, deformations and conditions are tested.

For the non-rigid registration phase, we have employed the Simulation Open Framework Architecture (SOFA) simulator [START_REF] Faure | Sofa: A multimodel framework for interactive physical simulation[END_REF], which enables to deal with various physical models and to evolve simulations in real-time.

A. Results on synthetic data

Relying on the SOFA framework, we have generated a sequence involving the deformations of a cylindrical elastic object, modeled by the FEM co-rotational approach. It has a Young Modulus of E = 0.130M P a and Poisson ratio of ν = 0.49. The tetrahedral mesh is made of 352 vertices with a circumferential/radial/height resolution of 25 × 7 × 2, for radius/height dimensions of 0.11 × 0.02m, and is featured in Fig. 4. An impulse elastic stretching force is applied in the -Z direction (see Fig. 5), on one point on the border of the object (point 1), for which few other points are fixed on the opposite border (points 4,5,6), and two compression forces are applied along Y and -Y (points 2,3). The applied forces result in a fast elongation deformation of the object, with a maximum elongation above 50% at frame 25. For the tracking phase, segmentation aspects are not considered in these experiments. We only process the visible vertices of the rendered object in the sequence, and as a ground truth, the positions of the whole set of points are stored for evaluation. The following models and methods have been compared: (CW) (proposed method) where for the mass-spring model, the vertices consist of point masses connected together by springs, deformations being solved through Newton's second law. Results can be visually observed in Fig. 5, and in Fig. 6 the 3D errors between the vertices of the registered mesh and the corresponding points in the point cloud are plotted (see also the attached video). The benefit of our method can also be observed in Fig. 5 featuring the original target (red) and the tracking 3D mesh (blue). The suitability of the co-rotational model can be stressed out and with the contour weighting technique, our method manages better to track the extensions on the extremities, even though some errors remain.

B. Results on real data

In order to carry out experiments on real data, the point cloud of the investigated scene is acquired from a calibrated RGB-D camera Asus Xtion, 320 × 240 RGB and depth images being processed. A standard laptop with an NVIDIA GeForce 720M graphic card has been used, along with a 2.4GHz Intel Core i7 CPU. Here the segmentation process is involved in the loop, and since fast real-time performance is required, it relies on a CUDA implementation. The results presented here deal with a pizza-like elastic object, lacking of texture and showing a smooth surface, and with an elastic cylindrical bar object made with modeling clay.

For the pizza-like object, the idea has been to test motions and deformations similar to the ones involved in the pizza making process, in the scope of the RoDyMan project. On the presented sequence, the object undergoes fast rigid motions and various deformations such as isometric or elastic ones. The involved mesh has a circumferential/radial/height Qualitative results are presented in Fig. 7, comparing our method using the co-rotational FEM approach with other models. On the first row are shown input RGB images, the second row features the corresponding segmented frame, the third row shows the 3D mesh tracking the object with the mass spring model, the fourth with the standard linear FEM model, the fifth with the co-rotational model, and the last along with contour weighting. We can notice the ability of the process of the proposed method to correctly segment the visible part of the object, to track rigid motions and, in contrast to the mass spring and standard FEM models, to accurately register deformations, while being robust to occlusions due to the hands manipulating the object (third column on 7) or segmentation errors. The slight advantage of the contour weighting technique can be observed when stretching the object. With the cylindrical bar object, a sequence featuring rigid motions, along with bending deformations, is featured in Fig. 9. Here the circumferential/radial/height resolution of the mesh, depicted in Fig. 8, is 10 × 20 × 2, resulting in 420 vertices. The material is here poorly elastic, the Young Modulus being empirically set to E = 0.900M P a and the Poisson ratio to ν = 0.3. Satisfactory results are also observed regarding occlusions, tracking rigid motions and isometric deformations, in comparison with other methods (mass spring and standard FEM models).

1) Computational costs: Regarding computational aspects, in Tab. I are shown the computation times of the various phases of the algorithm, for the different methods compared in this paper. Visibility corresponds to the process of determining the visible vertices of the rendered mesh, and in the case of using the contour weighting mode, extracting the vertices lying on the contour. Ext. forces is the step involving the determination of the closest points between the mesh and the point cloud, and the computation the subsequent external forces exerted on the mesh. Resolution consists in the resolution of the Lagrangian mechanical equations, to compute the deformations. The presented figures are the averages of the execution times per frame (in milliseconds) for the sequence presented in Fig. 7. As noticed, the suggested method (Co-rotational model with the contour weighting mode) runs on the sequence at around 35 fps. We can also observe that, the computational costs for the resolution phase being relatively small within the whole process, overall execution times are relatively independent of the selected model.

VII. CONCLUSION

The recent development of physics-based modeling methods for deformable elastic objects for registration purposes and the availability of real-time implementations have lead us to choosing such an approach to track a textureless object subjected to various large deformations, with an RGB-D sensor. The use of a pertinent linear FEM model, of an efficient segmentation method, and of classical point cloud registration techniques have made our system a promising real-time tracking method able to handle various deformations and motions. Regarding future works, efforts could be concentrated on different aspects to improve, such as segmentation, which could benefit from the depth data, the point cloud matching procedure, and the physical model, by extending it to other deformations such as plastic ones. A major issue would also consist in demonstrating the suitability of the approach to mimic the art of making pizzas with a dual arm/hand robot.

Fig. 1 :

 1 Fig. 1: Artistic views of the RoDyMan robotic platform and the pizza making process.

Fig. 2 :

 2 Fig.2: Overview of our approach for deformable object tracking.

Fig. 3 :

 3 Fig. 3: Temporal consistency for segmentation. Segmentation will be effective on the strip (grey area on (d)) around the contour of the previous segmented frame (b), through the distance map to the contour (c).

Fig. 4 :

 4 Fig. 4: External forces exerted on the vertices of the mesh, with k = 1.

•

  Mass-spring model • Standard FEM model, based on 3 • Co-rotational FEM model • Co-rotational FEM model along with contour weighting

Fig. 5 :Fig. 6 :

 56 Fig.5: Results of the deformable tracking process, for the cylindrical object. On the first row is featured the ground truth, on the second the tracking with the linear FEM, the third with the linear FEM and contour weighting, the fourth with the co-rotational FEM and the fifth adding contour weighting.

Fig. 7 :

 7 Fig. 7: Results of the tracking process for the pizza-like object, with the input images (first row), the segmented frames (second row), and the registered mesh reprojected in the input image, for the mass spring model (third row), the standard FEM model (fourth row) and finally with the co-rotational model, and with the contour weighting technique (CW).

Fig. 8 :Fig. 9 :

 89 Fig. 8: Meshes used for the pizza-like object and the cylindrical bar object.

TABLE I :

 I Execution times, in milliseconds, for the different phases of the approach, and the various models and methods employed in this paper.

A. Petit, V. Lippiello and B. Siciliano are with DIETI, Università degli Studi di Napoli Federico II, Italy, {antoine.petit, vincenzo.lippiello, bruno.siciliano}@unina.it

2 http://www.rodyman.eu/ The research leading to these results has been supported by the RoDyMan project, which has received funding from the European Research Council (FP7 IDEAS) under Advanced Grant agreement

number 320992. The authors are solely responsible for its content. It does not represent the opinion of the European Community and the Community is not responsible for any use that might be made of the information contained therein.