RÉSUMÉ DE LA THÈSE

Pour traiter de l'analyse d'objets déformables dans une séquence d'images, il est nécessaire de disposer d'une base mathématique préalable permettant de modéliser les déformations. Ce modèle de déformation régit un ensemble de règles permettant de prédire les déformations subies par l'objet. Toute approche de suivi et d'analyse des déformations doit reposer sur un modèle de déformation pour expliquer le comportement observé de l'objet qu'elle entend suivre. Dans la littérature, une combinaison de modèles géométriques, visuels et physiques a été utilisée en conjonction avec des techniques de vision par ordinateur standard pour suivre les déformations avec un degré de précision variable.

Au cours des dernières décennies, l'état de l'art sur le suivi d'objets non rigides a évolué rapidement. Toutefois, malgré ces progrès, il reste de nombreux défis à relever avant que le suivi des déformations puisse être considéré comme un problème "résolu". Dans cette thèse, nous examinons ces problèmes et proposons des méthodes novatrices pour relever certains de ces défis.

Cette thèse aborde le problème du suivi temporelle de la surface d'un objet déformable en utilisant uniquement des informations visuelles capturées à l'aide d'une caméra de profondeur (RGB-D). Les méthodes proposées dans cette thèse ont été développées en vue d'applications industrielles notamment la manipulation d'objets mous par des préhenseurs et des actionneurs robotisés. Bien que la manipulation robotique utilisant le suivi des déformations reste le principal domaine d'application de cette thèse, des approches similaires ou étroitement liées peuvent également être utilisées pour la robotique chirurgicale, la capture de mouvement, la réalité augmentée et le SLAM non rigide.

Au cours des dernières années, de nombreuses approches ont été proposées pour résoudre ce problème de suivi d'objets non rigides. Dans notre cas, nous avons choisi d'utiliser des approches basées sur la simulation physique pour la modélisation des déformations. Cette approche confère une réalité physique aux méthodes de suivi des déformations proposées, mais elle nous permet également d'étendre les algorithmes de suivi des déformations à des applications robotiques.

Motivations Dans la plupart des applications robotiques industrielles, une caméra monoculaire ou de profondeur est utilisée comme capteur principal. Pour manipuler des objets complexes en temps réel, il est nécessaire de connaître la position de l'objet (six degrés de liberté) par rapport au robot. Si l'objet se déforme, il est nécessaire de suivre non seulement les 6 degrés de liberté de l'objet, mais aussi de caractériser ses déformations. Cela peut être fait avec ou sans modèle CAO de l'objet. Cependant, sans modèle, le problème devient moins contraint. De plus, sans le modèle de l'objet, il est impossible d'avoir une idée des surfaces occultées ou cachées de l'objet. Dans un contexte industriel, un modèle approximatif de l'objet qui doit être manipulé est généralement disponible.

Cela nous amène à opter pour des solutions de suivi d'objets déformables basées sur des modèles. Cependant, nous nous assurons que toutes les approches proposées dans cette thèse sont efficientes avec un modèle grossier de l'objet. Le processus de reconstruction d'un maillage fin d'un objet quelconque de forme complexe est en effet sensiblement plus complexe (et coûteux) que l'acquisition d'un modèle 3D approximatif. De plus, la précision du suivi doit être suffisamment élevée et robuste pour gérer les occultations et le bruit de mesures.

Au cours de la seconde partie de la thèse, nous arrivons progressivement à aborder un problème secondaire : l'estimation des paramètres d'élasticité d'un objet tout en suivant visuellement ses déformations. Ceci a été rendu possible en grande partie grâce à la présence du modèle de suivi des déformations basé sur la simulation physique.

Contributions

Cette thèse apporte les contributions suivantes dans le domaine du suivi des objets non rigides à partir de séquences d'images fournies par une caméra de profondeur (RGB-D) :

Une méthode permettant de suivre des objets rigides et de formes complexes à partir d'un modèle approché de l'objet d'intérêt ;

Une méthode reposant sur la simulation physique pour suivre le mouvement nonrigide d'objets déformables. Cette approche repose sur la minimisation d'une erreur de distance géométrique entre le modèle de l'objet et un nuage de points 3D fourni par une caméra de profondeur. Elle se base sur une estimation en ligne du gradient de l'erreur à partir d'une simulation itérative de la déformation ;

Une méthode similaire qui se base sur les propriétés physiques de l'objet permet-tant la minimisation de fonctions de coût géométriques et photométriques. Cette méthode permet de déterminer analytiquement le gradient de l'erreur, évitant ainsi d'effectuer les simulations itératives de la déformation qui sont nécessaires à l'estimation du gradient des fonctions de coût et qui sont coûteuses en temps de calcul ; Une approche permettant d'estimer les paramètres d'élasticité d'un objet se déformant sous l'action d'un bras robotique. Cette approche utilise le résultat du suivi de la déformation et permet également d'estimer les forces externes appliquées sur l'objet uniquement à partir des informations visuelles.

Structure de la thèse

Le manuscrit de cette thèse est organisé comme suit :

Le premier chapitre présente les concepts fondamentaux de la vision et de la géométrie par ordinateur. La première partie du chapitre rappelle les méthodes classiques dédiées au suivi d'objets rigides. La deuxième partie présente une analyse des méthodes existantes dans la littérature pour le suivi d'objets non rigides. Nous y expliquons en détail les concepts pertinents et analysons les résultats de l'état de l'art.

Dans le second chapitre, nous développons une méthode permettant de suivre avec précision des objets rigides de formes complexes. La méthode se base sur l'utilisation d'un modèle approché de l'objet d'intérêt et utilise en entrée les données fournies par une caméra de profondeur de type RGB-D. Nous décrivons la méthodologie en détail et validons l'approche sur des données simulées et réelles. A l'aide de données simulées, nous comparons notre approche à une approche similaire de l'état de l'art et présentons les résultats permettant d'analyser les performances de l'approche proposée.

Le troisième chapitre présente notre première approche permettant de suivre des objets non rigides en utilisant les données RGB-D. Nous décrivons la méthodologie, les détails de la fonction de coût et la stratégie proposée pour sa minimisation. Nous validons notre approche sur des données simulées et réelles et démontrons expérimentalement que l'approche est invariante par rapport à la précision des paramètres physiques de l'objet qui sont fournis en entrée du modèle de déformation.

Dans le quatrième chapitre, nous reformulons l'approche de suivi des déformations afin de minimiser une combinaison d'erreurs photométriques et géométriques. Cette nouvelle méthode se base sur une formulation analytique du gradient des erreurs. Nous expliquons d'abord la méthode dans un cadre générique, de manière à fournir une solution quel que soit le choix du critère visuel à minimiser. Nous présentons ensuite son utilisation pour un exemple spécifique qui consiste à minimiser un terme d'erreur combinant des informations géométriques et photométriques. Les résultats de cette approche sont ensuite comparés aux approches de l'état de l'art et également à notre première contribution présentée dans le chapitre précédent. Nous validons également expérimentalement la méthode sur des objets réels et étendons l'approche pour permettre l'estimation des forces externes appliquées sur un objet déformable à partir d'une estimation préalable de ses paramètres d'élasticité.

Le cinquième chapitre traite des applications robotiques du suivi des déformations. Nous abordons le problème de l'estimation des paramètres d'élasticité d'un objet par le développement d'une méthode qui combine le suivi des déformations et les mesures de force provenant d'un actionneur robotique. Cette méthode permet d'estimer automatiquement les paramètres d'élasticité lors d'une tâche de manipulation robotique. Dès que les paramètres d'élasticité sont disponibles, l'approche proposée dans le troisième chapitre rend alors possible l'estimation des forces externes de déformation appliquées sur l'objet et ceci en utilisant uniquement les données fournies par la caméra de profondeur.

Le sixième chapitre présente une conclusion générale du travail de thèse et met en avant des perspectives à court et moyen termes.

ACKNOWLEDGEMENT

This thesis would not have been possible without the guidance and supervision of Alexandre Krupa and Eric Marchand. This thesis is a result of their efforts, suggestions and timely encouragement. I would also remember them for being extremely nice, especially Alexandre for his discernment and wit and Eric for his occasional words of wisdom that were extremely helpful. I would also like to thank Maud Marchal for her insightful comments and help during our brief interaction.

I would like to thank the members of the jury, Stéphane Cotin, Andrea Cherubini, Gilles Simon and Luce Morin for reading and reviewing my thesis as well as for the invigorating discussion during the defense.

I have to thank François Chaumette and Paolo Robuffo Giordano for welcoming and supporting me in Lagadic and Rainbow. Fabien Spindler had been wonderfully helpful to me on multiple occasions and I thank him for his efforts. And a lot of thanks to Hélène de la Ruée for the numerous helps during the entirety of my PhD. I thank Ekrem Misimi for all the wonderful interactions over the years, as well as for hosting me in Trondheim.

A lot of thanks to the entire team of Rainbow for the wonderful memories throughout the years. I would fondly remember the co-workers in my office including Alexander, Fabrizio, Souriya and Aly, as well as Sunny, Daniel and Steven. I want to thank Pedro, Usman, Ide-flore, Firas, Hadrien, Marco & Marco, Bryan, Rahaf, Quentin, Wouter, Samuel, Javad, Ramana and Florian for all the good moments. A special thanks to Romain Lagneau for not only being an incredible collaborator but also a good friend.

I want to take this opportunity to thank my friends at the AGOS for some wonderful experiences, including Sebastien, Raphaël and everyone else. And a special thanks to Patrick and Pierre for putting up with me in Fontainbleau. I must also mention that I will fondly remember the chats at the cafét with Vijay and Sudhanya, thanks for the wonderful time.

I thank Somdotta for supporting me throughout the thesis and bearing with me through this often difficult journey of PhD. And I thank my parents for enabling me to pursue my dreams in the first place. However, it is not nearly possible to thank my family sufficiently enough with a few words of acknowledgement.

TABLE OF CONTENTS

INTRODUCTION

Salvador Dalí's illustrious career of over six decades was well known for his famous artworks that interpret common, rigid objects as flexible. However, little is said or known about an obscure and unfinished project of his own, the hundred-meter horse. In 1980, Salvador Dalí conceptualized the model of a horse with its shoulder and rump stretched apart over a distance of several meters while it appeared to be proportionally appropriate when viewed from the right perspective [Banchoff et al., 2014]. Due to reasons which remains unclear, this project of designing the 'hundred-meter horse' never really took off. By 1982, Dalí had rechristened his original idea as Horse from the Earth to the Moon. In this renewed project, the shoulder of the horse was supposed to be on a mountaintop while the rump shall be on the moon. Evidently, none of these ideas ever materialized. However, despite the obvious absurdity of the concept, it is likely that Dalí's model of the horse stretched from the earth to the moon, if constructed and viewed from the right place at the right time, would appear to be in perfect proportions.

This example is representative of the conundrum faced by all systems that involve viewing deformed objects from an arbitrary point in space. The visual perception of a deformed shape is highly dependant on the perspective of the viewer. Any attempt made towards geometrically explaining these shapes tend to become a highly unconstrained problem. If the system used for observing the deformation involves a projection operation, we loose another dimension of space in the captured data. Projection makes it even more challenging to resolve the structure of the object that is being viewed.

The fundamental principles of computer vision were originally developed with rigid objects and rigid scene as the only input. However, the actual physical world is made up of a very large amount of non-rigid objects. To be of practical utility, computer vision and vision based control systems needed to evolve techniques for dealing with nonrigid objects as well. This is especially true when robotic devices attempt to interact with soft objects.

To deal with the analysis of deforming objects in an observed visual data, it is necessary to have a mathematical prior to explain the deformation. The mathematical priors form a deformation model, which is a set of rules that the deforming object can FIGURE 1 -Different approaches and techniques for deformation tracking have been used in diverse application areas such as face tracking [START_REF] Agudo | Good vibrations : A modal analysis approach for sequential non-rigid structure from motion[END_REF] (A), manipulation of soft objects using robotic grippers [START_REF] Sun | Autonomous clothes manipulation using a hierarchical vision architecture[END_REF] (B), estimation of material properties of deformable surfaces [START_REF] Davis | Visual vibrometry : Estimating material properties from small motion in video[END_REF] (C), augmented reality on deformable objects, including surgical applications [START_REF] Haouchine | Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery[END_REF] (D), motion capture [START_REF] Helten | Full-body human motion capture from monocular depth images[END_REF]] (E) or deformation tracking of generic objects [START_REF] Salzmann | Closed-form solution to non-rigid 3d surface registration[END_REF] (F) be expected to follow. Every deformation tracking approach needs to have some deformation model for explaining the observed behaviour of the object it intends to track. In the literature, a combination of geometric, visual and physics-based models have been used in conjunction with standard computer vision techniques to track deformations with varying degree of accuracy. In the last few decades, the field of non-rigid object tracking using visual information has evolved rapidly. Fig. 1 shows a pictorial summary of some of the relevant application areas that have been addressed by the non-rigid tracking research in the recent years. Moreover, the advent of modern GPUs has enabled the tracking of deforming surfaces at frame rate. However, despite these advances, there are many challenges that need to be addressed before deformation tracking using computer vision can be considered as a problem that has been 'solved'. In this thesis, we will look at those problems and offer some novel methods to address some of those challenges. This thesis addresses the problem of tracking the temporal evolution of the surface of a deforming object using visual information only. The visual information is captured using a depth camera. The thesis has been developed with future industrial applications in mind, such that the methodologies developed here can be utilized to aid and enable the manipulation of soft objects by robotic grippers and actuators.

Deformation tracking using depth cameras has many practical utilities. Although robotic manipulation using deformation tracking remains the primary application area of this thesis, similar or closely-related approaches can also be utilized for surgical robotics [START_REF] Moerman | Digital image correlation and finite element modelling as a method to determine mechanical properties of human soft tissue in vivo[END_REF], motion capture [Hughes, 2012], augmented reality [START_REF] Huang | Real-time finite element structural analysis in augmented reality[END_REF] and non-rigid SLAM [START_REF] Agudo | Fem models to code non-rigid ekf monocular slam[END_REF].

In the recent years, many approaches have been proposed to address the problem of tracking non-rigid objects. However, due to reasons discussed in details throughout the dissertation, we chose to utilize physics based approaches for deformation modelling. Not only does this approach impart physical reality to the deformation tracking methods proposed in this thesis, this approach also enables us to extend the deformation tracking algorithms towards robotic applications. These applications could have significant industrial use-cases by themselves.

Motivation

In most industrial robotic applications, monocular or depth camera is used as the primary sensor (apart from tactile sensing). To manipulate complex objects in realtime, it is necessary to know the position of the object along all 6 degrees-of-freedom (DoF) with respect to the robot. If the object happens to be deforming and changing its shape, it is necessary to track not just the 6-DoF of the object, but to track the entire surface. This can be done with or without a CAD model of the object. However, without a model, the problem statement becomes more unconstrained. Moreover, without the object model, it is not possible to have any idea about the occluded or hidden surface of the object. In an industrial setup, an approximate CAD model of the object that needs to be manipulated is usually readily available. There are many industrial robotic assemblies which deal only with known objects that have been identified beforehand. This leads us to opt for model based tracking of deforming objects. However, we ensure that all the approaches proposed in this thesis work accurately with a coarse model of the object, since the process of reconstructing a fine mesh of a random, complex shaped object is significantly more complex (and expensive) than acquiring an approximate 3D model. Moreover, the tracking accuracy has to be high enough and robust enough to handle occlusion and sensor noise.

Over the second half of the dissertation, we progressively arrive towards addressing a secondary problem statement. This involves estimating the elasticity parameters of an object while we visually track its deformation. This has been made possible largely due to the presence of the physics based model for deformation tracking.

Contributions

This dissertation makes the following contributions to the field of non-rigid object tracking:

-A rigid object tracking methodology is developed to track complex shaped objects using a coarse object model ;

-A physics based model is utilized to track deforming objects by minimization of depth based error by estimating the gradient of error using repeated simulation of deformations ;

-A similar physics based model is used for minimization of geometric and photometric cost functions using a novel approach to analytically approximate the gradient of error, thereby avoiding the expensive simulations for gradient estimation at every iteration ;

-Having developed a non-rigid object tracking methodology, we utilize this approach to estimate the physical elasticity parameter of a deforming object using a robotic actuator and also use the estimated parameter to track the forces being applied on the object using visual information only

Structure of the thesis

The manuscript of this dissertation is organized as follows:

Chapter 1 introduces some of the fundamental concepts of computer vision and geometry. The first part of this chapter recalls some of the classical algorithms for rigid object tracking. The second half devotes itself to analyzing the existing methods for non-rigid object tracking, as available in the literature. We explain some of the relevant concepts in details and analyze some of the important results and observations from the pertinent literature. In Chapter 2, we describe a method to accurately track rigid objects of complex shapes using a coarse object model with the help of RGB-D cameras. We describe the methodology in details and validate the approach on simulated and real data. On simulated data, we compare our approach to a similar approach from state-of-the-art and present the quantitative comparison. Chapter 3 presents the first approach to track non-rigid objects using RGB-D data.

We describe the methodology, the details of the cost function and the proposed strategy for minimization. We validate our approach on simulated and real data, demonstrating that the approach is invariant to the accuracy of the physical parameters provided as an input to the deformation model. In Chapter 4, we reformulate the deformation tracking approach to minimize a combination of photometric and geometric error using an approximately analytic gradient estimation technique. We first explain the methodology in a generalized sense, such that the framework remains valid for any kind of visual error minimization. We then proceed to explain the specific example of minimization using a combined geometric and photometric error term. We compare this proposed approach to state-of-the-art methodologies, as well as the method introduced in the previous chapter. We also validate the method on real objects as well as extend the approach to track the force applied on an object, when the physical elasticity parameters of the object is known.

Chapter 5 deals with robotic applications of deformation tracking. We address the problem of estimating the elasticity parameter of an object using a combination of deformation tracking and force measurements from a robotic actuator. Once the elasticity parameter has been measured, we proceed to utilize the approach proposed in Chapter 3 for estimating deforming forces acting on the object, using visual information only.

The approach proposed in Chapter 5 was done as a joint work with Romain Lagneau.

FIGURE 5 -Tracking of contact force using the approach of Chapter 5

Contributions

We present a brief summary of the contributions that were disseminated during this Ph.D.

Publications

This thesis resulted in the following publications being accepted in international conferences :

-Sengupta, A., Krupa, A. and Marchand, E., 2019, September. RGB-D tracking of complex shapes using coarse object models.

Dataset

The following dataset were created and made publicly available for aiding future research and for enabling future comparison with our proposed approaches :

-github.com/lagadic/nr-dataset.git -github.com/lagadic/VisualDeformationTracking.git

Videos

The following videos have been made publicly available for demonstrating our results from the methods proposed throughout this thesis :

-Demonstration of rigid object tracking methodology : youtu.be/_TPJkleBu3w -Demonstration of non-rigid object tracking using depth information only : youtu.be/RFd-Ix9hcdg

BACKGROUND

Non-rigid object tracking is a complex problem to solve. It becomes even more challenging when it is required to be done in real-time. This has only been made possible by the recent developments over the last decade or so. Rigid object tracking, on the other hand, is a very well-developed research area. There are multiple alternative approaches available in the state-of-the-art for handling the problem of tracking deforming objects using visual sensors. However, a many existing non-rigid tracking methods utilize some format of rigid object tracking as an initial registration step. Given this context, we commence our study of non-rigid object tracking by first analyzing some fundamental principles involving rigid object tracking. This is followed by a survey of the existing literature of deformation tracking, as well as a brief explanation of some underlying principles.

The first half of this chapter (Sec. 1.1) introduces some of the preliminary mathematical notations required for a detailed study of rigid object tracking (Sec. 1.2), while the second-half (Sec. 1.3) dives into the state-of-the-art for deformation tracking.

Preliminary Mathematical Definitions

Since the state-of-the-art is significantly diverse in terms of approaches used for tracking deformations, we confine ourselves to describing only those preliminary notations and fundamentals which are strongly relevant to the method proposed in the thesis. Some additional approaches are discussed in section 1.3, as required.

Since the overall approach for deformation tracking proposed in this paper broadly involves visual tracking using mechanical object models, the Sec. 1.1 is splitted into two sub-parts, one which recalls the classical mathematical tools used in visual tracking of rigid objects and the other describing the details of the mechanical models.

Frame Transformation

In this thesis, the Euclidean transformation of points, lines and planes are almost always restricted to the rotation group SO(3) and the special Euclidean group of transformation SE(3). An element R ∈ SO(3) ⊂ R 3 is a matrix denoting the group of 3D rotation. An element of SE(3) is usually denoted by the homogeneous matrix:

T =   R t 0 1×3 1   ∈ SE(3) R ∈ SO(3), t ∈ R 3 (1.1)
where t ∈ R 3 . The tangent space of SO(3) and SE(3), denoted by so(3) and se(3) respectively, is obtained using the logarithmic map:

ω = θ 2 sin θ      r 32 -r 23 r 13 -r 31 r 21 -r 12      =      ω 1 ω 2 ω 3      (1.2) where R =      r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33      (1.3) and θ = cos -1 tr(R) -1 2 (1.4)
such that ω = (ω 1 , ω 2 , ω 3 ) ∈ so(3) and the corresponding skew-symmetric matrix is given by

[ω] × =      0 -ω 3 ω 2 ω 3 0 -ω 1 -ω 2 ω 1 0     
(1.5)

An element ξ = ν ω ∈ se(3) can be derived by:

ν = I 3×3 + 1 -cos θ θ 2 [ω] × + θ -sin θ θ 3 [ω] 2 × -1 t (1.6)
The setup for visual tracking (in the context of this thesis) would always contain at-least three cartesian reference frames, one centered at the camera (or the visual sensor) F c , another centered at the object being tracked F o and the last one depic-ting the world reference frame F w . The coordinate frames belong to the 3D Euclidean space E 3 . An arbitrary 3D point P in real coordinate space R 3 is represented in homogeneous coordinates as a 4-vector P = (X, Y, Z, 1), while the point itself is represented as

P = (X, Y, Z) ∈ R 3 w.r.t F c .
The same point, when expressed w.r.t the world or object coordinate, gets represented as w P and o P respectively. Using standard convention, a transformation from an arbitrary reference frame A to another arbitrary reference frame B is denoted by B T A . With this representation, a 3D point can be transformed from the frame A to B using:

B P = B T A A P (1.7)
If T is represented as q = (t, θu), where u is the axis of the rotation R and θ its relative angle, the time derivative q can be used to represent the same expression as a velocity twist ξ = q, provided θ and t are very small.

Perspective Projection Model

When a 3D point P is imaged using the pinhole camera model of the general projective camera [START_REF] Hartley | Multiple view geometry in computer vision[END_REF], a 2D image point p = (u, v), with the corresponding homogeneous coordinate p = (u, v, 1), is formed on the image plane I, where

u = f x X Z + c x v = f y Y Z + c y (1.8) (f x , f y )
are the focal lengths of the camera, expressed in pixel units, whereas (c x , c y ) denotes the 2D coordinates of the principal point on the image plane.

For a point p, the projective transformation K p can be expressed as the mapping:

        X Y Z 1         →      f x X + c x Z f y Y + c y Z Z      = p = K p Π p P (1.9) P 2 = (0, Y, Z) P 1 = (0, 0, Z) p 2 = (u 2 , v 2 ) p 1 = (u 1 , v 1 ) |v 1 -v 2 | Y f F c FIGURE 1
.1 -Pinhole camera projection model. We represent image formation for two 3D points P 1 and P 2 that are co-planar and lies parallel to the image plane

⇒ p =      f x 0 c x 0 f y c y 0 0 1      Kp      1 0 0 0 0 1 0 0 0 0 1 0      Πp         X Y Z 1         (1.10)
The matrix K p is called the camera calibration matrix and denotes the intrinsic parameters for a CCD or CMOS type sensor. Note that generalized projection could be modelled with an additional skew parameter s p , such that:

K p =      f x s p c x 0 f y c y 0 0 1      (1.11)
s p is an axial skew parameter which denotes the shear distortion in the projected image. However, it can be assumed that s p = 0 for most normal cameras. Moreover, to generalize the projection even further, if p is not maintained at the camera-centered reference frame, an additional transformation can be utilized to orient the point o P from its objectcentered reference frame to that of the camera, such that:

p = K p Π p c T o camera projection matrix o P (1.12)
The R 3 → R 2 operation corresponding to the perspective projection of 3D points to 2D pixel on the image plane can be simplified by the projection operator Π • such that p = Π P can be expressed using the non-homogeneous coordinates of the 3D and 2D points.

I(u, v)

gives the grayscale intensity of the pixel at coordinates (u, v) of I. The pinhole camera model used for this projection operation is shown in Fig. 1.1. This perspective projection ideally allows us to represent light rays by straight lines, but with most cameras in real-life, certain radial distortions are observed in the pixel coordinates. These radial distortions can be modelled easily [Faugeras, 1993] [Hartley andZisserman, 2003], and is represented as:

u = u d (1 + k 1 s 2 + k 2 s 4 ) v = v d (1 + k 1 s 2 + k 2 s 4 ) (1.13)
where (u, v) is a point in the image plane which can be obtained using perspective projection only while (u d , v d ) is the corresponding point with distortion. Here,

s 2 = u 2 d +v 2 d
and k 1 and k 2 are the parameters of distortion coefficient, which can be estimated using classical calibration techniques, such as [Brown, 1971] [Stein, 1997].

Object Model and Pointcloud

The 3D model depicting the surface of an object is expressed using the tuple M = A V, X V , where:

A V = P 1 P 2 • • • P N =         P 1 X P 2 X • • • P N X P 1 Y P 2 Y • • • P N Y P 1 Z P 2 Z • • • P N Z 1 1 • • • 1        
(1.14)

and A pointcloud comprising of multiple 3D points is denoted by ψ, which has the same matrix structure as A V, i.e., ψ = P 1 P 2 • • • P K and A ψ = P 1 P 2 • • • P K , where K denotes the number of points in the pointcloud.Note that V and ψ denotes similar matrices that represent a set of 3D points. However, throughout this thesis, V has been used to denote the vertices of a mesh whereas ψ has been used to denote pointclouds. X V denotes the adjacency matrix of the simple, undirected mesh such that:

A V = P 1 P 2 • • • P N is a set of 3D
X V ij =      1 if (P i , P j ) is connected 0 else (1.15)
thereby depicting a model with N 3D points and M number of Qhedral surfaces (denoting a polygon with Q edges), as shown in Fig. 1.2b.

B V = B T A A V (1.16)
while X V remains independent of the inertial reference frame.

Sensor Setup

While discussing the representation of pointcloud, it is worthwhile to briefly discuss the depth sensor setup that has been used in this thesis to obtain the pointcloud data. The non-rigid tracking approaches proposed in this thesis have been developed using RGB-D sensors only. The capability to extract depth information along with color (RGB) information from the scene can be availed by using multiple sensor setups, such as a stereo rig, time-of-flight (ToF) sensors or structured-light based depth sensor. In this thesis, two different depth cameras are used for the experiments conducted in this thesis: Intel RealSense SR300 and Intel RealSense D435, as shown in Fig. 1.3. SR300 is a coded-light based depth camera [START_REF] Zabatani | Intel® realsense™ sr300 coded light depth camera[END_REF], wherein a special temporal optical code is projected by an infrared (IR) projector and the triangulation between projected patterns and images, captured by a dedicated IR sensor, is used to generate a dense depth map. The D435 uses an Active Stereo Depth system [Grunnet-Jepsen et al., ], by projecting a pattern to assist the stereoscopic depth estimation using the right and left IR sensor. The color camera of SR300 is rolling shutter, while D435 is equipped with a global shutter color camera.

Rigid Object Tracking

While tracking rigid objects using visual information, there are multiple approaches that can be utilized for frame-to-frame registration. We first describe the classical approaches for rigid object tracking and then take a look at some of the recent methods from the state-of-the-art for rigid object and scene tracking.

Classical Approaches

Since we utilize a depth camera, we can choose to track objects in terms of 3D-3D registration only, we can restrict ourselves to just the image data and track objects in a 2D-2D sense, we can switch back and forth between 2D-3D registration or we can use the hybrid data consisting of depth and image information together in RGB-D format. We shall present the basic principles related to the different approaches for registration of rigid objects:

3D-3D Registration

The first approach for rigid object tracking involves the registration of a pair of 3D data obtained from some suitable depth sensor. Given a stream of 3D points from the depth sensor, one of the simplest approach to align the points is using an Iterative Closest Point (ICP) algorithm which was first proposed as a method for registration of 3D shapes in [START_REF] Besl | Method for registration of 3-d shapes[END_REF]. It was later summarized by [START_REF] Pomerleau | A review of point cloud registration algorithms for mobile robotics[END_REF], outlining more than two decades of refinement since it was first proposed.

In its basic form, ICP has two variants, one involving the registration of 3D points with another set of 3D points (point-to-point ICP), while the other variant involves the minimization of distance between a set of 3D points with planar surfaces (point-to-plane ICP). The point-to-point ICP can be expressed as the minimization of the following cost function w.r.t the transformation between two pointclouds:

E(ψ P , ψ S ) = (p,s)∈N ψ p -s 2
(1.17)

where ψ P and ψ S are two pointclouds located close to each other, and the set N ψ denotes the association between the pointclouds, defined as:

N ψ = match(ψ P , ψ S ) = (p n , s m ) : ∀p n ∈ ψ P , s m = arg min j d(p n , s j ) (1.18)
where d(p, s) = p-s 2 . This matching operation is repeated iteratively. For this form of problem, there exists a closed form solution for determining the transformation between ψ P and ψ S that minimizes the error in Eqn. 1.17. Solutions have been proposed based on singular value decomposition (SVD) [START_REF] Arun | Least-squares fitting of two 3-d point sets[END_REF], quaternions [Horn, 1987], orthonormal matrices [START_REF] Horn | Closedform solution of absolute orientation using orthonormal matrices[END_REF] and dual quaternions [START_REF] Walker | Estimating 3-d location parameters using dual number quaternions[END_REF] (however, it must be noted that [Horn, 1987] and [START_REF] Walker | Estimating 3-d location parameters using dual number quaternions[END_REF] does not utilize exactly the same error function as given in Eqn. 1.17, interested readers may refer to [START_REF] Eggert | Estimating 3-d rigid body transformations : a comparison of four major algorithms[END_REF]). Despite the existence of closed form solution, it must the noted that Eqn. 1.17 is differentiable and therefore it can be fitted into a steepest gradient descent or Gauss-Newton optimization scheme to arrive at the optimal transformation [Solomon, 2015].

The point-to-plane ICP is the registration between a 3D pointcloud and a surface model, and it is expressed as the minimization:

E(ψ P , M S ) = K k=1 (p k -s l ) • n l 2 (1.19)
where n l is the surface normal of the model around the point s l ∈ V S and the model itself is represented as M S = (V S , X V S ) and has l vertices. K represents the total number of points in the pointcloud ψ P and the association between p k and s l is done using an external algorithm for point-to-plane correspondence. Interested readers may refer to [START_REF] Ramalingam | A theory of minimal 3d point to 3d plane registration and its generalization[END_REF] for an overview of correspondence methods. [START_REF] Pomerleau | A review of point cloud registration algorithms for mobile robotics[END_REF] provides a recap of a classical closed form solution to Eqn. 1.19, given in the form of:

GG A ξ p x = Gh b (1.20)
where ξ p = ω ν , the transformation aligning ψ P with M S , can be obtained by solving Eqn. 1.20 for x = A -1 b using Cholesky decomposition [Cholesky, 2005] of the matrix denoted by A. Here:

G =   • • • p k × n k • • • n k   [6×K]
(1.21)

and:

h =       . . . (s k -p k ) • n k . . .       [K×1]
(1.22) However, it is also possible to reach at a similar solution with non-linear least squares optimization, where the Jacobian relating the variation of ξ p with the change of E(ψ P , M S ) is given by:

J icp = ∂E(ψ P , M S ) ∂ξ p = p k × n k n k (1.23)
while the update ∆ξ p can be computed in the Gauss-Newton sense:

∆ξ p = -(J icp J icp ) -1 J icp E(ψ P , M S ) (1.24)
The update is combined with the initial estimate by (ξ p ) t = (ξ p ) t-1 + ∆ξ p where (ξ p ) t is the estimate of ξ p at t-th iteration of the minimization. This iterative gradient estimation and update of ξ p is continued till a certain convergence criterion is reached.

The exact convergence criterion is often determined empirically and depends on the application area. Interested readers are directed to [Salzo and Villa, 2012] for a detailed analysis of convergence properties and criteria for Gauss-Newton method.

Dense 2D-2D Registration

Having summarized some of the traditional methods for 3D-3D registration, we now take a look into some of the 2D-2D registration techniques based on just the image data. This is a more classical problem statement, since monocular cameras were popularized much earlier than affordable depth sensors. Tracking with images can be With classical image based tracking using the Lucas-Kanade algorithm [START_REF] Baker | Lucas-kanade 20 years on : A unifying framework[END_REF], p is usually the image pixel coordinates, where I * (•) is the target/template image and I is the current image, W is an image warping function based on the warp parameters x = (x 1 , x 2 , • • •, x n ), a vector of parameters defining the warp.

Minimizing the expression in Eq. 1.25 is a non-linear optimization task, even if W is linear in x. To optimize Eq. 1.25, the current estimate of x is assumed to be known. The increment ∆x is iteratively solved by minimizing

E(p) = p I W(p, x + ∆x) -I * p 2 (1.26)
and the parameters are updated by

x ← x + ∆x (1.27)
The Lucas-Kanade algorithm is specifically a Gauss-Newton based gradient descent algorithm. However, steepest gradient descent, Newton-Raphson [Acton, 1990] or Levenberg-Marquadt [Marquardt, 1963] [Solomon, 2015] based minimization can be equally applicable for a minimization of this nature. The expression for intensity difference can be linearized using the Taylor expansion:

I(W(p, x + ∆x)) -I * (p) = I(W(p, x)) + ∇I ∂W ∂x ∆x -I * (p) (1.28)
where ∇I = ∂I ∂u ∂I ∂v , the image gradients along x and y axis of the image plane. The term ∂W ∂x is the Jacobian of the warping function and is given by:

∂W ∂x =   ∂Wx ∂x 1 ∂Wx ∂x 2 • • • ∂Wx ∂xn ∂Wy ∂x 1 ∂Wy ∂x 2 • • • ∂Wy ∂xn   (1.29)
The formulation of eqn. 1.26 is termed the forward-additive formulation for direct image intensity based visual tracking. Forward-compositional formulation [Shum and Sze-liski, 2001], inverse-additive formulation [START_REF] Hager | Efficient region tracking with parametric models of geometry and illumination[END_REF]] and inversecompositional formulation [START_REF] Baker | Equivalence and efficiency of image alignment algorithms[END_REF] are used as well. The comparative outline of these methods are summarized below:

1. Forward-additive method The error function is given as by Eqn. 1.26, while the update is given by Eqn. 1.2.1.2.

Forward-compositional method

The error function is given by:

E(p) = p I W W(p, ∆x) , x -I * p 2 (1.30)
and the update is given by:

W(p, x) ← W(p, x) • W(p, ∆x) (1.31)
where the • operator denotes the composition of two warps, such that:

W(p, x) • W(p, ∆x) ≡ W W(p, ∆x), x (1.32)
3. Inverse-compositional method The error function is given by:

E(p) = p I * W(p, ∆x) -I W(p, x) 2 (1.33)
while the update is given by:

W(p, x) ← W(p, x) • W(p, ∆x) -1 (1.34)
4. Inverse-additive method The error function is the same as Eqn. 1.26, while the role of the template and the image is switched. This reformulates the Taylor expansion of Eqn. 1.28 into:

E(p) = p I W(p, x) + ∇I * ∂W ∂p -1 ∂W ∂x ∆x -I * (p) 2 (1.35)
and the parameter update is given by x ← x -∆x

The Lucas-Kanade algorithm described above enables us to track an image based on a template. We now focus on deriving a relationship between a small number of image points in two different images when a one-to-one correspondence between the image positions are given. Given a set of four 2D to 2D point correspondences a p i ↔ b p i , where a p i = ( b u i , a v i , 1) and b p i = ( b u i , b v i , 1) are two points on the image plane in homogeneous coordinates, it is possible to derive a 3×3 matrix H such that b p i = H a p i . To enforce a scale constancy, an additional constraint can be imposed in the form of b p i × (H a p i ) = 0. This leads to a set of three equations of the form:

     0 -a p i a p i a p i 0 -a p i -a p i a p i 0      3×9      h 1 h 2 h 3      9×1 = 0 (1.36)
where h n gives the n-th row of H. With four point correspondences, a non-zero solution of H can be determined up to a non-zero scale factor [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. This forms the simplified 2D representation of the Direct Linear Transform (DLT) algorithm, and can be utilized in a 2D-2D object tracking framework. However, it can be also utilized in tandem with 3D data resulting in a more stable variant of tracking, as explained in the next paragraph.

2D-3D Registration

For a calibrated camera, i.e., when K p from Eqn. 1.12 is estimated using an external calibration method, the transformation c T o , that describes the relative pose between the object frame and the camera frame, can be determined using a small number of 2D-3D point correspondences and this method is more stable and reliable for object tracking applications, as compared to classical DLT based techniques. [START_REF] Fischler | Random sample consensus : a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] coined the term Perspective-n-Point (PnP) problem, where n stands for the number of correspondences, to describe this class of algorithms and a vast amount of research has been done to come up with various solutions to this PnP problem.

The PnP problem can be solved either non-iteratively [START_REF] Fischler | Random sample consensus : a paradigm for model fitting with applications to image analysis and automated cartography[END_REF], Dhome et al., 1989, Gao et al., 2003, Kneip et al., 2011] or through iterative techniques [Lowe, 1991,Dementhon and Davis, 1995,Lu et al., 2000]. Interested readers may refer to [START_REF] Marchand | Pose estimation for augmented reality : a hands-on survey[END_REF] for a detailed survey of 3D pose estimation techniques.

State-of-the-art for Rigid Object Tracking

The rigid object tracking literature is extensive and heterogeneous. An exhaustive survey of all such techniques is beyond the scope of this thesis. Therefore, we restrict the discussion to a few of the more important and relevant object tracking methodologies that were proposed in the last three decades. A brief summary of some of the notable research in the field of rigid object tracking is given below.

As mentioned in Sec. 1.2, [START_REF] Besl | Method for registration of 3-d shapes[END_REF] is one of the earliest paper to explain the ICP in details. The approach involves calculating the closest distance from a point to either a parametric entity: viz., curves or surfaces, or implicit geometric entity: like a vector-valued, multivariate function. A Newtonian minimization approach is used for the former, while an augmented Lagrange multiplier system [START_REF] Luenberger | Linear and nonlinear programming[END_REF] is used for the later. The closest point calculation and registration is done iteratively till the mean square error falls below a preset threshold. This forms the basic ICP algorithm described in [Wikipedia, 2016]. Alternate methods of point cloud registration has been proposed by [START_REF] Pottmann | Registration without icp[END_REF] for point cloud registration without using the original ICP proposed by [START_REF] Besl | Method for registration of 3-d shapes[END_REF]. [START_REF] Pottmann | Registration without icp[END_REF] proposes a kinematic model-based approach to 3D point cloud registration. [START_REF] Newcombe | Kinectfusion : Real-time dense surface mapping and tracking[END_REF] and [START_REF] Izadi | Kinectfusion : real-time 3d reconstruction and interaction using a moving depth camera[END_REF] provided a major advance in realtime tracking and mapping. It allowed users to create very accurate map of the scene while tracking the camera reliably, leading to implementation in applications like 3D modelling, high quality, tracking of robots mounted with these depth cameras, etc. The technique used for KinectFusion is broadly divided into four steps: A) measurement, B) pose estimation, C) Reconstruction and D) Surface Prediction. The measurement of the raw data coming in from the sensor is stored as a dense vertex map and a pyramid of normal map. The reconstruction of the scene, at every step, is stored as a Truncated Signed Distance Function (TSDF). It eases the integration of new measurement into the reconstructed model. If the transformation of the camera (or Kinect) with respect to the world is known, the appearance of the depth map at the next iteration can be predicted by transforming the reconstruction using the camera's rotation and translation, and then projecting it into the image plane using ray-casting techniques. Pose estimation, on the other hand, is done using ICP.

Many subsequent papers have tried to improve upon the accuracy of KinectFusion. [START_REF] Henry | Patch volumes : Segmentation-based consistent mapping with rgb-d cameras[END_REF] is one of the primary research work dealing with improvement of tracking accuracy of KinectFusion by implementing global optimization on the pose frame graph. This was done using g2o [START_REF] Kümmerle | g2o : A general framework for graph optimization[END_REF]. [START_REF] Roth | Moving volume kinectfusion[END_REF], [START_REF] Nießner | Real-time 3d reconstruction at scale using voxel hashing[END_REF], [START_REF] Zeng | A memoryefficient kinectfusion using octree[END_REF] and [START_REF] Whelan | Deformation-based loop closure for large scale dense rgb-d slam[END_REF] improves the mapping capabilities of KinectFusion. [START_REF] Choi | RGB-d object tracking : A particle filter approach on GPU[END_REF] proposed a particle filtering based approach to track the degrees of freedom of an object from RGB-D data, wherein the likelihood of each particle was evaluated using a combination of geometric and photometric information.

Extending this approach, [START_REF] Ren | Real-time tracking of single and multiple objects from depth-colour imagery using 3d signed distance functions[END_REF] proposes to express the probability of each pixel as a joint distribution over a combined function of the camera pose, Signed Distance Function (SDF) [START_REF] Chan | Level set based shape prior segmentation[END_REF] of the object, RGB value of the pixel and a step function which determines if the object belongs to background or foreground. The pose is determined by a Levenberg -Marquardt minimization of the Negative Log-Likelihood (NLL) of this distribution. [START_REF] Slavcheva | Sdf-2sdf : Highly accurate 3d object reconstruction[END_REF] proposes a real-time SDF to SDF registration method using a combination of geometric and surface normal based constraints to generate a colored pointcloud which is globally registered.

Among the recent approaches, [START_REF] Kehl | Deep learning of local rgb-d patches for 3d object detection and 6d pose estimation[END_REF] uses a convolutional auto-encoder for matching scene-patch descriptors with synthetic model patches to detect and localize all 6 DoF of an object. [START_REF] Garon | Deep 6-dof tracking[END_REF] proposes a CNN based 6DoF object tracker using Microsoft Kinect. In 2017, [START_REF] Xiang | Posecnn : A convolutional neural network for 6d object pose estimation in cluttered scenes[END_REF] proposed PoseCNN, a CNN based semantic labelling and 6DoF object tracking framework. [START_REF] Wu | Real-time object pose estimation with pose interpreter networks[END_REF] proposes a realtime technique for 6DoF object localization using CNN based on RGB images only.

Non-rigid Object Tracking

We shall now present a brief overview of the existing state-of-the-art on non-rigid object tracking, as well as the summary of some of the relevant and notable methodologies used in the literature. However, we begin by defining the objective of this thesis.

Problem Statement

Given a 3D mesh model M (which can be a triangular or tetrahedral mesh) and a 3D point cloud ψ, we define a cost function d(M , ψ) that provides a similarity mea-FIGURE 1.4 -Categorization of some of the selected literature about deformation tracking using visual and/or depth information. The highlighted region denotes the area of interest of this thesis sure between the pointcloud and the model. We also define a deformational function F which acts on the model M to produce a deformed model F (M ). F can be parameterized using a deformation or force field. The objective of the non-rigid tracking method is to minimize the cost function:

E(ψ) = arg min F d F (M ), ψ (1.37)
With this objective, tracking of deforming objects can be handled using multiple approaches, based on the application area and the practical constraints involved with the targeted application. The available literature on this subject can be broadly classified into two categories: a) template/model based tracking, and b) model-free reconstruction and tracking of deforming objects. Even though this thesis primarily deals with model based tracking, we also present the details of the research from the relevant reconstruction and tracking literature as well.

For the sake of completeness, we discuss the state-of-the-art for both these paradigms. An overview of some of the notable approaches that we plan to discuss are given in Fig. 1.4. The explanation and the relevant discussion about these research articles follow in the subsequent paragraphs. In this thesis, we use a physically based deformation model, the co-rotational FEM, for modelling the deformation of objects. The following paragraphs are an attempt to explain the rationale behind this choice and to highlight the advantage of using physically based models for deformation tracking, over other alternative approaches.

Physically-based Models

Mechanical models for modelling bio-mechanical deformation of organic objects can be done using various advanced mechanical models, such as Finite Element Method (FEM), Smoothed-Particle Hydrodynamics (SPH) or porous media model [START_REF] Wei | Toward real-time simulation of blood-coil interaction during aneurysm embolization[END_REF] based fluid flow simulation. However, in this thesis, we confine our study of physics based deformable model to FEM.

FEM aims to discretize a dynamic system involving elastic materials (in the context of mechanical engineering), such that the temporal evolution of the state of the system when subjected to external, unbalanced forces can be simulated across time steps. To do so, multiple formats of mechanical models can be used, such as triangular, tetrahedral or hexahedral meshes. This mechanical model alone is sufficient to evaluate the evolution of the FEM. However, since we are interested in visual tracking, it is important to map this mechanical model with a visual model of the surface of the object being tracked.

The mapping between the vertices of the mechanical mesh V m and the visual model V s , as shown in Fig. 1.5, can be expressed by V s = J (V m ) (the notation for the mesh vertices is defined in Sec. 1.1.3) while the velocities are related by U s = J sm U m , where U = V, the differentiation w.r.t time, and the Jacobian:

J sm = ∂V s ∂V m (1.38)
relates the displacement of the vertices of V m to V s . Here, J represents the barycentric mapping.

V m V s V m → V s FIGURE 1
.5 -The mapping between the visual and the mechanical mesh One of the purpose of using FEM is to convert the harmonic analysis of free or damped vibrations of physical objects in continuous domain, which is usually expressed as Partial Differentials Equations (PDEs), into a discrete formulation such that the same system can be expressed by Ordinary Differential Equations (ODEs), which enables it to be solved efficiently by modern computers. A simple example of FEM for a 1-dimensional bar is summarized below. Let us assume an uniform, homogeneous bar of length L, as shown in Fig. 1.6. The one dimensional wave equation of this bar [START_REF] Wood | Geometrically nonlinear finite element analysis of beams, frames, arches and axisymmetric shells[END_REF], under an axial load, is given by

∂ 2 us ∂t 2 = V 2 b ∂ 2 us ∂x 2 such that V b = E ρ and u s is the displacement in x direction at time t.
E and ρ are the Young's modulus and density of the bar respectively, while V b gives the wave propagation velocity. For FEM based analysis, it is necessary to represent the force-displacement relationship in a matrix representation of the format:

F = Ku s (1.39)
where K is the stiffness matrix, u s is the vector of displacements and F is the external force vector. It can be shown [Rao, 1995] that for the i-th node of the bar:

K i = nC k   1 -1 -1 1   (1.40)
and:

F = l 0 F b (x, t)φdx (1.41)
Here, A is the cross-sectional area of the bar, C k = EA, l is the length of the current element, F b (x, t) is a time-varying distributed load and φ is the shape function [Logan, 2011]. Representing the stiffness matrix for individual elements as:

K i =   K 11 i K 12 i K 21 i K 22 i   (1.42)
we can assemble the global stiffness matrix as:

K n×n =                  K 11 1 K 12 1 K 21 1 K 22 1 + K 11 2 K 12 2 K 21 2 K 22 2 + K 11 3 • • • K 22 n                  (1.43)
Since the left end of the bar in Fig. 1.6 is fixed, the row and column corresponding to node 0 will be set to zero, thereby enforcing the boundary condition. If the forces are known, the nodal displacement vector can be obtained as: This scheme for relating force with displacement can be generalized to arbitrarily oriented beam elements and 3D triangular, tetrahedral or hexahedral elements. Only tetrahedral elements are used to model the mechanics for the FEM used in this thesis.

u s = K -1 F (1.44)
Tetrahedral meshes have the advantage of producing low dispersion error [START_REF] Wu | The advantages of triangular and tetrahedral edge elements for electromagnetic modeling with the finite-element method[END_REF] when used in the context of FEM. This encourages us to use tetrahedral mesh for the mechanical model (Fig. 1.7), while using triangular mesh for the surface model.

Frame 2: Example of FEM in a 3D Tetrahedral Element An example of FEM formulation for a single tetrahedral element and its assembly into a global mechanical mesh is discussed below. A typical tetrahedral element of FEM comprises of four vertices p 1 , p 2 , p 3 and p 4 as shown in Fig. 1.8. Each point is represented by its 3D coordinates, e.g:

p 1 = x 1 y 1 z 1 etc. Let force vectors F 1 , F 2 ,
F 3 and F 4 be applied at the four vertices of this tetrahedron, causing a deformation of u 1 , u 2 , u 3 and u 4 respectively. The relation between the forces applied and the displacement caused is governed by the equation of motion derived from Newton's second law, given by:

M [12×12] ü + D [12×12] u + K [12×12] u [12×1] = F [12×1] = F 1 , F 2 , F 3 , F 4 (1.45)
Here, M is the mass matrix, D is the damping matrix and K is the stiffness matrix. M can be approximated by a diagonal matrix from the total mass of the system under the assumption of uniform mass distribution. There are multiple alternatives for constructing D, the simplest being the Rayleigh damping formulation given by:

D = α e M + β e K (1.46)
where α e and β e are proportionality constants (usually determined by modal analysis of the structure). The stiffness matrix is given by:

K = V K scalar B K [12×6] D [6×6] B K [6×12]
(1.47)

Where V K is the volume of the tetrahedral element, B K is the strain matrix and D is the material matrix. For the rest of this thesis, we assume that all objects used are isotropic. For isotropic materials, D can be expressed as:

D =   E(1 -ν) (1 + ν)(1 -2ν)                 1 ν 1-ν ν 1-ν 0 0 0 ν 1-ν 1 ν 1-ν 0 0 0 ν 1-ν ν 1-ν 1 0 0 0 0 0 0 1-2ν 2(1-ν) 0 0 0 0 0 0 1-2ν 2(1-ν) 0 0 0 0 0 0 1-2ν 2(1-ν)               (1.48)
where E and ν are the Young's modulus and Poisson ratio of the material respectively. Equivalent matrices can be derived for orthotropic [START_REF] Li | Stable orthotropic materials[END_REF] and anisotropic [START_REF] Zhao | Standardized compliance matrices for general anisotropic materials and a simple measure of anisotropy degree based on shear-extension coupling coefficient[END_REF] materials, if needed. The strain matrix can be expressed as:

B K = 1 6V K G (u) (1.49)
where G (u) can be expanded as:

G (u) =               a 1 0 0 a 2 0 0 a 3 0 0 a 4 0 0 0 b 1 0 0 b 2 0 0 b 3 0 0 b 4 0 0 0 c 1 0 0 c 2 0 0 c 3 0 0 c b 1 a 1 0 b 2 a 2 0 b 3 a 3 0 b 4 a 4 0 0 c 1 b 1 0 c 2 b 2 0 c 3 b 3 0 c 4 b c 1 0 a 1 c 2 0 a 2 c 3 0 a 3 c 4 0 a              
(1.50)

where:

a 1 = -(y 3 z 4 -z 3 y 4 ) + (y 2 z 4 -y 4 z 2 ) -(y 2 z 3 -y 3 z 2 ) b 1 = (x 3 z 4 -x 4 z 3 ) -(x 2 z 4 -x 4 z 2 ) + (x 2 z 3 -x 3 z 2 ) c 1 = -(x 3 y 4 -y 3 x 4 ) + (x 2 y 4 -x 4 y 2 ) -(x 2 y 3 -x 3 y 2 ) a 2 = (y 3 z 4 -y 4 z 3 ) -(y 1 z 4 -y 4 z 1 ) + (y 1 z 3 -y 3 z 1 ) b 2 = -(x 3 z 4 -x 4 z 3 ) + (x 1 z 4 -x 4 z 1 ) -(x 1 z 3 -x 3 z 1 ) c 2 = (x 3 y 4 -x 4 y 3 ) -(x 1 y 4 -x 4 y 1 ) + (x 1 y 3 -x 3 y 1 ) a 3 = -(y 2 z 4 -y 4 z 2 ) + (y 1 z 4 -y 4 z 1 ) -(y 1 z 2 -y 2 z 1 ) b 3 = (x 2 z 4 -x 4 z 2 ) -(x 1 z 4 -x 4 z 1 ) + (x 1 z 2 -x 2 z 1 ) c 3 = -(x 2 y 4 -x 4 y 2 ) + (x 1 y 4 -x 4 y 1 ) -(x 1 y 2 -x 2 y 1 ) a 4 = (y 2 z 3 -y 3 z 2 ) -(y 1 z 3 -y 3 z 1 ) + (y 1 z 2 -y 2 z 1 ) b 4 = -(x 2 z 3 -x 3 z 2 ) + (x 1 z 3 -x 3 z 1 ) -(x 1 z 2 -x 2 z 1 ) c 4 = (x 2 y 3 -x 3 y 2 ) -(x 1 y 3 -x 3 y 1 ) + (x 1 y 2 -x 2 y 1 ) (1.51)
The volume of the tetrahedron is available using elementary geometry, given by the expression

V K = 1 6 p 1 -p 4 • (p 2 -p 4 ) × (p 3 -p 4 ) .
Having summarized the FEM formulation for a single tetrahedral element, we now describe how to assemble these element-wise matrices into a global equation for the entire object.

Global Assembly

We denote the global assembled matrices with the suffix (•) g , thereby providing us with global stiffness matrix K g , global force vector F g and so on. Let us consider the case of an object that has been modelled with N tetrahedral elements. Let F d be the force vectors of the N elements stacked column-wise, i.e., F d

[12N×1] = F 1 [12×1] , F 2 , . . . , F N .
It is possible to construct an assembly matrix A such that:

F d [12N×1] = A [12N×N V ] F g [N V ×1] ⇒ F g [N V ×1] = A [N V ×12N] F d [12N×1] (1.52)
where N V is the number of vertices in the global mechanical model of the object. It is possible to use the assembly matrix to construct the global stiffness matrix, given by:

K g [N V ×N V ] = A [N V ×12N] K d [12N×12N] A [12N×N V ]
(1.53) where:

K d =          K 1 0 0 . . . 0 K 2 . . . 0 . . . . . . . . . K N          [12N×12N]
(1.54)

Similarly, u g can be constructed using the approach of Eqn. 1.52 and M g . D g can be constructed using the approach of Eqn. 1.53.

Co-rotational FEM

We utilize the co-rotational formulation [START_REF] Müller | Interactive virtual materials[END_REF] for modelling deformation using FEM since the co-rotational model is more robust and computationally efficient. The main idea of the co-rotational method is to introduce a local coordinate system that continuously rotates and translates with the system. From the literature [Rankin, 1988, Felippa, 2000], some of the advantages of the co-rotational method can be summarized as:

-Co-rotational FEM is better suited to handle problems with large rotational motion but with small strains -It decouples the non-linearity and possible anisotropic behavior of the material from geometric non-linearities of motion We now present an introductory summary of co-rotational formulation for FEM. Let us assume that the model undergoes a rigid rotation as well as a deformation, as shown in Fig. 1.9. Let the rotated centroid at the new frame be given by F C R . An arbitrary point p x in the first frame gets deformed to p in the next frame. The base/un-deformed frame is F C and the co-rotated/deformed frame is F C R (all variables with • are expressed in the corotated frame F C R ). c is the displacement of the centroid of this element from F C to F C R . X p and Y p are the position vectors of p x at F C and F C R respectively. Using elementary geometric transformations [Felippa, 2000], we get:

dp = R d p = R u p -c + X p -X p (1.55)
where R is the rotation matrix between F C and F C R , u p is the total displacement of p x , expressed in the base frame F C , and dp is the deformational displacement of p x , expressed in the corotated frame F C R and d p is the same displacement in the base frame. Assuming a nodal force fr acting on p x and the associated deformational stiffness is K, if the system is modelled to be conservative, we get:

U( dp , λ r ) = U( dp ) -W( dp , λ r ) (1.56)
where U( dp , λ r ) is the element-level discrete potential energy, λ r is the load parameter, while U( dp ) and W( dp , λ r ) are the internal strain energy and the external work function respectively. Differentiating Eqn. 1.56 w.r.t dp yields:

r = ∂U( dp , λ r ) ∂ dp = ∂U ∂ dp - ∂W ∂ dp = f I r -fr = 0 (1.57)
Clearly, f I r is the vector of internal forces. This gives us the formulation for the residual force vector r in the co-rotated frame. This leads to the formulation:

r = τ r = f I r -f r (1.58) f r = τ fr (1.59) f I r = τ f I r (1.60) K = τ Kτ + r • Q (1.61)
where the matrix τ denotes the change of position of dp with ũp and the variation of the i-th vertex w.r.t the j-th vertex's displacement is given by:

τ ij = ∂ dp i ∂ũ p j (1.62)
and:

Q ijk = ∂ 2 dp i ∂ũ p j ∂ũ p k (1.63)
τ and Q forms the transformation array for this co-rotated configuration. When r = 0, we can write K = τ Kτ . Now, applying similar motion to the nodes of the tetrahedron, we can represent the internal, elastic forces acting on the nodes by:

F e = R e K e ŨR (1.64) Here, ŨR = ( p1 , p2 , p3 , p4 ) 12×1 , K e 12×12
is the stiffness matrix and R e

12×12

is the block diagonal matrix of four R rotation matrices stacked diagonally. F e

12×1

is the elastic forces acting on the nodes of this tetrahedral element.

To determine the interaction between forces and their resulting displacement using the FEM, we solve a second order differential equation, given by:

Mp + D ṗ + K g p = F ext + F e (1.65)
where M and D are the model's mass and damping matrices respectively. The elements of M are given as :

M ij = Ψ φ i φ j dV (1.66)
where M ij denotes the element of M at the i-th row and j-th column, φ i , φ j is the shape function at the i-th and j-th vertex of V M respectively, V is the volume of the tetrahdron associated with i and j and Ψ is the domain of V M . In this thesis, we use a Rayleigh style damping matrix D given by :

D = r M M + r K K g (1.67)
where r M and r K are the Rayleigh mass and Rayleigh stiffness respectively. Non-Rayleigh style damping matrices can be used as well [Pilkey, 1998]. K g is the global stiffness matrix. F ext are the external forces acting on the vertices.

Interested readers may refer to [Hughes, 2012] for a detailed treatment of standard FEM formulation.

Tracking using Physics-based Models

Using a triangular finite element method, [START_REF] Agudo | Modal space : A physics-based model for sequential estimation of time-varying shape from monocular video[END_REF] proposes a method to tackle deformation of planar objects using modal analysis using the shape basis of the modes of deformation. [START_REF] Agudo | Modal space : A physics-based model for sequential estimation of time-varying shape from monocular video[END_REF] is based on the seminal work of [START_REF] Bregler | Recovering non-rigid 3d shape from image streams[END_REF], which proposed a new method to recover 3D non-rigid shape models from image sequences by representing the 3D model as a combination of basis shapes. In [START_REF] Agudo | Learning shape, motion and elastic models in force space[END_REF] the elastic model of the object is estimated by approximating the full force-field from a low-rank basis. This approach estimates the entire force field acting on all points of the considered object while estimating the elastic model of that object and the pose of the camera using an Expectation Maximization (EM) algorithm. [START_REF] Agudo | Good vibrations : A modal analysis approach for sequential non-rigid structure from motion[END_REF] uses modal analysis from continuum mechanics to model the shape as discretized, linear elastic triangles and the solution to the force balance equations for an undamped free vibration model of the underlying FEM is used to recover the shape of the deforming object. Using a slightly different approach, [START_REF] Malti | Elastic shape-from-template with spatially sparse deforming forces[END_REF] proposes a new class of method, termed Sparse Linear Elastic -SfT (SLE-SfT), which replaces the 2 -norm minimization of standard SfT approaches with 0 -norm for minimizing the cardinal of the non-zero components of the deforming forces. An alternate formulation minimizing the 1 -norm of the sum of absolute values of force component has also been proposed. None of these approaches uses the Solid Boundary Constraints (SBC) to rigidly position the shape in the object frame. [START_REF] Royer | Real-time target tracking of soft tissues in 3d ultrasound images based on robust visual information and mechanical simulation[END_REF] proposed an elastic model based technique for tracking of deformable internal organs of human using ultrasound data. [Petit et al., 2015a] proposes a real-time approach for tracking 3D deformable object using a volumetric mesh in realtime, with the help of FEM simulations. In this approach, the images from the RGB-D sensor are segmented to isolate the object to be tracked using a grabcut [START_REF] Rother | Grabcut : Interactive foreground extraction using iterated graph cuts[END_REF] approach, which is in turn utilized to extract a segmented pointcloud. To track the deformation of this object, an external force is applied to the mechanical model (simulated with FEM) of the object. The per-vertex force exerted is given by:

F ext i = ζ i k(P Vm i -P ψ i ) (1.68)
where F ext i is the external force vector for the i-th element of V m which is given by

P Vm i , ζ i = σ 1 e -( d i σ 2
) such that d i is the closest distance of P Vm i from the projected 2D contour of the object, σ 1 and σ 2 are two empirically chosen constants, k is the stiffness of the element corresponding to the i-th node, and the variable P ψ i is obtained from the segmented pointcloud ψ by the formulation:

P ψ i =        αNN ψ (P Vm i ) + (1 -α) 1 #ψ i P j ∈ψ i P j if #ψ i > 0 αNN ψ (P Vm i ) + (1 -α)P Vm i else (1.69)
where NN ψ (P Vm i ) denotes the nearest neighbor of

P Vm i in ψ, ψ i is a subset of ψ such that ∀(ψ i ) j ∈ ψ i there exists NN Vm ((ψ i ) j ) = P Vm i
and α is a tunable parameter. An example of the tracking achieved by this approach is shown in Fig. 1.10. [START_REF] Petit | Tracking fractures of deformable objects in real-time with an rgb-d sensor[END_REF] extends the same approach to track fractures in the deforming objects, while tracking their deformation at the same time. Both these approaches use co-rotational FEM as the mechanical model and simulates the full volumetric deformation of the object to track deformation. [START_REF] Petit | Capturing deformations of interacting non-rigid objects using rgb-d data[END_REF] also extends this method to capture collision among interacting objects while tracking their deformation simultaneously.

[ Wang et al., 2015] proposes a method to capture deformation of soft objects using co-rotational FEM, where the tracking is handled with a probabilistic approach. Given a set of points ψ i ∈ ψ in a pointcloud, the problem statement is to fit a surface model V s to the pointcloud to maximize the probability (represented by the function p(•)) of vertex position, given the measurement of the pointcloud, such that:

V s = arg max Vs p(V s |ψ) (1.70)
This maximum-aposteriori-estimation (MAP) problem is solved by regulating the external force acting on the mechanical model V m which is connected with the surface model V s in a scheme similar to [Petit et al., 2015a], as shown in Fig. 1.5. This external, virtual force is expressed as:

f j = η i p(z ji )σ -1 j (P ψ i -P Vs j ) (1.71)
where η is a scaling factor, σ j is the diagonalized co-variance matrix of P ψ i and z ji is a latent variable that indicates if P ψ i has any contribution towards P Vs j or not. A graphical outline of this approach is shown in Fig. 1.11.

It must be noted that [Wang et al., 2015] is an improved variant of the approach proposed in [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF] and most of the fundamental concepts of tracking pointclouds using virtual force is based on Schulman et. al. However, unlike [Schulman extends the deformation tracking approach to also estimate the material properties of the object being tracked by studying the oscillation of the object as it settles down to its original, undeformed configuration after a simple twisting or pulling interaction. A closely related problem statement was also considered by [Petit et al., 2017a] wherein the material properties of a deforming object that is being manipulated by a robotic arm was estimated, as well as the force on the contact points were tracked after the estimation of the material property has been done using the tracking mechanism of [START_REF] Petit | Tracking fractures of deformable objects in real-time with an rgb-d sensor[END_REF].

Geometric Models

As delineated in Fig. 1.4, not all model based tracking methodology uses physically based models to explain the deformation. There are many approaches in the nonrigid object tracking literature that utilizes geometric models, often inspired from the computer graphics community, to track deformation. We now describe some of the fundamental concepts required to model and track deformations geometrically.

To control the deformation of a mesh model geometrically, we need a regularizing function or a method that can propagate the displacement of a single vertex, often called the control handle, to the adjacent mesh vertices. It is preferable to utilize a method that propagates the deformation without causing any un-realistic mesh deformation or decimation. Obviously, these methodologies have many different sub-categories, which are frequently combined with each other to derive numerous variations of geometric deformations models. However, it is still possible to cluster most of these methods into some frequently recurring themes, some of which are highlighted below:

P 1 P 2 P 3 P 4 P 1 P 2 P 3 P 4 FIGURE 1.
12 -Regularization of deformation using embedded deformation graphs

Embedded Deformation Graph

In this approach, the spatial deformation of an object is represented by a collection of affine transformations organized in a graph structure. Each node of the graph has a particular transformation associated with it and the deformation is propagated to the nearby space by a series of affine transforms, as explained below. Let a particular node P i has the rotational matrix R i and translation t i associated with it. This transformation is propagated to a node P j (which is connected to P i through a graph of undirected edges) to produce its deformed configuration P j via the relationship:

P j = n i=1 w i (P j ) R i (P j -P i ) + P i + t i (1.72)
where w i (P j ) is a weighting parameter. As shown in Fig. 1.12, with this deformation propagation mechanism in place [START_REF] Sumner | Embedded deformation for shape manipulation[END_REF], a set of rotational and regularizational constraints can be imposed on this graph structure to result in spatially coherent and smooth deformation when some controlled nodes are deformed arbitrarily (either by user specified commands, or by minimization of some secondary error terms).

Thin plate spline (TPS)

TPS warp presents an efficient shape interpolation and smoothing function for controlling or regularizing the deformation of non-rigid surfaces in 2D or 3D. In its sim-plest form, the 3D interpolation is given by:

P = a 0 + a 1 X P + a 2 Y P + a 3 Z P + N i=1 λ i U || P -P i || (1.73)
where

P i = (X P i , Y P i , Z P i ) | i = 1, 2, • • • , N (1.74)
are the landmark points, a 0 , a 1 , a 2 , a 3 , λ 1 , • • •, λ N are weighting coefficients and P = (X P , Y P , Z P ) are the interpolated points. U(x) = x 2 log(x 2 ) is the radial basis function.

As-rigid-as-possible (ARAP) regularizer

Assuming that we have a triangular mesh V which gets deformed to a new configuration V , while two connected model vertices are denoted by P i and P j , ARAP [START_REF] Bregler | Recovering non-rigid 3d shape from image streams[END_REF] proposes to regularize deformation by minimizing the energy term:

E ARAP = n i=1 w i j∈X V w ij || P i -P j -R P i -P j || 2 (1.75)
where w i and w ij are suitable cell and edge weights, while R is the approximate rigid rotation matrix between V and V .

Coherent Point Drift (CPD)

In its simplest form, CPD [START_REF] Myronenko | Point set registration : Coherent point drift[END_REF] offers a mechanism to align two pointclouds with each other, irrespective of the nature of the transformation between them, i.e., it can be used to estimate both rigid and non-rigid transformations. Assuming that a set of target point cloud is denoted by ψ

X = {p x i } | i = 1, • • •, N , while ψ * Y denotes the centroids of the Gaussian Mixture Model (GMM) of another set of source pointcloud ψ Y = {p y j } | j = 1, • • •, M , CPD based
alignment involves the minimization of the following term:

E CPD = - N n=1 log M+1 m=1 P(m)p(p x | m) (1.76)
where

p(p x | m) = 1 (2πσ 2 ) e -||px-py m || 2 2σ 2
(1.77) FIGURE 1.13 -The problem statement of SfT (Image courtesy [Chhatkuli, 2016])

V τ ⊂ R 3 Ω ⊂ R 2 S ⊂ R 3 I ⊂ R 2
Here, membership probability P(m) = 1 M while σ 2 is the isotropic covariance of the GMM components. The parameters of the GMM as well as σ 2 can be estimated using an Expectation Maximization (EM) [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF] strategy, thereby providing us with a method to register both rigidly transforming and/or deforming points.

Shape-from-Template (SfT)

As demonstrated in Fig. 1.13, the basic problem statement of SfT involves determining the function ϕ ∈ R 3 that embeds a flat template image Ω ⊂ R 2 into a 3D deforming surface S ⊂ R 3 , given that the 3D undeformed model V τ ⊂ R 3 is given and the warping function ⊂ R 2 between the 2D template and the image of the deformed surface can be determined externally. [START_REF] Bartoli | Shape-from-template[END_REF] provides the basic solution to this problem. The embedding function can be represented as:

ϕ = χ • ∆ (1.78)
where χ is the isometric deformation of V τ and ∆ is the function that embeds Ω into V τ . On such a system, two different constraints are imposed. The first constraint aims to preserve the isometry, and is given by:

J ϕ J ϕ = J ∆ J ∆ (1.79)
where J ϕ and J ∆ are the Jacobians obtained by differentiating ϕ and ∆. The second constraint ensures the consistency between the warp and the projection of the embedding in the image Π, and is given by:

= Π • ϕ (1.80)
Eqn. 1.79 and eqn. 1.80 forms a system of partial differential equations and an analytic solution for the unknown parameters has been established in [START_REF] Bartoli | Shape-from-template[END_REF]. Many variants of the standard SfT formulation exists in the literature and is briefly reviewed in the following paragraph.

Tracking using Geometric Models

A method to detect and recover three dimensional shape from monocular video sequence was proposed in [START_REF] Salzmann | Surface deformation models for nonrigid 3d shape recovery[END_REF]. This approach creates a low dimensional model of deformable, planar 3D surfaces and based on the assumption that the deformation is strictly isometric, a dimensionality reduction technique is used to the deformation models needed for tracking and detection. [START_REF] Pilet | Fast non-rigid surface detection, registration and realistic augmentation[END_REF] provided a method to detect and register non-rigid, planar surfaces based on wide-baseline point matches between the undeformed and deformed image of an object. [START_REF] Salzmann | Closed-form solution to non-rigid 3d surface registration[END_REF] offers a closed form solution to the problem of recovering 3D shape from non-rigid, elastic 3D-to-2D correspondences. The set of 3D-to-2D correspondences constitutes a linear system, which can also be expressed as a weighted sum of eigen vectors, the solution of which accounts for the inextensibility constraints between neighboring mesh vertices.

[ [START_REF] Bartoli | Shape-from-template[END_REF] proposes a first-order method using both image point loca-FIGURE 1.14 -Tracking of a deforming object using the approach proposed in [START_REF] Salzmann | Closed-form solution to non-rigid 3d surface registration[END_REF] (Image Courtesy [START_REF] Salzmann | Closed-form solution to non-rigid 3d surface registration[END_REF] tions and the first-order differential structure extracted from the warp from template of an object to an input, deformed image to reconstruct the deformed shape of the surface (the SfT problem). [START_REF] Bartoli | Shape-from-template[END_REF] dealt with planar meshes only. However, it was extended to handle volumetric meshes in [START_REF] Parashar | Asrigid-as-possible volumetric shape-from-template[END_REF]. Two methods were proposed for interpolating the volumetric shape, one was based on a global smoothness prior, while the other is based on a locally rigid transform at every point of the visible surface, somewhat similar to ARAP. [START_REF] Collins | Realtime shape-fromtemplate : System and applications[END_REF] describes a realtime SfT based approach. [START_REF] Haouchine | Single view augmentation of 3d elastic objects[END_REF] combined SfT with elasticity priors with SfT to achieve realtime augmentation on planar deforming surfaces. [START_REF] Willimon | An energy minimization approach to 3d non-rigid deformable surface estimation using rgbd data[END_REF] proposes a joint energy minimization of sparse feature correspondences and depth based error terms along with smoothness priors to track deforming laminar objects from RGB-D data.

Non-rigid Tracking and Reconstruction

Model free tracking of deformation involves tracking the surface of the deforming object as well as iteratively generative the mesh model of the entire surface of the object or scene under consideration. Many of the fundamentals used for geometric model-based tracking are re-utilized for reconstruction and tracking as well. Tracking and reconstruction of non-rigid objects using RGB-D cameras have been developed in recent years. [START_REF] Zollhöfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF] tracked a deformable template in real-time, without having any limitations about apriori knowledge of the template. It must be noted that this research is entirely based on the output of a stereo rig and not from a depth camera. The approach works by minimizing the global rigid transformation as well as the per-vertex displacement between consecutive frames. This is done by minimizing four error terms: a point-to-point distance, a point-to-plane distance, direct color intensity difference and a geometric prior term acting as the ARAP regularizer.

Frame 3: Truncated Signed Distance Field (TSDF)

TSDF and Signed Distance Function (SDF) has been used extensively in rigid and nonrigid reconstruction from depth sensors in the recent years, including [START_REF] Newcombe | Kinectfusion : Real-time dense surface mapping and tracking[END_REF], [START_REF] Izadi | Kinectfusion : real-time 3d reconstruction and interaction using a moving depth camera[END_REF], [START_REF] Newcombe | Dynamicfusion : Reconstruction and tracking of non-rigid scenes in real-time[END_REF], [START_REF] Slavcheva | Sdf-2sdf : Highly accurate 3d object reconstruction[END_REF], [Slavcheva et al., 2017a] and [START_REF] Slavcheva | Sobolevfusion : 3d reconstruction of scenes undergoing free non-rigid motion[END_REF], among many others. The TSDF algorithm provides a simple method to reconstruct rigid scene from multiple depth scans (using any depth sensor) and it is possible to modify the TSDF algorithm to effectively reconstruct non-rigid scenes too. Given its efficacy and popularity, it can be useful to take a closer look at the TSDF methodology for the sake of completeness. TSDF was introduced for rigid object reconstruction by [START_REF] Newcombe | Kinectfusion : Real-time dense surface mapping and tracking[END_REF] and [START_REF] Izadi | Kinectfusion : real-time 3d reconstruction and interaction using a moving depth camera[END_REF]. However, the principle of TSDF is heavily based on SDF, introduced by [START_REF] Curless | A volumetric method for building complex models from range images[END_REF]. SDF is a voxel based implicit representation of a scene, where each voxel is associated with a function measuring the distance of the voxel from the nearest surface. Given a depth sensor, the SDF algorithm proposes to maintain a set of voxels which contain the distance function d sdf that gives the distance of a voxel from the nearest surface S sdf and has a positive sign if it is closer to the sensor w.r.t the viewing direction and negative if it is farther, as shown in Fig. 1.15. For every new data frame from the depth sensor, the range surface (S k sdf for the k-th frame) is a piecewise linear surface created by connecting nearest neighboring points with triangles in the depth scan. Every voxel is also associated with a weight function w sdf , which maybe an elaborate function modeling the uncertainty in measurement at each updated voxel, or in the simplest case of [START_REF] Curless | A volumetric method for building complex models from range images[END_REF], proportional to the angle between the viewing direction of the sensor and the surface normal of S sdf at a point closest to the given voxel. The algorithm for updating the SDF for the i-th voxel in a new data frame ((k + 1)-th range image) is given below :

Algorithm 1: Computing SDF from depth data 1 Construct S k+1
sdf from (k + 1)-th range data ; 2 for i ← 1 to no. of voxels in (k + 1)-th range image do 3 Compute ( dk+1 sdf ) i and ( ŵk+1 sdf ) i , which are i-th value of d sdf and w sdf respectively, computed just from S k+1 sdf ; [START_REF] Izadi | Kinectfusion : real-time 3d reconstruction and interaction using a moving depth camera[END_REF] modified the method of Algorithm 1 to store the SDF only for a truncated region around the actual surface, resulting in the TSDF value d tsdf , given by :

4 (d k+1 sdf ) i = ( dk+1 sdf ) i ( ŵk+1 sdf ) i +(d k sdf ) i (w k sdf ) i ( ŵk+1 sdf ) i +(w k sdf ) i ; 5 (w k+1 sdf ) i = ( ŵk+1 sdf ) i + (w k sdf ) i 6 end
(d k+1 tsdf ) i =      min 1, (d k+1 sdf ) i σmax If (d k+1 sdf ) i > 0 max -1, (d k+1 sdf ) i σ min else (1.81)
where σ max and σ min are the maximum and minimum truncation depth required by the algorithm. An example of surface reconstruction from 3 consecutive RGB-D frames (captured at 30 fps) using this TSDF methodology is demonstrated in Fig. 1.16.

FIGURE 1.16 -Example of rigid reconstruction using TSDF methodology (similar to [START_REF] Izadi | Kinectfusion : real-time 3d reconstruction and interaction using a moving depth camera[END_REF]) for 3 consecutive frames coming from a RGB-D camera, as shown in (a). The TSDF volume is computed and the isosurface is extracted using the Marching Cubes method [START_REF] Lorensen | Marching cubes : A high resolution 3d surface construction algorithm[END_REF] with a truncation distance of 5mm, 1cm and 1m for (b), (c) and (d) respectively

σ min , σmax = ±5mm σ min , σmax = ±1cm σ min , σmax = ±1m
DynamicFusion [START_REF] Newcombe | Dynamicfusion : Reconstruction and tracking of non-rigid scenes in real-time[END_REF] extended the premise of KinectFusion into non-rigid objects. It enabled the tracking and modeling of deformable objects on the go, without using a previously known template. In DynamicFusion, the objective is split into three steps, estimating the warping function of the current depth frame from the canonical model, fusion of the live depth frame into the canonical model once the warping function has been determined and adaption of the warp field structure to track the camera movement using ICP. The non-rigid warp field is represented as a 6D transformation in SE3, but to avoid memory overhead, this 6D transformation is kept sparse, while all the dense points in between are interpolated using a weighted blending of dual quaternion. The weight alters the radius of influence of each transformation node. Fusion between the current depth frame and the canonical model is done using the TSDF fusion technique. The warp field is estimated by minimizing an energy function comprising of a data term, which is the dense model-to-frame ICP cost, and a regula-FIGURE 1.17 -Comparison between [Slavcheva et al., 2017a] and [START_REF] Slavcheva | Sobolevfusion : 3d reconstruction of scenes undergoing free non-rigid motion[END_REF]. The highlighted regions show the over-smoothing tendencies of [Slavcheva et al., 2017a] (Image Courtesy [START_REF] Slavcheva | Sobolevfusion : 3d reconstruction of scenes undergoing free non-rigid motion[END_REF]) RGB-D Data [Slavcheva et al., 2017a] [ [START_REF] Slavcheva | Sobolevfusion : 3d reconstruction of scenes undergoing free non-rigid motion[END_REF] RGB-D Data [Slavcheva et al., 2017a] [Slavcheva et al., 2018] rization parameter. The results of DynamicFusion has been demonstrated on a variety of dataset consisting of 3D toys and human body.

[ [START_REF] Gao | Surfelwarp : Efficient nonvolumetric single view dynamic reconstruction[END_REF] replicates the approach of DynamicFusion using surfels instead of volumetric TSDF, thereby imporving computational time requirements. [START_REF] Innmann | Volumedeform : Real-time volumetric non-rigid reconstruction[END_REF] extended the work of DynamicFusion to produce even more robust and accurate results. VolumeDeform is quite similar to DynamicFusion. The notable differences between the two approach is that VolumeDeform does not use dual quaternion based interpolation to convert sparse correspondence to dense. Instead, it uses a novel point-to-plane alignment strategy as the dense term of the energy function. It also uses a SIFT feature based sparse color point correspondence to robustify the tracking of the model. A regularization term is maintained as before. [Slavcheva et al., 2017a] proposes a non-rigid reconstruction technique using a combination of three data terms: the first penalizing Euclidean distance between the voxel positions, the second ensuring that the non-rigid deformation follows the Killing property [Ben- Chen et al., 2010] and the third ensuring that the gradient magnitude of SDF always remains unity. Named after Wilhelm Killing [Coleman, 1989], Killing vector fields are used in the context of deformation tracking as an isometric or nearly-isometric regularizer. It is surprisingly straightforward to adapt the Killing conditions to act as a geodetic distance preserving constraint on deforming, discrete manifolds. Assuming a warp field denoted by ϕ = (α, β, γ) | ϕ ∈ R 3 acting on a set of points which can be represented in the form P = (X, Y, Z), the Jacobian of the warp field is given by J ϕ = ∂ϕ ∂P . On such a vector field, the Killing condition is denoted by J ϕ + J ϕ = 0. However, to allow nearly isometric deformations, most authors prefer to penalize deformation by minimizng the Killing energy :

E AKVF = X,Y,Z || J ϕ + J ϕ || 2 F (1.82) where || • || 2
F is the Frobenius norm. Assuming that Υ gives the current TSDF volume, a comparison between [START_REF] Newcombe | Dynamicfusion : Reconstruction and tracking of non-rigid scenes in real-time[END_REF], [START_REF] Gao | Surfelwarp : Efficient nonvolumetric single view dynamic reconstruction[END_REF], [START_REF] Innmann | Volumedeform : Real-time volumetric non-rigid reconstruction[END_REF] and [Slavcheva et al., 2017a] can be delineated as given in Table 1.1. [START_REF] Slavcheva | Sobolevfusion : 3d reconstruction of scenes undergoing free non-rigid motion[END_REF] alters the approach of [Slavcheva et al., 2017a] by proposing a Tokhonov regularization strategy of Sobolev gradient flow to replace the level-set data term and approximate Killing condition based regularization of [Slavcheva et al., 2017a].

DynamicFusion [START_REF] Newcombe | Dynamicfusion : Reconstruction and tracking of non-rigid scenes in real-time[END_REF] SurfelWarp [START_REF] Gao | Surfelwarp : Efficient nonvolumetric single view dynamic reconstruction[END_REF] VolumeDeform [START_REF] Innmann | Volumedeform : Real-time volumetric non-rigid reconstruction[END_REF] KillingFusion [Slavcheva et al., 2017a] Cost Function

E = E data + λE reg E = E depth + λE reg E = w d E dense + w r E reg + w s E sparse E = E data + w r E levelset +

Non-Rigid Structure from Motion (NR-SfM)

NR-SfM is an excellent technique to resolve the shape of a deforming object along with camera motion parameters. It is inherently a batch technique, requiring sequential observation of deforming keypoints on the object. This makes it somewhat unsuitable for sequential, fast tracking of deforming surfaces. However, for the sake of completeness, we take a brief look at some of the notable approaches developed from this technique.

Let the shape of a non-rigid object be described by a set of K keyframe basis, denoted by S i 3×P denoting P points. The overall shape can be described using:

S = K i=1 l i • S i | l i ∈ R (1.83)
Under scaled orthographic projection, the basis set can be used to express the image coordinates (u i , v i ) of the keypoints as:

  u 1 • • • u P v 1 • • • v P   = l 1 R [2] • • • l K R [2]            S 1 • • • S K            (1.84)
where R [2] gives the first two rows of the camera rotation matrix. Assuming we have L data frames for the sequence, we can re-arrange Eqn. 1.3.4 as:

ς 2L×P =            u 1 1 • • • u 1 P v 1 1 • • • v 1 P • • • • • u L 1 • • • u L P v L 1 • • • v L P            =            l 1 1 R 1 [2] • • • l 1 K R 1 [2] • • • • • • • • • • • • • • • l L 1 R L [2] • • • l L K R L [2]            Q 2L×3K            S 1 • • • S K            S l 3K×P (1.85)
ς is termed as the tracking matrix and can be formed by observing a set of keypoints across multiple frames (e.g: using a feature detector/matching technique). Applying a FIGURE 1.18 -Results from applying NR-SfM using the approach of [START_REF] Kumar | Multi-body non-rigid structure-from-motion[END_REF]. The top row shows the input images, while the row at the bottom shows the extracted meshes.

(Image Courtesy [START_REF] Kumar | Multi-body non-rigid structure-from-motion[END_REF])

SVD decomposition on ς, we get SVD: ς = UΣV = QS l . Solving this equation, the rotational pose of camera R and shape configuration S can be factored simultaneously.

[ Gotardo and Martinez, 2011a] proposed a kernel trick to better constraint the weakened low rank constraint, without requiring additional basis shapes to model keypoints moving along curved paths. [START_REF] Gotardo | Non-rigid structure from motion with complementary rank-3 spaces[END_REF] improved the standard NR-SfM representation by considering K complementary spaces of rank 3 as the 3D basis shapes for factorization. [START_REF] Kumar | Multi-body non-rigid structure-from-motion[END_REF] extends NR-SfM to multibody deformation tracking using a subspace clustering approach (Fig. 1.18).

Positioning this Thesis

Having provided an overview of the existing state-of-the-art for deformation tracking, we now discuss the aim of this thesis and compare our contribution with the available literature from an application perspective. Before we begin discussing the technical details in the subsequent chapters, it is important to justify the approach that we utilize to track deforming objects. It is important to weigh the relative advantages and disadvantages of the state-of-the-art approaches in the context of this thesis.

The prominent research approaches presented in Sec. 1.3.1 through Sec. 1.3.4 can be grouped on the basis of the input data, as shown in [START_REF] Kumar | Multi-body non-rigid structure-from-motion[END_REF] SfT : [START_REF] Bartoli | Shape-from-template[END_REF]] [Malti et al., 2015] (planar) ; [START_REF] Parashar | Asrigid-as-possible volumetric shape-from-template[END_REF] (volumetric)

Planar model [START_REF] Salzmann | Surface deformation models for nonrigid 3d shape recovery[END_REF]] [Salzmann et al., 2008] [Pilet et al., 2008] [Agudo et al., 2014 approaches that primarily confine themselves to monocular input images. Inexpensive depth cameras are readily available as consumer-grade sensors (e.g.: the ones mentioned in Sec. 1.1.4) and not using the depth information is unnecessary and counterintuitive for robotic applications. We also want to deal with approaches that track objects based on volumetric information, i.e., laminar objects are not of much interest to this thesis.

Once we remove all the monocular camera based approaches and laminar object tracking methodologies, we are left with the geometric and physical model based approaches. We are also left with reconstruction and tracking algorithms. At this stage, it is important to recall the objective for the non-rigid tracking approaches proposed in this dissertation. This thesis has been primarily directed towards aiding robotic manipulation of soft objects. In the context of robotic manipulation, it can be noted that only with volumetric and physical-model based approaches do we get the following advantage:

-A well-developed deformation model that is suitable for small and large deformation, i.e. FEM, co-rotational FEM etc.

-The ability to aide and enable robotic grasping and manipulation of deforming objects -Tracking deformation model using coarse model of arbitrary resolution (i.e., suitable for tracking using object model of very low polygonal density) and mechanical model of arbitrary physical parameters -Deformation tracking using physically-based models can be readily extended to act as a contact force tracker and also utilized to estimate the physical parameters of a deforming object Given these strong advantages, we dedicate the rest of this dissertation to the study of deformation tracking using physically-based models.

Conclusion

In Sec. 1.1 of this chapter, we presented a summary of classical techniques involving visual tracking of rigid objects and physical modelling of deforming objects using standard and co-rotational FEM. In this thesis, a combination of these two paradigms are going to be used to design a set of methodologies that enable the tracking of deforming objects.

In Sec. 1.3 of this chapter, we discuss the state-of-the-art for non-rigid object tracking using visual information. Only a small subset of these approaches involve physical models for deformation tracking. We consider volumetric physical model based deformation tracking as the most relevant framework, since it enables us to achieve the following objectives (some of which has already been implemented while some which remains a future work [ marked with a * ]):

-Accurate and fast 3D tracking of deforming surface, with and without the knowledge of the physical parameters (Young's modulus, Poisson's ratio etc.). Presence of advanced physical model enables us to track large deformations -Extending the 3D tracking framework to either track external, applied forces on the object or to estimate the physical parameters of the object -Tracking multi-object collision and non-isometric deformations (cutting, tearing, fracturing etc.) of the object * -Providing a deformation model that can be used to predict the future state of a deforming object when force is applied on it -which can have significant utilities in the area of robotic manipulation of compliant objects

Chapter 2 provides a framework for tracking of rigid objects, which is an essential initial step for subsequent non-rigid tracking. Chapter 3 details the first methodology developed to track deforming object using depth data only. This is followed by Chapter 4, which improves the approach of Chapter 3 to utilize both depth and photometric data simultaneously, while the tracking methodology evolves to become faster and more robust. Chapter 5 describes a robotic application of the approach developed in Chapter 3 involving estimation of physical parameters of objects and visual force tracking.

RIGID OBJECT TRACKING

In the previous chapter, we introduced some of the existing literature on non-rigid object tracking and discussed some of the fundamental concepts about deformation modelling and tracking. Multiple real-time non-rigid tracking approaches, e.g., [START_REF] Petit | Capturing deformations of interacting non-rigid objects using rgb-d data[END_REF], uses two separate methods for non-rigid object tracking. One that approximately tracks the object rigidly, and another that tracks the deformation independently from the rigid motion. This is especially useful for the physical model based deformation tracking that we intend to do for this thesis. To this end, in this chapter we present a method to track rigid motion in objects with complex shapes using approximate 3D models of those objects. This is entirely based on RGB-D sensors. The key contributions of this chapter are:

-Combining point-to-plane distance minimization and photometry for tracking of complex objects using coarse model ;

-Using the concept of 'keyframe' in object tracking for increased robustness ;

-Accurate tracking of objects with inaccurate and coarse object models.

In this thesis, a coarse model implies a 3D, surface based model of an object without high density of vertices (approximately < 6800 vertices/m 2 on the surface of the mesh) and with relatively higher registration error (∼ ±2cm in the worst cases) with the object.

Background

Although a brief summary of research about rigid object tracking was already provided in Sec. 1.2.2, it is worthwhile to look into some more state-of-the-art research related to rigid object tracking in the context of this chapter specifically. Research related to localizing and/or tracking simple geometric shapes is extensive [START_REF] Lepetit | Monocular model-based 3d tracking of rigid objects : A survey[END_REF] and has been studied since a long time [START_REF] Armstrong | Robust object tracking[END_REF]. Some approaches use probabilistic models for tracking shapes. [START_REF] Prisacariu | Pwp3d : Real-time segmentation and tracking of 3d objects[END_REF] aims to maximize the discrimination between background and foreground of a scene while simultaneously tracking the 6DoF pose of an object with known 3D model. This method used only RGB images as input. [START_REF] Teichman | Learning to segment and track in rgbd[END_REF] extended this method to depth cameras, adding more cues like local image appearance, depth discontinuities, optical flow, and surface normal to inform the segmentation decision in a conditional random field model. Similar approaches have been used to track articulated objects [START_REF] Cifuentes | Probabilistic articulated real-time tracking for robot manipulation[END_REF] with remarkable precision, but the tracking of articulated joints remain beyond the scope of our discussion. Particle filtering based approaches have been parallelized using GPU and implemented towards tracking rigid objects [START_REF] Choi | RGB-d object tracking : A particle filter approach on GPU[END_REF]. The second group of algorithms uses learningbased techniques to track objects. [START_REF] Hinterstoisser | Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes[END_REF] proposes a pose estimation technique for complex objects under heavy clutter, but the training model needs color gradient information. In our method, however, we assume that the object's accurate model and any model based on color / texture are not available. [START_REF] Doumanoglou | Recovering 6d object pose and predicting next-best-view in the crowd[END_REF], [START_REF] Michel | Global hypothesis generation for 6d object pose estimation[END_REF] and [START_REF] Xiang | Posecnn : A convolutional neural network for 6d object pose estimation in cluttered scenes[END_REF] propose increasingly accurate posing estimation techniques, but they can not be considered real-time. [Mc-Cormac et al., 2018] and [START_REF] Runz | Maskfusion : Realtime recognition, tracking and reconstruction of multiple moving objects[END_REF] are both a dynamic SLAM algorithm that uses R-CNN Mask for segmentation and performs object tracking as a by-product. The accuracy of the per frame object tracking has not been evaluated explicitly. The third group of algorithms use well known minimization techniques to track objects using geometric and photometric constraints. [START_REF] Pomerleau | A review of point cloud registration algorithms for mobile robotics[END_REF] has already been introduced in Sec. 1.2 as an excellent summary of classical ICP. Many variants of ICP have been proposed for probabilistic [START_REF] Segal | Generalized-icp[END_REF] implementation of point set registration. In this context, SLAM and object tracking have been tackled interchangeably by some authors [START_REF] Salas-Moreno | Slam++ : Simultaneous localisation and mapping at the level of objects[END_REF]. CoFusion [START_REF] Rünz | Co-fusion : Real-time segmentation, tracking and fusion of multiple objects[END_REF] is a dynamic SLAM system that uses a combination of dense ICP and photometric error minimization between the model and the current frame to track objects. The approach we propose, in terms of the framework for object tracking, is somewhat closer to CoFusion [START_REF] Rünz | Co-fusion : Real-time segmentation, tracking and fusion of multiple objects[END_REF]. However, we do not undertake explicit motion segmentation, we are not interested in reconstruction and we use the concept of keyframes. Moreover, the presence of coarse object model makes it a different problem statement altogether.

In this chapter we focus on the high-accuracy tracking of rigid, complex shapes with approximately known geometry using depth cameras. We consider the coarse model to be a highly decimated, minimal representation of the object model (instead of a high-resolution and detailed CAD model), containing very few triangular faces at best. The model does not contain any color or textural information. This is easy to generate and can also be rendered without a depth scanner (e.g: using manual measurements), making it suitable for various industrial applications. Our approach is significantly robust to measurement errors in the coarse model. We are interested in tracking all the 6 degrees of freedom [START_REF] Yilmaz | Object tracking : A survey[END_REF]] of the object. The proposed approach is validated both quantitatively and qualitatively in the subsequent sections.

Method

The approach proposed here to track rigid objects is a combination of point-toplane distance minimization and photometric error minimization between frames. We work with two types of input data: 1) registered and calibrated depth and grayscale images from the depth camera, and 2) simulated data of 3D points along with their corresponding grayscale values.

We denote the depth data (obtained from the sensor in the RGB-D format) as:

ψ = (P 1 , c 1 ), • • • , (P N , c N )
(2.1)

where P i = (X i , Y i , Z i ) and c i is the intensity value of the point P i , expressed in the camera centered coordinate frame. A function Π(•) acts upon a 3D point and projects it to an image plane using the pinhole camera model such that p i = Π(P i ). The function

c i = I(p i )
provides the image intensity of the point p i . ∇I i,x and ∇I i,y gives the gradient of the image along x and y axis.

The model of the object is also presumed to be known for the purpose of tracking. We represent it as a surface 3D mesh composed of a set of planes given by O V, X V and n j = (n X j , n Y j , n Z j ) is the normal to the j-th plane of the mesh which lies at a perpendicular distance of d j from the origin of the camera-centered coordinate frame.

The n-th frame from the depth sensor (or from a simulated data) is denoted by C n , while certain frames get tagged as keyframes C n ⇒ C K n . F o denotes the object centered coordinate frame. These keyframes serve as the reference for photometric tracking and is explained in details in Sec. 2.2.3.

Tracking

For each data frame, we propose to minimize two cost functions that depend on a geometric term based on point-to-plane alignment and a photometric term that minimizes the difference in intensity between the predicted image (based on an estimate of the interframe transformation) and the projected image (based on the pointcloud transformation).

Point-to-plane Distance Minimization

For the geometric term, we minimize the point-to-plane distance between the 3D points in the n-th frame ψ n , with respect to the set of planes registered with the point cloud in the previous frame ψ n-1 . This is given by minimizing the distance error:

e dist i ( n q n-1 ) = n R n-1 P i + n t n-1 • n k -d k (2.2)
where i denotes a specific point in the pointcloud and k is the index of the plane in the object model, to which it corresponds. We address the topic of this correspondence in the next subsection.

This cost function is optimized using Gauss-Newton optimization and the Jacobian is given by partial differentiation of eq. (2.2) with respect to n q n-1 , given by:

J dist i = n k n k × P i (2.3)

Photometric Minimization

The estimate at the first iteration for the transformation between the last keyframe C K p and the current data frame C n used in photometry is given by:

n T p = n T O p T O -1 (2.4)
where p denotes the data frame number for the last keyframe. From this initial estimate of the transformation between the last keyframe and current frame, the image intensity error that we seek to minimize for every image point is given as:

e img i n q p = I p Π P i -I n Π n R p P i + n t p (2.5) 68 2.2. Method m-th frame m + 1-th frame J dist tx J dist ty J dist tz J dist θux J dist θuy J dist θuz J img tx J img ty J img tz J img θux J img θuy J img θuz FIGURE 2
.1 -Jacobian of depth and photometric error between two data frames m and m + 1. J dist and J img are the gradient of the geometric and photometric error respectively, while the suffix t x , t y , t z , θu x , θu y , θu z represents the component of the gradient along the 6 dimensions of the (t, θu) vector (the gradient intensities have been thresholded for visualization)

The Jacobian used for this minimization is:

J img i [1×6] = ∇I i [1×2]   f x 0 0 f y   [2×2]   -1 Z 0 X Z 2 XY Z 2 -(1 + X 2 Z 2 ) Y Z 0 -1 Z Y Z 2 -(1 + Y 2 Z 2 ) -XY Z 2 -X Z   [2×6]
(2.6)

where

∇I i = ∇I i,x ∇I i,y (2.7)
and f x and f y are the focal lengths. The value of the Jacobians expressed in eq. 2.3 and eq. 2.6 can be projected on the image plane and visualized, as shown in Fig. 2.1.

Optimization

Having computed the error and the Jacobian matrices for the two different types of error function, we now describe the method used for computing the pose update that minimizes these two errors jointly. The optimization that we use is the Iteratively Reweighted Least Squares (IRLS). The weighting function, as described below, allows the optimization to handle the presence of additional objects that occludes the object being trackes, as well as minor changes to illumination and the presence of sensor noise. The pose update is given by:

ξ = -λ WJ +

We

(2.8)

where:

J = J dist 1 , J dist 2 , • • •, J img 1 , J img 2 , • • • (2.9)
and:

e = e dist 1 , e dist 2 , • • •, γe img 1 , γe img 2 , • • • (2.10)
are the stacked Jacobian matrices and error vectors respectively. ξ can be interpreted as the velocity screw acting on the camera that transforms it from data frame n-1 to n. The pose update is given as:

n T n-1 = (∆T) n Tn-1 (2.11)
where ∆T = exp(ξ) represents the transformation between the last estimate and the updated estimate of the pose of the pose of the current frame and n Tn-1 represents the previous estimate of the transformation between the (n -1)-th frame and the current frame. Here :

γ = e dist e img
(2.12) is computed only once per frame, at the first iteration. γ serves as a scaling factor to ensure that the point-to-plane distance error and the photometric error are at a similar order of magnitude. W is a diagonal matrix of weights obtained from the m-estimator for being robust to outliers. We use Tukey biweight operator as the m-estimator [START_REF] Malis | Experiments with robust estimation techniques in real-time robot vision[END_REF]. For any given residual e i , we define the Tukey operator as:

ρ(e i , κ) =      e 6 i 6 + κ 2 e 4 i 2 + κ 4 e 2 i 2 if |e i | < κ 1 6 κ 6 else (2.13)
where κ = 4.7σ and σ = 1.48 × Median(|e i -ẽj |) and:

W =                ρ(e dist 1 , κ) ρ(e dist 2 , κ) . . . ρ(e img 2 , κ) ρ(e img 2 , κ) . . .               
(2.14)

The error vectors, the Jacobian matrices and weight matrices are re-computed for every iteration of the optimization. At this point, it must be noted that the Jacobian of the two error vectors J dist and J img are both full rank, thereby making it unnecessary to prioritize one error minimization over the another.

Point Correspondences

As indicated in eq. (2.2), there is a need to associate every 3D point P i with one of the planes of the model, which can be represented by the tuple ( O V k , X V k ), the k-th plane of the model. At every frame, we know the value of C n-1 T O . We project the 3D points O V k for all visible planes into the image obtained in the current frame. This gives us a set of 2D polygons in the image, each representing one of the visible faces. All the points in the image (x i , y i ) that intersects with a particular plane obtained from Π O V k are considered to be associated with this plane. This method is demonstrated with a toy example in Fig. 2.2.

Selection of Keyframe

We use keyframes to reduce drift in frame-to-frame photometric tracking, as shown in Fig. 2.3. The estimate of the transformation between the last keyframe and the current frame is given by Cn T C k . We decompose this transformation matrix into the translational component (t x , t y , t z ) and the θu rotational component. We define a variable p k , such that: The distance threshold is expressed in millimetre while the angular threshold is given in radian. At the very first frame, we set k = n = 0, thereby providing an initialization for the keyframe.

p k =      1 if ||(t x , t y , t z )|| > 0.05 or θ > 0.15 0 else (2.15)

Results

For benchmarking the performance of the tracker, we generated three short sequences of simulated RGB-D data, comprising of three objects: a marmalade container, a coffee machine and a simulated car. A very simple model is used to track these objects. The model is not required to be continuous or topologically closed. We end up with a coarse approximation of the shape of the object, comprising of 24, 48 and 62 faces for the three objects respectively. The tracker is initialized using the ViSP library [START_REF] Marchand | Visp for visual servoing : a generic software platform with a wide class of robot control skills[END_REF], by matching some keypoints detected in the very first image with those extracted in the training images using an approach similar to [START_REF] Choi | Real-time 3d model-based tracking using edge and keypoint features for robotic manipulation[END_REF].

Very few model-free tracking and reconstruction algorithms are publicly available along with their open source code (e.g: [START_REF] Izadi | Kinectfusion : real-time 3d reconstruction and interaction using a moving depth camera[END_REF], [START_REF] Rünz | Co-fusion : Real-time segmentation, tracking and fusion of multiple objects[END_REF]), but it is not trivial and requires significant modifications to compare reconstruction and tracking methods with model-based object tracking using RGB-D data. Hence, we compare the proposed method with two approaches: a) 'edge + keypoint + depth

2.3. Results C K i C K i+5 FIGURE 2.3 -
The photometric tracking is based on a keyframe-to-frame error minimization. This figure shows an example of keyframes being generated after every 5-th frame. The exact criteria for generating a new keyframe is given in Eqn. 2.15 tracker'from [START_REF] Trinh | A modular framework for model-based visual tracking using edge, texture and depth features[END_REF] (denoted in the figures as ViSP), and b) stacked error minimization of point-to-plane distance with photometric error, without using keyframes (denoted as No KF ). The proposed approach is denoted as PA in the figures.

The quantitative comparison of the various tracking methods1 with the ground-truth is summarized in Table 2.1. The marmalade container sequence (Fig. 2.4a) shows accurate tracking with all the approaches. The proposed approach manages to outperform the other two methods, although by a small margin. However, in the coffee machine sequence, the proposed approach outperforms ViSP significantly. As shown in frame 275 of Fig. 2.4b, ViSP shows a noticeable drift while tracking a set of coplanar faces with not enough image features in it. This is the only instance among all the sequences tested across all the available methods, where the tracking resulted in a visually noticeable drift. No KF alone, does not solve the issue of the drift completely, but the proposed approach eliminates the visible drift and the positional tracking is 83.09% better than that of ViSP. Simulated car is a bit more challenging sequence due to larger inter-frame motion towards the end. No KF shows very low accuracy in this sequence. In the rotation, ViSP outperforms the proposed approach by average of 0.51°over the entire sequence. This drift is not noticeable visually, and happens because a combination of a large number of robust feature points and edges makes it easier for ViSP to track the overall orientation of the car, while larger inter-frame motion disadvantages the photometric minimization. However, the proposed approach is more accurate than ViSP in terms of translation. Across the three sequences, it can be concluded that the proposed approach performs better than both No KF and ViSP. We show the tracking error plots of only ViSP and the proposed approach in Fig. 2.5a and Fig. 2.5b. Some additional results on realworld, complex shaped objects from the dataset of [START_REF] Choi | RGB-d object tracking : A particle filter approach on GPU[END_REF] is shown in Fig. 2.7.

For the real objects, all the object models were constructed using manual measurements of the object. The data was captured using Intel RealSense SR300. The banana model is only a rough approximation of the shape of the real banana. Both the box and the car got accurately tracked (for visual validation, refer to Fig. 2.6), despite the obvious inaccuracies in the model. The proposed approach does not get affected by moderate occlusion of the objects by the hand. There were some slippage of the model from the actual object while tracking the banana. However, it never completely looses tracking.

The proposed approach has been implemented on C++ with occasional utilization of the PCL library [START_REF] Rusu | 3D is here : Point Cloud Library (PCL)[END_REF] and the ViSP library for I/O and the IRLS implementation respectively. The simulated data has been generated using the Blender software [Blender, 2018].The proposed algorithm has been tested on an Intel Core i7-6600U CPU with 16 GB RAM. Running on a single core of the system, without SSE optimization and without using GPU in any format, the basic C++ code written for implementing the proposed approach achieves a runtime of 100 -160 ms per frame, including data capture, tracking and display on a simple GUI. It can be envisaged that with either SSE optimization or with the use of GPU, the overall algorithm can run much faster, if required. [START_REF] Trinh | A modular framework for model-based visual tracking using edge, texture and depth features[END_REF] NoKF PA [START_REF] Trinh | A modular framework for model-based visual tracking using edge, texture and depth features[END_REF] NoKF PA [START_REF] Trinh | A modular framework for model-based visual tracking using edge, texture and depth features[END_REF] NoKF PA

Results

Conclusion

In this chapter, we present an algorithm to accurately track the pose of rigid and complex-shaped objects. The tracking is robust to occlusion and partial specularity of the scene. We provide validation on both simulated and real data. The proposed approach outperforms one of the best among the open-sourced, model-based, 6DoF object tracking methods. It also outperforms a partial re-implementation of a state-ofthe-art tracking method from recent advances in the field of tracking and reconstruction. The proposed approach is an efficient method to track complex objects that a) does not require detailed object model (reducing the setup time in practical applications), b) tracks objects with better accuracy than comparable state-of-the-art approaches. FIGURE 2.7 -Tracking results on the real data of the dataset from [START_REF] Choi | RGB-d object tracking : A particle filter approach on GPU[END_REF] CHAPTER 3

DEPTH BASED NON-RIGID OBJECT TRACKING

Demonstration of the method for deformation tracking proposed in this chapter, utilized for the tracking of the deforming surface of a pizza

In the previous chapter, we discussed a method to track rigid objects using a coarse 3D model. This is meant to serve as a prior to the non-rigid object tracking methodologies discussed in the rest of this thesis. In this chapter, we present a method to track deformable objects using co-rotational Finite Element Method (FEM), by controlling the force applied on the physical object model at certain nodes of the mesh.

We describe a method to handle the problem of accurate tracking of the surface of non-rigid objects undergoing deformation. A commodity-level RGB-D camera is used for sensing. It is assumed that we know the visual-surface model of the object (which can also be a CAD model), but this model does not have to be precise. The approach proposed in this chapter tracks the deforming objects by regulating virtual forces acting on the surface of the simulation of a physically-based mechanical mesh.

The primary contribution of this chapter is a method for tracking the entire visible surface of a deforming, non-rigid object in 3D, such that:

-it requires only a very coarse estimate of the physical properties (Young's modulus and Poisson ratio) of the object ;

-it performs accurate frame-to-frame tracking of deformation of the non-rigid object ;

-it has been validated on simulated data with ground truth, as well as real data.

Background

We use the co-rotational FEM to simulate the physics behind the deformation of non-rigid objects. For simulating the FEM, we use a tetrahedral mesh as the physical model of the object. But, unlike the state-of-the-art approaches in the literature, we extend the method to minimize an error function with a closed-loop optimization based on a point-to-plane geometric error. This additional step effectively reduces the dependency of the system on the accuracy of the physical properties of object matter used for the finite element modelling. Moreover, we dissociate the rigid and non-rigid tracking into two parallel processes. The rigid tracking is purely based on the surface model, while the FEM is used strictly for tracking the non-rigid deformations. This parallel system is capable of real-time performance. Our approach has been both quantitatively and qualitatively evaluated using a combination of ground-truthed, simulated data and real data.

Having decided to utilize physically based models for deformation tracking, in this chapter we begin the study of tracking approaches based on depth information only. Since depth data is an important information available from the RGB-D sensors and can be conveniently utilized for tracking the temporal evolution of non-rigid objects, we design a methodology for tracking this depth data using the co-rotational FEM model. However, such approaches are not completely unprecedented.

As discussed in Sec. 1.3.1.1, physics based framework for deformation tracking has been demonstrated using mass-spring-damper systems [START_REF] Royer | Real-time target tracking of soft tissues in 3d ultrasound images based on robust visual information and mechanical simulation[END_REF], finite element model (FEM) [Petit et al., 2015a] and kinematic chains [START_REF] Schmidt | Dart : Dense articulated real-time tracking[END_REF]. For model based tracking of non-rigid objects, the representation of the model is usually done by a set of triangular faces representing the surface of the object. However, tetrahedral, volumetric model is preferred [START_REF] Paloc | Online multiresolution volumetric mass spring model for real time soft tissue deformation[END_REF] for tracking using mass-spring-damper systems, while the FEM based systems available in the literature FIGURE 3.1 -Overview of the approach for deformation tracking proposed in this chapter typically preferred to use a combination of surface model and volumetric model. Haouchine et. al. [START_REF] Haouchine | Impact of soft tissue heterogeneity on augmented reality for liver surgery[END_REF] proposed a linear tetrahedral co-rotational FEM based model for tracking large deformations. [Petit et al., 2015a] [Petit et al., 2018] use co-rotation FEM based model to track deformation by estimating the direction and magnitude of elastic force acting on the object using depth information. As summarized in [START_REF] Nadon | Multi-modal sensing and robotic manipulation of non-rigid objects : A survey[END_REF], many FEM based approaches in the available literature suffer from excessive dependency on the availability of the accurate physical properties of the object being tracked.

The method that we propose in this chapter removes this dependency using a novel, closed loop minimization technique. The same lack of dependency on the accuracy of the physical parameters of the model is also applicable to [Petit et al., 2015a]. However, as demonstrated in Sec. 3.4, the approach proposed here outperforms the underlying basic framework of [Petit et al., 2015a] under practical constraints. Moreover, a closedloop error minimization improves robustness to occlusion and noise.

Method

The methodology that we use for tracking deformable objects will be described in this section. We begin by describing the preliminary notations and the elastic deformation model that we use for interpreting the physics behind the objects we track. This is followed by a brief description of the rigid registration process (Sec. 3.2.3). The description of matching the deformable object between consecutive frames is discussed next (Sec. 3.2.4). We follow this up with the mechanism for computing the Jacobian that links the variation in the geometric error with the variation of force applied on a particular vertex. We use this Jacobian for minimizing the error using an Iteratively Reweighted Least Square (IRLS) formulation. We finish this section by summarizing the steps for minimizing the non-rigid error.

In the approach proposed here, first the deforming object is tracked rigidly using its 3D object model. After the rigid tracking, the remaining residual error occurs purely due to the deformation of the model. In the second step, this non-rigid error is minimized by applying a deforming force on the physical object model. The method for obtaining this deforming force that minimizes the non-rigid error is the primary contribution of this chapter and is described in Sec. 3.2.4. The overview of this approach is described in Fig. 3.1.

Notations

We use two types of 3D model for the non-rigid tracking. The first is a visual-surface model, denoted by V S , and the other one is a tetrahedral, volumetric, internal model, denoted by V M . Continuing from the definition in Sec. 1.1.3, C V S and O V S denotes the visual-surface model in the camera (F c ) and object frame (F o ) respectively while C V M and O V M denotes the internal model in a similar manner. The visual-surface model is described using a set of planes which are represented using the nodes defining the boundary of the surface (usually a triangle) and the connectivity between these nodes. We can represent this surface as

V S = p 1 p 2 ... p M (3.1)
where M is the number of vertices in V S and the i-th vertex is p i = X i , Y i , Z i , the 3D coordinate of a point in the reference frame of the base model. As given in Sec. 1.1.3, we have a corresponding, connectivity map, given by X V S . M is the number of vertices and N is the number of triangular faces in V S . The 3D coordinates of the three vertices of the triangle representing a given face can be derived from X V S . Clearly, the normal vector n i = (n X i , n Y i , n Z i ) and distance to origin d i for any given face can be derived from V S and X V S using elementary geometry. We similarly define a tetrahedral, volumetric, internal model V M along with its connectivity map X V M , such that every element of X V M has four indices, instead of three. As shown in Fig. 3

.2, V M represents F c O V M O V S O V C V F c F c F o C T O FIGURE 3.2 -
set { O V, X V }, where O V = O V S O V M and X V = {X V S , X V M }.
The reference frame used for this setup of non-rigid tracking is given in Fig. 3.2. C T O denotes the transformation from the camera-centered coordinate frame F C to the object in the pointcloud. For rigid tracking, we define a tracked model using the tuple

{ C V, X V }, such that: O V = O R C C V + O t C • • • O t C 3×M (3.2)
During non-rigid deformation of the base model ( O V) centered at F C , the transformation of (3.2) is repeated, so that the model at the object centered reference frame always stays updated according to the deformation.

Deformation Modelling

We use co-rotational FEM as the deformation model, for developing the deformation tracking method proposed in this chapter. As explained in Sec. 1.3.1, co-rotational FEM offers us a method to model large deformations efficiently.

We can now briefly recall some of the relevant concepts used for simulating the deformation using FEM. For a tetrahedral mesh, we can represent the internal elastic forces acting on the nodes by: gives the elastic forces acting on the nodes of this tetrahedral element.

F e = R e K e ŨR (
To resolve the interaction between forces and their resulting displacement using the FEM, we solve a second order differential equation, given by:

Mp + D ṗ + K g p = F ext + F e (3.4)
where M and D are the model's mass and damping matrices respectively, derived using Eqn. 1.66 and Eqn. 1.67. K g is the global stiffness matrix. F ext are the external forces acting on the vertices. The method for determining F ext is discussed in the subsequent sections. To solve this differential equation, we use a linear solver based on conjugate gradient descent [START_REF] Faure | Sofa : A multi-model framework for interactive physical simulation[END_REF]. To impose additional constraint, Eqn.

(3.4) can be multiplied using a projection matrix that sets the values of certain indices to zero. We use the projective constraint to eliminate the rigid motion of the object. This is discussed in the next section It must be noted that the force F ext is just the deforming force acting on the model, and does not include the forces causing the rigid transformation of the body. In fact, the deformation model proposed here is completely independent of rigid motion that results from the effects of gravity and interaction with other contact surfaces. As described in Sec. 3.2.3, the rigid motion is tracked separately. This separation of rigid and non-rigid tracking method, along with the minimization described Sec. 3.2.4, makes the overall system independent from the inaccuracies of the physical parameters.

Rigid Registration

The rigid registration is a joint minimization of two error terms: depth based geometric error and keypoint based feature tracking [START_REF] Trinh | A modular framework for model-based visual tracking using edge, texture and depth features[END_REF]. The two error terms are explained below:

Depth based geometric error

Assuming that we know the accurate C T n-1 at the (n-1)-the frame and n T n-1 gives the initial estimate of transformation between previous and current frame, the error is given as:

e D ( n q n-1 ) = n R n-1 P + n t n-1 • n k -d k (3.5)
where n k and d k are the normal and distance to origin respectively, for the k-th planar face that corresponds with the point P in the pointcloud. The Jacobian that links the variation of the error e D ( n q n-1 ) with the variation of n qn-1 , is given by:

J D i = n k n k × P i (3.6)

Feature based minimization

The Harris corner features are used for tracking with keypoints. This is based on the classical KLT algorithm [START_REF] Baker | Lucas-kanade 20 years on : A unifying framework[END_REF]. Let u = (x, y, 1) be the homogeneous 2D coordinate of a feature point in the (n -1)-th image and u * = (x * , y * , 1) be the matched coordinate for the same point in the n-th frame. We define the error term as:

e K ( n q n-1 ) =   x( n q n-1 ) -x * y( n q n-1 ) -y *   (3.7) where (x( n q n-1 ), y( n q n-1 ), 1) = n H n-1 u (3.8)
Here, u = (x, y, 1) and u * = (x * , y * , 1) are the homogeneous pixel coordinates of the tracked feature point in the (n -1)-th and n-th data frame respectively. Moreover:

n H n-1 = n R n-1 + n t n-1 d n (3.9)
where n and d are interpreted in the same way as (3.5). The corresponding Jacobian that relates the variation of e K ( n q n-1 ) with the time variation of n q n-1 is given by:

J K =   -1 Z 0 x Z xy -(1 + x 2 ) y 0 -1 Z y Z (1 + y 2 ) -xy -x   (3.10)
where (x, y) are the pixel coordinates and Z is in meters. Given these two errors and their corresponding Jacobian, the combined error e( n q n-1 ) is obtained by stacking the vectors e D ( n q n-1 ) and e K ( n q n-1 ), while the combined Jacobian J is given by J = (J D , J K ). The combined error is minimized with the update given by:

ξ = -λ WJ + We (3.11)
where W is the weight matrix for outlier removal using the Tukey m-estimator [START_REF] Beaton | The fitting of power series, meaning polynomials, illustrated on band-spectroscopic data[END_REF], and ξ ∈ se(3). This becomes an iteratively re-weighted least squares problem.

Non Rigid Tracking

Before starting the non-rigid tracking using a minimization technique, it is necessary to determine the possible points of application of forces that needs to be tracked. It is sufficient to track only a small subset of nodes, depending upon a set of criteria. These criteria, as detailed below, is obtained by clustering the residual error from the rigid tracking step. Only regions surrounding the clusters with higher value of errors are considered to be relevant for non-rigid tracking. It must be noted that only the point-toplane error based depth information is used for determining the point of application of forces.

At the end of the rigid registration process, we can construct a map that links the points in the 3D pointcloud to their corresponding geometric error from (3.5). Given that e D i denotes the error for the i-th point P i in the entire pointcloud P, it is possible to derive a new pointcloud P * using a linear thresholding operation, such that:

P * = {P i ∈ P |e D i | ≥ θ D } (3.12)
where θ D is a threshold that depends on the geometry of the object being tracked.

We subject the pointcloud P * to a clustering step using Eucledian distances [START_REF] Hartigan | Algorithm as 136 : A k-means clustering algorithm[END_REF]. Very small clusters and clusters spanning more than half of the size of the entire pointcloud are discarded. Let us assume that we obtain j clusters from this operation, denoted by

K 1 , K 2 , • • •, K j ,
k * i = C T O -1 k i (3.13)
For all these j centroids, we determine their nearest neighbors in C V M using the knearest neighbor algorithm [START_REF] Fukunage | A branch and bound algorithm for computing k-nearest neighbors[END_REF]. Let the nearest neighbors

of k * 1 , k * 2 , • • •, k * j in C V M be denoted by C V M k 1 , C V M k 2 , • • •, C V M k j .
For the following discussion, let us consider an arbitrary k i -th cluster alone, which we refer to as C V M i for the sake of simplicity. At this stage, we must modify (3.5) to accommodate a slightly different variant of the same error function. We re-define it as e D C V S i , assuming the r-th plane corresponds to the s-th point in the pointcloud, and the normal n r and the distance to origin d r is derived from C V S i . This change in notation is necessary because the error term e D is no longer a function of n q n-1 at this point of non-rigid tracking, it is rather a function of C V S i , i.e., the position of the surface model of the object w.r.t the s-th point in the pointcloud. The correspondence between the point and the plane is done using a strategy similar to Sec. 2.2.2. The propagation of the vertex displacement from C V M i to C V S i happens through the barycentric mapping J (defined in Sec. 1.3.1).

Jacobian Computation

The Jacobian that relates the variation of e D C V S i with the variation of the applied force is computed numerically by perturbing the node O V M i by a very small, constant force ∆F J successively along the three axes, as shown in Fig. 3.3. After applying the force on the node, a simulation based on FEM is done using the minimization described in Sec. 3.2.2 -Eqn. 3.4. After the conjugate gradient solver optimizes (3.4), the system will attain a static condition. There will be six such configurations of the mesh, obtained by the perturbation along the positive and the negative direction of the three axes. We denote these new six configurations of O V M i by using the axis direction X+, X-,

• • • , Z- FIGURE 3.
3 -The node closest to the centroid of a cluster is perturbed by a small force along the three axes in both positive and negative direction, producing six deformed configurations per node as a superscript, given as:

( O V M i ) F = ( O V M i ) X+ ( O V M i ) X-( O V M i ) Y + ( O V M i ) Y -( O V M i ) Z+ ( O V M i ) Z- (3.14)
Following the definition of Sec. 1.3.1, the deformed meshes from Eqn. 3.14 is propagated down to the visual-surface model via the barycentric map:

( O V S i ) F = J ( O V M i ) F (3.15)
We take these modified, visual mesh and transform it back to the registered pointcloud using:

( C V S i ) F = C R O ( O V S i ) F + C t O • • • C t O 3×6M (3.16)
where M is the number of vertices of V S . The relation of the variation of the external force with the variation in error can be expressed as:

ėD C V S i = J i Ḟext (3.17)
The term J i is obtained numerically by finite difference computation using central differences. The final Jacobian used is:

J i = J X i J Y i J Z i (3.18) ⇒ J i = ( O V M i ) X+ -( O V M i ) X- 2F J ( O V M i ) Y + -( O V M i ) Y - 2F J ( O V M i ) Z+ -( O V M i ) Z- 2F J (3.19)

Minimization

So far, j nodes have been selected for application of external force to deform the model. We have p points in the pointcloud, and their correspondence with the model is known. If the iteration of minimization is denoted by n, the initial estimate of the vertically stacked force vector at the (n -1)-th iteration is denoted by F n-1 ext 3j×1 and the values are set to zero. The Jacobian matrices are stacked up horizontally, such that:

J p×3j = J 1 J 2 • • • J j (3.20)
The update is computed as:

∆F = -λ W p×p J + We D C V S i p×1 (3.21)
W is a weighting matrix obtained from the Tukey based m-estimator, similar to Sec. 3.2.3. λ is a scaling factor. The force vector is updated by:

F n ext = F n-1 ext + ∆F (3.22)
This force is applied on the base model and the final node displacements are determined, once again, using (3.4).

Implementation

In order to compute the cost function, it is necessary to associate every 3D point P i with one of the planar surfaces of the visual-surface model O V S . Given that we know the value of C T O obtained at the previous frame, we project the 3D points O V S for all visible planes into the image obtained in the current frame. The visibility is checked using the classical ray-casting algorithm [Roth, 1982]. The ray-casting is also needed for imposing the projective constraints. The nodes invisible to the camera are considered to be immobile under the influence of external force. Moreover, we do not consider the volumetric, internal mesh to be an input to the system. The mesh is rather generated apriori, using Dirichlet tessellation, followed by Delaunay tetrahedralization of the input mesh using the Bowyer-Watson algorithm [Si, 2015].

In contrast to [Petit et al., 2015a] and [START_REF] Petit | Capturing deformations of interacting non-rigid objects using rgb-d data[END_REF], the approach proposed here does not require a visual segmentation for separating the region of interest from the background. This is done by initializing the pose of the object at the first frame using pre-trained markers on the object. It is done using the ViSP library [START_REF] Marchand | Visp for visual servoing : a generic software platform with a wide class of robot control skills[END_REF], by matching the keypoints detected in the very first image with those extracted in the training images using an approach similar to [START_REF] Dementhon | Model-based object pose in 25 lines of code[END_REF].

The algorithm is implemented using a parallel framework, where two different processes are involved for the rigid and non-rigid tracking of the object respectively, as shown in Figure 3.4. The rigid tracking process is capable of processing each frame at < 100 ms, thereby ensuring real-time interaction. The un-optimized, non-rigid tracking code runs at 800ms to 1.3s (approximately) per frame, depending on the size of the object and its proximity to the camera. The results reported here have been achieved using processes running on a single core of a i7-6600U CPU with 16GB of RAM. An Intel Skylake GT2 GPU has been utilized, but only for the ray-casting. The FEM solvers are implemented using the SOFA library [START_REF] Faure | Sofa : A multi-model framework for interactive physical simulation[END_REF], without using CUDA. The rigid tracking process runs at > 10 fps, while the non-rigid tracking process handles every 10-th/12-th frame coming from the sensor.

Results

The results are quantitatively validated on two sets of simulated data, as well as on multiple real objects 1 . FIGURE 3.4 -The parallel implementation of the rigid and non-rigid tracking processes enable us to track the deforming surface at frame rate

Simulation

For the simulated data, the deformation of the objects were generated using simulation of co-rotational FEM. Two objects are considered, a cube and a rectangular board. They are made to undergo simple but large deformations, as shown in Fig. 3.5. We made the dataset available at: github.com/lagadic/nr-dataset. The visual models produced as an output of the simulation were subjected to texturing, shading and rendering using the Blender software, followed by the generation of the pointcloud using a RGB-D camera simulator. In the simulation, both the objects were modelled using Young's modulus (YM) of 50000 Pa and Poisson's ratio of 0.3. Rayleigh mass was assumed to be 0.1 and Rayleigh stiffness to be 0.3.

Comparison

We utilize the simulated dataset not only to validate our approach, but also to compare the proposed method to existing paradigm for deformation tracking from the stateof-the-art literature. As a comparison, it must be noted that merely idea of applying controlled forces to mesh vertices for deformation tracking is not entirely novel. As noted in Sec. 1.3.1.1, similar ideas have been proposed by [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], [Wang et al., 2015] and [Petit et al., 2015a]. The novelty of the approach proposed by us lies in FIGURE 3.5 -The undeformed model of the board and the cube. The arrows in the image shows the approximate direction of application of force the implementation of a geometric error minimization mechanism using a numeric optimization scheme. This introduces a novel paradigm in non-rigid object tracking using volumetric, physically-based meshes.

In this context, it becomes necessary to compare our approach with the fundamental deformation tracking mechanism of [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF], [Wang et al., 2015] and [Petit et al., 2015a]. The source code or results on a benchmarked dataset is not available from any of these authors. Hence, we design a minimalistic strategy to obtain a comparable tracking mechanism, as explained below.

[ [START_REF] Schulman | Tracking deformable objects with point clouds[END_REF] uses a standard FEM as the deformation model while [Wang et al., 2015] and [Petit et al., 2015a] uses a co-rotational FEM. We restrict our comparison to the co-rotational model only, since it provides a more evolved formulation over the standard FEM. The external, elastic forces, f ext , acting on the object is given by:

f ext,i = k ext,i (x i -y i ) (3.23)
where k ext is the stiffness corresponding to the external elastic force and y i is a point in the pointcloud which has been matched to x i , a vertex on the mesh. Eqn. 3.23 forms the default, per-vertex force estimation method for both [Wang et al., 2015] (Eqn. 1.71) and [Petit et al., 2015a] (Eqn. 4.29). We re-implement this force computation, whereas the per vertex displacement vector (x iy i ) is obtained from the ground-truth in the simulation. The force is applied on the vertices selected by clustering and the FEM simulation is allowed to resolve the mesh displacements over the entire object model for a fixed number of iterations. The number of iterations allowed to converge is constant for both this re-implementation and the approach proposed by us. This implies that in the re-implementation, the displacement vector associated with the deformation is provided exactly from the groundtruth, providing it with a significant advantage over the approach proposed by us. In general, the fundamental difference between the method proposed in this chapter and the approach of [Petit et al., 2015a] lies in the fact that we propose an optimization based method for deformation tracking while [Petit et al., 2015a] relies on force-estimation from pointcloud matching using a modified k-nearest neighbor search.

In the subsequent text, the results obtained from the approach proposed in this chapter are denoted by Numeric-optimization, while the results from this partial reimplementation scheme (as discussed above) are denoted by Linear-force-estimate.

To analyze the robustness of the two approaches, experiments were repeated by varying the Young's modulus of the model for determining the variance in error with change in physical parameters. The physical parameters used for simulation are considered unknown in the implementation of the proposed approach. The tracking was repeated over the values of 5 Pa, 500 Pa, 50000 Pa, 5 × 10 6 Pa and 5 × 10 8 Pa respectively. The results are summarized in Fig. 3.6 and Fig. 3.7. In the figure, the ground-truth of the model undergoing deformation is provided in the left-most column. The rest of the images show the 3D model (the tracked model) obtained from the tracking algorithm, placed into the pointcloud with the latest estimate of C T O .

Since the ground-truth for the deformation is known, it is possible to compute the Hausdorff distance metric [Henrikson, 1999] between the output of the tracking and the ground-truth, to quantify the error between the ideal and the actual output. Fig. 3.8 delineates the Hausdorff distance of the output for various values of Young's modulus.

As given in Table 3.1, the mean error in terms of Hausdorff distance using Numericoptimization and Linear-force-estimate are 0.305 and 1.622 units for the cube dataset and 0.834 and 1.100 units for the board dataset respectively. As a reference, the length of the largest diagonal is 69.28 and 55.19 units for the cube and the board models respectively (the data is simulated using geometrically defined values for the dimension of the objects and the FEM is simulated without applying the gravity vector, hence the exact unit of length cannot be expressed in SI units directly). Hence, the tracking accuracy of the proposed approach is 81.18% better than Linear-force-estimate in the cube data and 24.20% better than Linear-force-estimate in the board data. The tracking accuracy of the proposed approach in terms of Hausdorff distance varies only by an average of 5.006% when the Young's modulus is increased by a factor of 10 8 . It can therefore be claimed that the system is significantly robust to error in the estimation of the physical properties used to model the deforming object. It must be noted that the results shown for Linear-force-estimate contains only the error in estimating the magnitude of deformation of the objects. As stated before, the direction of application of force is provided to Linear-force-estimate from the ground-truth in our re-implementation (while Numeric-optimization makes no such exception), which explains the impressive accuracy of Linear-force-estimate when using Young's modulus equal to the ground-truth value for simulation (50000 Pa).

Real Data

The real data has been captured using the Intel RealSense SR300 RGB-D camera. It is a commodity level depth sensor that produces RGB-D images of reasonably good resolution between 20 to 150 cm depth. Three objects have been tracked for the validation: a) a pizza, b) a cuboidal soft toy and c) a rectangular sponge. A Poisson's ratio of 0.3 was assigned to all the objects uniformly. All the objects were tracked using multiple values of Young's modulus which were set coarsely and empirically. The pizza data was tracked with an Young's modulus of 5 × 10 3 Pa, 5 × 10 4 Pa and 5 × 10 5 Pa, the toy was tracked using 5 × 10 4 Pa, 5 × 10 5 Pa and 5 × 10 6 Pa, and the sponge was tracked using 8 × 10 5 Pa, 8 × 10 6 Pa and 8 × 10 7 Pa. The tracking results are shown in Fig. 3.9, Fig. 3.10 and Fig. 3.11.

Given these results, the average, per pixel point-to-plane error across the entire sequence for multiple values of Young's modulus are given in Fig. 3.12.

Conclusion

We have presented an algorithm for tracking deforming objects using RGB-D camera with the help of physically-based models. The deformation tracking approach is not strongly dependant on the accuracy of the estimate of the physical parameters. A minimization of depth based error term enables us to track the deforming surface accurately. The algorithm can be enabled to track objects in real-time with the help of a parallel architecture. The approach has been qualitatively validated on simulated data, as well as tested on real objects with unknown physical parameters.

However, it is obvious that the depth data provided by the RGB-D sensor is only a fraction of the entire information available to us. It is not prudent to discard the photometric information entirely. Tracking based on additional information can increase the accuracy of the system. Moreover, it is also obvious that the computation of Jacobian by the numerical estimation scheme via repetitive simulation of deformation is extremely expensive in terms of time requirement. Any possible mechanism to compute the Jacobian analytically would greatly improve the per-frame time required for tracking. These two concerns are addressed in the following chapter, where we extend our deformation tracking with the addition of photometric information as well. We also propose a mechanism to approximate the Jacobian analytically, thereby resulting in significantly reduced time requirements.

NON-RIGID OBJECT TRACKING WITH DEPTH AND PHOTOMETRY

A method for model based tracking of deformation and deforming forces is presented in this chapter: (A) the deformation is captured using a RGB-D sensor, (B) a combination of geometric and photometric errors are minimized, (C) the minimization is done using FEM, (D) the deformed mesh is the default output of the system, while the force applied can also be estimated if the material properties of the object are known

Introduction

In the previous chapter, we presented a method to iteratively minimize depth based geometric error using FEM. The method has been utilized to accurately track both simulated and real deforming objects. However, in the formulation presented in the previous chapter, there are two aspects which can be improved.

The first concern arises from computing the gradient of error using repeated simulation of mesh deformation with the FEM model. It requires a significant amount of time. We managed to extricate fast runtime from the system by utilizing a parallelization scheme, but this type of implementation tracks the deformation while skipping certain frames, because the rigid tracking runs significantly faster than the non-rigid tracking method. It is not trivial to modify this algorithm into a sequentially real-time system. It is not possible to optimize ∼ 20 -30 iterations of complete FEM simulation, each containing ∼ 40 -60 iterations of the conjugate gradient solver into a real-time system. However, skipping intermediate frames is also not desirable.

The second aspect which merits closer inspection is the fact that purely depth based object tracking (especially using a point-to-plane distance based cost function) has scope for improvement. In Chapter 2, we already noticed that combining photometric error minimization with depth based error lead to better accuracy in the case of rigid tracking. It can be interpreted intuitively that addition of more error terms into the minimization loop of Chapter 3 should usually lead to better accuracy in the case of non-rigid object tracking too. The availability of images from the RGB sensor of the depth camera leads us to conclude that a direct photometric intensity based error term is an obvious choice for this additional error.

In this chapter, we address these concerns arising out of Chapter 3. A framework is proposed to track non-rigid objects using RGB-D cameras by utilizing physicallybased models. The physics of the model is similar to that proposed in Chapter 3, i.e., co-rotational FEM. The methodology proposed here integrates classical computer vision based error minimization using photometric and geometric cost functions with the deformation simulation capabilities of co-rotational FEM.

The main contributions of the methodology proposed in this chapter are:

-Combined minimization of error terms based on depth and photometric information ;

-A new technique for approximating the gradient of these errors w.r.t displacement of vertices ;

-An improved methodology for determining the vertices which are to be controlled for tracking ;

-Improved runtime

Background

This chapter combines classical computer vision with geometric analysis of deforming manifolds and physics-based structural models from mechanical engineering.

Tracking deformation using RGB-D cameras is an advanced area of research in computer vision. However, purely computer vision based algorithm for deformation tracking does not provide any real world information about the nature of physical interactions that the object is undergoing. The objective of this chapter is to treat the deforming object from a physics based perspective, such that both the deformation and the interaction causing the deformation can be accounted for.

Deformation has been tracked in real-time using mass-spring-damper models, as well as linear Finite Element Models (FEM) by either computing a point-to-point correspondence [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] for estimating the deformation of the surface, or by using a steepest gradient descent based approach [START_REF] Royer | Real-time target tracking of soft tissues in 3d ultrasound images based on robust visual information and mechanical simulation[END_REF] to minimize a SSD/SSE type cost function. FEM is being extensively used to solve mechanical engineering problems involving complex elasticity and structural analysis for the last eight decades [Hrennikoff, 1941] [Courant et al., 1943] or more. Consequently, FEM offers us a wide variety of highly evolved mathematical tools to model mechanical properties of real objects. A model based minimization strategy to track deformation has been proposed in in the previous chapter, but it cannot be considered real-time. Tracking of topological changes such as cutting and tearing has also been demonstrated using simple feature correspondences [START_REF] Paulus | Augmented reality during cutting and tearing of deformable objects[END_REF]]. An approach has been proposed to combine NRSfM with FEM by representing the deformation as a linear combination of modal shapes [START_REF] Agudo | Good vibrations : A modal analysis approach for sequential non-rigid structure from motion[END_REF], but the method has been validated only on laminar models.

Interesting use-cases, such as estimation of material properties, begin to appear when tracking vibrations of objects using computer vision [START_REF] Davis | Visual vibrometry : Estimating material properties from small motion in video[END_REF], but these approaches have not been developed for deformation tracking of volumetric objects. External force sensing by vision has been proposed by [START_REF] Zhang | Calibration and external force sensing for soft robots using an rgb-d camera[END_REF], but the deformation tracking method proposed here is a variant of feature matching, which is partially similar to [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF], and hence can be expected to suffer all the drawbacks of point matching based object tracking approaches. The approach proposed in this chapter has been compared against point matching based approaches for deformation tracking (see Section 4.4) and has been demonstrated to outperform the state-of-the-art for these types of physically based tracking algorithms.

It is evident that there is a clear and strong separation between the evolved computer vision techniques used in physical model free deformation tracking, and the relatively simpler techniques used while tracking using physics based models. In this chapter, we propose to bridge this divide between vision based tracking and mechanical models by enabling an iterative minimization strategy for generic, vision-based cost functions using FEM. The method described here involves the minimization of a combination of a geometric and a photometric cost functions. A textureless, coarse model is required as the input of the system while the output consists of the tracked 3D model. Moreover, the availability of an advanced physically based model enables us to track the forces acting on the deformed object, whenever an approximate estimate of the material properties are available.

Proposed Approach

The input to the system consists of RGB-D data stream from a depth camera and a coarse 3D model of the surface of the object being tracked. Unlike similar methods in the literature, we do not require the material properties of the object being tracked or the textured model of the object. For each frame, the first step is to approximately track the rigid pose of the object w.r.t the camera (described in Section 4.3.3). This is followed by determining the approximate areas where the error in rigid registration has a higher magnitude (see Section 4.3.2). The mesh vertices lying close to these areas with high rigid registration error are termed as control handles. These control handles are utilized to minimize a set of error terms (explained in Section 4.3.1), which are defined w.r.t the surface model of the object.

Non-rigid Tracking

We briefly recap our motivation behind proposing the non-rigid object tracking method of this chapter before explaining the technical details.

Motivation

The procedure for real-time, RGB-D based tracking of non-rigid objects without using mechanical model usually involves the iterative minimization of one or multiple highly non-linear objective function, such as point-to-point distance [START_REF] Wang | Templateless non-rigid reconstruction and motion tracking with a single rgb-d camera[END_REF], point-to-plane distance [START_REF] Newcombe | Dynamicfusion : Reconstruction and tracking of non-rigid scenes in real-time[END_REF], combination of point-to-plane distance and sparse feature alignment error [START_REF] Innmann | Volumedeform : Real-time volumetric non-rigid reconstruction[END_REF], etc. However, in the absence of a physics based framework for modelling the real-world properties of the deforming object, a regularizing or smoothing data term is used to prevent unnatural deformations while tracking the object. For tracking using physics based models, the problem statement becomes much more constrained due to the presence of a model that already mimics the physical properties of the actual object. The tracking methodology changes from an iterative minimization to displacing one or multiple vertex/vertices to follow the object deformation, since estimating the gradient of the non-linear error term with respect to the vertex displacement involves time-consuming deformation simulation, which is not suitable for online tracking. However, we have demonstrated in Chapter 3 that iterative optimization of depth based error term is possible using physically-based model and results in accurate tracking. Through the mechanism proposed in this chapter for tracking non-rigid objects, we aim to bridge this divide between the physical-model free and physical-model based approaches even further. This is done by the minimization of photometric error term along with depth based error while the gradient of error in approximated analytically. We first describe the overall framework for generic objective functions. The specific error term is described thereafter.

Methodology

We now describe our proposed methodology for tracking non-rigid objects. Let us denote the non-linear error term that we seek to minimize by E. We first discuss the procedure that we follow to minimize this error using the physically based model, i.e., co-rotational FEM. Throughout the rest of the chapter, the notation P M is used to denote the vertices of the mechanical tetrahedral mesh V M , while P = (P X , P Y , P Z ) denotes the vertices of the surface model V S w.r.t the object centered reference frame F o . The surface model consists of triangular faces and the j-th face is denoted by:

V j 3×3 = P i P i+1 P i+2 (4.1)
starting from the i-th vertex of V S . The mapping V S → V M is done via a standard master-slave configuration using barycentric mapping. The RGB-D camera is assumed to be at a reference frame F c . For the sake of brevity, the baseline between the depth and the color camera is not mentioned explicitly and all data presented here are assumed to be aligned with the color camera.

Let us assume that the method chosen for the deformation tracking involves a physical model P that describes the interaction between the applied forces and displacement of the vertices of V M . Recall that V M is coupled with the surface geometry V S of the object. We propose to track the deformation of this object using the minimization of one or multiple error term(s) defined strictly on the triangular elements of the faces of V S . This is done by defining a set of control handles C on V M , such that the entire deformation becomes optimizable by regulating the displacement of C. Note that any displacement of C gets propagated to all vertices of V M using the properties of P. With these constraints in place, let us assume that we are trying to optimize for a set of N objective functions E 1 , E 2 , ..., E N , defined on V S . We are interested in minimizing

E S = (E 1 , E 2 , • • •, E N ) (4.2)
with the help of the displacement vector on C, given by u C . However, to optimize in a non-linear least squares fashion, we need to obtain the gradient of the stacked/combined cost function, which can be done numerically. But with any modern physical model P, numeric estimation can be computationally expensive as determining the node displacements typically involves multiple iteration of conjugate gradient descent for solving an equation of the nature of (3.4). This is beyond the reasonable limits of performance optimization and, to the best of our knowledge, online method for deformation tracking using minimization of visual error term(s) w.r.t node displacements of a mechanical model is unavailable in the literature. We propose an assumption to solve this problem, as described below.

Let us analyze the case of an arbitrary point p of the RGB-D pointcloud, which lies on the triangle P i P i+1 P i+2 of V S . Let us assume that a small displacement:

∆u C = ∆u C X ∆u C Y ∆u C Z (4.3)
produces the displacement ∆x, ∆y and ∆z on the j-th surface plane of V S comprising of three vertices P i , P i+1 and P i+2 respectively. This vector of vertex positions is given by:

ϑ j = P i P i+1 P i+2 (4.4)
Subsequently, the gradient of the error E S (p) w.r.t u C is given by the Jacobian:

J S (p) = ∂E S (p) ∂u C = ∂E S (p) ∂ϑ j 1×9 ∂ϑ j ∂u C 9×3 (4.5)
Estimating ∂E S (p) ∂ϑ j is usually a straightforward modification of a series of well established techniques in computer vision [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. The details of how this can be done in our case is demonstrated in section 4.3.4. On the other hand, the term ∂ϑ j ∂u C is somewhat similar to the classical strain-displacement matrix, as expressed in conventional FEM literature. As stated before, determining the exact value of ∂ϑ j ∂u C for any sufficiently advanced choice of P will require computationally expensive simulation of model deformation at every iteration of the non-linear least squares solver.

We propose that this additional complexity can be avoided by estimating the deformation of the mechanical model V M , for every new control handle explored by our proposed approach, only once. For the j-th control handle C j , this is done by displacing the vertex in V M corresponding to C j by a small distance along the positive direction of X, Y and Z axis. The magnitude of this deformation ∆u C is empirically determined such that it slightly exceeds the RMSE of the sensor noise (e.g: we use Intel RealSense D435 for capturing the real data, and the magnitude of ∆u C was set to 2 mm, which successfully overwhelmed the sensor noise [START_REF] Ahn | Analysis and noise modeling of the intel realsense d435 for mobile robots[END_REF]). This estimation step produce three deformed meshes per control handle, which can be directly utilized for determining ∂ϑ j ∂u C using forward finite differences. Assuming that u C points to the c-th vertex of V M , the values of ∂P i ∂u C can be mapped to a matrix Γ O by a mapping function C such that:

Γ O = C ∂P i ∂u C (4.6)
given that:

∂ϑ j ∂u C = ∂P i ∂u C , ∂P i+1 ∂u C , ∂P i+2 ∂u C (4.7)
Here, C represents the mapping ∂P i ∂u C → Γ O and it is a one-to-one, bijective map as long as the number of vertices in V M and V S are equal. Assuming that M denotes the

∆u C X → ∆u C Y → ∆u C Z ∂(Pi) X ∂u C X ∂(Pi) Y ∂u C X ∂(Pi) Z ∂u C X ∂(Pi) X ∂u C Y ∂(Pi) Y ∂u C Y ∂(Pi) Z ∂u C Y ∂(Pi) X ∂u C Z ∂(Pi) Y ∂u C Z ∂(Pi) Z ∂u C Z FIGURE 4
.1 -This figure shows the deformational displacement ∆u C being applied to the c-th vertex of the mechanical mesh. The effect of this displacement on the i-th vertex gets stored in Γ O . This procedure is repeated for all c, i ∈ M number of vertices in V M and V S , C can be expressed as:

Γ O = C ∂P i ∂u C = M i=0 M c=0 ∂P i ∂u C ⊗ J i,c M×M (4.8)
where ⊗ gives the Kronecker product and J i,c M×M denotes the conventional single-valued matrix of dimension [M × M], such that:

J i,c M×M x,y =      1 if x = i ∧ y = c, ∀x, y ∈ M 0 else (4.9)
The construction of Γ O is demonstrated graphically in Fig. 4.1. Based on the matrix indices, Γ O can be represented as:

Γ O aM+i,bM+c = ∂P i ∂u C a,b | ∀i, c ∈ M ∧ a, b ∈ {0, 1, 2} (4.10)
In the expanded form, Γ O for the i-th vertex P i under the effect of the displacement vector u C on the c-th vertex can be expressed as:

Γ O 3M×3M = P 1 • • • P c • • • P M P 1 • • • P c • • • P M P 1 • • • P c • • • P M                                                                               • • • • • • • • • • • • • • • P • • • • • • • • • • • • • • • • . . . • • ∂(Pi) X ∂u C X • • • • ∂(Pi) X ∂u C Y • • • • ∂(Pi) X ∂u C Z
represent a modified matrix R Γ representing the rotation such that:

R Γ 3M×3M = C R O ⊗ 1 M×M =               diag(r 11 ) M diag(r 12 ) M diag(r 13 ) M diag(r 21 ) M diag(r 22 ) M diag(r 23 ) M diag(r 31 ) M diag(r 32 ) M diag(r 33 ) M               (4.12)
where:

diag(r x ) M =      r x . . . r x      M×M (4.13)
and 1 M×M is an identity matrix of dimension [M × M]. This allows us to rotate Γ O (obtained from offline computation) to the current camera reference frame. This transformed matrix Γ C is given by:

Γ C = R Γ Γ O (4.14)
We then derive the matrix Γ by thresholding Γ C such that its value at an arbitrary index m, n associated with the i-th vertex P i is given by:

Γ m,n =      1 if ε > 3α||∆u C || 2 0 else (4.15)
given that:

ε = 2 a=0 2 b=0 Γ C m+a,n+b 2 (4.16)
where ∆u C is the calibrating deformation (implying that ||∆u C || is a constant for a particular Γ O ). α is a tunable parameter which regulates the area for which the value of Γ will be 1.

At this point, it is natural to question the necessity of thresholding the value of Γ C . using Eqn. (4.15). This thresholding is done purely because we intend to utilize the point-to-plane distance as an error term in the minimization, as explained in Sec. 4.3.4. Point-to-plane distance minimization using planar (or nearly planar) surface model re- sults in the model 'sliding' over the 3D data it aims to minimize [START_REF] Chen | Object modeling by registration of multiple range images[END_REF]. This makes point-to-plane distance minimization ineffective for tracking planar surfaces. However, this is exactly what happens if we directly use the value of Γ C . The calibrating deformation ∆u C tends to exaggerate the deformation close to the vertex index c, i.e., Γ C highly prioritizes vertices close to c. This, in turn, converts the Jacobian in Eqn. (4.5) to effectively consider only a small patch of surface near the vicinity of the c-th vertex while neglecting the small magnitude of deformation observed in the vertices farther away from the c-th vertex. To overcome this problem, we employ this trick of binarizing Γ C . The parameter α in Eqn. (4.15) is used to expand the area of influence of Γ C artificially, such that it covers more orthonormal (or nearly orthonormal) faces, thereby eliminating the problem of drift. As explained in Sec. 4.3.4.1, this manner of approximation for the value of Eqn. (4.5) does not affect the minimization of error in a significant manner.

We term this matrix Γ, after the threshold has been applied, as the influence matrix, and the value is kept constant (for a given control handle), once it has been determined. It has been demonstrated experimentally that the assumption of keeping the influence matrix constant after the initial calibration results in negligible loss of accuracy in tracking (see Table 4.1) while providing significant improvement in the time required to handle each data frame (see Table 4.2). Intuitively, these values after applying the threshold in the influence matrix merely determines the faces of V S for which the error term E S will be minimized, as shown in figure 4.2. Replacing the matrix elements with 1 turns M into a pseudo-rigid object around the vicinity of u C . As long as the interframe deformation remains small, this assumption remains valid and does not affect the outcome of the minimization in a significant manner.

Determining the Control Handles

The determination of the control handles can also be handled in a generic way. Given the set of N objective functions of E S acting on L different points of the pointcloud, such that p = p 1 p 2 • • • p L . We define a function I En Π(p L ) which represents the error value using the n-th cost function E n (p l ) projected on the image. The combined error matrix I E S Π(p) on the image plane is obtained by:

I E S Π(p) = I E 1 Π(p) • • • I E N Π(p) (4.17)
where denotes the Hadamard product and • gives the normalized matrix. I E S Π(p) is clustered into multiple clusters, and the centroid of each cluster is associated with the nearest vertex Π(r), the projection of the visible vertices of V M on the image plane. These associated vertices of V M are identified as the control handles for that particular frame. This approach of determining the point of interest is highly effective, since it treats the determination of control handles as a function of the objective functions in use.

Approximate Rigid Tracking

Before commencing the minimization of the objective function(s) used for deformation tracking, the rigid pose of the object in the current frame needs to be determined approximately. This enables us to track the deforming motions acting on the surface of the object alone, without having to care about the interaction with the surrounding scene (forces causing rigid motion, friction, collision etc.). This is done in a manner exactly similar to Sec. 3.2.3.

Non-rigid Error Terms

Once an approximate estimate of p T p-1 has been obtained, a combination of depth based geometric error and direct photometric error is minimized to achieve the tracking of the deformation using the method described in Section 4.3.1. We now define the error term that needs to be minimized for a single point p of the pointcloud at the p-th data frame. The point-to-plane distance based geometric error is given by:

E N (p p ) = n j • p p l -d j (4.18)
assuming that the j-th surface of V S has been associated with the l-th point of pointcloud p l . We propose a photometric error term defined on the (p-1)-th frame by:

E P (p p-1 ) = I p (p p-1 e ) -I p-1 (p p-1 ) (4.19)
The updated point position p p-1 e is determined by a barycentric map

p p-1 e = P i P i+1 P i+2 B (4.20)
such that p p-1 corresponded with the triangle P i P i+1 P i+2 and P i gives the updated vertex position of P i when subject to the update ϑ, given that B is a column vector denoting the barycentric coordinates of p p-1 w.r.t P i , P i+1 and P i+2 .

The combined cost function is given by E S = (E N , µE P ), where µ = E N E P is used for bringing the geometric and photometric error terms to the same scale. Following the outline provided by (4.5), we can now develop the partial derivatives ∂E N (p) ∂ϑ and ∂E P (p) ∂ϑ .

Jacobian

We describe the method to obtain the Jacobian for the geometric and photometric error term described in Eqn. 4.18 and Eqn. 4.19. The combined cost function that we seek to minimize is given as:

E S = (E N , µE P ) (4.21)
We aim to minimize (4.21) w.r.t u C , the displacement of the control handle. The Jacobian relating the change of u C to the change of E S is given by:

J = ∂E S ∂u C =   ∂E N ∂ξ µ ∂E P ∂ξ   Γ (4.22)
where:

∂E N ∂ξ =       -n j -n l A P i+2 -P i+1 × ) -n l (A P i -P i+2 × -n l A P i+1 -P i ×       (4.23)
where

A = 1 n j (1 3×3 -n j n j ) (4.24)
given that:

n j = (P i+2 -P i ) × (P i+1 -P i ) (4.25)
and n l = p p l -P i . On the other hand:

∂E P ∂ξ = ∇I u p ∇I v p               b 1 0 0 b 1 b 2 0 0 b 2 b 3 0 0 b 3                    H P i H P i+1 H P i+2      1 9×9 (4.26)
where: The optimization method of our choice is Iteratively Re-weighted Least Squares (IRLS) solver, and the update is given by:

H Pq =   fx (Pq) Z 0 -f x (Pq) X (Pq) 2 Z 0 fy (Pq) Z -f y (Pq) Y (Pq) 2 Z   ∀q ∈ i, (i + 1), (i + 2) (4.
∆u C = -λ WJ + WE S (4.28)
where W is a weighting matrix using Tukey based M-estimator [START_REF] Meer | Robust regression methods for computer vision : A review[END_REF], and λ is a scaling factor. The new position u C +∆u C of C j is to be applied on the mechanical model such that it deforms to minimize the errors from (4.18) and (4.19).

Implementing the Mechanical Model

The goal of this chapter is to formalize a method for dissociating the cost function E S from the physical model P while using them together to solve a non-linear least squares problem. To this end, as indicated in Sec. 1.3.1, we intend to base our choice of P on FEM. In practice, we validate our approach using two mechanical models: a linear FEM and a co-rotational FEM using a tetrahedral mesh.

As indicated in section 4.3.3, the rigid tracking of the object is handled without using the mechanical model (i.e., only based on the surface model), thereby allowing the deformation estimation to tackle a better posed problem. The object is assumed to be attached to a fixed base via projective constraints on the vertices of M that lies on the ground plane, thereby eliminating the necessity of modelling friction or contact forces.

Force Tracking

With the deformation tracking methodology in place, it is possible to utilize this approach to online estimate the deforming forces acting on the object. To do this, two additional constraints are required: the material properties of the object being tracked, i.e., Young's modulus, Poisson's ratio, Rayleigh stiffness etc. and the approximate point of contacts on the object. The control handles derived in Section 4.3.2 are virtual points that may or may not have a physical meaning. However, to track forces, we need the actual points of application of force. All our force tracking examples have been performed by tracking the tool applying the force using external, fiducial markers (classical hand or finger tracking algorithms [Letessier andBérard, 2004] [Hamer et al., 2009] can be used in unison with the approach proposed here to freely track applied forces on hand-manipulated objects).

Once a point P C has been identified as the point of contact, its nearest mechanical mesh vertex P M C is obtained using a nearest neighbor search. Thereafter, the force applied on P C is given as

i F P M i
, for all indices i such that P M C is a neighbor of P M i , wherein the force vectors F P M i is derived using (3.3).

Results

To validate the deformation tracking methodology and to quantitatively compare with the state-of-the-art, we propose a synthetic RGB-D dataset of deforming objects. There are some dataset of non-rigid objects already available in the public domain. However, to quantitatively validate the approach proposed in this chapter, we require a dataset that has: a) color and depth data from the scene, b) a 3D CAD (or similar), textureless model or some 3D template of the object that needs to be tracked, and c) the object being tracked should be non-rigid and must undergo deformation without the presence of any articulated joints and without any change of topology (cutting, tearing etc.). Among the publicly available dataset, [START_REF] Slavcheva | KillingFusion : Non-rigid 3D Reconstruction without Correspondences[END_REF] is not applicable since the current approach does not handle modelling of articulated joints. [START_REF] Jensen | A benchmark and evaluation of non-rigid structure from motion[END_REF] is a 2D dataset, not suitable to our problem statement. The synthetic dataset from [START_REF] Sengupta | Tracking of non-rigid objects using rgb-d camera[END_REF] does not have valid textural information.

Our proposed synthetic dataset consists of two objects undergoing non-rigid deformation, created using the Blender software [Blender, 2018]. This dataset will be made available publicly. For comparing the output of the algorithm with the groundtruth, the error metric of our choice is the Hausdorff distance between the object model from the 3D scene used for creating the data and the output of the algorithm.

In the implementation of our method, all the FEM models have been developed using the SOFA framework [Faure et al., 2012a]. For the physical model, the input to our system is a coarse 3D model of the surface of the object at the inital, un-deformed configuration. The volumetric, tetrahedral mesh is generated using Dirichlet tessella-tion and by Delaunay tetrahedralization of the input mesh using the Bowyer-Watson algorithm [Si, 2015]. The initialization of C T O at the very first frame is done by matching pre-trained markers with those extracted from the initial images, similar to the approach of [START_REF] Choi | Real-time 3d model-based tracking using edge and keypoint features for robotic manipulation[END_REF]. Since we track the model frame-to-frame, no segmentation is required for this approach.

Frame 4: Re-implementation of [Petit et al., 2015a] In [Petit et al., 2015a], the images from the RGB-D sensor are segmented to isolate the object to be tracked using a grabcut approach, which is in turn utilized to extract a segmented pointcloud. To track the deformation of this object, an external force is applied to the mechanical model (simulated with FEM) of the object. The per-vertex force exerted is given by:

F ext i = ζ i k(P Vm i -P ψ i ) (4.29)
where F ext i is the external force vector for the i-th element of V m which is given by

P Vm i , ζ i = σ 1 e -( d i σ 2
) such that d i is the closest distance of P Vm i from the projected 2D contour of the object, σ 1 and σ 2 are two empirically chosen constants, k is the stiffness of the element corresponding to the i-th node, and the variable P ψ i is obtained from the segmented pointcloud ψ by the formulation:

P ψ i =        αNN ψ (P Vm i ) + (1 -α) 1 #ψ i P j ∈ψ i P j if #ψ i > 0 αNN ψ (P Vm i ) + (1 -α)P Vm i else (4.30)
where NN ψ (P Vm i ) denotes the nearest neighbor of P Vm i in ψ, ψ i is a subset of ψ such that ∀(ψ i ) j ∈ ψ i there exists NN Vm ((ψ i ) j ) = P Vm i and α is a tunable parameter. Some additional weights are assigned to the vertices based on the distance to the nearest contour of the mesh, as visible from the projection of the model on the image plane. We re-implement this method from [Petit et al., 2015a], while implementing co-rotational FEM using SOFA library and repeat the experiments while fine-tuning the empirical parameters. We keep only the best results among all the experiments. Moreover, to avoid ambiguity, we restrict the comparison to dataset with small deformation only (deformation confined inside the contour of the projected model).

Cube Icosphere

Std. FEM Constant Γ 5.76 % 4.83 % Updated Γ 5.75 % 5.42 % CR FEM Constant Γ 2.14 % 5.25 % Updated Γ 2.14 % 5.34 % [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] 2.93 % 7.93 % [START_REF] Sengupta | Tracking of non-rigid objects using rgb-d camera[END_REF] 3.28 % 6.67 % TABLE 4.1 -Comparison of tracking accuracy for the two synthetic sequences in terms of Hausdorff distance between tracking output and groundtruth. Standard (Std.) and co-rotational (CR) FEM has been tested while maintaining Γ constant, as well as while updating it numerically in every iteration. The same sequences have been compared with [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] and [START_REF] Sengupta | Tracking of non-rigid objects using rgb-d camera[END_REF] in the last two rows. The Hausdorff distances are expressed as percentage of the longest side of the 3D bounding box of the model of the object in undeformed configuration.

The assumption introduced in Eqn. 4.15 (i.e., Γ for a particular mesh vertex can be thresholded, binarized and kept constant throughout a sequence of deformation) forms the theoretical basis of the entire methodology proposed in this chapter. We demonstrate that the difference in accuracy between maintaining the influence matrix Γ constant (after it has been computed only once during the initial estimation step) and re-computing Γ at each frame numerically is not significant. To establish this proposition, we run tests on the synthetic data with and without holding Γ constant for both standard, linear FEM and co-rotational FEM. The results are summarized in Table 4.1. It can be clearly seen that the variation in accuracy of tracking is between 0 to 0.59 % for both the sequences tested here. This comes in exchange for a large improvement in runtime (> 53% improvement in per frame time requirement, see Table 4.2) of the entire algorithm, since multiple FEM simulations per iteration of the IRLS solver is highly expensive. Fig. 4.3 shows the visual comparison of all the approaches tested on the synthetic sequences.

For the synthetic dataset, the approach proposed in this chapter outperforms the accuracy of the state-of-the-art methods for physical model based tracking proposed in [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] and [START_REF] Sengupta | Tracking of non-rigid objects using rgb-d camera[END_REF]. Table 4.1 describes the variance in the accuracy when the two synthetic sequences were tested with [START_REF] Petit | Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot[END_REF] and [START_REF] Sengupta | Tracking of non-rigid objects using rgb-d camera[END_REF]. The proposed deformation tracking approach is highly tolerant to variance in the estimate of the Young's modulus (YM) and Poisson's ratio (PR) used to model the FEM. While varying the YM between 5000 Pa to 5 MPa, the accuracy varied by less than 0.001%, which is practically negligible. While varying the Poisson's ratio between 0.2 and 0.5, the accuracy drifted by < 0.129%.

To validate our approach on real objects, we utilized two objects: a deformable dice made of foam and a block of sponge in the shape of a cuboid. The results are visually demonstrated in Fig. 4.4. The tracking results are visually coherent while suffering occlusions/outliers in the range of ±5 cm. The dice was compressed for more than 14 cm (> 87% of its own height), but our approach still tracked the entire deformation accurately.

Validation of Force Tracking

To validate the force tracking methodology, we use a 6-DOF anthropomorphic robot arm (a Viper 850 from ADEPT) fitted with a ATI Gamma IP65 force/torque sensor and a 3D-printed stylus as an end-effector distal tool. The robot is used only to utilize its force sensor to obtain a groundtruth of the force that is being applied on the object. To get a proper estimate of the force using the proposed method, we need to know the material property (Young's modulus, Poisson's ratio, etc.) of the object, and the point of contact needs to be measured separately. To demonstrate this feature, we use the sponge and a ball made of foam. First, the Young's modulus of the sponge and the ball is determined by repeated indentation tests. It is determined that the sponge and the ball has a Young's modulus of 460 kPa and 160 kPa respectively. Next, these objects are subjected to a strong deformation using the robot's end-effector, while our proposed method is used to track the deformation and the force applied simultaneously. The 3D printed end-effector is tracked using pre-trained markers from the image data of the RGB-D camera. The mesh vertex nearest to the tip of the end-effector gives us the point of contact. The results of the force tracking approach are summarized in Fig. 4.5.

We observe a force tracking accuracy of ∼ 97% for the sponge and of ∼ 90% for the ball, when the deformation is optimally observable. For very small deformations (< 1 cm), the sensor noise causes small variances in the force measurement while at very large deformation, the tip of the end-effector occludes the deformed surface to a very large extent, causing the accuracy to decrease in the experiment with the sponge. However, with properly observable deformations (> 1 cm and free from large occlusions), the approach is very accurate. The measured forces show a consistent Pearson correlation coefficient of > 0.92 with the actually applied forces over a large

Runtime Evaluation

The experiments discussed in this chapter were performed on a Intel Xeon CPU working at 3.70GHz. Utilizing only a single core of the computer, the un-optimised code was able to achieve a runtime of 350 msec -550 msec/frame while tracking deformations using the constant value of Γ, showing that it can achieve real-time performance at frame rate. The time taken to estimate the value of Γ for each node is approximately 0.5 sec -0.6 sec per frame, which is a one time cost that can be either done offline or computed when needed, as per the requirements of the application. The runtime for the real data used in this chapter is summarized in

Conclusion

The chapter presented here describes a new method for combining visual error minimization with FEM to create an accurate and fast deformation tracking method. The algorithm has been tested on synthetic and real data and has been shown to outperform state-of-the-art methods in tracking accuracy for generic deforming objects. The algorithm proposed here can be extended to other, more complex physical models without loss of generalization. The method proposed here has been demonstrated as a reliable visual force tracking system, when the material properties of the object being tracked are available. We believe that there is a large potential for such applications in the field of robotics and augmented/mixed reality.

In this dissertation, we have so far presented two different approaches for deformation tracking. In the next chapter, we are going to present some applications of deformation tracking, especially highlighting the benefits of using a FEM based approach as the primary deformation model. We will present a robotic application involving estimation of elasticity parameters of a deforming object while it is being tracked by a RGB-D camera.

CHAPTER 5

ROBOTIC APPLICATIONS FOR NON-RIGID OBJECT TRACKING

The preceding chapters dealt with different mechanisms to track deforming object using mechanical model. As indicated in Chapter 1, the usage of mechanical models enable us to re-utilize these deformation tracking methodologies into some interesting use-cases, which would not have been straightforward using physical model-free deformation tracking approaches. In this chapter, we take a look into some of the practical applications that arises out of using physical models for deformation tracking. More specifically, we describe a method to estimate the physical properties of an object using the deformation tracking algorithm and an external force sensor mounted on a robotic arm. Later in this chapter, we extend that approach to track freely applied forces on the object, once the physical properties have been determined.

Robotic manipulation of rigid objects is a well-studied and well-developed area of research. However, non-rigid object manipulation using robotic end-effector is a more difficult problem to address. Part of the problem arises from the fact that real-time tracking of non-rigid objects had been an unsolved problem until the last few years. Even with the recent development of non-rigid object tracking methods using computer vision, there remains significant scope for improvement in tracking the entire surface model of the object under challenging conditions. Another important question arising while interacting with soft objects is the determination of the elasticity parameters of the object, especially the Young's modulus. Simultaneous manipulation and tracking of deformable objects could improve different use-cases such as humanoid robots interacting with soft objects, compliance testing in product manufacturing or automated, preventive healthcare [Stults, 1984]. The aim of this approach is to simultaneously track and estimate the elasticity parameters of a soft object that is being deformed by the end-effector of a robot. In this chapter, we propose a closed-loop method consisting of: a) a deformation tracking method and b) an elasticity estimation algorithm. This closed-loop method allows to achieve two objectives: STEPE (Simultaneous Tracking and Estimation of Parameters of Elasticity) and remote force estimation. The required inputs of STEPE are a coarse 3D geometric model (mesh) of the object to track and the external measurements of the forces deforming the object. The deformation tracking method uses a depth sensor to capture the deformation and tracks it using a physicsbased simulation of the object deformations. The elasticity estimation algorithm uses the result of the deformation tracking method and measurements of the deformation forces, obtained from an external force sensor, in order to estimate the elasticity parameters of the object. Once the elasticity estimation is achieved, the parameter can be used by the deformation tracking method, thereby closing the loop. The method will iteratively converge towards a correct estimation of the elasticity parameters of the object. The elasticity parameters obtained at the convergence of the method are thereafter used for remote force estimation. From this stage, the deformation tracking method can be used to estimate the deformation forces acting on the object without any external force sensor.

The main contributions of this chapter are summarized below:

-We propose a novel method to simultaneously track the deformation of soft objects and estimate their elasticity parameter. The tracking of the deformable object is performed by combining the visual information acquired with a RGB-D sensor with interactive simulation of the object deformation based on FEM ;

-In parallel, the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor ;

-Once the elasticity parameters are estimated, our tracking algorithm can then also be used to estimate the deformation forces applied on an object without the use of a force sensor

Background

As discussed in Sec. 1.3, non-rigid object tracking can be done using either a physics-based model, or a geometry-based model for computing the deformations. Both physics-based model such as [START_REF] Newcombe | Dynamicfusion : Reconstruction and tracking of non-rigid scenes in real-time[END_REF], Innmann et al., 2016] or geometric-based models such as [START_REF] Bronte | Modelbased real-time non-rigid tracking[END_REF], Bronte et al., 2014] offer mecha-126 nism to track complex 3D shapes without utilizing any kind of machine-learning based approach. However, iteratively refining the physical parameters of the object being tracked is not possible using any of these techniques. Zhang et al. use a RGB-D camera to track the external forces applied on an object, but they utilized very specific soft-robots for the purpose, instead of doing it for any arbitrary deformable objects [START_REF] Zhang | Calibration and external force sensing for soft robots using an rgb-d camera[END_REF]. The approach of Petit et al. [START_REF] Petit | Capturing deformations of interacting non-rigid objects using rgb-d data[END_REF] is better suited for the purpose of deformation tracking using a physics-based model. However, since it relies solely on depth data and a point-to-plane correspondence matching scheme to determine the direction and magnitude of deformation, this approach is more susceptible to error induced from incorrect correspondences due to heavy occlusions. Among the machine learning based approaches for deformation tracking, Varol et al. use a constrained latent variable model for learning the deformational behavior of an object from its latent state to an output state [START_REF] Varol | A constrained latent variable model[END_REF]. The results from this approach has however only been demonstrated on planar objects. From the literature, the methodology proposed by Frank et al. [START_REF] Frank | Learning the elasticity parameters of deformable objects with a manipulation robot[END_REF] has some resemblance to the method proposed in this chapter for simultaneously tracking deformation and estimating parameters. The approach utilizes a combined tracking and elasticity estimation modules using a point-to-point ICP [START_REF] Besl | Method for registration of 3-d shapes[END_REF] for pointcloud registration along with a linear, tetrahedral FEM as the deformation model. The use of linear FEM makes this approach not suitable when large rotational deformations occur. Additionally, point-topoint ICP is not tolerant towards occlusion adequately, which compels the authors to utilize a narrow wooden probe (attached to the end-effector) to avoid occlusion. To the best of our knowledge, this approach has not been used as a force estimator.

Manipulation of soft objects becomes an easier problem statement when the elasticity parameters of the objects are known. Sedef et al. proposed a method using a haptic device and a force sensor to compute the linear viscoelastic parameters of a homogeneous soft biological tissue [START_REF] Sedef | Real-time finiteelement simulation of linear viscoelastic tissue behavior based on experimental data[END_REF]. Bickel et al. proposed a method for handling elastic objects [START_REF] Bickel | Capture and modeling of non-linear heterogeneous soft tissue[END_REF] of heterogeneous composition. A soft object, monitored by a marker-based stereovision system, is stimulated by a probe equipped with a force sensor to estimate the Lamé's coefficients of heterogeneous deformable objects. Data-driven methods are also available to determine elasticity parameters of heterogeneous elastic objects. Wang et al. proposed an approach that iteratively performs deformation tracking and parameter estimation using a physics-based model with the help of a two-step optimization method [Wang et al., 2015]. A method based on static Euler-Bernouilli deformation model along with a tracking methodology using multi-section parameterization, was developed for simultaneous tracking and estimation of deformable objects in [START_REF] Fugl | Simultaneous estimation of material properties and pose for deformable objects from depth and color images[END_REF]. This method has two assumptions: the first is that the object is homogeneous and linearly elastic, the second assumption is that the bending of the object is subject to gravity. However, this method has only been tested on planar object.

In this chapter, we propose the simultaneous tracking and elasticity parameter estimation of arbitrary soft objects using the same physical model, which can be refined iteratively, as opposed to [START_REF] Frank | Learning the elasticity parameters of deformable objects with a manipulation robot[END_REF]. The deformation tracking methodology used here is similar to Chapter 3. It does not use any fiducial markers on the deforming object. An approximate model of the object is enough for both deformation tracking and elasticity parameter estimation. As opposed to [START_REF] Zhang | Calibration and external force sensing for soft robots using an rgb-d camera[END_REF], the remote force estimator does not require a prior knowledge of the elasticity parameters. The STEPE method can run on inexpensive hardware requiring only a consumer-grade RGB-D sensor and force measurements.

This chapter is divided into three sections. The next section details the proposed methodology. The results are presented next. The discussions and the concluding remarks are conducted in the last two sections.

Elasticity Estimation from Deformation

The objective of the methodology proposed in this chapter is to track a deformable object while estimating its elasticity parameters simultaneously. In this chapter, we achieve this objective using a method, that we named STEPE (Simultaneous Tracking and Estimation of Parameters of Elasticity), made of three different modules: one performing the deformation tracking, one estimating the elasticity parameters and one measuring the external deformation forces applied on the object. In STEPE, all the modules are involved. When doing remote force estimation, only the module using the deformation tracking method is considered. The overview of the principle of the method is depicted in Fig. 5.1.

Modeling

The deformation model proposed for this method is similar to Chapter 3. The corotational formulation of FEM [START_REF] Müller | Interactive virtual materials[END_REF]] is used to avoid large deformational artifacts. The fundamental principle involves the Hooke's law that relates the stress and the strain applied on an object using a linear equation that depends on the elastic parameters of the object, i.e., the Young's modulus and the Poisson ratio.

Let the number of vertices of the mesh V M be n, M the mass matrix, D ∈ R 3n×3n the damping matrix and K ∈ R 3n×3n the stiffness matrix which depends on the Young's modulus and Poisson ratio of the entire object. Let p(t) ∈ R 3n denote the position of each vertex of the mesh at time t, f ext ∈ R 3n the external forces applied onto the object and f int ∈ R 3n the internal forces resulting from the deformation. The linear algebraic equation of motion of an object is given by Eqn. 3.4. This equation is used to estimate the elasticity parameters from the deformation observations as well as to compute the external deformation forces from deformation observations when the elasticity parameters have converged.

STEPE

The STEPE methodology consists primarily of three sub-modules: the first one measures the external forces and controls the robotic end-effector to deform the object gradually by applying a pre-determined displacement along a direction which minimizes the geometric error defined on the surface model of the object. Second module tracks the object using the deformation tracking methodology that will be described here. Third module estimates the elasticity parameters of the object using the output of deformation tracking and with the sensor data obtained from the measurement module. We now describe these modules in details.

External Force Measurements Module

The external measurements module is responsible for measuring the deforming forces that are applied on the tracked object. To ensure that the experiments are repeatable, we decided to use a robotic arm equipped with a force sensor in order to both manipulate the object and measure the resulting forces. The end-effector deforms the object by sequential, compressive motions approximately along the direction of the normal to the surface of contact while recording the forces that are exerted. The end-effector motion is stopped at a certain point in order to maintain a static deformed state. This module stores the external force measurements and timestamps at which they were acquired in order to transmit them to the estimation algorithm when a steady-state has been reached in the deformation tracking.

Deformation Tracking Method

In this chapter, we implemented a deformation tracking method similar to the one proposed in [START_REF] Sengupta | Tracking of non-rigid objects using rgb-d camera[END_REF]. The initial pose of the object(O) w.r.t the camera (C) is obtained in the first image frame using pre-trained markers with an approach similar to [START_REF] Marchand | Pose estimation for augmented reality : a hands-on survey[END_REF]. The pose of the camera w.r.t the object, denoted by the homogeneous transformation matrix C T O and is updated at the beginning of each frame, using an approximate rigid object tracking method with the assumption that the object is rigid. The rigid tracking is done with the technique described in Section 3.2.3 of Chapter 3.

The geometric residual error is computed according to Eqn. 3.5. As given in Sec. 3.2.4, the minimization of this residual error by regulating the virtual forces acting on the surface of the mechanical object model results in the deformation tracking. The Jacobian (J) is computed using the axial perturbation method, which is somewhat similar to the approach demonstrated in shown in Fig. 3.3. Every node is subjected to a small force given by ∆F x , ∆F y and ∆F z , acting along the three axes. The displacement of all vertices of the mesh are computed by solving Eqn. 3.4 using an Euler-Implicit mechanism of a conjugate gradient based linear solver [Faure et al., 2012a]. The Jacobian J is computed using finite differences in the value of e N obtained from the three deformed configurations. The only difference between the Jacobian computation in this chapter w.r.t the method described in Chapter 3.2.4 is the modification of the axial perturbation from central-differences to forward-finite differences, which results in performance optimization.

Once the Jacobian J relating the variation of e N w.r.t the variation of the force vector F is obtained, the force update is given by:

∆F = -λ WJ + We N ( k-1 P) (5.1)
where W is a weighting matrix from Tukey based M-estimator [START_REF] Meer | Robust regression methods for computer vision : A review[END_REF], and

λ = λ X λ Y λ Z (5.2)
is a scaling factor. The deformation caused in the mesh due to the Jacobian computation is discarded and the update:

k-1 F = k-1 F + ∆F (5.3)
is applied on the node, where k-1 F is the external force already existing on the node at the previous image frame. This Iteratively Re-weighted Least Squares scheme is allowed to repeat for a fixed number of iteration at every image frame. These estimated forces using Eqn. 5.3 result in deformation of the mechanical mesh using the co-rotational FEM formulation, as described in Sec. 1.3.1. The vertices of the mesh V M obtained by the procedure given above, i.e., tracking of the observed RGB-D data, is denoted by p * , and will be used for elasticity parameter estimation, as described in the following section.

Estimation algorithm

In this chapter, we chose to use the Hooke's law evoked in 5.2.1 to represent the deformable objects. Consequently the elasticity parameters that can be estimated are the Young's modulus and the Poisson ratio. In order to estimate the elasticity parameters at a contact point of a deformable object, the algorithm relies on the tracking results p * and the external measurements of the deformation forces f * taken at the same point. Let V S v (the visible mesh and vertices are denoted by the suffix (•) v ) be the vertices that the camera can see at any given frame (determined by a raytracing algorithm) and p v denotes every vertex of the visible mesh. A cost-function e(V) is minimized using the Levenberg-Marquardt algorithm [Moré, 1978] for updating the value of υ, the Young's modulus of the object, and is given by:

e(V) = pv∈V S v e v (υ) 2
(5.4)

In this expression, e v is the error function defined such as:

e v (υ) = p * v -p s v (f * , υ) (5.5)
and is equal to the Euclidean distance between p * v and p s v , which are respectively the tracked and simulated 3D positions of V S v . p * v is taken in at the final deformed state at the end of tracking and p s v at the end of the simulation. The estimated parameters are the ones for which the cost-function e(V) is minimized.

The simulation applies forces on the simulated object, which replicate the ones that were recorded during the experiment. The Jacobian of the cost-function is needed to compute the update of the elasticity parameters between two steps k and k + 1 of the minimization algorithm. This Jacobian is numerically estimated by running several corotational FEM simulations using different values of elasticity parameters and computing the error between the experimental data and the simulation result. Let ∆υ(k) be the update of the elasticity parameters at the k-th iteration step, e be the Jacobian at this step of the function p s (f * , υ) w.r.t. the elasticity parameters and µ be a damping factor that is adjusted at each Levenberg-Marquardt step. The update of the elasticity parameters can be computed by solving for ∆υ(k) the linear equation : In this chapter, we focus on estimating the Young's modulus . The Poisson ratio is assumed to be known a priori. Thus, the elasticity parameters vector is given by υ = (E) ∈ R. At this stage, the estimated value of Young's modulus obtained from the STEPE method described above is defined by Ê.

Remote force estimation

When doing remote force estimation, the deformation tracking module can be used without external force sensor. Let A be the set of vertices representing the active region of the object. The active region is an approximation of the surface of the model where the deformation is happening. This region is determined using some practical heuristics e.g., neither the vertices lying on the ground plane nor the vertices not visible from the camera are taken into account. Active vertices, denoted by a, are the vertices belonging to the active region. The method is able to estimate the external deformation forces, given by:

f ext = a f ext a (5.8)
which are applied on the object in the active region, as shown in Fig. 5.2. The estimated forces are projected onto the normal of the active surface:

n A =
a n a a n a (5.9) because the tracking method cannot have information about tangential forces, since (4.18) is a point-to-plane distance. This produces the following estimate of the magnitude of the deformation forces:

f ext ≈ f ext • n A (5.10) FIGURE 5
.2 -Force estimation in the active region.

Experiments

Setup

The experimental setup shown in Fig. 5.3 consists of a 6-DOF anthropomorphic robot arm (a Viper 850 from ADEPT) equipped with a ATI's Gamma IP65 force/torque sensor and a 3D-printed stylus used as an end-effector distal tool. A RGB-D Intel Realsense D435 camera is used for the tracking. The implementation has been done on a computer that has an Intel Xeon CPU working at 3.70GHz with 16 logical cores. We tested our method with different soft objects: a foam block, a soft ball and a complexshaped plush toy. The transformation between the robot frame and the object frame is computed only once, i.e., at the initialization of the method. Thereafter, it is utilized to express measured forces into the coordinate frame of the object. The mesh of the plush toy has been generated by photogrammetry using the Meshroom software [Moulon et al., 2012, Jancosek and[START_REF] Jancosek | [END_REF]. The corotational FEM simulations are performed using SOFA framework [Faure et al., 2012a].

Results

We now describe a series of experimental results and the necessary implementation details associated with these experiments. The experiments are categorized into certain conditions. Each of these condition represents a particular position for a speci- fic object. E.g., the first condition described below denotes the rectangular foam being deformed at the center of its largest face, while being placed horizontally on a table. To evaluate the accuracy of the estimation process, the ground truth of Young's modulus E GT has been determined through indentation tests for each condition of experiments [START_REF] Mckee | Indentation versus tensile measurements of young's modulus for soft biological tissues[END_REF]. The indentation tests can be described as: repeated, slow and incremental vertical displacements applied on the objects using the robot while measuring the forces and the displacements. Forces are measured from the force sensor fitted in the robot's arm while displacements are obtained using the odometry of the robot. For each condition of experiment, the average of several indentation tests is considered to be the ground truth. We denote that as: E GT . Let Ê(i) be the estimation of the Young's modulus obtained from the tracking results for the i th deformation. The Young's modulus that is thereafter used for the external force estimation, denoted by E est , is the one for which the following convergence criterion is respected: Ê(i) -Ê(i -1) Ê(i -1) < 0.05 (5.11) Some examples of the output of the deformation tracking method are shown in Fig. 5.4. The mean of the norm of the error ( e N ) was found to be 1.53 mm, 0.51 mm and 0.21 mm for the foam block, soft ball and plush toy, while the standard deviations were 7.2 mm, 0.7 mm and 1.4 mm respectively, across all the experiments reported in this article. In the three example sequences shown in Fig. 5.4, the mean of the norm of error varied between a maximum of 0.381 mm (for the plush toy, deformed by its nose) to as low as 3 µm (for the undeformed sponge block), despite having outlying correspondence or noise in the range of -5.5 cm to +9.0 cm (which gets rejected by the M-estimator). In the experiments shown in Table 5.1, the time required for tracking varies between 1.21 to 3.1 sec/frame. However, it was possible to run the same algorithm on the foam block at ∼ 800 ms/frame without using any GPU based computation, with negligible loss of accuracy (< 10%).

Our method can be evaluated on different criteria. The first set of experiments was conducted to evaluate the time-performance of the tracking and the quality of the Young's modulus estimation w.r.t. the number of vertices of both the visual mesh and the mechanical mesh. The results of this set of experiments are grouped in Table 5.1. In this table, #Visual designates the number of vertices of the visual mesh while #Mechanical designates the number of vertices of the mechanical mesh. t tr is the average deformation tracking time, std(t tr ) is its standard deviation, t st is the time that was required to deform the object and reach a steady-state and t est is the time to estimate the Young's modulus. All the times are expressed in seconds. E GT and E est correspond to the average ground truth of Young's modulus and estimated Young's modulus respectively, both expressed in kilo Pascals. Finally, Error designates the percentage of error of the estimation and is given by Error = 100 * abs(E GT -E est )/E GT where abs designates the absolute value. The second set of experiments was conducted to evaluate the consistency of the Young's modulus estimation w.r.t. the initial estimate E 0 . Low initial estimate are usually of the order of 0.1 × E GT and high initial estimate of the order of 10 × E GT . The results of this set of experiments are grouped in Table 5.2.

The last set of experiments was conducted to evaluate the estimation of the defor- 5.3. f GT and f est designate the average norm of the ground truth forces measured by the force sensor and the norm of the forces estimated by the deformation tracking method respectively, expressed in Newtons. ∆f = f est -f GT represents the error between the estimated and ground truth forces. The estimated forces are projected onto the measured forces when compared with the ground truth. The force experiments have been conducted with different orientations of the objects (horizontal and tilted by approximately 25 degrees w.r.t. the z-axis of the object frame). Different depths of stimulation have also been tested and are indicated in the depth column of Table 5.3. Finally, Fig. 5.5 depicts a possible use case of our method to estimate deformation forces exerted on an object in an environment where no force measurements are available.

Discussion

The first set of experiments whose results are summarized in Table 5.1 shows that the accuracy of the Young's modulus estimation is improved as the resolution of the mechanical mesh increases. This results from the fact that the deformation can be represented with finer details and is thus closer to the reality. Table 5.2 shows that our method is able to estimate the Young's modulus of an arbitrary object even when the initial estimate is much different from the ground truth. A future work would consist in using our method to estimate both Poisson ratio and Young's modulus within an iterative process. Table 5.3 shows that the estimated Young's modulus could be used by our method to accurately estimate the deformation forces from visual information only. The use case scenario depicted in Fig. 5.5 shows one possible application of our method in environments where no force sensor could be used, for instance in human-performed manipulation tasks. This application opens novel perspectives for better estimating the forces applied on a deformable object by different operators, as well as on a cooperative task on such objects.

Conclusion

In this chapter, we described a method, that we named STEPE, to simultaneously estimate the elasticity parameters of a deforming object while tracking their surface using a RGB-D camera. The fundamental technique for tracking the surface was already developed in Chapter 3, while this chapter produced a more optimized method of deformation tracking, enabling the estimation of elasticity parameters simultaneously. Moreover, we utilize the estimated parameters to track the deforming forces acting on the object remotely, without using any external force sensor. These results pave the way for better control of deformable objects manipulated by a robot.

CONCLUSION AND PERSPECTIVES

Conclusion

Here, we first briefly recall the concepts presented throughout this thesis. This is followed by a concluding summary of the individual chapters. We then proceed to end this thesis by discussing some of the short and long term perspective research work that can be spawned by the approaches proposed in this thesis.

Brief Summary of the Thesis

In Chapter 2, we present a framework for precisely tracking objects of complex shapes with joint optimization of depth-based geometric and photometric error terms using a low-accuracy 3D object model with the help of RGB-D cameras. Object tracking with coarse 3D model is useful for industrial applications, where scanning and reconstructing exact 3D model is often prohibitively difficult. In this chapter, a method is proposed that uses a combination of point-to-plane distance minimization and photometric error minimization to track objects accurately. 'Keyframes' are used in this mechanism of object tracking for minimizing drift. The approach proposed in this chapter has been validated on both simulated and real data. It has been shown that our approach is more accurate than existing state-of-the-art approaches on benchmarked, simulated dataset, especially while dealing with low-textured objects with co-planar faces.

In Chapter 3, a method to track deformable objects with a RGB-D camera using a coarse 3D model is presented. The model of deformation is based on corotational FEM. The physical model of the deforming object does not need to be precise and we do not require the precise physical properties of the object to track it accurately. The vertex positions of the surface mesh of the tracked object is regulated using virtual forces which are obtained by minimizing a point-to-plane distance based geometric error between the pointcloud and the mesh. The point of application of force is obtained by thresholding and clustering of the error obtained from rigid tracking, which is run in parallel with the non-rigid tracking algorithm. This parallel implementation also enables the overall system nearly real-time at frame-rate. The proposed approach has been evaluated on a synthetic dataset with ground-truth, as well as on real data.

In Chapter 4, we propose a methodology for tracking non-rigid objects with RGB-D cameras by minimizing a combination of depth and photometric error. The approach proposed in this chapter does not depend on the accuracy of the material properties provided as input for deformation tracking. However, if an accurate estimate of the material properties are known, it can be used to measure the external, deforming forces acting on the object with just the visual data obtained from the RGB-D camera. The mechanism proposed in this chapter combines computer-vision based methodologies with physical model based deformation representation without computing expensive numerical optimization for minimizing the non-linear error terms. The method proposed here has been validated on synthetic and real data and has been shown to outperform some of the state-of-the-art techniques in physical model based deformation tracking, while opening up new applications, such as visual force estimation on deforming objects.

In Chapter 5, a new approach to simultaneously track the deformation of soft objects and estimate their elasticity parameters has been suggested. This approach also allows the estimation of deforming forces applied on an object from the visual information only. This is done by combining corotational FEM simulations and minimization algorithms to track the deformation and estimate the elasticity parameters of a soft object jointly. A commercial RGB-D sensor is used for obtaining the visual information needed for tracking. This information is utilized to deform the object model according to the virtual forces computed using the deformation tracking algorithm. The module involved in the elasticity parameter estimation minimizes the error between the tracked object and a simulated object deformed by the forces that are measured using a force sensor mounted on a robotic arm. Once the elasticity parameter is estimated, the deformation tracking can then estimate the deforming forces applied on the object without the use of a force sensor. The accuracy of the elasticity parameter estimation and of the force estimation has been validated on several soft objects with various complexities of shape.

Perspectives

There are some important work which can be done to follow up the methodologies proposed in this dissertation. We categorize them based on the expected duration it would take to complete this perspective research work. The perspectives can also be differentiated from the possible application areas of deformation tracking, which can be numerous.

Short-term Perspectives

The approaches proposed here have been demonstrated to work with depth and photometry based error minimization. However, in Chapter 4, we have already set the groundwork for easily incorporating multiple new cost functions e.g., sparse-feature based, edge-based or wavelet based error terms. This can be a relatively straightforward extension of our approach in Chapter 4 that could lead to further improvement in tracking accuracy.

All of the methods implemented in this thesis have been implemented on a single core of a CPU (OpenMP has been used sparingly for the STEPE method presented in Chapter 5). Therefore, all run-time performances reported in this thesis can be subjected to extensive performance optimization, thereby decreasing the runtime. There is a strong potential for improving the runtime significantly, given that some of the computations involved are highly parallelizable.

This thesis makes the use of control handles for tracking deformation. In Chapter 3 and Chapter 4, two different methods for determining these control handles from the input data has been proposed. However, determining the exact sequence of control handles that maximizes the accuracy of the tracking requires some heuristics and can be sensitive to noise and occlusion. Investigating a novel and robust mechanism for selecting these control handles to maximize the non-rigid tracking accuracy can be an interesting research area. This can possibly use the help of deep learning based approaches.

Long-term Perspectives

The approaches proposed in Chapter 3 and Chapter 4 can be extended to perform deformation tracking using monocular data. Given that the initial object to camera transformation at the first frame is known, a frame-to-frame tracking of the deforming surface can be done using a similar method, but with suitably modified cost functions to suit the monocular data.

Iterative mechanical mesh refinement using adaptive remeshing [START_REF] Narain | Folding and crumpling adaptive sheets[END_REF]] could be combined with the approaches proposed in this thesis for tracking the tearing or fracturing of volumetric objects without the prior knowledge of the physical properties of the object being tracked. On the same note, adaptive remeshing can also be used to reconstruct a physically-based model from RGB-D based voxel data. This online reconstructed physical mesh can be used to do the non-rigid object tracking.

A challenging direction for future research could possibly involve developing an unified, physics-based framework that is capable of tracking multiple mutually-interacting rigid objects, non-rigid objects and fluid surfaces using visual information only. Abstract : Tracking soft objects using visual information has immense applications in the field of robotics, computer graphics and automation. In this thesis, we propose multiple new approaches for tracking both rigid and non-rigid objects using a RGB-D camera. There are four main contributions of this thesis. The first contribution is a rigid object tracking method which utilizes depth and photometry based errors for tracking complex shapes using their coarse, 3D template. The second contribution is a non-rigid object tracking method which uses co-rotational FEM to track defor-ming objects by regulating the virtual forces acting on the surface of a physics based model of the object. The third contribution is a deformation tracking approach which minimizes a combination of geometric and photometric error while utilizing corotation FEM as the deformation model. The fourth contribution involves estimating the elasticity properties of a deforming object while tracking their deformation using RGB-D camera. Once the elasticity parameters have been estimated, the same methodology can be re-utilized for tracking contact forces on the surface of deforming objects.

  Mathematical Definitions . . . . . . . . . . . . . . . . . . . . 1.1.1 Frame Transformation . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Perspective Projection Model . . . . . . . . . . . . . . . . . . . . 1.1.3 Object Model and Pointcloud . . . . . . . . . . . . . . . . . . . . 1.1.4 Sensor Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Rigid Object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Classical Approaches . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1.1 3D-3D Registration . . . . . . . . . . . . . . . . . . . . 1.2.1.2 Dense 2D-2D Registration . . . . . . . . . . . . . . . . 1.2.1.3 2D-3D Registration . . . . . . . . . . . . . . . . . . . . 1.2.2 State-of-the-art for Rigid Object Tracking . . . . . . . . . . . . . . 1.3 Non-rigid Object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Physically-based Models . . . . . . . . . . . . . . . . . . . . . . 1.3.1.1 Tracking using Physics-based Models . . . . . . . . . . 1.3.2 Geometric Models . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2.1 Tracking using Geometric Models . . . . . . . . . . . . 1.3.3 Non-rigid Tracking and Reconstruction . . . . . . . . . . . . . . . 1.3.4 Non-Rigid Structure from Motion (NR-SfM) . . . . . . . . . . . . 1.4 Positioning this Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Rigid Object Tracking 2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1.1 Point-to-plane Distance Minimization . . . . . . . . . .

FIGURE 2 -

 2 FIGURE 2 -A banana being tracked by the approach of Chapter 2

FIGURE 3 -

 3 FIGURE 3 -The deformation of a sponge being tracked by the approach of Chapter 3

FIGURE 4 -

 4 FIGURE 4 -The deformation of a soft dice being tracked by the approach of Chapter 4

  FIGURE 1.3 -The two depth sensors from Intel: RealSense SR300 and RealSense D435. (b) shows the color-coded Structured Light (SL) pattern emitted by the IR projector of SR300, as viewed when projected on a flat surface. (d) shows the Structured Light for D435, used to improve the quality of stereo based active depth sensing by the sensor

1. 2 .

 2 Rigid Object Tracking usually expressed as the minimization of the sum of squared error term:

Frame 1 :

 1 FIGURE 1.6 -A continuous bar (top), represented by discretizing it into n nodes (bottom)

FIGURE 1

 1 FIGURE 1.8 -A tetrahedral element of a FEM with four vertices

FIGURE 1

 1 FIGURE 1.9 -Demonstration of the kinematics of a single element of a corotated, tetrahedral mesh. In the corotated frame, the red outline shows the deformed surface

FIGURE 1 .

 1 FIGURE 1.10 -Physical model based tracking of deforming objects with the approach of [Petit et al., 2015a]. Top: input images and bottom: tracked model. (Image Courtesy: [Petit et al., 2015a])

  FIGURE 1.11 -Outline for deformation tracking by fitting a FEM based object model to a pointcloud by controlling the forces applied on the mesh vertices, as given in Eqn. 1.71 (Image Courtesy [Wang et al., 2015])

  FIGURE 1.15 -SDF computation from depth scan

  FIGURE 2.2 -A toy example demonstrating the point-plane correspondence methodology. The correspondence is decided entirely on the image plane, with the green, blue and red dots corresponding to the planes marked A, B and C

[

FIGURE 2

 2 FIGURE 2.4 -Comparison of tracking results from 'ViSP Generic Tracker' [Trinh et al., 2018] (ViSP -first row), 'Point-to-plane + Photometry without Keyframes' (No KF -second row) and the 'Proposed Approach' (PA -third row)

FIGURE 2 . 6 -

 26 FIGURE 2.6 -Tracking results from real data captured using Intel RealSense for a) a box, b) a toy car, c) a banana

  FIGURE 3.2 -The depth camera is centered at F C while the tracked model is used for the rigid registration of the base model with the pointcloud. F c = F o at the beginning of the sequence, when C T O has not been initialized

  FIGURE 3.6 -Tracking deformation of the cube across different values for Young's modulus

  FIGURE 3.7 -Tracking deformation of the board across different values for Young's modulus

  FIGURE 3.8 -Comparison of Hausdorff distance between output and ground-truth for Numericoptimization and Linear-force-estimate for multiple values of Young's modulus

  FIGURE 4.2 -Given the control handle marked by red in (a), the numerically obtained values of Γ are plotted on the pointcloud registered to the object model, as shown in (b). The intensity of Γ linearly decreases as we go farther from the cotrol point. The values of Γ after applying the threshold (4.15) are shown in (c)

  27) and B = b 1 , b 2 , b 3 are the barycentric coordinates.

FIGURE 4 . 3 -

 43 FIGURE 4.3 -Comparison between groundtruth (red edges) and tracking output (black edges) for the two synthetic sequences cube and icosphere. The red edges and vertices shows the object model from the groundtruth, the black ones are from the tracking output.

  Deformation tracking results from the dice

FIGURE 4 . 4 -Force

 44 FIGURE 4.4 -Results from deformation tracking of a dice (a) and a sponge (b). Due to the presence of the physical model (co-rotational FEM), it is possible to track large deformations without producing any un-natural deformation in the object's mesh model

FIGURE 4 . 5 -

 45 FIGURE 4.5 -Setup and results for force tracking on the sponge. The Viper 850 robot is being used deform the object while being tracked. The red plot is the force measurement from the robot's force sensor, the blue plot is the estimated force reported by the proposed approach

  FIGURE 5.1 -Overview of the STEPE and Remote Force Estimation architecture.

  + 1) = υ(k) + ∆υ(k) (5.7)

  FIGURE 5.3 -The experimental setup shown with and without annotation in Fig. 5.3a and Fig. 5.3b. The robot (at reference frame F w ) is deforming a plush toy (at reference frame F o ) while the data is being captured from an Intel RealSense D435 RGB-D Camera (at reference frame F c ). The transformation between these reference frames are shown in Fig. 5.3a, following the convention established in Sec. 1.1.1

FIGURE 5 . 4 -

 54 FIGURE 5.4 -Visual results from the deformation tracking methodology. The first and the fourth column shows the color image, the second and the fifth column shows the tracked object model and the third and the sixth column shows the value of e N (the geometric error computed using Eqn. 3.5), color-coded and augmented on the object

FIGURE 5

 5 FIGURE 5.5 -Demonstration of force estimation while the plush toy gets freely deformed by hand. The deformation forces (13.8 N) are in the expected range.

  Suivi visuel, suivi des déformations, FEM Resumé : Le suivi d'objets déformable à partir d'informations visuelles à de nombreuses applications dans le domaine de la robotique, de l'animation ou de la simulation. Dans cette thèse, nous proposons de nouvelles approches pour le suivi d'objets rigides et non rigides à l'aide d'une caméra RGB-D. Cette thèse comporte quatre contributions principales. La première contribution est une nouvelle approche de suivi d'objets dans des images RGB-D qui utilise des erreurs basées sur la profondeur et la photométrie pour suivre et localiser des formes complexes en utilisant leur modèle 3D grossier. La seconde contribution porte sur une méthode de suivi d'objets non rigides reposant sur une approche par éléments finis (FEM) pour suivre et caractériser les déformations. La troisième contribution est une approche de suivi de la déformation qui minimise une combinaison d'erreurs géométriques et photométriques tout en utilisant la FEM comme modèle de déformation. Finalement, la quatrième contribution consiste à estimer les propriétés d'élasticité d'un objet analysant ses déformations toujours à l'aide d'une caméra RGB-D. Une fois les paramètres d'élasticité estimés, la même méthodologie peut être réutilisée pour caractériser les forces de contact. Title : Visual Tracking of Deformable Objects with RGB-D Camera Keywords : Visual Tracking, Deformation Tracking, FEM

TABLE 1 .

 1 1 -Comparison of the error terms that is minimized by some selected non-rigid tracking and reconstruction approaches. λ, w d , w r and w s are weighting terms

  Table 1.2. We discard the

	Monocular	NR-SfM [Gotardo and
		Martinez, 2011b] [Go-
		tardo	and	Martinez,
		2011a] [

TABLE 2

 2

			.1 -Summary of RMSE values
		Marmalade Container Coffee Machine	Simulated Car
		Translation Rotation Translation Rotation Translation Rotation
	No KF	0.045	0.139	0.284	0.032	0.265	0.072
	[Trinh et al., 2018] 0.053	0.004	0.118	0.008	0.183	0.027
	PA	0.043	0.003	0.048	0.006	0.179	0.036

  and their corresponding centroids are represented by k 1 , k 2 , • • •, k j , where k = (X, Y, Z, 1) is the homogeneous 3D coordinate of the point w.r.t F C . These points are inverse transformed with the last estimate of

C T O obtained after minimization with (3.11), such that

Pa 500 Pa 5 × 10 4 Pa 5 × 10 6 Pa 5 × 10 8 Pa

	Young's Modulus: Numeric-optimization 0.3118 0.3118 0.3009 5 Cube Linear-force-estimate 4.6448 2.3788 0.0051	0.3009 0.3455	0.3009 0.7383
	Board	Numeric-optimization 0.8239 0.8239 0.8761 Linear-force-estimate 2.0562 1.5674 0.064	0.8237 0.9092	0.8246 0.9081

TABLE 3 .

 3 1 -Table for comparison of Hausdorff distance between output and ground-truth for Numeric-optimization and Linear-force-estimate for multiple values of Young's modulus. Lower Hausdorff distance indicates higher accuracy

Table 4

 4

	.2.

TABLE 4 .

 4 2 -Comparison of the per frame time requirement for deformation tracking with constant and updated values of Γ using Standard (Std.) and Co-rotational (CR) FEM

TABLE 5 .

 5 1 -Evaluation of tracking and estimation times and Young's modulus (in kPa) estimation accuracy w.r.t. different quality of visual and mechanical meshes. (s) std(t tr )(s) t st (s) t est (s) Ē GT E est Error(%)

	Objects No. of vertices in Visual Mesh	No. of vertices in Mechanical Mesh
	foam 35	1049	1.94 0.14	62 310 454 431 5.08
	foam 415	1049	2.37 0.15	122 180 454 438 3.62
	foam 178	554	1.21 0.02	60 180 454 497 9.47
	ball 404	627	1.38 0.03	126 70 156 136 12.8
	ball 404	1060	1.61 0.02	118 96 156 148 5.0
	ball 404	1954	3.09 0.07	130 330 156 148 5.1

ttr

TABLE 5 .

 5 2 -Evaluation of the estimation of the Young's modulus (in kPa).

	Objects	E 0	E GT E est Error
		200	454 480 5.7%
	foam middle	1500 454 451 0.7%
		15000 454 488 7.5%
	foam comer	200	247 224 9.3%
	ball	1000 156 154 1.3%
	toy nose	200	51	53 3.790
	toy leg	50	42 43.1 2.6%

TABLE 5 .

 5 3 -Comparison of the average norm of the measured and estimated forces.Objects Orientation Indentation Depth (cm) f GT (N) f est (N) ∆f (N) mation forces in different conditions of experiments. For these experiments, the estimated Young's modulus values summarized in Table5.2 were used by the deformation tracking method. The results of this set of experiments are grouped in Table

	foam	horizontal	2	18.8	20.6	1.8
	foam	horizontal	2	21.36	23.11	1.75
	foam	horizontal	4	45.46	42.67 -2.79
	foam	tilted	4	38.94	35.59 -3.35
	ball	horizontal	2.5	9.49	8.57	-0.92
	toy	horizontal	2	5.32	5.81	0.49

1.2. Rigid Object Tracking

All results can be viewed at: youtu.be/_TPJkleBu3w

Please visit the following URL for all results on real data: https://youtu.be/RFd-Ix9hcdg

Γ O can be conveniently computed offline once per every object model and stored. During tracking, Γ O is loaded from memory when required.

For the next step, let us recall that the transformation from the object to the camera reference frame is given by O T C , following the standard notation defined in Sec. 1.1.1. We reconstruct the rotation matrix C R O using the element indices given in Eqn. 1.3 to