I would like to express my gratitude to my supervisors Adrien and Dani who always inspired and guided me during my PhD. With their immense knowledge about the field, they helped me a lot in looking for the right answers which led to my research. I am very thankful for their kindness, patience and a friendly assistance. I highly appreciate the fact that Adrien has always been available for a discussion. I am very grateful to Dani for his support. I have always found him insightful and his contributions are very significant in the development of this thesis. Both of them have been very friendly and caring, which has made this PhD a great learning experience for me.

I would also like to thank Toby for his guidance. His knowledge and perspective about research has helped me a lot in my research. It was a great pleasure to have spent time in this lab where everyone is so friendly and helpful. I have learnt a great deal from all of them and I will always cherish the memories of this time spent together.

In addition, I would like to thank the thesis jury reporters, Mathieu Salzmann and Alessio Del Bue for carefully reading the document and providing constructive remarks that has provided further encouragement to me for research in the field. I would also like to thank Marie-Odile Berger for her role of the President of the jury and her valuable comments. I am also thankful to my parents and Alok for their unconditional love which helped a lot during these PhD years. v

Abstract

Reconstructing the 3D shape of objects from multiple images is an important goal in computer vision and has been extensively studied for both rigid and non-rigid (or deformable) objects.

Structure-from-Motion (SfM) is an algorithm that performs the 3D reconstruction of rigid objects using the inter-image visual motion from multiple images obtained from a moving camera. SfM is a very accurate and stable solution. Deformable 3D reconstruction, however, has been widely studied for monocular images (obtained from a single camera) and still remains an open research problem. The current methods exploit visual cues such as the inter-image visual motion and shading in order to formalise a reconstruction algorithm. This thesis focuses on the use of the inter-image visual motion for solving this problem. Two types of scenarios exist in the literature: 1) Non-Rigid Structure-from-Motion (NRSfM) and 2) Shape-from-Template (SfT). The goal of NRSfM is to reconstruct multiple shapes of a deformable object as viewed in multiple images while SfT (also referred to as template-based reconstruction) uses a single image of a deformed object and its 3D template (a textured 3D shape of the object in one configuration) to recover the deformed shape of the object.

We propose an NRSfM method to reconstruct the deformable surfaces undergoing isometric deformations (the objects do not stretch or shrink under an isometric deformation) using Riemannian geometry. This allows NRSfM to be expressed in terms of Partial Differential Equations (PDE) and to be solved algebraically. We show that the problem has linear complexity and the reconstruction algorithm has a very low computational cost compared to existing NRSfM methods. This work motivated us to use differential geometry and Cartan's theory of connections to model NRSfM, which led to the possibility of extending the solution to deformations other than isometry. In fact, this led to a unified theoretical framework for modelling and solving both NRSfM and SfT for various types of deformations. In addition, it also makes it possible to have a solution to SfT which does not require an explicit modelling of deformation. An important point is that most of the NRSfM and SfT methods reconstruct the thin-shell surface of the object. The reconstruction of the entire volume (the thin-shell surface and the interior) has not been explored yet. We propose the first SfT method that reconstructs the entire volume of a deformable object. Chapter 1

Resumé

Introduction

Background

An important task in 3D computer vision is to recover 3D information from 2D images obtained by the camera. This task is widely termed as 3D reconstruction. Although there are active image sensors such as the Kinect and Time-of-Flight (ToF) cameras available which can obtain the depth of the view under consideration, passive 3D reconstruction from images remains an interesting topic for researchers because the scope of 3D sensors is limited due to the various constraints of size, cost and accuracy. 3D reconstruction methods rely on various visual cues from images (such as shading, texture, silhouettes, contours and motion) in order to find 3D descriptors such as the depth map and local surface orientation (or normals) of the objects.

The objects found in nature can be roughly classified into rigid or non-rigid (or deformable) objects. The 3D reconstruction of rigid objects using motion cues, also known as Structure-from-Motion (SfM) [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] (see figure 1.1a), has been widely studied for the past few decades and there are solutions available which are stable and accurate. SfM exploits the inter-image visual motion information in order to reconstruct 3D from multiple 2D images taken from different views of a rigid object. Rigidity allows the inter-image visual motion to be expressed in terms of the rotation and translation of the camera coordinate frames of the images. However, SfM cannot be extended to deformable objects as between any two images, the deformable object may undergo a deformation and therefore, the inter-image visual motion cannot be expressed only in terms of the camera rotation and translation.

In the past decade, the deformable 3D reconstruction problem has been studied extensively in over a hundred research papers. Some of the methods combine motion with other visual cues to disambiguate the problem and make it well-posed. For example, [START_REF] Liu-Yin | Better Together: Joint Reasoning for Non-rigid 3D Reconstruction with Specularities and Shading[END_REF][START_REF] Moreno-Noguer | Capturing 3D Stretchable Surfaces from Single Images in Closed Form[END_REF]Varol et al., 2012b] combine shading with motion, [START_REF] Gallardo | Using Shading and a 3D Template to Reconstruct Complex Surface Deformations[END_REF][START_REF] Gallardo | Dense Non-Rigid Structure-from-Motion and Shading with Unknown Albedos[END_REF] combine shading and contours with motion and [START_REF] Choe | 3-D Shape from a Shaded and Textural Surface Image[END_REF][START_REF] White | Combining Cues: Shape from Shading and Texture[END_REF] combine shading with texture. However, the existing solu-tions are not close to SfM in terms of accuracy and stability. Deformable 3D reconstruction is an important problem to solve as it has a wide range of applications such as in the medical, sports, entertainment and advertising domains. Some applications are explored in augmented reality: 1) [START_REF] Smith | Measuring the Accuracy of Softball Impact Simulations[END_REF] shows how to study the impact of a soft ball on various surfaces. This is useful in designing and testing sports equipments. 2) [START_REF] Haouchine | Image-guided Simulation of Heterogeneous Tissue Deformation for Augmented Reality during Hepatic Surgery[END_REF][START_REF] Haouchine | Using Contours as Boundary Conditions for Elastic Registration During Minimally Invasive Hepatic Surgery[END_REF][START_REF] Koo | Deformable Registration of a Preoperative 3D Liver Volume to a Laparoscopy Image using Contour and Shading Cues[END_REF][START_REF] Maier-Hein | Comparative Validation of Single-shot Optical Techniques for Laparoscopic 3-D Surface Reconstruction[END_REF] show how to augment the deformations of the body organs in order to aid minimally invasive medical surgeries. 3) [START_REF] Collins | Realtime Shape-from-Template: System and Applications[END_REF][START_REF] Ngo | Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture[END_REF] recover the deformation of the objects in real-time which can be useful in various industries. For example, they can be used by online shopping companies to enable the customers to try clothes and accessories virtually. Figure 1.2 shows some of these applications.

Figure 1.1: 3D reconstruction methods. For rigid objects, SfM is a widely used method (Images taken from [START_REF] Snavely | Modeling the World from Internet Photo Collections[END_REF]). The 3D reconstruction of deformable objects can be performed by either NRSfM or SfT methods.

This thesis focuses on the monocular deformable 3D reconstruction methods that use motion as a visual cue in monocular imaging conditions. We now define the problem in detail and describe our contributions.

1.1. BACKGROUND ! " "

# $ $ $% $ $ $% " &$ ' $ ( % ! ! $"% ) $ $
$" ! * $ * $ ! * $ ! * $ + , % ! ! $"% -" ! ! " " -$ $ " $ " $ ! " $ ! % Figure 1.2: Some applications of the deformable 3D reconstruction methods.

3D Reconstruction of Deformable Objects

As mentioned earlier, the 3D reconstruction of rigid objects by SfM cannot be directly extended to deformable objects. As these objects may undergo deformations, the inter-image visual motion (strictly induced by the change in camera coordinate frame in the case of SfM) is now dependent on both camera motion and object deformation. Exploiting this visual motion (coupled with deformations) becomes a challenging task as the constraints are weaker in this case.

The goal of this thesis is to propose a general framework for modelling and solving deformable 3D reconstruction. At this point, we classify deformable objects as thin-shell (objects with an infinitesimal thickness, such as a piece of cloth, paper, etc.) and volumetric objects (objects with non-negligible thickness such as a sponge, cushion, etc.). Volumetric objects can be considered as thin-shell objects in cases they are represented by their outershells only but at the price of losing inner constraints. We now discuss the two categories of deformable 3D reconstruction problems that arise in computer vision.

Shape-from-Template

SfT (see figure 1.1b) is the generic name for a set of methods which perform the monocular 3D reconstruction of deformable objects using a 3D template of the object. It is also called template-based (or model-based) reconstruction in the literature. The inputs of SfT are a single image and the object's template, and its output is the object's deformed shape. The template (sometimes also called model) is a very strong object-specific prior as it includes a reference shape, a texture-map and a deformation model. Some SfT problems, such as the reconstruction of isometric thin-shell objects, have been extensively studied. Some of these methods are [START_REF] Bartoli | Shape-from-Template[END_REF][START_REF] Brunet | Monocular Template-based 3D Surface Reconstruction: Convex Inextensible and Nonconvex Isometric Methods[END_REF]Chhatkuli et al., 2016b;[START_REF] Haouchine | Single View Augmentation of 3D Elastic Objects[END_REF][START_REF] Moreno-Noguer | Capturing 3D Stretchable Surfaces from Single Images in Closed Form[END_REF][START_REF] Oswald | Fast and Globally Optimal Single View Reconstruction of Curved Objects[END_REF][START_REF] Perriollat | Monocular Template-based Reconstruction of Inextensible Surfaces[END_REF][START_REF] Salzmann | Linear Local Models for Monocular Reconstruction of Deformable Surfaces[END_REF][START_REF] Vicente | Balloon Shapes: Reconstructing and Deforming Objects with Volume from Images[END_REF]. SfT methods with real-time implementation are [START_REF] Collins | Realtime Shape-from-Template: System and Applications[END_REF][START_REF] Ngo | Template-based Monocular 3D Shape Recovery using Laplacian Meshes[END_REF].

Most of the SfT methods use the thin-shell isometric deformation model which implies that the geodesic distances between any two points on the object do not change due to the deformation. Isometry can be seen as local rigidity. Isometry is a very good approximation as most of the objects in nature undergo near-isometric deformations. Mathematically, it is also relatively easier to model isometry than other deformations. Our work focuses on isometry but we do explore other deformations as well and present a general modelling framework which makes it easier (and practical) to express various deformations.

Non-Rigid Structure-from-Motion

NRSfM (see figure 1.1b) is the generic name for a set of methods which perform the monocular 3D reconstruction of deformable objects from multiple images only. It is also called templatefree (or model-free) reconstruction in the literature. The inputs to NRSfM are multiple images and its output is the object's 3D shape for every image. In NRSfM, the rigidity constraint of SfM is replaced by constraints on the object's shape and deformation model. NRSfM methods were proposed initially with the low-rank shape basis [START_REF] Bregler | Recovering Non-rigid 3D Shape from Image Streams[END_REF], the trajectory basis [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF], isometry [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF][START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF][START_REF] Vicente | Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction[END_REF], inextensibility [Chhatkuli et al., 2016a] and elasticity [START_REF] Agudo | Sequential Non-Rigid Structure from Motion using Physical Priors[END_REF].

Existing methods suffer from one or several limitations amongst solution ambiguities, low accuracy, ill-posedness, inability to handle missing data and high computation cost. NRSfM thus still exists as an open research problem.

Based on the modelling framework, existing NRSfM methods can be divided into two main categories: 1) methods with statistics-based modelling and 2) methods with physicsbased modelling. physics-based modelling is the most recent. Most of the NRSfM methods use a statistics-based modelling. While statistics-based modelling does not take the object's nature into account, physics-based modelling is usually limited to thin-shell objects only.

Current Limitations

With this discussion, we want to emphasize the following limitations of existing SfT and NRSfM methods:

1) Methods with physics-based modelling are capable of handling more complex deformations than methods with statistics-based modelling and they proved to be very successful in SfT. However, physics-based modelling is seldom used in NRSfM.

2) Most of the existing thin-shell NRSfM methods work with the orthographic projection model which suffers from flip ambiguities. Therefore, the focus of the new techniques should be towards solving NRSfM using perspective projection as these solutions are more accurate.

3) Most of the methods deal with isometry or near-isometry which is relatively easier to model. Other deformations have been less explored.

4) Volumetric SfT has not been explored yet. There are some methods that recover the closed thin-shell of the object but a complete 3D reconstruction of a deformable object has not been achieved yet.

Now we discuss our contributions to SfT and NRSfM in detail.

Contributions

This thesis has three main contributions. Our first contribution is about solving NRSfM with the use of Riemannian geometry. This is a local formulation which means that the points on a surface can be reconstructed independently. The particles on an object are attached to each other and therefore the force causing deformation acts globally. However, the impact of the force is not necessarily uniform throughout the object which makes it interesting to study the deformations locally. This local formulation using Riemannian Geometry allows NRSfM to be expressed in terms of polynomial expressions whose variables are independent of the number of images under consideration. Our second contribution is about proposing a modelling framework for NRSfM and SfT (using differential geometry) which is general and makes it convenient to handle various kinds of deformations. These solutions are obtained in terms of the differential or local quantities expressed at a surface, as a set of Partial Differential Equations (PDE) that hold at each point on the surface. In order to solve the PDE, we convert them to algebraic expressions by replacing the differentials in the PDE with algebraic variables. Given enough constraints, these algebraic equations yield a local solution.

This solution can be obtained independently for each point. However, it may not always be possible to find such a solution. We discuss such conditions. The third contribution is a solution to SfT for volumetric objects.

To sum up, this thesis contributes in finding an answer to the following questions:

1) Is it possible to the extend the differential physics-based modelling of SfT [START_REF] Bartoli | Shape-from-Template[END_REF] for isometric deformations to NRSfM? Can we solve NRSfM locally from a PDE formulation?

2) Is it possible to the extend the local formulation of NRSfM to deformations other than isometry?

3) Can we reconstruct the entire volume of a deformable object in a model-based scenario?

A fundamental assumption. Our framework relates the 3D shapes using the inter-image warps. These are the functions that register one image to another. Registering wide-baseline images can be accomplished by Scale Invariant Feature Transform (SIFT) [START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF] which is a sparse-registration method. Dense or semi-dense registration can be achieved using SIFT Flow [START_REF] Liu | SIFT Flow: Dense Correspondence Across Scenes and Its Applications[END_REF] and DeepFlow [START_REF] Weinzaepfel | DeepFlow: Large Displacement Optical Flow with Deep Matching[END_REF] respectively. In order to register short-baseline images (for example, images from a video sequence), optical flow methods can be used. It usually gives a dense-registration. Some of the efficient methods are [START_REF] Brox | High Accuracy Optical Flow Estimation Based on a Theory for Warping[END_REF]Garg et al., 2013b;[START_REF] Sundaram | Dense Point Trajectories by GPU-accelerated Large Displacement Optical Flow[END_REF]. These methods yield a dense registration. The first and higher order derivatives of the registration can be computed from the keypoint correspondences (obtained from the previous methods) using [START_REF] Bookstein | Principal Warps: Thin-plate Splines and the Decomposition of Deformations[END_REF][START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. We make the assumption that in SfT and in NRSfM, the image registration can be established by using existing methods. Nevertheless, we only need to find these warps locally at each point. We use the first and the second-order derivatives of the warps in the first two contributions while in the third we only need the first order derivatives.

The second-order derivatives are usually noisy, we correct them using Schwarps [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. We show that the use of Schwarps is theoretically justified as well.

Contribution 1: Non-Rigid Structure-from-Motion using Riemannian geometry.

We present a solution to NRSfM using the thin-shell isometric deformation model, that we hereinafter denote as Iso-NRSfM. We model Iso-NRSfM using concepts from Riemannian geometry.

We model the object's 3D shape for each image as a Riemannian manifold and deformations as isometric mappings. We parametrise each manifold by embedding the corresponding retinal plane. This allows us to reason on advanced surface properties, namely the metric tensor and the Christoffel Symbols (CS), directly in retinal coordinates, and in relationship to the warps. These metric properties allow us to express the differential properties of surfaces, such as length, which are to be preserved under isometric deformations.

We formulate Iso-NRSfM locally with five variables which are functions of the first and the second-order derivatives of the inverse-depth of the surface undergoing deformation. We write the metric tensor and the CS in terms of these variables. We prove two new theorems

showing that for isometric deformations, the metric tensor and the CS may be transferred between views using only the warps. This limits the number of variables to only five for any number N of views.

First, we solved Iso-NRSfM in [START_REF] Parashar | Isometric Non-Rigid Shape-From-Motion in Linear Time[END_REF] (see figure 1.3) by assuming that the surface is planar in the infinitesimal neighbourhood of each point. This is the assumption of Infinitesimal Planarity (IP) which lets us get rid of the second-order derivatives in the expression of the CS. This limits the variables to only two. These variables correspond to the ratio of first-order derivatives of the inverse-depth function to the inverse-depth function.

We obtained a system of two cubics in two variables that involve the first and the second-order derivatives of the warps. This system holds at each point on the surface.

Then, we extended the solution to Iso-NRSfM without the assumption of IP. Our solution is obtained in two steps. 1) We solve for the first-order derivatives assuming that the secondorder derivatives are known. This is initialised using the solution with IP. 2) We solve for the second-order derivatives with the first-order derivatives obtained in the previous step.

We obtain a system of 4N -4 linear equations in three variables which is solved using Linear Least Squares (LLS). We iterate these two steps until the first-order derivatives converge.

The solution gives an estimate of the metric tensor field, and thus of the surface's normal field, in all views. The shape is finally recovered by integrating the normal field for each view. The proposed method has the following features. 1) It has a linear complexity in and accuracy, as we experimentally verified using synthetic and real datasets.

!!" #$%&' #(" #$%&'
Contribution 2: a unified framework for the 3D reconstruction of deformable objects using Cartan's connections. Unlike SfM which is modelled using algebraic projective geometry, there is no consensus on the modelling framework of NRSfM yet. We present a modelling framework for NRSfM using the differential geometry of surfaces. In mathematics, differential geometry is the basis to study the properties of curves and surfaces.

Recently, [START_REF] Fabbri | Multiview Differential Geometry of Curves[END_REF] proposed to solve SfM using the differential geometry of 3D curves. [START_REF] Fabbri | Multiview Differential Geometry in Application to Computer Vision[END_REF] proposed pose estimation and camera calibration using differential geometry of 3D curves. However, it is not widely used in modelling surfaces for deformable 3D reconstruction. Recently, [START_REF] Bartoli | Shape-from-Template[END_REF] proposed solutions to SfT using differential geometry. These solutions are analytic and therefore, they are very fast and need not be initialised. The success of our first contribution where we solve NRSfM with Riemannian geometry (which is a special case of differential geometry) is a motivation for us to use differential geometry to propose a general framework to model NRSfM and SfT. Riemannian geometry is limited to isometric (geodesic-distances preserving) and conformal (angles preserving) deformations whereas differential geometry is more general and can model a wider range of deformations.

This framework can therefore handle a wide variety of deformation models (including isometry) in a convenient way and therefore is a practical approach towards NRSfM. We model surfaces as smooth manifolds [START_REF] Lee | Introduction to Smooth Manifolds[END_REF] and extract differential properties of the surfaces using differential geometry. Our work is essentially based on Cartan's theory of connections [START_REF] Cartan | Sur les variétés à connexion affine, et la théorie de la relativité généralisée (première partie)[END_REF][START_REF] Cartan | Sur les varietes a connexion projective[END_REF][START_REF] Cartan | Espaces à connexion affine, projective et conforme[END_REF] devised using the differential geometry of smooth manifolds and the theory of moving frames [START_REF] Cartan | La théorie des groupes finis et continus et la géométrie différentielle traitées par la méthode du repère mobile[END_REF]. The connections were at first formalised as the entities that enable movement along the curves as a parallel transport i.e., the orientation of a vector on the curve or surface does not change when it moves in a closed curve. This is known as a Levi-Civita connection [START_REF] Lee | Riemannian Manifolds : An Introduction to Curvature[END_REF]. Cartan generalised the idea of connections as the entities that transport tangent plane vectors along the curve. Cartan's connections are not limited to parallel transport along the curves and therefore, they are more generic. In this thesis, we always work with Cartan's connections.

A moving frame is defined as a local frame of reference defined at a point on a surface (or a manifold). The differential properties of the surfaces such as lengths, angles and areas can be described using the moving frames. The connections are derived using the moving frame and its derivatives. They are related to the first, second and the third fundamental forms of the surfaces [START_REF] O'neill | Elementary Differential Geometry[END_REF]. Cartan proved that connections are necessary and sufficient to study the properties of 3D surfaces. From these properties, we derive differential constraints on the surfaces that lead to a solution to NRSfM and SfT. We solve these constraints algebraically.

We use moving frames and connections to design a modelling framework for the study of thin-shell deformable objects. Our framework has the following characteristics.

1) Our framework relies on the Infinitesimal Linearity (IL) assumption [START_REF] Kock | Synthetic Geometry of Manifolds[END_REF].

Under this assumption, any smooth deformation may be considered to be linear in the in-1.3. CONTRIBUTIONS finitesimal neighbourhood of a point while globally it could still be non-linear. This allows us to express the moving frame and the connections in terms of two variables (the first-order derivatives of the inverse-depth) only.

2)

We prove a theorem stating that connections on any two surfaces can be related to each other for any kind of smooth (IL) deformation they undergo. This allows the number of variables to be only two for any number of views used in the reconstruction.

3) We express the physical properties of surfaces (such as lengths, angles and areas) locally in terms of the moving frames. We express deformation constraints as the physical properties they preserve. For example, isometry preserves both lengths and angles. We express isometric deformation constraints as the preservation of lengths and angles defined using moving frames.

We express constraints for other deformations such as conformal (angles made by any three points on a surface do not change under deformation) and equiareal (areas are preserved under deformations). We propose a deformation that is a combination of anisotropic scaling (along surfaces' frame-basis) and a conformal deformation. We call it the skewless deformation. We explain it further in chapter 5. Moving frames and connections are the generalisation of the concepts of metric tensors and CS from Riemannian geometry. We use them to express differential constraints in terms of PDE. The manipulation of these constraints with or without the IL assumption leads to reconstruction equations that are also PDE. IL is not necessary to find these equations, however we use it to simplify the problem. These reconstruction equations may or may not have an algebraic solution. Here, an algebraic solution implies that the equations can be solved locally at each point. In this thesis, we solve the equations with possible algebraic (or local) solution.

pressing the deformation constraints in terms of the moving frame and using our theorem of transfer of connections, we present solutions to deformable 3D reconstruction for various deformations like isometry, conformity, skewless or equiareal. The solution to NRSfM under a) isometric/conformal deformations is given by solving a system of two cubics, b) skewless deformations is given by solving a system of two septics (exploiting the first and second-order derivatives of the warps) in terms of two variables only.

We show that Iso-NRSfM (our first contribution) can be obtained using this framework as well. In this solution, we chose isometry to be solved as conformity as it makes the problem simpler to solve.

Our framework is directly extended to SfT. The existing solutions to isometric and conformal SfT [START_REF] Bartoli | Shape-from-Template[END_REF] can be derived using this framework. We obtain an SfT solution to a) isometric/conformal and equiareal deformation as two linear expressions, b) skewless deformation as a system of two cubics in terms of two variables. These expressions exploit the first and the second-order derivatives of the warps.

Due to our theorem of transfer of connections across smooth surfaces using the inter-image warps, we propose a solution to SfT which is independent of the deformation constraints and only imposes deformation to be locally smooth. SfT has previously been solved under the assumption of smooth deformation in [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF]. We compare it with our results of SfT. [START_REF] Salzmann | Deformable Surface Tracking Ambiguities[END_REF] discussed that such a solution is not well-posed, however, we show that our solution to smooth deformations is well-posed. We discuss the reasons which make it well-posed.

Summing up, the proposed framework has the following features. 1) It is a unified modelling framework for NRSfM (and SfT) using differential geometry assuming IL, which can be extended to various deformations. 2) It brings the solutions to isometric/conformal and skewless NRSfM as a set of two cubic and septic polynomials respectively in terms of two variables for any number of views. 3) It brings the solutions to isometric/conformal and equiareal SfT using a linear system of two equations in two variables only. 4) It brings the solution to skewless SfT by solving a cubic system of two equations in two variables only. 5)

It also brings a solution to SfT for smooth deformations which is also a system of two linear equations in two variables only.

Contribution 3: Shape-from-Template for volumetric objects. As discussed earlier, existing SfT methods are thin-shell SfT essentially, as they are designed for thin objects such as a piece of paper. However, while thin-shell SfT handles thicker objects such as the woggle of figure 1.5 or a foam ball, it does not fully exploit the strong constraints induced by the object's non-empty interior.

We bring SfT one step further by introducing volumetric SfT, defined as an SfT method which uses a deformation constraint on the object's outer surface and interior. An example is shown in figure 1.5. Volumetric SfT reconstructs the object's interior deformation, which is not reconstructed by thin-shell SfT, and reconstructs the object's outer surface more accurately than thin-shell SfT thanks to the stronger deformation constraint it uses. Volumetric SfT is challenging as only the front part of the object's surface is visible in the image: the object's back surface and interior have to be inferred with no direct visual observations. It is important to note that strictly speaking, isometry leads to rigidity in volumes. Only rigid volumetric objects can preserve geodesic-distances while undergoing deformation. We For instance, [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF] uses isometric constraints on the object's visible (front) surface and reconstructs the object partially, while [ Östlund et al., 2012] uses isometric constraints on the object's whole closed outer surface and reconstructs it entirely. Volumetric SfT uses deformation constraints on the object's surface and interior. This greatly improves reconstruction accuracy and facilitates reconstruction of the object's interior. In this example, the thin-shell SfT methods [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF][START_REF] Östlund | Laplacian Meshes for Monocular 3D Shape Recovery[END_REF] reach a 3D error of 20 mm and 13 mm respectively on the visible surface, while the proposed volumetric SfT method reaches a 3D error of 7 mm. It reconstructs the non-visible (back) surface, for which no visual data is available, with a 3D error of 17 mm.

propose to instantiate volumetric SfT using the As-Rigid-As-Possible (ARAP) deformation model (a relaxation of isometry), which has been used extremely successfully in Computer

Graphics [START_REF] Sorkine | As-rigid-as-possible Surface Modeling[END_REF][START_REF] Zhang | Skeleton Based As-Rigid-As-Possible Volume Modeling[END_REF]. The ARAP model maximises local rigidity while penalising stretching, sheering and compression. More specifically, ARAP has been widely used to perform mesh editing of animated characters [START_REF] Zhou | Large Mesh Deformation using the Volumetric Graph Laplacian[END_REF][START_REF] Zollhöfer | GPU based ARAP Deformation using Volumetric Lattices[END_REF] because the resulting deformations locally preserve the object's structure.

Contrary to thin-shell SfT, volumetric SfT is largely unexplored. Recently, [START_REF] Innmann | VolumeDeform: Real-Time Volumetric Non-rigid Reconstruction[END_REF] proposed a method that reconstructs the closed thin-shell surface of the deformable object in real-time. This method is named "VolumeDeform" however it does not reconstruct the interior of the object. The closest method to volumetric SfT is perhaps [START_REF] Vicente | Balloon Shapes: Reconstructing and Deforming Objects with Volume from Images[END_REF], where SfT has been combined with silhouette-based reconstruction.

However this method requires stronger image cues, including silhouette and point correspondences, and recovers two-way ambiguous shape solutions. In contrast, we solve volumetric SfT without restricting the topology of the object and using the perspective camera. By using ARAP, our method preserves the object's interior structure while jointly reconstructing the deformation of the object's full outer surface and interior, as illustrated in figure 1.6. ARAP volumetric SfT involves solving a non-convex constrained variational optimisation problem.

We discretise the object's volume and relax the constraints to convert the variational problem into an unconstrained non-linear least-squares optimisation problem. This problem can then be solved with standard numerical solvers such as Levenberg-Marquardt. We propose two Thesis layout. We have divided this thesis into 7 chapters. We discuss the state of the art in chapter 2, mathematical preliminaries in chapter 3. Chapters 4, 5 and 6 give our first, second and third contributions. Chapter 7 presents our conclusions and perspectives for future work.

Chapter 2

Related Work

In this chapter we discuss the existing works on SfT and NRSfM that use motion as a visible cue in two sections. We sub-categorise these methods based on the constraints they use.

Most of these methods are designed for thin-shell objects but we also discuss the works that are related to volumetric objects.

Shape-from-Template

The SfT methods were introduced much later than NRSfM but they evolved quickly. Now there are stable and real-time SfT methods [START_REF] Collins | Realtime Shape-from-Template: System and Applications[END_REF][START_REF] Ngo | Template-based Monocular 3D Shape Recovery using Laplacian Meshes[END_REF].

In general, SfT uses a 3D template of a thin-shell object. This is a very strong prior and makes SfT a better-posed problem than NRSfM. We classify current SfT methods into two categories: initialisation and refinement methods. The initialisation methods are the ones which achieve a fast solution to SfT using deformation constraints. They do not employ a heavy non-convex optimisation to minimise a cost which consists of a set of constraints such as deformation, smoothness or reprojection which is the case with refinement methods.

Refinement methods are computationally expensive but more accurate. However, they need to be initialised. The performance of these methods depends on the accuracy of initialisation.

A good initialisation can significantly reduce their computation time. We recall that the current SfT solutions are for thin-shell objects only, SfT for volumetric objects has not been proposed however, we discuss some of the works that use non thin-shell models.

Most of the SfT methods exploit physics-based modelling. Most of them use isometry as a physical prior but there are some solutions that use elasticity [START_REF] Haouchine | Single View Augmentation of 3D Elastic Objects[END_REF][START_REF] Malti | Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity[END_REF], the particle model [ Özgür and Bartoli, 2016] or smoothness [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF]. We organise these two categories of initialisation and refinement into methods that employ isometric and non-isometric constraints. We now discuss these two categories of methods.

Thin-Shell Initialisation Methods

Isometric Constraints

Most of the initialisation methods use isometry as a deformation prior. Isometry is a physical prior on deformation which can be seen as local rigidity. Isometry preserves the geodesic distances between points on a surface undergoing deformations. Therefore stretching or shrinking of the surfaces is not allowed. Inextensibility is a relaxation of isometry. It means that the Euclidean distances between the neighbouring points on the deformed surfaces are always lower than or equal to the corresponding geodesic distances on the original surface. tensibility constraints on the points (P 1 , P 2 ) on S 1 and (Q 1 , Q 2 ) on S 2 in terms of distances between the points can be written as

Q 2 -Q 1 g = P 2 -P 1 g isometry constraint Q 2 -Q 1 2 ≤ P 2 -P 1 g inextensibility constraint (2.1)
where . 2 represents the Euclidean distance and . g represents the geodesic distance between two points on a surface. Therefore, we can see that intextensibility is a relaxed form of isometry. However, expressing geodesics on an arbitrary surface is not easy. Therefore, most of the methods approximate the geodesics with euclidean distances by assuming that the points are very close to each other. For example, the geodesic of (Q 1 , Q 2 ) on S 2 can be written as the Euclidean distance between them given that Q 2 is close enough to Q 1 . The sense of closeness or neighbourhood of these points are defined by the methods. Inextensibility needs to be combined with maximum depth in order to prevent the reconstructed surface from shrinking. This is called as Maximum Depth Heuristics (MDH).

We now discuss the initialisation methods that use inextensibilty and isometry constraints in two sections. [START_REF] Perriollat | Monocular Template-based Reconstruction of Inextensible Surfaces[END_REF] was the first method to model isometry using the inextensibility constraint (2.1). It is based on the MDH. It finds a solution to SfT by maximising depth heuristically while imposing inextensibility constraints. Figure 2.2 shows two surfaces related with an isometric mapping. Consider Q 1 at a distance μ 1 from the camera. Assuming that Q 2 is a neighbouring point of Q 1 , it can be parametrised with the angle α 12 between their

Inextensibility Constraints

(Q 1 , Q 2 ) on S 2 .
The sightlines of the two points (Q 1 , Q 2 ) from the camera C pass through (q 1 , q 2 ) on the image plane.

sightlines from the camera C. Therefore, we can write

Q 1 = ⎡ ⎢ ⎣ μ 1 0 0 ⎤ ⎥ ⎦ , Q 2 = ⎡ ⎢ ⎣ μ 2 cos (α 12 ) μ 2 sin (α 12 ) 0 ⎤ ⎥ ⎦ . (2.2)
Using this parametrisation of the points, the inextensibility constraint in equation (2.1) gives the upper bound of μ 1 as

μ 1 ≤ d 12 sin α 12 , (2.3)
where d 12 is the geodesic distance between the 3D points P 1 and P 2 in the template. This is the upper bound on the depth of each point. It is chosen to be the minimum upper bound of all the neighbouring points such that the inextensibility constraint (2.1) is satisfied. [START_REF] Salzmann | Linear Local Models for Monocular Reconstruction of Deformable Surfaces[END_REF] made an improvement by modelling this problem as an Secondorder Cone Program (SOCP) which can be globally solved using convex optimization. The method parametrises 3D points as the back-projection of 2D image points. Therefore any point Q i can be expressed as

Q i = z i q i 1 , (2.4)
where z i is the depth at the i th point and q i is the normalized image point. The idea is to maximise the sum of all the depths z i such that the inextensibility constraint (2.1) is satisfied.

This method uses a learned space of deformations using linear local models for small patches.

This limits the applicability of this method to surfaces whose linear local models are known.

However, it shows good performance when there is enough perspective in the images. [START_REF] Brunet | Monocular Template-based 3D Surface Reconstruction: Convex Inextensible and Nonconvex Isometric Methods[END_REF] proposed an initialization method based on inextensibility constraints solved using MDH. It used a parametric representation of surfaces using cubic B-splines [START_REF] Dierckx | Curve and Surface Fitting with Splines[END_REF] which reduces the dimensionality of the problem and provides a solution faster.

[ [START_REF] Ngo | Template-based Monocular 3D Shape Recovery using Laplacian Meshes[END_REF] proposed a modified approach, where the method uses a laplacian smoothness prior along with inextensibility constraints. The laplacian of the template is calculated which is assumed to be preserved under the deformation. The laplacian is linearly parametrised and therefore, the problem can be solved using LLS. It is used as an initialisation method to the real-time solution to SfT proposed in [START_REF] Ngo | Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture[END_REF].

The above-mentioned methods use inextensibility constraints which is a relaxation on isometry and therefore, it is not a strict physical constraint. A more accurate representation of deformation constraints is possible by using differential modelling which we discuss next.

First-Order Differential Isometric Constraints

Recently, [START_REF] Bartoli | Shape-from-Template[END_REF] proposed a local analytical solution to SfT using a warp and first-order differential isometric constraints. It shows that SfT is a well-posed problem for isometric deformations. It expresses the constraints in terms of first-order PDE and finds an algebraic solution to it. Since the method is analytical, it is very fast. However, it suffers with instabilities under near-affine conditions. [Chhatkuli et al., 2016b] proposed an improvement on these solutions to find an analytical stable solution to depth using the gradient of the depths which were otherwise discarded in [START_REF] Bartoli | Shape-from-Template[END_REF].

The success of these SfT methods inspires us to extend the physics-based differential modelling of deformation to NRSfM as well. We use differential geometry to formulate the NRSfM and SfT problems in terms of PDE and we find algebraic solution to them.

Non-Isometric Constraints

[ [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF] proposed to solve SfT by using only smoothness as a deformation prior. The solution is unique and obtained by solving an LLS problem. It finds the solution to SfT using reprojection constraints and a smoothness constraint for a fixed scale. The problem with this method is that smoothness is a very weak constraint which can make this method unstable.

[ [START_REF] Bartoli | Shape-from-Template[END_REF] also proposed an analytical solution to conformal SfT. This solution was also formulated using PDE and solved algebraically. Even though this method suffers form instabilities, it usually performs better or as well as isometric SfT [START_REF] Bartoli | Shape-from-Template[END_REF].

Therefore, differential modelling proves to be a good tool for non-isometric deformations as well.

Thin-Shell Refinement Methods

These methods formulate global deformation constraints and solve them by using a nonconvex optimisation. Therefore, these methods need to be initialised. The initialisation methods discussed in the previous section can be used to initialise these methods. In fact, initialisation methods should always be combined with refinement methods in order to get the best possible reconstruction. The accuracy of initialisation methods makes the refinement significantly faster. [Chhatkuli et al., 2016b] showed results by initialising [START_REF] Brunet | Monocular Template-based 3D Surface Reconstruction: Convex Inextensible and Nonconvex Isometric Methods[END_REF] with their result. They achieved an almost real-time reconstruction. [START_REF] Collins | Realtime Shape-from-Template: System and Applications[END_REF] made an improvement on this and achieved a real-time reconstruction. We now discuss the refinement methods that use isometric and non-isometric constraints in the next two sections. [START_REF] Brunet | Monocular Template-based 3D Surface Reconstruction: Convex Inextensible and Nonconvex Isometric Methods[END_REF] proposed the first solution to SfT which optimises a statistically optimal cost. The cost is composed of three errors: 3D back-projection, differential isometric constraints and smoothness. The 3D back-projection error (E reprojection ) accounts for the difference when the 3D points of the deformed shape project back to the corresponding input image points. The differential isometric constraint error (E isometry ) forces that isometry holds at infinitesimal level. It ensures that the deformed shape is isometric. The smoothness error (E smooth ) forces the solution to be smooth. This problem is non-convex and relies on iterative local optimization such as Levenberg-Marquardt which requires to be initialised and has a high computation time. The cost is written as

Isometric Constraints

Cost = E reprojection + l isometry E isometry + l smooth E smooth , ( 2.5)
where the two parameters l isometry and l smooth are weights to the isometric and smoothness constraints and need to be tuned. We use a similar cost in order to find a solution to volumetric SfT.

[ [START_REF] Yu | Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video[END_REF] introduced a temporal smoothness constraint in addition to the above mentioned constraints in order to improve the refinement. [START_REF] Malti | A Pixel-Based Approach to Template-Based Monocular 3D Reconstruction of Deformable Surfaces[END_REF][START_REF] Ngo | Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture[END_REF] model deformation as conformal and use the pixel intensity error instead of the reprojection error. [START_REF] Ngo | Dense Image Registration and Deformable Surface Reconstruction in Presence of Occlusions and Minimal Texture[END_REF] is initialised with [START_REF] Ngo | Template-based Monocular 3D Shape Recovery using Laplacian Meshes[END_REF] and handles occlusions and poorly textured surfaces.

Non-Isometric Constraints

[ Özgür and Bartoli, 2016] proposed a solution to SfT by expressing the object as a set of particles where deformation acts as a set of forces on it. It uses deformation and reprojection constraints and finds a solution by evolving particles to achieve a global equilibrium due to the action of various forces (including gravity). It uses boundary points to fix the solution.

[ [START_REF] Haouchine | Single View Augmentation of 3D Elastic Objects[END_REF][START_REF] Malti | Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity[END_REF] proposed a solution to SfT for extensible surfaces by modelling deformation with elasticity. The idea is to minimise the stretching energy such that the reprojection constraint and boundary points are satisfied.

SfT methods using non-isometric constraints are mostly solved using non-convex optimisation.

Shape-from-Template for Volumetric Objects

Volumetric objects have non-zero thickness. SfT methods using elasticity [START_REF] Haouchine | Single View Augmentation of 3D Elastic Objects[END_REF][START_REF] Malti | Monocular Template-Based 3D Reconstruction of Extensible Surfaces with Local Linear Elasticity[END_REF] require the surface model to include thickness, which must however be 'small' so that extension along normal direction may be neglected. In continuum mechanics, this means that the thickness is at least ten times smaller than the object's largest dimension.

Therefore, these methods are categorised as thin-shell methods. They require one to provide the Young modulus of the object's material and, more importantly, boundary conditions expressed as a set of known 3D point coordinates, which may restrict their applicability.

A related goal was pursued in [START_REF] Vicente | Balloon Shapes: Reconstructing and Deforming Objects with Volume from Images[END_REF] where a silhouette-based method was combined with SfT. The template is also reconstructed from a reference image using a silhouette-based method inspired from [START_REF] Oswald | Fast and Globally Optimal Single View Reconstruction of Curved Objects[END_REF]. This method reconstructs objects that have a plane of symmetry parallel to the image plane and does not infer concavities, which is also a limitation of most silhouette-based methods [START_REF] Oswald | Fast and Globally Optimal Single View Reconstruction of Curved Objects[END_REF][START_REF] Prasad | Single View Reconstruction of Curved Surfaces[END_REF]]. The template is then deformed using a data term based on silhouette, area and orthographic reprojection constraints. The deformation model extends thin-shell isometry by placing virtual nodes in the object's interior, with the objective of preserving the object's volume.

Relationship to our Work

In chapter 4, we propose a modelling framework for SfT using differential geometry. This framework is coherent with methods based on differential modelling [START_REF] Bartoli | Shape-from-Template[END_REF] and is general, therefore, other deformation models can be used. We also propose a solution to

SfT assuming that the deformation is smooth. This means that SfT can be solved analytically for any kind of deformation without explicitly modelling the deformations.

In chapter 5, we propose volumetric SfT which, in contrast to thin-shell SfT, recovers the deformation of the object's outer surface and interior. It formulates the deformation globally in terms of a cost function and minimises it using non-convex optimisation.

Non-Rigid Structure-from-Motion

The first solution to NRSfM for thin-shell objects was proposed in [START_REF] Bregler | Recovering Non-rigid 3D Shape from Image Streams[END_REF] which modelled deformation using a low-rank shape-basis. It assumes that the shape of an object can be represented as a linear combination of a low-dimensional shape-basis. We discuss the two categories of NRSfM methods based on the modelling framework: statistics-based modelling and physics-based modelling.

Methods with Statistics-based Models

Starting from the work of [START_REF] Bregler | Recovering Non-rigid 3D Shape from Image Streams[END_REF], the low-rank shape-basis has been the most commonly used shape prior in NRSfM. It is a statistical prior on a set of 3D point correspondences expressed in terms of point correspondences in images that forces the matrix containing these correspondences to have a fixed low-rank. This matrix can be further decomposed into a shape-basis and their weights. Inspired from this method, [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF] proposed NRSfM which modelled deformation as a set of trajectory basis. We discuss statistics-based methods under these two categories.

Low-Rank Shape-Basis

For N images, the image observation matrix consisting of the P matched point correspondences across the images is written as .6) where (u j i , v j i ) represents the image coordinates of the i th point on the j th image. Any shape can be written as a linear combinations of the K shape-basis B i . Therefore the shape S i of the i th image can be written as

W = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ u 1 1 . . . u 1 P v 1 1 . . . v 1 P . . . . . . . . . u N 1 . . . u N P v N 1 . . . v N P ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ , ( 2
S i = K t=1 l i t B t , (2.7)
where each shape-basis B t is 3 × P matrix and l i t is the set of weights that decide the scale. Projecting these shape-basis on images under a scaled orthographic projection, we can write the observation matrix as

W = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ u 1 1 . . . u 1 P v 1 1 . . . v 1 P . . . . . . . . . u N 1 . . . u N P v N 1 . . . v N P ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ = ⎡ ⎢ ⎢ ⎣ l 1 1 R 1 . . . l 1 K R 1 . . . . . . . . . l N 1 R N . . . l N K R N ⎤ ⎥ ⎥ ⎦ ⎡ ⎢ ⎢ ⎣ B 1 . . . B k ⎤ ⎥ ⎥ ⎦ = QB, (2.8)
where R i consists of the first two rows of the camera projection matrix. The goal is to decompose W into two matrices Q and B which contain the information of the scale and the set of shape-basis respectively. Once Q and B are found any shape can be written using equation (2.7). Q can be further decomposed in order to find the pose.

[ [START_REF] Bregler | Recovering Non-rigid 3D Shape from Image Streams[END_REF] used Singular Value Decomposition (SVD) to decompose W into Q and B by fixing K as a low positive integer. Q and B are called the coefficient matrix and shape-basis matrix respectively. This problem is non-convex and suffers from ambiguities in the solution to the shape-basis. [START_REF] Del Bue | Non-rigid Structure from Motion using Non-Parametric Tracking and Non-linear Optimization[END_REF] proposed a non-linear refinement to improve the solution. However in order to deal with these ambiguities, different kinds of priors were proposed by various methods:

1) [START_REF] Del Bue | A Factorization Approach to Structure from Motion with Shape Priors[END_REF] proposed to use shape-basis priors to constrain the B matrix. The idea is to estimate the shape-basis for some known 3D shapes and use them along with the unknown shape basis in order to estimate the shapes for all the images. This means that some of the basis in B are already calculated using few known 3D shapes and W is decomposed in a way that these known shape bases do not change. Given that the first b elements of B are known, the shape prior L according to [START_REF] Del Bue | A Factorization Approach to Structure from Motion with Shape Priors[END_REF] is given by

L = N ⎡ ⎢ ⎢ ⎣ B 1 . . . B b ⎤ ⎥ ⎥ ⎦ = NB.
(2.9)

The joint decomposition of W and L into Q, N and B improves the conditioning of the shape. [START_REF] Tao | Non-rigid Structure from Motion with Diffusion Maps Prior[END_REF] extended this idea to allow shapes to have non-linear deformations by allowing L to be non-linear. B is found from a manifold whose embeddings are learnt from a representative training dataset of the deformable object under consideration. This is not a traditional NRSfM method (as it heavily relies on training to find B), however it does highlight the success of manifold learning.

2) [START_REF] Torresani | Tracking and Modeling Non-Rigid Objects with Rank Constraints[END_REF] solves this problem by adding a shape regulariser term that imposed spatial smoothness in the observed shape using an iterative optimisation.

3) [START_REF] Olsen | Implicit Non-Rigid Structure-from-Motion with Priors[END_REF] solves this problem by optimising the shape-basis using spatial and temporal smoothness priors.

4) [START_REF] Bartoli | Coarse-to-fine Low-rank Structure-from-Motion[END_REF] uses deformation modes and closeness of points in the mean shape as priors. It solves the ambiguity by using a low-rank coarse-to-fine shape model which prioritises the deformation modes that give the coarsest deformations.

5) [START_REF] Fayad | Piecewise quadratic reconstruction of non-rigid surfaces from monocular sequences[END_REF] proposed to model the deformations (patch-wise) with quadratic models where the linear modes allow shearing and stretching, quadratic modes allow bending and the mixed modes allow the twisting of the surfaces. These modes are optimised for each (overlapping) patch using the temporal smoothness priors. Then they are stitched together to obtain a global surface.

5) [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF] shows that an ambiguity in the SVD does not necessarily lead to an ambiguous reconstruction. They introduced a correction matrix that can be used to obtain a unique solution. They showed that the camera orthonormality conditions are sufficient to find the correction matrix. [START_REF] Dai | A Simple Prior-free Method for Non-rigid Structure-from-Motion Factorization[END_REF] proposed a more efficient solution to find this matrix by minimising the trace of the shape-basis. Trace-minimisation is a tighter constraint than rank-minimisation (used in [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF]). [Garg et al., 2013a] also used trace minimisation of the correction matrix in order to find the low-rank shapes. They formulated the problem as a global variational energy minimisation problem. The goal is to minimise the trace of the shape-basis, reprojection error and the total spatial variation. This method performs dense reconstruction and yields good results for faces. However, this algorithm is computationally very expensive.

Most of these methods express shapes as a linear combinations of shape-basis. This forces the deformations to be linear. Therefore these methods are applicable to simple deformations such as a talking face. They do not cope up with the larger deformations such as a walking man or a tree moving due to the wind unless more constraints are provided.

Low-Rank Trajectory-Basis

[ [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF] proposed to replace the shape-basis by a trajectory-basis in the formulation suggested by [START_REF] Bregler | Recovering Non-rigid 3D Shape from Image Streams[END_REF]. This method can easily reconstruct larger deformations than the above-mentioned methods. The fundamental idea behind this method is to express the trajectory of a point on an image as a linear combination of the trajectorybasis. The trajectory-basis are obtained using a Discrete Cosine Transformation (DCT)

basis. [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF] proposed to decompose the image observation matrix W in equation (2.6) into R and T using SVD, where R contains the R i (the first two rows of camera projection matrix for the i th image) and T is the trajectory-basis matrix.

[ [START_REF] Gotardo | Kernel Non-Rigid Structure from Motion[END_REF] proposed a solution that uses additional higher frequencies of the DCT basis in order to estimate large deformations better than [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF].

This method first estimates the 3D using trajectory-basis. It then estimates the shape-basis by applying a kernel transformation which generalises the inner product with a radial basis function.

[ [START_REF] Torresani | Non-rigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors[END_REF]] also uses a parametric representation of shape and trajectorybasis and finds 3D by estimating these parameters using Probabilistic Principal Component Analysis.

These methods can handle large deformations better than the low-rank shape model methods but they still need video-sequences or short-baseline data to achieve good results.

Table 2.1 summarises the statistics-based NRSfM methods. All these methods use the orthographic projection model. All these methods solve NRSfM with a non-convex formulation except [START_REF] Dai | A Simple Prior-free Method for Non-rigid Structure-from-Motion Factorization[END_REF] which uses a convex relaxation.

Most of the statistics-based methods use orthographic camera projection which may lead to flip ambiguities in the reconstruction. Also, these methods are designed for short-baseline images and therefore, they cannot handle large deformations. For good results, these methods need a large number of views.

Method Type of basis Additional priors

Complexity of basis [START_REF] Bregler | Recovering Non-rigid 3D Shape from Image Streams[END_REF]] Shape -Linear [START_REF] Del Bue | A Factorization Approach to Structure from Motion with Shape Priors[END_REF] Shape Shape Linear [START_REF] Tao | Non-rigid Structure from Motion with Diffusion Maps Prior[END_REF] Shape Shape Non-linear [START_REF] Dai | A Simple Prior-free Method for Non-rigid Structure-from-Motion Factorization[END_REF] Shape -Linear [START_REF] Torresani | Tracking and Modeling Non-Rigid Objects with Rank Constraints[END_REF] Shape Spatial smoothness Linear [START_REF] Olsen | Implicit Non-Rigid Structure-from-Motion with Priors[END_REF] Shape Spatio-temporal smoothness Linear [START_REF] Bartoli | Coarse-to-fine Low-rank Structure-from-Motion[END_REF] Shape Deformation modes, point closeness Linear [START_REF] Akhter | Nonrigid Structure from Motion in Trajectory Space[END_REF] Trajectory -Linear [START_REF] Torresani | Non-rigid Structure-from-Motion: Estimating Shape and Motion with Hierarchical Priors[END_REF] Shape, trajectory Spatio-temporal smoothness Linear [START_REF] Gotardo | Kernel Non-Rigid Structure from Motion[END_REF] Shape, trajectory -Non-Linear

Methods with Physics-based Models

For a long time, the focus of the research community has been on the statistics-based modelling of deformation. Physics-based modelling for NRSfM is rather recent. In general, these methods use the physical properties of thin-shell objects to model deformation as in SfT.

They can handle more complex deformations and work with fewer images than methods with statistics-based modelling. Most of these methods, for example [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF][Chhatkuli et al., , 2016a;;[START_REF] Collins | Locally Affine and Planar Deformable Surface Reconstruction from Video[END_REF][START_REF] Russell | Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes[END_REF][START_REF] Taylor | Non-rigid Structure from Locally-rigid Motion[END_REF][START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF][START_REF] Vicente | Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction[END_REF] use isometry as a deformation model except [START_REF] Agudo | Sequential Non-Rigid Structure from Motion using Physical Priors[END_REF] which models deformations using elasticity. [START_REF] Agudo | Sequential Non-Rigid Structure from Motion using Physical Priors[END_REF] first reconstructs the surface by assuming that there are no deformation acting on it (basically SfM) and then uses this solution the predict the deforming shapes. Therefore, this method resembles SfT even though it is presented as a NRSfM method by the authors.

[ [START_REF] Collins | Locally Affine and Planar Deformable Surface Reconstruction from Video[END_REF][START_REF] Taylor | Non-rigid Structure from Locally-rigid Motion[END_REF] approximate isometry with a rigid rotation and translation at a local (or piecewise) level. For example, [START_REF] Collins | Locally Affine and Planar Deformable Surface Reconstruction from Video[END_REF][START_REF] Taylor | Non-rigid Structure from Locally-rigid Motion[END_REF] solved NRSfM by expressing isometry as local rigidity with an orthographic camera projection. [START_REF] Taylor | Non-rigid Structure from Locally-rigid Motion[END_REF] finds sets of 3 rigid points reconstructed using SfM whereas [START_REF] Collins | Locally Affine and Planar Deformable Surface Reconstruction from Video[END_REF] performs automatic clustering of point sets. These methods rely on finding the 3 points in a close neighbourhood in order to make sure that the assumption of rigid transformation holds. Another approximation was made by [START_REF] Russell | Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes[END_REF][START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF], they exploited isometry as piecewise-rigidity. [START_REF] Russell | Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes[END_REF] computes fundamental matrices [START_REF] Hartley | Multiple View Geometry in Computer Vision[END_REF] to obtain the solution to surface normals. However, fundamental matrices may be unstable in case of small patches.

An improvement on [START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF] was proposed by [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF] which defines isometric constraints between points that are infinitesimally close to each other while [START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF] defines these constraints on a small patch (assuming piecewiserigidity). Both of them assume perspective projection. [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF] assumes the surface to be a set of infinitesimal planes while [START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF] assumes the surfaces to be represented as a set of planar patches. They then obtain a homography between the corresponding normalised image points of the planes and use homography decomposition [START_REF] Malis | Deeper understanding of the Homography Decomposition for Visionbased Control[END_REF] to obtain the surface normals. The surface normals thus obtained have a two-fold ambiguity which is resolved by using spatial smoothness in [START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF] while [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF] uses additional views to disambiguate the normals. The surface normals thus obtained are integrated to obtain an up-to-scale representation of the surface.

[ [START_REF] Vicente | Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction[END_REF] expressed the deformation using isometric constraint (see equation ( 2.1)) and proposed a global solution to NRSfM by performing a discrete nonconvex optimisation based on energy minimization of isometry constraints of all the points considered. They provided solutions for both orthographic and perspective projection. The method does not yield a globally optimal solution. The solution assuming orthographic projection suffers from flip ambiguities.

Recently, [Chhatkuli et al., 2016a] proposed a solution to NRSfM by modelling deformation with the inextensibility constraint (2.1) using the perspective camera model. Relaxing isometry to inextensibility makes the problem convex and a globally optimal solution is obtained using second-order cone programming. The problem is formulated using the MDH [START_REF] Perriollat | Monocular Template-based Reconstruction of Inextensible Surfaces[END_REF] where the goal is to maximise the point depth for each image correspondence in the retinal frame under the inextensibility constraints. The point depths are bounded by the sum of the unknown template distances in order to make sure that a global minimum is reached.

[ [START_REF] Agudo | Good Vibrations: A Modal Analysis Approach for Sequential Non-Rigid Structure from Motion[END_REF] solves NRSfM using a mix of physics-based deformation prior and statistics-based priors. [START_REF] Agudo | Sequential Non-Rigid Structure from Motion using Physical Priors[END_REF] solves NRSfM for potentially extensible surfaces.

It requires the surface to be represented as a thin-plate (a surface with considerable thickness), however it does not reconstruct the volumes. It represents the object's mechanics in terms of in-plane stress and out of plane bending. It models the deformation using Navier's equation which are solved by using Finite Element Method. However, this method works only for video sequences. [START_REF] Agudo | Good Vibrations: A Modal Analysis Approach for Sequential Non-Rigid Structure from Motion[END_REF][START_REF] Agudo | Sequential Non-Rigid Structure from Motion using Physical Priors[END_REF] require an initialisation which is obtained by using SfM on the first few frames. Therefore these methods resemble SfT in their approach rather than NRSfM.

Table 2.2 summarises the NRSfM methods using physical priors. The use of physical priors in NRSfM is limited to isometry. Other deformation priors are unexplored. Most of these methods are applicable to both short-baseline and wide-baseline data except [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF][Chhatkuli et al., , 2016a] ] which break on short-baseline data.

NRSfM methods based on physics-based modelling of deformations (for example, isometry can be modelled with a rotation and translation at a local level, which incorporates camera's rotation and translation as well) do not decouple camera motion and deformation unlike the NRSfM methods based on statistics-based modelling. The motion of the object or the camera are both treated as a deformation and hence, camera-motion is not recoverable. A deformation is therefore regarded as the change in the object from one view to another.

Relationship to our Work

Most of the current NRSfM methods suffer from ambiguities and poor performance in reconstruction. Deformations other than isometry are very rarely explored. The success of

Method

Physical prior Constraint Camera model [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF] Local rigidity Local Perpective [START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF] Local rigidity Piecewise Perspective [START_REF] Russell | Video Pop-up: Monocular 3D Reconstruction of Dynamic Scenes[END_REF] Local rigidity Piecewise Orthographic [START_REF] Taylor | Non-rigid Structure from Locally-rigid Motion[END_REF] Chapter 3

Mathematical Formulation

Notation

We define a set of rules that we follow in the next chapters in order to make them easier to understand. We describe here the general rules and the exceptions we make.

1)

We use small-case Latin letters to denote scalars. Exception: In chapter 6, we use ρ and α to denote scalars.

2) Bold Latin letters denote 2D and 3D vectors.

3) We use Greek letters to denote functions. We express the inverse-depth function in chapters 4, 5 as β (in the case of planar surfaces) and α (in the case of non-planar surfaces).

Exception: In chapters 4 and 5, we use Γ to express CS and components of a connection respectively. In chapter 5, f is a multi-valued smooth function defined in R n with the coordinates f i and their respective basis as e i where i ∈ [1, ..., n].

4)

We use a subscript to index the images and a superscript to index the coordinates of a point.

5)

We use calligraphic letters for objects and images. In chapter 6, we use calligraphic letters for sets, and |A| for the size of set A.

6)

We use the operator J ϕ for the Jacobian of a function ϕ.

7)

In chapter 4, we use g to denote the metric tensor and Γ to denote the CS matrix.

8)

In chapter 5, differential 1-forms are represented as w. d is the operator for exterior differentiation. The origin is denoted as O.

9)

In chapters 4 and 5, we give our modelling for a pair of views. It straightforwardly generalises to any number of views. We consider two surfaces M i and M j , which are represented by images I i and I j . A point in I i is denoted by x and the corresponding one in I j by y. We name the points this way to avoid the subscripts in the equations. Similarly, a point on the surface M i is denoted by z and the corresponding point on M j by w.

10)

In chapter 6, the 3D points on the template and object are given by P and Q. Their corresponding points on the images are given by p and q respectively. A tetrahedron attached to a 3D point, for example, P is given as a set of 3D points (P n1 , P n2 , P n3 , P n4 ).

11)

In chapters 4 and 5 we omit x and just write the functions described at x as α i , β i and φ i (instead of α i (x), β i (x) and φ i (x) ) in order to make the equations compact.

12) In chapters 4 and 5, we also write ∂α i ∂x t as α i,t , chapters 4 and5, we write (k 1 , k 2 , k 3 , k 4 , k 5 ) as the expressions that represent the ratio of first and second-order derivatives of the inverse-depth of the surface M i to the inverse-depth, (p 1 , p 2 , p 3 ) which are known quantities on M i , written in terms of the secondorder derivatives and (c 1 , c 2 , c 3 , c 4 , c 5 , c 6 ) which represent the CS at M i . On M j , we write these expressions with a bar as ( k1

∂β i ∂x t as β i,t , ∂φ i ∂x t as φ i,t , ∂ 2 α i ∂x t ∂x s as α i,ts , ∂ 2 β i ∂x t ∂x s as β i,ts and ∂ 2 φ i ∂x t ∂x s as φ i,ts . 13) In
, k2 , k3 , k4 , k5 ), (p 1 , p2 , p3 ) and (c 1 , c2 , c3 , c4 , c5 , c6 ).
We remind some of these notations in the chapters.

Manifolds and Surfaces

In general, a manifold is a topological space that resembles the Euclidean space R n locally. Therefore, at each point of the manifold one can find a neighbourhood that is homeomorphic to the Euclidean space of dimension n. 2D manifolds represent surfaces. If embedded in 3D, they represent 3D surfaces.

Infinitesimal Planarity

IP refers to the assumption that a surface at each point is approximately planar in its infinitesimal neighbourhood. This is fundamentally different from piece-wise planarity: in IP, the surface is globally curved, but in an infinitesimal neighbourhood, it may be represented by a plane. In other words, each infinitesimal model agrees with the global surface at the point where IP is assumed, but this agreement holds only at the zeroth order. We define the IP approximation of any surface as the IP surface where at each point the infinitesimal plane is the tangent plane of the original surface. Note that while the surface can be globally curved, the IP approximation is point-wise planar. It is a very interesting concept as it makes it simpler to derive properties of surfaces.

Infinitesimal Linearity

IL refers to the assumption that a smooth mapping between two surfaces can be represented by a set of linear mappings which map the infinitesimal neighbourhoods of the corresponding points on the surfaces. Figure 3.1 shows two curves related by a smooth mapping ψ.

According to the formulation of IL in synthetic differential geometry [START_REF] Kock | Synthetic Geometry of Manifolds[END_REF] (it uses IL to formalise theory of connections), given that ψ maps P to Q, there exists at least one linear function ψ L that maps the infinitesimal neighbourhood of P to Q. Therefore, ψ is represented with an infinite set of linear mappings ψ L that map infinitesimal neighbourhoods of the curves. ψ L has the same first-order differentials as ψ. It only assumes that the second or higher-order differentials are zero as it is a linearization of ψ.

Projection

A surface is mathematically related to an image with an image projection function. Figure 3.2 shows a surface M ∈ R 3 being projected into the image I ∈ R 2 with the function Π : R 3 → R 2 . We model projection with the perspective camera, where Π takes as input the point

z = z 1 z 2 z 3
on the surface M and outputs its retinal coordinates x = x 1 x 2 in the image:

x = x 1 x 2 = Π(z) = z 1 z 3 z 2 z 3 . ( 3

Image Embedding

The image embedding, denoted as φ : I → M (see figure 3.2), represents the inverse of Π restricted to the surface M ∈ R 3 , as it maps retinal coordinates to the 3D surface. It must satisfy the following identity:

x = (Π • φ)(x). (3.2)
Smooth functions that comply with equation (3.2) can be expressed with a depth function ρ ∈ C ∞ (I, R), where:

φ(x) = ρ(x) x 1 . (3.3)
Alternatively, let α = ρ -1 be the inverse-depth function. This allows us to re-define the image embedding in equation (3.3) as:

φ(x) = 1 α(x) x 1 . (3.4)
where α is a function that represents the inverse of the depth of the surface at a point

x = (x 1 , x 2 ) in I. A point on the surface M is given by z = φ = α -1 x 1 x 2 1 . (3.5)
For general surfaces, α is a non-linear function but for planar surfaces it is linear. Due to the assumption of IP, the restriction of α to a point becomes linear.

In chapter 4, we will show that working with the inverse-depth for defining the image embedding has an important role while defining the differential properties of surfaces.

Introduction

Research in NRSfM is still at an early stage. As we discussed in chapter 2, statistics-based modelling in NRSfM limits the applicability to short-baseline images only. Also, the use of orthographic camera projection leads to flip ambiguities. Physics-based modelling in NRSfM is very recent. It is widely used in SfT [START_REF] Salzmann | Linear Local Models for Monocular Reconstruction of Deformable Surfaces[END_REF] and recent works [START_REF] Bartoli | Shape-from-Template[END_REF]Chhatkuli et al., 2016b] show that the differential modelling of surfaces results in local analytical solution to SfT and makes is a well-posed problem for isometric deformations.

Therefore, local SfT methods are powerful and computationally cheap.

The success of differential modelling in SfT is our motivation for this work. Our goal is to extend this modelling to NRSfM. We use Riemannian geometry for differential modelling of NRSfM. The differential quantities metric tensors and CS, defined in this geometry, represent the properties of surfaces such as length, angles, areas and curvature. For isometric deformations, we found that these properties are preserved across surfaces. This leads to a solution for NRSfM using isometry as a deformation prior which we present in this chapter.

This solution performs significantly better than the compared state-of-the-art methods.

The method handles missing data and occlusions, needs very few images to perform reconstruction, handles large images conveniently without affecting the computation time and works for both short and wide-baseline images.

Chapter outline. We present the mathematical background of our solution in section 4.2.

It describes the modelling of NRSfM, the concepts of metric tensor and CS, the effect of the IP assumption on these quantities, the preservation of these quantities under isometry. Section 4.3 shows how to use these concepts to write the reconstruction equations for NRSfM under isometric deformations with and without the assumption of IP. Section 4.4

explains the algorithms and analyses their computational complexity. Section 4.5 discusses the experiments and section 4.6 concludes.

Mathematical Background

General Model

Our model of NRSfM is shown in figure 4.1. We have N input images I 1 , . . . , I N that show the projection of different isometric deformations of the same surface. The registration warps (η ij and η ji ) between the pair of images (I i , I j ) are known. In this framework, we compute these warps using [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. This choice is explained and justified by theorem 4.

Abusing notation, we also use I i to denote an image's retinal plane, with I i ⊂ R 2 . Surfaces in 3D are modeled as Riemannian manifolds. This allows us to define lengths, angles and tangent planes on the surface [START_REF] Lee | Riemannian Manifolds : An Introduction to Curvature[END_REF]. We denote M i ⊂ R 3 as the i th manifold, which can be seen as a two-dimensional subset embedded in 3D. We use the extrinsic definition of M i , where a function embeds a subset of the plane R 2 into R 3 . With embedding functions, one can easily compute manifold characteristics [START_REF] Lee | Introduction to Smooth Manifolds[END_REF] such as the metric tensor and the CS. However, these characteristics change according to the coordinate frame. We use the retinal plane I i as coordinate frame for M i and define as φ i ∈ C ∞ (I i , R 3 ) the image embedding for M i . We define as ψ ij the isometric mapping between manifolds M i and M j .

The Metric Tensor

The metric tensor (see appendix A for more details) is a differential quantity used to define lengths, angles and areas on the surface [START_REF] Lee | Riemannian Manifolds : An Introduction to Curvature[END_REF]. The metric tensor of φ i (in figure 4.2)

is denoted as g mn [φ i ]. We use the standard Einstein tensor notation and thus g mn [φ i ] is a combined reference to all elements of the metric tensor, a 2 × 2 matrix in this case. According to the Einstein summation convention, the summation is done over the indices appearing twice in the expression. Also, the free indices in an expression (the ones that do not appear twice in the expression) can be seen as both the indexed element or the whole arrangement. The indices m and n refer to the components of the coordinate frame of φ i . In figure 4.2, we have z = φ i (x) and:

J φ i = ⎛ ⎜ ⎜ ⎜ ⎜ ⎝ ∂z 1 ∂x 1 ∂z 2 ∂x 1 ∂z 3 ∂x 1 ∂z 1 ∂x 2 ∂z 2 ∂x 2 ∂z 3 ∂x 2 ⎞ ⎟ ⎟ ⎟ ⎟ ⎠ . (4.1)
The metric tensor of φ i is then:

g mn [φ i ] = J φ i J φ i = ∂z s ∂x m ∂z k ∂x n δ sk , (4.2)
with δ sk the Kronecker delta function. We recall that according to the Einstein summation convention, the summation in equation ( 4.2) is done over indices s and k. The inverse of the metric tensor is expressed with raised indices g mn [φ i ]. Given the change of coordinates:

x = η(y) with y = y 1 y 2 , (4.3)
the metric tensor of φ i • η is obtained as:

g st [φ i • η] = J η J φ i J φ i J η = ∂x m ∂y s ∂x n ∂y t g mn [φ i ]. (4.4)

Christoffel Symbols

CS (see appendix B for more details) of the second kind are function arrays that describe several properties of a Riemannian manifold, such as the curvature tensor, the geodesic equations of curves and the parallel transport of vectors on surfaces [START_REF] Lee | Riemannian Manifolds : An Introduction to Curvature[END_REF]. We denote the CS of embedding φ i as

Γ p mn [φ i ]. It is useful to represent the CS of φ i as two 2 × 2 matrices Γ 1 mn [φ i ] and Γ 2 mn [φ i ],
where the upper indices 1 and 2 make reference to the 2D image coordinates x = x 1 x 2 , where φ i is defined. The CS are given by:

Γ p mn [φ i ] = 1 2 g pl [φ i ] (g lm,n [φ i ] + g ln,m [φ i ] -g mn,l [φ i ]) , ( 4.5)
where g lm,n = ∂ n g lm . Given a change of coordinates x = η(y), the CS in the new coordinates are given as: Theorem 1 (CS Structure). Let x ∈ I i , then Γ p mn [φ i (x)] is given by:

Γ q st [φ i • η] = ∂x m ∂y s ∂x n ∂y t Γ p mn [φ i ] ∂y q ∂x p + ∂y q ∂x l ∂ 2 x l ∂y s ∂y t . ( 4
Γ 1 mn [φ i (x)] = - 1 α i 2α i,1 α i,2 α i,2 0 + (α i ) 2 A i D i α i,11 α i,12 α i,12 α i,22 Γ 2 mn [φ i (x)] = - 1 α i 0 α i,1 α i,1 2α i,2 + (α i ) 2 B i D i α i,11 α i,12 α i,12 α i,22 , (4.7)
where

α i,k = ∂α i ∂x k , α i,nm =
∂ 2 α i ∂x n ∂x m and:

A i = -x 1 α i + 1 + x 1 2 α i,1 + x 1 x 2 α i,2 B i = -x 2 α i + 1 + x 2 2 α i,2 + x 1 x 2 α i,1 D i = α i -x 1 α i,1 -x 2 α i,2 2 + (α i,1 ) 2 + (α i,2 ) 2 . (4.8)
Proof. From the definition of φ i (x) in equation (3.4), we can write the Jacobian matrix of φ i (x) as:

J φ i (x) = 1 α 2 i ⎛ ⎜ ⎝ α i -x 1 α i,1 -x 1 α i,2 -x 2 α i,1 α i -x 2 α i,2 -α i,1 -α i,2 ⎞ ⎟ ⎠. (4.9)
Next we compute the metric tensor by substituting the Jacobian matrix from equation (4.9) in equation (4.2). The metric tensor is given by

g 11 [φ i (x)] = 1 α 4 i 2 (α i,1 ) 2 + (α i ) 2 -2x 1 α i α i,1 g 12 [φ i (x)] = 1 α 4 i 2 α i,1 α i,2 -x 1 α i α i,2 -x 2 α i α i,1 g 22 [φ i (x)] = 1 α 4 i 2 (α i,2 ) 2 + (α i ) 2 -2x 2 α i α i,2 .
(4.10)

where 2 = 1 + x 1 2 + x 2 2
. The inverse of metric tensor is given by

g 11 [φ i (x)] = g 22 [φ i (x)] det(g[φ i (x)]) = (α i ) 8 g 22 [φ i (x)] D i g 12 [φ i (x)] = - g 12 [φ i (x)] det(g[φ i (x)]) = - (α i ) 8 g 12 [φ i (x)] D i g 22 [φ i (x)] = g 11 [φ i (x)] det(g[φ i (x)]) = (α i ) 8 g 11 [φ i (x)] D i . (4.11)
The derivatives of the metric tensor are given by

g 11,1 [φ i (x)] = - 4α i,1 α i g 11 [φ i (x)] + 2E i α i,11 (α i ) 4 g 12,1 [φ i (x)] = - 4α i,1 α i g 12 [φ i (x)] - H i (α i ) 4 + E i α i,12 + F i α i,11 (α i ) 4 g 22,1 [φ i (x)] = - 4α i,1 α i g 22 [φ i (x)] + 2L i (α i ) 4 + 2F i α i,12 (α i ) 4 g 11,2 [φ i (x)] = - 4α i,2 α i g 11 [φ i (x)] + 2H i (α i ) 4 + 2E i α i,12 (α i ) 4 g 12,2 [φ i (x)] = - 4α i,2 α i g 12 [φ i (x)] - L i (α i ) 4 + E i α i,22 + F i α i,12 (α i ) 4 g 22,2 [φ i (x)] = - 4α i,2 α i g 22 [φ i (x)] + 2F i α i,22
(α i ) 4 , (4.12)

where

E i = 1 + x 1 2 + x 2 2 α i,1 -x 1 α i , F i = 1 + x 1 2 + x 2 2 α i,2 -x 2 α i , H i = x 2 (α i,1 ) 2 + α i α i,2 -x 1 α i,1 α i,2
, and

L i = x 1 (α i,2 ) 2 α i,1 α i + -x 2 α i,1 α i,2 .
According to equation (4.5), the CS are given by

Γ p mn [φ i (x)] = 1 2 g p1 [φ i (x)] (g 1m,n [φ i (x)] + g 1n,m [φ i (x)] -g mn,1 [φ i (x)]) + 1 2 g p2 [φ i (x)] (g 2m,n [φ i (x)] + g 2n,m [φ i (x)] -g mn,2 [φ i (x)]) . (4.13)
Using the metric tensor and its derivatives from equations (4.10) and ( 4.12) in the expression for the CS given in equation ( 4.13), gives the result in equation (4.7). For example, Γ 1 11 [φ i (x)] is given by

Γ 1 11 [φ i (x)] = 1 2 g 11 [φ i (x)] (g 11,1 [φ i (x)]) + 1 2 g 12 [φ i (x)] (2g 21,1 [φ i (x)] -g 11,2 [φ i (x)]) = (α i ) 8 g 22 [φ i (x)] D i - 2α i,1 α i g 11 [φ i (x)] + E i α i,11 (α i ) 4 - (α i ) 8 g 12 [φ i (x)] D i - 4α i,1 α i g 12 [φ i (x)] - 2H i (α i ) 4 - (α i ) 8 g 12 [φ i (x)] D i F i α i,11 (α i ) 4 + 2α i,2 α i g 11 [φ i (x)] = -2α i,1 α i + (α i ) 2 A i D i . (4.14)

Commutativity under Isometry

Images and surfaces in Iso-NRSfM follow the commutative diagram shown in figure 4.2.

Therefore,

φ j = ψ ij • φ i • η ji J φ j = J ψ ij J φ i J η ji . (4.15)
The metric tensor of φ j can be written according to equation (4.2). It is given by

J φ j J φ j = J η ji J φ i J ψ ij J ψ ij J φ i J η ji = J η ji J φ i J φ i J η ji . (4.16)
The fact that mappings between manifolds are isometric (J [START_REF] Bartoli | Shape-from-Template[END_REF] allows us to derive the following fundamental result: in Iso-NRSfM, both metric tensors and CS commute between surfaces with a change of variable given by the image warps. This result is formalised with Theorem 2 and Corollary 1.

ψ ij J ψ ij = I 3×3 ) [
Theorem 2 (Metric Tensor Commutation). Let ψ ij be an isometric mapping between the manifolds M i and M j , then

g mn [φ j ] = g mn [φ i • η ji ] with (i, j) ∈ [1, N] × [1, N].
Proof. We first write φ j in terms of φ i using the isometric mapping ψ ij :

φ j = ψ ij • φ i • η ji . (4.17)
From equations (4.4) and ( 4.17) we have:

g mn [φ j ] = g mn [(ψ ij • φ i ) • η ji ] = ∂x s ∂y m ∂x t ∂y n g st [ψ ij • φ i ]. (4.18)
By definition, isometric mappings do not change the local metric and so g[

ψ ij • φ i ] = g[φ i ],
which applied to equation ( 4.18) gives:

g mn [φ j ] = ∂x s ∂y m ∂x t ∂y n g st [φ i ]. ( 4
[φ j ] = g mn [φ i • η ji ].
Corollary 1 (CS Commutation). Let ψ ij be an isometric mapping between the manifolds M i and M j , then

Γ p mn [φ j ] = Γ p mn [φ i • η ji ] with (i, j) ∈ [1, N] × [1, N].
Proof. As described in equation (4.5), Γ p mn [φ j ] is a function of g mn [φ j ] and its derivatives. From Theorem 2 we have that g mn [φ j ] = g mn [φ i • η ji ]. By multiplying this expression in both sides by g mn [φ j ], we get:

g mn [φ j ]g mn [φ j ] = g mn [φ j ]g mn [φ i • η ji ] = δ mn , (4.20) from which we deduce that g mn [φ j ] = g mn [φ i • η ji ]. Also, by differentiating g mn [φ j ] = g mn [φ i • η ji ]
on both sides we have:

∂ l g mn [φ j ] = ∂ l g mn [φ i • η ji ], (4.21) giving g mn,l [φ j ] = g mn,l [φ i • η ji ]
. By substitution of these identities in equation ( 4.19) we obtain:

Γ p mn [φ j ] = 1 2 g pl [φ i • η ji ](g lm,n [φ i • η ji ] + g ln,m [φ i • η ji ] -g mn,l [φ i • η ji ]),
and thus the equality

Γ p mn [φ j ] = Γ p mn [φ i • η ji ] holds.
These results show that, given the metric tensor and the CS for one image embedding, they can be transferred to the rest of the embeddings using the warps, which are known entities in Iso-NRSfM. Note that this result cannot be generalised to non-isometric mappings. This establishes the ground rules for developing a local solution to Iso-NRSfM where the number of unknowns does not grow with the number of images. The main idea is to define the unknowns in a reference image and to use the warps to transfer all constraints into it.

Infinitesimal Planarity

We study the differential properties of the image embedding when the surface is a plane. We then invoke IP to extend these properties point-wise to non-planar surfaces. In this regard we present Theorem 3 and Corollary 2.

Theorem 3 (Linear Inverse-Depth of a Plane). If M is a plane then its image embedding at

x ∈ I is φ(x) = β(x) -1 (x 1
) with β a linear function.

Proof. Suppose M is a plane described by the equation n z + d = 0, where z = z 1 z 2 z 3 and n is the plane's normal. From equation (3.3), the embedding is expressed with a depth function φ(x) = ρ(x) x 1 . By combining the depth parametrisation with the plane equation, we have:

n ρ(x) x 1 + d = 0, (4.22)
from which we compute ρ as:

ρ(x) = - d n x 1 . (4.23)
By defining β(x) = (ρ(x)) -1 , φ is written as:

φ(x) = β(x) -1 (x 1) , (4.24)
where β(x) is linear in x.

Corollary 2 (CS of a Plane). Let M be a plane and φ(x) the image embedding at x ∈ I, the CS Γ p mn [φ(x)] are given by:

Γ 1 mn [φ(x)] = 1 β(x) -2β 1 (x) -β 2 (x) -β 2 (x) 0 Γ 2 mn [φ(x)] = 1 β(x) 0 -β 1 (x) -β 1 (x) -2β 2 (x) , (4.25)
where

β 1 (x) = ∂β(x) ∂x 1 and β 2 (x) = ∂β(x) ∂x 2 .
Proof. From the definition of φ(x) in equation ( 4.24), we can write the Jacobian matrix of φ(x) as:

J φ (x) = 1 β(x) 2 ⎛ ⎜ ⎝ β(x) -x 1 β 1 (x) -x 1 β 2 (x) -x 2 β 1 (x) β(x) -x 2 β 2 (x) -β 1 (x) -β 2 (x) ⎞ ⎟ ⎠. (4.26)
Using equation (4.2), the metric tensor at φ(x) can be written as J φ(x) J φ(x) . The expression is given by

g 11 [φ(x)] = 1 β 4 2 (β 1 ) 2 + β 2 -2x 1 ββ 1 g 12 [φ(x)] = 1 β 4 2 β 1 β 2 -x 1 ββ 2 -x 2 ββ 1 g 22 [φ(x)] = 1 β 4 2 (β 2 ) 2 + β 2 -2x 2 ββ 2 .
(4.27)

where 2 = 1 + x 1 2 + x 2 2
. The derivatives of the metric tensor are given by:

g 11,1 [φ(x)] = - 4β 1 β g 11 [φ(x)] g 12,1 [φ(x)] = - 4β 1 β g 12 [φ(x)] - x 2 (β 1 ) 2 + ββ 2 -x 1 β 1 β 2 (β) 4 g 22,1 [φ(x)] = - 4β 1 β g 22 [φ(x)] + 2 x 1 (β 2 ) 2 β 1 β + -x 2 β 1 β 2 (β) 4 g 11,2 [φ(x)] = - 4β 2 β g 11 [φ(x)] + 2 x 2 (β 1 ) 2 + ββ 2 -x 1 β 1 β 2 (β) 4 g 12,2 [φ(x)] = - 4β 2 β g 12 [φ(x)] - x 1 (β 2 ) 2 β 1 β + -x 2 β 1 β 2 (β) 4 g 22,2 [φ(x)] = - 4β 2 β g 22 [φ(x)]. (4.28)
Note that there are no second-order derivatives in the above expression because they vanish in the case of planes. This leads to the CS in equation (4.25).

Theorem 3 shows that the inverse-depth β of a planar surface is a linear function. Corollary 2 is derived from Theorem 1 and Theorem 3. It shows that the CS have a simplified structure under infinitesimal planarity, where at any point they have 3 degrees of freedom: β and its first-order derivatives. Moreover this also shows that both the metric tensor and the CS share the same 3 unknowns.

From Corollary 2 we find the following constraints over the elements of the CS:

Γ 1 22 [φ(x)] = Γ 2 11 [φ(x)] = 0 2Γ 2 12 [φ(x)] = Γ 2 22 [φ(x)] Γ 1 11 [φ(x)] = 2Γ 2 12 [φ(x)]. (4.29)
We derive Theorem 4 from equation (4.29). It shows that the warps must comply with the 2D Schwarzian derivatives [START_REF] Sasaki | Schwarzian Derivatives and Uniformization[END_REF], which are second-order bilinear PDE that arise in the field of projective differential geometry. Proof. The elements of the CS for M i with i = [1, . . . , N] have the form of (4.25), and thus must comply with the following algebraic constraints:

Γ 1 22 [φ i ] =Γ 2 11 [φ i ] = 0 2Γ 2 12 [φ i ] = Γ 2 22 [φ i ] Γ 1 11 [φ i ] = 2Γ 2 12 [φ i ]. (4.30)
From Corollary 1 we have

Γ p mn [φ j ] = Γ p mn [φ i • η ji ]. Now we use equation (4.6) to compute Γ p nm [φ i • η ji ] from Γ p nm [φ i ]
given in equation (4.25). Given that x = η ji (y), we write

Γ p nm [φ i • η ji ] = ∂ 1 y p -2 β i,1 β i ∂ m x 1 ∂ n x 1 - β i,2 β i ∂ m x 1 ∂ n x 2 + ∂ m x 2 ∂ n x 1 + ∂ 2 y p -2 β i,2 β i ∂ m x 2 ∂ n x 2 - β i,1 β i ∂ m x 1 ∂ n x 2 + ∂ m x 2 ∂ n x 1 + ∂ 1 y p ∂ 2 mn x 1 + ∂ 2 y p ∂ 2 mn x 2 .
(4.31)

By forcing conditions in equation (4.30) in Γ[φ i •η ji ] we obtain the following four second-order PDE only in η ji

∂ 2 11 x 1 ∂ 1 x 2 -∂ 2 11 x 2 ∂ 1 x 1 = 0 ∂ 2 22 x 1 ∂ 2 x 2 -∂ 2 22 x 2 ∂ 2 x 1 = 0 ∂ 11 x 1 ∂ 2 x 2 -∂ 11 x 2 ∂ 2 x 2 + 2 ∂ 12 x 1 ∂ 1 x 2 -∂ 12 x 2 ∂ 1 x 1 = 0 ∂ 22 x 1 ∂ 1 x 2 -∂ 22 x 2 ∂ 1 x 1 + 2 ∂ 12 x 1 ∂ 2 x 2 -∂ 12 x 2 ∂ 2 x 1 = 0. (4.32)
These are the 2D Schwarzian equations introduced in [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF], where point-wise projective warps were investigated.

The 2D Schwarzian derivatives were used in [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF] as a penalty to compute 'Schwarps', smooth warps that preserve the deformation's local projective structure.

Schwarps were shown to improve accuracy in both SfT and NRSfM with respect to other smoothing penalties based on the bending energy. Theorem 4 theoretically justifies our choice to use Schwarps for computing our image warps. Nonetheless our method can also be used with any means to compute the local image warps.

Reconstruction Equations

We study local solutions to Iso-NRSfM, based on the differential properties derived in the previous section. We show that Iso-NRSfM can be posed as a non-linear PDE system and that we can find non-holonomic solutions of this system. We do not deal with boundary conditions in the PDE as we find algebraic solutions of the system in terms of the nonholonomic variables. This follows the same path as [START_REF] Bartoli | Shape-from-Template[END_REF] for finding local solutions in Iso-SfT.

For planes, there is a unique linear relationship between the metric tensor and the CS, which is why Iso-NRSfM is solvable under the assumption of IP. Corollary 2 shows that both of them can be expressed in terms of the first-order derivatives of the inverse-depth of the surface only. We explore this relationship for non-planar surfaces, where we need the first and the second-order derivatives of the inverse-depth of the surface to express the metric tensor and the CS. We argue that there is no uniqueness in the relationship between the metric tensor and the CS anymore, and therefore, there is not a unique solution to Iso-NRSfM locally. Then, we propose to solve Iso-NRSfM by solving for the first and the second-order derivatives separately.

Relating the Metric Tensor and the Christoffel Symbols

For a non-planar surface, the CS at z ∈ M i are given by equation (4.7). We define them as:

Γ 1 mn [φ i (x)] = c 1 c 3 c 3 c 5 Γ 2 mn [φ i (x)] = c 2 c 4 c 4 c 6 , ( 4.33)
where c 1 , c 2 , c 3 , c 4 , c 5 and c 6 are expressed in terms of the first and second-order derivatives of α i (x) defined in equation (3.4). The expressions in equation ( 4.7) are: (4.34) with:

c 1 = -2k 1 + k 3 A i c 2 = k 3 B i c 3 = -k 2 + k 4 A i c 4 = -k 1 + k 4 B i c 5 = k 5 A i c 6 = -2k 2 + k 5 B i ,
A i = -x 1 + 1 + x 1 2 k 1 + x 1 x 2 k 2 B i = -x 2 + 1 + x 2 2 k 2 + x 1 x 2 k 1 D i = 1 -x 1 k 1 -x 2 k 2 2 + (k 1 ) 2 + (k 2 ) 2 , ( 4.35)
where

k 1 = α i,1 α i , k 2 = α i,2 α i , k 3 = α i,11 α i D i , k 4 = α i,12 α i D i and k 5 = α i,22 α i D i
. The jacobian and hence the metric tensor at z can be written in terms of (k 1 , k 2 ). Our goal is to find a relationship between the metric tensor parametrised with (k 1 , k 2 ) and the CS (c 1 , c 2 , c 3 , c 4 , c 5 , c 6 ). Having such a relationship, we can formulate a system of equations exploiting the transfer of variables in the CS and metric tensor from one surface to another. From c 1 and c 2 in equation ( 4.34), we can write:

c 1 + 2k 1 c 2 = A i B i . (4.36)
Similarly, from c 5 and c 6 in equation ( 4.34), we can write:

c 5 c 6 + 2k 2 = A i B i . (4.37)
From c 3 and c 4 in equation ( 4.34), we find (4.36) and (4.37), we obtain:

A i B i = c 3 + k 2 c 4 + k 1 . We substitute A i B i in equations
(c 1 + 2k 1 )(c 4 + k 1 ) = c 2 (c 3 + k 2 ) (c 6 + 2k 2 )(c 3 + k 2 ) = c 5 (c 4 + k 1 ). (4.38)
From the first expression in equation ( 4.38), we find

k 2 = (c 1 + 2k 1 )(c 4 + k 1 ) c 2 -c 3 and
substitute it in the second expression. We obtain the following quartic in k 1 :

(c 4 + k 1 ) 8k 3 1 + 8 (c 1 + c 4 ) k 2 1 + 2 (c 1 (c 1 + 4c 4 ) + c 2 (c 6 -2c 3 )) k 1 + 2c 2 1 c 4 + c 1 c 2 (c 6 -2c 3 ) -c 2 2 c 5 = 0. (4.39)
This gives up to four possible solutions to k 1 , which means that there is not a unique relationship between the CS and the metric tensor.

Combining equation ( 4.38) with equation ( 4.34), we can express (k 1 , k 2 ) in terms of the CS (c 1 , c 2 , c 3 , c 4 , c 5 , c 6 ) as a rational expression of degree two. This gives a system of two polynomials of degree 8 in 6 variables for each pair of views. Existing solvers such as [START_REF] Henrion | Gloptipoly: Global optimization over polynomials with matlab and sedumi[END_REF]] cannot solve such high degree polynomial systems. We conclude that the first and second-order derivatives of α i (x) cannot be solved jointly via estimating the CS.

However, we see that the expressions of the CS in equation ( 4.34) are linear in terms of (k 1 , k 2 ) and (k 3 , k 4 , k 5 ). By assuming (k 3 , k 4 , k 5 ) to be known, we can find a unique relationship between the metric tensor and the CS and vice versa. Therefore, splitting the problem in two steps of solving for the first and the second-order derivatives of α i (x) separately leads to a solution to Iso-NRSfM.

Solving for the First-Order Derivatives

We assume that the second-order derivatives of α i (x) are known. They can be assumed to be zero (as in the case of the infinitesimal planarity assumption) or they can be obtained by the method we describe next. We show how to solve for the first-order derivatives of α i (x). We also show that this solution has a similar structure as the solution to Iso-NRSfM under IP assumption. We first select a pair of surfaces (M i , M j ) (see figure 4.2) and a

point x = (x 1 , x 2 ) ∈ I i . We evaluate Γ p mn [φ i ] at x, namely Γ p mn [φ i (x)].
According to equation (4.7), it is given by: (4.40) where

Γ 1 mn [φ i (x)] = -2k 1 + A i p 1 -k 2 + A i p 2 -k 2 + A i p 2 A i p 3 Γ 2 mn [φ i (x)] = B i p 1 -k 1 + B i p 2 -k 1 + B i p 2 -2k 2 + B i p 3 ,
k 1 = β 1 (x) β(x) and k 2 = β 2 (x) β(x)
. The expressions (p 1 , p 2 , p 3 ) are functions of second-order derivatives of α i (x) and therefore, they are known. A i and B i are linear expressions in (k 1 , k 2 ) according to equation (4.8). Next we compute J φ i in terms of (k 1 , k 2 ):

J φ i (x) = 1 β(x) ⎛ ⎜ ⎝ 1 -k 1 x 1 -k 2 x 1 -k 1 x 2 1 -k 2 x 2 -k 1 -k 2 ⎞ ⎟ ⎠ . (4.41)
By substitution of equation ( 4.41) in equation ( 4.2) we have:

g 11 [φ i (x)] = 1 β(x) 2 k 1 2 + k 1 x 1 -1 2 + k 1 x 2 2 g 12 [φ i (x)] = 1 β(x) 2 k 1 k 2 1 + x 1 2 + x 2 2 -k 2 x 1 -k 1 x 2 g 22 [φ i (x)] = 1 β(x) 2 k 2 2 + k 2 x 1 2 + k 2 x 2 -1 2 . (4.42) We define G mn [φ i (x)] = β(x) 2 g mn [φ i (x)], which only depends on (k 1 , k 2 ). Let x = η ji (y).
We use equation (4.6) and Corollary 1 to compute

Γ k mn [φ j (y)] = Γ k mn [(φ i • η ji )(y)] as: Γ 1 mn [(φ i • η ji )(y)] = -2 k1 + A j p1 -k2 + A j p2 -k2 + A j p2 A j p3 Γ 2 mn [(φ i • η ji )(y)] = B j p1 -k1 + B j p2 -k1 + B j p2 -2 k2 + B j p3 , ( 4.43)
where according to equation (4.6), ( k1 , k2 ) are linear combinations of (k 1 , k 2 ) and (p 1 , p2 , p3 ).

(p 1 , p2 , p3 ) are known. A j and B j are linear expressions in ( k1 , k2 ) according to equation (4.8).

From equation (4.42) one can find g st [φ j (y)] in function of ( k1 , k2 ), and thus in function of

(k 1 , k 2 ).
Alternatively, from equation (4.4) and using the definition of G mn [φ i (x)] and G mn [φ j (y)]

we have the following identity:

1 β(x) 2 G st [φ j (y)] = 1 β(y) 2 ∂x m ∂y s ∂x n ∂y t G mn [φ i (x)].
(4.44)

We cancel β(x) and β(y) by converting system (4.44) into the following two equations:

G 11 [φ j (y)] ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] -G 12 [φ j (y)] ∂x m ∂y 1 ∂x n ∂y 1 G mn [φ i (x)] = 0 G 22 [φ j (y)] ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] -G 12 [φ j (y)] ∂x m ∂y 2 ∂x n ∂y 2 G mn [φ i (x)] = 0. (4.45)
We recall that both G mn [φ i (x)] and G st [φ j (y)] are only functions of (k 1 , k 2 ) and x.

Equation (4.45) is a system of two cubics in variables (k 1 , k 2 ) modeling Iso-NRSfM for manifolds M i and M j at point x ∈ I i . We denote the two equations as

Q ij (x, η 1j (x), k 1 , k 2 ).
By keeping the first index as the reference manifold, for instance i = 1, and obtaining the polynomials for the rest of the views we have 2N -2 polynomial equations in two vari-

ables Q 1 (k 1 , k 2 ) = {Q 1j (x, η 1j (x), k 1 , k 2 )} N j=2 .
The solution (k 1 , k 2 ) to the polynomial system Q 1 (k 1 , k 2 ) at the point x = x 1 allows us to reconstruct the metric tensor, the CS and the tangent plane for point x 1 in view I 1 . Using equation ( 4.41) we can reconstruct J φ 1 (x 1 ) up to an unknown scale β(x 1 ) -1 . It is not necessary to recover this scale to estimate a unit normal, which is computed by taking the cross product of the two columns of J φ 1 (x 1 ) and normalising.

We solve system Q 1 (k 1 , k 2 ) by finding the values of (k 1 , k 2 ) that minimise the sum-ofsquares of all polynomials in the system. This optimisation is solved globally using moment based convex optimisation [START_REF] Henrion | Gloptipoly: Global optimization over polynomials with matlab and sedumi[END_REF]. Given (k 1 , k 2 ), we calculate ( k1 , k2 ) by using equation ( 4.6) at each point.

Notice that the structure of the CS given in equation ( 4.25) for planes is very similar to equation (4.40) with (p 1 , p 2 , p 3 ) as zeros. This shows that the solution to Iso-NRSfM with the IP assumption is a special case of this solution. We express the system with zero second-order derivatives as P 1 (k 1 , k 2 ), which is solved in a similar way as Q 1 (k 1 , k 2 ).

Solving for the Second-Order Derivatives

We now show how to solve for the second-order derivatives of α i (x), assuming that the firstorder derivatives of α i (x) are known from the previous step (we start by solving for the first-order derivatives assuming that second-order derivatives are null). The expressions for the CS in equation ( 4.40) become linear in the second-order derivatives of α i (x). This means that Γ p mn [φ i (x)] is a linear function of (k 3 , k 4 , k 5 ). Given that x = η ji (y) and equation (4.6), Γ p mn [(φ i • η ji )(y)] is given by:

Γ 1 mn [(φ i • η ji )(y)] = c1 c3 c3 c5 Γ 2 mn [(φ i • η ji (y))] = c2 c4 c4 c6 , ( 4.46)
where (c 1 , c2 , c3 , c4 , c5 , c6 ) are expressed as a linear combination of (k 3 , k 4 , k 5 ). Therefore, at M j , ( k3 , k4 , k5 ) are given by the following expressions:

k3 = c1 + 2 k1 A j + c2 B j A 2 j + B 2 j k4 = c3 + k2 A j + c4 + k1 B j A 2 j + B 2 j k5 = c5 A j + c6 + 2 k2 B j A 2 j + B 2 j .
(4.47)

These expressions show that ( k3 , k4 , k5 ) can be expressed as a linear combination of (k 3 , k 4 , k 5 ).

In order to solve for the second-order derivatives of α i (x), we differentiate the first-order reconstruction equations (4.45). The expressions are given in equation (4.48).

∂G 11 [φ j (y)] ∂x 1 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] - ∂G 12 [φ j (y)] ∂x 1 ∂x m ∂y 1 ∂x n ∂y 1 G mn [φ i (x)] +G 11 [φ j (y)] ∂ ∂x 1 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] -G 12 [φ j (y)] ∂ ∂x 1 ∂x m ∂y 1 ∂x n ∂y 1 G mn [φ i (x)] = 0 ∂G 11 [φ j (y)] ∂x 2 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] - ∂G 12 [φ j (y)] ∂x 2 ∂x m ∂y 1 ∂x n ∂y 1 G mn [φ i (x)] +G 11 [φ j (y)] ∂ ∂x 2 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] -G 12 [φ j (y)] ∂ ∂x 2 ∂x m ∂y 1 ∂x n ∂y 1 G mn [φ i (x)] = 0 ∂G 22 [φ j (y)] ∂x 1 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] - ∂G 12 [φ j (y)] ∂x 1 ∂x m ∂y 2 ∂x n ∂y 2 G mn [φ i (x)] +G 22 [φ j (y)] ∂ ∂x 1 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] -G 12 [φ j (y)] ∂ ∂x 1 ∂x m ∂y 2 ∂x n ∂y 2 G mn [φ i (x)] = 0 ∂G 22 [φ j (y)] ∂x 2 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] - ∂G 12 [φ j (y)] ∂x 2 ∂x m ∂y 2 ∂x n ∂y 2 G mn [φ i (x)] +G 22 [φ j (y)] ∂ ∂x 2 ∂x m ∂y 1 ∂x n ∂y 2 G mn [φ i (x)] -G 12 [φ j (y)] ∂ ∂x 2 ∂x m ∂y 2 ∂x n ∂y 2 G mn [φ i (x)] = 0. (4.48)
The derivatives of G mn [φ i (x)] in equation ( 4.45) are given in equation (4.49).

∂G 11 [φ i (x)] ∂x 1 = 2k 1 x 1 k 1 -1 + 2 k 1 -x 1 k 3 D -k 1 2 ∂G 12 [φ i (x)] ∂x 1 = k 2 2x 1 k 1 -1 + k 2 -x 2 k 3 D -k 1 2 + k 1 -x 1 (k 4 D -k 1 k 2 ) ∂G 22 [φ i (x)] ∂x 1 = 2x 1 k 2 2 + 2 k 2 -x 2 (k 4 D -k 1 k 2 ) ∂G 11 [φ i (x)] ∂x 2 = 2x 2 k 1 2 + 2 k 1 -x 1 (k 4 D -k 1 k 2 ) ∂G 12 [φ i (x)] ∂x 2 = k 1 2x 2 k 2 -1 + k 2 -x 2 (k 4 D -k 1 k 2 ) + k 1 -x 1 k 5 D -k 2 2 ∂G 22 [φ i (x)] ∂x 2 = 2k 2 x 2 k 2 -1 + 2 k 2 -x 2 k 5 D -k 2 2 , with D = 1 -x 1 k 1 -x 2 k 2 2 + k 1 2 + k 2 2 and = 1 + x 1 2 + x 2 2 .
(4.49) Equation ( 4.49) shows that the derivatives are linear functions of (k 3 , k 4 , k 5 ). Using equation (4.47), the reconstruction equations (4.48) form a linear system in three variables only, which can be solved using LLS. Therefore, for each pair of manifolds (M i , M j ) at point

x ∈ I i , equation (4.48) is a system of four linear equations in variables (k 3 , k 4 , k 5 ). We denote the four equations as S ij (x, y, k 3 , k 4 , k 5 ). Fixing the i th manifold as a reference, for instance i = 1, we obtain 4N -4 linear equations in three variables which are written as

S 1 (k 3 , k 4 , k 5 ) = {S 1j (x, η 1j (x), k 3 , k 4 , k 5 )} N j=2 .
The solution (k 3 , k 4 , k 5 ) to the linear system S 1 (k 3 , k 4 , k 5 ) for the point x = x 1 gives the second derivatives of α i (x). We use them to compute a better estimate of the CS in equation (4.7). Using equation ( 4.47), we can obtain ( k3 , k4 , k5 ) using (k 3 , k 4 , k 5 ).

Algorithms

We describe our solutions to Iso-NRSfM based on the theoretical results from the previous sections. First, we describe the algorithm for solving Iso-NRSfM with the IP assumption and then we describe the algorithm for the general solution. This uses the solution with the IP assumption as initialisation. The inputs to our system are N images of a deforming object and their inter-image warps with respect to the first image η j1 and η 1j . The outputs of our system are the 3D points and normals corresponding to the point correspondences for the N images. In our formulation, our goal is to find the jacobian of the image embedding. The normals are then obtained from the jacobian and the 3D points are calculated by integrating the normal field as in [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF].

Solution under Infinitesimal Planarity

With the assumption of IP, we can write the metric tensor of equation ( 4.40) and the CS of equation ( 4.42) on a manifold M i in terms of two variables which correspond to the ratio of first-order derivatives of the inverse-depth β of the image embedding φ i with the inversedepth function. We can also write the metric tensor and the CS on the rest of the images in terms of the variables in the first image (equations (4.4) and (4.6) respectively), which leads to two variables for N images. We solve the system of two cubics in two variables of equation ( 4.45) for all images by minimising the sum-of-squares of the polynomials. The algorithm takes the following steps:

Inputs: Warps η j1 , j ∈ [2, N].
1) Find point correspondences. Select a grid of points on the first image and using the warps η 1j , find the corresponding grid of points on the rest of the images. We evaluated our method on a 20 × 20 grid of points.

2) Find (k 1 , k 2 ). Evaluate the polynomial P 1 (k 1 , k 2 ) and solve by minimising the sum of squares using [START_REF] Henrion | Gloptipoly: Global optimization over polynomials with matlab and sedumi[END_REF]. This gives (k 1 , k 2 ). Find ( k1 , k2 ) in terms of (k 1 , k 2 ) and the η j1 warps, j ∈ [2, N], using equation (4.6).

3) Find normals and 3D points. Find the jacobian in terms of (k 1 , k 2 ) using equation (4.41). Compute the normals by taking the cross-product of the jacobian's columns and normalising it. Use the method in [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF] to recover the 3D surfaces by integrating the normal fields.

Outputs: Points and normals on 3D surfaces.

General Solution

We still have the metric tensors on a manifold M i in terms of two variables but the CS are now written in terms of five variables, ratio of the first and the second-order derivatives of the inverse-depth α i of the image embedding φ i with the inverse-depth function. We iteratively solve for the first and the second-order derivatives alternatively until the first-order derivatives of α i (x) converge. Our algorithm is: and the depth error (mean difference between computed and ground truth 3D coordinates in mm). We denote Iso-NRSfM with IP as infP and as iso otherwise. We compared our method with six other NRSfM methods: diffH [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF], mdhI [Chhatkuli et al., 2016a], kerF [START_REF] Gotardo | Kernel Non-Rigid Structure from Motion[END_REF], plaH [START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF], pieceI [START_REF] Taylor | Non-rigid Structure from Locally-rigid Motion[END_REF], inextI [START_REF] Vicente | Soft Inextensibility Constraints for Template-Free Non-rigid Reconstruction[END_REF]. All the codes for these methods were obtained from the authors' websites except plaH which we re-implemented.

Synthetic Datasets

We simulated random views of a cylindrical surface deforming isometrically. The image size is 640p × 480p and the focal length is 400p. We tracked 400 points. We compared all methods by varying the number of views and noise in the images. kerF needs a temporal sequence, 10 views are not enough for reconstruction especially for short-baseline viewpoints. It therefore did not do well. Also, mdhI needs the views to be very different and therefore, it also gave very bad results on this sequence. [Chhatkuli et al., 2016a] mentioned that their method fails on such sequences. The results are shown in figure 4.4. The results are obtained after averaging the errors over 50 trials (the default is 1p noise and 10 views).

Varying the Number of Views

infP gives a very good reconstruction for three views which improves when more images are added. iso performs much better than infP as it does not assume IP; this helps in reconstructing the high curvature deformations more accurately. The errors of iso are almost half of infP. plaH and diffH give higher errors than infP and iso with plaH being better than diffH. However, diffH improves faster with the number of views and gives better results than plaH for 8-10 views. The performance of these methods is off by almost 5 degrees compared with infP. pieceI and inextI give decent results only with 8-10 views but their performance is worst amongst the compared methods. Overall, infP and iso consistently show lower errors than all other methods.

Varying Noise

For the 10 images of the synthetic dataset, we observe that all methods degrade linearly when noise varying between 1-5 pixels is added. inextI and pieceI show a good tolerance to noise, even though their performance is worse than all other methods. diffH and plaH give a slightly better performance than inextI and pieceI. Their performance degrades faster with noise compared to inextI and pieceI. Even though the normal errors of diffH and plaH are comparable to inextI and pieceI, their depth errors are lower because they smooth normals while calculating the depth. infP and iso give best performance with noise. iso performs better than infP and degrades more slowly than infP. The normal error for infP and iso is almost half compared to other methods.

We also made an experiment to study the influence of noise on the warps. This is because we use warps to represent the image transformation and that estimating these could be reducing some of the noise applied to the correspondences. We simulated two synthetic images (I1 and I2) and added a 1-5 pixels noise to point correspondences in I2. We computed the schwarps [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF] (using B-splines) between I1 and I2 using our default setup (30 control points and a fixed value for the hyperparameter controlling the Schwarps smoothing). We then computed the standard deviation of the pixel noise after computing the warps. The result is given in the last row of table 4.1, where we can see that the amount of noise is almost unaffected. We did the same experiment with fewer control points, which resulted in improved noise values (see the table 4.1). However, we found that using fewer points has an impact on the final accuracy as they degrade the derivatives, which directly influence the output of our method. We conclude that the warps do not remove the noise from the point correspondences.

Varying Curvature

We simulated a cylindrical surface with a varying radius. The curvature is the inverse of the radius. We simulated 10 surfaces with the radius varying from 2 to 11 and used 10 different views of each surface for the experiment. We see that the performance of iso and infP is best amongst the compared methods. Their performance is very similar when the surfaces are almost flat or less bent. As the curvature of the surfaces increase, we can see that iso performs much better than infP. iso handles high deformations better than infP because it estimates the second-order derivatives of the surface while infP assumes them to be zero. diffH and plaH need wide baseline views with different deformations, therefore, their performance is worse or comparable with inextI and pieceI.

Real Datasets

We conducted experiments with the four datasets shown in figure 4.3 and performed two types of experiments. The tshirt dataset is a wide baseline dataset and the rest of them are short baseline datasets. Our observations are summarised below.

Short Sequences

The rug, table mat and shows that iso works best among the compared methods while the performance of mdhI and infP is quite good as well.

Rug dataset. iso and infP have the best performance on this dataset; iso being better than infP. mdhI improves with the number of views and gives comparable results to infP for 7-10 views. diffH and plaH show a worse performance than mdhI, with plaH being better. inextI and pieceI are orthographic methods and therefore, they did not do well on this dataset as it has too much perspective in the deformations. inextI's depth error is comparable to plaH but normal error is much higher. This indicates flattening of surfaces during the reconstruction. methods. Their errors are almost twice as the best performing methods mdhI, infP and iso. diffH and plaH show better performance in terms of normals as compared to inextI and pieceI. The performance of diffH, plaH, inextI and pieceI in terms of depth error is similar.

Tshirt dataset. iso has the best performance on this dataset with infP and mdhI being very close to iso. diffH is slightly worse than infP and mdhI. plaH follows a similar trend as diffH, but its performance is worse. inextI and pieceI have poor results on this dataset because they cannot handle such deformations.

Summary of experiments on short sequences iso and infP give the best performance on the rug and table mat datasets while mdhI gives best results on the kinect paper dataset.

It is important to note that iso and infP converge quickly as compared to the rest of the methods. They show very good results for as few as three views. iso converges much quicker than infP; the errors seem to have been stabilised with four views only. Figure 4.6 shows 3D reconstruction error maps for the rug, table mat, kinect paper and tshirt datasets. We showed results for iso, infP and mdhI only because they are the most competitive methods amongst the compared methods. In case of occlusion, iso and infP can reconstruct the entire surface as long as the surface is visible in at least three images, therefore, even if the kinect paper dataset is occluded by a hand, it can be reconstructed by them. However, mdhI reconstructs only the visible part of each surface. Therefore we observe the occlusion in the reconstruction and rendering as well.

Long Sequences

The rug, table mat and kinect paper datasets are long sequences with 60, 159 and 191 images.

Since our method can easily handle a large number of images, it is important to show results on large sequences by considering all images in the dataset. A limitation of current NRSfM methods is that they cannot handle a large number of views. Also, several NRSfM methods such as diffH and plaH [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF][START_REF] Varol | Template-free Monocular Reconstruction of Deformable Surfaces[END_REF] reconstruct the reference image only and are computationally expensive to recover the other shapes. kerF reconstructs the entire image set in one execution and therefore, we compare our method with kerF on long sequences. The cat dataset is a relatively short sequences (60 images) therefore, we added the results of inextI for this dataset on the entire sequence. Figure 4.7 shows the comparison of our method with others. It is very clearly visible that our method performs much better than the compared methods. mdhI and infP show good results as well. Table 4.2 summarises the results of these methods.

Rug dataset. iso gives the best results among the compared methods on this dataset. infP's performance is slightly worse than iso. mdhI shows better performance than kerF, but it is worse than iso and infP.

Rug

Table mat

Elastic Objects

Our methods model deformations with isometry. In case of non-isometric deformations, theorem 2 and corollary 1 do not hold and therefore, there is no meaningful theoretical solution guaranteed. However, we solve Iso-NRSfM by finding a set of CS that minimise the sum of squares of polynomials in equation (4.45). Therefore, we made an experiment to test Iso-NRSfM with objects deforming non-isometrically in order to test the limitation for our methods infP and iso. We used the partially stretched surface dataset introduced in [ Özgür and Bartoli, 2016]. It consists of 50 shapes of an elastic surface partially stretched from its longest side in a sequential order. The images are shown in figure 4.8. We made five experiments on this dataset. These experiments include 10, 20, 30, 40 and 50 images respectively. The experiment with 10 images has the least elastic deformation (this can be seen in figure 4.8) and the one with 50 images has the most elastic deformation. For each experiment, we calculate the shape error for each image. The mean shape error calculated over the entire image set in each experiment increases about linearly with the degree of extension. Therefore, we see that the method does not collapse completely for non-isometric deformations but gracefully degrades. Table 4.3 summarises the results of this experiment.

Computation Time Comparison

We compared the performance of our methods with the others in terms of computation time on a standard computer with 16 GB RAM. Ours and the rest of the methods are implemented in MATLAB. infP takes ≈ 10 seconds for any number of images. iso is initialised with infP, and the solution to the first and the second-order derivatives of α i (x) is found iteratively upto 5 iterations. Solving for the first-order derivatives takes a similar duration as infP while solving for second-order derivatives and integrating normals have a linear complexity but they are very fast. We made an experiment with 10, 30 and 60 views. We observed that the computation time of iso (≈ 60 seconds) and infP (≈10 seconds) is almost the same (very small increase) in the three experiments. mdhI and kerF also are very fast but the computation time increases significantly as the number of images increases. plaH, diffH, pieceI and inextI show a drastic increase in computation time on changing the number of images from 10 to 30. We did not compute these timings for 60 images. Table 4.4 summarises the results.

Nearly-Stationary Objects

We made an experiment with a nearly-stationary object. In the table mat dataset, we picked the first 5 frames which are shown in figure 4.9. The mat is nearly-stationary. We observed that our methods iso and infP did get a decent reconstruction for these datasets. The errors are higher as compared to the experiments with the full sequence, as expected. Figure 4.9

shows the images and the reconstruction. The results are shown for iso. The performance of infP was almost the same as iso. This shows that the solution to Iso-NRSfM is well-posed.

There is always a solution for N ≥ 3, as long as the images are not exactly the same.

Conclusions

We proposed a theoretical framework for modelling and solving NRSfM locally for surfaces deforming isometrically. It uses Riemannian manifolds and applies to the minimal and redundant cases of N ≥ 3 views. Unlike existing methods, the proposed method has only five variables to solve for N views. Therefore, it can handle a large number of views without a significant increase in the computation time. The complexity is linear, which is a substantial improvement from the current state-of-the-art. Since the method is local, it handles missing data and occlusions. However, it does not handle self-occlusions but they could be inferred a posteriori. We proposed two methods that solve NRSfM with and without the assumption of IP. The performance of these methods is quite similar (except in computation time). This

shows that IP is a good assumption to make. We tested our methods on datasets with widebaseline and short-baseline viewpoints, large and small deformations. Our results show that the proposed methods consistently give significantly better results than the state-of-the-art methods even for as few as three views.

Introduction

In the previous chapter, we obtained a solution to NRSfM (Iso-NRSfM) for deformable thin- solutions to NRSfM (Iso-NRSfM) and SfT [START_REF] Bartoli | Shape-from-Template[END_REF] with differential modelling.

This framework allows us to go one step further in NRSfM and SfT and to model these problems for various types of deformations in terms of PDE. With the assumption of IP, we show that these PDE can be solved algebraically. We show that nonetheless equiareal NRSfM cannot be solved locally.

An interesting result in this chapter is the solvability of SfT using geometric constraints instead of deformation constraints. We show that the solution is unique and well-posed.

This chapter contributes a modelling framework for deformable 3D reconstruction. It is general, practical and easy to use.

Chapter outline. We present the mathematical background of this framework in section 5.2. It describes the connections, moving frames and shows how to express deformation constraints using these quantities. Section 5.3 proposes a framework for model-based reconstruction of deformable objects (SfT) and the reconstruction algorithm. Section 5.4 proposes a framework for model-free reconstruction of deformable objects (NRSfM) and the reconstruction algorithm. Section 5.5 discusses the experiments and section 5.6 concludes.

Mathematical Background

We describe surfaces and deformations using concepts from differential geometry and the theory of manifolds. We discuss next some of the deformations relevant to this framework.

Affine Connections

In differential geometry, a connection is a geometric object attached to a point on a smooth manifold. It transports some of the geometric properties (such as lengths, angles and areas)

of the surface at this point to its neighbourhood.

Definition 1. Cartan's affine connection is a set of geometric relations (expressed in terms of the affine moving frame defined locally at a point in space) that relate a point to its infinitesimal neighbourhood.

In order to define an affine connection in an n-dimensional space, we fix an origin O

and n-linearly independent basis vectors (e 1 , e 2 , . . . , e n ) originating from O which describe a frame at f as (5.1) where f t are the coordinates of the point f on the manifold. An affine connection is described using the following first order differential system which represents the vectorial variations of f and the coordinate basis:

f = O + f 1 e 1 + f 2 e 2 + ....... + f n e n ,
df = w 1 e 1 + w 2 e 2 + . . . + w n e n de t = w 1 t e 1 + w 2 t e 2 + . . . + w n t e n , (5.2)
where the 1-forms (see appendix C for details) w t and w s t are known as dual and connection forms respectively. Note that this formulation of connections is applicable to any kind of frame. In case of moving frames, a connection 1-form describes the change in the moving frame as one moves to the infinitesimal neighbourhood of a point on the manifold. An affine connection defines the local geometric and physical properties (such as unit lengths and areas) of the affine space around a point in the manifold. Cartan's vision behind connections [START_REF] Cartan | Sur les variétés à connexion affine, et la théorie de la relativité généralisée (première partie)[END_REF][START_REF] Cartan | Sur les varietes a connexion projective[END_REF][START_REF] Cartan | Espaces à connexion affine, projective et conforme[END_REF] was to define geometric properties on an object without defining the object itself. He derived these laws using the theory of moving frames [START_REF] Cartan | La théorie des groupes finis et continus et la géométrie différentielle traitées par la méthode du repère mobile[END_REF] which we describe next.

Moving Frames on Surfaces

The moving frame in a 3D surface M is a set of 3 linearly independent vectors. It can be defined in several ways. The connection forms defined in equation ( 5.2) are dependent on the particular choice of the moving frame. In general we assume that any point on the surface (manifold) admits a local parametrisation described by the function f (x 1 , x 2 ) (see figure 5.1).

A natural choice is to use the tangent vectors of the surface to define the moving frame:

e 1 = ∂f ∂x 1 e 2 = ∂f ∂x 2 e 3 = ∂f ∂x 1 × ∂f ∂x 2 ,
(5.3) with e 1 e 3 = 0 and e 2 e 3 = 0. The expression of the total derivative of f in terms of the moving frame is given by

df = ∂f ∂x 1 dx 1 + ∂f ∂x 2 dx 2 , ( 5.4)
By identifying the terms in equation ( 5.4) with equation ( 5.2), we obtain the dual forms w 1 = dx 1 , w 2 = dx 2 and w 3 = 0. The connection forms can be found by taking the total derivative of the basis vectors (5.5) and finding the representation of equation (5.5) in the basis formed by the moving frame (e 1 , e 2 , e 3 ): (5.6) where w s t = Γ s t1 dx 1 + Γ s t2 dx 2 and Γ s tk are scalar functions. In case of the frame used in equation ( 5.3), Γ s tk for s, t, k ∈ {1, 2} are CS of the surface and Γ s tk with s = 3 and t, k ∈ {1, 2} contain the coefficients of the second fundamental form of the surface.

de t = ∂e t ∂x 1 dx 1 + ∂e t ∂x 2 dx 2 ,
de t = w 1 t e 1 + w 2 t e 2 + w 3 t e 3 ,
Example 1. Given the following surface of a plane:

f (x 1 , x 2 ) = ux 1 + vx 2 + o, (5.7)
where u and v are three-dimensional unit vectors that define the plane and o is a threedimensional displacement vector, we define the moving frame of equation (5.3):

e 1 = u e 2 = v e 3 = u × v.
(5.8)

The total derivative is given by

df = dx 1 dx 2 u v = ω 1 ω 2 e 1 e 2 .
(5.9)

As e 1 and e 2 are constant vectors then de 1 = 0 and de 2 = 0. This makes all connection forms w s t = 0 for t ∈ {1, 2} and s ∈ {1, 2, 3}. We remind that this result is correct for the particular parametrization of the plane given in equation (5.7) and the moving frame in equation (5.8).

Example 2. If the plane of equation (5.7) is projected into the image with coordinates

(x 1 , x2 ), it can be alternatively parametrised using the image embedding φ from equation (3.3) as

φ = 1 β(x 1 , x2 ) ⎛ ⎜ ⎝ x1 x2 1 ⎞ ⎟ ⎠ ,
(5.10)

with β(x 1 , x2 ) = - n ⎛ ⎜ ⎝ x1 x2 1 ⎞ ⎟ ⎠ n o n = u × v.
(5.11)

The moving frame on the surface M described by the image embedding (5.11) is given by

e 1 = φ 1 = 1 β 2 β -x1 β 1 , -x 2 β 1 , -β 1 e 2 = φ 2 = 1 β 2 -x 1 β 2 , β -x2 β 2 , -β 2 e 3 = φ 1 × φ 2 = 1 β 3 β 1 , β 2 , β -x1 β 1 -x2 β 2 .
(5.12)

Using the frame and its first-order derivatives, we obtain the connection forms w s t related to φ. We write w s t = Γ s tk dx k and obtain

⎛ ⎜ ⎝ Γ 1 11 Γ 2 11 Γ 3 11 Γ 1 21 Γ 2 21 Γ 3 21 Γ 1 31 Γ 2 31 Γ 3 31 ⎞ ⎟ ⎠ = - 1 β ⎛ ⎜ ⎝ 2β 1 0 0 β 2 β 1 0 0 0 3β 1 ⎞ ⎟ ⎠ ⎛ ⎜ ⎝ Γ 1 12 Γ 2 12 Γ 3 12 Γ 1 22 Γ 2 22 Γ 3 22 Γ 1 32 Γ 2 32 Γ 3 32 ⎞ ⎟ ⎠ = - 1 β ⎛ ⎜ ⎝ β 2 β 1 0 0 2β 2 0 0 0 3β 2 ⎞ ⎟ ⎠ .
(5.13)

As expected, the connections in equation ( 5.13) are different from those of Example 1.

Equation (5.13) plays an important role in this paper as it shows the general structure of the connection's coefficients for planar surfaces and by extension the IP approximation of surfaces described by their image embeddings.

We now show how to write moving frames and connections on planar surfaces.

Moving Frames and Parametrisations

Figure 5.2 shows a surface M defined using two different parametrizations f (x) and g(x).

The domains of f and g are mutually homeomorphic and η is the diffeomorphic mapping that expresses the change of coordinates: .14) By differentiating, we obtain the relation between the dual forms of the two parametrisations as

x = η(x) = η 1 (x) η 2 (x) . ( 5
dx 1 = ∂x 1 ∂ x1 dx 1 + ∂x 1 ∂ x2 dx 2 dx 2 = ∂x 2 ∂ x1 dx 1 + ∂x 2 ∂ x2 dx 2 .
(5.15) Given that g = f • η in figure 5.2, we obtain J g = J f J η . Therefore, the moving frames vectors (ē 1 , ē2 , ē3 ) and (e 1 , e 2 , e 3 ) defined using (x 1 , x2 ) and ( x (5.17)

In these equations, we express e 1 e 2 e 3 as e 1 e 2 J η |J η |e 3 J -1 η |J η | -1 and using equation ( 5.16), we write the relation between the connection forms as w1

1 w1 2 w2 1 w2 2 = J -1 η w 1 1 w 1 2 w 2 1 w 2 2 J η + J -1 η dJ η w3 1 w3 2 = |J η | -1 w 3 1 w 3 2 J η w1 3 w2 3 = J -1 η w 1 3 w 2 3 |J η | w3 3 = w 3 3 + d (|J η |) |J η | -1 .
(5.18) Equations ( 5.16) and (5.18) show that the moving frame and the connections of the 3D surface M (in figure 5.2) derived using the functions f and g are linearly related in terms of the first and second-order derivatives of η. We refer to these equations as the change of variable equations of moving frames and connections. In the next section, we show how to draw relations between different surfaces.

Smooth Mappings between Surfaces

Mappings are defined by functions that connect points between two surfaces. Figure 5.3

shows a mapping ψ : M 1 → M 2 between surface M 1 and M 2 , respectively parametrized by embeddings f (x) and g(x). The mapping ψ takes a point z on M 1 and transports it to z on M 2 . Therefore, z = ψ(z). We assume that mappings between surfaces are diffeomorphic which implies that they are smooth, bijective and with a smooth inverse.

Among this general class of mappings we study the following types: 1) isometric (distances-preserving), 2) conformal (angle-preserving), 3) equiareal (area-preserving) and 4) skewless (orthogonal frame basis' angle-preserving). The set of isometric mappings is a subset of 2), 3) and 4) and is also given by the intersection of 2) and 3). The set of skewless mappings includes 1), 2) and a subset of 3). These properties are described in figure 5.4. All these kinds of mappings are identified by how they affect the metric tensor (first fundamental form) between the two surfaces. They are thus defined with first order differential constraints. Given that g and ḡ are the metric tensors of M 1 and M 2 respectively, where:

g = J f J f ḡ = J g J g (5.19)
the four categories of mappings are described by the following invariants:

Type of mapping Invariant

Isometric g = ḡ Conformal g ∝ ḡ Equiareal |g| = |ḡ| Skewless γ(g) = γ(ḡ) (5.20)
where

γ(g) = g 2 12 g 11 g 22 (5.21)
Given the moving frames (e 1 , e 2 , e 3 ) at M 1 and (ē 1 , ē2 , ē3 ) at M 2 from equation ( 5.3), the coefficients of the metric tensor can be described as follows from equation (5.19):

g = e 1 e 1 e 1 e 2
e 2 e 1 e 2 e 2 ḡ = ē 1 ē1 ē 1 ē2 ē 2 ē1 ē 2 ē2 (5.22)

From equations (5.22) and ( 5.20), the four categories of mappings can be transferred to constraints on the frames.

Isometric Mappings

Isometric mappings preserve the lengths and angles of the moving frames on the corresponding points on the two surfaces. The constraints are ē 1 ē1 = e 1 e 1 ē 2 ē2 = e 2 e 2 ē 1 ē2 = e 1 e 2 .

(5.23)

Conformal Mappings

Conformal mappings preserve the angles of the moving frames on the corresponding points on the two surfaces but the lengths of the frame vectors are isotropically scaled. The constraints are ē 1 ē1 ē 2 ē2 ē 1 ē2 ∝ e 1 e 1 e 2 e 2 e 1 e 2 .

(5.24)

Equiareal Mappings

Equiareal mappings preserve the area (expressed as the squared norm of the cross product of tangent vectors) of the tangent plane defined by the moving frames on the corresponding points on the two surfaces. The constraint is

|ē 1 × ē2 | 2 = ē 1 ē1 ē 2 ē2 -ē 1 ē2 2 = e 1 e 1 e 2 e 2 -e 1 e 2 2 = |e 1 × e 2 | 2 .
(5.25)

Skewless Mappings

Skewless mappings preserve the angles along the orthogonal frame basis on the surface. It is composed of two local anisotropic scaling along the orthogonal frame basis followed by a conformal mapping (see figure 5.5 for more details). Next we prove a theorem to formalise the construction of these mappings.

Theorem 5 (Skewless Mappings).

A mapping is skewless iff it can be only decomposed into a conformal mapping and two anisotropic scaling along the orthogonal frame basis.

Proof. First we prove the reverse implication of the theorem, i.e., two anisotropic scaling and a conformal mapping lead to a skewless mapping. In figure 5.5, we have

ψ xyc = ψ c • ψ y • ψ x .
(5.26)

Since ψ x and ψ y are anisotropic local scaling along the orthogonal frame basis, they preserve the angles along the orthonormal frame basis. Therefore, ψ x , ψ y and their composition is a skewless mapping. ψ c is a conformal mapping and therefore, it is skewless too, which makes ψ xyc a skewless mapping.

Note that ψ c • ψ y • ψ x and ψ c • ψ x • ψ y are both skewless mappings as the anisotropic scalings are commutative. However, ψ x • ψ c • ψ y is not a skewless mapping anymore due to the non-commutativity of the conformal mapping.

In order to prove the forward implication, we need to show that a skewless mapping ψ xyc can always be decomposed into a conformal mapping ψ c and two anisotropic local scalings along the orthogonal frame basis.

We can always express ψ xyc as

ψ xyc = ψ c • ψ y • ψ x • ψ u , ( 5.27)
where ψ u is an unknown mapping. On decomposing ψ xyc as in equation ( 5.27), we have that ψ y • ψ x • ψ u must be a skewless mapping. ψ u cannot be a conformal mapping as it is non-commutative with anisotropic scalings. ψ u can be expressed as (5.28) where ψ x and ψ y are transformations along the orthogonal frame basis. ψ x y represents a transformation that is not along the orthogonal frame basis and therefore it does not preserve the angles along the orthogonal frame basis.

ψ u = ψ x • ψ y • ψ x y ,
Therefore ψ y • ψ x • ψ u can only be a skewless mapping if ψ x y is identity. Otherwise, it causes a scaling which is not along the orthogonal frame basis which makes ψ y • ψ x • ψ u a non-skewless mapping.

Hence a skewless mapping can only be defined as a combination of anisotropic scaling followed by a conformal mapping.

The constraint for skewless mappings is

ē 1 ē2 2 ē 1 ē1 ē 2 ē2 = e 1 e 2 2
e 1 e 1 e 2 e 2 .

(5.29)

It is possible to propose more kinds of mappings and express the properties they preserve in terms of the moving frames. In the next section we will show how to exploit these properties (expressed in terms of moving frames) for the reconstruction of deformable objects.

Infinitesimally Linear Mappings between Surfaces

In this section we show that linear mappings preserve the connection forms across surfaces. This is a very important property that we use in this chapter to propose a reconstruction algorithm (without having to categorise the mapping according to equation (5.20)) by extending these properties to the so-called infinitesimally linear maps.

Theorem 6. (Linear Mappings) A linear mapping between two planes preserves the connection forms (w 1 1 , w 1 2 , w 2 1 , w 2 2 ).

Proof. Given that g = ψ • f in figure 5.3, we have J g = 0. The moving frames (e 1 , e 2 , e 3 ) and

(ē 1 , ē2 , ē3 ) on the planes M 1 and M 2 are related by

ē1 ē2 = J ψ•f e 1 e 2 = J ψ•f e 1 J ψ•f e 2 .
(5.30) Equation (5.30) shows the relation between the moving frames (e 1 , e 2 , e 3 ) at M 1 and (ē 1 , ē2 , ē3 ) at M 2 . Therefore, on differentiating this relation we obtain

dē 1 dē 2 = J ψ•f de 1 de 2 + dJ ψ•f e 1 e 2 .
(5.31)

Given that ψ is a linear mapping, dJ ψ•f = 0 in the previous equation. We multiply the two expressions obtained in the above equation with ē 1 and expand the expression using the connection relations in equation ( 5.2) and obtain

w1 1 ē1 ē 1 + w2 1 ē2 ē 1 = J ψ•f w 1 1 e 1 + w 2 1 e 2 ē 1 w1 2 ē1 ē 1 + w2 2 ē2 ē 1 = J ψ•f w 1 2 e 1 + w 2 2 e 2 ē 1 .
(5.32) Since w t s are scalar functions (they are differential forms), we employ equation ( 5.30) and rewrite the above expression as

w1 1 ē1 ē 1 + w2 1 ē2 ē 1 = w 1 1 ē1 ē 1 + w 2 1 ē2 ē 1 w1 2 ē1 ē 1 + w2 2 ē2 ē 1 = w 1 2 ē1 ē 1 + w 2 2 ē2 ē 1 .
(5.33)

Since (ē 1 , ē2 ) are linearly independent, on expressing w s t in terms of Γ s tk dx k in the above equation we obtain Γs tk = Γ s tk and therefore, w1

1 w1 2 w2 1 w2 2 = w 1 1 w 1 2 w 2 1 w 2 2 .
We use the concept of IL mappings which allows us to extend the result of Theorem 6

for mappings between two generic surfaces. To summarise, it is possible to find constraints on smooth mappings (with an assumption of IL) without adding a deformation prior (like isometry or conformity).

In the next section, we show how to use theorem 6, the moving frame constraints derived for smooth mappings in equations ( 5.23)-( 5.25) and the change of variable for moving frames in equation (5.16) and connections in equation ( 5.18) for 3D reconstruction of deformable objects.

Model-Based 3D Reconstruction

We now propose a general framework based on the theory of connections presented in the previous section to model and to solve SfT for thin-shell objects.

φ 1 (x 1 , x 2 ) = 1 β 1 (x 1 , x 2 ) ⎛ ⎜ ⎝ x 1 x 2 1 ⎞ ⎟ ⎠ φ 2 (x 1 , x2 ) = 1 β2 (x 1 , x2 ) ⎛ ⎜ ⎝ x1 x2 1 ⎞ ⎟ ⎠
(5.34)

Following equation ( 5.3) the moving frames (e t 1 , e t 2 , e t 3 ) and (ē 1 , ē2 , ē3 ) on T and M respectively are given as

e t 1 = φ 11 = 1 β 2 1 β 1 -x 1 β 11 , -x 2 β 11 , -β 11 e t 2 = φ 12 = 1 β 2 1 -x 1 β 12 , β 1 -x 2 β 12 , -β 12 e t 3 = φ 11 × φ 12 = 1 β 3 1 β 11 , β 12 , β 1 -x 1 β 11 -x 2 β 12 , (5.35) ē1 = φ21 = 1 β2 2 β2 -x1 β21 , -x 2 β21 , -β21 ē2 = φ22 = 1 β2 2 -x 1 β22 , β2 -x2 β22 , -β22 ē3 = φ21 × φ 22 = 1 β3 2 β21 , β22 , β2 -x1 β21 -x2 β22 .
(5.36)

Using equation (5.22) we compute the elements of the metric tensor in T as:

e t 1 e t 1 = 1 k 2 2 k 2 1 + 1 -2x 1 k 1 = E t k 2 e t 2 e t 2 = 1 k 2 2 k 2 2 + 1 -2x 2 k 2 = G t k 2 e t 1 e t 2 = 1 k 2 2 k 1 k 2 -x 1 k 2 -x 2 k 1 = F t k 2 ,
(5.37)

where 2 = 1 + x 1 2 + x 2 2 , k = β 1 , k 1 = β 11 β 1 and k 2 = β 12 β 1
. Under the assumption of infinitesimal planarity, the first-order derivatives of these expressions are given by

∂E t ∂x 1 = -2k 1 E t ∂F t ∂x 1 = -k 2 E t -k 1 F t ∂G t ∂x 1 = -2k 2 F t ∂E t ∂x 2 = -2k 1 F t ∂F t ∂x 2 = -k 1 G t -k 2 F t ∂G t ∂x 2 = -2k 2 G t .
(5.38) Similarly, we can write ( Ē, F , Ḡ) and their first-order derivatives for ( ē1 , ē2 , ē3 ) at M in terms of ( k1 , k2 , k).

In order to compare the moving frames at the two surfaces T and M, we need to define them in the same parametrisation space. Therefore, we derive (E, F, G) at M 1 in terms of (x 1 , x2 ) by using the change of variable as suggested in equation (5.16)

E = ∂x 1 ∂ x1 2 E t + 2 ∂x 1 ∂ x1 ∂x 2 ∂ x1 F t + ∂x 2 ∂ x1 2 G t F = ∂x 1 ∂ x1 ∂x 1 ∂ x2 E t + ∂x 1 ∂ x1 ∂x 2 ∂ x2 + ∂x 2 ∂ x1 ∂x 1 ∂ x2 F t + ∂x 2 ∂ x1 2 G t G = ∂x 1 ∂ x2 2 E t + 2 ∂x 1 ∂ x2 ∂x 2 ∂ x2 F t + ∂x 2 ∂ x2 2 G t .
(5.39)

The derivatives of (E, F, G) according to equation (5.38) are given by

∂E ∂ x1 = -2 ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 E + 2 ∂ 2 x 1 ∂ (x 1 ) 2 A1 + 2 ∂x 2 ∂ (x 1 ) 2 A2 ∂E ∂ x2 = -2 ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 F + 2 ∂ 2 x 1 ∂ x1 ∂ x2 A1 + 2 ∂x 2 ∂ x1 ∂ x2 A2 ∂G ∂ x1 = -2 ∂x 1 ∂ x2 k 1 + ∂x 2 ∂ x2 k 2 F + 2 ∂ 2 x 1 ∂ x1 ∂ x2 C1 + 2 ∂x 2 ∂ x1 ∂ x2 C2 ∂G ∂ x2 = -2 ∂x 1 ∂ x2 k 1 + ∂x 2 ∂ x2 k 2 G + 2 ∂ 2 x 1 ∂ (x 2 ) 2 C1 + 2 ∂x 2 ∂ (x 2 ) 2 C2 ∂F ∂ x1 = - ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 F - ∂x 1 ∂ x2 k 1 + ∂x 2 ∂ x2 k 2 E + ∂ 2 x 1 ∂ x1 ∂ x2 A1 + ∂x 2 ∂ x1 ∂ x2 A2 + ∂ 2 x 1 ∂ (x 1 ) 2 C1 + ∂x 2 ∂ (x 1 ) 2 C2 ∂F ∂ x1 = - ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 G - ∂x 1 ∂ x2 k 1 + ∂x 2 ∂ x2 k 2 F + ∂ 2 x 1 ∂ x1 ∂ x2 C1 + ∂x 2 ∂ x1 ∂ x2 C2 + ∂ 2 x 1 ∂ (x 2 ) 2 A1 + ∂x 2 ∂ (x 2 ) 2 a2
where

A1 = ∂x 1 ∂ x1 E t + ∂x 2 ∂ x1 F t , A2 = ∂x 1 ∂ x1 F t + ∂x 2 ∂ x1 G t C1 = ∂x 1 ∂ x2 E t + ∂x 2 ∂ x2 F t , C2 = ∂x 1 ∂ x2 F t + ∂x 2 ∂ x2 G t .
(5.40)

Now that we have both (E, F, G) and ( Ē, F , Ḡ) with respect to (x 1 , x2 ), we write the constraints for various kinds of deformations expressed in equations ( 5.23)- (5.25). Since (k 1 , k 2 , k) are known, our goal is to find ( k1 , k2 , k) in order to obtain the normal and depth at M.

Reconstruction Equations for Smooth Mappings

We now derive the reconstruction equations for the various types of deformations we discussed in section 5.2.

Isometric Mappings

Given that ψ is an isometric mapping, the relationship between the moving frames at M and T are described in equation (5.23). This gives the following three constraints:

E k 2 = Ē k2 F k 2 = F k2 G k 2 = Ḡ k2 .
(5.41)

We differentiate these constraints under the assumption of IP (using the expressions in equations (5.38) and ( 5.40)) and obtain the following equations

k1 = ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 - 1 2E ∂ 2 x 1 ∂ (x 1 ) 2 A1 + ∂ 2 x 2 ∂ (x 1 ) 2 A2 k2 = ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 - 1 2G ∂ 2 x 1 ∂ (x 2 ) 2 C1 + ∂ 2 x 2 ∂ (x 2 ) 2 C2 ,
where

A1 = ∂x 1 ∂ x1 E + ∂x 2 ∂ x1 F, A2 = ∂x 1 ∂ x1 F + ∂x 2 ∂ x1 G, C1 = ∂x 1 ∂ x2 E + ∂x 2 ∂ x2 F, C2 = ∂x 1 ∂ x2 F + ∂x 2 ∂ x2 G.
(5.42)

These expressions are linear in ( k1 , k2 ), independent of (k, k) and exploit the first and the second-order derivatives of the warp η. k can be recovered from equation (5.41) using ( k1 , k2 ) obtained from the solution of equation (5.42).

Conformal Mappings

Given that ψ is a conformal mapping, the relationship between the moving frames at M and T are described in equation (5.24). This gives the following three constraints:

1 k 2 E G F ∝ 1 k2 Ē Ḡ F 1 k 2 E G F = α 2 1 k2 Ē Ḡ F , (5.43)
where α 2 is the scale of conformity. On differentiating these expressions, we obtain equation (5.42) with ( k1 , k2 ) replaced with α 2 ( k1 , k2 ). We get rid off the (k, k) and ∝ in equation (5.43) by taking ratios of (E, F, G) with ( Ē, F , Ḡ). This leads to

E Ḡ = ĒG F Ḡ = F G.
(5.44)

Using ( k1 , k2 ) obtained from the solution of equations (5.42) in this equation, we obtain a quadratic equation in α 2 which leads to two possible solutions (upto ambiguity) for conformal mappings. k can be recovered from equation (5.43) using ( k1 , k2 ) and α 2 .

[ [START_REF] Bartoli | Shape-from-Template[END_REF] proposed an analytical solution for isometric and conformal mappings by solving the quadratic expressions in (5.41).

Equiareal Mappings

Given that ψ is an equiareal mapping, the relationship between the moving frames at M and T is described in equation (5.25). This is the only constraint due to the equiareal mappings.

By differentiating this constraint, we obtain two first-order constraints.

EG k 4 - F k 4 2 = Ē Ḡ k4 - F 2 k4 ∂ ∂ xi Ē k 2 Ḡ k2 + Ē k2 ∂ ∂ xi Ḡ k 2 -2 F k2 ∂ ∂ xi F k 2 = ∂ ∂x 1 E k 2 ∂x 1 ∂ xi + ∂ ∂x 2 E k 2 ∂x 2 ∂ xi G k 2 + ∂ ∂x 1 G k 2 ∂x 1 ∂ xi + ∂ ∂x 2 G k 2 ∂x 2 ∂ xi E k 2 -2 F k 2 ∂ ∂x 1 F k 2 ∂x 1 ∂ xi + ∂ ∂x 2 F k 2 ∂x 2 ∂ xi ∀i ∈ (1, 2) (5.45)
Using the expressions of differentials from equations (5.38) and ( 5.40), the above-mentioned constraints are written as

EG k 4 - F k 4 2 = det (J η ) 2 E t G t k 4 - F t k 4 2 = Ē Ḡ k4 - F 2 k4 k1 Ē Ḡ k4 - F 2 k4 = E t G t k 4 - F t k 4 2 det (J η ) 2 k 1 ∂x 1 ∂ x1 + k 2 ∂x 2 ∂ x1 - E t G t k 4 - F t k 4 2 det J η ∂x 1 ∂ x1 ∂ 2 x 2 ∂ x1 ∂ x2 + ∂x 2 ∂ x2 ∂ 2 x 1 (∂ x1 ) 2 + E t G t k 4 - F t k 4 2 det J η ∂x 2 ∂ x1 ∂ 2 x 1 ∂ x1 ∂ x2 + ∂x 1 ∂ x2 ∂ 2 x 2 (∂ x1 ) 2 k2 Ē Ḡ k4 - F 2 k4 = E t G t k 4 - F t k 4 2 det (J η ) 2 k 1 ∂x 1 ∂ x2 + k 2 ∂x 2 ∂ x2 - E t G t k 4 - F t k 4 2 det J η ∂x 1 ∂ x1 ∂ 2 x 2 (∂ x2 ) 2 + ∂x 2 ∂ x2 ∂ 2 x 1 ∂ x1 ∂ x2 + E t G t k 4 - F t k 4 2 det J η ∂x 2 ∂ x1 ∂ 2 x 1 (∂ x2 ) 2 + ∂x 1 ∂ x2 ∂ 2 x 2 ∂ x1 ∂ x2 . (5.46)
Using the first constraint in the rest, we end up with the following expressions

k1 det J η = det J η k 1 ∂x 1 ∂ x1 + k 2 ∂x 2 ∂ x1 - ∂x 1 ∂ x1 ∂ 2 x 2 ∂ x1 ∂ x2 + ∂x 2 ∂ x2 ∂ 2 x 1 (∂ x1 ) 2 + ∂x 2 ∂ x1 ∂ 2 x 1 ∂ x1 ∂ x2 + ∂x 1 ∂ x2 ∂ 2 x 2 (∂ x1 ) 2 k2 det J η = det J η k 1 ∂x 1 ∂ x2 + k 2 ∂x 2 ∂ x2 - ∂x 1 ∂ x1 ∂ 2 x 2 (∂ x2 ) 2 + ∂x 2 ∂ x2 ∂ 2 x 1 ∂ x1 ∂ x2 + ∂x 2 ∂ x1 ∂ 2 x 1 (∂ x2 ) 2 + ∂x 1 ∂ x2 ∂ 2 x 2 ∂ x1 ∂ x2 .
(5.47)

Given that (k 1 , k 2 ) and the first and second-order derivatives of η are known, these expressions are linear in ( k1 , k2 ). k can be found by using ( k1 , k2 ) obtained from equation (5.47) in equation (5.45).

Skewless Mappings

Given that ψ is a skewless mapping, the relationship between the moving frames at M and T is described in equation (5.29). We write the constraint as

F 2 Ē Ḡ = F 2 EG. (5.48)
This expression is independent of (k, k). This constraint can be rewritten as

Ē Ḡ -F 2 F 2 = (det J η ) 2 EG -F 2 F 2 Ē Ḡ -F 2 EG = (det J η ) 2 EG -F 2 Ē Ḡ.
(5.49)

By differentiating the skewless constraint in equation ( 5.48), we get

2F ∂F ∂x 1 ∂x 1 ∂ xi + ∂F ∂x 2 ∂x 2 ∂ xi Ē Ḡ + F 2 ∂ Ē ∂ xi Ḡ + Ē ∂ Ḡ ∂ xi = 2 F ∂ F ∂ xi EG + F 2 ∂E ∂x 1 ∂x 1 ∂ xi + ∂E ∂x 2 ∂x 2 ∂ xi G + F 2 E ∂G ∂x 1 ∂x 1 ∂ xi + ∂G ∂x 2 ∂x 2 ∂ xi ∀i ∈ (1, 2)
(5.50)

We expand these expressions using equations (5.38) and ( 5.40) and obtain

EF Ē Ḡ -F 2 ( ∂x 1 ∂ x2 k 1 + ∂x 2 ∂ x2 ) -F Ē(EG -F 2 ) k2 = F Ē Ḡ ∂ 2 x 1 ∂ x1 ∂ x2 A1 + ∂x 2 ∂ x1 ∂ x2 A2 + ∂ 2 x 1 ∂ (x 1 ) 2 C1 + ∂x 2 ∂ (x 1 ) 2 C2 F 2 E ∂ 2 x 1 ∂ x1 ∂ x2 C1 + ∂x 2 ∂ x1 ∂ x2 C2 + G ∂ 2 x 1 ∂ (x 1 ) 2 A1 + ∂x 2 ∂ (x 1 ) 2 A2 GF Ē Ḡ -F 2 ( ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 ) -F Ḡ(EG -F 2 ) k1 = F Ē Ḡ ∂ 2 x 1 ∂ x1 ∂ x2 C1 + ∂x 2 ∂ x1 ∂ x2 C2 + ∂ 2 x 1 ∂ (x 2 ) 2 A1 + ∂x 2 ∂ (x 2 ) 2 A2 F 2 G ∂ 2 x 1 ∂ x1 ∂ x2 A1 + ∂x 2 ∂ x1 ∂ x2 A2 + E ∂ 2 x 1 ∂ (x 2 ) 2 C1 + ∂x 2 ∂ (x 2 ) 2 C2
where

A1 = ∂x 1 ∂ x1 E + ∂x 2 ∂ x1 F, A2 = ∂x 1 ∂ x1 F + ∂x 2 ∂ x1 G, C1 = ∂x 1 ∂ x2 E + ∂x 2 ∂ x2 F, C2 = ∂x 1 ∂ x2 F + ∂x 2 ∂ x2 G.
Using equations (5.48) and (5.49) in the above expressions, we obtain the following two constraints for skewless deformation

EG F k2 -EF Ḡ k 1 ∂x 1 ∂ x2 + k 2 ∂x 2 ∂ x2 + B 1 Ḡ + det J -1 η D 1 EG Ḡ = 0 EG F k1 -GF Ē k 1 ∂x 1 ∂ x1 + k 2 ∂x 2 ∂ x1 + B 2 Ē + det J -1 η D 2 EG Ē = 0.
where

A 1 = ∂x 1 ∂ x1 E t + ∂x 2 ∂ x1 F t , A 2 = ∂x 1 ∂ x1 F t + ∂x 2 ∂ x1 G t , C 1 = ∂x 1 ∂ x2 E t + ∂x 2 ∂ x2 F t , C 2 = ∂x 1 ∂ x2 F t + ∂x 2 ∂ x2 G t , B 1 = G A 1 ∂ 2 x 1 (∂x 1 ) 2 + A 2 ∂ 2 x 2 (∂x 1 ) 2 + E C 1 ∂ 2 x 1 ∂x 1 ∂x 2 + C 2 ∂ 2 x 2 ∂x 1 ∂x 2 , B 2 = F A 1 ∂ 2 x 1 ∂x 1 ∂x 2 + A 2 ∂ 2 x 2 ∂x 1 ∂x 2 + E C 1 ∂ 2 x 1 (∂x 2 ) 2 + C 2 ∂ 2 x 2 (∂x 2 ) 2 , D 1 = - ∂x 2 ∂ x2 ∂ 2 x 1 (∂x 1 ) 2 + ∂x 1 ∂ x2 ∂ 2 x 2 (∂x 1 ) 2 + ∂x 2 ∂ x1 ∂ 2 x 1 ∂x 1 ∂x 2 - ∂x 1 ∂ x1 ∂ 2 x 2 ∂x 1 ∂x 2 , D 2 = - ∂x 2 ∂ x2 ∂ 2 x 1 ∂x 1 ∂x 2 + ∂x 1 ∂ x2 ∂ 2 x 2 ∂x 1 ∂x 2 + ∂x 2 ∂ x1 ∂ 2 x 1 (∂x 2 ) 2 - ∂x 1 ∂ x1 ∂ 2 x 2 (∂x 2 ) 2 .
(5.51) Given that (k 1 , k 2 ) and the first and second-order derivatives of η are known and ( Ē, F , Ḡ) are quadratic in ( k1 , k2 ) (written according to equation (5.37)), these expressions are cubic in ( k1 , k2 ). These expressions are independent of (k, k) and therefore, k cannot be found for skewless mappings.

Reconstruction Equations for Infinitesimally Linear Mappings

Given that ψ an infinitesimally linear mapping, according to theorem 6, the connection forms

( w f 1 1 , w f 1 2 , w f 2 1 , w f 2
2 ) defined using φ 1 at T are the same as the respective connection forms ( w1 1 , w1 2 , w2 1 , w2 2 ) defined using θ at M. Using equation (5.18), the connection forms at M defined using θ are related by a change of variable with the CS at the same surface defined using φ 2 .

Therefore, the connections at T (parametrised with x 1 and x 2 ) are related to the connections at M (parametrised with x1 and x2 ) by a change of variable suggested in equation (5.18) for any smooth and IL mapping ψ between them. The components of these connection forms (5.52) where k = (1, 2) and

= ∂ xk ∂x 1 J -1 η Γ 1 11 Γ 1 12 Γ 1 21 Γ 1 22 J η + ∂ xk ∂x 2 J -1 η Γ 2 11 Γ 2 12 Γ 2 21 Γ 2 22 J η + J -1 η ∂J η ∂x k ,
J -1 η = ⎛ ⎜ ⎝ ∂ x1 ∂x 1 ∂ x1 ∂x 2 ∂ x2 ∂x 1 ∂ x2 ∂x 2 ⎞ ⎟ ⎠.
These components of the connection forms for the image embedding described in equation (3.4) are evaluated in equation ( 5.12). For T and M, these expressions are written only in terms of (k 1 , k 2 ) and ( k1 , k2 ) respectively. This leads to the following two constraints

k1 = ∂x 1 ∂ x1 k 1 + ∂x 2 ∂ x1 k 2 - ∂ x2 ∂x 1 ∂ 2 x 1 ∂ x1 x2 - ∂ x2 ∂x 2 ∂ 2 x 2 ∂ x1 x2 k2 = ∂x 1 ∂ x2 k 1 + ∂x 2 ∂ x2 k 2 - ∂ x1 ∂x 1 ∂ 2 x 1 ∂ x1 x2 - ∂ x1 ∂x 2 ∂ 2 x 2 ∂ x1 x2 .
(5.53)

Since (k 1 , k 2 ) are known, these equations are linear in terms of ( k1 , k2 ). These expressions are independent of (k, k) and therefore, k cannot be found for IL mappings.

We rewrite this solution as

1 0 0 1 k1 k2 = J η k 1 k 2 - ⎛ ⎜ ⎝ ∂ x2 ∂x 1 ∂ 2 x 1 ∂ x1 x2 + ∂ x2 ∂x 2 ∂ 2 x 2 ∂ x1 x2 ∂ x1 ∂x 1 ∂ 2 x 1 ∂ x1 x2 + ∂ x1 ∂x 2 ∂ 2 x 2 ∂ x1 x2 ⎞ ⎟ ⎠ .
(5.54)

The right hand side of the expression is known and therefore we see that there is a unique solution for IL mappings. This is coherent with the solution in [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF] (which also shows that non-isometric SfT has a unique solution). [START_REF] Salzmann | Deformable Surface Tracking Ambiguities[END_REF] argued that it is not possible to solve SfT at zeroth-order without using a deformation prior.

Warps provide additional priors in terms of the first and the second-order derivatives which leads to additional constraints and makes our solution well-posed.

Reconstruction Algorithm

We wrote reconstruction equations for various kinds of smooth deformations in equations (5.42), (5.47), (5.51) and (5.53). All of these equations are expressed in terms of the unknowns ( k1 , k2 ). We present the following algorithm to solve these equations.

Inputs:

The warp η and (k, k 1 , k 2 ) on the corresponding points of T and M.

1) Find ( k1 , k2 ). For isometric and conformal deformations, the solution to equation (5.42) gives ( k1 , k2 ). These equations are linear in ( k1 , k2 ). For skewless deformations, equations (5.51) are cubic in ( k1 , k2 ). We solve them by minimising the sum-of-squares using [START_REF] Henrion | Gloptipoly: Global optimization over polynomials with matlab and sedumi[END_REF]. For equiareal and infinitesimally linear deformations, equa- tions (5.47) and ( 5.53) are also linear in ( k1 , k2 ) and therefore easily solvable.

3) Find normals at M. Compute unit normal at each point on M according to equation (5.12) in terms of ( k1 , k2 ).

4) Find depth at M. For isometric, conformal and equiareal deformations, the scale can be evaluated by using ( k1 , k2 ) in equations ( 5.41), (5.43) and ( 5.45) respectively. For skewless deformations, the scale cannot be recovered.

Outputs: Points (for isometric and equiareal deformations) and normals on 3D surfaces.

Model-based 3D reconstruction of deformable thin-shell objects is summarised in table 5.1.

Model-Free 3D Reconstruction

We propose local solutions to 3D reconstruction of a deformable thin-shell object from multiple views. Figure 5.7 shows the modelling. We have N input images I 1 , . . . , I N representing between the pair of images (I i , I j ). ψ ij is the deformation function between M i and M j modelled using the image embedding described in equation (3.4). We write the constraints for a pair of views i and j. This modelling can be extended to any number of image pairs. The moving frames (e i 1 , e i 2 , e i 3 ) and (ē 1 , ē2 , ē3 ) on M i and M j are written according to equation (5.12). The expressions (E i , F i , G i ), ( Ē, F , Ḡ) and their first-order derivatives are expressed according to equations (5.37) and (5.38). Using equation ( 5.39), we can write (e 1 , e 2 ) at M i which allows us to write (E, G, F ) and its derivatives with respect to (x 1 , x2 ). In this case, we can write the reconstruction equations in a similar fashion to the previous section but the unknowns are both (k 1 , k 2 , k) and ( k1 , k2 , k). Under the assumption of IL, we can express ( k1 , k2 , k) in terms of (k 1 , k 2 , k) using equation (5.53) which allows us to restrict the unknowns to (k 1 , k 2 , k) only.

Reconstruction Equations

Now we derive the reconstruction equations for the various types of smooth deformations we discussed in this chapter.

Isometric/Conformal Mappings

Given that ψ is an isometric/conformal mapping (with the isometric and conformal mapping constraints in equations ( 5.41) and ( 5.43) respectively), the reconstruction equations for a pair of views I i and I j are given in equation ( 5.44) as a set of two cubic expressions in terms of (k 1 , k 2 ) and ( k1 , k2 ). Using equation (5.53) in equation (5.44), we obtain two cubic equations in terms of (k 1 , k 2 ) only. We proposed a solution to these equations for isometric deformations in chapter 4. It is important to note that the solution in previous chapter uses metric tensors and CS. Using the proposed framework it becomes clear that isometry and conformity share the same constraints in the context of NRSfM. This result was straightforward from the theory developed in chapter 4.

Equiareal Mappings

Given that ψ is an equiareal mapping, the reconstruction equations are given in equation (5.47) in terms of (k 1 , k 2 ) and ( k1 , k2 ). Using equation (5.53) in equation ( 5.47) leads to a system of equations independent of ( k1 , k2 ) and (k 1 , k 2 ). Therefore, these expressions cannot be used to solve for (k 1 , k 2 ). Next we prove that equiareal NRSfM cannot be solved algebraically.

Theorem 7 (Non-solvability of Equiareal NRSfM). Equiareal NRSfM is not locally solvable.

Proof. The constraint in equiareal mappings is given by equation (5.45). Without the assumption of IP, these expressions are in terms of β, β and their first and second-order derivatives. Using equation (5.53), this results in 7 variables (β, β and the first and secondorder derivatives of β) in 3 equations. This system is not solvable. Even if we differentiate equation (5.45) further, it always results in higher number of variables than equations.

1) Find point correspondences. Select a grid of points on the first image and using the warps η 1j , find the corresponding grid of points in the rest of the images. We evaluated our method on a 20 × 20 grid of points.

2) Find (k 1 , k 2 ). For isometric, conformal and skewless deformations, the solution to equations (5.44) and ( 5.51) gives (k 1 , k 2 ). We solve these equations by minimising the sumof-squares using [START_REF] Henrion | Gloptipoly: Global optimization over polynomials with matlab and sedumi[END_REF].

3) Find ( k1 , k2 ). ( k1 , k2 ) can be written in terms of (k 1 , k 2 ) and the first and second order derivatives of η ji using equation (5.53). 4) Find normals at M. Compute unit normal at each point on M according to equation (5.12) in terms of ( k1 , k2 ) and (k 1 , k 2 ). 5) Find depth at M. The depth can be evaluated by using the method described in [START_REF] Chhatkuli | Non-Rigid Shape-from-Motion for Isometric Surfaces using Infinitesimal Planarity[END_REF].

Outputs: Points (for isometric deformations only) and normals on 3D surfaces.

Model-free 3D reconstruction of deformable thin-shell objects is summarised in table 5.2.

Experiments and Discussion

We tested our proposed methods for model-based and model-free reconstruction of thin-shell deformable objects on two synthetic and four real datasets. These datasets show objects undergoing different types of deformations according to the properties of their material.

The Cylinder dataset is a synthetic dataset which consists of isometric deformations of a cylindrical surface viewed in images of the size 640p × 480p with a focal length of 400p. 400 points are tracked across the isometrically deformed surfaces. The Paper dataset (introduced in [Varol et al., 2012a]) consists of 190 images (with 1500 tracked points) of a paper deforming isometrically. The synthetic Rubber dataset (introduced in [ Özgür and Bartoli, 2016]) consists of 50 partially-stretched surfaces of a rubber with 400 tracked points. are linear solutions to SfT (exploiting the first and second-order derivatives of the warp η ) while the solutions for IsoFS and ConFS are obtained by solving quadratic equations (see equation (5.20)) consisting of only the first-order derivatives of the warp and the embedding.

The constraints in our methods are obtained by using equation (5.20) and its derivatives which forces the solution to be locally smooth implicitly. Therefore, IsoS, ConS, EqArS and NoDefS show better performance than IsoFS and ConFS. SkewS (assumes the deformation to be skewless) does not show good results on this dataset. This is probably because it is less-constrained than other deformation models. IsoFS, ConFS and LinModS show a similar performance on this dataset. The performance of these methods is very good.

NoIsoS is the best performing method out of the five methods proposed in [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF]. It needs high perspective in images and therefore it performed very well in this dataset. The performance of this method is quite close to other SfT methods (IsoS, ConS, IsoFS, ConFS and LinModS). In general, this method is unstable in the sense that the performance of the proposed methods in [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF] is usually very different from each other. We picked the best performing method for each image in the dataset by comparing the reconstruction with the ground truth.

Performance of methods under noisy conditions. We evaluated the performance of the methods under noisy conditions. We added 1-5 pixel noise to the images of the dataset and obtain the results for each method. We report the normal and depth errors for each experiment averaged over 20 trials. noise even though their performance on this dataset is not so good. MDHN and KerN do not perform well on this dataset and therefore it is largely affected by noise as well.

Curvature test of our methods. The best performing methods on this dataset are EqArS and NoDefS (with NoDefS being better). These solutions (like the rest of our methods) are dependent on the first and second-order derivatives of the warps. The second-order derivatives may be unstable however they can be corrected by warp refinement methods such as [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. However it may be difficult to find them in some cases, therefore it is interesting to see the performance of these methods without second-order derivatives. We we do not perform the refinement step proposed for these methods. This is because our goal is to compare the analytical solutions of these methods to ours.

Balloon Dataset

This dataset consists of 20 images uniformly sampled from a video sequence of 80 images.

We observe that the mean normal error for all NRSfM methods on this dataset is around 20 degrees which is very high as compared to the performance of these methods on other datasets.

However, the mean depth error for these methods is quite comparable to the rest of the datasets. This indicates to the flattening of the reconstructed surface. However, IsoConN

shows the best performance amongst compared NRSfM methods. Our SfT methods except SkewS show a very good performance on this dataset and their results are very similar.

The reconstruction from SkewS is quite flat. IsoFS, ConFS and NoIsoS, all of these methods also lead to flat reconstructions. LinModS performed quite well on this dataset.

Its performance is quite close to EqArS and NoDefS which are the best performing methods on this dataset. This dataset is near-affine and therefore, most of the methods did not do well. The reconstructions of two surfaces of this dataset using the compared methods are shown in figure 5.14.

Sock Dataset

This dataset consists of 20 images from a video sequence of a sock being stretched in one direction only. Analytical methods show a very good performance on this dataset. Amongst NRSfM methods, SkewN showed the best performance as the sock is undergoing almost skewless deformation. The reconstruction of normals by IsoConN is quite good but it cannot cope with the stretching of the surface and therefore, it leads to a higher depth error. MDHN and KerN did not do well on this dataset. SfT methods except NoIsoS (it completely broke on this dataset) showed a very good reconstruction even though the object was stretched.

NoDefS shows the best performance. However, the rest of the methods are quite competitive.

An important thing to note is that even though Discussion on smoothness. An important point to note here is that NoIsoS and NoDefS both reconstruct the surface without modelling deformation explicitly. However, we found NoIsoS to be unstable. [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF] proposed five solutions to cope with the stability issue. Also, it needs the input image to have a high amount of perspective in order to give decent results. The main idea of this method is to use smoothness (expressed in terms of the second-order differentials of the surface) in order to reconstruct surfaces. Our solution to NoDefS (along with other solutions) also relies on the second-order derivatives of the surfaces under the assumption of IL. Our experiments show that our solution to NoDefS is not unstable. We discuss the difference between the two solutions now. [START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF] exploit smoothness by minimising the second order derivatives of the surface. For a function f described in equation ( 5.1), it tries to minimise d 2 f . Using equations ( 5.3), (5.4) and (5.5), we write .55) Therefore, this solution tries to minimise (de 1 , de 2 ) whereas our solution (NoDefS) is obtained by imposing a structure to the components of (de 1 , de 2 ) (connection forms in equation (5.6)). This is the underlying difference between the two methods. This structure of (de 1 , de 2 ) cannot be studied without differential geometry which makes this framework a comprehensive study of deformations.

d 2 f = d e 1 dx 1 + e 2 dx 2 = de 1 dx 1 + de 2 dx 2 . ( 5

Summary of Experiments

The results of all the compared methods on each dataset are summarised in figure 5.9. On these datasets, the best performing NRSfM methods are IsoConN and MDHN. For SfT methods, our methods (except SkewS) showed a very good performance on all the datasets.

NoDefS showed the best performance on most of the datasets. LinModS showed a very good performance on these datasets, it showed the best performance in the rubber and paper datasets. However, NoDefS and EqArS have been very competitive with LinModS.

SkewN gives very good results on the sock dataset. This is because the sock is undergoing a skewless deformation. An important point to note about skewless deformation is that it should be along the two directions of the orthogonal frame basis of the surface. Therefore, SkewN and SkewT can perform better if the basis is orthogonal and is aligned to the direction of deformation. This means that the frame basis on a surface should be aligned to the direction of deformation which needs to be known apriori. Our SfT methods (except SkewS) are linear and therefore, computationally very cheap. In general, for 50 images and 400 points, it takes 2-5 seconds for these methods to evaluate the results. The computation time for our NRSfM methods is about 15-20 seconds for around 100 points tracked over 30 images. These computation times are reported on a standard PC.

Conclusions

We presented a theoretical modelling framework for deformable 3D reconstruction using differential geometry and Cartan's theory of connections. This includes the model-based and model-free cases. We showed how to obtain solutions to isometric, conformal (same as the isometric solution) and skewless NRSfM under the assumption of IP using this framework.

We showed that equiareal NRSfM cannot be solved locally. We used this framework to find solutions to isometric, conformal, skewless and equiareal SfT under the assumption of IP.

These solutions are computationally very cheap. Using this framework, we also proposed a solution to SfT (under the assumption of IL) without modelling the deformation explicitly.

Our methods rely on the first and second-order derivatives of the warp function which are known to be unstable when obtained using sparse registration algorithms. We used a warp refinement method presented in [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. As discussed in the previous chapter, it is a theoretical requirement for our methods to comply with the Schwarzian equations presented in [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. Therefore, it makes sense to use this method for warp correction. Our NRSfM methods solve the minimum data case, have linear complexity in the number of points, handle missing data, work for both short and wide-baseline datasets, work with both large and small number of images. Our experiments showed that the proposed NRSfM and SfT methods outperform most of the compared methods in terms of accuracy and computation time.

Introduction

The objective in SfT is to obtain the object's deformed shape in the camera coordinate frame using a deformation constraint formulated from the object's physical material. Existing

SfT methods use deformation constraints on the object's outer surface, whose thickness is considered infinitesimal. We thus call them thin-shell SfT methods. Thin-shell SfT is very well adapted to thin objects, such as a piece of paper or a balloon, whose outer surfaces may be well approximated by an open or a closed surface. Even though thin-shell SfT methods can handle thicker objects such as a foam ball, they do not fully exploit the strong constraints induced by the object's non-empty interior.

Volumetric SfT reconstructs the object's interior deformation (which is not reconstructed by thin-shell SfT) and reconstructs the object's outer surface more accurately than thin-shell SfT due to the stronger deformation constraint it uses. The biggest challenge for volumetric SfT is to reconstruct the object's back surface and interior.

We solve volumetric SfT using the ARAP deformation model. It maximises local rigidity while penalising stretching, sheering and compression. It helps in preserving the object's interior structure and enables to reconstruct the object's full outer surface and interior.

We present volumetric SfT as an unconstrained non-linear least-squares optimisation problem which can be solved using standard numerical solvers such as Levenberg-Marquardt.

We propose two heuristic initialisation methods. Experimental results on synthetic and real data show that volumetric SfT improves accuracy to a large extent compared to state-of-theart thin-shell SfT methods.

Chapter outline. We present the mathematical modelling of volumetric SfT in section 6.2.

It describes the modelling of volumetric SfT model and the deformation. Section 6.3 defines volumetric SfT and proposes a solution. Section 6.4 discusses the experiments and section 6.5 concludes.

Mathematical Modelling

Geometric Model

Figure 6.1 shows a general diagram of volumetric SfT extending a thin-shell framework described in the previous chapter. We denote the 3D template as the volume V T ⊂ R 3 , the unknown deformed volume as V S ⊂ R 3 , and their respective outer surfaces as ∂V T and ∂V S .

We denote as S ⊂ ∂V S the deformed object's visible surface part, i.e., the part which is directly observed in the input image I ⊂ R 2 , and T ⊂ ∂V T the corresponding part in the template surface. We use a 2D surface parameterisation space F, called the flattened template. This allows us to represent the template's outer surface ∂V T by a known invertible embedding Δ ∈ C 2 (F, R 3 ). In practice, F and Δ may be computed from ∂V T by any flattening method; we use conformal flattening [START_REF] Sheffer | ABF++: Fast and Robust Angle Based Flattening[END_REF]. Using F, the unknown 6.2. MATHEMATICAL MODELLING deformed surface S may be represented by an embedding ϕ ∈ C 2 (F, R 3 ). The deformation between V T and V S is the unknown mapping ψ ∈ C 2 (V T , R 3 ). The task in volumetric SfT is not only to compute the volume V S of the deformed object, but to find a full volume deformation function ψ ∈ C 2 (V T , R 3 ), matching points between the object's template and deformed states. This is a challenging task, as most of V S is not directly observed in the image: assuming the object is opaque, the only visual information comes from the outer surface's visible part. The surface embedding ϕ may of course be directly recovered from the volume deformation ψ computed by volumetric SfT as ϕ = ψ • Δ. Depending on the formulation, thin-shell SfT computes either the surface embedding ϕ [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF] or a 3D surface deformation, which is a restriction of ψ to ∂V T [ Östlund et al., 2013]. The full volume deformation ψ cannot be directly recovered in either case. Our initialisation strategy for volumetric SfT involves inferring ψ from ϕ through two new solutions which we name volume interpolation.

A point in the 3D template is given by P and the corresponding point on the flat template is p. In a similar way, a point on the deformed object is given by Q. The corresponding point in the image is given by q. Finally, we define η ∈ C 2 (F, R 2 ) as the registration warp between F and the image. Estimating the warp directly gives two pieces of information. First, it identifies the subset G ⊂ F corresponding to the surface's visible part in the flattened template. Second, it establishes the reprojection constraint on ϕ and ψ as:

η = Π • ϕ = Π • ψ • Δ, (6.1)
where Π denotes perspective projection in coordinates normalised with respect to the camera's intrinsics Π

(Q) = 1 Q 3 (Q 1 Q 2 ) with Q = (Q 1 Q 2 Q 3 ) .

Deformation Model

Thin-shell isometry allows SfT to resolve the visible surface part uniquely [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF] and to extrapolate the non-visible surface part [ Östlund et al., 2013]. Applied to an object's interior volume, isometry yields the following differential constraint on the mapping ψ:

J ψ J ψ = I 3×3 . (6.2)
According to the Mazur-Ulam theorem [START_REF] Rassias | On the Mazur-Ulam Theorem and the Aleksandrov Problem for Unit Distance Preserving Mappings[END_REF], equation (6.2) constrains ψ to be a rigid transformation. So as to model deformations, one must relax equation (6.2).

One possibility is the so-called ARAP heuristic [START_REF] Sorkine | As-rigid-as-possible Surface Modeling[END_REF], which means searching for ψ such that J ψ J ψ -I 3×3 2 p (p denotes the type of norm) is minimised over

V T . We propose to combine ARAP with the reprojection constraint to preserve the object's local structure while driving its deformation to comply with the image constraints.

Volumetric Shape-from-Template

We present ARAP volumetric SfT, which finds the deformation ψ that transforms the volume V T into the unknown volume V S whose surface is partially observed in I.

Formulation and Non-Convex Solution

Problem formulation. Combining the reprojection constraint (6.1) with ARAP leads to the following variational problem:

min ψ ρ V T J ψ J ψ -I 2 p d V T ARAP penalty +(1 -ρ) G η -Π • ψ • Δ 2 2 d G Reprojection . (6.3)
The reprojection constraint is convex, but the ARAP penalty is not. Problem (6.3) is thus difficult to solve, as it involves integrals and equality constraints. Local analytical solutions, as the ones proposed in the previous chapter and in [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF] for thin-shell SfT,

are not applicable at non-visible points since they do not have a data term. This is because the reprojection constraint applies on the visible surface part S only, corresponding to the subset G of the flattened template.

Discretisation and optimisation. We evenly discretise the template volume V T with a set of 3D points P V T . We define the deformation functional d [ψ] :

d [ψ] = 1 |P V T | P∈P V T J ψ (P)J ψ (P) -I 3×3 2 p , ( 6.4)
where |P V T | represents the size of set P V T .

We write r [ψ] over a regular discretisation P G of G:

r [ψ] = 1 |P G | p∈P G η(p) -Π(ψ(Δ(p))) 2 2 . (6.5)
Finally, we optimise the following unconstrained non-linear least-squares problem:

ψ = arg min ψ ρ d [ψ] + (1 -ρ) r [ψ] 0< ρ < 1, (6.6)
where ρ is a weight that balances the ARAP penalty and the reprojection constraint.

In order to find a numerical solution to problem (6.6), we use a parametric representation of the solution ψ ∈ C 2 (V T × R n , R 3 ), where n is the dimension of the parameter space. Let L ∈ R n be the parameter vector and Q ∈ V T , we have ψ(Q, L) ∈ V S . We have multiple choices for ψ such as the popular linear basis expansion representations (the NURBS [START_REF] Piegl | On NURBS: A Survey[END_REF], the Thin-Plate Splines (TPS) [START_REF] Bookstein | Principal Warps: Thin-plate Splines and the Decomposition of Deformations[END_REF], the B-Spline [START_REF] Carl | A Practical Guide to Splines[END_REF],

tetrahedron mesh displacements, etc.). We use the TPS representation.

Problem (6.6) is then optimised using Levenberg-Marquardt. Iterative methods can be highly accurate but because problem (6.6) is non-convex due to the ARAP penalty, the iterations may converge to a non-global minimum. Therefore, it is important to provide an initial solution close to the global minimum.

Convex Initialisation

Our initialisation strategy finds an approximate solution ψ 0 to problem (6.3) in two main steps.

1) Isometric thin-shell SfT. We first compute the embedding ϕ that represents the visible surface S. We approximate the deformation from T to S by thin-shell isometry, giving the following problem reformulation:

Find ϕ s.t. ⎧ ⎨ ⎩ J ϕ J ϕ = J Δ J Δ η = Π • ψ • Δ on G, (6.7)
where J ϕ is a 3 × 2 matrix. Problem (6.7) has an analytical solution given in [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF].

2) Volume interpolation. We use ϕ to infer ψ representing the full volume deformation.

We propose two strategies.

i) Global Smoothness (GS).

Our first strategy is based on the assumption that the deformation of the volume is smooth. We can, therefore, formulate the problem as finding the smoothest volumetric deformation such that the deformation at the surface agrees with the solution from thin-shell SfT. We write the discretised transport error:

e [ψ] = 1 |P G | p∈P G ϕ(p) -ψ(Δ(p)) 2 2 .
(6.8)

Because ϕ was computed in step 1), this is a linear least-squares cost in ψ. We then compute ψ 0 as the solution of the following system: (6.9) where s =

ψ 0 = arg min ψ α e [ψ] + (1 -α) s [ψ] 0 < α < 1,
d 2 ψ dp 2 2
2 dp is a smoothing term called the bending energy and α is a weight balancing the transport error and smoothness. As in the non-convex solution, we use a TPS representation of ψ. The bending energy is then a quadratic function of the TPS parameters [START_REF] Bookstein | Principal Warps: Thin-plate Splines and the Decomposition of Deformations[END_REF], making problem (6.9) linear least-squares, thus convex and easily solvable. GS is a natural way of initialising ψ from ϕ, but as smoothness is the only constraint it uses to propagate the visible surface deformation, it may spoil the object's inner local structure by causing local shear, shrinking and extension. Our second volume interpolation method addresses this issue.

ii) Local Rigidity (LR). This method is based on the idea that from thin-shell SfT, we can compute a local rigid transform at every point on the visible surface to propagate shape through the object's volume, in an ARAP manner. The key idea is to initialise ψ on the surface's visible part from ϕ, and use local rigidity to iteratively 'complete' ψ. This is implemented by iteratively drawing local rigid transformations to locally extrapolate the deformation. Concretely, we first find correspondences to all points in P V T (which may be seen as a discretisation of ψ) and then fit a continuous parametric representation of ψ. We write the corresponding point of P ∈ P V T as Q(P).

We first use a Delaunay triangulation of the point set P V T to define a tetrahedral mesh.

A given tetrahedron has four vertices, which we denote as P n 1 , P n 2 , P n 3 and P n 4 . Drawing for i = 1, . . . , 3. At the early iterations, this means that these three vertices will have to be in the surface's visible part, and that Q n i = ϕ(Δ -1 (P n i )), i = 1, . . . , 3. From these three correspondences {P n i ↔ Q n i }, i = 1, . . . , 3, we fit a rigid transform Ω in the least-squares sense using [START_REF] Horn | Closed-Form Solution of Absolute Orientation Using Orthonormal Matrices[END_REF]. Completing ψ by local rigidity is then simply done by setting Q(P n 4 ) = Ω(P n 4 ). At each iteration, we cycle through all tetrahedra with three vertices lying in the completed domain of ψ. This obviously causes the fourth vertex of many tetrahedra to receive multiple predictions, as several tetrahedra may share it as their single unknown vertex. In order to approximate ARAP as best as possible, we keep the prediction for which Ω was fitted with the lowest error. We stop the iterations when all points in P V T have been given a correspondence. We finally define the discretised transport error: (6.10) and obtain ψ 0 as the solution of the following optimisation problem: (6.11) where α is a weight balancing the transport error and smoothness. Equation (6.11) is linear least-squares, thus convex and easily solved.

e [ψ] = 1 |P V T | P∈P V T Q(P) -ψ(P) 2 2 ,
ψ 0 = arg min ψ α e [ψ] + (1 -α) s [ψ] 0 < α < 1,

Experimental Results

We report experiments with synthetic data and three sets of real data with different geometries and materials: a woggle, a sponge and an arm. The refinement solution (6.3) is tested using the L1 and L2 norms, and is then called L1-refinement and L2-refinement respectively. The two initialisation solutions are called GS-initialisation and LR-initialisation.

We also compare with two isometric thin-shell SfT methods [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF][START_REF] Östlund | Monocular 3D Shape Recovery using Laplacian Meshes[END_REF], which were discussed as being representative of the state-of-the-art in the introduction. We use a constant weight ρ = 0.005 in the refinement problem (6.6) (for both L1-refinement and L2-refinement) and a constant weight α = 0.0001 in both equation (6.9)

for GS-initialisation and equation (6.11) for LR-initialisation. We noticed that the algorithms were not very sensitive to these values up to an order of magnitude.

Synthetic Datasets

We test our method for volumetric SfT in various conditions of noise, deformation and correspondences. We simulate a box of dimension 20 × 20 × 10cm 3 and deform it by bending each of its layers along a vertical rule with some varying bending angle. The higher the bending angle, the more important the box's deformation. If the bending angle is zero, the box is undeformed. We then create a virtual image of the box by projecting it using a perspective

Real Datasets

We evaluate the performance of the methods with three real-world objects captured across a range of deformed states.

Test Data and Ground Truth Acquisition

The three objects are a foam tube called a woggle (figure 6.4), a sponge (figure 6.5) and a human arm (figure 6.6). We construct the 3D template of each object using Photoscan, a dense rigid SfM package [START_REF] Photoscan | Agisoft Photoscan 1.0.4[END_REF]. To achieve this we photograph the objects in a rigid pose from a number of different viewpoints in order to capture the full 3D geometry (we use 47, 55 and 78 images for the three objects respectively). We apply a small amount of manual post-processing to fill holes and make the templates watertight. Then we physically apply forces to the real objects to obtain a set of deformed shapes, which we grouped into [START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF] and Thin-shell SfT 2 [ Östlund et al., 2013].

three levels: low, medium and high deformation. For each level we compute the ground truth shape by photographing again the deformed object from approximately 50 viewpoints and then running Photoscan. Because Photoscan provides the reconstructed object and the pose for each camera image, this provides us with the ground truth shape of the object's outer surface (including the back surface) in camera coordinates.

Similarly to the vast majority of previous SfT methods, ours takes as input point correspondences between the 3D template and the input image. These can be computed automatically using for instance SIFT combined with outlier detection [START_REF] Pilet | Real-Time Non-Rigid Surface Detection[END_REF][START_REF] Pizarro | Feature-based Deformable Surface Detection with Self-occlusion Reasoning[END_REF]. However, to keep the results independent of the matching algorithm, we define correspondences manually. For the three objects this gives between 50 to 350 correspondences per image. We click between 30 and 40 correspondences per image and create the others using TPS interpolation [START_REF] Bookstein | Principal Warps: Thin-plate Splines and the Decomposition of Deformations[END_REF].

Performance Metrics and Method Comparison

We calculate two types of 3D errors, E f and E b , both expressed in mm, for the visible and non-visible surface parts respectively, as the RMSE discrepancy between the true and reconstructed 3D points at the correspondences. For each of the three datasets, and each of the three deformation levels, the top of each figure shows the template, the input image and the ground truth shape. On each figure, the deformation goes through low, medium and high level from left to right. The rows then show the results of both initialisation methods and their use to initialisation both refinement methods, giving a total of six combinations.

The 3D errors E f , including the two thin-shell methods, are finally summarised in table 6.1.

We observe that LR-initialisation gives consistently better results than GS-initialisation, which is in accordance with our observations made on simulated data. The difference becomes very important for stronger deformations. This has a very small impact on the refinement results, for both refinement methods. We can observe small differences between the two refinement methods. However, refinement+LR converges faster than refinement+GS because LR-initialisation is closer to the correct solution. However, none of them is consistently better than the other, even if for the woggle L2-refinement is slightly more accurate, whereas for the sponge and arm L1-refinement is slightly more accurate. The 3D error E f for volumetric SfT for both refinement methods is consistently smaller than for thin-shell SfT. Depending on the dataset and the deformation level, it is between two and ten times smaller. This confirms our observations made on simulated data that, even if the surface's visible part is handled naturally by both volumetric and thin-shell SfT, the stronger volume deformation constraints used by the former allows it to obtain a much more accurate result. The results shown in figures 6.4, 6.5 and 6.6 show that the reconstructed object shape is visually close to the true shape. This means that volumetric SfT could allow a user to handle a physical object as a proxy interactor in applications such as virtual shape editing.

Quantitatively, the woggle, sponge and arm are 37 cm, 15 cm and 20 cm long, respectively.

The relative highest error over the whole reconstructed volume deformation, for the highest level of deformation, is thus smaller than 5%, 7% and 5% of the objects' size, respectively.

Our unoptimised MATLAB implementation on a standard desktop with 3.1GHz processor takes between 10 -25 seconds for the refinement to converge. The computation time for LR-initialisation is 3-5 seconds while for GS-initialisation it is 1-2 seconds.

In practice, volumetric SfT always converges with LR or GS-initialisation. But, in order to create conditions of failure, we initialised the refinement at equation (6.11) very far from the optimal solution. Figure 6.7 shows a failure case in the sponge dataset.

Conclusions

We presented volumetric SfT, which reconstructs an object from a single image and a 3D template, by using deformation constraints on the object's outer surface and interior. Pre-vious thin-shell SfT methods use constraints on the object's outer surface only. Volumetric

SfT is thus to be used with non-empty and non-flat objects. We proposed an implementation of volumetric SfT using ARAP. Our implementation uses non-convex refinement and has an initialisation procedure following an ARAP propagation of a surface deformation obtained by thin-shell SfT through the object's volume. Our method has significantly more accurate results than state-of-the-art isometric thin-shell SfT, reducing the error of an order of magnitude in some cases. Experiments with synthetic and real data show that our method has a typical maximum relative error of 5% in reconstructing the deformation of an entire object, including its back and interior for which no visual data is available. ARAP volumetric SfT opens the way to doing Human-Computer Interaction using a proxy object such as a cushion and a simple monocular webcam. Volumetric SfT may also be instantiated with other deformation models, such as biomechanical models.

Chapter 7

Conclusions

In this thesis, we presented solutions to thin-shell deformable 3D reconstruction in both model-based (SfT) and model-free (NRSfM) cases. We also proposed an SfT solution for reconstructing volumetric objects. We conclude the deformable 3D reconstruction for thinshell and volumetric objects and also discuss the future prospects.

Thin-Shell Deformable 3D Reconstruction

We proposed a unified framework for modelling NRSfM and SfT using differential geometry and Cartan's theory of connections. This framework is general and easy to use for the physics-based modelling of various kinds of deformations. We also proposed a deformation model (skewless deformation) and showed how to perform deformable 3D reconstruction with the skewless deformation prior. We presented a solution to SfT which does not require an explicit modelling of deformation. We also showed that this framework is coherent with existing SfT solutions [START_REF] Bartoli | Shape-from-Template[END_REF]. It is also coherent with our solution to NRSfM (proposed in chapter 4) for objects undergoing isometric deformations, which was developed using Riemannian geometry. This method, in fact, served as an inspiration to look beyond isometry in deformable 3D reconstruction.

Our methods are a significant improvement on the state-of-the-art in terms of accuracy, simplicity, computational complexity and applicability to wider situations. These methods are applicable to both wide and short-baseline images, handle a large number of images, yield a very accurate reconstruction with very few images and deal with missing data and occlusions implicitly.

These methods are local and can be solved analytically. Therefore, one can conclude that local and analytical methods are very quick and accurate. They are capable of playing a very important role in achieving real-time solutions to NRSfM and SfT without using high-performance computing devices.

This thesis is a new step towards local and analytical construction of NRSfM and SfT solutions. However there is still a lot to be explored about this local framework and its

Appendix B

Christoffel Symbols

In the previous section we saw that the metric tensor in a Cartesian coordinate system (equation (A.1)) is written as an identity matrix, but this may not be true for the metric tensors in other coordinate systems (equation (A.2)). In such coordinate systems, since the metric tensor keeps on changing with the coordinates, we can define its change using CS.

Therefore, CS is a set of numbers which are the components of vectors that represent the change in metric tensor. They are defined as the CS of first kind (Γ cab ) and the CS of second kind (Γ d ab ) related by the expression Γ cab = g cd Γ d ab . Therefore it is very easy to recover the CS of one kind given the other ones. In our work, we found that the expressions of the CS of second kind were simpler and therefore, we use only these. In the Cartesian coordinate system, the metric tensor is the identity and therefore, all the CS of second kind are zero.

However, in the spherical coordinate system, the metric tensor is variable and the CS of second kind are:

Γ r = ⎛ ⎜ ⎝ 0 0 0 0 -r 0 0 0 -r sin 2 θ ⎞ ⎟ ⎠ Γ θ = ⎛ ⎜ ⎝ 0 r -1 0 r -1 0 0 0 0 -sin θ cos θ ⎞ ⎟ ⎠ Γ φ = ⎛ ⎜ ⎝ 0 0 r -1 0 0 cotθ r -1 cot θ 0 ⎞ ⎟ ⎠ . (B.1)
They are expressed in terms of the metric tensor and its first-order derivatives. Therefore, we can define them at various surfaces and use them in our framework, just like the metric tensor. The CS are given by: Γ p mn = 1 2 g pl (g lm,n + g ln,mg mn,l ) , (B.2)

where g lm,n = ∂ n g lm and g pl = (g pl ) -1 . We write the derivatives of the metric tensor for the spherical coordinate system as:

  La reconstruction 3D d'objets à partir de plusieurs images est un objectif important de la vision par ordinateur. Elle a été largement étudiée pour les objets rigides et non rigides (ou déformables). Le Structure-from-Motion (SfM) est un algorithme qui effectue la reconstruction 3D d'objets rigides en utilisant le mouvement visuel entre plusieurs images obtenues à l'aide d'une caméra en mouvement. Le SfM est une solution très précise et stable. La reconstruction 3D déformable a été largement étudiée pour les images monoculaires (obtenues à partir d'une seule caméra) mais reste un problème ouvert. Les méthodes actuelles exploitent des indices visuels tels que le mouvement visuel inter-image et l'ombrage afin de construire un algorithme de reconstruction. Cette thèse se concentre sur l'utilisation du mouvement visuel inter-image pour résoudre ce problème. Deux types de scénarios existent dans la littérature: 1) le Non-Rigid Structure-from-Motion (NRSfM) et 2) le Shape-from-Template (SfT). L'objectif du NRSfM est de reconstruire plusieurs formes d'un objet déformable tel qu'il apparaît dans plusieurs images, alors que le SfT (également appelé reconstruction à partir d'un modèle de référence) utilise une seule image d'un objet déformé et son modèle 3D de référence (une forme 3D texturée de l'objet dans une configuration) pour estimer la forme déformée de l'objet. Nous proposons une méthode de NRSfM pour reconstruire les surfaces déformables soumises à des déformations isométriques (les objets ne s'étirent pas ou ne se contractent pas sous une déformation isométrique) en utilisant la géométrie riemannienne. Cela permet d'exprimer le NRSfM en termes d'équations aux dérivées partielles et de le résoudre algébriquement. Nous montrons que le problème a une complexité linéaire et que l'algorithme de reconstruction proposé a un coût de calcul très bas comparé aux méthodes existantes de NRSfM. Ce travail nous a motivé à utiliser la géométrie différentielle et la théorie des connexions de Cartan pour modéliser le NRSfM, ce qui nous a permis d'étendre la solution à des déformations autres que l'isométrie. En fait, cela a conduit à un cadre théorique unifié pour modéliser et résoudre le NRSfM et le SfT pour différents types de déformations. Ce cadre permet également d'avoir une solution au SfT qui ne nécessite pas de modélisation explicite de la déformation. Un point important est que la plupart des méthodes de NRSfM et de SfT reconstruisent la surface de l'objet (hypothèse coque mince). La reconstruction de l'ensemble d'un volume (la surface et l'intérieur d'un objet) n'avait pas encore été explorée. Nous proposons la première méthode de SfT qui reconstruit le volume complet d'un objet déformable.

Figure 1 . 3 :

 13 Figure 1.3: Comparison of Iso-NRSfM (with and without IP) with [Chhatkuli et al., 2016a].

4 )Figure 1 . 4 :

 414 Figure 1.4:A broad overview of the problem. Moving frames and connections are the generalisation of the concepts of metric tensors and CS from Riemannian geometry. We use them to express differential constraints in terms of PDE. The manipulation of these constraints with or without the IL assumption leads to reconstruction equations that are also PDE. IL is not necessary to find these equations, however we use it to simplify the problem. These reconstruction equations may or may not have an algebraic solution. Here, an algebraic solution implies that the equations can be solved locally at each point. In this thesis, we solve the equations with possible algebraic (or local) solution.

Figure 1 . 5 :

 15 Figure 1.5: Volumetric SfT versus thin-shell SfT. Existing methods are thin-shell SfT. They use deformation constraints on the object's surface. For instance,[START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF] uses isometric constraints on the object's visible (front) surface and reconstructs the object partially, while[ Östlund et al., 2012] uses isometric constraints on the object's whole closed outer surface and reconstructs it entirely. Volumetric SfT uses deformation constraints on the object's surface and interior. This greatly improves reconstruction accuracy and facilitates reconstruction of the object's interior. In this example, the thin-shell SfT methods[START_REF] Bartoli | On Template-based Reconstruction from a Single View: Analytical Solutions and Proofs of Well-posedness for Developable, Isometric and Conformal surfaces[END_REF][START_REF] Östlund | Laplacian Meshes for Monocular 3D Shape Recovery[END_REF] reach a 3D error of 20 mm and 13 mm respectively on the visible surface, while the proposed volumetric SfT method reaches a 3D error of 7 mm. It reconstructs the non-visible (back) surface, for which no visual data is available, with a 3D error of 17 mm.

Figure 1 . 6 : 4 )

 164 Figure 1.6: As opposed to thin-shell SfT, volumetric SfT reconstructs the object's interior deformation. In this example using the data from figure 1.5, a virtual cylinder is placed inside the woggle's template. It is then deformed using the deformation reconstructed by volumetric SfT to aid visualization of the object's reconstructed interior deformation. The second deformation is the one shown in figure 1.5.

Figure 5 .

 5 Figure 5.1 shows two surfaces S 1 and S 2 related by a deformation. The isometric and inex-

Figure 2 .

 2 Figure 2.1: A surface S 1 transforms to surface S 2 due to a deformation. The points (P 1 , P 2 ) on S 1 transform to (Q 1 , Q 2 ) on S 2 .

Figure 2 . 2 :

 22 Figure 2.2:A surface S 1 transforms to surface S 2 due to an isometric deformation. The points (P 1 , P 2 ) on S 1 transform to (Q 1 , Q 2 ) on S 2 . The sightlines of the two points (Q 1 , Q 2 ) from the camera C pass through (q 1 , q 2 ) on the image plane.

Figure 3 . 1 :

 31 Figure 3.1: Illustration of IL. Two smooth curves are related by a mapping ψ. According to IL, there exists a linear map ψ L that relates P and Q and agrees with ψ at zeroth and first-order.

Figure 3 . 2 :

 32 Figure 3.2: An image embedding φ that relates the 3D surface M with its image I.

Figure 4 . 1 :

 41 Figure 4.1: The proposed model of NRSfM, where each surface M i is a Riemannian manifold defined by embedding the corresponding retinal plane I i .

Figure 4 . 2 :

 42 Figure 4.2: Simplified notation for two images.

  .6) Note that even though CS are expressed with tensor notation, they are not tensors and thus equation (4.6) does not correspond to the way tensors change coordinates. The CS of the image embedding, defined via the inverse-depth in equation (3.4), has a special structure given in Theorem 1.

Theorem 4 (

 4 2D Schwarzian Equations for Planes). Given that M i with i ∈ [1, N] are planes, the registration warps η ij with (i, j) ∈ [1, N] × [1, N] are point-wise solutions of the 2D Schwarzian equations.

Figure 4 . 3 :

 43 Figure 4.3: Some images of the rug, table mat, kinect paper and tshirt datasets. The five rightmost images of the table mat dataset are zoomed in to improve visibility.

Figure 4 . 4 :

 44 Figure 4.4: Synthetic data experiments. Average normal and depth errors with respect to number of views, noise and curvature. Best viewed in colour.

  kinect paper datasets are long sequences (159, 60 and 191 images with 350, 300 and 1500 point correspondences) and the t-shirt is a short dataset (10 images, 85 point correspondences). The long sequences are uniformly reduced by picking 10 images at regular intervals in the sequence. The results are shown in figure 4.5. The results for the tshirt dataset are averaged over 20 trials of randomly sampled images. The figure clearly

Figure 4 . 5 :

 45 Figure 4.5: Experiments on short sequences. The average normal and depth error for each experiment with number of views varying between 3-10 is shown. The views of the rug, table mat and kinect paper datasets are selected by uniform sampling the long sequences. The views of the tshirt dataset are selected by randomly sampling the dataset and the results are averaged over 20 trials. Best viewed in colour.

Figure 4 . 6 :

 46 Figure 4.6:Reconstruction error maps and renderings for the rug, table mat, kinect paper and tshirt datasets. We remind that mdhI reconstructs only the visible part of the surface. Therefore, the rendering and error map for the kinect paper dataset is broken for this method. Best viewed in colour.

Figure 4 . 7 :

 47 Figure 4.7: Experiments on long sequences. The average normal and depth error for each frame is shown. The experiment with inextI is only performed for the table mat dataset. Best viewed in colour.

Figure 4 . 8 :

 48 Figure 4.8: Images (10, 20, 30, 40, 50) of a partially stretched rubber like surface. The first image has the least deformation and the last one has the most.

Figure 4 . 9 :

 49 Figure 4.9: Experiment with an almost stationary object. The first five images of the table mat sequence are used. The reconstruction of iso is shown in red. The ground truth is indicated with black. E s represents the mean shape error (in degrees). E d represents the mean depth error (in mm).The performance of iso is almost the same as infP on these five images. Best viewed in colour.

Figure 5 . 1 :

 51 Figure 5.1: A moving frame on a surface M defined using a local parametrisation (x 1 , x 2 ).

Figure 5 . 2 :

 52 Figure 5.2:A moving frame on a surface M i defined using local parametrisations (x 1 , x 2 ) and (x 1 , x2 ) related by η.

  1 , x 2 ) respectively are related by the following relationship ē1 ē2 = e 1 e 2 J η ē3 = ē1 × ē2 = |J η |(e 1 × e 2 ) = |J η |e 3 ē1 ē2 ē3 = e 1 e 2 J η |J η |e 3 .(5.16) Differentiating the above relation and expressing the derivatives of frame bases according to equation η | + e 3 d|J η |.

Figure 5 . 3 :

 53 Figure 5.3: Two surfaces M 1 and M 2 related by ψ are parametrised using (x 1 , x 2 ).

Figure 5 . 4 :

 54 Figure 5.4: Classification of various types of smooth mappings.

Figure 5 . 5 :

 55 Figure 5.5: An example of skewless deformation. A surface grid undergoes anisotropic scaling in two orthogonal directions and then undergoes a conformal transformation. Therefore, only the angles between the basis are preserved.

Figure 5 . 6 :

 56 Figure 5.6: Modelling of model-based 3D reconstruction of deformable objects from a single view.

Figure 5 .

 5 Figure 5.6 shows the general modelling of SfT. Given a 3D model T (template) of the object, our goal is to find the surface M as observed in the image I. η is the image warp from I to the flattened 3D model T f . η is known and can be estimated in practice with dense image registration methods. ψ is the deformation function between T and M. For rigid objects ψ is a Euclidean transformation. We model the surface M and the model T using image embeddings φ 1 and φ 2 :

Figure 5 . 7 :

 57 Figure 5.7: Modelling N views of a deforming 3D surface.

Figure 5 . 8 :

 58 Figure 5.8: Images from the sock and balloon datasets. Some of the tracked points are shown. Best viewed in colour.

Figure 5 .

 5 10 shows our results. Our SfT (IsoS, ConS, SkewS, EqArS, NoDefS) and NRSfM (IsoConN, SkewN) methods show a very stable performance in noisy conditions. The performance of other methods is very stable as well.Our methods (both SfT and NRSfM) exploit the first and second-order derivatives of the warps. These derivatives (especially the second-order ones) may be largely affected by the noise. Therefore, the methods EqArS, NoDefS, IsoConN (which solve SfT/NRSfM in terms of the first and second-order derivatives of the warp only) show a larger increase in the errors compared to other methods (such as IsoFS and ConFS) that use the first-order derivatives of the warps only. LinModS, NoIsoS, ConS and IsoS are very stable in the presence of noise. The skewless methods SkewN and SkewS also show a good tolerance to

Figure 5 . 10 :

 510 Figure 5.10: Normal and depth errors on the cylinder dataset by varying the noise from 1 to 5 pixels. The mean of errors on the entire imageset of each dataset is reported. Best viewed in colour.

Figure 5 . 11 :Figure 5 . 12 :

 511512 Figure 5.11: Normal and depth errors on the cylinder dataset by varying the radius from 2 to 10. The surface with radius 2 is the most curved. The mean of errors on the entire imageset of each dataset is reported. Best viewed in colour.

5 . 5 . 2 . 4 Figure 5 . 14 :Figure 5 . 15 :Figure 5 . 16 :

 5524514515516 Figure 5.14: Error maps for two surfaces the balloon dataset. The depth error maps show the difference in the reconstruction and ground truth. Best viewed in colour.

Figure 6 . 1 :

 61 Figure 6.1: General diagram of volumetric SfT. The visible surface part is shown in blue.

Figure 6 . 2 :

 62 Figure 6.2: Volume interpolation using Local Rigidity.

Figure 6 . 5 :

 65 Figure 6.5: Results on the sponge. The green boxes show the best performing algorithm for each deformation level.

Figure 6 . 6 :

 66 Figure 6.6: Results on the arm. The green boxes show the best performing algorithm for each deformation level.

Figure 6 . 7 :

 67 Figure 6.7: Testing with the top image led to the local minimum shown in the bottom image.

Table 2 .

 2

1: Summary of statistics-based NRSfM methods

Table 2 . 2 :

 22 Summary of physics-based NRSfM methods

Table 4 . 1 :

 41 Performance of warps in noisy conditions. The average pixel error due to added noise w.r.t. the B-spline control points are shown. The warps reduce the noise only when fewer control points are used, but then significantly degrade their derivatives.

	Noise (in pixels)→ B-spline control points	1	2	3	4	5
	10	.53	.92 1.55 1.94 2.46
	20	.99 2.00 2.94 3.93 4.90
	30	1.00 2.06 3.02 4.07 5.14

  Table mat dataset. iso has the best performance on this dataset. infP and mdhI show a similar performance in this dataset, and they are closest to iso compared with other methods.

	1RUPDO (UURU LQ GHJUHHV	_	[H]^\ (UURU LQ PP
			1XPEHU RI YLHZV	1XPEHU RI YLHZV
				(a) Rug dataset
	1RUPDO (UURU LQ GHJUHHV	_	[H]^\ (UURU LQ PP
			1XPEHU RI YLHZV	1XPEHU RI YLHZV

diffH and plaH show a worse performance than mdhI, with plaH being a little bit better than diffH. inextI and pieceI are orthographic methods and therefore, they did not do well. Kinect paper dataset. mdhI has the best performance on this dataset. infP and iso are very close to mdhI. inextI and pieceI have the worst normal error as compared to other (b) Table

mat dataset

Table 4 . 2 :

 42 Summary of experiments on long sequences. The average normal (E n ) and depth error (E d ), are measured in degrees and mm respectively and shown over the entire sequences.

	Kinect paper

Table 4 . 3 :

 43 Mean shape error (in degrees) for the experiment with partially stretched surface. The error increases with the number of images as the deformation increases. iso performs better than infP. In the experiments that we performed, we observed that iso and infP show the best results amongst all compared methods. mdhI shows a good performance. Its results are comparable to ours for the tshirt and kinect paper datasets. This method is based on the MDH, it usually requires a lot of images with different viewpoints

	Summary of experiments on long sequences.	Table 4.2 summarises the results of
	the compared methods on the rug, table mat and kinect paper datasets. iso and infP give
	the best performance on rug and the table mat datasets while mdhI gives the best results

Table mat dataset. iso has the best performance with infP being very close to it. mdhI shows better performance than kerF, but both of them are worse than iso and infP. diffH and plaH need to compute homographies between image pairs, therefore, they grow nonlinearly with the number of views. For 60 images, the execution time goes upto 45 min for a single reconstruction. Therefore, we did not compare with them. pieceI breaks on this sequence, therefore we did not include it. The comparison with inextI is done only for this sequence as it is a relatively smaller sequence. One must also note that inextI, pieceI grow with the number of views and point correspondences, therefore, they are not very efficient with a large number of views. Kinect paper dataset. mdhI shows the best results on this dataset. iso and infP have similar performance but they are worse than mdhI. kerF has the worst performance; the errors are almost double of the rest of the methods. It is because this sequence has outliers; and therefore the performance of kerF is affected. One must note that iso, infP and kerF reconstruct the occluded part of the paper while mdhI only recovers the visible paper in each frame. on the kinect paper dataset. kerF gives decent results on the rug and table mat datasets but does not do well on the kinect paper dataset. However, its performance is always worse than iso, infP and mdhI. inextI was only compared on the table mat dataset; it gives the worst results compared to the other methods.

Summary of experiments.

to give good results. diffH and plaH are based on homography decomposition. They suffer from ambiguities in the normals. They disambiguate the normals assuming the smoothness of the surfaces which is not a strong assumption to make. These methods work well with wide baseline datasets. kerF is a method based on statistical modeling designed to work for video sequences. It needs a good estimation of the radius of the kernel in which the similarities between the two shapes are measured. pieceI and inextI are methods based on orthographic projection. They suffer from convex-concave flip ambiguities.

Table 4 . 4

 44 table mat sequence are used. The reconstruction of iso is shown in red. The ground truth is indicated with black. E s represents the mean shape error (in degrees). E d represents the mean depth error (in mm). The performance of iso is almost the same as infP on these five images. Best viewed in colour.

	infP iso mdhI kerF plaH diffH pieceI inextI
	10 11.2	51	8.9	14.1	65	86	180	210
	30 13.1 52.4	21.3	26.3	3090	2280	850	1299
	60 14.8 55.1	65.8	44.8	-	-	-	-

: Comparison of computation time (in seconds) for 10, 30 and 60 views. The best performing method is highlighted in bold. infP and iso show a much lower increase in computation time while plaH, diffH, pieceI and inextI show a drastic increase. They could not be evaluated for 60 views.

  shell objects undergoing isometric deformations. It uses the differential quantities of metric tensor and CS to model NRSfM. It is a significant improvement on the state-of-the-art. It is a local solution which has limited variables for any number of images, which makes it computationally very cheap. However, it gives very good results with very few images. It handles missing data and occlusions implicitly and works with both short and wide-baseline images. It shows that physics-based modelling in NRSfM leads to better solutions than existing ones.In this chapter, we extend NRSfM to deformations other than isometry. Our goal is to propose local solutions which maintain the strengths of Iso-NRSfM. We generalise the concepts of metric tensor and CS from Riemannian geometry (defined for isometry and conformity only) to moving frames and connections in differential geometry. Differential geometry is a very good tool for physics-based modelling as in pure mathematics, it is well-established that it is the comprehensive study of surfaces. Cartan's theory of connections not only generalises the concept of CS on surfaces but also gives the laws that combine moving frames

and connections in order to completely describe the properties of surfaces. It then leads to interesting results. With differential geometry and Cartan's connections, we propose a unified framework for modelling NRSfM and SfT. This framework is local and coherent with other

Table 5 . 1 :

 51 Summary of model-based 3D reconstruction of deformable thin-shell objects. For each deformation we obtain two constraints in ( k1 , k2 ). The degree of these constraints is shown in the table and discuss whether depths and normals for deformation are recoverable.

	Deformation	Assumption Degree of constraints Scale Normals
	Isometric	IP	1	Y e s	Y e s
	Conformal	IP	1	Y e s	Y e s
	Skewless	IP	3	No	Yes
	Equiareal	IP	1	Y e s	Y e s
	Smooth	IL	1	No	Yes
	Isometric [Bartoli et al., 2015]	-	2	Y e s	Y e s
	Conformal [Bartoli et al., 2015]	-	2	No	Yes

, ConS, EqArS, NoDefS, IsoFS and

  The Tissue dataset is a piece of elastic tissue with 100 points matched to a undeformed model of the tissue. It has only one image. This dataset was introduced in[START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF]. The Balloon dataset consists of 20 surfaces (3000 point tracks) of a balloon deformed conformally. This dataset Normal and depth errors on all datasets. The errors shown are evaluated by averaging the errors on the entire imageset. The first four methods shown are NRSFM methods and the rest are SfT methods. In general, the performance of SfT methods is better than NRSfM methods on these datasets. The methods ending with N and S are NRSfM and SfT methods respectively. NoIsoS represents the best performing SfT method in[START_REF] Bartoli | A Perspective on Non-Isometric Shape-from-Template[END_REF]. Best viewed in colour.reported this as a failure scenario for MDHN and therefore, it did not perform well for this dataset. The model-based (SfT) methods, however, showed a much better performance as compared to NRSfM methods on this dataset. IsoS

	IsoConN SkewN IsoConN SkewN Figure 5.9: ConFS are all analytical solutions to SfT. Our methods (IsoS, ConS, EqArS, NoDefS) MDHN KerN IsoS ConS SkewS EqArS NoDefS IsoFS ConFS LinModS NoIsoS Method 5 10 15 20 25 30 35 40 Normal Error (in degrees) Synthetic Paper Balloon Sock Rubber Tissue MDHN KerN IsoS ConS SkewS EqArS NoDefS IsoFS ConFS LinModS NoIsoS Method 5 10 15 20 25 30 35 40 Depth Error (in mm) Synthetic Paper Balloon Sock Rubber Tissue

  Table 6.1: 3D visible surface error E f for the datasets shown in figures 6.4, 6.5 and 6.6. Thin-shell SfT 1 [

	Error Analysis		Tube Data		Sponge Data		Arm Data
	(in mm)	Low Med. High Low Med. High Low Med. High
	Thin-shell SfT 1	11.8 16.0	19.8 8.0	15.3	31.8 18.5 20.1	33.4
	Thin-shell SfT 2	10.4 11.4	12.7 6.0	25.5	35.4 15.5 18.9	35.7
	GS-initialisation	11.1 13.5	17.3 6.9	15.1	31.6 15	18.3	31.6
	LR-initialisation	10.8 11.3	13.8 6.9	13.0	28.5 13.5 14.2	25.6
	L1-refinement + GS 2.1	6.7	11.0 5.4	5.2	7.0	3.4	4.1	5.2
	L1-refinement + LR 1.8	6.4	9.0	5.3	5.1	6.9	2.9	4.1	5.1
	L2-refinement + GS 2.2	5.6	7.1	3.4	4.9	8.5	3.0	4.5	7.3
	L2-refinement + LR 2.1	5.6	7.1	3.6	4.5	8.3	2.7	4.2	6.8

Acknowledgements

List of Abbreviations

ARAP As-Rigid-As-Possible

CS

Chapter 4

Non-Rigid Structure-from-Motion with Riemannian Geometry

Summary

In this chapter, we propose Isometric Non-Rigid Structure-from-Motion (Iso-NRSfM) using a theoretical framework based on the Riemmanian manifold to represent the unknown 3D surfaces as embeddings of the camera's retinal plane. This allows us to use the manifold's metric tensor and CS fields. These are expressed in terms of the inverse-depth of the 3D surfaces and its first and second-order derivatives. These are the unknowns for Iso-NRSfM. We prove that the metric tensor and the CS are related across images by simple rules depending only on the warps. We show that Iso-NRSfM is solvable from local image warps. It proves that NRSfM can be solved locally from a PDE formulation. We propose two solutions to Iso-NRSfM: with and without the assumption of IP. This chapter is based on our published work [START_REF] Parashar | Isometric Non-Rigid Shape-From-Motion in Linear Time[END_REF].

Inputs: Warps η j1 , j ∈ [2, N].

1) Find point correspondences. This step is the same as step 1) from the previous algorithm.

2) Initialise (k 1 , k 2 ) using the solution under IP. Run step 2) from the previous algorithm.

3) Find the second-order derivatives (k 3 ,k 4 ,k 5 ). Evaluate the linear system S 1 (k 3 , k 4 , k 5 )

and solve using LLS. This gives (k 3 , k 4 , k 5 ). Find ( k3 , k4 , k5 ) using equation (4.47). 4) Find (k 1 , k 2 ). Evaluate the sum-of-squares polynomials of the system Q 1 (k 1 , k 2 ) and find (k 1 , k 2 ) by minimising it using [START_REF] Henrion | Gloptipoly: Global optimization over polynomials with matlab and sedumi[END_REF]. Find ( k1 , k2 ) in terms of (k 1 , k 2 ) and the warps η j1 , j ∈ [2, N], using equation (4.6). 5) Repeat steps 3) and 4) until the solution to (k 1 , k 2 ) converges. The maximum number of iterations is set to 5. 6) Find normals and 3D points. Run step 3) from the previous algorithm.

Outputs: Points and normals on 3D surfaces.

Complexity Analysis

We discuss the complexity of the algorithm under the IP assumption and in the general solution. For both solutions, we assume that the warps are provided. We calculate the warps using schwarps [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF], that impose the Schwarzian equations ( 4.29) as a soft constraint. However we would like to point out that we do not require warps between all possible pairs of images in the sequence. We use a reference view and thus require the computation of N -1 warps only, not N 2 .

Solution with Infinitesimal Planarity

The solution to Iso-NRSfM under IP solves only one sextic polynomial for N images. This polynomial is formed by computing the sum-of-squares of 2(N -1) cubic polynomials (4.45).

Forming this sextic polynomial has a linear complexity but solving it is independent of N .

General Solution

The solution to the general case solves for the first and the second-order derivatives of α i (x) in parts. The approach for solving for the first-order derivatives of α i (x) is similar to the solution under IP assumption. It also solves one sextic polynomial and therefore, the solution is independent of N . For the second-order derivatives of α i (x), we obtain 4(N -1) linear equations and they are solved using LLS. Therefore, there is a linear complexity in forming these equations and solving them as well.

Experimental Results

We tested Iso-NRSfM on synthetic and real datasets. Figure 4.3 shows some images from real datasets on which the methods were evaluated. For quantitative comparison, we measured the normal error (mean difference between computed and ground truth normals in degrees)

Chapter 5

A Modelling Framework for Deformable 3D Reconstruction Under the assumption of IP, the simplified constraint is given in equation (5.47). Using equation (5.53) in this constraint leads to expressions independent of ( k1 , k2 ) and (k 1 , k 2 ).

Therefore, it is not solvable. Differentiating equation (5.47) to find more constraints under the assumption of IP does not make sense as it contradicts the assumption of IP. Hence, we

show that the PDE (5.45) for equiareal NRSfM does not possess a local solution.

Skewless Mappings

Given that ψ is a skewless mapping, the reconstruction equations are given in equation (5.51) in terms of (k 1 , k 2 ) and ( k1 , k2 ). Using equation (5.53) in equation ( 5.51), we obtain two septic equations in terms of (k 1 , k 2 ) only. These equations can be solved by minimising the sum-of-squares.

Smooth Linear Mappings

Given that ψ is a smooth infinitesimal mapping, there can only be two constraints obtained from equation (5.53) in terms of (k 1 , k 2 ) and ( k1 , k2 ) respectively. Therefore, a solution for (k 1 , k 2 ) cannot be obtained. This is an important result as it becomes clear that NRSfM cannot be solved based only on deformation smoothness.

Reconstruction Algorithm

We wrote reconstruction equations for various kinds of smooth deformations in equations (5.44), (5.47), (5.51) and (5.53). All of these equations are expressed in terms of the unknowns (k 1 , k 2 ). We present the following algorithm to solve these equations.

The index 1 corresponds to the first image in the sequence.

It can be chosen randomly.

was recorded using Kinect2.0 and the point tracks were obtained using [START_REF] Sundaram | Dense Point Trajectories by GPU-accelerated Large Displacement Optical Flow[END_REF]. The Sock dataset consists of 20 surfaces (3500 point tracks) of a sock that undergoes elastic deformations on one direction only. This dataset was recorded using Kinect2.0 and the point tracks were obtained using [START_REF] Sundaram | Dense Point Trajectories by GPU-accelerated Large Displacement Optical Flow[END_REF]. A few images of the Balloon and Sock datasets are shown in figure 5.8.

For quantitative comparison, we measured the normal error (mean difference between computed and ground-truth normals in degrees) and the depth error (mean difference between computed and ground-truth 3D coordinates in mm).

In the model-free scenario (NRSfM), we proposed solutions using isometric, conformal and skewless deformation constraints. Our solution for conformal and isometric deformations is obtained by solving same equations. Therefore, our method IsoConN represents a solution to NRSfM assuming that the object undergoes isometric or conformal deformation and SkewN represents the solution to NRSfM assuming skewless deformations. We compare our results with MDHN (maximum depth heuristics based NRSfM method proposed in [Chhatkuli et al., 2016a]) and KerN (low-dimensional shape-basis based NRSfM method proposed in [START_REF] Gotardo | Kernel Non-Rigid Structure from Motion[END_REF]).

In the model-based scenario (SfT), we proposed solutions using isometric, (IsoS), conformal (ConS), skewless (SkewS) and equiareal (EqArS) deformation constraints. We also proposed a solution (NoDefS) which does not need deformation to be modelled explicitly.

It assumes IL. We compare our results with isometric (IsoFS) and conformal (ConFS)

SfT proposed in [START_REF] Bartoli | Shape-from-Template[END_REF] and LinModS (also an isometry based SfT proposed in [START_REF] Salzmann | Linear Local Models for Monocular Reconstruction of Deformable Surfaces[END_REF]). We also compare our results with NoIsoS (a smoothness based SfT proposed in [ Özgür and Bartoli, 2016]). [ Özgür and Bartoli, 2016] proposed five solutions, we report only the best working solution for this method as NoIsoS. We now discuss the results of these methods on each dataset separately.

Synthetic Datasets

We evaluated the performance of the compared methods on two synthetic datasets: Cylinder and Rubber.

Cylinder Dataset

The cylinder dataset consists of images of a cylindrical surface deforming isometrically. We added a random noise with a Gaussian distribution of 1 pixel of standard deviation to the images of the dataset. The mean normal and depth error of each compared method is evaluated by averaging the result of the experiment over 20 trials. For evaluating SfT methods, the results are shown in figure 5.9. The first four methods (IsoConN, SkewN, MDHN and KerN) are NRSfM methods. IsoConN shows the best performance amongst the NRSfM methods in this dataset. The images of this dataset do not come from a video sequence, therefore KerN did not do well for this dataset. In this dataset, the object and the camera do not move while deforming. This makes the problem ill-posed. [Chhatkuli et al., 2016a] make an approximation to the solution of NoDefS (SfT under IL assumption) by disregarding the second-order derivatives. The new solution is called NoDefS0. Figure 5.11 shows the results of all methods with respect to the varying curvature of the surface. The curvature is the inverse of the radius. In this experiment we show reconstruction for 10 surfaces of the cylinder with the radius varying from 2 to 10. In general the reconstruction errors increase with bending. However, IsoConN, SkewN and NoDefS do not show a sharp increase in errors due to bending. NoDefS0 is affected by bending (especially with strong bending), however it still shows decent reconstruction for highly bent surfaces.

The interesting thing about NoDefS is the simplicity of the solution. The reconstruction algorithm is only two lines of code given that (k 1 , k 2 ) in equation ( 5.53) are obtained from the template. This means that the computational cost of this method is very low as it involves simple addition and multiplications only. Even by ignoring second-order derivatives, NoDefS0 gives decent results.

Rubber Dataset

We evaluated all compared SfT and NRSfM methods on the first 20 images of this dataset.

The rubber is partially stretched from its longest side in a sequential order. Figure 5.9

summarises the performance of different methods on this dataset. The reconstruction of surfaces 10 and 20 by the compared methods are shown in figure 5.12. Amongst NRSfM methods, IsoConN shows the best performance. The rest of the NRSfM methods do not perform well. Amongst SfT methods, LinModS shows the best performance with EqArS and NoDefS being close (in terms of depth). Figure 5.12 shows that the curvature of the surface 10 is best captured by EqArS with NoDefS whereas surface 20 is decently reconstructed by LinModS only. For low deformations of this dataset, EqArS with NoDefS show better results than LinModS but they cannot cope up with the higher deformations of this dataset.

Real Datasets

Now we discuss the results of the compared methods on real datasets.

Paper Dataset

This dataset consists of 191 images. We picked 20 images by uniformly sampling the dataset.

This makes the dataset more closer to wide-baseline and therefore, KerN does not have good results. Figure 5.9 summarises the performance of the compared methods on this dataset.

Amongst NRSfM methods, MDHN shows the best performance, with IsoConN being very close to it. SkewN also shows good results on this dataset. Our SfT methods show a very good performance on this dataset. NoIsoS also leads to decent reconstruction in this dataset.

However, LinModS shows the best performance. The reconstructions of two surfaces of this dataset are shown in figure 5.13. IsoFS and ConFS do not show a good performance on the selected images of this dataset. We recall that in order to compare them with our methods, Chapter 6

Volumetric Shape-from-Template This chapter is based on our published work [START_REF] Parashar | As-rigid-as-possible Volumetric Shapefrom-Template[END_REF]. The three graphs on the left column of figure 6.3 show the 3D volume error in mm, computed as the Root Mean Square Error (RMSE) over a dense grid of points sampled over the object's outer surface and interior. The results on these graphs thus only concern the proposed volumetric SfT methods. We observe that the refinement methods all do significantly better than all initialisation methods. LR-initialisation does consistently and substantially better than GS-initialisation. This is explained by the fact that LR-initialisation follows the ARAP methodology for local propagation, while GS-initialisation simply uses smoothness, which is The same observations which we made for the refinement methods on the 3D volume error can be made, and the general trends also apply to the two tested thin-shell SfT methods. Importantly, we observe that volumetric SfT does consistently and in several case substantially better than thin-shell SfT, even if the measured error concerns only the visible surface part, CHAPTER 7. CONCLUSIONS applicability to real-life 3D reconstruction scenarios. Now we discuss future prospects: 1) Image embedding. We have proposed an embedding to describe the surface. This embedding is very efficient but one needs to explore more embeddings in order to find an efficient way to model the surfaces.

2) Warp issue. We use warps in all the solutions we proposed. Warps can be a source of error (especially when higher order derivatives are required). We refine the warps according to [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF]. However, they are the bottleneck of our methods. We suspect that higher accuracy in warps can be achieved by using learning methods such as Convolutional Neural Networks.

3) Warp correction. Our methods rely on the second-order derivatives of the warps which are usually a big source of error. Like [START_REF] Pizarro | Schwarps: Locally Projective Image Warps Based on 2D Schwarzian Derivatives[END_REF] presented a warp correction method based on Schwarzian equations, other methods are also possible in the model-based scenario. 4) Combining visual cues. So far, our framework only exploits inter-image motion. Some of the deformable reconstruction methods like [START_REF] Wang | Template-Free 3D Reconstruction of Poorly-Textured Nonrigid Surfaces[END_REF] use other visual cues, such as shading and are capable of reconstructing challenging objects. It would be interesting to find out if other cues, like contours and shading can also be used in our framework. 5) Non-smooth objects. Since our methods are local, they need the surfaces to be locally smooth only. Our methods could be extended to non-smooth objects and a reconstruction similar to [START_REF] Yu | Direct, Dense, and Deformable: Template-Based Non-rigid 3D Reconstruction from RGB Video[END_REF] be achieved. Obtaining warps for non-smooth objects should be explored. 6) Real-time NRSfM and SfT solutions. Since the local methods are computationally very efficient, making real-time applications with these methods should be explored.

Volumetric Deformable 3D Reconstruction

The challenge for volumetric reconstruction is to reconstruct the parts of the objects that are not seen in the camera: the interior and the back surface. It is, therefore, not possible to reconstruct volumes in a model-free case. We presented the first SfT solution to reconstruct volumes. It involved a non-convex optimisation which could be initialised by two of our proposed methods. We also showed that volumetric SfT yields a better reconstruction than the thin-shell SfT methods. This is because volumetric constraints are much stronger than the constraints on the surfaces.

A complete volumetric reconstruction should yield a more accurate registration of objects that lie in the interior, such as tumours inside the body organs. However most of the body organs do not deform near-isometrically, therefore it is important to model volumetric SfT with a deformation model less strict than ARAP. One such deformation model is conformity.

In future, we intend to explore volumetric SfT with incompressibility (volume-preserving)

deformations. This constraint is widely used in biomechanical modelling. [START_REF] Delingette | Toward Realistic Soft-tissue Modeling in Medical Simulation[END_REF] discussed the aptness of this constraint for modelling soft body organs.

Appendices

Appendix A

The Metric Tensor

In order to describe a physical surface, one must define a coordinate system where measurements like lengths, angles and areas can be defined. The metric tensor [START_REF] Lee | Riemannian Manifolds : An Introduction to Curvature[END_REF] is a function which is defined on a physical surface to obtain these measurements. We use a simple example to develop a better understanding of the metric tensor. For this purpose, we consider an infinitesimal vector v in the Euclidean 3-space.

If we use a Cartesian coordinate system, we can define the length ds of v using Pythagoras' law for distances as:

where dx, dy and dz represent the components of v in x, y and z coordinates respectively and the identity matrix in equation (A.1) is the metric tensor. The metric tensor is denoted as g. The identity metric tensor implies that the distances remain constant as one moves along the coordinate frame. Now if we measure the length of the same infinitesimal vector v in a spherical coordinate system (r, θ, φ), we have:

Here, the metric tensor g is not the identity and changes at each point. This is necessary in order to measure the same distances at various locations in the coordinate frame. shows two points A and B represented on a spherical coordinate system. Moving these points to A and B , in the direction of the r-coordinate, will also change the distance between them. However, this does not happen in a Cartesian coordinate system. On further elaboration, we see that the change of coordinates from Cartesian to spherical (described by the transformation function f ), results in the following expression for the metric tensor:

In this case g depends only on J f as the metric tensor in the Cartesian coordinate system is the identity.

Moving on to the application of this theory in our work, we have a surface M i (see Fig. 4.2) undergoing an isometric deformation which leads to M j . A point z ∈ M i becomes w ∈ M j . Since ψ ij is an isometric deformation, the infinitesimal distances around w will be the same as that of z, as both of them represent the same point on different isometric surfaces. This is analogous to equations (A.1) and (A. Appendix C

Differential K-forms

A differential K-form represents a smooth section on the infinitesimal tangent space of the manifold. For example, a 0-form describes a point on the manifold, a 1-form describes a line element, a 2-form describes an area element, a 3-form describes a volume element, and so on.

In differential geometry, differential forms [START_REF] Cartan | Differential Forms[END_REF][START_REF] O'neill | Elementary Differential Geometry[END_REF] are used to perform the multivariate calculus independently of the coordinates. A scalar function f , parametrised with m variables (x 1 , x 2 , . . . x m ) such that a point f = f (x 1 , x 2 , . . . x m ) is a 0-form. In this case, the exterior derivative of f is the same as the total derivative of f . Now, the differential 1-form expressing the exterior derivative of f is given by df = m t=1 ∂f ∂x t dx t , (C.1) where dx t are the 1-forms and ∂f ∂x t represents a linear function on the tangent space of the function f in R n . Differential forms are defined locally, in terms of the local coordinates. Hence they are easily transferable from one coordinate system to another and therefore, very useful for defining local properties of the surfaces.