CVPR Edmond Boyer

Marc Pollefeys

Federica Bogo

Bugra

François Le Clerc

Martin Hachet Now

Vagia Tsiminaki

Jean-Sébastien Franco

Benjamin Allain

Edmond Boyer Cvpr

Jinlong Yang

Franck Hétroy-Wheeler

Stefanie Wuh- Rer

Adnane Boukhayma

Chun-Hao Huang

Nassir Navab

Slo- Bodan Ilic

Vincent Leroy

Tony Tung

Multi-view object segmentation with calibrated camera networks

Keywords: 4.2 Student Supervision acquisition. Rigid structure detection. Collaboration with OMG/Vicon, BBC, HHI, Artefacto, University

la stéréo multi-vues et l'estimation monoculaire de forme monoculaire 3D. Deuxièmement, je présenterai l'apport d'a prioris faibles mais très généraux sur la rigidité des surfaces ou en volume, et l'utilisation d'a priori d'un espace des formes humaines, et comment les mettre à profit pour le suivi de surface et l'alignement de sujets humains en vêtements amples comme serrés. Troisièmement, je discuterai des modèles statistiques spatio-temporelles et des représentation d'apparence que nous avons proposés pour estimer et stocker des cartes de couleur pour les sujets acquis.

iii

ABSTRACT

Acquiring and representing 3D shapes in motion from one or multiple views, coined 4D modeling, has been a topic of interest to computer vision and computer graphics for several decades, for 3D content production, virtual reality, telepresence and human motion analysis applications. In this time frame, the topic has gone from theoretical research subject to partial industrialisation, including nowadays many acquisition studios from major world companies, and a successful Inria startup with 15 years of activity. To achieve this, a set of hard problems had to be addressed, with advances in what can be classified as three subfields of shape acquisition and representation, motion retrieval and modeling, and appearance and texture estimation. This document summarizes my journey and contributions toward these goals with the students I was privileged to work with, with a particular focus on PhDs defended in the last five years. First I will discuss our work with multi-view stereo and monocular shape estimation. Second, I will present how weak but general priors on surface or volumetric rigidity, and using the a priori of an underlying human shape space, can be leveraged for surface tracking and alignment of human subjects in tight or loose clothing. Third, I will discuss the space-time statistical and representational models of appearance we proposed to estimate and store color map information of acquired subjects.

RÉSUMÉ

Acquérir et représenter des formes 3D en mouvement à partir d'une ou plusieurs vues, aussi appellé modélisation 4D, a été un sujet d'intérêt pour la vision par ordinateur et l'infographie depuis plusieurs décennies, pour la production de contenu 3D, les applications liées à la réalité virtuelle, la téléprésence et à l'analyse du mouvement humain. Dans ce laps de temps, le sujet est passé de la recherche théorique à une industrialisation partielle, avec de nos jours de nombreux studios d'acquisition dans les entreprises majeures du domaine de l'informatique, et une startup Inria avec actuellement à son actif 15 ans d'activité. Pour atteindre ce palier majeur, un ensemble de problèmes difficiles a dû être résolu, avec des progrès dans ce qui peut être classé en trois champs d'investigations, l'acquisition et la représentation de forme, la modélisation et l'estimation de mouvement, et l'estimation de l'apparence et de la texture. Ce document résume mon parcours et mes contributions dans le sens ces objectifs avec le étudiants avec lesquels j'ai eu le privilège de travailler, avec un accent particulier sur les thèses soutenues ces cinq dernières années. J'aborderai tout d'abord notre travail avec CHAPITRE 1 Introduction

RESEARCH PROBLEM : 4D MODELING

The past decades have seen an ever increasing interest in automated 3D dynamic content creation, for various applications such as 3D content production, advertizing, entertainment, virtual reality, telepresence. It is also becoming a popular research topic in the learning era, where works are increasingly showing that 3D constraints and inference can be built on deep learning methods to increase result quality and scene understanding.

This trend has been fueled by the increased availability of multi-camera systems, such as our 68-camera capture plaftorm Kinovis [kin], sometimes comprised of dozens or hundreds of cameras, that can be used for performance capture (e.g. [SH07, dAST + 08, LDX10, CCS + 15, JLT + 15]). Such systems can readily produce video streams of the same subject from multiple viewpoints, allowing indirect and passive access to the 3D geometry, motion, and appearance of filmed subjects. This technology offers numerous promises with respect to rugged, but sparse previous generation motion capture techniques, where only a predefined set of points and no colorimetry was observed, and relies on instrumentation of the captured subjects with active or passive markers. Thanks to spatially and temporally dense color and sometimes depth observations now offered with multi-view platforms, a much richer set of possibilities arise toward estimation of full shapes in motion, with detailed surface reconstruction, motion tracking, and access to the appearance through the acquired images. This is all with the advantage of passive, non invasive protocols where the subject or actor can come in full clothing and expect to be captured with all his apparel, props, and ultimately interaction with other people or with the set and scenery. But with this richer data comes a vastly increased complexity, and extracting these problems is still a very active research field to this day. This document is centered on the problem of 4D modeling, which is the process of producing this 3D content and appearance representations from temporal sequences obtained from a set of cameras. In most works in this topic, access to a controlled setup with pre-calibrated cameras is assumed, such that the focus is on obtaining the 3D model geometry, its motion and its appearance as texture map as opposed to egenral scene structure. 4D modeling differs from simple, one shot reconstructions over each frame of the sequence, by the main property that it exploits time continuity and redundancy to enhance the quality of the models, or produces models that are aligned, e.g. with identical surface topology and connectivity but deforming shape or, by means of varying model parameters or vertex positions.

CHALLENGES

Our general goal is to produce highest quality possible and easy to use models, and produce the best 4D modeling algorithms to this goal. This is a generally very challenging topic because, in its most general form, we are observing a moving shape or set of moving shapes with very indirect raw data, namely sets of image sequences

I = {I t i } t∈{1,•••m} i∈{1,••• ,n}
obtained from a set of calibrated n cameras during m temporal frames, and how to extract the shape, motion and appearance of these single underlying objects, in some sense a large scale multi-variate and interdependent regression problem. Some of the main obstacles to achieve these goals are the following :

Dimensionality

The key information is buried in several high frequency video streams that quickly comprise Terabytes of data. On a typical platform such as Kinovis, acquisitions produce data at a rate of 40Gb/s.

STRUCTURE AND CONTRIBUTIONS

The dimensionality of the output is typically also very large, even if it exploits and removes key input redundancies, but still comprises large representations, e.g. mesh and textures, with full or partial updates that need to be periodically refreshed.

Representation

Finding the best intermediate and output representations for the 4D model is still to this day an open problem. Exactly what granularity shall be stored for the geometry, temporal evolution and appearance can lead to compact results, but the desirable properties of the representations may not be the same at different stages of the pipeline. Should we use surface meshes, or volumetric primitives for the inference ? How exactly should the color information be stored, as a separate per-frame texture map, or using a temporally refactored representation ? What is the best way of co-storing shape, geometry, motion and appearance ? These are some of the questions examined in this document.

Choosing Priors

The image set we observed is a noisy, partial observation of the quantities we wish to estimate, and our problem is an inverse and ill-posed problem. Choosing the correct priors and how to embed them in our method is going to be important and in fact we have explored many different directions.

First the assumptions inherent to the model itself involve relying on regularizing behaviors, and usually in their simplest form involve geometric, motion and appearance continuity over the full shape. But choosing the correct prior is non-trivial. Typically building the prior that the motion is from a human subject is a key tradeoff we examine in this work, on one hand dealing with very basic motion hypothesis or at the opposite of the spectrum having a strong constraining human model. One will favor generality and the other will be more stable but may not account for out-of-model situations such as object or multi-person interaction or loose clothing.

STRUCTURE AND CONTRIBUTIONS

As the previous challenges hint at, the 4D modeling problem requires examining a collection of subproblems together to target a single goal. To break the complexity of the algorithm, some kind of stratification is usually chosen. Throughout our works during the years, the stratification that we opted for has been relatively uniform, first extracting the shape at individual time frames, then performing motion analysis for tracking and alignment on the individual reconstructed shapes directly in 3D, and finally extracting appearance information. This is of course not the only possible stratification, but to this day remains a very popular way to break down the problem, with variants in how geometric and temporal tasks are connected, e.g. [CCS + 15, MVK + 20].

CHAPITRE 1. INTRODUCTION

This manuscript follows these three main stages in their logical order of application in the pipeline, which in particular is quite different than the chronological order in which these works were actually executed.

In Chapter 2, I describe how we approached the problem of 3D shape estimation with one frame or small group of frames simultaneously :

-I first present two of our contributions toward designing a customized multiview stereo pipeline particularly suitable for the context of performance capture and 4D modeling tasks. A careful analysis of our needs and acquisition scenarios lead us to build-in key features and exploit local temporal redundancy to increase precision ; we also examine how deep learning can be used to optimize the feature extraction and reconstruction decisions. -Second, I present a method whereby we used an end-to-end deep learning to extract a full 3D from one viewpoint, with the idea of testing the limits of the monocular reconstruction setup, and as preliminary step to possible multicamera applications as well.

In Chapter 3, I present several works focused on temporal analysis and alignment of the 3D models previously extracted -I first summarize the works achieved on a series of models based on weak but quite general, quasi-rigid patch-based priors, using adaptive versions of patchbased constraints either at the surface or the volumetric level. -Then I examine how a stronger human shape-space model can be used to constrain the estimation of shapes in motion, and estimate the underlying body shape of acquired subject under clothing.

In Chapter 4, the discussion is on two appearance estimation works we did to retrieve an appearance map measuring dense radiance information observed :

-First, we present a principled method by which the appearance estimation of a 3D model under small motions can be treated as a specialized 3D version of the well-known video superresolution problem -Second, I discuss how this work can be extended to longer sequences by storing long term appearance variations as a linear combination of 3D mapped Eigen textures.

Before getting to the technical part of the presentation, I will describe some contextual information, in particular about the people and contexts I worked with.

PEOPLE

This manuscript is intended to give an overview of my research work as an associate professor ("maître de conférences") in computer science at Grenoble INP -Ensimag (part of the Université Grenoble Alpes, France) with a focus on PhDs defended 1.4. PEOPLE 5 in the last 5 years. During my career, I had the pleasure and privilege to work with a wide range of colleagues and students on the aforementioned topics. I will give a brief overview of these encounters in the following paragraphs. In 2007, I was appointed to my first associate professor position at Université Bordeaux 1 with Pascal Guitton and the IPARLA team, Inria Bordeaux Sud-Ouest. I co-supervized the PhDs of Yann Savoye and Robin Skowronski with Pascal Guitton, working respectively on non-rigid human tracking and calibration for drones. I also collaborated in the supervision of with PhD student Benjamin Petit with whom I interacted a lot with for the Dalia ANR project on telepresence between two multi-camera platforms Hemicyclia and GrImage.

I then moved to the Ensimag (School of Computer Science and Applied Mathematics, INP Grenoble University) as associate professor of Computer Science, and as a researcher at the Inria Grenoble Rhône-Alpes, France, with the Morpheo team in 2011 and have been co-supervising many PhD students since then, Abdelaziz Djelouah on multi-view segmentation with Patrick Perez and Francois LeClerc at Technicolor, Vagia Tsiminaki and Adnane Boukhayma on multi-view appearance estimation and superresolution, Benjamin Allain on shape tracking and temporal surface alignement, Vincent Leroy on probabilistic Multi-View Stereo, with Edmond Boyer ; and I have had a non-official supervizing role throughout Jinlong Yang's PhD on model-based human body tracking under clothing, a collaboration with Morpheo team members Stefanie Wuhrer and Franck Hetroy.

Out of the master students I supervized too numerous to enumerate here, one stands out recently as it opened a collaboration cycle with Inria Thoth team members Gregory Rogez and Cordelia Schmid, through the supervision of Valentin Gabeur. This collaboration would yield a major conference paper at ICCV 2019 and paved the way to several Morpheo-Naver Labs Europe collaborations after Gregory Rogez and Vincent Leroy joined Naver.

I am currently co-supervizing Mathieu Armando on mesh and appearance superresolution, Boyao Zhou on using deep learning for human space-time human shape reconstruction, with Federica Bogo and Edmond Boyer, as part of a Microsoft Grant ; I am also co-supervizing Abdullah Haroon Rasheed on learning based cloth physics estimation with Florence Bertails, Stefanie Wuhrer, and Mathieu Marsot on learning space-time human shape and motion generative models, with Stefanie Wuhrer, and Anne-Hélène Olivier from Inria Rennes who is a ANR project collaborator. These have all been exciting projects that are followups of the research projects described in this document, and have benefited from these previous experiences.

Manuscript Focus

It will be apparent to the reader at this point that this document is in no way meant as an exhaustive review of works and student projects. The fact that I left them out of this manuscript says nothing about those works, and everything about having to keep this document contained in the interest of time and by focusing on key emblematic stages and developments we had with the understanding of the problems in the last five years.

I have in particular left out the excellent work of Li Guan [GSFP06, GFP07, GFP08a, GFP08b, GFBP10, GFP10], the first PhD student I co-supervized with Marc Pollefeys during my post-doc, of Abdelaziz Djelouah [DFB + 12, DFB + 13, DFB + 15, DFB + 16], which have in common the fact that they are focused on earlier silhouettebased extraction efforts.

I have also taken the stance of leaving out works that were not defended yet as PhDs, leaving this discussion at the legacy stage at the end of this work, this includes the excellent work of Mathieu Armando [AFB19, AFB20], Boyao Zhou [ZFB + 20], and Abdullah Haroon Rasheed (CVPR 2020 oral paper [RRBD + 20]).

I did include Valentin Gabeur's work on monocular reconstruction with Deep Learning [GFM + 19], defended as master student, since it so well represents the Deep Learning leap and transition we took these last years, in continuity with the work of Vincent Leroy.

CHAPITRE 2 3D Shape Estimation

INTRODUCTION

Among the first problems we examine in this document, and one of the first problems to solve in stratified 4D Modeling, is that of 3D shape estimation from images. This is in fact one of the fundamental problems of computer vision and has often been treated in generic form in the geometric era of computer vision [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. Using a preconfigured platform camera rig, we have been naturally strongly inclined to pursue research in methods that assume calibration is available. This was notably the case during my thesis and post-doc years where I pursued research in silhouette-based methods [START_REF] Franco | Exact polyhedral visual hulls[END_REF][START_REF] Franco | A distributed approach for real-time 3d modeling[END_REF][START_REF]Fusion of Multi-View Silhouette Cues Using a Space Occupancy Grid[END_REF][START_REF]Efficient polyhedral modeling from silhouettes[END_REF][START_REF]3D Object Reconstruction with Heterogeneous Sensor Data[END_REF][START_REF] Guan | Probabilistic 3d occupancy flow with latent silhouette cues[END_REF].

While this push to enhance silhouette-based methods yields quite useful results and has been the basis of many other works discussed in the other chapters, during the last five years, we also pursued research on enhancing 3D modeling, first in the direction of significantly improving model quality given the abundant input data acquired through our 68-camera Kinovis platform, and second gaining better understanding of how deep learning can contribute to improving 3D modeling methods, in particular toward any of the common objectives of increasing model quality, precision, robustness to input corruption and lack of texture detail routinely encountered in everyday, casual clothing. Dealing with fast motions has also been a desirable target due to the presence of motion blur when subjects execute fast movements typical to dancing or sports. This drive was to open new research avenues in the team, as they have proven largely successful.

To illustrate the push in these directions, I will discuss in this chapter the two most relevant efforts we pursued, first toward novel improving Multi-View Stereo in the context of performance capture applications, and second toward the problem of monocular 3D shape estimation, where Deep Learning has proven to be instrumental for both. 12 CHAPITRE 2. 3D SHAPE ESTIMATION

MULTI-VIEW STEREO EFFORTS

Innovation on the quality of the shape result was a major drive throughout the last decade in the team ; and multi-view stereo had the potential to offer these benefits. But multi-view stereo is both a widely studied and notoriously difficult subject to work with on a technical level, which has stumped a number of students we attempted to tackle this problem with. A testament to this is the endurance of several widely respected benchmarks that have survived as beacon references to the community for the better part of the last decades, such as the Middlebury, the DTU and more recently tanks and temples datasets [SCD + 06, SvHG + 08, JDV + 14, SSG + 17, KPZK17], which all have registered small-step but continuous improvements in the domain over large time frames.

Also, the technical variety of the frameworks and representations devised to tackle this problem is quite astounding spanning several decades, from level sets [START_REF] Faugeras | Variational principles, surface evolution, pde's, level set methods, and the stereo problem[END_REF], voxel carving [START_REF] Kutulakos | A Theory of Shape by Space Carving[END_REF], depth map sweeping [GFM + 07], Delaunay decomposition and graph cut optimization [START_REF] Labatut | Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts[END_REF], sparse points from image features [START_REF] Furukawa | Accurate, dense, and robust multiview stereopsis[END_REF] spatiotemporal integration [START_REF] Goldlücke | Space-time isosurface evolution for temporally coherent 3D reconstruction[END_REF][START_REF] Mustafa | Temporally coherent 4d reconstruction of complex dynamic scenes[END_REF], model-based integration [SH07, dAST + 08], convex grid optimization [START_REF] Cremers | Multiview stereo and silhouette consistency via convex functionals over convex domains[END_REF] or probabilistic inference [START_REF] Osman Ulusoy | Towards probabilistic volumetric reconstruction using ray potentials, 3D Vision[END_REF], to name only a few.

It thus takes a special type of character to confront this problem, with perseverance, heterogeneous literature comprehension, and large programming and technical proficiency, which we found with PhD student Vincent Leroy. In this journey, we implemented two state of the art MVS pipelines with a classic plane-sweeping articulation but several key practical improvements, then substituted the photoconsistency core with a Deep Learning replacement that considered local photoconsistency volumes to compute a depth indicator function.

DAISY-Based Plane Sweep Stereo Pipeline with Local Temporal Integration

Our first effort in this direction was to implement a classic pipeline based on what stood out in the literature as desirable characteristics for our capture scenarios. First and foremost, we wanted to be able to enhance the quality of the surface thanks to temporal smoothing and refinement, which has been a longstanding goal [GM04, APSK07, MKGH16] seldom achieved for centrimetric detail in performance capture scenarios. Of particular inspiration both for the remarkable detail accumulation in dynamic capture environments and for the popularization of Truncated Signed Distance Functions (TSDF) as depth map volumetric fusion representation, were methods of the Kinect and Dynamic fusion family [NIH + 11, NFS15, IZN + 16, DKD + 16], a key difference with our goals being that they process RGB-D streams instead of multi-RGB inputs.

Second, depth-map plane sweep methods have been notable for one very desirable feature : their spatially dense, per-pixel monotonous depth parameterization of visible surface geometry for every viewpoint, with first-point visibility built-in the parameterization [Col96, MAW + 07]. This yields very efficient depth map extraction methods which are cast as extracting the first visible point at each pixel, gracefully dealing with the visibility problem. Third, among the various classic photoconsistency characterizations based on normalized window correlation (e.g. ZNCC, SSD, SHD, etc) or gradient feature correlation [Low04, BTG06, MS03], DAISY features [START_REF] Tola | DAISY : an efficient dense descriptor applied to wide-baseline stereo[END_REF] was the state of the art at the time for dense, wide-baseline stereo and computational efficiency, and as such was a natural contender for our photoconsistency function and evaluation.

Method and Contribution

We proposed a pipeline illustrated in Fig. 2.1 implementing together these key characteristics, assuming images I i , silhouettes Ω i , and calibration matrices π i are given.

-First we compute per-view depth maps based on line searches that maximize a DAISY-based photoconsistency criterion, filtering correct matches using a visual hull of the subject obtained from the silhouettes -Second we provide an initial estimate of a multi-view merged shape based on the fused multiple depth maps obtained at a single frame. Because of the detail density, classic TSDF on regular grids is particularly inefficient and memory heavy, so we proposed an implicit TSDF form which can be computed with sparse storage. -Third, using this initial per-frame shape estimate for a time-window of neighboring frames, we construct sparse surface feature matches to neighboring frame shapes using MeshHOG features [START_REF] Zaharescu | Keypoints and local descriptors of scalar functions on 2d manifolds[END_REF], which we densify by propagating them on the surface. -We use these matches to fold the shape contributions to the center frame in each temporal window using a simple locally rigid deformation model. -These folded shapes bring new sparse implicit TSDF constraints to the central frame, which can then be seamlessly integrated. The process is then iterated between alternate steps of matching and reconstructing, leading to a refined final shape. -The final surface can be extracted from the implicit TSDF using Centroid Voronoi Tesselations [DFG99, LWL + 09] of a carefully selected set of points samples in the vicinity of the final surface as hinted by the TSDF induced occupancy function, and clipping the CVT tesselation to the zero level set of the implicit TSDF function to extract surface polygon geometry, as illustrated in Fig. 2.2 Our results showed that the quality of the reconstructions, measured with the widely adopted Chamfer-based metrics of surface accuracy and completeness [SCD + 06, JDV + 14], was significantly improved using the various components of our proposed approach. The approach also demonstrated measurable improvements with respect to state of the art methods in the context of performance capture. Details of the method and results can be found in the ICCV 2017 publication [START_REF] Leroy | Multi-View Dynamic Shape Refinement Using Local Temporal Integration[END_REF].

Volume Sweeping : Learned Photoconsistency

During the time frame of Vincent Leroy's thesis, it became quite evident that the benefits of Deep Learning could reach beyond 2D vision problems which were the main focus in the early DNN years vision [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. As my personal and our team expertise in the subject was initially limited, our approach has been largely progressive and at first geared toward testing the contribution of learning techniques in well identified stages of time-honored vision pipelines.

In this respect, the MVS work with Vincent was an ideal candidate. As illustrated in Fig. 2. 3 showing how we modified the classic MVS pipeline, the photoconsistency function ρ used to characterize surface hits for every depth candidate along each pixel's viewing line surface depths has a well identified local support in the images at the projection of the candidate, which can be described with a small size network. The surface hit decision is typically a per pixel decision in plane sweep algorithms with optional image-domain regularization of the depth maps based on local per-pixel depth data terms. The photoconsistency function was thus ripe for this kind of trial as it also clas-FIGURE 2.3 -Volume sweeping method overview. Depth maps, for all input image I i , are obtained by maximizing, along viewing lines, a learned function ρ that measures photoconsistency at a given depth d along the viewing line of a given pixel p. Depth maps are then fused into an implicit form from which the zero set surface is extracted.

sically relies on a hand-crafted measure and empirically selected image features. This is typically where deep learning can contribute by automatically extracting relevant features from example data.

In fact at the time several works had started pushing in the direction of shortbaseline MVS with symmetric combination of 2D learned features [HGH + 17], or wide baseline sparse capture scenarios [HLC + 18, GVCH18] that followed learning research on short baseline stereo [vL16, LSU16, ZK15, UZU + 17].

Method and Contribution

While some learning methods for MVS were starting to emerge, at the time little or none of them were particularly geared or tested for the specifics of the multi-camera performance capture scenario, where in particular the attention is on moving subjects that cover only a minority fraction of all frame pixels, which we informally referred to as the "mid-range scenario". This use case is in particular very different to the case where objects are fully framed in every view, for close static objects or full-views of architectural scenes which are typical used as benchmark cases [SCD + 06, JDV + 14, SvHG + 08].

Our feeling as well was that, in our case with known calibration, we would be under-using the input data if we only used 2D patches to characterize matches in the learned photoconsistency function, as several methods were proposing [vL16, LSU16, ZK15, UZU + 17]. Several methods were also starting to propose 3D inference in the volume for these kind of reasons [CXG + 16, JGZ + 17, KHM17]. Reasoning only on 2D patches deprives the network from information on relative orientation of views and the local geometric context that can be additionally useful to build a decision. This intuition turned out to be correct and verified in our results, where we compared 2D versus 3D volumetric support of the learning function.

To these goals, we proposed to formulate the inference on a small projective volume surrounding each query point of interest along a viewing line. Each voxel of this volume is assigned two R,G,B triplets from the reference view whose depth map we are computing, and another view used to check photoconsistency. The support region of the network, and its architecture is illustrated in Fig. 2.4. The architecture is intentionally simple and classically inspired [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], with the main characteristic that the contribution of all pairs of views needs to be merged using a symmetric function computing a result independent of the number of views considered, with the mean giving the best empirical results in practice. The network was trained on a large set of samples built from the standard DTU dataset, which for this compact network can provide thousands of training samples per set of multi-view input frames given the local support of the learned function ; we also notably used no view from our Kinovis acquisition platform, a testament to the generalization ability of the method.

We obtained impressive results that, quite frankly, exceeded our expectations by a large margin. As Fig. 2.5 illustrates, the method was able to infer more details and exhibited less errors and more completeness even in color-ambiguous, partially reflective regions (e.g. thigh, boots) than the classic version of our pipeline. More unexpected, the FIGURE 2.5 -Challenging scene captured with a passive RGB multi-camera setup [kin]. (left) one input image, (center) reconstructions obtained with classical 2D features [START_REF] Leroy | Multi-View Dynamic Shape Refinement Using Local Temporal Integration[END_REF], (right) proposed solution. Our results validate the key improvement of a CNN-learned disparity to MVS for performance capture scenarios. Results particularly improve in noisy, very low contrast and low textured regions such as the arm, the leg or even the black skirt folds. method recovered fold details of the black dress in portions of the image with almost zero contrast. The results were shown to outperform several off-the-shelf available methods, including classic methods [FP10, TSF12, CVHC08] and recently published learning based methods [YLL + 18, JGZ + 17]. Interestingly, the method also recovers results of quality comparable to [CCS + 15] who use two modalities RGB and infrared pattern projections, despite the fact that we were using only the RGB inputs of their datasets for our comparison.

Details are available in the ECCV 2018 publication [START_REF] Leroy | Shape reconstruction using volume sweeping and learned photoconsistency[END_REF] but the more thorough comparisons and comments are published in IJCV 2021 [START_REF] Leroy | Volume Sweeping : Learning Photoconsistency for Multi-View Shape Reconstruction[END_REF], which is the one I am providing in the appendix A.8. We refer to the supplemental video for more results 1

MOULDING HUMANS : LEARNING 3D SHAPE ESTIMATION FROM A SINGLE IMAGE

The success of deep learning on a multi-view stereo pipeline naturally brought us to another research question : can we transfer these benefits to the less data rich case of monocular 3D shape estimation ?

The question of how much can be recovered from a single view has intrigued researchers for quite a while. From a purely geometric standpoint, the fundamental FIGURE 2.6 -Our non-parametric representation for human 3D shape : given a single image, we estimate the "visible" and the "hidden" depth maps from the camera point of view. The two depth maps can be seen as the two halves of a virtual "mould". We show this representation for one of the images of our new dataset.

projection ambiguity prohibits recovering absolute depths and scales in that situation [START_REF] Hartley | Multiple view geometry in computer vision[END_REF]. It has been however identified quite early that data-driven priors may provide the possibility of finding a solution, e.g. for the problem of recovering a human 3D [START_REF] Agarwal | Recovering 3d human pose from monocular images[END_REF]. In fact, estimating 3D poses is one of the key problems by which deep learning was thrown at monocular 3D estimation [CWL + 16, RWS19] along with denser monocular 3D estimation applications such as dense surface correspondance [WHC + 16, GNK18].

[RWS19] happens to be a Thoth team contribution down our corridor at Inria Rhone-Alpes with Gregory Rogez and Cordelia Schmid. The Thoth team was getting more and more interested in addressing 3D problems with human subjects, as evidenced by various other collaborative publications to estimate 3D shapes from images [VRM + 17, VCR + 18]. So we came to join forces with Gregory and Cordelia and co-supervized a master student together, Valentin Gabeur.

One of the issues with using DNNs for human 3D data is that straightforwardly extending CNNs from the 2D to the 3D domain, while relying on well defined extensions of the usual CNNs convolution stages in the 3D domain such as with [VCR + 18], are subject to the curse of dimensionality as this creates a large memory and computational burden in the training process. This limits the amount of training data which can be processed in a single batch and also means a large number of training parameters, with potential for overfitting.

For this reason, we set our main quest as one of finding a lean representation that would allow us to retrieve human 3D models of comparable or better quality at a fraction of the cost. In fact the lower parameter dimensionality would probably help us achieve the quality goal. The idea emerged in the discussions of performing depth inference in the image domain, but to output a surface, we need to regress not one but two depth maps, one for the front view and one for the back view of the human subject, a representation we informally coined a "mould", for the reason apparent in Fig. 2.6.

MOULDING HUMANS : LEARNING 3D SHAPE ESTIMATION FROM A SINGLE IMAGE 19

Both depth maps assembled together create a point cloud which can yield a surface reconstruction using standard techniques [START_REF] Kazhdan | Screened poisson surface reconstruction[END_REF]. Although technically only having two depth maps encoding doesn't cover all possible situations (such as self-occlusion which leads to four or more surface crossing and depth values along a given viewing ray), this first dual-depth map encoding, with the advantage of a fixed output inference representation, provided surprisingly good results already. The inference can then be expressed in terms of an image-wide regression task, for which we proposed a stacked hourglass architecture [START_REF] Newell | Stacked hourglass networks for human pose estimation[END_REF], previously shown successful for human pose regression -but not full surfaces. While generally successful, we did encounter one significant issue in that the succession of reduced latent space at the bottlenecks of the hourglass networks were not sufficiently regularizing for the network to fully encode the "humanness" prior underlying the training set, yielding humans with additional limbs or no limbs for example. For this reason, we additionally introduced a discriminator network, trained in adversarial fashion, to allow this to happen. This proposed pipeline is illustrated in Fig. 2.7. FIGURE 2.7 -Overview. Given a single image, we estimate the "visible" and the "hidden" depth maps. The 3D point clouds of these 2 depth maps are combined to form a full-body 3D point cloud, as if lining up the 2 halves of a "mould". The 3D shape is then reconstructed using Poisson reconstruction [START_REF] Kazhdan | Screened poisson surface reconstruction[END_REF]. An adversarial training with a discriminator is employed to increase the humanness of the estimation.

We devised a training dataset, 3D HUMANS, with a large synthetic portion from [VRM + 17], complemented by real captures from the Kinovis platform to capture some clothing and shape variability absent from the latter. We obtained very good to excellent results, improving over state of the art approaches, with shorter training and inference times. This includes on test datasets that had nothing to do with the training such as the DeepFashion dataset, with images of women in completely different clothing or postures, as evidenced in Fig. 2.9, Fig. 2.8. The contribution of the GAN can also be appreciated in these figures under severe input frame occlusion. More details and results can be found in the attached paper A.7 (Appendix A.7), which was published at ICCV 2019 [GFM + 19] with the master student, and the supplemental video 2 .

2. https://hal.inria.fr/hal-02242795/file/Moulding-Humans-ICCV2019. mp4

MOULDING HUMANS : LEARNING 3D SHAPE ESTIMATION FROM A SINGLE IMAGE 21

Ours HMR [18] Bodynet [33] FIGURE 2.9 -Comparison between HMR [KBJM18] (left), Bodynet [VCR + 18] (middle) and our method (right). Unlike [KBJM18, VCR + 18], we do not train on inthe-wild images but estimate 3D shapes of clothed subjects.

CHAPITRE 3 3D Motion Estimation

INTRODUCTION

In this chapter, we examine the problem of tracking shapes and estimating surface alignments over complete temporal sequences of shapes seen from multiple calibrated cameras.

When estimating 3D shapes per frame, one is left with a representation that has no temporal consistency, meaning the geometric representations have no primitive correspondence. In our case with the Kinovis platform multi-RGB video streams, and applying the algorithms of the type described in chapter 2, that means we retrieve a set of meshes per time frame that are all independent and possess completely different, unmatched sets of vertices and triangles. 1These per-frame representations can be used as-is for some applications, such as streaming and displaying the models in sequence, but this has a number of limitations. The first one is that it is memory and bandwidth inefficient. All resources must be transferred from scratch per frame, typically to the GPU for displaying or through the network for streaming. Pre-loading the sequences doesn't circumvent this bottleneck as it introduces a transfer latency at the beginning of a sequence. Additionally, this may not be possible on a GPU because each mesh can be several megabytes depending on resolution and a sequence of meshes can saturate GPU memory quite fast. All of these problems are drastically worse when one starts attaching attribute data to the mesh, typically appearance texture data which allows to rendering perceived detail on the surface of the shape.

CHAPITRE 3. 3D MOTION ESTIMATION

More generally, all temporal-related tasks requiring surface tracks or speed or a motion representation, such as motion editing and motion-driven interactive setups, are not possible with this inadequate frame-to-frame representation.

Fundamentally, throwing away the previous reconstructed shape information is sub-optimal : in typical acquisition scenarios involving human or non-human subjects, the underlying shape observed in motion has a fixed physical envelope and topology, or a set of discrete geometric topologies, depending on whether self-contacts are considered as merges of the geometic envelope of the object [START_REF] Letouzey | Progressive shape models, Computer Vision and Pattern Recognition[END_REF]. Using the hypothesis of a common underlying surface means recovering the sequence as a moving shape, whose topology is at least temporarily stationary. In concrete terms one can then store a single shape topology or sparse set of topologies for the whole sequence, often referred as template, and store frame updates as a set of relative or absolutes motions, expressed as raw vertex displacements, local piecewise transforms using a shape decomposition such as patches [CBI10b], or updates in a parametric or kinematic representation [START_REF] De Aguiar | Marker-less deformable mesh tracking for human shape and motion capture[END_REF].

This provides a wide set of benefits, notably reducing the memory and bandwidth requirements of the representation, since only the motion updates need to be transferred per frame. This is especially true when one attaches additional surface attribute information, such as a appearance, to the fixed-topology templates rather than each time frame [CCS + 15], thus avoiding needless duplication. From a broader standpoint, subject and human motion analysis then become possible, such as monitoring specific attributes of motion, e.g. trajectories of specific landmarks, building motion statistics, using the model to compute dynamic collisions with virtual objects in a immersive digital world, with a wide range of applications for sports, medical, virtual reality, entertainment and interactive systems. This type of representation is also a pre-requisite to make the motion sequence editable [START_REF] Stoll | A volumetric approach to interactive shape editing[END_REF][START_REF] Matsuyama | Generation, visualization, and editing of 3d video[END_REF], to use it in a content production pipeline where artists or designers wish to use real-life full shape captures as the starting point for 3D asset creation, in which modification and stylisation of the motion are targeted.

Alignment Problem

The benefits are clear, but the problem itself is challenging due to various aspects. Looking at it from a global perspective, it can be seen as retrieving from a set of image sequences I = {I t i } t∈{1..m} i∈{1..n} a single, time-independent set of parameters of a shape model S of the subject on one hand, and a set of (most often per-frame t) motion parameters of a motion model Θ = {Θ t } t∈{1..m} on the other hand, such that {S , Θ} optimally explain the full set of image sequences I. The shape representation S will contain geometric attributes of the shape, describing its surface, volume, pose, etc., and may also include non-geometric attributes such as appearance, or other features.

Although some methods do treat the problem this way as a single, global sequence retrieval problem [START_REF] Goldlücke | Space-time isosurface evolution for temporally coherent 3D reconstruction[END_REF], in particular in some recent publications [START_REF] Niemeyer | Occupancy flow : 4d reconstruction by learning particle dynamics[END_REF], it is very complex and technically challenging for a number of reasons. First the size of the image sequence data I, and shape and motion parameter dimensionality, usually prohibits global optimization update steps relating directly all images I to all motion parameters Θ and the shape S . Second because in several ways this is a chicken and egg problem : updates to the shape require updates to the motion parameters, and vice versa. Matching shape surface points to identified 2D projection in the viewpoints in the different temporal frames requires updating the motion model, and vice versa. Third it is intrinsically difficult to formulate spatio-temporal surfaces directly in terms of multi-view and temporal image content variation, which this view of the problem implies.

Interestingly some formulations close to the latter standpoint exist, e.g. [START_REF] Goldlücke | Space-time isosurface evolution for temporally coherent 3D reconstruction[END_REF] where the moving shape is described as an spatiotemporal isosurface optimized from image variations. While quite elegant, as in many methods, the formulation however needs to make practical compromises, e.g. it forgoes describing the single underlying shape S , and in practice makes partial updates in frame batches that are slow and susceptible to local minima.

Because of these difficulties, the vast majority of estimation methods in the literature propose stratified approaches, breaking the problem down into more tractable sub-problems, and this is still the case today with state of the art approaches, e.g. [BHKH13, MVK + 20]. This can be done along various dimensions, e.g. Simon et al. [START_REF] Simon | Separable spatiotemporal priors for convex reconstruction of time-varying 3d point clouds[END_REF] consider spatiotemporal priors on point trajectories, putting more emphasis on temporal connections than shape connectivity. But a majority of techniques first pre-process whole shapes in individual frames, yielding a set of independent shapes {S t } t∈{1,••• ,m} substituted for inputs of the 3D temporal alignment stage, which is the strategy we follow. This is also a natural path to tackle the problem by stepping up from our expertise in 3D reconstruction techniques from a calibrated set of images.

Model-Based Approaches

In the stratified view of the problem relying on per-frame reconstructions, we must first select and conceive shape and motion models. Various strategies exist to this end, but in this document and in the work described in this chapter we took interest in so-called model-based approaches. These adopt a further simplification to the general paradigm described earlier in §3.1.1, by restricting the shape S either to a family of parametric shapes [ASK + 05, HAR + 10, NH13, PWH + 17] or to a single template shape [SH03, BC08, VBMP08, GSA + 09, LGS + 13], to name only a few. In our work, we explored both possibilities. This simplification thus trades the complexity of matching of spatio-temporal 4D representations to input images, for the problem of fitting a model by optimizing its deformation parameters Θ t such that it best explains per-time step 3D reconstructions.

Examining the literature, one can see that most such approaches mostly share a common general canvas : once the motion model is chosen, they all require defining a 3D matching cost or loss to provide a metric measuring the disparity between the deformation estimate and the 3D reconstructions used as input of the fitting. This loss can be either directly minimized for certain forms of the loss function, e.g. Chamfer which considers maximum surface to surface distances, or the minimization can be interlea-ved with an association step, where shape primitives are matched to the primitives of per-frame reconstructions in ICP fashion [START_REF] Besl | A method for registration of 3-d shapes[END_REF][START_REF] Granger | Multi-scale EM-ICP : A fast and robust approach for surface registration[END_REF][START_REF] Mundermann | Accurately measuring human movement using articulated icp with soft-joint constraints and a repository of articulated models[END_REF].

I will here discuss various template-based approaches we explored based on motion expressed on detecting rigid parts for surface or volume patch decomposition of a single template shape to track ( §3.2), and an approach based on the human shape space S-SCAPE which is equipped with a multi-linear deformation framework for identity and kinematic motion ( §3.3), all of which illustrate particularly well our research interests and progress on this subject.

SURFACE AND VOLUME PATCH-BASED SHAPE TEMPLATES

With the first model, our goal was to explore different improvements of a fixed template object S that allow more robust and more precise fitting. A very popular strategy which we follow in this work is to use a particular capture or scan of the exact subject to be tracked as template shape, popularized by various works in the 2000s [dAST + 08]. With this simplification advantage of a fixed shape S , the essence of the alignment approach is to define a deformation model and its motion parametrization Θ t .

Many such parametrizations exist in prior works. One is to rig a generic human model using kinematic chain which can be pre-fitted to the template [BC08, VBMP08, GSA + 09, LGS + 13] or in some cases automatically extracted [START_REF] Baran | Automatic rigging and animation of 3D characters[END_REF]. Another is to relax this strong kinematic prior by using local surface rigidity constraints, with the idea that looser skin or clothing can then also be fitted by the model. Typically, the idea is to preserve local intrinsic surface properties, e.g. isometric deformations [MS04, BBK06, OMMG10, SY10, BPC13], conformal deformations [BGC + 15], inextensibility of the template mesh [START_REF] Mathieu Salzmann | Closed-form solution to non-rigid 3d surface registration[END_REF], or general neighborhood preservation through functional warps [START_REF] Del Bue | Multiview 3d warps, 13th Internationl Conference on Computer Vision (ICCV 2011)[END_REF]. Others use properties related to local rigidities, e.g. as-rigid-as-possible deformations or elasticity minimization in [WSSC11, ZWG + 13, BGC + 15]. Among the methods allowing to express rigid cohesion of the template surface, and a particular topic of interest to us, patch-based methods [CBI10a, CBI10b, BHH11] offer an interesting compromise as they decorrelate the surface support of the deformation model from the geometry resolution, by formulating elasticity-like constraints between a set of pre-computed surface patches.

To summarize, on one hand at the time we had a large set of methods with very strong priors on human articulation, which couldn't deal with non rigid surface aspects such as clothing or objects held by the subject, on the other hand we have a set of generic surface fitting methods that use weak rigidity constraints that can deal with many kinds of surfaces but are completely agnostic to the underlying characteristics of the shape, assuming instead uniformly distributed local cohesion properties.

This raised my curiosity and sparked a research effort aimed at finding a middle ground between the two families, i.e. a deformation model that could be applied to most situations, e.g. for humans with or without loose clothing or even holding objects, able to model both rigid and non-ridid aspects in a single method. One question this FIGURE 3.1 -Illustration of the convergence of algorithm [START_REF] Franco | Learning temporally consistent rigidities, Computer Vision and Pattern Recognition[END_REF] for a single frame of the LOCK sequence, courtesy [START_REF] Starck | Surface capture for performancebased animation[END_REF]. Each rigid patch assignment is given by a different color.

raised for me was whether an alignment inference method could become automatically aware of the distribution of rigid regions on the template surface.

Preliminary Effort on Rigidity Learning

My first attempt toward this goal [START_REF] Franco | Learning temporally consistent rigidities, Computer Vision and Pattern Recognition[END_REF] was to devise a method that introduced rigid segmentation parameters K v ∈ {∅, 1..k} associated to every vertex v on the template surface. Each of the k rigid components was assigned a set of rigid motion parameters Θ t k per time step t and rigid component k. An additional robustness outlier class label ∅ was included for every K v to allow for a vertex to also be drawn from a free uniform motion component unexplained by the k rigid components. Each input observed surface point o at time t was also given an association variable V t o basically attaching a matched template vertex v to every observed point of the input reconstructions at time t.

A joint probability distribution p(K v , V t o , Θ t ) was then expressed on the full set of statistical variables K v , V t o and Θ t to allow for a Maximum A Posteriori (MAP) to be computed, using Gaussian priors for the distribution of labels on the template mesh, and Gaussian noise distributions on the difference between observed surface points o, t and their deformed template associated vertex given by V t o . By treating K v and V t o as hidden variable sets in the inference, one could then write a formal Expectation Maximization algorithm akin to a specialized GMM, that basically extracted a locally optimal point estimate of the rigid-component transforms, a set of per-vertex probability tables over K v that amounted to fuzzy and automatic rigid-guided patchsegmentation extension of [CBI10a], and sparse probability tables over V t o for each observed vertex o at time t, giving probabilistic associations to the data points at each time frame, which essentially builds EM-ICP in the method. This was published in CVPR 2011 [START_REF] Franco | Learning temporally consistent rigidities, Computer Vision and Pattern Recognition[END_REF]. The method had some excellent qualities, in simultaneously exhibiting a datadriven solution to rigid segmentation of the template, rigid motion and closest point assignments with closed form updates resembling those of a standard GMM, essentially giving a principled solution to the template-based multi-rigid EM-ICP problem. However, this initial attempt was also undermined by possible losses of tracking for our practical data, with complete stretching and disconnection of the rigid patches. This is because it had not built strong inter-vertex cohesion constraints in the method and the general surface cohesion provided by the rigid components was too weak to prevent a rigid component to go astray.

Detecting Rigidities on a Patched-Based Surface Template

We were still convinced that a more performance capture-suitable analysis was possible and that it would yield improvements in results quality with respect to purely agnostic models. So we included this as PhD topic in European Project React that was aimed at creating new vision-driven workflows for 3D digital content creation, which sparked a 4-year effort on this topic with PhD student Benjamin Allain. The first work was also motivated by discussions with Tony Tung of Kyoto University at the time, and the paper became a collaboration.

Our first effort was a better attempt to a surface-based template with patch-based deformation. To this goal, we opted for a much simpler model than previously. First we computed a fixed patch decomposition of the template surface into k uniformly distributed patches (see Fig. with each C k,l ∈ {0, 1}, for every pair k, l in the set N of immediately adjacent patches on the fixed template geometry. C k,l is meant as a time-independent inference variable describing how rigidly correlated two neighboring patches are.

In some sense this can be seen as a simpler way to implement the initial idea of §3.2.1 [START_REF] Franco | Learning temporally consistent rigidities, Computer Vision and Pattern Recognition[END_REF], first because the rigidity variables C k,l can together express arbitrary rigid aggregations of the initial and smaller "primordial" k patches. Second, while the number of rigid components were fixed in the previous effort [START_REF] Franco | Learning temporally consistent rigidities, Computer Vision and Pattern Recognition[END_REF], the arbitrary rigid aggregations afforded by the C k,l variables allow support for any composed set of k -component patch aggregates by this method, as long as k <= k. Third, the strong surface cohesion previously missing is now built-in the inter patch distance prior in the expression of a surface prior term which basically quadratically penalizes neighboring patches that come apart. When C k,l = 0, that is when patches k and l are non-rigidly correlated, the surface term is chosen to allow neighboring patches to loosely hinge. When C k,l = 1, that is when pacthes k and l belong to the same rigid group the surface intrinsic penalty term penalized any non rigid movement between the two patches. The probability assigned ot each C k,l is then expected to be estimated based on the data at inference time thanks to these prior constraints.

An additional contribution of this model is in how it computes inter-patched distances. We had noticed a bias in how Cagniart computed inter-patch penalties [CBI10a] that basically favored estimates close to the default template pose. For this reason, we included a mean pose Θ as a point estimate to be retrieved during the optimization for a full sequence, which allows the model to significantly deviate from the default template pose, and also acts as a regularizer to stabilize estimates patch pose etsimates in the sequence.

Similarly to §3.2.1, each (now constant) patch k is again given a set of rigid transform parameters Θ t k at each time frame t of the processed sequence of reconstructions, and each observed point o on the input 3D meshes at each time step t is again given a template vertex assignment index V t o . The inference can then again be expressed as a MAP over the joint probability p(C, V, Θ, Θ), treating C k,l as hidden. Using Expectation Maximization, one then alternates between refining point estimates of the patch transforms Θ t and mean patch pose Θ in the maximization step, and computing probabilities over each patch pair (k, l)'s rigidity variable in C k,l on one hand, and EM-ICP-like template vertex assignment probabilities V t o on the other hand, both as part of the E-step.

The results obtained with this simple idea confirmed the expected regularizing behavior of sharing a common rigidity inter-patch property over all frames of a sequence (see Fig. 3.2 and Fig. 3.4), leading to better temporal alignment and more robustness, quite apparent in Fig. 3.3. The features we built in the method also acted as a form of damping and stabilization in the inference when applied in sliding-window fashion across longer sequences. Detailed results are available in the original ECCV 2014 publication [START_REF] Allain | On mean pose and variability of 3d deformable models[END_REF] (available as Appendix A.2) and supplemental video2 .

Volume-Patch Rigidity-Enforced Template

Although encouraging, it appeared from previous results that one of the remaining limitations of the previous method was in fact a common limitation of the surfacebased family of methods : the method exhibits some stretching, squeezing and generally rubbery looking artifacts in non rigid zones.

As with LBS, this was prominent at kinematic joints, where a non smooth split between two neighboring rigid or quasi-rigid is expected ; however the instrinsic surface priors we chose, when weighted by an intermediate, undecided probability of rigid correlation in the neighboring patches, tend to smoothen the energy required to follow the limb joint across limbs rather than putting all the deformation at the joint location (see which highlights these cases).

Our hope was in some sense that the rigidity inference and regularizing effect would at least partially mitigate some aspects of the problem, which turned out to be correct, but we also sensed that this could be better enforced. One idea could have been to use a different norm as support for the surface tension expression, which in fact has later been explored at the time of this writing, e.g. [GXW + 15] in the case of depth camera streams. Another possible path which we ended up following was to enforce rigidity constraints at the volumetric level, where volume preservation could then be built in the model to alleviate rubber and squeezing effects. A source of inspiration was to observe what had been done in the graphics community to express volumetric deformations and to mitigate problems of the LBS model [ACOL00, ZHS + 05, BPWG07], which in fact had already been used for performance capture applications [dATSS07, dAST + 08, BH10].

As illustrated in Fig. 3.5(a), those approaches are however based on tesselations of surface points of the input reconstructions and do not introduce any inner vertices. This allows some volumetric constraints to be taken into account for the deformation energy, but prevent a full volumetric treatment of the alignment problem. Other tesselations of the volume are possible (Fig. 3.5(b) and (c)), such as a regular voxel grid, but it is non isotropic and biased along the axis directions. Using Voronoi cells of points randomly drawn inside the volume is insufficient as it yields irregular inner cells.

Instead we propose a fully dense an regular volumetric treatment of the volume using a CVT of the volume (Fig. 3.5(d)). Informally CVT are a particular type of Voronoi tessellation where the samples are iteratively repositioned to coincide with the center of mass of their cell, which achieves the desired properties [DFG99] : isotropy, rotational invariance, uniform cells of compact and regular form factor, regular intersection of boundary cells and surface, independent cardinality and practical computa- tion. On can also choose the number of final cells desired, thus adapting the complexity to the desired problem granularity.

The tracking methodology and deformation model is then strictly analogous to §3.2.2, the main initialization difference being that both the inputs and template shape are now tesselated into a set of CVT cells, on which the whole algorithm operates. Namely, the initial patch decomposition is now expressed on volumetric cell groupings instead of surface vertices, and observation-to-template matches are also expressed from input volumetric cells to template volumetric cells. We keep the rigidity inference on C k,l variables based on pairs of adjacent volumetric patches in the template as a complementary regularizer of rigid behavhior over each inferred sequence as in the previous method.

We observed significant qualitative and quantitative improvements on Kinovis and BBC React acquired datasets 3 , as measured with both the silhouette reprojection error for template-fitted sequences and sparse ground truth marker RMS error for some sequences which had been simultaneously captured with multi-RGB and sparse markers [LGS + 13], as illustrated Fig. 3.6. One can notably observe the more natural folds at kinematic joints, and the increase robustness even with truly nightmarish data, e.g. GOALKEEPER has some frames where the subject gets up after leaning on the floor, with a very messy visual hull reconstruction, and the volumetric approach does not lose the main anatomic features. This paper, provided in appendix A.3 was presented as an oral at CVPR 2015 [START_REF] Allain | An efficient volumetric framework for shape tracking[END_REF]. More results can be seen in the supplemental video 4 .

Volume-Based Tracking-by-Detection

For the record, I will briefly mention here an interesting ramification of this work demonstrating the usefulness of a CVT-based volume approach. Previously, the deformation model was the focus of our discussion and work, and associating observations to the template was performed with EM-ICP type terms. We showed, with a collaboration with Chun-Hao Huang, Federico Tombari, Slobodan Ilic and Nassir Navab, that is also possible to use the volumetric CVT template decomposition as the basis for the association component of the loop. For this, we directly took inspiration from work on the Vitruvian Manifold [START_REF] Taylor | The Vitruvian manifold : Inferring dense correspondences for one-shot human pose estimation, Computer Vision and Pattern Recognition[END_REF], where the depth-anchored, learned data-driven association model based on Random Forests [SFC + 11] was generalized to continuous regression of depth points to a generic template surface, then applied as a one-shot association component to drive a non-rigid human body model fitting scheme.

We proposed in [HAF + 16, HAB + 18] to substitute into a Vitruvian-like pipeline CVT-based representations for the volumetric feature extraction and regression stages with Random Forests, this time driving the volume-based deformation model of volumetric templates described in the previous section, with of course some technical adjustments. The results showed drastically improved stability of the results and better trainability for a CVT-decomposition of the shape versus a training based on a regular grid volumetric feature support. We show results where the alignment is recovered even with truly challenging situations where the subject is a performing a full cartwheel, without any tracking loss. The PAMI version of this article explains the original methods, as well as additional temporal stability improvements, is provided as appendix A.6, with more results in the supplemental video 5 . FIGURE 3.7 -Overview of the proposed pipeline. From left to right : input frame, automatically computed landmarks using [START_REF] Zuffi | The stitched puppet : A graphical model of 3d human shape and pose[END_REF], result after estimation of initial identity and posture, final result, and overlay of input and final result.

INFERRING SHAPE UNDER CLOTHING USING SHAPE SPACES

In the work discussed above, we have used priors over rigidity but not complete priors of observing humans, only using the template shape but no constraint on human kinematics. This has both advantages and disadvantages. On one hand the generic nature of the deformation and fitting model allows uniform treatment of a wide variety of scenarios, such as people in clothing, people holding objects, and some hair detail, as long as the geometry of these features are pre-observed as part of the template used for tracking the sequence. Those added feature would elude methods purely based on a generic human shape template with no clothing for example. On the other hand, one can argue that, since we are observing humans, it only makes sense to use that prior in some way.

Various methods have gone in the direction of either explicitly including a clothing model, for example a specific skirt or type of clothing [RKP + 08]. More recently more and more layered models have been proposed that use the underlying body and kinematic structure as a shape and motion prior guiding an outer cloth layer [START_REF] Zhang | Detailed, accurate, human shape estimation from clothed 3d scan sequences[END_REF].

Before these latest works, in 2016, we worked on model that estimates shape under clothing, that is by only using the outer envelope over a fully observed sequence as a guide constraining the underlying human shape possibilities and narrows them down to the most plausible data given the indirect observations of it.

But this only works if we are given a low dimensional representation describing the full set of plausible human body shapes. This is why we rely on S-SCAPE, a mul-CHAPITRE 3. 3D MOTION ESTIMATION tilinear variant of SCAPE that is easier to manipulate and optimize. This was an ideal topic to start collaboration and PhD co-supervision of student Jinlong Yang, with then newly arrived Inria Researcher Stefanie Wuhrer, who joind the team in 2015 and had prior expertise in this field as a co-author of the S-SCAPE model. The well-known SCAPE family of methods has been introduced in the 2000s as a means to encode body shapes with two sets of parameters, a set of kinematic parameters governing the body pose observed, and a set of intrinsic body shape parameters. The kinematic set is the same controlling as other kinematic model based methods, but the body shape parameters are essentially a set of coordinates in a PCA-shape basis, which is learned from a database of human shapes with different body characteristics, such as height, gender and corpulence. The immense advantage of such a model is its ability to describe a specific body shape instance, also called "identity", with a few dozen parameters, and a description of shape and pose together only spanning around a hundred parameters.

We cast the fitting problem as a minimization procedure over the set of identity parameters β, which basically encode the estimated intrinsic shape S of the subject, and the set of m per-frame poses {Θ t } t∈{1,••• ,m} , which in this case are the set of joint parameter of the S-SCAPE LBS-based kinematic model. We define a loss to minimize over the set of observed input 3D shapes over time {S } t∈{1,••• ,m} , with

E sequence (β, Θ t , S ) = m t=1 ω lnd E lnd (β, Θ t , S ) + ω data E data (β, Θ t , S ) + ω cloth E cloth (β, Θ t , S ), (3.1)
where E lnd (β, Θ t , S ), E data (β, Θ t , S ) and E cloth (β, Θ t , S ) are energy terms weighted by scalars ω lnd , ω data and ω cloth . The landmark term E lnd (β, Θ t , S ) measures the distance between a set of automatically computed landmarks on S and their corresponding anatomical points on c(β, Θ t ). This energy allows to obtain a rough estimate of the body shape and posture at each frame. The data term E data (β, Θ t , S ) measures the distance of points c(β, Θ t ) to the nearest neighbors on scan S and serves to pull the estimate towards the observed scan surface. This term allows to obtain a good estimate for the identities β. The clothing term E cloth (β, Θ t , S ) accounts for the loose clothing by encouraging c(β, Θ t ) to be located inside the observation S . Since the cloth term is applied to all frames, it allows to take advantage of the motion cues observed throughout the sequence ; as the clothing moves close to localized regions of the body in different frames it restricts the underlying shape §encoded by β to essentially lie inside the observed cloth for all frames.

To acquire our solution over full sequence which may span hundreds of frames, as illustrated in Fig. 3.7, we begin by extracting the matching landmarks c(β, Θ t ). Second, we initialize the identity parameters β by minimizing (3.1) for a small subsequence at the beginning of our full sequence. Third we solve (3.1) for the pose parameters Θ t sequentially over the full sequence, while fixing β. Fourth, we refine the identity parameters β.

To evaluate the proposed framework, we have captured a database of six subjects (three male, three female) performing three different motions and wearing three clo- thing styles each using the Kinovis platform, which has become popular with the community, (e.g. [START_REF] Zhang | Detailed, accurate, human shape estimation from clothed 3d scan sequences[END_REF]), the conference paper [START_REF] Yang | Estimation of Human Body Shape in Motion with Wide Clothing[END_REF]. We make it available as Appendix A.4 and refer to the supplemental video for more results 6 . Figure 3.8 shows six representative frames of the database. Figure 3.9 evaluates the posture estimation on manually placed markers for walking sequences captured in tight clothing. Note that the use of S-SCAPE as statistical prior prevents drift. Figure 3.10 shows further qualitative results of our method. Due to these very interesting results and originality of the work, the paper became quite popular inspiring for instance several great followups in other teams, e.g. [START_REF] Zhang | Detailed, accurate, human shape estimation from clothed 3d scan sequences[END_REF]. We also had a followup collaboration with Jinlong Yang and Stefanie Wuhrer examining the statistics of the outer shape as a vertex displacement layer, which I'm not discussing here in detail in the interest of space [START_REF] Yang | Analyzing Clothing Layer Deformation Statistics of 3D Human Motions[END_REF]. 6. https://hal.inria.fr/hal-01344795/file/supplementaryVideo.mp4

CHAPITRE 4

Appearance Estimation and Refinements

INTRODUCTION

Multi-camera capture platforms such as the 68-camera Kinovis produce a lot of observations on the same scene and from the same surfaces. Thanks to the sequence alignments described in the previous chapter, we have access to an additional source of observation redundancy. A natural research goal that has been the focus of our attention in the last decade, is how to best exploit this redundancy to enhance the quality of the models produced. This is a particularly important aspect of our work since, as was previously discussed, the subjects we acquire cover only a fraction of the input image pixels, limiting the actual amount of appearance data present in each input frame.

We discussed several of our works toward this goal concerning geometry acquisition in chapter 2, but another aspect to enhance the perceived realism and appeal of our output models is to acquire the fine-grain color appearance and texture of those subjects. To this goal we can leverage the multiple RGB streams relating to the same underlying surface.

We here discuss two families of works in this direction that we carried, to acquire appearance data, first in the direction of multi-view superresolution ( §4.2), second in how to efficiently describe and estimate a temporally evolving appearance representation ( §4.3).

HIGH RESOLUTION 3D SHAPE TEXTURE FROM MULTIPLE VIDEOS

Retrieving appearance information from all views of the subject raises interesting questions. In the case of a single video, a vast literature on 2D superresolution methods exists, which by analogy raises the possibility of retrieving appearance detail from the multi-view streams, with a quality beyond that of any given input frame in isolation.

When we took on this project with PhD student Vagia Tsiminaki, co-supervized with Edmond Boyer for the React European project on content production, the literature on the subject mostly looked at multi-view texture estimation without looking at the temporal redundancy, by aligning the various contributing images onto a single texture to avoid ghosting [TAL + 07, LHS01], or building a texture patchwork of singleview contributions optimized with graph cuts [START_REF] Victor | Seamless mosaicing of imagebased texture maps[END_REF], extended to the temporal domain as one of the rare methods addressing time redundancy in the multi-view case [START_REF] Janko | Spatio-temporal image-based texture atlases for dynamic 3-D models[END_REF] at the time. Only a handful of particularly relevant works has started to examine this explicitly as a multi-view superresolution problem, but without considering temporal frames [START_REF] Goldluecke | Superresolution texture maps for multiview reconstruction[END_REF][START_REF] Goldluecke | A super-resolution framework for high-accuracy multiview reconstruction[END_REF].

The literature in the monocular video case is however abundant. An interesting thing we learned from reviewing those works is that a well identified generative model of low resolution image formation had emerged, as a geometric warping, blurring and sub-sampling process of the initial high-resolution image [START_REF] Baker | Limits on super-resolution and how to break them[END_REF]. Of particular interest to us are that this model can be represented by a stack of linear transforms and that Bayesian noise models have been developed to explicit the noise dependencies and statistical priors over the image and warps to estimate [START_REF] Fransens | Optical flow based super-resolution : A probabilistic approach[END_REF], some using the L1-norm based priors and total variation (TV-)minimization popular for image restoration tasks [START_REF] Liu | A Bayesian approach to adaptive video super resolution[END_REF]. Notably, super-resolving multiple videos of a moving subject was examined in a performance capture context, but only for the input viewpoints [START_REF] Tung | Simultaneous super-resolution and 3D video using graphcuts[END_REF].

Proposed Methodology and Contributions

Essentially no multi-view superresolution technique existed that used the timetested elements of 2D superresolution generative models to retrieve a common appearance map for a 3D model, nor was the temporal aspect really examined in this context either. Yet the building blocks for that were there, including for example papers such as [EDM + 08], which hinted at how optical-flow-driven warps could be substituted for the 2D optical flow driven warps in 2D superresolution pipelines. We looked at the problem as one of inferring a single appearance map T attached to a reconstructed geometry pre-aligned with [START_REF] Allain | On mean pose and variability of 3d deformable models[END_REF], from all input frames

I = {I t i } t∈{1,••• ,m} i∈{1,••• ,n} .
We proposed to generalize the linear generative pipeline of 2D superresolution technique [START_REF] Liu | A Bayesian approach to adaptive video super resolution[END_REF], which explains the low resolution input views as warped, blurred, subsampled versions of an underlying high resolution (HR) image ; by filling the gap in explaining the high resolution images as a projected version of the high resolution common texture. This framework is illustrated in Fig. 4.1.

To generate an input image from each HR image H t i , the HR image is first warped according to the different sources of variability apparent in the image -calibration error, distortion, model geometry error -using a dense warp field W t i . This warp results in a linear operator over the HR image, which we note F W t i . The image then traverses the optical system, where it is blurred and captured by the CCD which performs light integration at every photosite. Following 2D super-resolution literature [START_REF] Baker | Limits on super-resolution and how to break them[END_REF][START_REF] Fransens | Optical flow based super-resolution : A probabilistic approach[END_REF] this is generally modeled using a Point Spread Function (PSF) with the form of a Gaussian blur kernel, followed by an image subsampling stage. Both operations can be written as linear operators, the image-wide blur operator K and subsampling operator S, which are applied to the HR image to obtain a view's observed image I t i = SKH t i . Remarkably, in its noiseless form, the full image formation model can thus be noted as a single, sparse linear operator A t i = SKF W t i P t i for each view {i, t}, with 3×w T ×h T rows and 3 × w I t i × h I t i columns, such that I t i = A t i T for each view {i, t}. This linear model is then used as the foundation to describe the noisy dependencies in the model and ultimately make this a Bayesian generative model, whose MAP we estimate by alternating the minimization of the common appearance map T and the warps W t i , in analogous fashion to the original 2D algorithm.

Results

We compared a static version of our algorithm to the then state of the art static multi-view resolution technique of [START_REF] Goldluecke | A super-resolution framework for high-accuracy multiview reconstruction[END_REF], with results significantly improved, with less noise and artifacts, as illustrated in Fig. 4.2. The method was also successfully tested on several datasets acquired with the Kinovis platform and with the BBC React platform, shown in Fig. 4.3, on small temporal windows, in particular comparing the results obtained with 1, 2 or 3 frames with each time a measurable improvement. A more detailed version of the framework, which was published at CVPR 2014 [START_REF] Tsiminaki | High Resolution 3D Shape Texture from Multiple Videos[END_REF], and additional results and comments are available as Appendix A.1 and the supplemental video1 .

EIGEN APPEARANCE MAPS

In the latter work, we were able to retrieve high quality textures, by performing a principled fusion of the appearance data present in the different views and the different time frames. This however can only be valid over a relatively short temporal window before the perceived appearance changes. This is because the images provide a reading of the radiance of the surface patches each pixel sees and not an intrinsic value ; and this radiance changes with the surface motion and orientation, and with significant changes in the lighting conditions that may occur over a larger temporal window. Appearance changes may also encode desirable appearance variations, such as change in lighting or personal expression of the subject, that have a negligible impact on geometry and could not be encoded at that level. One strategy is therefore to store estimated textures for as long as the texture doesn't significantly change, e.g. as in [CCS + 15]. But how to represent this information for longer sequences is still an open problem and needs to account for all these different sources of variability. With PhD students Vagia Tsiminaki and Adnane Boukhayma, and given previously encouraging results described in the previous section, it came as a natural extension of their work to tackle this challenge and examine how to retrieve and represent appearance maps over full acquired 4D sequences.

We proposed to advance this aspect by providing a view-independent appearance representation and estimation algorithm, to encode the appearance variability of a dynamic subject, observed over one or several temporal sequences. Compactly representing image data from all frames and viewpoints of the subject can be seen as a non-linear dimensionality reduction problem in image space, where the main nonlinearities are due to the underlying scene geometry. Our strategy is to remove these non-linearities with state-of-the-art geometric and image-space alignment techniques, so as to reduce the problem to a single texture space, where the remaining image variabilities can be straightforwardly identified with PCA and thus encoded as Eigen texture combinations. To this goal, we identify two geometric alignment steps. First, we co-FIGURE 4.4 -Overview : Time consistent shape modeling provides datasets of appearance maps. Our proposed method exploits the manifold structure of these appearance information through PCA decomposition to generate the Eigen appearance maps relative to a shape. arsely register geometric shape models of all time frames to a single shape template, for which we pre-computed a single reference surface-to-texture unwrapping. We use the techniques of chapter 3 for this task. Second, to cope with remaining fine-scale misalignments due to registration errors, we estimate realignment warps in the texture domain. Because they encode low-magnitude, residual geometric variations, they are also advantageously decomposed using PCA, yielding Eigen warps. The full appearance information of all subject sequences can then be compactly stored as linear combinations of Eigen textures and Eigen warps. Our strategy can be seen as a generalization of the popular work of Nishino et. al. [START_REF] Ko Nishino | Eigen-texture method : Appearance compression and synthesis based on a 3d model[END_REF], which introduces Eigen textures to encode appearance variations of a static object under varying viewing conditions, to the case of fully dynamic subjects with several viewpoints and motions.

The pipeline is shown to yield effective estimation performance. In addition, the learned texture and warp manifolds allow for efficient generalizations, such as texture interpolations to generate new unobserved content from blended input sequences, or completions to cope with missing observations due to e.g. occlusions.

Method

To eliminate the main geometric non-linearity, we first align sequence geometries to a single template shape and extract the texture maps of a subject over different motion sequences in a common texture space using the previously described method in §4.2 [START_REF] Tsiminaki | High Resolution 3D Shape Texture from Multiple Videos[END_REF]. From these subject specific textures, Eigen textures and Eigen warps that span the appearance space are estimated. The main steps of the method below are depicted in Fig. 4.5.

1. Texture deformation fields that map input textures to, and from, their aligned versions are estimated using optical flows. Given the deformation fields, Poisson reconstruction is used to warp textures.

2. PCA is applied to the aligned maps and to the texture warps to generate the Eigen textures and the Eigen warps that encode the appearance variations due to, respectively, viewpoint, illumination, and geometric inaccuracies in the reference model.

Hence, The main modes of variation of aligned textures and deformation fields, namely Eigen textures and Eigen warps respectively, span the appearance space in our representation.

Note that due to texture space discretization, the warps between textures are not one-to-one and, in practice, two separate sets of warps are estimated. Forward warps map the original texture maps to the reference map. Backward warps map the aligned texture maps back to the corresponding input textures (see Fig. 4.5).

Given the Eigen textures and the Eigen warps, and as shown in Fig. 4.6, a texture can be generated by first creating an aligned texture by linearly combining Eigen textures and second de-aligning this new texture using another linear combination of the Eigen warps.

Results

We evaluated the method on a number of Kinovis acquired datasets and measure the quality using the SSIM metric [START_REF] Wang | Simoncelli, Image quality assessment : From error visibility to structural similarity[END_REF] that is more tolerant to small shifts, both in image space after reprojection and texture space before reprojection by comparing it to the short term maps estimated with §4.2 [START_REF] Tsiminaki | High Resolution 3D Shape Texture from Multiple Videos[END_REF]. The sequences are previously tracked using the method from §3.2.3 with a single template that is used as support for the common texture space. The results show that our strategy successfully encodes 2048x2048 datasets of 200 or 300 frames with virtually no error (0.98 SSIM) with 50 PCA components. We also show that using an equivalent number of parameters, a simple baseline PCA strategy without eigen warps achieves much lower performance, substantiating that the eigen warps successfully correct small geometry induced texture slippage. We also show the applicability of our method for two tasks : interpolation between two template poses, by interpolating in both eigen spaces (warp and texture) and texture completion to correct artifacts in estimated textures e.g. due to poor visibility, by projecting the textures to a pre-established space of our eigen representation, in both cases with encouraging results as illustrated in figures

CHAPITRE 5 Conclusion

In this chapter, we give a short recap of the contributions, discuss some insights, and mention some of the interesting followups related to this work that are happening in the team or that are envisioned as future work.

SUMMARY AND INSIGHTS

This manuscript presents some of the prominent works with my students, for PhDs defended in the last five years. With the general goal of advancing 4D modeling to digitize captured subjects, in the form of high quality time evolving representations estimated from multi-viewpoint video, we advanced the field in the following directions.

In chapter 2, we first examined estimating 3D shapes in the static case, with several simultaneous images inputs from one or several cameras. We have crafted a classic MVS pipeline tuned for performance capture applications, but also successful with more general data, then successfully demonstrated the benefit of learning the photoconsistency function and substituting it in this pipeline. We then extended our experience of learning-based approaches to end-to end monocular 3D models estimations, which is currently the basis for new works in the team for monocular sequence analysis and new multi-view reconstruction algorithms. All contributions presented became state of the art in reconstruction quality at the time of their publication and to this day serve as input to subsequent methods explored in the team. Some main limitations are the relatively slow training and processing times for e.g. Vincent Leroy's algorithms, due to the very fine level of detail targeted and particular training procedure, and this also drives are future research to improve efficiency.

In chapter 3, we have presented several principled motion models for 3D sequence tracking and alignment, using surface and volume rigidity priors, and volume support for improved data-driven shape matching, with state of the art results at the time of publication. We also presented human-prior driven tracking under wide clothing. They are all grounded in the previous extraction of 3D models per time step discussed in the previous chapter. While these have been great achievements and have definitely contributed to advance understanding of motion models, improving the precision, robustness, in particular breaking the centimetric error barrier in broad acquisition situations, are still prominent challenges. Even better cooperation between the reconstruction, 3D sequence alignment and appearance modeling layers, over longer terms, are probably a key to break this deadlock, as well as better accounting for clothing in our models.

In chapter 4, we presented a principled approach to estimate appearance maps from multi-view sequences over small time windows, treating it as a generalized multicamera superresolution problem. We further show how this estimation can be used a building block to build an Eigen space of varying textures to encode the appearance of a 4D sequence. These contributions were made possible by the advances with the surface reconstruction and alignment previously presented. One of the challenges facing us is to make the superresolution model more practical as it is currently very compute-intensive. We believe finer detail is to be accessed with better cooperation between the geometric and appearance stages as well. This has largely been the basis for new work with PhD student Matthieu Armando on new encodings for appearance on the shape [START_REF] Armando | Adaptive Mesh Texture for Multi-View Appearance Modeling[END_REF], and fine detail correction and restoration on the shape surface [START_REF] Armando | Mesh Denoising with Facet Graph Convolutions[END_REF], for which we are obtaining very encouraging results.

FUTURE DIRECTIONS

The work has brought me to very different fields and was rich with new discoveries, which helped forge some intuitions.

Breaking stratified assumption and representations.

Mostly one stratification approach was examined in this document, however others are possible. For example, some works examine the 3D motion problem and priors in trajectory space [START_REF] Akhter | Nonrigid structure from motion in trajectory space[END_REF][START_REF] Simon | Separable spatiotemporal priors for convex reconstruction of time-varying 3d point clouds[END_REF]. I think there are powerful priors to be learned by examining the problem in such orthogonal directions, not necessarily abandoning the existing geometric priors in fact, but using those in complementary fashion.

Learning offers new paths to consider input data and output of shape, motion and data jointly representation. Graph-convolution [MBM + 17, VBV18] networks offer a promising avenue of research that we have began exploiting in the team, e.g. [START_REF] Armando | Mesh Denoising with Facet Graph Convolutions[END_REF]. Some methods also of great interest to us that redefine shape representation based on implicit networks [CAPM20] e.g. the recent occupancy flow works [START_REF] Niemeyer | Occupancy flow : 4d reconstruction by learning particle dynamics[END_REF], which echo some of our own works to a similar goal [START_REF] Guan | Probabilistic 3d occupancy flow with latent silhouette cues[END_REF], or NERF [MST + 20]. Thsi is driving some of our current explorations, e.g. with PhDs student Mathieu Marsot and Boyao Zhou [ZFB + 20].

FUTURE DIRECTIONS

51

Better priors for motion and clothing.

Modeling clothing is definitely becoming a hot topic, because our research is has hit the precision limits associated to body-only models. New models are increasingly accounting for the actually observed surface layer which sits on top of the other body and flesh layers. We have started examining this to some extent with Jinlong Yang's work, as presented in this document, with one of the team followups focused on predicting a vertex deformation layer of a template based on the underlying human motion parameters [START_REF] Yang | Analyzing Clothing Layer Deformation Statistics of 3D Human Motions[END_REF]. We've personally made a low level-oriented effort in collaboration with Stefanie Wuhrer, and Physical and Structural Mechanics specialist Florence Bertails and Arnaud Lazarus, in co-supervizing PhD student Abdullah Haroon Rasheed, to better understand how cloth simulations could be used in vision systems to build new priors [RRBD + 20].

One can measure the growing popularity of the topic with the number of papers and citations surrounding this work, and seeing interesting new work picking up where we left off with Jinlong Yan's thesis, e.g. [START_REF] Lassner | A generative model for people in clothing[END_REF][START_REF] Pons-Moll | Clo-thCap : Seamless 4D clothing capture and retargeting[END_REF]. And I believe much more can be done in this direction.

From old make new.

It would be quite lame and self-evident to state that Deep Learning will continue reshaping our field, 4D modeling being no exception. But our experience, and the literature shows that one of the key limits fo Deep Learning models, explainability, can be at least partially mitigated by focusing the learning method on well understood sub-problems of time-tested classical pipelines, wit a well identified support domain, which has guided some of our earlier works, notably with the MVS photoconsistency function. While more and more methods are able to look at the whole problem end-toend, I think this is still a valid approach to gain understandings on new idea and new problems.

Many times we can use the classic pipeline as a basis for a data-driven method to build on. This is in some sense what we were doing with our volumetric tracking by detection method in §3.2.4 [HAF + 16] by alternating a data-driven discriminative detection step with a classic generative model-based error minimization. Interestingly, during her post-doc after her thesis with us in Morpheo, Vagia Tsiminaki demonstrated another way to achieve this, by using a network to compute the fine detail residual of appearance maps with respect to the generative appearance map model presented here [RCO + 19]. These are all useful alternative ways to look at our problems.

Another aspect I am noticing and increasingly interested in integrating in my future learning approaches, is that some time time-tested pipeline structuring ideas that have proven to be essential for classic vision algorithms to work in practice and achieve top performance, such as coarse-to-fine processing, are finding their way in the way deep learning pipelines are structured. It is not a coincidence that coarse-to-fine pyramid networks are top scorers in today's benchmarks, such as PWC-net for optical flow [START_REF] Sun | Cnns for optical flow using pyramid, warping, and cost[END_REF] or for the MVS problem [START_REF] Yang | Cost volume pyramid based depth inference for multi-view stereo[END_REF]. This is exciting as it offers new

Introduction

Gathering appearance information of objects through multi-camera observations is a challenging problem, of particular interest for multi-view capture systems. In such systems, typically, a geometric model is reconstructed, tracked or refined to be as close as possible to reality. Adding an appearance or texture layer to this geometric information plays an essential part in the realism of the result, and is often more important than geometric detail to convey the object's visual aspect. Applications of this acquisition pipeline, such as broadcast, special effects or entertainment, among others, are very highly demanding in terms of quality. Yet, even with state of the art multi-camera studio equipment, simply reprojecting texture from any one of the high resolution video streams used in the acquisition process is not enough to guarantee good texture coverage and high quality renderings or close-ups. Because several such input video streams are available in this context, and in order to take advantage of all the information they carry, we naturally turn to the various sources of data redundancy to boost texture quality. Following 2D monocular superresolution techniques that successfully regain details from low resolution images, we consider here a similar framework for multi-viewpoint videos.

Such a framework is however significantly different from 2D super-resolution. First, dealing with multiple video streams is a different problem than using only one, where little parallax is usually assumed to occur. In a multi-view scenario, the intrinsic appearance of a single 3D object is only partially visible in each view, and observed only after being perspective projected, distorted by 3D geometry, and self-occluded. The 3D geometry itself is subject to reconstruction error and thus uncertain. Seamlessly blending and super-resolving the different input contributions into one single coherent texture space, while accounting for all such sources of variability is thus quite challenging. In fact it has only recently started to be addressed as such [10] for static objects.

But to fully exploit data redundancy, temporal accumulation of all views also needs to be examined. Not only is it an additional source of data, but interestingly temporal accumulation might make it possible to obtain high quality results with a sparser set of viewpoints than in the static case. This is not without its own source of difficulties. More often than not the subjects of interest are of arbitrarily deformable nature, such as human actors. This means that consistent temporal accumulation of texture data can only be done by realigning the relevant parts of the texture from one temporal frame to another, and accounting for sources of geometric variability. Fortunately, recent progress in non-rigid surface tracking methods [3] offer a path to resolve such issues, which we open with this work.

Overview. Generalizing existing multi-view appearance super-resolution work to the temporal domain requires a robust model of variability. As the appearance of subjects may drastically change over the long run, we focus in this paper on small non-rigid motions of the subject around a stable pose and observed appearance. We propose to deal with the largest non-rigid motion component using a surfacetracking method [3], and to compensate for any remaining geometry perturbations with a per-view, per-time frame warp. This per-view registration popularized in various rendering techniques [20,8] has the large advantage of uniformly dealing with all sources of error, calibration, reconstruction, temporal misalignments and ghosting for our texture super-resolution, and is one of the major contributions of the paper. This paper is also the first, to our knowledge, to deal both with multiple viewpoints and temporal frames to build one common super-resolved texture, as opposed to [21] which enhance the input views directly, and [10] which only deals with the static multi-view aspect. Warping is done on an intermediate, high-resolution projected proxy of the model texture, where variability can be appropriately densely compensated ( §3.1). We also expose a straightforward model and algorithm for this task, illustrated in Fig. 2. We notably show that some linear models [17] of the image formation can be generalized to the multi-view, multi-frame case ( §3), as well as the monocular noise models ( §4). We exhibit a two-stage iterative algorithm ( §5), whose convergence is illustrated in experiments ( §6). Our validation protocol also includes favorable comparison with the closest state of the art method [10], at the intersection of the validity domains of the methods (static, multi-view texture resolution case). Furthermore, we quantitatively demonstrate the convergence and temporal improvement of our method over using the same number of views in the static case.

Related Work

View-Dependent Texturing. Various strategies exist to retrieve and render the appearance of objects from input views and given a viewpoint, a geometric reconstruction being assumed available in general. One of the first proposed is to reproject and blend view contributions according to visibility and viewpoint-to-surface angle [7]. Viewdependent techniques have been generalized to model and approximate the plenoptic function for the scene object, capturing view dependent shading effects [2] but this requires many dense views. Imperfect proxies and other geometric errors create rendering misalignments (ghosting), which various techniques correct with an additional imagespace registration step [8], building a local basis of appearance variability [5], or refining the geometry proxy [19]. By nature, these methods are not targeted to capture intrinsic, view-independent texture properties and generally do not exploit viewpoint redundancy to super-resolve visual quality, nor do they easily extend to the time domain for deformable objects as proposed.

Multi-View Texture Estimation. To store intrinsic details of the acquired object and later render them, numerous methods build an image-based texture atlas to store appearance information, where each texel blends contributions from each view. Realignment is often proposed again to avoid ghosting [20,15], but a second strategy exists which instead builds the texture as a mosaics of unique-view contributions, whose seam locations are optimized to minimize appearance change between fragments [14]. Interestingly, this strategy was extended to the temporal domain [13]. Only a handful of particularly relevant works examine how to super-resolve fine appearance detail from viewpoint redundancy at a single time frame [12,10]. We propose an improved, unified model to deal with geometric variability due to reconstruction error and small deformation across time for multi-view super-resolution.

Video Super-Resolution While very few works exist concerning super-resolution techniques applied in a multiview context, the problem has been extensively studied in the monocular case. The image formation model is well identified, as a geometric warping, blurring and subsampling process of the initial high-resolution image [1]. Two features of particular interest to us are that this model can be represented by a stack of linear transforms, and that Bayesian models have been developed to explicit the noise dependencies and priors over the target image and estimated warps [9,17]. L1-norm based priors and total variation (TV-)minimization are increasingly popular [17] for their image restoration qualities. Notably, super-resolving multiple videos of a moving subject was examined in a performance capture context, but only for the input viewpoints [21]. Our model proposes temporal and multi-view super-resolution, yet super-resolves a single, intrinsic appearance map which can be re-used to render new viewpoints.

Image Formation Model

Our goal is to estimate an appearance map T of an object of interest from a set of input color images {I t i }, where i ∈ {1, • • • , n i } is the camera number and t ∈ {1, • • • , n t } the time. We assume a temporally coherent mesh model of the object, i.e. whose connectivity is time independent but of varying pose {M t }, obtained by tracking the surface tracking [3].

High Resolution Projection

We project the texture to a high resolution (HR) image {H t i } for each viewpoint {i, t}. Before reaching H t i , the texels of T undergo two geometric transformations.

Texture Mapping. For the appearance to be mapped to the mesh, a geometric mapping function must map each texel of T to the mesh surface. Thanks to fixed connectivity of the mesh across time, only one such function τ needs to be defined. Conformal mappings are preferred, because preservation of angles ensure low distortion during the transfer, such that the texel density of T is kept homogeneous on the 3D surface. Note that due to potential cuts and non-zero genus topology of the objects of interest, τ may not be continuous and may have a support region with several connected components (or charts) in the texture domain. To obtain τ , we use [18], which yields large charts with relatively few components, a useful feature for regularization and to avoid continuity artifacts.

Projection to High Resolution Image. We assume projections {π t i } are known for each view i and time t. A texel at texture location x is mapped to a geometric point τ (x) on model M t , this point is then projected in view {i, t} at point π t i • τ (x). This projection model is intended to provide a high resolution image space to be able to precisely compute a correction warp, which remaps the texture con-tributions with the matching content of input images I t i . In particular we do not model any optical blur here; rather for each HR pixel q we collect all texel contributions projecting within. Because calibration and 3D models are available, we can use GPU z-rendering to filter out non-visible texels [7]. Occasionally the density of projected texels is insufficient (i.e. in high curvature regions of the surface) for a pixel to receive any texture samples. In this situation we assume the underlying surface appearance perceived by this pixel is an interpolation of neighboring texels. For a uniform, continuous treatment of both cases, we combine all texel contributions falling in the vicinity of q by a spatial Gaussian weight with small variance σ 2 p , and normalize to one the sum of texel contribution weights for a pixel q. The continuity of this scheme ensures that no artificial discontinuity is created as a result of a discrepancy in the treatment of these cases. This insures that samples present at the pixel contribute overwhelmingly when present at the center of the pixel, and that the pixel is computed as a weighted sum of texels further away otherwise.

Note that, with this formulation, HR pixels are a linear combination of texels of T. Let P t i be the resulting sparse projection operator such that H t i = P t i T, appropriately collecting the weights previously discussed after being mapped and projected in view {i, t}. Each P t i can be stored as a sparse matrix with w H t i × h H t i rows and w T × h T columns, respectively HR image resolution and the chosen texture resolution.

Inputs as Warped, Downsampled HR Images

To generate an input image from each HR image H t i , the HR image is first warped according to the different apparent sources of variability impacting the input image -calibration error, distortion, model geometry error -using a dense warp field W t i . This warp results in a linear operator over the HR image, which we note F W t i . The image then traverses the optical system, where it is blurred and captured by the CCD which performs light integration at every photosite. Following 2D super-resolution literature [1,9] this is generally modeled using a Point Spread Function (PSF) with the form of a Gaussian blur kernel, followed by an image subsampling stage. Both operations can be written as linear operators, the image-wide blur operator K and subsampling operator S, which are applied to the HR image to obtain a view's observed image I t i = SKH t i . Remarkably, in its noiseless form, the full image formation model can thus be noted as a single, sparse linear operator

A t i = SKF W t i P t i for each view {i, t}, with w I t i × h I t i rows and w T × h T columns, such that I t i = A t i T
for each view {i, t}. This elegantly generalizes the linear formation models used in various 2D super-resolution models [17] to the 3D+t case.

Bayesian Generative Model

The linear model previously discussed describes how input pixels are obtained through warping and blending of texels in noiseless fashion. As in the 2D case, inverting the problem to estimate T and the warps W t i from I t i is illposed, non-convex, and noise ridden [1]. We thus introduce a noise model and priors for better problem conditioning, formulating the solution as a MAP estimation over T, and the warps {W t i } for all views and temporal frames {i, t}:

{ T, { Ŵt i }} = arg max T,{W t i } p(T, {W t i }|{I t i }), (1)
where the posterior is a product of prior and likelihood:

p(T, {W t i }|{I t i }) = p(T) i,t p(W t i ) i,t p(I t i |W t i , T). ( 2
)
Prior Terms. To ensure sparsity of variations of the estimated texture and warp, we impose minimal Total Variation (TV) constraints on appearance image T and each W t i :

p(T) = 1 Z T (λ) e -λ ∇T , (3)
p(W t i ) = 1 Z W (γ) e -ν( ∇u t i + ∇v t i ) , (4)
where ∇ is the gradient operator, ∇T = q ∇T(q) = q ( T x (q) + T y (q) ), the sum over pixel index q of the L 1 -norm over spatial image derivatives of T(q). The same definition holds for u t i and v t i , the x-and y-components of the warp W t i . Z T (λ), Z W (γ) denote the normalization constants of both distributions.

The TV constraint ensures that T is treated as a natural image to be restored with sparse and preserved edges. However, a discontinuity between some neighboring object surface points can be created due to necessary cuts in the mesh unwrapping algorithm, leading some mapped texels to appear in different charts despite their proximity on the surface [13]. For such texels, we carefully compute gradients by computing the transform of axis directions as reprojected in the chart where surface neighbors were mapped [10,11]. This minimizes discontinuities in treatment across chart boundaries in the estimation.

Data Term. Under the assumption that the noise is independent per pixel given the information about the texture, model and cameras, we impose a Gaussian prior for each frame {i, t} :

p(I t i |W t i , T) = 1 Z(D t i ) e -(I t i -A t i T) D t i (I t i -A t i T) , (5)
where D t i is a diagonal covariance matrix introduced to allow different noise characteristics per pixel q, and Z(D t i ) a normalization function of D t i . In 2D super-resolution models, a single variance per input image is usually used, with the i.i.d. noise assumption [9]. However when acquiring appearance in the 3D case it is well known that contributions need to be modulated according to the angle θ q between viewing vector and local surface normal [7]. This can be purposely identified in the generative model, where each diagonal element d(θ q ) of D t i materializes the breadth of the underlying Gaussian predictive model and thus the confidence in the pixel. We set this value as a robust, conservative function of θ q given in §6, which we assume fixed for the purpose of estimation, under small warp perturbations. Note that this is a valid assumption since visibility and grazing angles are generally stable, as we assume given the full poses of the model M t for all frames.

Inference

Directly maximizing all variables in (1) is intrinsically hard and seldom done in the literature. We opt for a coordinate descent scheme, alternating between T and W t i .

Appearance Map. We maximize (1) by minimizing its negative log, dropping all terms independent of T:

T = arg min T i,t (I t i -A t i T) D t i (I t i -A t i T) + λ ∇T , (6)
where the data term develops to a weighted sum of per-pixel L 2 -norms. Although not specifically using a robust data term norm here as opposed to some works, we nevertheless obtain excellent results enforcing robustness through the constant covariance matrix D t i , as will be shown. Optimizing a L 2 data term with a TV-regularizer has been specifically studied [4], yielding a family of forward-backward splitting solvers whose implementation are available offthe-shelf [6]. Let us note f d (T) and f TV (T) the data and the TV-term. Forward-backward splitting is an iterative algorithm for estimating T, alternating between computing a gradient update step and projection prox γ,fTV which computes an implicit subgradient descent step for the TV-norm.

T n+1 = prox γ,fTV (T n -γ∇f d (T n )) , (7)
where γ is a step-size parameter. Our re-weighted functional ( 6) only implies a modification of the gradient update with respect to the standard case, with

∇f d (T n ) = 2A t i D t i (A t i T n -I t i ).
Warp Estimates. We independently estimate each W t i for an input view {i, t}. Minimizing the negative log of (1), and dropping all terms independent of W t i yields:

Ŵt i = arg min W t i ν ∇u t i + ∇v t i (8)
+ (I t i -SKF W t i P t i T) D t i (I t i -SKF W t i P t i T),
which can be interpreted as a modified optical flow equation with a TV-regularizer, where the data term is re-weighted by D t i . The intuition here is that the minimization favors the TV prior of sparse variation over the data term for untrustworthy pixels according to D t i , and puts more relative importance on trying to follow data on reliable pixels. We opt for a similar strategy to [17] for solving this equation, and initialize the estimation of W t i with the result of a standard optical flow method [16], applied between H t i and an upsampled I t i at each iteration.

Experiments

We exhibit results with a MATLAB prototype implementation, and run experiments on a 16-core 2.4 GHz PC with 32GB RAM 1 . Our current implementation is mainly mono-thread, with the exception of the optical flow which we launch in 10 separate threads. To initialize the algorithm, we first use a small C++/OpenGL program to render visibility maps from texture to image space, then initialize the texture map with a simple weighted average of visible inputs. The visibility maps are also used to generate each projection matrix operator P t i . We use the Optical Flow package from Liu et al. [16] for per-iteration optical flow initialization, and the UNLocBOX package [6] for the texture re-estimation in the loop. We use a threshold on the relative norm of the objective function ( 6) as stopping criterion, and observe convergence in 30 to 70 iterations for a given λ. The execution time of the algorithm is in the range of 30 minutes to an hour per iteration depending on the dataset, number of views and number of frames. These are not a good indication of the final achievable performance as many enhancements are possible, including making the 1 See video results at http://hal.inria.fr/hal-00977755 flow and image update estimations massively parallel on a GPU, better inter-time flow bootstraps as suggested by [17], more compact data-structures, C++ inner loop.

Parameter values. We set the Gaussian variances with σ p = 0.25 and σ k = 0.1, respectively for the projection weight and PSF kernel K, for all datasets. Although these could be optimized alongside other parameters, we observe low sensitivity to these parameters when set in the [0.1, 1] range. Higher values introduce over-blurring, while lower values tend to reveal the underlying discretization of the texture map (σ p < 0.1) or the input image (σ k < 0.1). We also fix the convergence parameters to γ = 0.05 and λ = 5.10 -4 for all experiments, using a second and third round of iterations with λ = 5.10 -5 and λ = 5.10 -6 to down-weigh TV-regularization and thus reveal higher frequency detail. We set d(θ q ) = 1 C e -s tan θ q as a faster approximation of a normal distribution over the angles of the perceived surface, and use C to normalize these weight contributions to 1 among all pixels that see a common texel x to obtain homogeneous weights among pixels in the data term i,q=τ •π t i (x) d(θ q ) = 1. We use s = 7π/16 over all experiments. This weight is more conservative than the cos θ q weight usually used for blending in multi-view texturing techniques [7], and yields improved results in our experiments, as it downgrades unreliable contributions from surface points at a grazing angle.

Static Multi-View Comparison

We compare our model with the latest state of the art multi-view texture super-resolution technique of Goldlücke et al. [10]. As the latter does not deal with temporal sequences, the comparison is performed on the common applicability domain, i.e. static images, as shown Fig. 3. The authors provide a public dataset for three objects BEETHOVEN, BUNNY and BIRD, and kindly provided additional data on request, so we could reproduce the experiment in the closest possible setup. This included a high resolution output of their algorithm for the viewpoint originally reported per dataset in [10], to which we compare our high resolution output. We use the same super-resolution ratio of 3× the input resolution for the texture and high resolution image domains. Respectively 108, 52 and 52 calibrated viewpoints were originally used at resolution 768×576. We have used identical views, and also use identical 3D models except for the BIRD dataset, for which we observed large reconstruction and silhouette reprojection artifacts on the model provided. In fairness we thus only provide crops in regions where the 3D model geometry is not significantly different.

It can be generally observed that our outputs provide lower noise levels and artifacts. This is particularly visible in the BUNNY dataset, in the ear region and shadow region around the left eye. The BEETHOVEN exhibits some visibility difficulties due to the face geometry and presence of concave regions around the nose and hair, which generate artifacts for [10]. In contrast, our method is able to deal with these situations efficiently. A single texture domain cut is present on the nose but the discontinuity is barely visible thanks to the inter-chart terms we introduced. More accurate details and sharper pattern borders can be observed on the BIRD wing and tail, notably in the feather textures.

Temporal Superresolution Validation

To evaluate our approach on the temporal aspect, we introduce three synchronised datasets, GOALKEEPER, BACKPACK and ACTOR, in Fig. 4. The datasets were acquired with three different setups and camera models so as to maximize testing variability. All 3D models were obtained using silhouette-based reconstruction techniques and thus yield largely imperfect models. GOALKEEPER consists of 21 calibrated viewpoints at 1024 × 1024, which we downsample to 512 × 512 for the purpose of evaluation. BACKPACK consists of 15 viewpoints of a person, with resolution 1624 × 1224. ACTOR consists of 11 viewpoints in resolution 1920 × 1080. The ACTOR dataset is arguably the most difficult one, with lower views and higher noise lev-els both in the images and the reconstruction. We focus on small motions of the three subjects, and test the method for 2 to 7 frames. Significant improvements can be seen in the figure through temporal accumulation.

There are several difficulties in designing an experiment to quantify this improvement, such as the absence of ground truth data in texture space for real datasets. Synthetic datasets are less than ideal for image restoration and superresolution problems: a significant conclusion can only be achieved if the different sources of variability are correctly introduced and simulated: sensor noise, calibration error, local reconstruction errors, specularities, temporal misalignments. Instead, we focus here on showing the temporal improvement by running our algorithm on a 2× downsampled version of the GOALKEEPER dataset, and comparing our reprojected result with the higher resolution inputs using the mean squared error metric (MSE). Fig. 5 shows the result of this experiment, with convergence curves from one frame (static case) to three frames, and MSE's evaluated on the 21 input views. Several observations can be made from these curves. First, they illustrate convergence of the iterations toward the high resolution ground truth. Second the temporal improvement leveraged by our algorithm is validated in two forms: acceleration of the rate of convergence using more temporal frames, and improvement of the final result quality over using only one temporal frame.

Discussion

We have presented a novel method to retrieve a single, coherent texture from several viewpoints and temporal frames of a deformable subject. The noiseless formation model introduced is linear from texture space to image space, and noise and regularization are achieved using a Bayesian framework. We have demonstrated the usefulness of this approach with respect to state of the art, and quantified the convergence and temporal improvement. The method opens several interesting research possibilities. First, more of the parameters and variability could be automatically learned, such as the projection parameters and regularization weight. The framework proposed enables this, with adapted convergence algorithms. Second, the trade-off between using more views or more temporal frames could be further explored to understand how each modality contributes to the result. Third, longer term resilience could be explored as an extension of this model.   Recent years have seen the emergence of many solutions for the capture of dynamic scenes, where a scene observed by several calibrated cameras is fully reconstructed from acquired videos using multiview stereo algorithms [24,12,1,20]. These techniques have many applications for media content production, interactive systems [2] and scene analysis [28] since they allow to recover both geometric and photometric information of objects' surface, and also their shape and evolution over time. Since these temporal evolutions were initially reconstructed as a sequence of topologically inconsistent 3D models, significant research work has been done for full 4D modeling and analysis of geometrically time-consistent 3D sequences.

On Mean Pose and Variability of 3D Deformable Models

In particular, several techniques propose to deform and match a template to either image data, or to intermediate 3D representations of the surface [25,17,9,26].

These methods allow the recovery of both shape and motion information. However they usually do not consider the intrinsic dynamic properties of a surface. These are either assumed, for instance through a kinematic structure (rigging) or through the surface tension parameters, or are simply ignored. Hence, there is a large interest in better understanding rigidity and motion properties of shapes, with the prospect of improving dynamic models, extracting more useful information, and better automation. In this work, we take the estimation a step further and investigate how to infer dynamics or statistical properties of shapes given temporal sequences.

Recovering this information is yet a largely open research topic with only few exploratory representations proposed for dynamics characteristics of surfaces, e.g. [11,29]. We propose a novel inference framework for the analysis of complex object shapes in motion that learns local surface rigidity probabilities (i.e., deformations), and estimates a mean pose over a temporal sequence. Based on recent advances in surface tracking techniques, we formulate a generative model of 3D temporal sequences using a probabilistic framework, which conditions shape fitting over all frames to a simple set of intrinsic surface rigidity properties. Surface tracking and rigidity variables can then be obtained iteratively using Expectation-Maximization inference by alternatively minimizing two nested fixed point iterations. Thus, our main contribution is a framework that allows the simultaneous tracking and inference of dynamic properties of object surfaces given temporal observations. We show how these properties contribute to a better understanding of surface motion and how they can be used for the dynamic analysis of 3D surface shapes through mean pose estimation and rigidity-based segmentation, while achieving competitive surface tracking.

The remainder of the paper is organized as follows. The next section discusses related work. Details on the mean pose inference model are given in Sect. 3. Section 4 presents various applications and experimental results. Section 5 concludes with a discussion on our contributions.

Related Work

The analysis of deformable surfaces captured by multi-video systems has gained lot of interest during the last decade due to the rapid progression of computer and image sensing technologies. We focus here on works that relate to dynamic properties of shapes. Kinematic structures. Many popular tracking methods propose to rigidly constrain a model using an articulated structure, for instance a skeleton or a cage, which must be scaled and rigged to a 3D template, and optimally positioned through a sequence of models representing the observed subjects [4,30,17,19]. The template is usually deformed using a skinning technique, according to the optimized structure across the sequence [5]. Such kinematic structures provide intrinsic information on the associated shapes through their parameter evolutions (e.g. their averages can define a mean pose). These approaches require a priori knowledge on the observed shapes, such as the topology and the rigid parts, and cannot be applied to arbitrary object shapes. Moreover global template deformation across time is subject to loss of local details such as cloth wrinkles and folds.

Locally rigid structures. The literature also contains several methods that relax the constraint on the shape structure using looser rigidity priors. A body of works consider deformations that preserve local intrinsic surface properties, e.g. isometric deformations [21,8,22,23]. Such properties relate to local rigidities, for instance in [31,32] local surface distorsions are constrained, however they are usually known priors. While efficient to register or match surfaces, intrinsic surface properties are not necessarily sufficient to track complex shapes such as human bodies. In that case, several approaches introduce local deformation models to drive surface evolutions. For instance, in [9], the observed surface is treated as a piece-wise body with locally rigid motions. We consider a similar model to represent surface deformations which is used to learn local rigidities as well as mean poses along with the tracking. Interestingly, recent approaches also in this category were proposed to characterize local surface deformations. In [11], the authors propose a probabilistic framework for rigid tracking and segmentation of dynamic surfaces where the rigid kinematic structure is learned along time sequences. Our framework does not assume such structure but learns instead local rigidities and mean poses. In [29], the authors model complex local deformation dynamics using linear dynamical systems by observing local curvature variations, using a shape index, and perform rigidity-based surface patch classification. The latter approach assumes surface alignment is given, in contrast to our proposed generative model that simultaneously performs surface tracking and local rigidity estimation. Shape Spaces. Following the work of Kendall [18], a number of works consider shape spaces that characterize the configurations of a given set of points, the vertices of a mesh for instance. This has been used in medical imaging to estimate mean shapes through Procrustes analysis, e.g. [16]. In this case, the shape of the object is the geometrical information that remains when the pose (i.e., similarities) is filtered out. Thus Procrustes distances can be used to measure shape similarities and to estimate shape averages with Fréchet means. We follow here a different strategy where a shape space represents the poses of a single shape and where we estimate a mean pose instead of a mean shape. This relates to other works in this category that also consider shapes spaces to model shape poses with mesh representations. They can either be learned, e.g. [3,15] or defined a priori, e.g. [27] and are used to constrain mesh deformations when creating realistic animations [3,27] or estimating shape and poses from images [15]. While sharing similarities in the deformation model we consider, our objective is not only to recover meaningful shape poses but also to measure pose similarities and intrinsic shape properties. Unlike [3,15], we do not need a pose or shape database and the associated hypothesis of its representativeness. Moreover, our methodology specifically addresses robust temporal window integration. We assume given a temporal sequence of 3D reconstructions, incoherent meshes or point clouds, obtained using a multi-view reconstruction approach, e.g. [12,1,20]. We also assume that a template mesh model of the scene is available, e.g. a particular instance within the reconstructed sequence under consideration. The problem of local surface rigidity and mean pose analysis is then tackled through the simultaneous tracking and intrinsic parameter estimation of the template model. We embed intrinsic motion parameters (e.g. rigidities) in the model, which control the motion behavior of the object surface. This implies that the estimation algorithm is necessarily performed over a sub-sequence of frames, as opposed to most existing surface tracking methods which in effect implement tracking through iterated single-frame pose estimation. We first describe in details the geometric model ( §3.1) illustrated with Fig. 1, and its associated average deformation parameterization for the observed surface ( §3.2). Second, we describe how this surface generates noisy measurements with an appropriate Bayesian generative model ( §3.3). We then show how to perform estimation over the sequence through Expectation-Maximization ( §3.4).

Mean Pose Inference Model

Shape Space Parameterization

To express non-rigid deformability of shapes, while de-correlating the resolution of deformation parameters from mesh resolution, we opt for a patch-based parameterization of the surface similar to [9]. The reference mesh is partitioned in an overlapping set of patches, pre-computed by geodesic clustering of vertices. Each patch P k is associated to a rigid transformation T t k ∈ SE(3) at every time t. Each position x k,v of a mesh vertex v as predicted by the transform of P k can then be computed from its template position x 0 v as follows:

x k,v = T k (x 0 v ). (1)
We thus define a pose of the shape space as the set of patch transforms T = {T k } k∈K that express a given mesh deformation. Note here that a pose in the shape space does not necessarily correspond to a proper geometric realization of the reference mesh and, in practice, patch deformations are merged on the template to preserve the mesh consistency.

Mean Pose

To retrieve the mean pose of a given sequence, we provide a definition suitable for the analysis of complex temporal mesh sequences. Following Fréchet's definition of a mean [13], we introduce the mean pose T of a given set of poses {T t } t∈T over the time sequence T as the pose minimizing the sum of squared distances to all poses in the set:

T = arg min T t∈T d 2 (T, T t ), (2)
where d() is a distance that measures the similarity of two poses. This distance should evaluate the non-rigidity of the transformation between two poses of a shape and hence should be independent of any global pose. Such a distance is not easily defined in the non-Euclidean shape space spanned by the rigid motion parameters of the patches. However using the Euclidean embedding provided by the mesh representation, we can define a proper metric based on the vertex positions. Inspired by the deformation energy proposed by Botsch et al. [7] our distance is expressed as an internal deformation energy between two poses. Let T i and T j be two poses of the model, the distance can be written as a sum of per patch pair squared distances:

d 2 (T i , T j ) = (P k ,P l )∈N d 2 kl (T i , T j ), (3)
with d 2 kl (T i , T j ) = v∈P k ∪P l T i k-l (x 0 v ) -T j k-l (x 0 v ) 2 , (4)
where

T i k-l = T i l -1
• T i k is the relative transformation between patches P k and P l for pose i, and N is the set of neighboring patch pairs on the surface. The distance sums, for every pair of patches of the deformable model, its rigid deviation from pose i to j. This deviation is given by the sum over each vertex v belonging to the patch pair, of the discrepancy of relative positions of the vertex as displaced by P k and P l . It can be verified that d 2 defines a distance as it inherits this property from the L 2 norm used between vertices.

Generative Model

The expression (2) is useful to characterize the mean over a set of poses already known. Our goal however is to estimate this mean in the context where such poses are indirectly observed through a set of noisy and sparse 3D point clouds of the surface. Thus we cast the problem as the joint estimation of mean pose and fitting of the model to each set of observations. For our purposes, we assume the set of poses {T t } t∈T are defined for a set T corresponding to observations in a temporal sequence. The observed point clouds are noted Y ={Y t } t∈T , where Y t ={y t o } o∈O t is the set of point coordinates y t o for an observation o among the set of observations O t at time t. Note that this set O t is different than V in general as it is obtained from a 3D reconstruction or depth camera, without any direct correspondence to the deformable shape surface model earlier defined.

To express the noisy predictions of observations, we follow the principle of EM-ICP [14] by introducing a set of assignment variables k t o indicating, for each observation o, which patch this observation is assigned to. We are also interested in retrieving information about the variations of the rigid deformation with respect to the mean shape. To keep this information in its simplest form, we express in the generative model that each pair of patches (k, l) ∈ N is assigned a binary rigidity variable c kl ∈ {0, 1}, which will condition the patch pair to accordingly be rigid or flexible. This variable is an intrinsic parameter attached to the original deformable model and is thus time-independent. We note the full set of rigidity variables C = {c kl } (k,l)∈N . This in turn will allow during inference the estimation of a rigid coupling probability for each patch pair (k, l). We express the generative model through the following joint probability distribution:

p( T, T, Y, C, K, σ) = p( T) t∈T   p(T t | T, C) o∈O t p(y t o | k t o , T t , σ t )   , (5)
p(y t o | k t o T t σ t ) = N (y t o | T t k t o (x 0 v ), σ t ). (6)
Pose constraining model. We constrain the fitted poses to be close to the mean pose, using the distance defined earlier (3). We embed the influence of rigidity variables in this term, by computing two versions of the distance, biased by rigidity variables C:

p(T t | T, C) ∝ exp   - (k,l)∈N d 2 kl ( T, T t , c kl )   , (7)
where

d 2 kl ( T, T t , c kl ) = v∈P k ∪P l β kl (v, c kl ) T i k-l (x 0 v ) -T j k-l (x 0 v ) 2 , (8)
with β kl (v, c kl ) a uniform function over all vertices of the patch pair if c kl = 1, which encourages common rigid behavior of the two patches, and a non-uniform function encouraging more elasticity when c kl = 0:

β kl (v, 0) ∝ exp(- b kl (v) η D ), (9)
where b kl (v) is the distance between the vertex v and the border between P k and P l on the template, D is the average patch diameter and η is a global coefficient controlling the flexibility. The β kl (•, 0) has larger values on the border between the patches, which allows more flexibility while enforcing continuity between the patches. The coefficients β kl (v, 0) are normalized such that

P k ∪P l β kl (v, 0) = P k ∪P l β kl (v, 1
) in order to make both modes as competitive.

Mean model prior. In the absence of any prior, the mean pose is unconstrained and could theoretically have completely loose patches unrelated to each other.

To avoid this and give the mean pose a plausible deformation, we consider the following a prior which expresses that the intrinsic mean pose should not significantly deviate from the original reference pose (represented by the identity transform Id):

p( T) ∝ exp(-d 2 ( T, Id)) ∝ exp   (P k ,P l )∈N v∈P k ∪P l Tk (x 0 v ) -Tl (x 0 v ) 2   , (10)

Expectation-Maximization Inference

We apply Expectation-Maximization [10] to compute Maximum A Posteriori (MAP) estimates of the tracking and average shape parameters given noisy 3D measurements, using the joint probability described in (5) as described in [6]. The assignment variables K and rigidity coupling variables C are treated as latent variables, which we group by the name Z = {K, C}. For the purpose of clarity let us also rename all parameters to estimate as Θ = { T, T, σ}. Expectation-Maximization consists in iteratively maximizing the following auxiliary function Q given the knowledge of the previous parameter estimate Θ m :

Θ m+1 = arg max Θ Q(Θ|Θ m ) = arg max Θ Z p(Z |Y, Θ m ) ln p(Y, Z |Θ). (11)
The E-Step consists in computing the posterior distribution p(Z |Y, Θ m ) of latent variables given observations and the previous estimate. It can be noted given the form of ( 5) that all latent variables are individually independent under this posterior according to the D-separation criterion [6], thus following the factorization of the joint probability distribution:

p(Y, Z |Θ m ) = t∈T   (k,l)∈N p(c kl |Θ m ) o∈O t p(k t o |Y, Θ m )   , (12)
where

p(c kl |Θ m ) = a • exp   - v∈P k ∪P l -d 2 kl (T t,m , Tm , c kl )   (13)
and

p(k t o |Y, Θ m ) = b • N (y t o | T t,m k t o (v), σ t,m ), ( 14
)
where a, b are normalization constants ensuring the respective distributions sum to 1, and v is the closest vertex on patch k. Equations ( 13) and ( 14) are the E-step updates that need to be computed at every iteration for every latent variable. ( 13) corresponds to a reevaluation of probabilities of rigid coupling between patches, based on the previous m-th estimates of temporal and mean poses. ( 14) corresponds to the probability assignment table of time t's observation o to each patch in the model. This corresponds to the soft matching term commonly found in EM-ICP methods [14].

The M-Step maximizes expression (11), which can be shown to factorize similarly to ( 5) and ( 12), in a sum of three maximizable independent groups of terms, leading to the following updates:

T t,m+1 = arg min T t (k,l)∈N c kl p(c kl |Θ m ) d 2 kl ( Tm , T t , c kl ) (15)
+ o∈O t k t o p(k t o |Y, Θ m ) y t o -T t k t o (x t v ) 2 , σ t,m+1 2 = 1 3 o∈O t k t o p(k t o |Y, Θ m ) y t o -T t,m+1 k t o (x t v ) 2 o∈O t k t o p(k t o |Y, Θ m ) , (16)
Tm+1 = arg min T d 2 ( T, Id) + t∈T (k,l)∈N c kl p(c kl |Θ m )d 2 kl ( T, T t,m+1 , c kl ). (17)
Expression (15) corresponds to simultaneous updates of all patch transformations for a given time t, weighed by E-step probabilities. ( 16) updates the pertime frame noise parameter with an E-step weighed contribution of each observation. ( 17) computes the mean pose, accounting for all poses in the time sequence. Note that, for ease of resolution, we decouple the estimation of T t,m+1 and Tm+1 , which is why (17) uses the result T t,m+1 . We solve both systems with Gauss-Newton iterations, using a parametrization of the rigid transforms as a rotation matrix and translation.

Experiments

We evaluate the proposed generative model using 3D sequences reconstructed from real human performances captured by multiple view videos. We propose two datasets, GoalKeeper and Dancer, which provide two different actions and clothing situations with high resolution inputs. These were processed by extracting visual hull reconstructions, and two neutral topology frames were selected to provide the template model after smoothing and simplifying the obtained mesh down to 5k vertices. Additionally, we also validate using two public datasets made available by the community. The Free [25] dataset consists of a photocoherent mesh sequence of a dancer with approximately 135k vertices per frame, exhibiting particularly fast and difficult dancing motion. The Marker dataset [19] provides another type of challenging situation with a two-person sequence of reconstructions, with martial art motions. It also provides markers on one of the persons which we will use for quantitative evaluation. For both these public sequences, we use the templates provided downsampled to 5k vertices. In all visualizations, we render mesh poses by computing vertex position x t v at time t as a linear blend of positions x t k of expression (1), weighed by a set of Gaussian weights α k (v) materializing the region of influence of patch P k on the mesh. These weights are maximal at the center of mass of P k and their sum over all non-zero patch influences are normalized to 1 for a given vertex v:

x t v = k α k (v) x t k . (18)
We visualize the rigidity coupling probabilities over the surface with heat-colored probabilities, by diffusing this probability over vertices of influence of patch pairs to obtain a smooth rendering. We provide a supplemental video3 with the processed results for these datasets.

Tracking Evaluation

We first evaluate the tracking performance of the algorithm. Full sequences may be processed but because of the motion of subjects in the sequence, all poses of the sequence cannot be initialized with a single static pose, as this would surely be susceptible to local minima. We thus process the four datasets using a sliding window strategy for T , where processing starts with a single pose, then additional poses are introduced in the time window after the previous window converges. We provide tracking results with sliding window size 20 which corresponds to approximately one second of video. We show the resulting poses estimated by our algorithm on the four datasets in Fig. 4, Fig. 5a and Fig. 5b.

Runtime is approximately 15 seconds per time step on a recent workstation and can be further improved.

We also provide a comparison with state of the art methods Liu et al. [19] and with a purely patch-based strategy [9], on the Marker dataset. We reproduce [9] results by neutralizing mean updates and rigid coupling updates from our method, which corresponds to removing these terms from the energy and closely mimics [9]. Note that [19] is a kinematic tracking strategy, where both subjects are rigged to a kinematic skeleton providing a strong, fixed and dataset specific rigidity prior. On the other hand, [9] only use patch rigidity and inter-patch elasticity priors, that are weaker than [19] and our method. The Marker dataset provides sparse marker positions, at which we estimate geometric positional error with respect to the surface. To this purpose we match the closest vertex on the template model provided, and follow it with the different methods, computing geometric errors in position with respect to the corresponding marker's position in these frames. The average errors are shown in Table 1. We also provide a temporal error graph for our method and [9] in Fig. 2. Table 1 shows our method achieves comparable tracking performance to state of the art surface tracking techniques. The slightly higher error with respect to [19] is not unexpected given that they use a stronger kinematic skeleton prior. Regarding [9], the graph and table show a small advantage in error for our method along the sequence, as well as a smaller variance of the error, showing the better constraining provided by our framework. The graph also shows significantly higher error values with [9] than with our method around frames 60, 250, 325, 390 and 460. These error peaks are imputable to difficult segments of the input sequence where [9] loses track of limbs (see Fig. 3a and Fig. 3b) while our method does not. The high error values around frame 390 are due to ambiguous input meshes where the head of the second character (not seen in Fig. 3b) is out of the field of view. Around this frame, our method still outperforms [9] which misaligns an arm (see Fig. 3b). These results substantiate stronger robustness for our method over [9].

Regarding limitations, the model may fall into local minima when the noise level of inputs is too high similarly to all patch-based methods but this was not a strong limitation on the datasets. As the model favours rigidity and isometric surface deformations, the surface sometimes overfolds in non-rigid sections (as sometimes seen in video), which we will address in future work. Fig. 3. Input mesh (left), tracked mesh with [9] (middle) and with our method (right).

Mean Pose and Rigidity Estimation

Fig. 5a shows tracking results with color coded rigidity coupling probabilities with sliding window size 20. The method accurately reports instantaneous rigidity deviation, such as when the subject folds his elbows or shoulders. Blue regions correspond to regions of the mesh that have no non-rigid distortion with respect to the estimated mean pose. Fig. 5b shows estimates of mean poses for full sequences, colored with the estimated rigidity coupling probabilities over full sequence (no sliding window). It can be noted that the method accurately reports where the most common deviations occur. The supplemental video shows mean pose sequences for several sliding window sizes. We observe a temporal smoothing of the initial deformation: fast deformation is filtered out. This effect is stronger with wide windows. We interpret this phenomenon as follows: when the temporal window slides along the sequence, it produces a mean pose sequence analogous to the convolution of the estimated pose sequence with a gate function, with the same size as the window size. This process can be seen as a low-pass filtering of the sequence poses.

We also observe that the mean pose is not affected by global rigid motion of the shape (noticeable with the Dancer dataset). This is an expected consequence of using a pose distance that is invariant under global rigid transforms in (2).

Conclusions

We present a novel methodology for the analysis of complex object shapes in motion observed by multiple cameras. In particular, we propose a generative model of 3D temporal sequences using a probabilistic framework that simultaneously learns local surface rigidity probabilities and estimates a mean pose over temporal sequence. Hence, rigidity-based surface segmentation can be achieved using local deformation properties, while motion synthesis or surface alignment for compression or morphing applications can be achieved using a mean pose as a sequence keyframe or a cluster prototype.

Our model can also perform surface tracking with state of the art performance, and does not require a priori rigid (kinematic) structure, nor prior model learning from a database. Surface tracking and rigidity variable probabilities are obtained by solving an Expectation-Maximization inference problem which alternatively minimizes two nested fixed point iterations.

To our knowledge, this is the first model that achieves simultaneous estimation of mean pose, local rigidity, and surface tracking. Experimental results on real datasets show the numerous potential applications of the proposed framework for complex shape analysis of 3D sequences.

A. 3

Edmond Boyer

Abstract

Recovering 3D shape motion using visual information is an important problem with many applications in computer vision and computer graphics, among other domains. Most existing approaches rely on surface-based strategies, where surface models are fit to visual surface observations. While numerically plausible, this paradigm ignores the fact that the observed surfaces often delimit volumetric shapes, for which deformations are constrained by the volume inside the shape. Consequently, surface-based strategies can fail when the observations define several feasible surfaces, whereas volumetric considerations are more restrictive with respect to the admissible solutions. In this work, we investigate a novel volumetric shape parametrization to track shapes over temporal sequences. In constrast to Eulerian grid discretizations of the observation space, such as voxels, we consider general shape tesselations yielding more convenient cell decompositions, in particular the Centroidal Voronoi Tesselation. With this shape representation, we devise a tracking method that exploits volumetric information, both for the data term evaluating observation conformity, and for expressing deformation constraints that enforce prior assumptions on motion. Experiments on several datasets demonstrate similar or improved precisions over state-of-the-art methods, as well as improved robustness, a critical issue when tracking sequentially over time frames.

Introduction

The capture of shapes and their evolutions has been a very active research topic for the last decade, motivated by many applications for which dynamic shape models are useful. This ability is of interest for several fields of activity such as computer-assisted design, virtual reality, entertainment, medical imaging, gesture and sports analysis. Ever since the initial promises of free viewpoint video [9], many models of shape capture have been explored. Initially examined as a per-time reconstruction problem, e.g. [24,14], temporal integration and tracking of the shape in the time domain have then been considered, e.g. [11,3]. In any case, however, surface-based models, such as meshes, have been largely dominant to represent and track shapes. This is due to several factors, primarily to the fact that visual observations generally lie on the shape surface, but also to the popularity of surface-based representations in the vision and graphics communities and the availability of efficient tools to manipulate them. Yet it has been observed that certain forms of volume-preserving deformations may be beneficial to model shape deformations in graphics applications such as [1,5], or to enforce volumetric constraints, nevertheless based on surface tesselations, in dynamic shape capture [10].

While the idea has led to interesting preliminary results, a full volumetric treatment of dynamic shape capture is still to be investigated and its benefits evaluated. Among the expected advantages of this approach are its ability to express volume conservation as well as its ability to enforce local volumetric deformation constraints. In this paper, we address this problem with a twofold contribution: we first propose a dedicated volumetric deformation model based on Centroidal Voronoi Tesselations (CVT) [13], which integrates the latest advances of recent tracking models, and second we propose an evaluation of the method based on a hybrid multi-camera and marker-based capture dataset [21].

Previous Work

A large set of techniques exist to capture moving shapes as a time independent sequence of meshes representing the object's surface [24,14]. For this process, many volumetric parameterizations have also been devised, based on regular or hierarchical Eulerian grid discretizations [30,20], although they are mainly dedicated to single time occupancy representation. Some approaches have taken these representations a step further, by examining short term motion characteristics of the shape using regular volume grids [33,17,32], yet they do not retrieve long term motion information of the sequence, nor do they embed spe-1 cific motion models in the volume.

Various methods attempt leveraging time consistency to retrieve temporally coherent shape models, in the vast majority of cases manipulating a surface model. While in some cases this process is purely data-driven, by aligning surfaces across frames using sparse matching and stereo refinement [29], in most cases a deformation prior is used to drive the method toward the solution within a plausible state space. In its weakest form and without further assumptions, pure spatio-temporal continuity of the observed surface can be used [16]. At the other end of the spectrum a full kinematic rigging of a template model can be assumed, where the surface is expressed from kinematic parameters using e.g. the linear blend skinning deformation framework [23] popularized for 3D animation in the computer graphics community. These parameters can then be estimated for best fitting the model reprojections to image and silhouette data [34,3,18,21]. For tracking more general subjects and situations, more generic surface deformation frameworks have been explored to bypass the rigging stage and allow for more general non-rigid motion components. Central to these methods is the idea of enforcing a cohesive behavior of the surface, such as locally rigid behavior [15], Laplacian deformation [11,10,6], inextensibility [25], or elasticity between piecewise-rigid surface patches [7,6].

Among the existing surface capture methods, only a handful use volumetric representations. Some methods have proposed reparameterizing temporally aligned sequences using a volumetric cage embedding [26,31] inspired from the animation community [27,19]. However, no volume deformation model strong enough to solve the full tracking problem has yet emerged from these works. Among the methods successfully using volume preserving constraints, most use a Delaunay tetrahedrization of reconstructed template surface points [11,10,6] to enforce asrigid-as-possible or Laplacian deformation constraints common to 3D animation techniques [1,28]. It can be noted that the proposed decomposition is not fully volumetric as it only involves tesselating surfaces. In contrast, we propose a fully volumetric treatment of the problem, using an intrinsically volumetric tesselation, deformation model and data terms for rewarding volume alignment.

Approach Overview

We formulate the tracking problem as the MAP estimation of multiple poses of a given geometric template model, non-rigidly adjusted to a set of temporally inconsistent shape measurements. In multi-view camera systems, these measurements typically take the form of time independent 3D mesh reconstructions obtained from a visual hull or multi-view stereo method, which is what we assume here. To efficiently make use of volumetric information, we need to express volume conservation and overlapping constraints from the template to the observed shape volumes. For representational and computational efficiency, we thus need a proper discretization of the interior of the shape. While uniformly located in the volume, regular grids are inherently anisotropic and biased toward the axis of the template basis. Furthermore, their intersection with the object surface yields boundary voxels of irregular shape (Fig. 1(a)). On the other hand, the Constrained Delaunay tetrahedrization of the boundary vertices, previously used in [11,10,6], yields a set of highly non-uniform tetrahedra spanning the whole interior of the volume, whose cardinality is not controlled but imposed by the surface discretization (Fig. 1(b)). Taking instead the Voronoi diagram of a uniform set of samples of the interior volume decorrelates the cardinality of the decomposition from the geometry, but still yields cells of irregular shape (Fig. 1(c)). Rejection sampling may statistically impose additional regularity, but this would only come with asymptotic guaranties attainable at high computational cost. We therefore propose to use CVT (Fig. 1(d)), informally a Voronoi tesselation where the samples are iteratively repositioned to coincide with the center of mass of their cell, which achieves the desired properties [13]: isotropy, rotational invariance, uniform cells of compact and regular form factor, regular intersection of boundary cells and surface, independent cardinality and practical computation.

After introducing how to define and compute CVTs in the context of our approach ( §2), we show how this discretization can be used to define adequate volumetric deformation ( §3) and observation ( §4) models in the form of Bayesian prior and likelihoods. The MAP estimation proposed on this basis in §5 is evaluated in §6.

Centroidal Voronoi Tessellation (CVT)

Definitions. To tesselate the shape, we manipulate Voronoi diagrams that are restricted, or clipped, to its inner volume. More formally, let S be a set of 3D point samples of a volumetric domain ⌦, either the template to be fitted or the observed shape for our purposes. The Clipped Voronoi diagram of S in ⌦ is defined as the intersection of the Voronoi diagram of S in R 3 with the domain ⌦. Thus the Voronoi cell ⌦ s of a sample s is the set of points from ⌦ that are closer to s than to any other sample:

⌦ s = {x 2 ⌦ | 8s 0 2 S \{s} kx x s k < kx x s 0 k} , (1)
where cells ⌦ s are mutually exclusive and define a partition of ⌦:

[ s2S ⌦ s = ⌦ , (2)
where ⌦ s and ⌦ denote topolgical set closures. If the border @⌦ of ⌦ is a polyhedral surface, then each cell also has a polyhedral border. A Centroidal Voronoi tessellation is a clipped Voronoi tessellation of ⌦ for which each sample s is the center of mass of its (clipped) Voronoi cell ⌦ s . CVT cells are of regular size and shapes, and also define a regular connectivity of the samples set, two samples being connected if and only if their respective CVT cells share a face. This connectivity thus encodes the shape volume and topology, a property we will use in the following sections.

Computing a CVT. It has been proven [13] that local minima of the energy

E(S) = X s2S Z x2⌦ s kx x t s k 2 dV (3)
define CVTs on ⌦. Thus a CVT can be obtained by iteratively estimating the sample locations that minimize (3), with a quasi-Newton method such as the L-BFGS algorithm [22], for a sample population of desired size and uniform initial position.

Deformation Model

Principle

Once a regular, anisotropic volumetric decomposition of the template shape is obtained, we can use it as a fundamental building block to build a volumetric deformation model of the shape, which will constrain the estimation. Botsch et al. [5] show that a non-linear elastic deformation energy can be devised using small volumetric deformations, typically voxels. While such a reasoning could be advantageously transposed to the CVT discretization proposed, eliminating the grid orientation bias, doing so comes at a high computational cost. Cagniart et al. [7] show that the complexity of the deformation model is best decorrelated from the geometry itself, in their case by using rigid surface patches in lieu of the original surface vertices. Recent works have shown a way to improve the quality and temporal stability using a similar surface decomposition [2], by inferring a mean pose and sequence rigidity behaviour.

We extend the latter ideas to the case of a complete volumetric treatment of the deformation problem. In so doing, we cluster together groups of CVT cells in rigid volume patches P k using a k-medoids algorithm. Note that such patches can lie either on the surface or completely inside the template shape's volume, which is of particular interest to express non-rigid deformation of the model while preserving the local volume and averting over-compression or dilation. We associate to each patch a rigid transform T t k 2 SE(3) at every time t. Each position x k,q of a mesh vertex or inner sample is indiscriminately labeled as a point q. Its position can be written as a transformed version of its template position x 0 q as follows, once the rigid transform of its patch is applied:

x k,q = T k (x 0 q ). (4)
This makes it possible to define a pose of the shape as the the set of patch transforms T = {T k } k2K , which expresses a given volumetric shape deformation.

Formulation

To prevent patch poses of the shape from being arbitrary, we constrain the shape to be close to a sequence rest pose T and to follow constant rigidity characteristics C over the sequence. These rigid characteristics are defined for neighboring patch pairs in the volume (P k , P l ), as a binary valued property c kl , whose value in {0, 1} reflects wether the relative motion between patches P k and P l is respectively articulated or rigid. To define the rest pose T we rely on the following measure [5,2] of the relative deformation energy between two arbitrary poses T i and T j of the template shape, given a rigidity configuration C:

E (T i , T j |C) = X (P k ,P l )2N E kl (T i , T j |c kl ), with (5)
E kl (T i ,T j |c kl ) = X q2P k [P l kl (q,c kl )kT i k l (x 0 q ) T j k l (x 0 q )k 2 ,
where

T i k l = T i l 1
T i k is the relative transformation between patches P k and P l for pose i, and N is the set of neighboring patch pairs on the surface. The energy measures the rigid deviation from pose i to j of every neighboring patch pair, as the sum over each of the samples s of the pair, of the discrepancy in relative positions of the vertex as displaced by P k on one hand, and P l on the other. If the two patches are rigidly linked (c kl = 1), then the discrepancy of all samples of the pair should be equally penalized, therefore kl (s, 1) is chosen to be constant over all samples s of the pair. On the other hand, if the patch pair is articulated (c kl = 0), only samples that lie near the boundary between the two patch volumes should be penalized for deviating relative positions: those samples materialize the locus of the inter-patch articulation, whereas samples that aren't close the inter-patch boundary can move more freely. We express this using kl (q, 0) / exp( b kl (s) ⌘ D ) where b kl (s) is the distance between the sample s and the boundary between P k and P l on the template, D is the average patch diameter and ⌘ is a global coefficient controlling the flexibility.

Resulting Pose Likelihoods

The relative pose energy described in (5) makes it possible to express the expected behavior of the estimated models as a prior and likelihood over the poses:

p( T) / exp( E ( T, Id)), (6)
p(T t | T, C) / exp E (T t , T|C) . (7)
p( T) is the prior over the rest pose, which should minimize the relative displacement energy to the default template pose (transformed by identity Id). This terms ensures minimal cohesion of the volume patches of the rest pose model, as it enforces mutual patch elasticity. p(T t | T, C) is the likelihood of a given tracked pose at time t, which should minimize the relative displacement energy with respect to the sequence rest pose T given a current rigidity state C. This ensures the inter-patch cohesion of pose T t as well as a general proximity to the rest pose, which stabilizes the resulting pose estimates. In turn the rest pose will be simultaneously estimated as the pose which minimizes the relative deformation energy to all poses in the sequence.

Observation Model

Probabilistic Shape Fitting

The observed shape ⌦ t at time t is described by the point cloud Y t = {y t o } o2O t . To describe how a deformed template can explain the observed shape, we propose a generative data term following EM-ICP, expressing how a given deformed template point predicts the position of an observed shape point o. A set of cluster association variables k t o is therefore instantiated for every observed point in time, indicating which cluster generates this point. For simplicity, each observation o is associated to its cluster k t o via the best candidate q of cluster k t o . The best candidate is chosen as the closest compatible sample in the cluster during iterative resolution. We consider that each cluster P k generates observations perturbed by a Gaussian noise with isotropic variance 2 :

p(y t o | k t o , T t k , ) = N (y t o | T t k (x 0 q ), ). (8)
Note that o indiscriminately refers to surface or volume sample points of the observed shape, as the principles we describe here apply to both, with the restriction that observed surface points only associate to surface template points, and volume samples are associated to volume samples of the template. As often proposed in ICP methods, we additionally filter associations using a compatibility test, described in the following sections. The compatibility test is specific to the nature (surface or volume) of the observed point and is detailed in the next paragraphs. If there is no compatible candidate in the cluster, then we set conditional probability density (8) to zero. We deal with outliers by introducing an outlier class among values of k, which generate observations with a uniform probability density over the scene.

Compatibility Tests

Compatibility tests are useful for pruning the association graph for obvious mismatches that would perturb and otherwise slow down convergence. We use two compatibilty tests respectively designed for surface fitting and volumetric fitting.

Surface Observations. While surface points may be matched based on position only, obvious orientation incompatibilites can be filtered by detecting large discrepancies between the normal of the deformed template candidate point v, and the normal of surface observation vertex o:

ñt o . R t k (ñ 0 v ) cos(✓ max ), (9)
where ñt o is the surface normal of observation o, ñ0 v is the surface normal of the template at vertex v, R t k is the rotation component of T t k , and ✓ max is an arbitrary threshold.

Volume Observations. We introduce a compatibility test specific to volumetric fitting, by assuming that the distance of inner surface points to the shape's surface remains approximately constant under deformation. Let us define the distance between an inner shape point x and the shape's surface by:

d(x, @⌦) = min p2@⌦ d(x, p). (10)
In our observation model, this hypothesis can be leveraged by the following compatibility test: a volumetric observation o can be associated to a template point s only if

d(x 0 s , @⌦ 0 ) = d(y t o , @⌦ t ). (11)
To account for small deviations to this assumption, which might occur under e.g. slight compression or dilation of the perceived shape, we relax the equality constraint up to a precision ✏, where ✏ accounts for the distance-to-surface inconsistency caused by the discrete sampling of the template. Using the triangular inequality, it can be shown that this error is bounded by the maximum cell radius over the set of the template's CVT cells. This leads to the following compatibility test:

d(y t o , @⌦ t ) ✏  d(x 0 s , @⌦ 0 )  d(y t o , @⌦ t ) + ✏ (12)
For the particular case of silhouette-base observed shapes, it can be noted that reconstruction algorithms based on the visual hull inherently provides inflated estimates of the true shape. This phenomenon results in an overestimation of the distance to the surface when computed on the reconstructed shape. Hence, we only impose a volumetric inclusion constraint instead of complete depth correspondance, i.e. we only keep the right inequality from expression (12) in this case:

d(x 0 s , @⌦ 0 )  d(y t o , @⌦ t ) + ✏. ( 13
)
Contrary to the surface compatibility test, this test does not depend on pose parameters T, consequently it is robust to convergence failures of inference algorithms.

Inference

The model proposed with ( 6), ( 7) and ( 8), defines a joint likelihood over the rest pose, the rigidity configuration, all temporal poses, the observed points and their selection variables, and prediction noise :

p( T) Y t2T 0 @ p(T t | T, C) Y o2O t p(y t o | k t o , T t , t ) 1 A , (14)
It can be shown that this likelihood can be maximized using an Expectation Maximization algorithm [2,12,4], yielding maximum a posteriori estimates of the pose parameters T, T and prediction noise . This results in an algorithm iterating between two steps. Intuitively, The E-Step computes all observation cluster assignment probabilities over K, based on the distance to the predicted template positions under the currently estimated poses. Compatibility rules are applied at this stage. Probabilities over inter-cluster rigid links C are also estimated based on the current deformation energy of the poses. The M-Step updates the rest pose T, all poses T, and prediction noise , using the assignment and rigid link probabilities to weigh individual observation contributions to each cluster transform estimate.

Experiments

Datasets

We validate our framework using four synchronized and calibrated multiple-camera datasets, labeled GOALKEEPER-13, DANCER [2], MARKER [21], and the newly proposed BALLET, whose content reflect a wide variety of shape tracking situations. DANCER is a long sequence (1362 frames, 2048 ⇥ 2048 resolution, 48 viewpoints) showing slow and medium speed dance moves, and thus offers good opportunity to verify the tracking stability. GOALKEEPER-13 (2048⇥2048, 150 frames, 48 viewpoints) illustrates a specific soccer goalie plunging move, of particular interest when the goalie is on the floor, where the reconstruction data is of poor quality due to grazing camera angle and challenges the tracking performance. Both previous sequences otherwise have very high quality and detailed reconstructions. BALLET is a more challenging full HD (1920 ⇥ 1080) sequence we have acquired with fewer cameras (9 viewpoints and 500 frames) and thus coarser reconstructions, consisting in a number of ballet moves with various levels of difficulty, including fast moves, spinning and crossing legs. MARKER (1296 ⇥ 972, 500 frames, 12 viewpoints) is a sequence with two actors performing karate moves, illustrating the robustness to several subjects, and which was captured simultaneously with a set of sparse markers offering a reference and comparison basis method std. dev. (L) MARKER BALLET Cagniart et al. 2010 [8] 3 with [21]. The reconstructions are of coarser quality due to relatively noisy inputs and occasional jumps where actor heads get clipped.

Experimental Protocol

We first select a template among the better frames with correct reconstruction topology, then compute a CVT using §2 with 5'000 samples per person (10'000 for the two-person MARKER sequence) and 250 clusters per person (500 for MARKER), as illustrated in Fig. 3. Each shape reconstruction is obtained from a silhouette-based reconstruction algorithm [14] and CVTs are also extracted (1 minute/frame). We then apply the algorithm described using a sliding window strategy over a 10 frame window, where the rest position is computed for each time slice to locally stabilize the estimation. The sequences are initially solved for frames in the vicinity of the template pose, then the solution is classically propagated to future windows as initialization. Convergence has been achieved for all frames, typically in a few hundred iterations, with a convergence time per frame of the order of a minute to a few minutes. The provided supplemental video 1 illustrates the results obtained with these sequences.

Quantitive Analysis

Volume Stability. We verify here the assertion that the volumetric parameterization of the tracking produces poses with stable volumes. As we use silhouette based reconstructions, it is not relevant to compare the estimated volumes with the observed shape volumes. Instead, we compute the standard deviation to this volume, in Table 1 and provide a comparison of these results with best runs of two state of the art methods [8,2]. This comparison supports the initial intuition of volumetric stability in the sequence, as the standard deviation of the estimated volumes is significantly lower for our method.

Silhouette reprojection error. We evaluate the silhouette reprojection error as the symmetric pixel difference between the reprojected and the silhouette projection of the reconstructions used as observed input shapes. We then express this value as a percentage of the area of the silhouette region in each view. Table 2 shows favorable comparisons to state of the art methods [8,2]. In particular, the mean error and maximum error achieved by our method over the sequences is significantly lower, and exhibits lower variance.

Additionally we test the influence of the volumetric data term by comparing the results with a run where it is disabled (surface data-term only), all other parameters being equal. Interestingly, the method still achieves better mean error than state of the art, but with less stable behavior.

Marker reference error. We use the MARKER sequence provided by Liu et al. [21] to sparsely compare the output quality of our method against state of the art methods. This comparison is illustrated in Table 3 and plotted against time in the sequence in Fig. 2. Again we illustrate turning off the surface data term, in which case the estimation is slightly worse. The method proposed performs consistently better than comparable surface-based state of the art. Note that Liu et al. fully rig a skeleton to the template, which provides slightly better mean results than ours thanks to the stronger assumption. On the other hand, our method is generic and can be applied to arbitrary objects.

Qualitative Assessment

To illustrate the benefits of the approach, in particular where the improvements can be seen, we provide excerpts of the datasets (see supplemental video for further examples). Fig. 4 shows the improvements of the method over surface-based methods [8,2] in poses of strong contortion, such as an elbow or knee bending gesture. Because of their use of the elastic energy on the surface, these methods tend to dilute error compensation over a smooth and extended location of the folds, yielding curvy elbow and knee shapes in the tracking. A usual side effect here is the local decrease of the shape volume in the vicinity of the fold. In contrast, our method being volumetrically constrained, it penalizes such local volume changes and prefers to focus the bending energy on fewer volumetric patch articulations, yielding more consistent and accurate pose estimates. The GOALKEEPER-13 dataset illustrates the increased robustness in the presence of highly perturbed reconstructions thanks to the volumetric constraints, where other methods yield random results. The reconstructed visual hull input is very ambiguous on the shown frame because of the presence of concavities and the strong topology mismatch creates errors for surfacebased methods.

Discussion

We have presented a novel volumetric approach to shape tracking based on CVT volume decomposition. The approach leverages CVT desirable properties to build suitable volumetric deformable constraints, while formulating a discrete volume assignment scheme as data term through the uniform cell centroid coverage of the volume. Currently, the volumetric clustering proposed for volumes yields uniform sizes over the entire template shape, which can be a limitation for parts that are thinner than the cluster size, such as arms. We will address this in future work with adaptive cluster densities, ensuring the volumetric prior is equally efficient regardless of thickness. Numerical analysis nevertheless shows significant improvement over state of the art tracking methods, both in terms of tracking error over the surface and silhouette reprojection. The framework is also shown to conform to initial intuition in being more stable in terms of the errors and volume measures of the fitted template shapes. We believe the approach paves the way for proper use of volumetric priors in any shape tracking framework. A. Inria Grenoble Rhône-Alpes, France Laboratoire Jean Kuntzmann, Université Grenoble Alpes, France {jinlong.yang,jean-sebastien.franco,franck.hetroy,stefanie.wuhrer}inria.fr Abstract. Estimating 3D human body shape in motion from a sequence of unstructured oriented 3D point clouds is important for many applications. We propose the first automatic method to solve this problem that works in the presence of loose clothing. The problem is formulated as an optimization problem that solves for identity and posture parameters in a shape space capturing likely body shape variations. The automation is achieved by leveraging a recent robust pose detection method [1]. To account for clothing, we take advantage of motion cues by encouraging the estimated body shape to be inside the observations. The method is evaluated on a new benchmark containing di↵erent subjects, motions, and clothing styles that allows to quantitatively measure the accuracy of body shape estimates. Furthermore, we compare our results to existing methods that require manual input and demonstrate that results of similar visual quality can be obtained.

Introduction

Estimating 3D human body shape in motion is important for applications ranging from virtual change rooms to security. While it is currently possible to effectively track the surface of the clothing of dressed humans in motion [2] or to accurately track body shape and posture of humans dressed in tight clothing [3], it remains impossible to automatically estimate the 3D body shape in motion for humans captured in loose clothing.

Given an input motion sequence of raw 3D meshes or oriented point clouds (with unknown correspondence information) showing a dressed person, the goal of this work is to estimate the body shape and motion of this person. Existing techniques to solve this problem are either not designed to work in the presence of loose clothing [4,5] or require manual initialization for the pose [6,7], which limits their use in general scenarios. The reason is that wide clothing leads to strong variations of the acquired surface that is challenging to handle automatically. We propose an automatic framework that allows to estimate the human body shape and motion that is robust to the presence of loose clothing.

Existing methods that estimate human body shape based on an input motion sequence of 3D meshes or oriented point clouds use a shape space that models human body shape variations caused by di↵erent identities and postures as prior. Such a prior allows to reduce the search space to likely body shapes and postures. Prior works fall into two lines of work. On the one hand, there are human body shape estimation methods specifically designed to work in the presence of loose clothing [6,7]. These techniques take advantage of the fact that observations of a dressed human in motion provides important cues about the underlying body shape as di↵erent parts of the clothing are close to the body shape in di↵erent frames. However, these methods require manually placed markers to initialize the posture. On the other hand, there are human body shape estimation methods designed to robustly and automatically compute the shape and posture estimate over time [4,5]. However, these methods use strong priors of the true human body shape to track the posture over time and to fit the shape to the input point cloud, and may therefore fail in the presence of loose clothing.

In this work, we combine the advantages of these two lines of work by proposing an automatic framework that is designed for body shape estimation under loose clothing. Like previous works, our method restricts the shape estimate to likely body shapes and postures, as defined by a shape space. We use a shape space that models variations caused by di↵erent identities and variations caused by di↵erent postures as linear factors [8]. This simple model allows for the development of an e cient fitting approach. To develop an automatic method, we employ a robust pose detection method that accounts for di↵erent identities [1] and use the detected pose to guide our model fitting. To account for clothing, we take advantage of motion cues by encouraging the estimated body shape to be located inside the acquired observation at each frame. This constraint, which is expressed as a simple energy that is optimized over all input frames jointly, allows to account for clothing without the need to explicitly detect skin regions on all frames as is the case for previous methods [7,9].

To the best of our knowledge, existing datasets in this research area do not provide 3D sequences of both body shape as ground truth and dressed scans for estimation. Therefore, visual quality is the only evaluation choice. To quantitatively evaluate our framework and allow for future comparisons, we propose the first dataset consisting of synchronized acquisitions of dense unstructured geometric motion data and sparse motion capture data of 6 subjects with 3 clothing styles (tight, layered, wide) under 3 representative motions, where the capture in tight clothing serves as ground truth body shape.

The main contributions of this work are the following.

-An automatic approach to estimate 3D human body shape in motion in the presence of loose clothing.

-A new benchmark consisting of 6 subjects captured in 3 motions and 3 clothing styles each that allows to quantitatively compare human body shape estimates.

Related work

Many works estimate human posture without aiming to estimate body shape, or track a known body shape over time. As our goal is to simultaneously estimate body shape and motion automatically and in the presence of loose clothing, we will focus our discussion on this scenario.

Statistical shape spaces. To model human body shape variations caused by di↵erent identities, postures, and motions, statistical shape spaces are commonly used. These shape spaces represent a single frame of a motion sequence using a low-dimensional parameter space that typically models shape variations caused by di↵erent identities and caused by di↵erent postures using separate sets of parameters. Such shape spaces can be used as prior when the goal is to predict a likely body shape under loose clothing. Anguelov et al. [10] proposed a statistical shape space called SCAPE that combines an identity model computed by performing principal component analysis (PCA) on a population of 3D models in standard posture with a posture model computed by analyzing near-rigid body parts corresponding to bones. This model performs statistics on triangle transformations, which allows to model non-rigid deformations caused by posture changes. Achieving this accuracy requires solving an optimization problem to reconstruct a 3D mesh from its representation in shape space. To improve the accuracy of the SCAPE space, Chen et al. [11] propose to combine the SCAPE model with localized multilinear models for each body part. To model the correlation of the shape changes caused by identity and posture changes, Hasler et al. [12] perform PCA on a rotation-invariant encoding of the model's triangles. These models may be used as priors when estimating human body shape in motion, but none of them allow to e ciently reconstruct a 3D human model from the shape space.

To speed up the reconstruction time from the SCAPE representation, Jain et al. [13] propose a simplified SCAPE model, denoted by S-SCAPE in the following, that computes the body shape by performing PCA on the vertex coordinates of a training set in standard posture and combines this with a linear blend skinning (LBS) to model posture changes. Any posture variations present in the training data cause posture variation to be modeled in identity space, which is known to cause counter-intuitive deformations [8]. To remedy this, recently proposed shape spaces start by normalizing the posture of the training data before performing statistics and model shape changes caused by di↵erent factors such as identity and posture as multilinear factors [14,8,15]. We use the normalized S-SCAPE model [8] in this work; however, any of these shape spaces could be used within our framework.

Recently, Pons-Moll et al. [16] proposed a statistical model that captures fine-scale dynamic shape variation of the naked body shape. We do not model dynamic geometry in this work, as detailed shape changes are typically not observable under loose clothing.

Estimation of static body shape under clothing. To estimate human body shape based on a static acquisition in loose clothing and in arbitrary posture, the following two approaches have been proposed. Balan et al. [9] use a SCAPE model to estimate the body shape under clothing based on a set of calibrated multi-view images. This work is evaluated on a static dataset of di↵erent subjects captured in di↵erent postures and clothing styles. Our evaluation on 3D motion sequences of di↵erent subjects captured in di↵erent motions and clothing styles is inspired by this work. Hasler et al. [17] use a rotation-invariant encoding to estimate the body shape under clothing based on a 3D input scan. While this method leads to accurate results, it cannot easily be extended to motion sequences, as identity and posture parameters are not separated in this encoding.

Both of these methods require manual input for posture initialization. In this work, we propose an automatic method to estimate body shape in motion.

Estimation of body shape in motion. The static techniques have been extended to motion sequences with the help of shape spaces that separate shape changes caused by identity and posture. Several methods have been proposed to fit a SCAPE or S-SCAPE model to Kinect data by fixing the parameters controlling identity over the sequence [4,5]. These methods are not designed to work with clothing, and it is assumed that only tight clothing is present.

Two more recent methods are designed to account for the presence of clothing. The key idea of these methods is to take advantage of temporal motion cues to obtain a better identity estimate than would be possible based on a single frame. Our method also takes advantage of motion cues.

Wuhrer et al. [6] use a shape space that learns local information around each vertex to estimate human body shape for a 3D motion sequence. The final identity estimate is obtained by averaging the identity estimates over all frames. While this shape space leads to results of high quality, the fitting is computationally expensive, as the reconstruction of a 3D model from shape space requires solving an optimization problem. Our method uses a simpler shape space while preserving a similar level of accuracy by using an S-SCAPE model that prenormalizes the training shapes with the help of localized information.

Neophytou and Hilton [7] propose a faster method based on a shape space that models identity and posture as linear factors and learns shape variations on a posture-normalized training database. To constrain the estimate to reliable regions, the method detects areas that are close to the body surface. In contrast, our method constrains the estimate to be located inside the observed clothing at every input frame, which results in an optimization problem that does not require a detection.

Both of these methods require manual input for posture initialization on the first frame. Additionally, a temporal alignment is required by Neophytou and Hilton. Computing temporal alignments is a di cult problem, and manual annotation is tedious when considering larger sets of motion sequences. In contrast, our method is fully automatic and addresses both aspects.

S-SCAPE model

In this work, we use the S-SCAPE model as prior for human body shape changes caused by di↵erent identities and postures. While we choose this shape space, any shape space that models identity and posture as multilinear factors could be used [14,15]. Although such a simple shape space does not accurately model correlated shape changes, such as muscle bulging, it allows to e↵ectively separate the di↵erent variations and can be fitted e ciently to input scans.

This section briefly reviews the S-SCAPE model introduced by Jain et al. [13] that allows to separate the influence of parameters controlling identity and parameters controlling posture of a human body shape. In the following, we denote by and ⇥ the parameter vectors that influence shape changes caused by identity and posture changes, respectively. In this work, we use the publicly available posture-normalized S-SCAPE model [8], where each training shape was normalized with the help of localized coordinates [18].

In the following, let N v denote the number of vertices on the S-SCAPE model, let s ( , ⇥) 2 R 3N v denote the vector containing the vertex coordinates of identity in posture ⇥, and let e s ( , ⇥) 2 R 4N v denote the vector containing the corresponding homogeneous vertex coordinates. For the fixed posture ⇥ 0 that was used to train the identity space, S-SCAPE models the shape change caused by identity using a PCA model as

e s ( , ⇥ 0 ) = e A + e µ, (1)
where e µ 2 R 4N v contains the homogeneous coordinates of the mean body shape, e A 2 R 4N v ⇥d id is the matrix found by PCA, and d id is the dimensionality of the identity shape space. For a fixed identity 0 , S-SCAPE models the shape change caused by posture using LBS as

s i ( 0 , ⇥) = N b X j=1 ! ij T j (⇥) e s i ( 0 , ⇥ 0 ) , (2)
where s i and e s i denote the standard and homogenous coordinate vector of the i-th vertex of s, N b denotes the number of bones used for LBS, T j (⇥) 2 R 3⇥4 denotes the transformation matrix applied to the j-th bone, and ! ij denotes the rigging weight binding the i-th vertex to the j-th bone. Combining Eq. 1 and 2 in matrix notation leads to

s ( , ⇥) = T (⇥) e A + T (⇥) e µ, (3)
where T (⇥) 2 R 3N v ⇥4N v is a sparse matrix containing the per-vertex transformations. Using this notation, it is easy to see that S-SCAPE is linear in both and T (⇥), which allows for a simple optimization w.r.t. and ⇥.

Estimating model parameters for a motion sequence

We start by providing an overview of the proposed method. Fig. 1 shows the di↵erent parts of the algorithm visually. Given as input a trained S-SCAPE model and a motion sequence consisting of N f frames F i represented by triangle meshes with unknown correspondence, we aim to compute a single parameter vector controlling the shape of the identity (as the identity of the person is fixed during motion) along with N f parameter vectors ⇥ i controlling the postures in each frame, such that s i ( , ⇥ i ) is close to F i .

To fit the S-SCAPE model to a single frame F , we aim to minimize

E (F , ,⇥ )=! lnd E lnd (F , ,⇥ )+! data E data (F , ,⇥ )+! cloth E cloth (F , ,⇥ ) (4)
w.r.t. and ⇥ subject to constraints that keep in the learned probability distribution of parameter values. Here, ! lnd , ! data , and ! cloth are weights that trade o↵ the influence of the di↵erent energy terms. The energy E lnd measures the distance between a sparse set of provided landmarks, which correspond to distinctive positions on the human body, to their corresponding locations on s ( , ⇥). The provided landmarks are computed automatically in the following.

The energy E data measures the distance between s ( , ⇥) and F using a nearest neighbor cost. The energy E cloth is designed to account for loose clothing by encouraging s ( , ⇥) to be located inside the observation F .

For a motion sequence of N f frames, our goal is then to minimize

E ✓ F 1:N f , ,⇥ 1:N f ◆ = PN f i=1 E ( F i , ,⇥ i ) (5)
w.r.t. and ⇥ 1:N f subject to constraints that keep in the learned probability distribution of parameter values. Here,

F 1:N f = {F 1 , . . . , F N f } is the set of frames and ⇥ 1:N f = {⇥ 1 , . . . , ⇥ N f } is the set of posture parameters.
The energy E cloth allows to take advantage of motion cues in this formulation as it encourages the body shape to lie inside all observed frames.

In the following sections, we detail the prior that is used to constrain as well as the di↵erent energy terms. Optimizing Eq. 5 w.r.t. all parameters jointly results in a high-dimensional optimization problem that is ine cient to solve and prone to get stuck in undesirable local minima. After introducing all energy terms, we discuss how this problem can be divided into smaller problems that can be solved in order, thereby allowing to find a good minimum in practice.

Prior model for

A prior model is used to ensure that the body shape stays within the learned shape space that represents plausible human shapes. The identity shape space is learned using PCA, and has zero mean and standard deviation i along the i-th principal component. Similarly to previous work [9], we do not penalize values of that stay within 3 i of the mean to avoid introducing a bias towards the mean shape. However, rather than penalizing a larger distance from the mean, we constrain the solution to lie inside the hyperbox ±3 i using a constrained optimization framework. This constraint can be handled by standard constrained optimizers since the hyperbox is axis-aligned, and using this hard constraint removes the need to appropriately weigh a prior energy w.r.t. other energy terms.

Landmark energy

The landmark energy helps to guide the solution towards the desired local minimum with the help of distinctive anatomical landmarks. This energy is especially important during the early stages of the optimization as it allows to find a good initialization for the identity and posture parameters. In the following, we consider the use of N lnd landmarks and assume without loss of generality that the vertices corresponding to landmarks are the first N lnd vertices of s. The landmark term is defined as

E lnd (F , , ⇥) = N lnd X i=1 ks i ( , ⇥) l i (F )k 2 , (6)
where l i (F ) denotes the i-th landmark of frame F , s i ( , ⇥) denotes the vertex corresponding to the i-th landmark of s ( , ⇥), and k•k denotes the `2 norm. The landmarks l i (F ) are computed automatically with the help of the state of the art Stitched Puppet [1], which allows to robustly fit a human body model to a single scan using a particle-based optimization. Specifically, we once manually select a set of vertex indices to be used as landmarks on the Stitched Puppet model, which is then fixed for all experiments. To fit the Stitched Puppet to a single frame, randomly distributed particles are used to avoid getting stuck in undesirable local minima. We fit the Stitched Puppet model to frame F , and report the 3D positions of the pre-selected indices after fitting as landmarks l i (F ). While the Stitched Puppet aims to fit the body shape and posture of F , only the coordinates l i (F ) are used by our framework. Note that our method does not require accurate l i (F ), since l i (F ) are only used to initialize the optimization.

Using many particles on each frame of a motion sequence is ine cient. Furthermore, since the Stitched Puppet is trained on a database of minimally dressed subjects, using many particles to fit to a frame in wide clothing may lead to overfitting problems. This is illustrated in Fig. 2. To remedy this, we choose to use a relatively small number of particles which is set to 30. Starting at the second frame, we initialize the particle optimization to the result of the previous frame to guide the optimization towards the desired optimum.

Data energy

The data energy pulls the S-SCAPE model towards the observation F using a nearest neighbor term. This energy, which unlike the landmark energy considers all vertices of s, is crucial to fit the identity and posture of s to the input F as

E data (F , , ⇥) = N v X i=1 N N ks i ( , ⇥) N N (s i ( , ⇥) , F )k 2 , (7)
where N v denotes the number of vertices of s and N N (s i ( , ⇥) , F ) denotes the nearest neighbour of vertex s i ( , ⇥) on F . To remove the influence of outliers and reduce the possibility of nearest neighbour mismatching, we use a binary weight N N that is set to one if the distance between s i and its nearest neighbor on F is below 200mm and the angle between their outer normal vectors is below 60 , and to zero otherwise.

Clothing energy

The clothing energy is designed to encourage the predicted body shape s to be located entirely inside the observation F . This energy is particularly important when considering motion sequences acquired with loose clothing. In such cases, merely using E lnd and E data leads to results that overestimate the circumferences of the body shape because is estimated to fit to F rather than to fit inside of F , see Fig. 3. To remedy this, we define the clothing energy as

E cloth (F , ,⇥ )= P N v i=1 out NN k s i ( ,⇥ ) N N ( s i ( ,⇥ ),F )k 2 +! r k 0 k 2 , ( 8
)
where out is used to identify vertices of s located outside of F . This is achieved by setting out to one if the angle between the outer normal of N N (s i ( , ⇥) , F ) and the vector s i ( , ⇥) N N (s i ( , ⇥) , F ) is below 90 , and to zero otherwise. Furthermore, ! r is a weight used for the regularization term, and 0 is an initialization of the identity parameters used to constrain . When observing a human body dressed in loose clothing in motion, di↵erent frames can provide valuable cues about the true body shape. The energy E cloth is designed to exploit motion cues when optimizing E cloth w.r.t. all available observations F i . This allows to account for clothing using a simple optimization without the need to find skin and non-skin regions as in previous work [9,19,7]. The regularization k 0 k 2 used in Eq. 8 is required to avoid excessive thinning of limbs due to small misalignments in posture.

Fig. 3 shows the influence of E cloth on the result of a walking sequence in layered clothing. The left side shows overlays of the input and the result for ! cloth = 0 and ! cloth = 1. Note that while circumferences are overestimated when ! cloth = 0, a body shape located inside the input frame is found for ! cloth = 1. The comparison to the ground truth body shape computed as discussed in Sec. 6 is visualized in the middle and the right of Fig. 3, and shows that E cloth leads to a significant improvement of the accuracy of .

Optimization schedule

Minimizing

E ⇣ F 1:N f , , ⇥ 1:N f
⌘ defined in Eq. 5 over all N f frames w.r.t. and ⇥ i jointly is not feasible when considering motion sequences containing hundreds of frames as this is a high-dimensional optimization problem. To solve this problem without getting stuck in undesirable local minima, we optimize three smaller problems in order. Initial identity estimation. We start by computing an initial estimate 0 based on the first N k frames of the sequence by optimizing

E ⇣ F 1:N k , , ⇥ 1:N k ⌘ w.r.t. and ⇥ i .
For increased e ciency, we start by computing optimal i and ⇥ i for each frame using Eq. 4 by alternating the optimization of ⇥ i for fixed i with the optimization of i for fixed ⇥ i . This is repeated for N it iterations. Temporal consistency is achieved by initializing ⇥ i+1 as ⇥ i and i+1 as i starting at the second frame. As it su ces for the identity parameters to roughly estimate the true body shape at this stage, we set ! cloth = 0. In the first iterations, E lnd is essential to guide the fitting towards the correct local optimum, while in later iterations E data gains in importance. We therefore set ! data = 1 ! lnd and initialize ! lnd to one. We linearly reduce ! lnd to zero in the last two iterations. We then initialize the posture parameters to the computed ⇥ i , and the identity parameters to the mean of the computed i and iteratively minimize E

⇣ F 1:N k , , ⇥ 1:N k ⌘ w.r.t. ⇥ 1:N k
and . This leads to stable estimates for ⇥ 1:N k and an initial estimate of the identity parameter, which we denote by 0 in the following.

Posture estimation. During the next stage of our framework, we compute the posture parameters ⇥ N k +1:N f for all remaining frames by sequentially minimizing Eq. 4 w.r.t. ⇥ i . As before, ⇥ i+1 is initialized to the result of ⇥ i . As the identity parameters are not accurate at this stage, we set ! cloth = 0. For each frame, the energy is optimized N it times while reducing the influence of ! lnd in each iteration, using the same weight schedule as before. This results in posture parameters ⇥ i for each frame.

Identity refinement. In a final step, we refine the identity parameters to be located inside all observed frames F 1:N f . To this end, we initialize the identity parameters to 0 , fix all posture parameters to the computed ⇥ i , and minimize

E ⇣ F 1:N f , , ⇥ 1:N f ⌘ w.r.t.
. As the landmarks and observations are already fitted adequately, we set ! lnd = ! data = 0 at this stage of the optimization.

Implementation details

The S-SCAPE model used in this work consists of N v = 6449 vertices, and uses d id = 100 parameters to control identity and d pose = 30 parameters to control posture by rotating the N b = 15 bones. The bones, posture parameters, and rigging weights are set as in the published model [8].

For the Stitched Puppet, we use 60 particles for the first frame, and 30 particles for subsequent frames. We use a total of N lnd = 14 landmarks that have been shown su cient for the initialization of posture fitting [6], and are located at forehead, shoulders, elbows, wrists, knees, toes, heels, and abdomen. Fig. 1 shows the chosen landmarks on the Stitched Puppet model. During the optimization, we set N it = 6 and N k = 25 . The optimization w.r.t. uses analytic gradients, and we use Matlab L-BFGS-B to optimize the energy. The setting of the regularization weight ! r depends on the clothing style. The looser the clothing, the smaller ! r , as this allows for more corrections of the identity parameters. In our experiments, we use ! r = 1 for all the sequences with layered and wide clothing in our dataset.

Evaluation

Dataset

This section introduces the new dataset we acquired to allow quantitative evaluation of human body shape estimation from dynamic data. The dataset consists of synchronized acquisitions of dense unstructured geometric motion data and sparse motion capture (MoCap) data of 6 subjects (3 female and 3 male) captured in 3 di↵erent motions and 3 clothing styles each. The geometric motion data are sequences of meshes obtained by applying a visual hull reconstruction to a 68-color-camera (4M pixels) system at 30FPS. The basic motions that were captured are walk, rotating the body, and pulling the knees up. The captured clothing styles are very tight, layered (long-sleeved layered clothing on upper body), and wide (wide pants for men and dress for women). The body shapes of 6 subjects vary significantly. Fig. 4 shows some frames of the database.

To evaluate algorithms using this dataset, we can compare the body shapes estimated under loose clothing with the tight clothing baseline. The comparison is done per vertex on the two body shapes under the same normalized posture. Cumulative plots are used to show the results.

Evaluation of posture and shape fitting

We applied our method to all sequences in the database. For one sequence of a female subject captured while rotating the body in wide clothing, Stitched Puppet fails to find the correct posture, which leads to a failure case of our method (see Fig. 2). We exclude this sequence from the following evaluation. To evaluate the accuracy of the posture parameters ⇥, we compare the 3D locations of a sparse set of landmarks captured using a MoCap system with the corresponding model vertices of our estimate. This evaluation is performed in very tight clothing, as no accurate MoCap markers are available for the remaining clothing styles. Fig. 5 summarizes the per-marker errors over the walking sequences of all subjects. The results show that most of the estimated landmarks are within 35mm of the ground truth and that our method does not su↵er from drift for long sequences. As the markers on the Stitched Puppet and the MoCap markers were placed by non-experts, the landmark placement is not fully repeatable, and errors of up to 35mm are considered fairly accurate. To evaluate the accuracy of the identity parameters , we use for each subject the walking sequence captured in very tight clothing to establish a ground truth identity 0 by applying our shape estimation method. Applying our method to sequences in looser clothing styles of the same subject leads identity parameters , whose accuracy can be evaluated by comparing the 3D geometry of s ( 0 , ⇥ 0 ) and s ( , ⇥ 0 ) for a standard posture ⇥ 0 . Fig. 6 summarizes the per-vertex errors over all motion sequences captured in layered and wide clothing, respectively. The left side shows the cumulative errors, and the right side shows the color-coded mean per-vertex error. The color coding is visualized on the mean identity of the training data. The result shows that our method is robust to loose clothing with more than 50% of all the vertices having less than 10mm error for both layered and wide clothing. The right side shows that as expected, larger errors occur in areas where the shape variability across di↵erent identities is high.

Layered clothing

Wide clothing

Knees up

Rotate body Walk Fig. 7. Overlay of input data and our result.

Fig. 7 shows some qualitative results for all three types of motions and two clothing styles. Note that accurate body shape estimates are obtained for all frames. Consider the frame that shows a female subject performing a rotating motion in layered clothing. Computing a posture or shape estimate based on this frame is extremely challenging as the geometry of the layered cloth locally resembles the geometry of an arm, and as large portions of the body shape are occluded. Our method successfully leverages temporal consistency and motion cues to find reliable posture and body shape estimates.

Comparative evaluation

As we do not have results on motion sequences with ground truth for existing methods, this section presents visual comparisons, shown in Fig. 8. We compare to Wuhrer et al. [6] on the dancer sequence [20] presented in their work. Note that unlike the results of Wuhrer et al., our shape estimate does not su↵er from unrealistic bending at the legs even in the presence of wide clothing. Furthermore, we compare to Neophytou and Hilton [7] on the swing sequence [21] presented in their work. Note that we obtain results of similar visual quality without the need for manual initializations and pre-aligned motion sequences. In summary, we present the first fully automatic method for body shape and motion estimation, and show that this method achieves state of the art results.

Comparison to

Comparison to Wuhrer et al. [6] Neophytou and Hilton [7] Fig. 8. Per comparison from left to right: input, result of prior works, our result.

Conclusion

We presented an approach to automatically estimate the human body shape under motion based on a 3D input sequence showing a dressed person in possibly loose clothing. The accuracy of our method was evaluated on a newly developed benchmark1 containing 6 di↵erent subjects performing 3 motions in 3 di↵erent styles each. We have shown that, although being fully automatic, our posture and shape estimation achieves state of the art performance. In the future, the body shape and motion estimated by our algorithm have the potential to aid in a variety of tasks including virtual change rooms and security applications.

HAL Id

Introduction

The last decade has seen the emergence of 3D dynamic shape models of moving objects, in particular humans, acquired from multiple videos. These spatiotemporal models comprise geometric and appearance information extracted from images, and they allow for subject motions to be recorded and reused. This is of interest for applications that require real 3D contents for analysis, free viewpoint and animation purposes and also for interactive experiences made possible with new virtual reality devices. This ability to now record datasets of subject motions bolsters the need for shape and appearance representations that make optimal use of the massive amount of image information usually produced. While dynamic shape representations have been extensively studied, from temporally coherent representations over a single sequence, to shape spaces that can encode both pose and subject variabilities over multiple sequences and multiple subjects, appearance representations have received less attention in this context. In this paper, we investigate this issue.

Currently, appearance information is still most often estimated and stored once per frame, e.g. a texture map associated to a 3D model [1], and the leap to an efficient temporal appearance representation is still a largely open problem. This is despite the obvious redundancy with which the appearance of subjects is observed, across temporal frames, different viewpoints of the same scene, and often several sequences of the same subject performing different actions or motions. At the opposite of the spectrum, and given registered geometries, one can store only one texture for a sequence or even for a subject in several sequences, hence dramatically reducing sizes, but in so doing would drop the ability to represent desirable appearance variations, such as change in lighting or personal expression of the subject.

In this paper, we advance this aspect by providing a view-independent appearance representation and estimation algorithm, to encode the appearance variability of a dynamic subject, observed over one or several temporal sequences. Compactly representing image data from all frames and viewpoints of the subject can be seen as a non-linear dimensionality reduction problem in image space, where the main non-linearities are due to the underlying scene geometry. Our strategy is to remove these non-linearities with state-of-the-art geometric and image-space alignment techniques, so as to reduce the problem to a single texture space, where the remaining image variabilities can be straightforwardly identified with PCA and thus encoded as Eigen texture combinations. To this goal, we identify two geometric alignment steps. First, we coarsely register geometric shape models of all time frames to a single shape template, for which we pre-computed a single reference surface-to-texture unwrapping. Second, to cope with remaining fine-scale misalignments due to registration errors, we estimate realignment warps in the texture domain. Because they encode low-magnitude, residual geometric variations, they are also advantageously decomposed using PCA, yielding Eigen warps. The full appearance information of all subject sequences can then be compactly stored as linear combinations of Eigen textures and Eigen warps. Our strategy can be seen as a generalization of the popular work of Nishino et. al. [2], which introduces Eigen textures to encode appearance variations of a static object under varying viewing conditions, to the case of fully dynamic subjects with several viewpoints and motions.

The pipeline is shown to yield effective estimation performance. In addition, the learned texture and warp manifolds allow for efficient generalizations, such as texture interpolations to generate new unobserved content from blended input sequences, or completions to cope with missing observations due to e.g. occlusions. To summarize our main contribution is to propose and evaluate a new appearance model that specifically addresses dynamic scene modeling by accounting for both appearance changes and local geometric inaccuracies.

Related work

Obtaining appearance of 3D models from images was first tackled from static images for inanimate objects, e.g. [3,2], a case largely explored since e.g. [4,5]. The task also gained interest for the case of subjects in motion, e.g. for human faces [6]. With the advent of full body capture and 3D interaction systems [7, Fig. 1: Overview: Time consistent shape modeling provides datasets of appearance maps. Our proposed method exploits the manifold structure of these appearance information through PCA decomposition to generate the Eigen appearance maps relative to a shape. 1] the task of recovering appearance has become a key issue, as the appearance vastly enhances the quality of restitution of acquired 3D models.

A central aspect of the problem is how to represent appearance, while achieving a proper trade-off between storage size and quality. 3D capture traditionally generates full 3D reconstructions, albeit of inconsistent topology across time. In this context the natural solution is to build a representation per time frame which uses or maps to that instant's 3D model. Such per instant representations come in two main forms. View-dependent texturing stores and resamples from each initial video frame [8], eventually with additional alignments to avoid ghosting effects [9]. This strategy creates high quality restitutions managing visibility issues on the fly, but is memory costly as it requires storing all images from all viewpoints. On the other hand, one can compute a single appearance texture map from the input views in an offline process [1], reducing storage but potentially introducing sampling artifacts. These involve evaluating camera visibility and surface viewing angles to patch and blend the view contributions in a single common mapping space. To overcome the resolution and sampling limitations, 3D superresolution techniques have been devised that leverage the viewpoint multiplicity to build such maps with enhanced density and quality [10][11][12].

In recent years, a leap has been made in the representation of 3D surfaces captured, as they can now be estimated as a deformed surface of time-coherent topology [13,14]. This in turns allows any surface unwrapping and mapping to be consistently propagated in time, however in practice existing methods have only started leveraging this aspect. Tsiminaki et. al. [11] examines small temporal segments for single texture resolution enhancement. Volino et. al. [15] uses a view-based multi-layer texture map representation to favour view-dependant dy-namic appearance, using some adjacent neighbouring frames. Collet et. al. [1] use tracked surfaces over small segments to improve compression rates of mesh and texture sequences. Methods are intrinsically limited in considering longer segments because significant temporal variability then appears due to light change and movement. While global geometry consistency has been studied [16][17][18], most such works were primarily aimed at animation synthesis using mesh data, and do not propose a global appearance model for sequences. In contrast, we propose an analysis and representation spanning full sequences and multiples sequences of a subject.

For this purpose, we build an Eigen texture and appearance representation that extends concepts initially explored for faces and static objects [19,6,20,2]. Eigenfaces [19] were initially used to represent the face variability of a population for recognition purposes. The concept was broadened to built a 3D generative model of human faces both in the geometry and texture domains, using the fact that the appearance and geometry of faces are well suited to learning their variability as linear subspaces [6]. Cootes et. al. [20] perform the linear PCA analysis of appearance and geometry landmarks jointly in their active appearance model. Nishino et. al. [2] instead use such linear subspaces to encode the appearance variability of static objects under light and viewpoint changes at the polygon level. We use linear subspaces for full body appearance and over multiple sequences. Because the linear assumption doesn't hold for whole body pose variation, we use state of the art tracking techniques [21] to remove the non-linear pose component by aligning a single subject-specific template to all the subject's sequence. This in turn allows to model the appearance in a single mapping space associated to the subject template, where small geometric variations and appearances changes can then be linearly modeled.

Method

To eliminate the main geometric non-linearity, we first align sequence geometries to a single template shape and extract the texture maps of a subject over different motion sequences in a common texture space using a state-of-the-art method [11]. Other per-frame texture extractions may be considered. From these subject specific textures, Eigen textures and Eigen warps that span the appearance space are estimated. The main steps of the method below are depicted in Figure 2 and detailed in the following sections.

1. Texture deformation fields that map input textures to, and from, their aligned versions are estimated using optical flows. Given the deformation fields, Poisson reconstruction is used to warp textures. 2. PCA is applied to the aligned maps and to the texture warps to generate the Eigen textures and the Eigen warps that encode the appearance variations due to, respectively, viewpoint, illumination, and geometric inaccuracies in the reference model. Hence, The main modes of variation of aligned textures and deformation fields, namely Eigen textures and Eigen warps respectively, span the appearance space in our representation. The main steps of this method are depicted in Figure 2 and detailed in the following sections.

Note that due to texture space discretization, the warps between textures are not one-to-one and, in practice, two separate sets of warps are estimated. Forward warps map the original texture maps to the reference map. Backward warps map the aligned texture maps back to the corresponding input textures (see Figure 2).

Aligning texture maps

Appearance variations that are due to viewpoint and illumination changes are captured through PCA under linearity assumption for these variations. To this purpose, textures are first aligned in order to reduce geometric errors resulting from calibration, reconstruction and tracking imprecisions. Such alignment is performed using optical flow, as described below, and with respect to a reference map taken from the input textures. An exhaustive search of the best reference map with the least total alignment error over all input textures is prohibitive since it requires N 2 alignments given N input textures. We follow instead a medoid shift strategy over the alignment errors.

The alignment algorithm (see Algorithm 1) first initializes the reference map as one texture from the input set. All texture maps are then aligned to this reference map, and the alignment error is computed as the cumulative sum of squared pixel differences between the reference and the aligned texture maps. The medoid over the aligned texture maps, with respect to alignment error, then identifies the new reference map. These two steps, alignment and medoid shift, are iterated until the total alignment error stops decreasing.

Data: Texture maps {I k } k∈[[1..N ]] Result: Reference map A ref , aligned textures A k A ref ,
A ref = I k0 s.t. k 0 = arg min k l A k -A l 2 ;
end Algorithm 1: Texture alignment with iterative reference map selection.

Dense texture correspondence with optical flow The warps {w k } in the alignment algorithm, both forward and backward in practice, are estimated as dense pixel correspondences with an optical flow method [22]. We mention here that the optical flow assumptions: brightness consistency, spatial coherency and temporal persistence, are not necessarily verified by the input textures. In particular, the brightness consistency does not hold if we assume appearance variations with respect to viewpoint and illumination changes. To cope with this in the flow estimation, we use histogram equalization as a preprocessing step, which presents the benefit of enhancing contrast and edges within images. Additionally, local changes in intensities are reduced using bilateral filtering, which smooths low spatial-frequency details while preserving edges.

Texture warping Optical flows give dense correspondences {w} between the reference map and the input textures. To estimate the aligned textures {A}, we cast the problem as an optimization that seeks the texture map which, once moved according to w, best aligns with the considered input texture both in the color and gradient domains. Our experiments show that solving over both color and gradient domains significantly improves results as it tends to better preserve edges than with colors only. This is also demonstrated in works that use the Poisson editing for image composition, e.g. [23,24] or interpolation, e.g. [25,26]. We follow here a similar strategy.

We are given an input texture map I, a dense flow w from A ref to I, and the gradient image ∇I. The aligned texture A of I with respect to A ref is then the map that minimizes the following term:

E(A) = Σ x ∇ 2 A(x) - - → ∇.∇I(x + w) 2 + λ A(x) -I(x + w) 2 , (1)
where ∇ 2 is the Laplacian operator, -→ ∇. the divergence operator, and x denotes pixel locations in texture maps. The weight λ balances the influence of color and gradient information. In our experiments, we found that the value 0.02 gives the best results with our datasets.

Using a vector image representation, the above energy can be minimized by solving, in the least-squares sense, the overdetermined 2N × N system below, where N is the active region size of texture maps:

L Λ A = - → ∇.∇I(x + w) ΛI(x + w) , ( 2
)
where L is the linear Laplacian operator and Λ = diag N (λ). A solution for A is easily found by solving the associated normal equations:

L T L + Λ 2 A = L T - → ∇.∇I(x + w) + Λ 2 I(x + w). (3)
Figure 3 shows an example where a texture map is warped, given a warp field, using both direct pixel remapping and Poisson warping. The latter strategy achieves visually more compelling and edge preserving results.

Eigen Textures and Eigen Warps

Once the aligned textures and the warps are estimated, we can proceed with the statistical analysis of appearances. Given the true geometry of shapes and their motions, texture map pixels could be considered as shape appearance samples over time and PCA applied directly to the textures would then capture appearance variability. In practice, incorrect geometry causes distortions in the texture space and textures must be first aligned before any statistical analysis. In turn, de-alignment must be also estimated to map the aligned textures back to their associated input textures (see Figure 2). And these backward warps must be part of the appearance model to enable appearance reconstruction. In the following, warps denote the backward warps. Also, we consider vector representations of the aligned texture maps and of the warps. These representations include only pixels that fall inside active regions within texture maps. We perform Principal Component Analysis on the textures and on the warp data separately to find the orthonormal bases that encode the main modes of variation in the texture space and in the warp space independently. We refer to vectors spanning the texture space as Eigen textures, and to vectors spanning the warp space as Eigen warps.

Let us consider first texture maps. Assume N is the dimension of the vectorized representation of active texture elements, and F the total number of frames available for the subject under consideration. To give orders of magnitude for our datasets, N = 22438995 and F = 207 for the Tomas dataset, and N = 25966476 and F = 290 for the Caty dataset that will be presented in the next section. We start by computing the mean image Ā and the centered data matrix M from aligned texture maps

{A i } i∈[1..F] : Ā = 1 F k A k , M =   | | A 1 -Ā ... A F - Ā | |   . (4)
Traditionally, the PCA basis for this data is formed by the Eigen vectors of the covariance matrix M M T , of size N × N , but finding such vectors can easily become prohibitive as a consequence of the texture dimensions. However, it appears that the non zero eigen values of M M T are equal to the non zero Eigen values of M T M , of size (F × F ) this time, and that they are at most: min(F, N ) -1. Based on this observation, and since F << N in our experiments, we solve the characteristic equation det(M M T -αI N ) = 0 by performing Singular Value Decomposition on the matrix M T M , as explained in [27]:

M T M = DΣD T , D =   | | V 1 ... V F | |   (5)
where D contains the (F -1) orthonormal Eigen vectors {V i } of M T M , and Σ = diag(α i ) 1≤i≤F contains the eigen values {α i } 1≤i≤F -1 . We can then write:

M T M V i = α i V i , i ∈ [1..F -1], (6)
and hence:

M M T M V i Ti = α i M V i Ti , i ∈ [1..F -1], (7)
where T i are the Eigen vectors of M M T and therefore form the orthonormal basis of the aligned texture space after normalization, namely the Eigen textures.

In a similar way, we obtain the mean warp w and the orthonormal basis of the warp space {W i } 1≤i≤F -1 , the Eigen warps.

Texture generation

Given the Eigen textures and the Eigen warps, and as shown in Figure 4, a texture can be generated by first creating an aligned texture by linearly combining Eigen textures and second de-aligning this new texture using another linear combination of the Eigen warps.

Performance Evaluation

To validate the estimation quality of our method, we apply our estimation pipeline to several datasets, project and warp input data using the built eigenspaces, then evaluate the reconstruction error. To distinguish the different error sources, we evaluate this error both in texture space before projection, and in image domain by projecting into the input views, as compared to the original views of the object and the texture before any reconstruction in texture space, estimated in our pipeline using [11]. For the image error measurement, we use the 3D model that was fitted to the sequence, as tracked to fit the test frames selected [21], and render the model as textured with our reconstructed appearance map, using a standard graphics pipeline. In both cases, we use the structural similarity index (SSIM) [28] as metric to compare to the original. All of our SSIM estimates are computed in the active regions of the texture and image domains, that is on the set of texels actually mapped to the 3D model in the texture domain, and only among actual silhouette pixels in the image domain.

We study in particular the compactness and generalization abilities of our method, by examining the error response as a function of the number of eigen components kept after constructing the linear subspaces, and the number of training images selected. For all these evaluations, we also provide the results for a naive PCA strategy, where only a set of eigen appearance maps are built in texture space and use to project and reconstruct textures, to show the performance contribution of including the Eigen warps.

For validation, we used two multi-sequence datasets: (1) the Tomas dataset which consists of 4 different sequences left, right, run and walk with 207 total number of frames and 68 input views each captured at resolution 2048x2048 pixels per frame; and (2) the Caty dataset: low, close, high and far jumping sequences with 290 total number of frames and 68 input views each captured at resolution 2048x2048 pixels per frame.

Estimation Quality and Compactness

We study the quality and compactness of the estimated representation by plotting the SSIM errors of reconstructed texture and image estimates of our method against naive PCA, for the two multi-sequence datasets (Figure 5). Note that all texture domain variability could be trivially represented by retaining as many Eigen textures as there are input images, thus we particularly examine how the quality degrades with the fraction of Eigen components kept. In the case of image domain evaluations, we plot the average SSIM among all viewpoints. Our method outperforms naive PCA in image and texture domains on both datasets, achieving higher quality with a lower number of Eigen components, and only marginally lower quality as the number of components grows, where the method would be anyway less useful. Higher number of Eigen components marginally favors naive PCA, because naive PCA converges to input textures when increasing the Eigen textures retained by construction, whereas our method hits a quality plateau due to small errors introduced by texture warp estimation and decomposition. For both datasets, virtually no error (0.98 SSIM) is introduced by our method in the texture domain with as low as 50 components, a substantially low fraction compared to the number of input frames (207 and 290). This illustrates the validity of the linear variability hypothesis in texture domain. The error is quite higher in the image domain (bounded by 0.7) for both our method and naive PCA, because measurements are then subject to fixed upstream errors due to geometric alignments, projections and image discretizations. Nevertheless, visually indistinguishable results are achieved with 50 Eigen components (images and warps), with a significant compactness gain.

Generalization ability

In the previous paragraph, we examined the performance of the method by constructing an Eigen space with all input frames. We here evaluate the ability of the model to generalize, i.e. how well the method reconstructs textures from input frames under a reduced number of examples that don't span the whole input set. For this purpose, we perform an experiment using a varying size training set, and a test set from frames not in the training set. We use a training set comprised of randomly selected frames spanning 0% to 60% of the total number of frames, among all sequences and frames of all datasets, and plot the error of projecting the complement frames on the corresponding Eigen space (Figure 6). The experiment shows that our representation produces a better generalization than naive PCA, i.e. less training frames need to be used to reconstruct a texture and reprojections of equivalent quality. For the Tomas dataset, one can observe than less than half training images are needed to achieve similar performance in texture space, and a quarter less with the Caty dataset.

Applications

We investigate below two applications of the appearance representation we propose. First, the interpolation between frames at different time instants and sec- ond, the completion of appearance maps at frames where some appearance information is lacking due to occlusions or missing observations during the acquisition. Results are shown in the following section and the supplementary video.

Number of Eigen Vectors

Interpolation

In our framework, appearance interpolation benefits from the pre-computed warps and the low dimensionality of our representation to efficiently synthesize compelling new appearances with reduced ghosting-artefacts. It also easily enables extension of appearance interpolation from pairwise to multiple frames. Assume that shapes between two given frames are interpolated using a standard non-linear shape interpolation, for instance [29]. Consider then the associated aligned textures and associated warps at the given frames. We perform a linear interpolation in the Eigen texture and Eigen warp spaces respectively by blending the projection coefficients of the input appearance maps. Poisson warping, as introduced in section 3.1 is used to build de-aligned interpolated texture with the interpolated backward warp. Figure 7 compares interpolation using our pipeline to a standard linear interpolation for 4 examples with the Caty and Tomas datasets. Note that our method is also linear but benefits from the alignment performed in the texture space to reduce interpolation artefacts, as well as from the simplified computational aspects since interpolation applies to projection coefficients only.

Completion

As mentioned earlier, appearance maps can be incomplete due to acquisition issues. For instance, as shown in Figure 8, during the running sequence the actor Tomas bends his knees in such a way that the upper parts of his left and right shins become momentarily hidden to the acquisition system. This results in missing information for those body parts in the texture maps and over a few frames. Such an issue can be solved with our texture representation by omitting the incomplete frames when building our appearance representations, and then projecting these incomplete appearance maps in the Eigen spaces and reconstructing them using the projection coefficients and Poisson texture warping. Figure 8 shows two examples of this principle with occluded regions. Note however, that while effectively filling gaps in the appearance map, this completion might yet loose appearance details in regions of the incomplete map where information is not duplicated in the training set.

Conclusion

We have presented a novel framework to efficiently represent the appearance of a subject observed from multiple viewpoints and in different motions. We propose a straightforward representation which builds on PCA and decomposes into Eigen textures and Eigen warps that encode, respectively, the appearance variations due to viewpoint and illumination changes and due to geometric modeling imprecisions. The framework was evaluated on 2 datasets and with respect to: (i) its ability to accurately reproduce appearances with compact representations;

(ii) its ability to resolve appearance interpolation and completion tasks. In both cases, the interest of a global appearance model for a given subject was demonstrated. Among the limitations, the representation performances are dependent on the underlying geometries. Future strategies that combine both shape and appearance information would thus be of particular interest. The proposed model could also be extended to global representations over populations of subjects.

A. 3 D shape tracking is the process of recovering temporal evolutions of a template shape using visual information, such as images or 3D points. It finds applications in several domains including computer vision, graphics and medical imaging. In particular, it has recently demonstrated a good success in marker-less human motion capture (mocap). Numerous approaches assume a user-specific reference surface, with the objective to recover the skeletal poses [1], surface shapes [2], or both simultaneously [3]. A standard tracking process consists in an alternation of the following two steps. First, finding associations between the observed data, e.g. 3D points of the reconstructed visual hull, to the corresponding 3D template shape, typically based on the proximity in Euclidean space or a feature space. Second, given such associations, recovering the pose of the template under the constraint of a deformation model, typically based on the kinematic skeleton [1], [4], [5], [6], or the piecewise-rigid surface [2] parameterization, among others.

Most of these model-based methods can be viewed as extensions of Iterative-Closest-Point (ICP) framework [7], [8] to deformable shapes, which attempts to explain newly observed data using the previous outcomes. As long as the initialization is close to the optimum solution, it is able Manuscript received XX.XX, 2017; revised XX.XX, 2017. to produce outstanding results. However, they also suffer from inherent weaknesses of generative strategies, e.g. slow convergence. Moreover, when large deformations or many outliers occur, discovering associations becomes particularly difficult. Unreliable correspondences result in ambiguous situations that yield erroneous numerical solutions.

Recently, a number of alternatives and enhancements have been explored for both association and deformation stages independently. On one hand, improvements have also been proposed for the association problem by discovering them discriminatively [6], [9], [16], This in turn yields the possibility for 3D tracking techniques that are robust to failure. In contrast to those generative ICP variants, these discriminative approaches that 'detect' rather than track models have shown better robustness over the past decade, for instance, in human pose estimation with 2.5D data from Kinect [6], [10]. These approaches usually consider foreground human subjects to be pre-segmented, which is not a favorable assumption in full 3D data that generally contains substantial amount of outliers like Fig. 1(b). Including nonhuman objects into the reference shape so that more points are explained, i.e. less outliers, is one workaround adopted by many existing multi-view methods [17], [18], with the downside that further post-processing is required to analyze only humans' movements. There is a growing need to facilitate robust frame-wise observation-model associations for reconstructed complete 3D human shapes. Although surface-based features are commonly used for this purpose in the context of shape matching [9], volumetric features have also proven to be a promising direction for 3D shape description with surface-based templates [11].

On the other hand, progress has also been made in the deformation stage by introducing volumetric deformation models instead of purely surface-based ones, mainly motivated by the observation that human movements are largely volume-preserving. It has shown significantly improved robustness to various tracking situations, such as shape folding and volume bias of observed shapes [12]. As volumetric deformation models are gradually used in capturing actors' motions due to their inherent local volumepreserving properties, facilitating volumetric discriminative correspondences can be favorable. We investigate this direction and make the following two contributions in this paper.

First, two volumetric features are designed for human shape correspondence detection, operating respectively on surface and volumetric meshes. Inspired by Taylor et al. [6], we apply regression forests to improve the associations, with two learning strategies devised for different shape parameterizations. In the case of surface mesh representations, we convert shapes to the volumetric Truncated Signed Distance Field (TSDF) [13], where each surface vertex is fed into userspecific forests to predict correspondences in one shot. Meanwhile, we also tessellate both the observed and template shapes as a set of uniform and anisotropic cells (see Fig. 2) from Centroidal Voronoi Tessellation (CVT) [14] and, again leverage the similar distance-transform representations to predict volumetric correspondences for all CVT cells.

Second, by integrating these one-shot associations into the respective deformation models, we further present a discriminative human mocap framework, as depicted in Fig. 1, termed tracking-by-detection of 3D human shapes. In contrast to the ICP-like methods [2], [3], [4], it does not require close initializations from a nearby frame to estimate correspondences and thus better handles large deformations. Experiments demonstrate that, when combined with a generative tracking approach, this hybrid framework leads to better or comparable results than purely generative ones, e.g. [2], [15], reducing error accumulations and hence increasing the stability. The regression entropy is also augmented with the classification one to identify outliers. Very few prior arts afford the tracking or matching situation where the input describes mainly irrelevant outliers. Notably, in the case of CVT, our method is a unified volumetric pipeline where the shape representation, deformation model, feature description, and points association are all built on a single CVT representation that brings benefits at all stages of the pipeline. This fully volumetric tracking-by-detection method shows improved accuracy and memory performance compared to the surface-based counterpart [11].

template observations inner cells inner cells

Fig. 2. Centroidal Voronoi tessellations yields volumetric cells of uniform shape and connectivity with controllable complexity. The cells of the observed shape are matched discriminatively to those of the template.

RELATED WORK

Among the vast literature on human motion analysis [19], we focus on top-down approaches that assume a 3D template and deform it according to input data, either directly with pixels [4], [20], or with computed 3D points [2], [3], [15]. These methods typically decompose into two major steps: (1) data association, where observations are associated to the model, and (2) deformation stage, where motion parameters are estimated given the associations. As our primary objective in this paper is to improve the first part, existing approaches are discussed accordingly below.

Generative approaches

Methods of this category follow the association strategy in ICP while extending the motion model to more general deformations than the one in the original method [7], [8].

Correspondences are addressed by searching for closest points, with various distance measures such as point-topoint [2], point-to-plane [21], or Mahalanobis distances [20]. This strategy heavily relies on the fact that observations in consecutive frames are in vicinity. Klaudiny et al. [22], Huang et al. [3] and Collet et al. [17] generalize the idea from the previous frame to a certain key-frame in the considered sequences, finding the best non-sequential order to track, but the proximity assumption remains. On the other hand, since 3D data such as reconstructed point clouds often contain spurious fake geometries, another challenge consists in identifying online and dynamically irrelevant observations without any prior knowledge. Liu et al. [4] establish 3D-2D correspondences by considering both texture in images and contours in silhouettes and further include image segmentation information to differentiate multiple interacting subjects. Huang et al. [2], [3] relax the hard correspondence constraint to soft assignments and introduce an additional outlier class to reject noisy observations. Data is explained by Gaussian Mixture Models (GMM) in an Expectation-Maximization (EM) manner [23]. In [24], both source and target points are similarly modeled as GMMs and the registration problem is cast as minimizing the distance between two mixture models. Collet et al. [17] fuse information from various modalities attentively to generate high-quality textured meshes. Yet, to yield a temporal coherent mesh tessellation, the underlying tracking component is still ICPbased [25]. All these generative methods are highly likely to fail in large deformations. Furthermore, they are prone to error accumulations and, as a result of matching several successive frames wrongly (whether sequentially or not), they are prone to drift. Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

Discriminative approaches and 3D descriptors

Recently, discriminative approaches have demonstrated their strengths in estimating human [6], [26] and hand [27] poses from depth images. With the initial intention to substitute ICP-based optimization, Taylor et al. [6] propose a frame-wise strategy that yields decent dense correspondences without iterative refinements. The method replaces the step of proximity search in ICP-based tracking methods by learning the mapping from input 3D points from depth sensors, to the human template surface domain, termed the Vitruvian manifold. Later, Pons-Moll et al. [5] train forests with a new objective on surface manifolds, and increase the precision by finishing convergence with an ICP-based loop after the discriminative association stage. Both approaches operate frame-independently and are generally drift free.

Following the same weak pair-wise features and random forest framework, Dou et al. [18] learn to match two successive depth frames to avoid depending on a specific template. More informative descriptors and matching strategies have long been studied for shape recognition or retrieval with meshes [28] and point clouds [29]. The well known heat kernel signatures (HKS) [30] and wave kernel signatures (WKS) [31] exploit the Laplacian-Beltrami operator, the extension of the Laplacian operator to surface embeddings. Rodola et al. [9] later apply forests to learn the parameters of WKS during training. These features are nonetheless known for their lack of resilience to significant topology changes, an artifact frequently seen in noisy surface acquisitions. Mesh-HoG [32] and SHOT [33] attach a local coordinate frame at each point to achieve invariant representations and reach better performance for noisy surfaces. To enforce consistent matches over the whole shape, Chen and Koltun [34] and Starck et al. [35] formulate the matching problem as the inference of Markov random field (MRF).

Besides hand-crafted features, there is a recent trend that applies Convolutional Neural Network (CNN) [36] to discover the deep representation of non-rigid human shapes. Wei et al. [16] render depth images in several viewpoints, where the CNN feature transformation takes place, and average the descriptors from multiple views. Boscaini et al. [37] stay in 3D space but define the convolution function in the intrinsic manifold domain. While showing encouraging results in handling missing data, these methods do not consider matching human shapes in the presence of large amount of outliers, e.g. un-subtracted furniture in the background, and thus do not fit to our 'detection' purpose.

Another common trait of the aforementioned approaches is that the computation involves only surface points. We show in our early work [11] that surface features can be built based on local coordinate frames in a regular-grid volume. In this paper, we not only improve this feature but also propose a new one to address the need of fully volumetric correspondences. Both features, implicitly or explicitly, leverage distance-transform volumes to describe 3D geometry. Taking only surface vertices into account, the existing approaches rely on heterogeneous shape representations, deformation models, target primitives and feature spaces. Instead, our CVT-based tracking-by-detection proposal builds a unified framework for all these purposes and takes advantage of volumetric tracking strategies.

OVERVIEW

We implement discriminative associations using two different volumetric representations. In the first case, we convert the triangular surface meshes to the Truncated Signed Distance Field (TSDF) constructed with the regular 3D volumetric grid. In the second case, we use CVT representation which is not bound to the regular grids. As in Fig. 2, the interior space of a triangular surface is tessellated into a set of cells of uniform anisotropic shape whose seed location coincides with its centers of mass. Such an optimal discretization yields lower memory footprint than regular-grid volumes, in turn accommodating more training meshes. Moreover, we also associate CVT cells discriminatively and present volumetric correspondences.

Formally, a humanoid shape describes a continuous volumetric domain in 3D Ω ⊂ R 3 whose border ∂Ω defines a 2-manifold surface. The discretized mesh representation M contains a set of 3D points M and their connectivity T , i.e. M = (M, T ), where M is drawn from the surface (M ⊂ ∂Ω) or the whole volume (M ⊂ Ω ). The goal of 3D shape tracking is to register a source reference 1 mesh X = (X, T X ) to the observed target mesh Y = (Y, T Y ), such as fitting the shape in Fig. 1(a) to the one in Fig. 1(b).

Our method starts with surface meshes reconstructed by shape-from-silhouette method [38]. We refer only to points on surfaces as vertices v ∈ V , where V is the set of their indices. Suppose the reference surface X and the input visual hull Y are located at

X = {x v } v∈V X and Y = {y i } i∈V Y 2
respectively, the registration typically boils down to two steps: (1) association: matching each points in Y with those in X to build the correspondence set To discover the correspondences C discriminatively, we adapt the Vitruvian strategy [6] from matching 2.5D against 3D to 3D against 3D. This amounts to warping the input mesh Y to the reference one X , denoted as Ỹ = ( Ỹ, T Y ) = (r(Y), T Y ) where r is the warping function. A good r shall lead to a clean warp Ỹ as in Fig. 3. Incorrect warped points, however, can still be told from huge edges. Specifically, this R 3 → R 3 mapping r is learned by a regression forest [39]. We convert each surface into an implicit representation, a distance field, which is usually defined volumetrically. As stated above, we investigate two ways to define the volumetric elements s. The first one is a voxel from a regular axis-aligned volume, i.e. s ∈ N 3 , while the second one is a cell from a volumetric mesh, i.e. s ∈ S, where S is a group of CVT cells that tessellate only the surface interiors. Depending on the choice of s, our volumetric feature f is hence also realized in two different forms. Taking the feature f as input, multiple binary decision trees are trained with previously observed meshes. In the online testing phase, a input point obtains a prediction ỹi = r(y i ) that indicates the locations of potential matches since the warp Ỹ is 1. Several terms are used interchangeably in this paper: reference and template; correspondences and associations; point and primitive.

C = {(i, v)} ⊂ V Y × V X ; and (
2. The observations are always indexed by i regardless of the parameterization. This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.

The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org. learned to resemble X. Thus, C can be built swiftly by doing nearest neighbor search between Ỹ and X just once and the deformation parameter Θ that encodes the shape pose of the template is estimated accordingly. Notably, in the case of CVT, since cells comprise a volumetric mesh, the whole pipeline (discovering C and estimating Θ) can instead be conducted in a fully volumetric fashion. Fig. 3 illustrates this correspondence detection process. The details of training, prediction and deformation models are provided in § 5.

VOLUMETRIC FEATURES

The two volumetric features are introduced in this section.

Although both taking a volumetric point s as input, the first one actually aims to match surface vertices v, denoted as f (v) := f (s v ) while the second one matches s directly, i.e. f (s). Both are designed to be incorporated into forest training and prediction. A great advantage of decision trees is to learn the most discerning attributes among a large feature bank. One does not have to prepare the whole highdimensional vector f to draw predictions, because only a few learned attributes κ are needed to traverse the trees. As a result, features can be computed on the fly during testing. To make use of such property, the calculation of each f κ is assumed to independent. We hence avoid the histogram-based descriptors that requires normalization, such as MeshHOG [32] or SHOT [33], and resort to offset comparison features used in [40] for f (s v ) and Haar feature in [41] for f (s).

Regular-voxel-based features

Our first approach to discriminative associations considers regular-grid volumes (upper row in Fig. 3, s ∈ N 3 ). The warping function r is modeled as a composite one:

r : R 3 → N 3 → R 3
, where the former is voxelization and the regression trees account for only the latter. We first cast each mesh M into a volumetric scalar field D : N 3 ⊂ R 3 → R.

Truncated signed distance transform (TSDT)

Voxelizing a surface in general comprises two parts: (1) determining which voxel s that every vertex v maps to, and (2) testing the overlap between triangles and voxels. The first part can be viewed as a quantization mapping from Euclidean space to a discretized space s : R 3 → N 3 . The size of the volume is large enough to include all possible pose variations, and its center is aligned with the barycenter of the surfaces. The voxel size is chosen to be close to the average edge length of meshes, so that a single voxel is not mapped by too many vertices. To check the intersection of triangles with voxels, we apply separating axis theorem which is known to be efficient for collision detection [42]. Voxels occupied by the surface are referred to as s suf . We further identify voxels located inside and outside the surface, denoted respectively as s in and s out . Together they define a directional truncated signed distance transform: (1) d(s, M) denotes the shortest Euclidean distance from the voxel center to the mesh, which can be computed efficiently via AABB trees using CGAL library. If the distance is bigger than a threshold , we store only ± to indicate the inside/outside information. It is empirically set to be three times the physical length of diagonal of voxels. In the earlier version of this work [11], we store averaged surface normals at each s suf . However, such representations yield high memory footprint and thus limit the amount of training meshes we can incorporate later in § 5. The TSDT representation naturally encodes the spatial occupancies of a mesh and the required memory footprint is only one-third of the former (each voxel stores now just a scalar, not a vector). It shares a similar spirit with implicit surface representations, e.g. levelset, and has been widely employed in RGBD-based tracking or reconstruction [43], [44].

D(s) =                + if

Pair-wise offset features

Next, we present the features f for describing TSDT, which are later used to train the forests. Since we are interested in predicting correspondences for vertices instead of triangles, from now on we concentrate only on those surface voxels s suf occupied by mesh vertices v, denoted as s v . The feature is thus defined as a function of s v , i.e. f (v) := f (s v ). As depicted in Fig. 4, for each surface voxel s v (blue), we shoot two offsets (red vectors) ψ = (o 1 , o 2 ) ∈ N 3 × N 3 , reaching two neighboring voxels (green). To describe the local geometry, we take the TSDT values within a cuboid around two respective voxels (yellow squares), perform element-wise subtractions and sum them up. Let ε denotes this sum-of-difference operation. By definition, ε from different offsets ψ can be evaluated independently and thus fully parallelizable, which is an useful trait since this computation will be carried out multiple times during training with thousands of randomly generated ψ for the same s v .

The feature vector f consist of ε resulted from many offset pairs ψ. More precisely, it is a function of s v but takes an offset pair ψ, a binary variable η (whether to use Local Coordinate Frame (LCF) or not), and a rotational matrix R ∈ SO(3) (the orientation of LCF) as parameters. Every possible combination of offset pairs ψ and binary variables η results in one independent feature attribute κ, in notations:

f κ (s v ) = ε(s v ; R η (ψ)).
The dimensionality of f is virtually infinite. Binary variables η determines the alignment of the offset ψ with respect to a LCF, whose transformation is specified by R. The intuition behind this adjustment is to make features f invariant to poses, c.f. Fig. 4(b) and(c). Without re-orientations, ψ might land on different types of voxel pairs, c.f. Fig. 4(a) and (b), and hence cause different feature responses ε, despite the fact that the current voxels are located on the same position on the body. Both offset pairs ψ and binary variables η are learned during forest training, while the rotational matrix R is characterized by a LCF obtained as follows.

Local coordinate frame

Defining local coordinate frames for 3D primitives (voxels, vertices, points) has long been studied and usually comes with their 3D descriptor counterparts, see [45] for a comprehensive review. An ideal LCF is supposed to follow whatever transformations the meshes undergo, namely, as co-variant as possible, such that the consequent feature representations are as invariant as possible. Constructing a LCF boils down to defining three orthonormal vectors as [x, y, z] axes. To do that, the state-of-the-art methods in the field of LCFs for rigid matching of 3D meshes and point clouds mainly rely on the neighboring points within a local support [33], [46], [47], [48]. The way they leverage spatial distributions can in general be classified into two categories:

(1) EigenValue-Decomposition (EVD) [33], [47], [49], and (2) signed distance (SignDist.) [46], [48]. Since it is impractical to repeat EVD process for all surface voxels s v , in the following, we propose an adaptation of SignDist. approach to our volumetric representations [50]. This conclusion is drawn after an extensive study and comparison of three LCF approaches presented in our early work [50]. Specifically, for each s v , we consider its surface normals n v as z axis, and obtain y axis by z × x. The task left is to identify a repeatable x axis. To this end, the class of SignDist. approaches look for a discerning point within the support (yellow voxel in Fig. 5(b)). We first open an local cuboid support (pink) around each s v (green) as visualized in Fig. 5(a). The search involves only the peripheral voxels s (cyan) lying on the intersection of support borders and the surface. The discernibility is defined as the maximum signed distance to the tangent plane [46]:

ŝ = arg max s∈ S (s -s v ) n v , ( 2
)
where S is the intersection of support borders and the surface. The x axis is the projection of the vector directed from s v towards ŝ. Fig. 5(b) illustrates the full procedure. Note that there is no guarantee that the discerning point ŝ from Eq. 2 is always repeatable: in particular, if different directions yield similar values of the signed distance, the x axis will be ambiguous, hence the resulting LCFs could rotate about the z axis. Therefore, as shown in Fig. 5(c), this approach produces LCFs quasi-covariant to pose changes, and as a result, only quasi-pose-invariant features f . We leave such noise for forests to take care of during learning.

CVT-based features

The feature f (s v ) above describes surface geometries in volumes but is devised to match only surface vertices v.

A more intriguing question is: can one match these points s directly? In other words, instead of an auxiliary role of matching surfaces, can they also be associated to the template discriminatively and even participate in shape deformations (bottom row of Fig. 3)? We investigate this direction with a volumetric representation from centroidal Voronoi tessellations that haven shown some recent success in various applications [51], [52], i.e. s is a CVT cell. We use it to sample a distance field where every cell s stores the Euclidean distance from the centroid to the surface ∂Ω: d(x s , ∂Ω) = min p∈∂Ω d(x s , p), yielding a distancetransform like representation similar to the TSDT above.

Haar-like spherical feature

The offset feature f (s v ) above is nevertheless not applicable here since it relies on regular grids. We propose a new feature f (s) with the following principles in mind. It should be able to characterize the local neighborhood of any point of the volumetric shape. This rules out the descriptors that rely on surface normals such as MeshHOG [32] and SHOT [33]. To be able to match any deformed pose with the template, we would like our feature to be pose-invariant. Therefore, we build it on the distance transform because it naturally encodes the relative location with respect to the surface and it is invariant to rotations, translations and quasi-invariant to pose changes. Finally, our feature needs to be robust to the topological noise present in the input data. Given a distance field sampled by CVT cells S, our feature is similar in spirit to Haar feature in the Viola-Jones face detector [41], except that the rectangular neighborhood is replaced with a sphere. As depicted in Fig. 6, we open an L-layer spherical support region in the Euclidean space around each cell. An L-dimensional vector u is defined accordingly, where each element u l is the sum of the distances of all cells falling within layer l. The feature value is the linear combination of all u l , with coefficients c l chosen from a set Υ = {-1, 0, 1}. Formally, suppose c are L-dimensional vectors whose elements are the bootstrap samples of Υ. Let c κ denote one particular instance of c, i.e. , c κ ∈ Υ L . The feature value is then expressed as an inner product: u c κ , corresponding to one feature attribute κ. We consider all possible c κ and also take the distance d itself into account. f is hence a vector of (3 L + 1) dimensions, where 3 L is the cardinality of Υ L and each element f κ is defined as:

f κ u c κ = l c κ l u l , κ < 3 L , c κ l ∈ {-1, 0, 1} d(x s , ∂Ω), κ = 3 L . (3)
Since each dimension f κ is computation-wise independent, f is suitable for decision forests, which select feature channels κ randomly to split the data during training. Being derived from d(x s , ∂Ω), f inherits the invariance to rigidbody motions. As opposed to the early version of this work [53], we normalize the distances with respect to the averaged edge length of cells, achieving invariance to the body size to a certain extent. However, f is not invariant to pose changes as the contained cells in each layer vary with poses. Although considering geodesic spherical supports instead of Euclidean ones would overcome this issue and yield quasi-invariance to pose changes, the resulting feature would be highly sensitive to topological noise. Thus, we keep the Euclidean supports and let forests take care of the variations caused by pose changes in learning.

CORRESPONDENCES INFERENCE

Now that the features for both surface and volumetric associations, f (v) and f (s), are defined, we proceed on using them to train a regression forest, an ensemble of T binary decision trees, to learn the mapping r : R 3 → R 3 from the observation domain to the template domain. During training each tree learns the split functions that best separate data recursively at branch nodes, while during testing the input point is routed through each tree, reaching T leaves that store statistics as predictions. We discuss in § 5.1 a generic learning framework that applies to both shape parameterizations. A CVT-specific multi-template strategy is presented in § 5.2 to generalize the Vitruvian framework from single mesh connectivity to multiple ones.

Training and prediction

Broadly speaking, training a regression forest amounts to determining the following components: sample-label pairs, split functions, learning objectives and leaf-node statistical models. Readers are referred to [39] for a comprehensive analysis on different choices of these components.

Training data and split functions

First we elaborate the training scenario for surface representations. Since forests aim to map an observed 3D vertex back to the template domain ∂Ω X , usually chosen to be in the rest (T or A) pose, it requires meshes in various poses but with the same connectivity for training. To incorporate abundant training variations, we animate the template X 0 = x 0 v ⊂ ∂Ω X to a variety of poses with a method similar to [START_REF] Sumner | Deformation transfer for triangle meshes[END_REF]. After voxelizing all animated meshes, we associate each surface voxel to their locations at the rest pose, obtaining a pool of sample-label pairs D = {(s v , x 0 v )}. Each tree is trained with a randomly bootstrapped subset of D. While the split function may be arbitrarily complex, a typical choice is a stump where one single dimension κ is compared to a threshold τ , i.e. axis-aligned thresholding. Our splitting candidate φ is hence the pair of testing channels κ and thresholds τ , φ = (κ, τ ), where κ is represented by offset pairs ψ and binary variables η in § 4.1. Let D N denotes the samples arriving at a certain branch node. The training process is to partition D N recursively into two subsets D L and D R , based on randomly generated φ:

D L (φ) = {s v ∈ D N |f κ (s v ) = ε(s v ; R η (ψ)) ≥ τ }, (4a) D R (φ) = {s v ∈ D N |f κ (s v ) = ε(s v ; R η (ψ)) < τ }. (4b)
Similarly, given a set of CVTs corresponding to the template volumes Ω X deformed in various poses, we associate each cell s ∈ S X to its locations in the rest pose, denoted as x 0 s ∈ X 0 ⊂ Ω X , forming a pool of sample-label pairs D = (s, x 0 s ) as the dataset. The split candidate φ is again the pair of thresholds and feature attributes, φ = (κ, τ ), where features are instead computed according to Eq. 3 but the thresholding criteria in Eqs. 4a and 4b follows.

Learning objectives and leaf predictions

At branch nodes, many candidates φ are randomly generated and the one that maximizes the information gain I , φ * = argmax φ I (φ), is stored for the later prediction use. We follow the classic definition of information gain:

I (φ) = H(D N ) - i∈{L,R} |D i (φ)| |D N | H(D i (φ)), (5)
This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication. The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

where H is the entropy, measured as the variance in Euclidean space, i.e. H = σ 2 for both parameterizations. The tree recursively splits samples and grows until one of the following stopping criteria is met: (1) it reaches the maximum depth, or (2) the number of samples |D N | is too small. A Mean-Shift clustering [55] is performed in a leaf node to represent the distributions of x 0 as a set of confidenceweighted modes H = {(h, ω)}. h ∈ R 3 is the mode location and ω is a scalar weight.

In the prediction phase, a 3D input point i ∈ V Y or i ∈ S Y traverses down the trees and lands on T leaves containing different collections of modes: {H 1 • • • H T }. The final regression output r i is the cluster centroid with largest weight obtained by performing Mean-Shift [55] on them. Each observed point then gets a closest point p in the reference shape X 0 , either in surfaces, p = argmin v∈V X r ix 0 v 2 , or in CVTs, p = argmin s∈SX r ix 0 s 2 . The correspondence pair (i, p) serves as input to the subsequent deformation framework described in § 6.

Outliers such as false geometries, or un-removed background elements often exist in 3D data, drastically deteriorating tracking results. If their models are available, we also include them in the training process, so that forests can identify and reject them online. In this case, the goodness of a split φ is evaluated in terms of both classification and regression. We follow Fanelli et al. [START_REF] Fanelli | Random forests for real time 3d face analysis[END_REF] and extend the entropy to be:

H(D) = - c p(c|D) log p(c|D) + (1 -e δ α )σ 2 (D), (6)
where p(c|D) is the class probability of being foreground or background. It is the weighted sum of the aforementioned regression measure σ 2 and the classification entropy measure. Forests trained with Eq. 6 are often referred to as Hough forests. During training it learns simultaneously (1) how to distinguish between valid and invalid samples (outliers) and (2) how to match valid samples to the template. The regression part gets increasing emphasis when the current depth δ gets larger (i.e. the tree grows deeper), and the steepness is controlled by the parameter α.

Learning across multiple volumetric templates

So far we know how to utilize Vitruvian-based learning framework to match surface or volumetric data against the template. For the training purposes, one has to deform the reference mesh into various poses such that all meshes share a consistent topology T X and one can easily assign each sample a continuous label which is its rest-pose position X 0 . In this regards, the trained forest applies only to one mesh connectivity T X . Nevertheless, the amount of training data for one single template is often limited. To avoid over-fitting, the rule of thumb is to incorporate as much variation as possible into training. This motivates us to devise an alternative that learns across different template connectivities T X . Due to the high memory footprint of regular voxel grids, this strategy is unfortunately less practical for the surface feature f (v) in § 4.1 and we implement it only with CVTs.

Given U distinct CVT templates:

{S µ } U µ=1
3 , whose temporal evolutions are recovered with the method in [51],

3. The template suffix X is dropped to keep notations uncluttered. resulting in a collection of different templates deformed in various poses:

{{X t 1 } • • • {X t U }} as our dataset.
To include all of them into training, we take one generic template Ŝ as the reference. Intuitively, if there exists a mapping g that brings each cell s ∈ S µ to a new cell g(s) = ŝ ∈ Ŝ, one only needs to change the template-specific labels x 0 s to the corresponding x 0 ŝ , which are common to all templates, and the training process above can again be applied. In other words, we align topologies by matching every template S µ to Ŝ. Fig. 7 depicts this multi-template learning scheme.

Although various approaches for matching surface vertices exist, only a handful of works discuss matching voxels/cells. Taking skinning weights [START_REF] Lewis | Pose space deformation: a unified approach to shape interpolation and skeleton-driven deformation[END_REF] as an example, we demonstrate in the following how to adapt a surface descriptor to CVTs. Note that our goal is not to propose a robust local 3D descriptor. With proper modifications, other descriptors can be used as well for shape matching.

Generalized skinning weights.

Skinning weights are originally used for skeleton-based animations, aiming to blend the transformations of body parts (bones). Usually coming as a side product of the skeletonrigging process [START_REF] Baran | Automatic rigging and animation of 3d characters[END_REF], it is a vector w of B -dimensions, each corresponding to a human bone b and B is the number of bones. The non-negative weight w b indicates the dependency on that part and is normalized to sum up to one, i.e.

b w b = 1. As such, a skinning weight vector w is actually a probability mass function of body parts, offering rich information about vertex locations. To extend it from surface vertices to CVT cells, we first relax the unity-summation constraint as w is not used to average transformations of bones but only as a descriptor here. The intuition behind the adaptation is that, a CVT cell should have bone dependencies similar to the closest surface point. Therefore, for a cell whose normalized distance to the surface is d, its skinning weight is simply the one of its closest surface point, scaled by e d . We tackle scale changes by normalizing the distance field with the averaged edge length of cells in the shape. Since the shortest distance usually hits a triangle rather than a single vertex, we use barycentric coordinates as the coefficients to linearly combine the skinning weights of the three vertices. Note that this does not violate the unity-summation constraint for surface vertices as their distance d is still zero. We illustrate this concept in Fig. 8(a). The mapping g is then determined by searching for the nearest neighbor in the skinning weight space: g(s) = arg min ŝ∈ Ŝ w ŝw s 2 .

In practice, we use Pinocchio [START_REF] Baran | Automatic rigging and animation of 3d characters[END_REF] to computes skinning weights, extend them from surface vertices to CVT This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.

The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org. cells, and match all cells to those of the common template Ŝ.

The resulting skeletons are not used in our method. Fig. 8(b) visualizes one example of matching results. Our approach yields reasonable matches, regardless of the difference in body sizes. Due to the descriptiveness of skinning weights, symmetric limbs are not confused. Note that this computation is performed only between user-specific templates S µ and the generic one Ŝ off-line once. Input data S Y cannot be matched this way, because rigging a skeleton for shapes in arbitrary poses remains a challenging task.

TRACKING

Recall that our goal is not only to detect the associations C but eventually to estimate the deformation parameter Θ via Θ = argmin Θ E(Θ; C ), such that the resulting X( Θ) best explains Y. The choice of Θ could be raw point positions [START_REF] Sorkine | Laplacian surface editing[END_REF], [START_REF] Zollh Öfer | Real-time non-rigid reconstruction using an rgb-d camera[END_REF], skeletal kinematic chains [4], [START_REF] Ye | Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera[END_REF] and cage [START_REF] Duveau | Cagebased motion recovery using manifold learning[END_REF]. We opt for a patch-based deformation framework [2] for surfaces and a CVT cluster-based method [51] for volumetric meshes respectively. Both group the 3D points into a higher-level structure, where shape deformations are represented as the ensemble of their rigid-body motions θ. We briefly explain here the basic principles and how to apply the predicted correspondences in § 5 to track a sequence of temporally inconsistent observations.

Surface-based deformation

In [2], the reference surface is decomposed into several patches k. It serves as a intermediate deformation structure between vertex positions and anatomical skeletons. Without any prior knowledge of motion, patches are preferred to be distributed uniformly over X . Given correspondences C from above, a data term is formulated as:

E data (Θ; C ) = (i,p)∈C w ip y i -x p (Θ) 2 2 , (7)
which is a standard sum of weighted squared distances. Since evolving a surface with discrete observations (even with a good C) is ambiguous by nature, regularization terms are usually introduced to exert soft constraints. Given a vertex v, the rigidity constraint enforces the predicted positions x v (θ k ) and x v (θ l ) from two adjacent patches P k and P l ∈ N k to be consistent:

E r (Θ) = K k=1 P l ∈N k v∈P k ∪P l w kl x v (θ k ) -x v (θ l ) 2 2 , ( 8
)
where Θ is implicitly encoded in x v (θ k ) and x v (θ l ).

Given a fixed input Y, the regression forest returns a fixed response Ỹ, and in turn a fixed C . We therefore apply standard Gauss-Newton method directly to find the minimizer of final energy: E(Θ; C ) = λE data (Θ; C ) + E r (Θ), where λ defines the softness of the template and is empirically set to 10 throughout our experiments. Note anyway that refining C like non-rigid ICP does is always possible. In this case, our method provides better initializations than using last frame results, reducing the needed ICP-iterations.

Volumetric deformation

On the other hand, a similar deformation framework can be formulated for CVTs as well, only that surface patches k are replaced by clusters of cells. We follow [51] which is essentially a non-rigid ICP method. As opposed to the extensive correspondence search, we again directly use the association pairs (i, p) detected by the forest as initializations. This results in a faster pose estimation.

EXPERIMENTS

The presented method is evaluated extensively in this section. We verify the merits of the discriminative associations as well as the complete 3D tracking-by-detection pipeline, in both surface and CVT parameterizations. As summarized in Table 1 in the supplemental material, in total 15 datasets are considered for various evaluation purposes. Due to the availability of ground-truths, the input in § 7.1 is the nonrigid registration, whereas in § 7.2 it is the reconstructed visual hull from [38] or raw tessellated CVT from [START_REF] Wang | A hierarchical approach for regular centroidal Voronoi tessellations[END_REF].

Discriminative associations

Recall that the goal of discriminative correspondences is to guide the shape deformation not to match non-rigid 3D shapes accurately. We aim only to show that (1) the presented features are more or at least equally informative for matching humanoid surfaces than the existing state-of-thearts 3D descriptors, e.g. Heat Kernel Signature (HKS) [30], [START_REF] Bronstein | Scale-invariant heat kernel signatures for non-rigid shape recognition[END_REF] or Wave Kernel Signature (WKS) [31] and (2) CVTbased associations are more reliable than the surface-based counterparts. We describe every vertices with HKS, WKS, and our pair-wise offset features f (v) in § 4.1. CVT cells are, on the other hand, described by the Haar-like spherical features f (s) in § 4.2. The forests learn to match these 3D primitives against their own learning template, either a generic reference surface (FAUST) or a subject-specific CVT template (Goalkeeper, Ballet and Thomas).

Surface-based correspondences

Surface correspondences are validated on the publiclyavailable dataset FAUST [START_REF] Bogo | FAUST: Dataset and evaluation for 3D mesh registration[END_REF]. Following [16], [34], we use only the training set because of the availability of groundtruth vertex indices. It comprises 100 static 3D scans from 10 subjects in 10 poses. The accuracy on FAUST indicates how well the proposed method deals with human shape variations. Specifically, half the registrations (50 meshes) are chosen to train T = 50 trees and the other half are left out for testing. At branch nodes, 5000 splitting candidates φ are randomly generated and the best one is stored. The error measure is the geodesic distance between predicted vertices This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.

The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org. and ground-truths. If the distance is smaller than a certain threshold, we consider the point correctly matched. The percentage of correct matches in varying thresholds characterizes the performance of one algorithm and is commonly used in many matching papers [16], [34].

The results are shown in Fig. 10, where x-axis is normalized by the averaged edge length of the template. We partition the 100 meshes in two ways to test the generalization to unseen shapes or poses. The keyword pose means that the forest is trained with meshes in all 10 subjects but in only 5 poses, whereas shape represents the opposite. To compare fairly with other existing methods, we keep the Vitruvianmanifold label space unchanged (i.e. the same learning template) while replacing the voxel-based features with 30dimensional scale-invariant HKS or WKS feature vectors. The proposed TSDT-forest combination yields overall best accuracy in Fig. 10, suggesting that the voxel-based TSDT feature is indeed more informative than H/WKS in the chosen parameter range. Comparing the blue solid curve to the dashed one, we notice that our approach handles unseen shapes better than unseen poses. This is due to the fact that our feature relies mainly on 3D geometry, in which pose variations cause more significant changes than shape variations. Although this phenomenon is not observed in the curves of H/WKS because they exploit the spectral domain for better pose invariance, they suffer from the confusion between symmetric parts as visualized in Fig. 9.

We further visualize in Fig. 11 the predicted associations on noisy reconstructed visual hulls with outliers, where no ground truths are available. Black colors means that the predicted correspondences are either rejected by simple normal compatibility check [2] like those on the body, or rejected because they are recognized as the chair. In this experiment, we include chair meshes into training data and follow Eq. 6 as the entropy measure to grow the trees. As a result, we can identify observations on the chair online and remove them, so that they do not affect the subsequent tracking stage. The task of trees here is throwing away the points of known outlier classes and in the meanwhile also predicting correspondences for the remaining points.

As one can see, our approach is capable of predicting reasonable associations for noisy visual hulls while rejecting outliers. This is of importance since they are the real input data of the final tracking-by-detection pipeline. HKS and WKS are known for their sensitivity to topological noises, e.g. the merging of arms and torso. We however would like to remark that, as oppose to our feature vector f (v) that has a dimensionality virtually longer than 5000 from the randomly generated splitting candidates at each branch node, HKS and WKS are only 30-dimensional in our experiment. To fully conclude that the presented voxel-based feature is certainly better than HKS or WKS requires a more fair and thorough comparison but is not the main goal of this paper.

Volumetric correspondences

The discriminative CVT-based correspondence in § 4.2 is validated with 6 sequences from 3 subjects: Goalkeeper, Ballet and Thomas. We register each template to the corresponding raw CVT sequences using a EM-ICP based This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.

The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org. Training- [12] wo re-ori.

Testing- [12] wo re-ori.

Training- [12] w re-ori.

Testing- [12] w re-ori.

(b) Ballet method [51] to recover temporal coherent volumetric deformations (tracked CVTs). For each subject, up to 250 tracked CVTs are randomly chosen from the first sequence to form the training set, while the second sequences are completely left out for testing. We open L = 8 sphere layers for the feature computation. Each tree is grown up to depth 20 with 30% bootstrap samples randomly chosen from the dataset. The contributions of CVT on improving the correspondences detection are evaluated in two aspects. First, we keep using the Vitruvian manifold ∂Ω as the regressing domain but replace the voxel-based features f (v) with the spherical feature f (s), denoted as CVTfeature. Next, we further change the label space from surfaces ∂Ω to volumes Ω, termed fullCVT. We test on the tracked CVTs and report the results on all frames of both training sequences (Tr) and testing ones (Te). The drop between them indicates the ability to generalize. The same error measure as in the previous subsection is applied, only the geodesic distances are replaced by Euclidean ones. To yield a fair comparison with [11], here the forests are subject-specific and consist of T = 20 trees. Fig. 13 shows the percentage of correct matches in varying thresholds for Thomas and Ballet. Since CVTfeature and [11] are regressing to surfaces whereas fullCVT regresses to volumes, we normalize the x-axis by the average edge length of templates to yield fair comparisons. While the results of CVTfeature are comparable to [11] (green vs. red or orange), fullCVT attains the improved accuracies (blue vs. red or green), demonstrating the advantages of our fully volumetric framework. Some visual results of the fullCVT approach on raw CVT input are shown in Fig. 12.

It is worth a closer analysis to highlight the advantages of CVT-based feature f (s) against the voxel-based one f (v). Our early work [11] applied f (v) that takes 150 3 voxels for f (v) to describe a human shape, while CVT needs only 5k cells 4 . Consequently, [11] is not able to include a sufficient amount of training shapes, leading to a major drawback that forests are limited to one single subject. To further decrease the needed number of training meshes, [11] exploits skeletal poses to cancel the global orientation. This in turn makes every mesh in the training dataset face the same direction and we learn merely pose variations. It follows that during tracking the input data has to be re-oriented likewise using the estimated skeletal poses from the last frame. The CVTbased feature f (s), on the other hand, considers distance fields of cells which is naturally invariant to rotations and hence does not require re-orientations. We anyway compare to [11] in both settings. Orange curves in Fig. 13 shows the results with re-orientation, which is better than the proposed strategy in Ballet. Nonetheless, without re-orienting data, the accuracy drops substantially during testing (compare red to orange). The efficiency on memory and the invariance of our features are two determining reasons why the presented method is better than [11]. With the multi-template learning strategy in § 5.2, it takes just one forest for different subjects in the tracking-by-detection experiment in § 7.2.

4. Further note that [11] stores a 3D vector in each voxel, whereas we store a scalar in each CVT cell. So the ratio is 3 × 150 3 to 5k. This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.

The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org. Next, we use the sequences of Goalkeeper to verify the merits of this multi-template learning framework, which is unfeasible for voxel-based feature f (v) due to the high memory footprint. It is a particularly difficult dataset because motions in the testing sequence UpJump have little overlap with those in the training SideJump. We report in Fig. 14 the correctness of correspondences for UpJump (unseen during training) in fullCVT setting. Three situations are taken into account: training with the tracked CVTs of all three subjects (red), training only with those from Goalkeeper (blue) and without Goalkeeper (green). For red and green curves, we choose the Goalkeeper template as the common one Ŝ and follow the strategy in § 5.2 to align distinct CVT tessellations. Comparing the red curve to the blue one confirms the advantage of this cross-template approach, leading to a forest that applies to all three subjects without trading off much accuracy. Nonetheless, the training data of the testing subject is still indispensable, as the accuracy drops when there is no tracked CVTs of Goalkeeper (green vs. red or blue), even if the forest of green curve is trained with twice the amount of CVTs compared to the blue curve. This is partially due to the fact that template of Goalkeeper has much smaller size than the other two and suggests that the proposed Haar-like feature in Eq. 3 captures more shape than pose information.

Tracking-by-detection

Now we move on to evaluate the full tracking-by-detection pipeline. The predicted associations C of two parameterizations are fed into their respective shape deformation frameworks in § 6 and the tracking is carried out on a frameby-frame basis. The fidelity of estimated shapes is verified by the widely-used silhouette overlap error.

Surface-based

An individual forest is trained for each subject with up to 200 meshes, depending on the number of vertices in the template. For Baran and Vlasic, we train standard regression forest; for Lionel and Ben we apply the adaptation in Eq. 6 (α = 5) due to the un-properly segmented chairs and tables in input data. Growing T = 20 trees to depth 25 with 5000 testing offset pairs ψ takes about 3 hours. Although it is not the aim of this paper, we anyway augment the energies in § 6.1 with the skeleton energy in [2] and validate the estimated human poses in 2D.

For sequences without outliers, we compare with surface-based ICP (surICP) [2] and articulated ICP (ar-tICP) [15], both of which explain data with GMM using the Expectation-Maximization algorithm. We run an additional ICP step to reduce the errors (ours + ICP) for all testing sequences. The averaged overlap errors are shown in Fig. 15(ad). In general, our method performs much better than artICP and attains comparable results with surICP. However, ICPbased methods often fail when large deformation occurs between consecutive frames, which is usually the case in videos of low frame rates. We simulate this by tracking only every three frames. As reported in Table 1, surICP now yields higher errors because local proximity search fails to estimate correspondences properly, while our approach is able to handle large jumps between successive input.

Four of our testing sequences, Cutting, WalkChair1, Ham-merTable, and WalkChair2 contain tables or chairs in observations, which play the roles as static outliers. We compare with other outlier rejection strategies such as, fixed outlier proportion (fixOL) [2], removing outliers by body-part classifications with SVM (bpSVM) [START_REF] Huang | Robust human body shape and pose tracking[END_REF], and modeling outlier likelihood dynamically by aggregating over all patches (patchedOL) [3]. As shown in Fig. 15(e-h), conventional outlier strategy fixOL drifts quickly and soon fail to track (green curves). ICP with robust outlier treatment, patchedOL, is able to sustain noisy input to a certain extent. Once it starts drifting, the error only gets higher due to its ICP nature (yellow curves). When subjects and outliers are sperate components in visual hulls, we cast them into separately TSDT, and feed them into the joint classification-regression forest. If they are connected to each other, forests inevitably associate some outliers to the humanoid template, leading to undesirable deformations as suggested by the spike in blue curves in Fig. 15(f). Nonetheless, since we rely less on previous frames for data associations, the results can always get recovered when they are separate again. In average, we still yield low errors throughout the whole sequences. We remark that such ability to recover is the essence of our discriminative approach, which is the biggest advantages over the existing generative methods. The recovered shapes and poses, superimposed on original images, are also presented in Fig. 2(c) in the supplementary material.

Fully volumetric tracking-by-detection

After evaluating the surface-based tracking-by-detection framework, now we turn to evaluate the volumetric one. We compare in two quantitative metrics against the whole pipeline in [11], which is the early version of our surfacebased tracking-by-detection approach.

Unlike the matching experiment in the previous subsection, here we apply the multi-template strategy in § 5.2 to train one universal regression forest, with Goalkeeper chosen as the common template Ŝ. Training T = 50 trees up to depth 20 where each one is grown with around 200 CVTs (approximately one million samples) takes about 15 hours on a 24-core Intel Xeon CPU machine. For each subject, we track the testing sequence, which is not part of the training set. Tracking inputs are raw CVTs that have no temporal coherence. The number of clusters K is 250 for Ballet and Goalkeeper and 150 for Thomas. We evaluate our tracking approach with two different metrics. On one hand, evaluation with marker-based motion capture evaluates the correctness of the surface pose, but only for a sparse set of surface points. On the other hand, the silhouette overlap error evaluates the shape estimate but not the estimated pose. Hence these metrics are complementary. Some visual results are shown in Fig. 3 in the supplemental material and video 5 . Our approach is able to discover volumetric associations even in challenging poses found in Thomas and deform the templates successfully. As shown in Table 2-4, we evaluate the results by computing the overlap error between the ground truth silhouette and the projection of the estimated surface. The metric we use is the pixel error (number of pixels that differ). Statistics are computed on all frames of all cameras. The Ballet/Seq2 sequence has markerbased motion capture data: fifty markers were attached to the body of the subject, providing a sparse ground truth for surface tracking. First, each marker is associated to a vertex of the template surface. Then, for each marker, we measure the distance between its location and the estimated vertex location. Statistics on the distance are reported on Table 5. We observe that our approach attains slightly better performances than a state of the art ICP-based approach [51] and outperforms a surface-based tracking-by-detection [11] which mostly fails to correctly register the legs of the subject.

Discussion

Last but not lease, we make a short comparison between the two presented features. As discussed above, voxel-based volume in § 4.1 has the downside of high memory footprint, which limits the allowed training variations. Aligning the orientations is one way to reduce the training variation such that forests only need to learn the pose variations of one single subject. One has to repeat the same thing likewise during the testing phase. In [11], we rely on the skeletal poses of previous frames for this purpose and thus the forest predictions are not fully frame-independent, exposing tracking subject to the potential risk of drifting. To facilitate a fully 3D tracking-by-detection framework, the information of previous frames is preferred no to participate in the discriminative correspondence estimation. On the other hand, the spherical feature presented in § 4.2 attempts to incorporate rotational, pose, and even shape variations during training, yielding completely frame-wise forest predictions.

As reported in Fig. 13, without aligning rotations, the accuracies of correspondences drop substantially on the testing sequences for the method in [11]. This means that voxel-based framework and the corresponding features do not generalize well to unseen rotations. When deployed in tracking applications, such unreliable associations eventually result in failure. In particular, one can observe in Table 4 that [11] attains high silhouette overlap discrepancy, most likely due to the fact that the subject rotates himself in many orientations and thus confuses the forest. From these observations, we conclude that the CVT-based Haar-like feature and the derived fully volumetric tracking-by-detection framework is better than the voxel-based counterpart.

CONCLUSION

In this paper, we present two features for surface and CVT shape parameterizations respectively, both making use of volumetric distance fields to describe 3D geometry. Aiming to integrate with random forests, each feature attribute is computationally independent and can be obtained on the fly in testing. They facilitate the surface-based and CVT-based discriminative associations and in turn lead to the corresponding tracking-by-detection frameworks for 3D human shapes. While CVT-based approach is more robust to the surface counterpart, we show that both yield more stability compared with the respective generative ICP extensions. The reliability of the proposed method is confirmed by the experiments on numerous public sequences. Future directions include alleviating problems of topological changing and incorporating photometric information. Nassir Navab is a professor of computer science and founding director of Computer Aided Medical Procedures and Augmented Reality (CAMP) laboratories at Technische Universitt Mnchen (TUM) and Johns Hopkins University (JHU). He also has secondary faculty appointments at TUM and JHU medical schools. He received the Ph.D. degree from INRIA and University of Paris XI, France, and enjoyed two years of postdoctoral fellowship at MIT Media Laboratory before joining Siemens Corporate Research (SCR) in 1994. At SCR, he was a distinguished member and received the Siemens Inventor of the Year Award in 2001. In 2012, he was elected as a fellow member of MICCAI society. He received the 10 year lasting Impact Award of IEEE ISMAR in 2015. He holds 45 granted US patents and over 40 European ones. He has served on the program and organizational committee of over 80 international conferences including CVPR, ICCV, ECCV, MICCAI, IPCAI, and ISMAR. He has also been co-author of more than 20 awarded papers in international conferences. His current research interests include robotic imaging, computer aided surgery, computer vision and augmented reality.

Chun-Hao

Slobodan

Abstract

In this paper, we tackle the problem of 3D human shape estimation from single RGB images. While the recent progress in convolutional neural networks has allowed impressive results for 3D human pose estimation, estimating the full 3D shape of a person is still an open issue. Model-based approaches can output precise meshes of naked under-cloth human bodies but fail to estimate details and un-modelled elements such as hair or clothing. On the other hand, non-parametric volumetric approaches can potentially estimate complete shapes but, in practice, they are limited by the resolution of the output grid and cannot produce detailed estimates. In this work, we propose a non-parametric approach that employs a double depth map to represent the 3D shape of a person: a visible depth map and a "hidden" depth map are estimated and combined, to reconstruct the human 3D shape as done with a "mould". This representation through 2D depth maps allows a higher resolution output with a much lower dimension than voxel-based volumetric representations. Additionally, our fully derivable depth-based model allows us to efficiently incorporate a discriminator in an adversarial fashion to improve the accuracy and "humanness" of the 3D output. We train and quantitatively validate our approach on SURREAL and on 3D-HUMANS, a new photorealistic dataset made of semi-synthetic in-house images annotated with 3D ground truth surfaces.

Introduction

Recent works have shown the success of deep network architectures for the problem of retrieving 3D features such as kinematic joints [4,33] or surface characterizations [43] from single images, with extremely encouraging results. Such successes, sometimes achieved with simple, standard network architectures [30], have naturally motivated Figure 1. Our non-parametric representation for human 3D shape: given a single image, we estimate the "visible" and the "hidden" depth maps from the camera point of view. The two depth maps can be seen as the two halves of a virtual "mould". We show this representation for one of the images of our new dataset. the applicability of these methodologies for the more challenging problem of end-to-end full 3D human shape retrieval [2,18]. The ability to retrieve such information from single images or videos is relevant to a broad number of applications, from self-driving cars, where spatial understanding of surrounding obstacles and pedestrians plays a key role, to animation or augmented reality applications such as virtual change rooms that can offer the E-commerce industry a virtual fitting solution for clothing or bodywear.

Designing a deep architecture that produces full 3D shapes of humans observed in an input image or a sequence of input images raises several key challenges. First, there is a representational issue. While the comfort zone of CNNs is in dealing with regular 2D input and output grids, the gap must be bridged between the 2D nature of inputs and the 3D essence of the desired outputs. One solution is to follow a parametric method and estimate the deformation parameters of a predefined human 3D model [2,18]. These methods are limited to the level of details covered by the model. In contrast, non parametric approaches can potentially account for shape surface details but are prone to produce physicallyimpossible body shapes. This is the case of the recent volumetric approach proposed in [38] that encodes the human body as a voxel grid whose dimensionality directly impacts 1 Figure 2. Overview. Given a single image, we estimate the "visible" and the "hidden" depth maps. The 3D point clouds of these 2 depth maps are combined to form a full-body 3D point cloud, as if lining up the 2 halves of a "mould". The 3D shape is then reconstructed using Poisson reconstruction [19]. An adversarial training with a discriminator is employed to increase the humanness of the estimation.

the precision of the estimation. This highlights a second challenge: the dimensionality of the problem is considerably higher than what existing networks have been shown to handle, because the parametrisation sought is no longer restricted to a subset of the variability, e.g. kinematic pose of humans or body shape parameters, but to an intrinsically finer description of the body. Finally, the training data for this problem, yet to be produced, requires a particularly demanding definition and acquisition effort. The large data variability of 3D problems has motivated some initial efforts to produce fully synthetic training sets [39], where such variability can be partially scripted. Yet recent successful methods underscore the necessity for realistic data, for both the general applicability of the estimation, and to keep the underlying network architecture simple, as devoid as possible of any domain specific adaptations.

In order to overcome these difficulties, we propose a non-parametric approach that employs a double depth map representation to encode the 3D shape of a person: a "visible" depth map capturing the observable human shape and a "hidden" depth map representing the occluded surface are estimated and combined to reconstruct the full human 3D shape. In this encoding of the 3D surface, the two depth maps can be seen as the two halves of a virtual "mould", see Figure 1. This representation allows a higher resolution output, potentially the same as the image input, with a much lower dimension than voxel-based volumetric representations, i.e. O(N 2 ) vs O(N 3 ). We designed an encoderdecoder architecture that takes as input a single image and simultaneously produces an estimate for both depth maps. These depth maps are then combined to obtain a point cloud of the full 3D surface which can be readily reconstructed using Poisson reconstruction [19]. Importantly, our fully differentiable depth-based model allows us to efficiently incorporate a discriminator in an adversarial fashion to improve the accuracy and "humanness" of the 3D output, especially in the case of strong occlusions. See Figure 2. To train and quantitatively evaluate our network in near real-world conditions, we captured a large-scale dataset of textured 3D meshes that we augment with realistic backgrounds. To account for the large variability in human appearance, we took special care in capturing data with enough variability in movements, clothing and activities. Compared to parametric methods, our method can estimate detailed 3D human shapes including hair, clothing and manipulated objects.

After reviewing the related work in Section 2, we present our two-fold contribution: our new non-parametric 3D surface estimation method is explained in Section 3 while our large-scale dataset of real humans with ground-truth 3D data is detailed in Section 4. Experiments are presented in Section 5 and conclusions drawn in Section 6.

Related Work

3D object from single images. Various representations have been adopted for 3D object shape estimation. Voxelbased representations [5] consist in representing the 3D shape as an occupancy map defined on a fixed resolution voxel grid. Octree methods [36] improve the computability of volumetric results by reducing the memory requirements. Point-clouds are another widely employed representation for 3D shapes. In [7], Fan et al. estimate sets of 1024 points from single images. Jiang et al. [15] build on this idea and incorporate a geometric adversarial loss (GAL) to improve the realism of the estimations. AtlasNet [10] directly estimates a collection of parametric surface elements to represent a 3D shape. Our representation combines two complementary depth maps aligned with the image, similar in spirit to the two halves of a "mould", and shares the resolution of the input image, capturing finer details while keeping output dimensionality reasonable.

Similarly to the work of Tatarchenko et al. [35] on reconstructing vehicle images from different viewpoints, we combine the estimation of several depth maps to obtain a 3D shape. For human shape estimation, however, we work on a deformable object. Also, we focus on the visible and hidden depth maps rather than any other because of their direct correspondence with the input image. Our two depth maps being aligned with the image, details as well as contextual image information are directly exploited by the skip connections to estimate the depth values. Multi-views [35] do not necessarily have pixel-to-pixel correspondences with the image making depth prediction less straightforward. 3D human body shape from images. Most existing methods for body shape estimation from single images rely on a parametric model of the human body whose pose and shape parameters are optimized to match image evidence [2,11,20,29]. This optimization process is usually initialised with an estimate of the human pose supplied by the user [11] or automatically obtained through a detector [2,20,29] or inertial sensors [42]. Instead of optimizing mesh and skeleton parameters, recent approaches proposed to train neural networks that directly predict 3D shape and skeleton configurations given a monocular RGB video [37], multiple silhouettes [6] or a single image [18,25,28]. Recently, BodyNet [38] was proposed to infer the volumetric body shape through the generation of likelihoods on the 3D occupancy grid of a person from a single image.

A large body of work exists to extract human representations from multiple input views or sensors, of which some recently use deep learning to extract 3D human representations [8,13,21]. While they intrinsically aren't designed to deal with monocular input as proposed, multi-view methods usually yield more complete and higher precision results as soon as several viewpoints are available, a useful feature we leverage for creating the 3D HUMANS dataset.

More similar to ours are the methods that estimate projections of the human body: in [39], an encoder-decoder architecture predicts a quantized depth map of the human body while in DensePose [12] a mapping is established between the image and the 3D surface. Our method also makes predictions aligned with the input image but the combination of two complementary "visible" and "hidden" depth maps leads to the reconstruction of a full 3D volume. In [24], the authors complete the 3D point cloud built from the front facing depth map of a person in a canonical pose by estimating a second depth map of the opposite viewpoint. We instead predict both depth maps simultaneously from a single RGB image and consider a much wider range of body poses and camera views. All these methods rely on a parametric 3D model [2,18,20] or on training data annotated [12] or synthesised [39] using such a model. These models of humans built from thousands of scans of naked people such as the SMPL model [23] lack realism in terms of appearance. We instead propose to tackle real-world situations, modeling and estimating the detailed 3D body shape including clothes, hair and manipulated objects. 3D human datasets. Current approaches for human 3D pose estimation are built on deep architectures trained and evaluated on large datasets acquired in controlled environments with Motion Capture systems [1,14,34]. However, while the typology of human poses on these datasets captures the space of human motions very well, the visual appearance of the corresponding images is not representative of the scenarios one may find in unconstrained real-world images. There has been a recent effort to generate in-thewild data with ground truth pose annotation [26,32]. All these datasets provide accurate 3D annotation for a small set of body keypoints and ignore 3D surface with the exception of [20] and [41] who annotate the SMPL parameters in real-world images manually or using IMU. Although the resulting dataset can be employed to evaluate under-cloth 3D body shapes, its annotations are not detailed enough, and importantly, its size is not sufficient to train deep networks.

To compensate for the lack of large scale training data required to train CNNs, recent work has proposed to generate synthetic images of humans with associated ground truth 3D data [4,31,39]. In particular, the Surreal dataset [39], produced by animating and rendering the SMPL model [23] on real background images, has proven to be useful to train CNN architectures for body parts parsing and 2.5D depth prediction [39], 3D pose estimation [31,33], or 3D shape inference [38]. However, because it is based on the SMPL model, this dataset is not realistic in terms of clothing, hair or interactions with objects and cannot be used to train architectures that target the estimation of a detailed 3D human shape. We propose to bridge this gap by leveraging multicamera shape data capture techniques [3,40], introducing the first large scale dataset of images showing humans in realistic scenes, i.e. wearing real clothes and manipulating real objects, dedicated to training with full 3D mesh and pose ground-truth data. Most similar to ours are the CMU Panoptic dataset [17] that focus on social interactions and the data of [45] that contains dense unstructured geometric motion data for several dressed subjects.

Methodology

In this section, we present our new non-parametric 3D human shape representation and detail the architecture that we designed to estimate such 3D shape from a single image.

"Mould" representation

We propose to encode the 3D shape of a person through a double 2.5D depth map representation: a "visible" depth map that depicts the elements of the surface that are directly observable in the image, and a "hidden" depth map that characterises the occluded 3D surface. These two depth maps can be estimated and combined to reconstruct the complete human 3D shape as done when lining up the two halves of a "mould". See example in Figure 2.

Given a 3D mesh, obtained by animating a 3D human model or by reconstructing a real person from multiple views, and given a camera hypothesis, i.e. location and parameters, we define our two 2D depth maps z vis and z hid by ray-tracing. Specifically, we cast a ray from the camera origin, in the direction of each image pixel location (u, v) and find the closest intersecting point on the mesh surface:

z vis [u, v] = min k2Ray(u,v) ||p k || 2 (1)
for the visible map, and the furthest one for the hidden map:

z hid [u, v] = max k2Ray(u,v) ||p k || 2 , (2)
where 3D points {p i } = {(p x,i , p y,i , p z,i )} are expressed in camera coordinate system and the L2-norm ||.|| 2 is the distance to the camera center. Ray(u, v) denotes the set of points p i on the ray passing through pixel (u, v) obtained by hidden surface removal and visible surface determination.

To be independent from the distance of the person to the camera, we center the depth values on the center of mass of the mesh, i.e. z orig : z vis [u, v] 0 = z vis [u, v] z orig 8u, v, and similarly for z hid [u, v]. Since they are defined with respect to the same origin, the 2 depth maps z vis [u, v] and z hid [u, v] can be readily combined in 3D space by merging their respective 3D point clouds into a global one:

{p i } = {p i } vis [ {p i } hid (3)
An example of such a point cloud is depicted in Figure 2, where points corresponding to z vis [u, v] and z hid [u, v] are respectively colored in red and blue. In practice, to keep the depth values within a reasonable range and estimate them more accurately, we place a flat background a distance L behind the subject to define all pixels values in the depths maps in the range [ z orig . . . L]. Points p i of the point clouds are then selected as belonging to the human surface if p z,i  L ✏.

As in volumetric representation through voxel grid, our method also encodes 3D surfaces and point clouds of diverse sizes into a fixed size representation, making a 3D surface easier to consider as a deep network target. However, in our case, we can work at the image resolution with a much lower output dimensionality O(N 2 ) than voxel-based volumetric representations O(N 3 ), N being the size of the bounding box framing the human in the input image.

We numerically validated the benefit of our representation compared to a voxel grid approach by encoding a random set of 100 meshes (picked from our 3D HUMANS dataset presented in Section 4) at different resolutions and computing the 3D reconstruction error (average Chamfer Figure 3. Reconstruction error for voxel grid and our "mould" when augmenting the dimensionality D of the representation, D=N 3 for voxels grid and D=2N 2 for ours. distance) between ground-truth vertices and the resulting point clouds. This comparison is shown in Figure 3. The error obtained with our mould-representation decreases and converges to a minimum value that corresponds to surface details that cannot be correctly encoded even with high resolution depth maps, i.e. when some rays intersect more than twice with the human surface for particular poses. In practice, we show in Section 5 that this can be solved by employing a Poisson reconstruction to obtain a smooth 3D surface, including those areas. We can extrapolate from Figure 3 that voxel grids can reach perfect results with an infinity of voxels, but for manageable sizes, our representation allows to capture more details.

Architecture

We formulate the 3D shape estimation problem as a pixel-wise depth prediction task for both visible and hidden surfaces. Our framework builds on the stacked hourglass network proposed by Newell et al. [27] that consists of a sequence of modules shaped like an hourglass, each taking as input the prediction from the previous module. Each of these modules has a set of convolutional and pooling layers that process features down to a low resolution and then upsample them until reaching the final output resolution. This process, combined with intermediate supervision through skip connections, implicitly captures the entire context of the image. Originally introduced for the task of 2D pose estimation and employed later for part segmentation and depth prediction [39], this network is an appropriate choice as it predicts a dense pixel-wise output while capturing spatial relationships associated with the entire human body.

We designed a 2-stack hourglass architecture that takes as input an RGB image I cropped around the human and outputs the 2 depths maps z vis and z hid aligned with I. We use a L L1 loss function defined on all pixels of both depth maps. The loss function to be minimized is thus the average distance between the ground truth z p and the estimation b z p :

L L1 = 1 P P X p=1 |z p b z p |, (4)
with P being the number of pixels in the batch and b z p the network output for pixel p, including pixels in both z vis [u, v] and z hid [u, v] maps.

We also experimented with an L L2 loss but found that it overly penalizes outliers, i.e. pixels incorrectly assigned to background and vice versa, and therefore focuses only on that task. By using the L L1 norm, we force the network to not only segment the image correctly, i.e discriminate the subject from the background, but also provide an accurate estimation of the depth at each pixel.

Adversarial training

As observed with other non-parametric methods [38] but also with approaches relying on a model [18], our network can sometimes produce implausible shapes that do not look human, especially when a limb is entirely occluded by other parts of the body. To improve the accuracy and the "humanness" of our prediction, we follow an adversarial training procedure inspired by the Generative Adversarial Networks (GAN) [9]. Our fully derivable depth-based model allows us to efficiently incorporate a discriminator in an adversarial fashion, i.e., the goal for the discriminator will be to correctly identify ground truth depth maps from generated ones. On the other hand, the generator objective will be two-fold: fitting the training set distribution through the minimization of the L L1 loss (Equation 4) and tricking the discriminator into classifying the generated depth maps as ground truth depth maps through the minimization of the L GAN loss:

L GAN (G, D) = E I,z [log D(I, z)]+E I [log(1 D(I, G(I))].
(5) Our discriminator D will be trained to maximize the L GAN loss by estimating 1 when provided with ground-truth depth maps z and estimating 0 when provided with generated depth maps G(I). In order to weigh the contribution of each loss, we will use a factor , our full objective being modeled as a minimax game:

(G ⇤ , D ⇤ ) = arg min G max D (L GAN (G, D) + L L1 (G)).
(6) The L L1 loss will be used to learn the training set distribution by retrieving the low-frequency coefficients while the L GAN loss will entice the generator into predicting realistic and precise depth maps. It is important to note that the discriminator is only used to guide the generator during the learning. The discriminator is not used at test time.

The architecture employed as our discriminator is a 4 stack CNN. Each stack is composed of a convolutional layer (kernel size 3, stride 1), a group normalization layer (32 groups), a ReLu activation function and a MaxPool 2x2 operation. There are 64 channels for the first convolution and the number of channels is multiplied by 2 at each stack until reaching 512 for the 4th and last stack convolution. We then connect our 8x8x512 ultimate feature map with 2 fullyconnected layers of size 1024 and 512 neurons and then our final output neuron on which we apply a binary cross entropy loss. We jointly trained our generator and discriminator on 50,000 images for 40 epochs. Training is performed on batches of size 8 with the Adam optimizer. Given our small training batch size, we found the use of group norm [44] to be a great alternative to batch norm that was producing training instabilities. The learning rate is kept constant at 1e-4 during the first 20 epochs and is then decreased linearly to zero during the following 20 epochs. In practice, since our L L1 loss is much smaller than the L GAN loss, we multiply the L L1 loss by a factor equal to 1e4. With this adversarial training, we observed that the results are sharper and more realistic. In cases of deformed or missing limb, e.g. the legs in Figure 7 right, the use of a discriminator forces the generator to produce a better prediction.

Dataset generation

We introduce 3D HUMANS (HUman Motion, Activities aNd Shape), a realistic large-scale dataset of humans in action with ground-truth 3D data (shape and pose). It consists of semi-synthetic videos with 3D pose and 3D body shape annotations, as well as 3D detailed surface including cloths and manipulated objects. First, we captured 3D meshes of humans in real-life situations using a multi-camera platform. We then rendered these models on real-world background scenes. See examples in Figure 4a. Capture. We employed a state of the art 3D capture equipment with 68 color cameras to produce highly detailed shape and appearance information with 3D textured meshes. The meshes are reconstructed frame by frame independently. They are not temporally aligned and do not share any common topology. We divided the capture into 2 different subsets: in the first one, 13 subjects (6 male and 7 female) were captured with 4 different types of garments (bathing suit/tight clothing, short/skirt/dress, wide cloths and jacket) while performing basic movements e.g., walk, run, bend, squat, knees-up, spinning. In the second subset, 6 subjects, 4 male and 2 female, were captured while performing 4 different activities (talking on the phone, taking pictures, cleaning a window, mopping the floor) in 2 different ways: standing/sitting for talking on the phone, standing/kneeling for taking picture, etc. More than 150k meshes were reconstructed. The dataset was collected at Inria from consenting and informed participants. Rendering. We rendered all our videos at a 320 x 240 resolution using a camera of sensor size 32mm and focal length 60mm. Our videos are 100 frames in length and start with the subject at the center of the frame. For the first frame of the sequence, the subject is positioned at a distance of 8 meters of the camera, with a standard deviation of 1 meter. We used the images of the LSUN dataset [46] for background.

Annotations. We augment our dataset with ground-truth SMPL pose and body parameters. To do so, we use the Hu-man3.6M [14] environment as a "virtual MoCap room": we render the 3D meshes for which we want to estimate the 3D pose within that environment, generate 4 views using camera parameters and background images from the dataset and estimate the 2D/3D poses by running LCR-Net++, an off-the-shelf 3D pose detector particularly efficient on Hu-man3.6M. An optimum 3D pose is then computed using multi-view 3D reconstruction and used as initialization to fit the SMPL model, estimating pose and shape parameters that better match each mesh. The SMPL model is fitted to the point clouds both for naked and dressed bodies. Keeping the body parameters fixed (obtained from fits in minimal clothing) resulted in a lower performance of the baseline when evaluated against ground truth dressed bodies.

Experiments

We analyse quantitatively and compare our approach to the state-of-the-art on two datasets. First, the SURREAL dataset [39], a synthetic dataset obtained by animating textured human models using MoCap data and rendering them on real background images, and our 3D HUMANS dataset introduced in this paper. While SURREAL covers a wider range of movements since it has been rendered using thousands of sequences from [1], our data better covers shape details such as hair and clothing. In the following exper- iments, both training and test images are tightly cropped around the person using subjects segmentation. The smallest dimension of the image is extended to obtain a square image that is then resized to 256x256 pixels to serve as input for our network. Performance is computed on both 128x128 output depth maps as the distance between each ground truth foreground pixel and its corresponding pixel in the predicted depth map. Background depth L is set at 1.5m.

SURREAL

Recent methods [38,39] evaluate their performance on this dataset. First, we evaluated the performance of our architecture when estimating quantized depth values (19+1 for background) through classification as in [39] and our proposed regression method: with a maximum distance to groundtruth of 30mm, the quantity of pixels with a correct depth estimation increase by 5% when using regression instead of classification. Then, we compare in Figure 5a performance against the recent BodyNet voxel grid-based architecture from [38] who also reported numerical performance on SURREAL. Although good 3D performances are reported in the paper, we can see that when evaluating in the image domain, i.e., comparing depth maps, the performance of BodyNet drops. Our method makes 3D estimations aligned with the image and better recover details, outperforming BodyNet quite substantially. The GAN helps increase the "humanness" of the predictions.

3D HUMANS

We consider 14 subjects (8 male, 6 female) for training and the remaining 5 subjects (2 male, 3 female) for test. An interesting aspect of synthetic datasets is that they offer an almost unlimited amount of training data. In our case, the data generation relies on a capture process with a non-negligible acquisition effort. It is therefore interesting to analyze how adding more training data impacts the performance. Our results in Figure 5b show that training our architecture on 50,000 images is sufficient and that using more training images does not improve much the performance. The appearance of our images being quite different from SURREAL data, we first compare the performance of our method when considering different training strategies: training on SURREAL, training on 3D HUMANS, or training on a mix of both datasets. In Figure 6a, we can see that the best performances are obtained when SURREAL images are not used. The appearance of the images is too different and our architecture cannot recover details such as clothes or hair when trained on data obtained by rendering the SMPL model. This is verified by the result depicted in Figure 6b: we outperform, by a large margin, a baseline obtained by fitting the SMPL model on the ground truth meshes, effectively acting as an upper bound for all methods estimating SMPL meshes [2,18]. It shows the inefficiency of these methods to estimate clothed body shape since clothes are not included in the SMPL model. Finally, we analyse in Figure 6c how much our performance varies between front and back depth maps. As expected, we better estimate the visible depth map, but our hidden depth maps are usually acceptable. See examples in Figure 7 and Figure 8. The quality of the 3D reconstructions is remarkable given the low dimensionality of the input. Main failures occur when a limb is completely occluded. In such cases, the network can create non-human shapes. We proposed to tackle this issue by considering an adversarial training that we analyse in the next section. We note a higher performance on 3D HUMANS than on SUR-REAL. We attribute that to several factors including the higher pose variability in SURREAL (some subjects are in horizontal position) and the absence of lighting in 3D HU-MANS. We also analyzed the results on different subsets of the evaluation set @50mm and obtained with/without clothing: 83.30% and 85.43% respectively and with/without object: 79.11% and 84.55% respectively, confirming the nuisance introduced by these elements. 8. Generalisation to previously unobserved data. We apply our pipeline to images with 3D realistically rendered backgrounds (left), and with 3 real-world images from the LSP dataset (right). These poses, in particular the baseball player, have not been seen at training time but our model still generalizes well.

GAN

Severe occlusions (self-or by other elements of the scene) are a limitation of our model that we address with adversarial training. We carried out a dedicated experiment where we artificially generated such occlusions in train/test images to quantify improvements. We obtain a 7% chamfer distance error drop with adversarial training and a clear qualitative improvement which we illustrate in Figure 7. We highlight the differences by showing an error heat-map over a Poisson reconstruction of the point cloud for better visualization. The quantitative gain is limited due to the network sometimes hallucinating plausible limbs far from groundtruth (red hand in the left Figure 7), resulting in higher error than a network without GAN that does not estimate any limb at all. This is because the metric does not evaluate the overall plausibility of the produced estimation.

Generalisation

In order to quantitatively measure its generalisation capability, we have evaluated our network on an additional dataset: instead of static background images, we have rendered the meshes in realistic 3D environments obtained on the internet (examples in Figure 4b). The results (Figure 6d) show that a mix training on both SURREAL and 3D HU-MANS is ideal for generalisation. We suspect that jointly rendering the subject and the 3D background at the same time creates a more realistic image where the subject is more complicated to segment, hence the need for more variability in the training data. We also generated qualitative results for LSP images [16], depicted in Figure 8, and for the DeepFashion dataset [22], shown in Figure 9 where we compare our approach with HMR [18] and BodyNet [38]. We can observe that our approach captures more details, including hair, shirt and the belly of the pregnant woman (up), hair, skirt and body pose (middle) and dress (bottom). [18] (left), Bodynet [38] (middle) and our method (right). Unlike [18,38], we do not train on in-the-wild images but estimate 3D shapes of clothed subjects.

Conclusion

We have proposed a new non-parametric approach to encode the 3D shape of a person through a double 2.5D depth map representation: a "visible" depth map depicts the elements of the surface that are directly observable in the image while a "hidden" depth map characterises the occluded 3D surface. We have designed an architecture that takes as input a single image and simultaneously produces an estimate for both depth maps resulting, once combined, in a point cloud of the full 3D surface. Our method can recover detailed surfaces while keeping the output to a reasonable size. This makes the learning stage more efficient. Our architecture can also efficiently incorporate a discriminator in an adversarial fashion to improve the accuracy and "humanness" of the output. To train and evaluate our network, we have captured a large-scale dataset of textured 3D meshes that we rendered on real background images. This dataset will be extended and released to spur further research. The main difference between our generator architecture and the stacked hourglass by Newell et al. [27] is the output dimension. Newell et al. estimate a 64x64 resolution heatmap for each body joint. In our case, we estimate 2 depth maps and aim at a higher 128x128 resolution. Our hourglass output dimension is 128x128x2. Because of this difference in output resolution, we apply the following modifications to the stacked hourglass [27] architecture: We do not use a maxpooling operation after layer1, we increase the depth of the hourglasses from 4 to 5 skipped connections, we project the hourglass result on 2 channels (one for each depth map). Also, we use 2 stacked hourglasses and we replace batch normalization by group normalization [44] that performs better on small training batches. See architecture details in Table 1.

Layer

Discriminator

For our discriminator, we employed a 4 stacks CNN. It takes as input a set of 2 depth maps at resolution 128x128 and outputs a scalar: close to 1.0 if it believes they are sampled from the ground truth depth maps and close to 0 if it believes they have been generated by the generator. See Table 2 for details.

Introduction

In this paper, we examine the problem of multiview shape reconstruction of production-realistic performance capture sequences. Such sequences may contain arbitrary casual clothing and motions, and have specific capture set assumptions due to the particular lighting and camera positioning of these setups. Multiview 3D reconstruction is a popular and mature field, with numerous applications involving the recording and replay of captured 3D scenes, such as 3D content creation for broadcast and mobile applications, or the increasingly popular virtual and augmented reality applications with 3D user avatars. An essential and still improvable aspect in this matter, in particular with performance capture setups, is the fidelity and quality of the recovered shapes, our goal in this work. Multi-view stereo (MVS) based methods have attained a good level of quality with pipelines that typically comprise feature extraction, matching stages and 3D shape extraction. Interestingly, very recent works have re-examined stereo and MVS by introducing features and similarity functions automatically inferred using deep learning. The main promise of this methodological shift is to include better data-driven priors, either in 2D [1,2,3,4] as improvement over classic 2D features, or in 3D to account for relative view placement and local or global shape priors [5,6,7]. These novel MVS methods have been shown to outperform classic learning-free methods on static scene benchmarks [8].

Our main goal is to examine whether these datadriven improvements transfer to the more complex case of live performance capture, where a diverse set of additional di culties arise with respect to typical MVS setups. Typical challenges for these capture situations include smaller visual projection areas of objects of in-Fig. 1 Challenging scene captured with a passive RGB multi-camera setup [9]. (left) one input image, (center ) reconstructions obtained with classical 2D features [10], (right) proposed solution. Our results validate the key improvement of a CNN-learned disparity to MVS for performance capture scenarios. Results particularly improve in noisy, very low contrast and low textured regions such as the arm, the leg or even the black skirt folds, which can be better seen in a brightened version of the picture in Figure 17.

terest, due to wider necessary fields of view for capturing motion; occlusion and self-occlusion of several subjects interacting together; lack of texture content typical of real-life subject appearance and clothing; or motion blur with fast moving subjects such as sport action scenes (see Figure 14). To the best of our knowledge, existing learning-based MVS schemes report results on static datasets such as DTU [11] or ShapeNet [12] but have not yet been demonstrated on performance capture data with the aforementioned typical issues.

We present a detailed framework for this purpose, which casts the problem as a fusion of per view depth maps as inspired by recent fusion methods [13], each depth map extracted using a learned multi-view photoconsistency function. Our approach performs multiview matching within local volumetric units of inference. Contrary to previous methods, our volumetric unit is defined in a given view's own reference, so as to capture camera inherent 3D dependencies, specifically for the purpose of per-view decision. Instead of inferring occupancies, we infer disparity scores to ease training and to focus the method more on photometric configurations than local shape patterns. We sweep viewing rays with this volumetric receptive field, a process we coin volume sweeping, and embed the algorithm in a multi-view depth-map extraction and fusion pipeline followed by a geometric surface reconstruction. With this strategy, we validate that CNN-based MVS outperforms classical MVS approaches in performance capture scenarios. In particular, we obtain high precision geometric results on complex sequences, outperforming both existing CNN-based and classic non-learning methods on a large set of capture datasets. These diverse results are obtained using only a DTU subset as training data, which evidences the generalization capabilities of our network.

This article is an extended version of [14] that provides a complete and self-contained description of the proposed method, with more details about the pipeline from [10] along with the detailed volume sweeping and surface extraction algorithms. Several supplementary experiments were performed to give more insights on the contribution and study the influence of the parameters. We finally challenged the generalization properties of our network on multiple dataset that were not seen during training with competitive results compared to both hand-crafted and learned state of the art.

Related Work

Multi-view stereo reconstruction is an active and longstanding vision problem [15]. Stereo and MVS-based approaches are increasingly being used for high fidelity capture applications [16,17,18,19,11,20,21], possibly complementing other strategies such as depthbased reconstruction [13,22,23,24] by addressing shortcomings that include limited range, sensitivity to high contrast lighting, and interference when increasing the number of viewpoints. While considering various shape representations, for instance point clouds [16], fused depth maps [25], meshes [26,27], or volumetric discretizations [28,29,30], most MVS methods infer 3D shape information by relying on the photoconsistency principle that rays observing the same scene point should convey similar photometric information.

In it simplest form, such similarity can be measured by considering projected color variances among views, as used in early works [28] with limited robustness. In stereo and short baseline situations, simple normalized forms of 2D window correlation are sucient to characterize similarity under simple lighting and constrast changes, using e.g. ZNCC, SSD, SHD. For broader geometric and photometric resilience, various features based on scale-invariant gradient characterizations [31,32,33] have been designed, some specialized for the dense matching required for the MVS problem [34]. More recently, image features have been successfully applied to performance capture sequences in e.g. [20,10]. Generally, MVS methods characterize photoconsistency either with a symmetric, viewpoint agnostic, combination of all pairwise similarities [35], or with a per image depth map determination through sweeping strategies [36,25]. The latter sweeping approaches have the advantage of simplifying the scene parametrization of occlusions [37,38], which we leverage for our approach and show to yield a robustness advantage over other strategies in our experiments.

While classic MVS approaches have been generally successful, recent works aimed at learning stereo photoconsistency have underlined that additional priors and more subtle variability co-dependencies are still discoverable in real world data. Several works leverage this by learning how to match 2D patch pairs for short baseline stereo, letting deep networks infer what features are relevant [1,2,3,4]. More recent works extend this principle to short baseline MVS, with symmetric combination of 2D learned features [39], or wide baseline sparse capture scenarios [40,41].

Most of these methods however use a 2D receptive field for stereo matching. The intuition that volumetric 3D receptive fields may be more informative and ease CNN inference and has been explored by some recent works [5,6,7,8], an assertion that the presented approach further verifies. While casting correlations in 3D as well, our approach proposes several key di↵er-ences. Contrary to the latter, our volumetric receptive field is projective in the camera coordinate frame, similar to some binocular stereo [42] or image-based rendering [43] works. This allows for sweeping along viewing rays, which was proven to be a robust search strategy for binocular stereo plane sweeping [38]. It also enables a per frame approach, with depth estimations, that appears to be more flexible than a global reasoning over all frames. This scheme also avoids decorrelating camera resolution and 3D receptive field resolution, as with e.g. voxels, the volumetric receptive field being defined as a backprojection along pixel rays. Additionally, this volumetric receptive field learns local pairwise correlations, a lower level and easier task than learning occupancy grid patterns. Our evaluation substantiates the aforementioned robustness benefits on a number of qualitative (7.3) and quantitative experiments (7.2) with challenging dynamic capture datasets, showing in particular the improvements over 2D receptive fields (7.1).

Method Overview

Our main objective is to study multi-view photoconsistency within the context of multi-view stereo reconstructions. We consider for that purpose the reconstruction framework, largely adopted over the last decade, that consists in first estimating per camera depth maps, followed by depth fusion and surface extraction. This framework allows to reason at the pixel level, enabling therefore each camera to provide local details on the observed surface with local estimations. This is in contrast with global strategies that consider photoconsistency at the shape level, with for instance voxels as in [6]. Comparisons between strategies are provided in the experiment section (see section 7.2).

Regarding depth map estimation, we propose to replace the traditional handcrafted photoconsistency measures used to estimate depths with a learned version. This version is based on CNNs and exploits their ability to learn local photometric configurations near surfaces observed from multiple viewpoints. As depicted in Figure 2, our approach takes as input a set of calibrated images and outputs a 3D mesh obtained by fusing depth maps. Depths along pixel viewing rays are obtained using a volume sweeping strategy that samples multi-view photoconsistency along rays and identifies the maxima. For a depth point candidate along a viewing ray, the photoconsistency is estimated using a discretized 3D volumetric projective grid centered on that point. In such a 3D grid, color inputs from the primary camera are paired with color inputs from another camera at each volume element of the grid around the depth point candidate. For a given depth candidate, we collect all such paired color volume grids for every other camera than the primary. A trained CNN is used to recognize the photoconsistent configurations given pairs of color samples within the 3D grid. The key aspects of this strategy are:

-The per camera approach, which, by construction, samples the photoconsistency at a given location as captured and thus enables more local details to be revealed compared to a global approach, as shown in Figure 17. -The 3D receptive field for the photoconsistency evaluation, which resolves some 2D projection ambiguities that hindered 2D based strategies. -The learning based strategy using a convolutional neural network, which outperforms traditional photometric features when evaluating the photoconsistency in dynamic captured scenes, as demonstrated by our experiments.

The following sections focus on our main contributions, namely the 3D volume sampling in section 4.1 and the learning based approach in sections 4.2,4.3 for the photoconsistency evaluation. We then describe in section 5 our depth map evaluation procedure, derived from a winners-take-all strategy suitable to our capture scenario. These depth maps are then fused into an implicit form, from which, without loss of generality, we extract the zero-level set using the surface extraction technique described in section 6.

Learning Photoconsistency

Our reconstruction approach takes as input N images {I i } N i=1 , along with their projection operators {⇡ i } N i=1 , and computes depth maps, for the input images, which are subsequently fused into a 3D implicit form. This section explains how these maps are estimated. Given a pixel p in an input image I i , the problem is therefore to find the depth d at which its viewing ray intersects the observed surface. The point along the ray of pixel p at depth d is noted r i (p, d). Our approach searches along viewing rays using a likelihood function for a point to be on the surface given the input color pairs in the evaluation volume. In contrast to traditional methods that consider handcrafted photoconsistency measures, we learn this function from multiview datasets with ground-truth surfaces. To this purpose we build a convolutional neural network which, given a reference camera i and a query point x 2 R 3 , maps a local volume of color pair samples around x to a scalar photoconsistency score ⇢ i (x) 2 [0..1]. The photoconsistency score accounts in practice for color information from camera i at native resolution, and for other camera colors in addition to their relative orientations as implicitly encoded in the volume color pair construction. These important features allow our method to adapt to specific ray incidences. Its voluntarily asymmetric nature also allows subsequent inferences to automatically build visibility decisions, e.g. deciding for occlusion when the primary camera i's color is not confirmed by other view's colors. This would not have been possible with a symmetric photoconsistency function such as [39].

We thus cast the photoconsistency estimation as a binary classification problem from these color pairs around the location x, with respect to the reference image I i and the other images. In the following, we first provide details about the 3D sampling regions before describing the CNN architecture used for the classification and its training. We then explain the volume sweeping strategy that is subsequently applied to find depths along rays.

Volume Sampling

In order to estimate photoconsistency along a viewing ray, a 3D sampling region is moved along that ray at regular distances. Within this region, pairs of colors backprojected from the images are sampled. Each pair contains a color from the reference image I i and its corresponding color in another image I j . Samples within the 3D region are taken at regular depths along viewing rays in the reference image (see Figure 3). The corresponding volume is a truncated pyramid that projects onto a 2D region of constant and given pixel dimension in the reference image. This allows the 3D sampling to adapt to the camera properties, e.g. pixel resolution and focal length.

More precisely, we denote r i (p, d) the 3D location at depth d along the viewing line back-projected from Volume sampling is always performed with the same orientation and ordering with respect to the reference camera. Convolutions are thus consistently oriented with respect to the camera depth direction.

Volume Size In practice, we choose k = 8. Our strategy is to learn pairwise photoconsistent configurations along rays. This way, decisions for the surface presence are conditioned to the observation viewpoints, which implicitly enforce visibility rules since only one 3D point per ray can be detected. This is in contrast to more global strategies where such per viewpoint visibility is less easy to impose, as with regular voxel grids, e.g. [6] with 32 3 or 64 3 grids. In addition, by considering the surface detection problem alone, and letting the subsequent step of fusion integrate depth in a robust and consistent way, we simplify the problem and require little spatial coherence, hence allowing for small grids. We provide a more detailed study of the performances of the classifiers with various depth values in section 7.1.

Multi-View Neural Network

As explained in the previous section, at a given point x along a viewing ray from image I i we can build N 1 color volumes with pairs of views (I i , I j6 =i ). Each volume is composed of k 3 cells with pairs of RGB values. In order to detect whether the surface is going through x, we use siamese encoders similar in spirit to [39], with however 3D volumes instead of 2D patches. Each encoder considers as input a pairwise color volume and provides a feature. Features from all color volumes at x are then averaged and fed into a final decision layer. Weight sharing and averaging are chosen to achieve camera order invariance.

The network is depicted in Figure 4. The inputs are the N 1 color volumes of size k 3 ⇥ 6 where RGB pairs are concatenated at each sample within the volume. Convolutions are performed in 3D over the 6 valued vectors of RGB pairs. The first layers (encoders) of the network process every volume in parallel, with shared weights. Every encoder is a sequence of two convolutions followed by non-linearities, and max-pooling with stride. Both convolutional layers consist of respectively 16 and 32 filters of kernel 4 ⇥ 4 ⇥ 4, followed by a Rectified Linear Unit (ReLU) and a max-pooling with kernel 2 ⇥ 2 ⇥ 2 with stride 2. We then average the obtained 2 ⇥ 2 ⇥ 2 ⇥ 32 features and feed the result to a 128 filter 1 ⇥ 1 ⇥ 1 convolutional layer, followed by a ReLU and a final 1 ⇥ 1 ⇥ 1 decision layer, for a total of 72K parameters . The network provides a score ⇢ i (r i (p, d)) 2 [0..1] for the photoconsistency at depth d along the ray from pixel p in image I i .

We experimented this network using di↵erent configurations. In particular, instead of averaging pairwise comparison features, we tried max-pooling which did not yield better results. Compared to the volumetric solution proposed by [6], the number of parameters is an order of magnitude less. As mentioned earlier, we believe that photoconsistency is a local property that requires less spatial coherence than shape properties.

Network Training

The network was implemented using TensorFlow [44] and trained from scratch using the DTU Robot Image Dataset [11], which provides multiview data equipped with ground-truth surfaces that present an accuracy of 0.5mm. From this dataset 11 million k 3 color volumes were generated, from which we randomly chose 80 percent for training, and the remaining part for evaluation. Both positive and negative samples were equally generated by randomly sampling volumes up to 20cm away from ground truth points, where a volume is considered as positive when it contains at least µ ground truth points. In theory, the network could be trained with any number of camera pairs, however, in practice, we randomly choose from one up to 40 pairs. Training was performed with the binary cross entropy function as loss. Model weights are optimized by performing a Stochastic Gradient Descent, using Adaptive Moment Estimation [45] on 560, 000 iterations with batch size of 50 comparisons, and with a random number of compared cameras (from 2 up to 40). Since our sampling grids are relatively small and camera dependent, we are able to generate enough sample variability for training, without the need for data augmentation.

Depth Estimation

As previously noted, our main motivation is to reconstruct live dynamic scenes, typically humans in motion. In such cases, it is advantageous to focus on the foreground objects in the observed scene rather than modeling the full scene. To this purpose, we limit the search domain for depths along viewing rays to a region defined by image silhouettes. In the following we explain how such a region is defined and we detail then the volume sweeping we adopt to identify image depths.

Confidence Volume

We assume we are given a set of N images {I i } N i=1 observed with a set C of calibrated cameras with known projections {⇡ i } N i=1 and centers {c i } N i=1 . We assume we are also given a set of silhouettes {⌦ i } N i=1 , often available in multi-view scenarios by performing image segmentation, for instance background subtraction with constrained capture environments. The silhouettes are generally imprecise, as a result of multiple causes, including color ambiguities, that plague the segmentation. However, the redundant information they provide over several viewpoints can be used to restrict the search domains along viewing rays to segments that are likely to intersect the object surface. To this purpose we define the confidence volume V as:

V = {x 2 R 3 : 9 >↵ i (⇡ i (x) 2 I i ) ^9> i (⇡ i (x) 2 ⌦ i )}, (1)
as the locus of points in R 3 which project in i > ↵ images and  i silhouettes to which they belong. When = i, V is simply the visual hull with i images. ↵, are two user defined constants that restrict weakly supported depth predictions with ↵ and enable predictions away from the exact visual hull when < ↵. Intuitively, V is a dilated version of the visual hull in the space region seen by at least ↵ images, as shown in fig 5. As explained in the following section, the intersection of a viewing ray with V defines the starting point of the depth search interval along that ray.

Volume Sweeping

In order to estimate pixel depths, the sampling volume introduced in section 4.1 is swept along their viewing rays while computing multi-view photoconsistency using the network detailed in section 4.2. For every camera, we sample therefore along viewing rays, test possible depth values, and choose the best candidate with respect to the network score. In practice, a reference view I i is only compared to the other views I j such that cos(✓ ij ) > 0.5, where ✓ ij is the angle between the optical axes of camera i and j. Then, we sample rays from camera i through every pixel p and build colored volumes at every candidate depth, starting at the intersection with the confidence volume introduced in the previous section. Once the probability of surface presence is computed for every candidate, we define the estimated depth d i as:

d i = argmax
Depths for all pixels and from all images are further fused using a truncated signed distance function (TSDF) [46]. The following section explains how we define and extract the zero level-set of the TSDF.

Surface Extraction

We explained in the previous section how to compute depth maps for every viewpoint. We now have to fuse them into an implicit form, namely the TSDF [46] from which we can extract the zero-level set that corresponds to the reconstructed surface, which appears in black in Figure 6. Contrary to previous works [47,24,13,22], we do not store TSDF values in a regular voxel grid but we rather devise a simple yet e cient sampling procedure derived from Voronoï Tesselation strategies, that specifically accommodates multi-view capture scenarios. It is worth mentioning that other works such as [27] also make use of irregular sampling strategies for MVS, but in a volumetric graph-cut framework.

Implicit Form Definition

For a point x 2 R 3 , the truncated signed distance T D(x) 2 R to the surface is defined as the weighted average of all camera contributions F i (x), i 2 C:

where C x = {i 2 C : F i (x) 6 = ;} and ⇢ 0 i (x) the photoconsistency measure (4) of the estimated depth along the ray from camera i passing through x. If d i is undefined at x, e.g. x is outside the camera visibility domain, then camera i does not contribute to the TSDF. When no camera contributes at x but x is inside the confidence volume V then it is considered as inside, i.e. T D(x) < 0. Note that contributions are weighted by the normalized photoconsistency measure which means that when cameras disagree about the photoconsistency at x, cameras with higher measures have an increased impact whereas cameras with low detection probability measures only marginally impact the reconstruction.

Extraction Procedure

From the previously defined TSDF, we extract the surface using a sampling strategy based on ray casting and Voronoï Tessellation. Figure 6 provides a 2D example of the main steps of the algorithm that are as follows:

1. (orange) Sample points inside the implicit form defined by the TSDF. This is achieved by randomly selecting pixels in all images and computing the point, along each pixel rays, inside but close to the surface according to the TSDF. The process is iterated until a user defined number of 3D points is reached. In the above strategy, sampling points close to the surface, and originating from image viewpoints, ensures that the 3D discretization is denser on the surface than inside the volume and also denser on surface regions Fig. 6 Our surface extraction procedure. The zero-level set of the implicit form (black) is observed by di↵erent cameras (red). They are used to provide the inside samples (orange) that will be used as the centroids for the Voronoï tessellation. This tessellation is finally clipped at the zero-level set and the final surface (green) can be extracted.

observed by more images. The latter enables more precision to be given to surface regions for which more image observations are available.

We visualize in figure 7 an example of extracted surface. We show 2 of the 40 input views in the top row, and our reconstruction in the middle. The bottom side of the bust is never seen by any camera. We show in the bottom row the di↵erence in sampling resulting from the observation of the shape. The horizontal bottom side of the model is never observed, yet still correctly reconstructed. On the other hand, the triangles of the mesh in that area are much larger than the ones in the vertical upper part, which is observed by more cameras. This strategy allows for complete reconstructions of captured shapes with an adaptive sampling density depending on the observations of the object, focusing more samples in the regions where the details can be recovered.

Runtime The full pipeline allows us to reconstruct one time frame, i.e. 68 images (2048⇥2048), in approximately 30 to 40 minutes using two NVIDIA Titan X GPUs, depending on the number of pixels that observe the shapes.

Experiments

Our main goals in this section are (i) to evaluate whether and how our learned photoconsistency contributes with respect to existing methods and (ii) to ver- ify whether these transfer to the more complex case of generic 3D capture scenes in practice, e.g. humans with complex clothing. To this aim, we perform various evaluations to verify and quantify the benefit of our learned multi-view similarity. We start by providing multiple validation experiments to justify the choices for the learning and reconstruction strategies in 7.1. Second, for comparison purposes, we apply in 7.2 our depth estimation approach in the static case using the [11] benchmark and compare it to state of the art MVS methods, both handcrafted and learning based. We make use of the standard accuracy and completeness metrics, both averaged and median, for which the evaluation code is provided by the authors.

Finally, we build experiments to test the main claim of improvement with production capture data in 7.3. To this goal we use several dynamic sequences captured on di↵erent platforms, which exhibit typical di culties of such data. In particular, we mainly focus on the Kinovis acquisition platform [9], which consists of 68 RGB cameras, of resolution 2048 ⇥ 2048 with focal lengths varying from 8mm to 25mm. We achieve very significant qualitative improvements compared to the state of the art approaches both learning-based [6,8], and handcrafted [10], without fine-tunning and despite the di↵erence of capture setup used for training. We also compare to [23] on an example provided by the authors and achieve slightly better quality using only half the available information.

Validation

We previously formulated the problem of surface detection along viewing rays as a binary classification problem, as explained in section 4. In order to assess the benefit of our volumetric strategy, we first focus on different classifiers performances. We provide in 7.1.1 receptive field comparisons on the training dataset, this to enhance the advantage of casting and learning correlations in 3D. Additionally, section 7.1.2 provides a study of the depth hyperparameter of the receptive field of our network. Then, since preliminary results of [14] seemed to show a better robustness to a larger baseline, we design an experiment with cameras that are further apart to better quantify this improvement in section 7.1.3. We finally provide in 7.1.4 an ablation study of the accumulator described in 3 to validate its importance in the depth estimation procedure, in the performance capture scenario.

Section 7.1.2 shows that a volume size of 8 ⇥ 8 ⇥ 8 is a preferred trade-o↵, thus will be used from now on, when not specified.

Classifiers Study

In this paragraph, we compare performances of di↵erent classifiers based on various receptive fields:

1. Zero-Mean Normalized Cross Correlation (ZNCC ):

ZNCC is applied over the samples within the volumetric support region. 2. Learning (CNN) with a planar support: a planar equivalent of our volumetric solution, with the same architecture and number of weights, in a frontofacing plane sweeping fashion. 3. Learning (CNN) with a volumetric support: our solution described in the previous sections.

Figure 8 shows, with the classifiers' ROC curves, that the most accurate results are obtained with a volumetric support and learning. Intuitively, a volumetric sampling region better accounts for the local non-planar geometry of the surface than planar sampling regions. This graph also emphasizes the significantly higher discriminative ability of learned correlations compared to deterministic ones.

Volume Sampling

To further demonstrate this, we then proceed to a study on the impact of the depth parameter of the sampling volume. While keeping a 8⇥8 pixels reprojection on the images, we study the performances on classifiers with receptive fields varying in depth. Figure 9 shows classifiers performances with depth values ranging from 1 to 12. To perform this experiment, we had to diminish the networks number of parameters to fit the 12 depth training in memory and keep reasonable training and testing times, explaining the worse performances compared to previous ROC curves. This experiment demonstrates that the more information the network gathers along the ray the better the detection of the surface is. We choose a depth of 8 as it gives the best trade-o↵ between computational complexity and performance.

Baseline Study

We now evaluate the robustness to various baselines by accounting for a higher number of cameras and more distant cameras in the classification. Table 1 shows the accuracy of the classifiers with a varying number of cameras and for the optimal threshold values in Figure 8. As already noticed in the literature, e.g. [16,19], a planar receptive field gives better results with a narrow baseline and the accuracy consistently decreases when the inter-camera space grows with additional cameras. In contrast the classifier based on a volumetric support exhibits more robustness to the variety in the camera baselines. This appears to be an advantage with large multi-camera setup as it enables more cameras to contribute and reduces hence occlusion issues. To push this experiment further, we design an experiment to test the robustness of our approach on a sparse capture platform, with lower scene coverage and wider baseline. Since no ground truth exists for this kind of performance capture scenario, we simulate it using of a realistic rendering engine to create a synthetic dataset. Similar to [9] in terms of camera parameters and capture volume, we chose to render only 10 randomly placed cameras, evenly distributed on an hemisphere around the capture volume. The average spacing between a camera and its 10 closest neighbors is 8.03m in this case, where it is 2.5m for the 68 POV kinovis platform and 0.188m in the 49 POV DTU case. For this experiment, we set the neighboring camera acceptation threshold cos(✓ ij ) to 0.1, meaning that we accept almost orthogonal cameras. The synthetic cameras render the scene using Filmic Blender [48], a photorealistic configuration for Blenders Cycles ray-tracing engine. The images are generated with random parameters, i.e. the cameras parameters vary, in terms of position, orientation, focal length, and pixels number of samples, the latter directly a↵ecting sensor noise. With this platform, we rendered a dozen of models such as procedu- rally generated geometric shapes, real life reconstructions or CAD models with various appearances. The multiview networks are trained from scratch on these synthetic examples, and evaluated on unseen synthetic data. Figure 10 shows an example of our synthetic platform as well as the generated synthetic data. We show in figure 11 the impact of a volumetric support: when the baseline between the cameras becomes extreme, it o↵ers more robustness compared to a planar support, which appears very slanted in the compared view. Even though it is only a synthetic dataset, we believe that it gives interesting insights on the versatility of our volume sweeping strategy for the performance capture scenario. A qualitative result of this improved robustness is shown in figure 12. The area of the face is highly occluded, and the volumetric support helps recovering a smoother surface. Also note the details of the belt: the volume allows a sharp reconstruction of finer details, where a plane cannot handle finer geometry details.

Accumulation Term

We now provide a qualitative experiment to justify the use of the accumulation term in equation 3 in figure 13. This figure demonstrates the importance of the accumulation scheme in the performance capture scenario. The noisy photoconsistency in this case leads to a lot of false positives, creating extreme holes in the reconstructions when not using the accumulation scheme, i.e. ⇢ max ! 1. The addition of this term (⇢ max = 1.6) allows for smooth and faithful reconstructions, still containing most of the important geometric details.

Quantitative Comparisons

In this section, we compare our solution to various state-of-the-art methods using the DTU Robot Image Dataset [11]. We use the standard accuracy and completeness metrics to quantify the quality of the esti- mated surface as described in [49], that is we define accuracy for a point of the reconstructed shape as the smallest Euclidean distance to the ground-truth, and the completeness of a point of the ground-truth as the smallest Euclidean distance to the reconstructed shape. For both metrics, we compare the average and median values over all the points of the shapes. To diminish the impact of far outliers in the metrics, we make use of the default thresholding parameter of [49]. We compare to Furukawa et al. [16], Campbell et al. [50] and Tola et al. [34], that are well-known handcrafted strategies, as well as to additional learning-based results from Ji et al. [6] and Hartmann et al. [39]. To conduct a fair comparison with [39], that is a patch based approach building a depthmap with a network comparable to ours, we use the result of our volume sweeping approach on the same depth map. When performing reconstructions on the DTU, we did not use the accumulation scheme in 3, i.e. ⇢ max ! 1. To speed up computations, we limit the search along a viewing ray to 5mm around a coarse depth estimation based on image descriptors [51]. Depths are sampled every 0.5mm. As a post processing step, we simply add a soft bilateral filter, similarly to [39], accounting for color, spatial neighborhood, and probability of the detection.

Reconstruction results are depicted in table 2. We achieve quality on par with the best performing methods on this dataset, with a median accuracy and completeness in the range of the ground truth accuracy that we measured around 0.5mm. It should be noticed that the best accuracy is obtained by Tola et al. [49] which tend to favor accuracy against completeness whereas Campbell et al. [50], in a symmetric manner, tend to favor completeness against accuracy. We obtain more balanced results on the 2 criteria, similarly to the widely used approach by Furukawa et al. [16], with however better performances. We also outperform the recent learning based method Surfacenet [6] on most measures in this experiment.

Compared to Hartmann et al. [39], and under similar experimental conditions, our approach give better results with 2 orders of magnitude less parameters, thereby confirming the benefit of volumetric supports over planar ones. Compared to Surfacenet. [6] (cube size 64 ⇥ 64 ⇥ 64, sample step 0.4mm) we obtain reconstructions of slightly better quality with an order of magnitude less parameters.

Qualitative Evaluation and Generalization

One of our main goals is to verify whether a learning based strategy generalizes to real life dynamic data and how it compares to state-of-the-art approaches in this case. To this purpose, we focus our qualitative evaluation on two di↵erent dynamic capture datasets, both drastically di↵erent from the training one. We first perform, in section 7.3.1, reconstructions of dynamic RGB sequences captured by the Kinovis platform [9]. We then test, in section 7.3.2, our reconstruction method on a di↵erent real life dynamic dataset, captured with the active setup of [23] and compare to their results. It is important to note that the network previously trained on the DTU Dataset [11] was kept as such without any fine tuning at all times in this section.

Kinovis Data

We first focus on data captured by [9], that is a hemispherical setup with 68 cameras of various focal lengths. In this scenario, standard MVS assumptions are often violated, e.g. wide baseline, specular surfaces, motion blur and occlusions, challenging therefore the reconstruction methods. A video demonstrating our results and providing comparisons on dynamic sequences is available online: https://hal.archives-ouvertes. fr/hal-01849286.

Most general purpose MVS methods we tested tend to fail in the performance capture scenario, either providing incomplete or low resolution results, or being extremely noisy. Figure 16 illustrates the reconstruction obtained using COLMAP [52], which is a handcrafted general purpose MVS pipeline based on Patch-Match Stereo [53]. Both methods perform overall correctly, as seen on the left side of the figure. However, [52] (top-right) struggles to recover fine-grained details while keeping the noise and artifacts level low, contrary to our approach (bottom-right). This results demonstrates the benefit of a dedicated method in the context of performance capture.

In order to assess the performances of our learned photoconsistency term, we compared in figure 1 to [10], which is a patch based sweeping method using traditional image features and specifically designed for this scenario. Both methods share a significant part of their pipeline, except for the photoconsistency evaluation, thus providing good insights about the benefits of the proposed learned term. Even though [10] performs well in contrasted regions, the patch based descriptors reach their limits in image regions with low contrast or low resolution. Figure 15 and 14 give such examples. They show that our strategy helps recovering finer surface details, while strongly decreasing noise in low contrast regions. The results obtained also demonstrate strong improvements in surface details, such as dress folds, that were undetected by the deterministic approach. In addition, they demonstrate lower levels of noise, particularly in self-occluded regions, and more robustness to motion blur as with the toes or tongue-in-cheek details that appear in Figure 14-bottom.

We then compare to the recent learning based approach [6] using the code available online (see Figure 17). Reconstructions with this approach were limited to a tight bounding box and di↵erent values for the volume sampling step were tested. The best results were obtained with a 2mm step. To conduct a fair comparison with our method, all points falling outside the visual hull were removed from the reconstruction. In this sce- nario, the point cloud obtained using [6] appeared to be very noisy and incomplete (see Figure 17 [6] wrongly reconstruct many surface points inside the shape volume (top figure), as a result of the ambiguous appearance of the dress. In contrast, our approach (bottom figure) correctly identify surface points by maximizing learned correlations along viewing rays.

In addition to this, we also compare to results of [8] provided by the authors in Figure 18. This method outputs a rather dense colored point cloud but similarly to results from [6], extracting a smooth surface from this point cloud remains a di cult task due to strong noise and missing data. Since the method uses custom and undocumented calibration parameters, it was not straightforward to remove points lying outside the visual hull. Moreover, the precision of the point cloud from [8] restricts its usage for performance capture and realistic reconstructions rendering. Figure 19 provides a close-up of the face of a subject. The level of detail of the result from [8] is not fine enough to correctly capture facial details, compared to the density of our output surface.

Active Capture Platform

Finally, we compare our reconstructions of a scene captured with results from the active system of [23]. This setup consists of 52 RGB cameras mounted as stereo pairs but also di↵ers from the previous dynamic capture scenario, as it also features an active system, projecting random infrared dots on the shape. 52 infrared cameras, also paired on stereo rigs then capture the reprojected spots on the shape, resulting in highly contrasted images, allowing to disambiguate the photoconsistency computation, especially in textureless regions without interfering with the visible appearance of the subject. In figure 20, we compare to results provided by the authors. While [23] make use of all the data available, we restrict our method to work with RGB images only. On the other hand, we allow cameras that are far apart to participate in the computation of the photoconsistency. Our results demonstrate the quality of our method's results, showing detailed reconstructions competitive with the results of [23] even though we only use the passive system, i.e. half of the available information. Figure 21 displays a close-up of the face of the subject. Our method allows to recover high-frequency facial details, such as the shape of the nostrils or the lips commissures, thus providing highly faithful reconstructions.

Conclusion and Future Works

We presented a learning framework for surface reconstruction in passive multi-view scenarios. Our solution consists in a N -view volume sweeping, trained on static scenes from a small scale dataset equipped with ground truth. Thanks to this new model, we validate the improvement of CNN-learned MVS photoconsistency in the case of complex and dynamic performance capture, with significant challenges typical of these datasets such as low light areas and low texture content and perceived resolution. This result is achieved with an order of magnitude less training parameters than previous comparable learned MVS works, showing significant network generalization from a training performed only on static DTU inputs, fully leveraging the high quality ground truth now available with these datasets. Thanks to our local strategy, our method achieved significantly improved detail recovery and noise reduction in complex real life scenarios, outperforming all existing approaches in this case.

The discretization of the volume around a query point involves a lot of redundancy and is a computationally expensive step for both training and inference. Moreover, even when optimized to process several neighboring depths in parallel, it remains rather memory ine cient. A possible future work could be to find a continuous representation for colored rays crossing the volume of interest, that could be used to infer surface presence probability in a similar manner with a much lighter computational cost.

Finally, we believe our approach is a first step towards a data-driven method to unify shape from silhouette and multi-view stereo inference, as made possible by the wide baseline robustness and general volumetric receptive field of our network, with the prospect of increased automation and quality.

FIGURE 1 . 1 -

 11 FIGURE 1.1 -Visualization of the geometry of a 4D aligned template model.

1. 4 . 1

 41 Contexts I obtained my PhD from the Institut National Polytechnique de Grenoble in 2005 with the INRIA MOVI team headed by Radu Horaud, supervized by Edmond Boyer. I then joined UNC Chapel Hill as a post-doctoral research assistant to Marc Pollefeys in 2006-2007, during which I co-supervized PhD students Li Guan and collaborated with Sudipta Sinha on probabilistic visual hulls.

FIGURE 2 . 1 -

 21 FIGURE 2.1 -Spatiotemporal refinement framework.

FIGURE 2 . 2 -

 22 FIGURE 2.2 -Obtained spatio-temporally refined results. (a) Detailed shapes obtained for complex, dynamic, multi-person performance captures scenes. (b) Comparison of the method without, and with the temporal integration proposed.

FIGURE 2 . 4 -

 24 FIGURE 2.4 -(left) The 3D volume used to estimate photoconsistency along rays from the reference image I i . k 3 samples within the volume are regularly distributed along viewing rays and contain color pairs as back-projected from images I i and I j . At a given depth along a ray from I i any image I j =i can define such a pairwise comparison volume. (right) CNN architecture. Each cube is a pairwise comparison volume with k 3 samples that contain 6 valued vectors of RGB pairs and over which 3D convolutions are applied. The output score ρ i (r i (p, d)) ∈ [0..1] encodes the photoconsistency measure at depth d along the ray from pixel p in image I i .

FIGURE 2 . 8 -

 28 FIGURE 2.8 -(a) Performance on 3D-HUMANS dataset in presence of severe occlusions on three frames : (top) input images, (left) with GAN, (right) without GAN. Errors above 15cm are shown in red. The GAN helps increase the "humanness" of the predictions. (b) Generalisation to previously unobserved data. We apply our pipeline to images with 3D realistically rendered backgrounds (left), and with 3 real-world images from the LSP dataset (right). These poses, in particular the baseball player, have not been seen at training time but our model still generalizes well.

FIGURE 3 . 2 -

 32 FIGURE 3.2 -Tracking excerpts from the DANCER dataset. Colors code patches.

  FIGURE 3.3 -(left) Input mesh, (middle) tracked mesh without rigidity inference -Cagniart et al. [CBI10b] and (right) with rigidity inference.The absence of the head on the top-left input mesh is imputable to the visual hull computation method, which assumes full visibility for each camera. It can be observed that our method corrects substantial artifacts.

FIGURE 3 . 4 -

 34 FIGURE 3.4 -Tracking excerpts from GOALKEEPER, MARKER [LGS + 13] and FREE datasets showing a heatmap coded values of the C k,l rigidity variables : red when very likely non rigid, blue when very likely rigid. Note that the inference is performed on a time window, i.e. the rigid inference corresponds to rigidities as observed in that window.

FIGURE 3 . 5 -

 35 FIGURE 3.5 -Possible volumetric decompositions of the template and observed shapes. (top) 2D schematic view. (bottom) 3D decomposition example. (a) Voxels on a regular grid. (b) A sliced Constrained Delaunay tetrahedrization showing the elongated inner tetrahedra generated. (c) Voronoi cells with random centroids shown in red, center of mass of each cell in green. (d) Centroidal Voronoi tesselation cells, where the center of mass and cell centroid coincide.

FIGURE 3 . 6 -

 36 FIGURE 3.6 -Frames of the BALLET (top and middle) and GOALKEEPER datasets (bottom). (a) Visual hull input. (b) Tracking result of Cagniart et al. [CBI10b]. (c) Allain et al. [AFBT14b]. (d) Our method. Note the improved angular shapes on the dancer's knee (top) and elbow (middle), and the improved robustness (bottom).

FIGURE 3 . 8 -

 38 FIGURE 3.8 -Six representative examples of frames of our motion database. From left to right, a female and male subject is shown for tight, layered, and wide clothing each.

FIGURE 3 . 9 -

 39 FIGURE 3.9 -Accuracy of posture estimation over the walking sequences of all subjects in tight clothing. Left : cumulative landmark errors. Right : average landmark error throughout each sequence. Figure from [YFHWW16].

FIGURE 3 . 10 -

 310 FIGURE 3.10 -Overlay of input data (grey) and our result (blue).Figure adapted from [YFHWW16].

  FIGURE 3.10 -Overlay of input data (grey) and our result (blue).Figure adapted from [YFHWW16].

FIGURE 4 . 2 -

 42 FIGURE 4.2 -Comparison on BUNNY, BEETHOVEN and BIRD datasets. Left column : input images. Middle : output of [GAKC13]. Right : our algorithm.

FIGURE 4 . 3 -

 43 FIGURE 4.3 -Top : GOALKEEPER. Middle : BACKPACK. Bottom : ACTOR. The figure illustrates various temporal improvements. Top : Input is compared to Frame 1 and Frame 3. Middle : Input on left, Frame 1 and Frame 3 comparisons. Details are revived on the backpack, T-shirt and pants. Bottom : Input, Frame 1 and 2 comparisons.

FIGURE 4 . 5 -

 45 FIGURE 4.5 -Method pipeline from input textures (left) to eigen maps (right).

FIGURE 4 . 6 -

 46 FIGURE 4.6 -Texture map generation by linear combination.

Fig. 4 .

 4 7 and Fig. 4.8. More details are available in the original ECCV 2016 publication [BTF + 16] (Appendix A.5) and with the supplemental video 2 .

FIGURE 4 . 7 -

 47 FIGURE 4.7 -Interpolation examples using linear interpolation (left) and our pipeline (right). From left to right : Input frames, Interpolated models, and a close-up on the texture maps (top) and the rendered images (bottom).

FIGURE 4 . 8 -

 48 FIGURE 4.8 -Completion examples. From left to right : Input and completed models, close-up on input and completed texture maps (top) and rendered images (bot-tom).

Figure 1 .

 1 Figure 1. Input view 768 × 576 resolution with up-sampling by factor of three, BEETHOVEN dataset. Super-resolved 2304 × 1728 output of our algorithm rendered from identical viewpoint.

Figure 2 .

 2 Figure 2. Summary of image formation model and problem notation.

Figure 3 .

 3 Figure 3. Comparison on BUNNY, BEETHOVEN and BIRD datasets. Left column: input images. Middle: output of [10]. Right: our algorithm. Best viewed magnified and in color.

Figure 5 .

 5 Figure 5. Results from GOALKEEPER dataset. We computed the mean value over frames of the Mean Square Error between our output and high resolution ground truth image. The resolution of input images is 512 × 512 and of the super-resolved output images is 1024 × 1024. We use a time step of 2 in experiments, corresponding to an acquisition frequency of 15Hz.

Figure 4 .

 4 Figure 4. This figure illustrates various temporal improvements and detail enhancements obtained with various acquired datasets, comparing different convergence using one or several temporal frames. Top: GOALKEEPER dataset. Left: output of Frame 3. Input is compared to Frame 1 and Frame 3 for each close-up. Middle: BACKPACK dataset. Input on left, Frame 1 and Frame 2 comparisons for close-ups. Details are revived on the backpack, T-shirt and pants. Bottom: ACTOR; left to right: full result with three frames, close-up comparison between input, against Frame 1 and Frame 3. Best viewed magnified and in color.

A. 2 .

 2 ON MEAN POSE AND VARIABILITY OF 3D DEFORMABLE MODELS 65 ∴ A.2 ON MEAN POSE AND VARIABILITY OF 3D DEFORMABLE MODELS Benjamin Allain, Jean-Sébastien Franco, Edmond Boyer, Tony Tung, ECCV 2014 -European Conference on Computer Vision, Sep 2014, Zurich, Switzerland. Springer HAL Id: hal-01016981 https://hal.inria.fr/hal-01016981 Submitted on 2 Jul 2015

  Benjamin Allain, Jean-Sébastien Franco, Edmond Boyer, Tony Tung To cite this version: Benjamin Allain, Jean-Sébastien Franco, Edmond Boyer, Tony Tung. On Mean Pose and Variability of 3D Deformable Models. ECCV 2014 -European Conference on Computer Vision, Sep 2014, Zurich, Switzerland. pp.284-297, 10.1007/978-3-319-10605-2_19. hal-01016981 1 Introduction

Fig. 1 .

 1 Fig. 1. Example of patch template used.

Fig. 2 .

 2 Fig. 2. Mean error for temporal evolution over Marker dataset.

Fig. 4 .

 4 Fig. 4. Tracking excerpts from the Dancer dataset. Colors code patches.

Fig. 5 .

 5 Fig. 5. Tracking excerpts from GoalKeeper, Marker and Free datasets. Best viewed in color. Please watch supplemental video for more visualizations.

Figure 1 .

 1 Figure 1. Possible volumetric decompositions of the template and observed shapes. (top) 2D schematic view. (bottom) 3D decomposition example. (a) Voxels on a regular grid. (b) A sliced Constrained Delaunay tetrahedrization showing the elongated inner tetrahedra generated. (c) Voronoi cells with random centroids shown in red, center of mass of each cell in green. (d) Centroidal Voronoi tesselation cells, where the center of mass and cell centroid coincide.

Figure 2 .

 2 Figure 2. Mean error for temporal evolution over MARKER dataset.

Figure 3 .Figure 4 .

 34 Figure 3. Patched volumetric decompositions of the template for sequences (a) BALLET, (b) GOALKEEPER-13, (c) MARKER and (d) DANCER. A color has been assigned to each patch for a visualization purpose.

Fig. 1 .

 1 Fig. 1. Overview of the proposed pipeline. From left to right: input frame, result of Stitched Puppet [1] with annotated landmarks, result after estimation of initial identity and posture, final result, and overlay of input and final result.

Fig. 2 .

 2 Fig. 2. Left: overfitting problem of Stitched Puppet in the presence of clothing. Input frame, Stitched Puppet result with 160 particles, and Stitched Puppet result with 30 particles are shown in order. Right: the failure case from our database caused by mismatching of Stitched Puppet.

Fig. 3 .

 3 Fig. 3. Influence of E cloth on walking sequence. Left: input data overlayed with result with ! cloth = 0 (left) and ! cloth = 1 (right). Middle: cumulative per-vertex error of estimated body shape with ! cloth = 0 and ! cloth = 1. Right: color-coded per-vertex error with ! cloth = 0 (left) and ! cloth = 1 (right).

Fig. 4 .

 4 Fig. 4. Six representative examples of frames of our motion database. From left to right, a female and male subject is shown for tight, layered, and wide clothing each.

Fig. 5 .

 5 Fig. 5. Accuracy of posture estimation over the walking sequences in tight clothing. Left: cumulative landmark errors. Right: average landmark error for each sequence.

Fig. 6 .

 6 Fig. 6. Summary of shape accuracy computed over the frames of all motion sequences of all subjects captured in layered and wide clothing. Left: cumulative plots showing the per-vertex error. Right: mean per-vertex error color-coded from blue to red.

Fig. 2 :

 2 Fig. 2: Method pipeline from input textures (left) to eigen maps (right).

Fig. 3 :

 3 Fig. 3: Poisson versus direct texture warping.

Fig. 4 :

 4 Fig. 4: Texture map generation by linear combination.

Fig. 5 :

 5 Fig. 5: Reconstruction Error for Tomas and Caty Dataset from top to down in Texture and Image Domain from left to right.

Fig. 6 :

 6 Fig. 6: Generalization Error for Thomas and Caty Dataset from top to down in Texture and Image Domain from left to right.

Fig. 7 :

 7 Fig. 7: Interpolation examples using linear interpolation (left) and our pipeline (right). From left to right: Input frames, Interpolated models, and a close-up on the texture maps (top) and the rendered images (bottom).

Fig. 8 :

 8 Fig. 8: Completion examples. From left to right: Input and completed models, close-up on input and completed texture maps (top) and rendered images (bottom).

Fig. 1 .

 1 Fig. 1. Given a reference shape (a) and input data (b), our method discovers reliable data-model correspondences by random forests, colorcoded in (c). This strategy detects user-specific shapes in a frame-wise manner, resulting in better sustainability. In (d) the reference model (a) is deformed with correspondences (c) to fit the input data (b).

  This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308

2 )

 2 optimization: estimating the motion parameter Θ by minimizing an energy E that describes the discrepancies between pairs in C , i.e. Θ = argmin Θ E(Θ; C ), such that X( Θ) resembles Y as much as possible.

Fig. 3 .

 3 Fig. 3. The pipeline of our tracking-by-detection framework. Data-model associations are visualized in the same color. Upper row: surface-based associations (black means no correspondence found for that vertex); bottom row: volumetric associations.

Fig. 4 .

 4 Fig. 4. The intuition of adjusting offsets. (a) original offset pair ψ. (b) η = 0 results in ψ without re-orientation, i.e. R = I. (c) η = 1. ψ is orientated by a rotation matrix R = [e 1 , e 2 , e 3 ] characterized by a LCF.

Fig. 5 .

 5 Fig. 5. Our method leads to quasi pose-covariant LCFs.

Fig. 6 .

 6 Fig. 6. CVT-based feature. Left: CVT cells S sample a distance field, where each cell stores the distance d(xs, ∂Ω). Blue to red colors means from close to far. Red dot: cell s to be described. Right: a toy example of our feature f , where L = 5. Shadowed and transparent layers have coefficients c l = -1 and 1 respectively. See text for more explanations.

Fig. 7 .

 7 Fig. 7. The schematic flowchart of the multi-template learning framework. Red arrows: mappings g µ that associate the indices from each subject-specific template S µ to the common one Ŝ. X t µ are the temporal evolutions of each template. Blue: training; green: prediction.

Fig. 8 .

 8 Fig. 8. (a): illustration of our strategy adapting skinning weights to CVT cells. Distances to the surface d(xs, ∂Ω) are reflected in the normalization constants e d . (b): result of matching two templates.

Fig. 9 .Fig. 10 .

 910 Fig. 9. Qualitative results of surface matching on FAUST. Best viewed in pdf.

Fig. 11 .

 11 Fig. 11. Predicted data-model associations on noisy visual hulls with Hough forests. Black color means that the points are either outliers, or the inferred correspondences are rejected due to incompatible normals.

Fig. 12 .(

 12 Fig. 12. Qualitative results of volumetric matching on the raw CVTs. Best viewed in pdf.

Fig. 13 .

 13 Fig. 13. Cumulative matching accuracy of different approaches. The xaxis is normalized w.r.t. the average edge length of the templates. The number of trees T is 20 in this experiment. Dashed and solid lines are accuracies on training (Tr) and testing (Te) sequences respectively.

Fig. 14 .

 14 Fig. 14. Cumulative matching accuracy on Goalkeeper.

Fig. 15 .

 15 Fig. 15. Pixel overlap error of 8 sequences, averaged over all cameras. Image resolution: (a-d): 1920×1080; (e-h): 1000×1000. Best viewed in pdf.

Federico

  Tombari is a research scientist and team leader at the Computer Aided Medical Procedures Chair of the Technical University of Munich (TUM). He has co-authored more than 110 refereed papers on international conferences and journals in the field computer vision and robotic perception, on topics such as visual data representation, RGB-D object recognition, 3D reconstruction and matching, stereo vision, deep learning for computer vision. He got his Ph.D at 2009 from University of Bologna, at the same institution he was Assistant Professor from 2013 to 2016. He is a Senior Scientist volunteer for the Open Perception foundation and a developer for the Point Cloud Library, for which he served, in 2012 and 2014, respectively as mentor and administrator in the Google Summer of Code. In 2015 he was the recipient of a Google Faculty Research Award. His works have been awarded at conferences and workshops such as 3DIMPVT'11, MICCAI'15, ECCV-R6D'16.

Figure 4 .

 4 Figure 4. Data generation. (a)We captured 3D meshes of humans, wearing real clothes, moving and manipulating objects using a multicamera platform. We then rendered these models on real-world background scenes and computed ground-truth visible and hidden depths maps. (b) We also generated a test set by rendering our meshes on realistic 3D environments.

Figure 5 .

 5 Figure 5. Comparaison with state-of-the-art on the SURREAL dataset: (a) we first compare against the BodyNet [38] baseline. (b) We analyse the impact of varying the size of the training set on performance on our new 3D HUMANS dataset.

  our

Figure 6 .

 6 Figure 6. Evaluation on our 3D HUMANS dataset: we first analyze the influence of the training data on performance (a). Then, we compare against the SMPL baseline (b). We compare the performance on visible and hidden depth map separately (c). Finally, we analyse the training data on a dataset rendered in realistic backgrounds and observe that SURREAL data is important for generalisation (d).

Figure 7 .

 7 Figure 7. Performance on 3D-HUMANS dataset in presence of severe occlusions on three frames: (top) input images, (left) with GAN, (right) without GAN. Errors above 15cm are shown in red. The GAN helps increase the "humanness" of the predictions.

Figure

  Figure 8. Generalisation to previously unobserved data. We apply our pipeline to images with 3D realistically rendered backgrounds (left), and with 3 real-world images from the LSP dataset (right). These poses, in particular the baseball player, have not been seen at training time but our model still generalizes well.

Figure 9 .

 9 Figure 9. Comparison between HMR[18] (left), Bodynet[38] (middle) and our method (right). Unlike[18, 38], we do not train on in-the-wild images but estimate 3D shapes of clothed subjects.

Fig. 2

 2 Fig. 2 Method overview. Depth maps, for all input image I i , are obtained by maximizing, along viewing lines, a learned function that measures photoconsistency at a given depth d along the viewing line of a given pixel p. Depth maps are then fused into an implicit form from which the zero set surface is extracted.

Fig. 3

 3 Fig.3The 3D volume used to estimate photoconsistency along rays from the reference image I i . k 3 samples within the volume are regularly distributed along viewing rays and contain color pairs as back-projected from images I i and I j . At a given depth along a ray from I i any image I j 6 = i can define such a pairwise comparison volume.

Fig. 4

 4 Fig. 4 CNN architecture. Each cube is a pairwise comparison volume with k 3 samples that contain 6 valued vectors of RGB pairs and over which 3D convolutions are applied. The output score ⇢ i (r i (p, d)) 2 [0..1] encodes the photoconsistency measure at depth d along the ray from pixel p in image I i .

Fig. 5

 5 Fig. 5 Left: the Confidence Volume with ↵ = = 54, equivalent to the Visual hull with the 54 cameras that see the subject; Right: the Confidence Volume with ↵ = = 10.

  d2[dmin,dmax] (⇢ i (r i (p, d))), (2) where ⇢ i (r i (p, d)) is the consistency measure along the ray from p in image I i , as estimated by the network. [d min , d max ] is the search range with: d min = d V (p) the intersection of the viewing ray at p with the confidence volume; d max such that the search is stopped when the accumulated photoconsistency score reaches a given value ⇢ max , in a winner-takes-all surface detection strategy. Z dmax x=dmin ⇢ i (r i (p, x))dx  ⇢ max

F⇢

  i (x) = ⇢ min(µ, ⌘(x)) iif ⌘(x) µ, ; otherwise, ⌘(x) = d i (⇡ i (x)) kc i xk, i (x)F i (x) P i2Cx ⇢ i (x) ,

2 .

 2 (blue) Determine the Voronoï diagram: given the points inside the shape surface, a Voronoï diagram of this set of points is computed. 3. (green) Clip the Voronoï diagram with the zero level set of the TSDF. This operation extracts the intersection of the Voronoï cells with the surface to form an oriented mesh.

Fig. 7

 7 Fig. 7 Two points of view of a synthetic model (top) and the result of our reconstruction (middle). A close-up of the extracted surface (bottom) at the limit between well-observed and unseen regions. The top part of the close-up is seen by many cameras whereas the bottom part is never observed.

Fig. 8

 8 Fig.8ROC Curves of three di↵erent classifiers, ZNCC, planar and volumetric supports, on the DTU Dataset[11]. Circles represent thresholds that optimize sensitivity + specificity with the values 0.2, 0.5 and 0.5 respectively.

Fig. 9

 9 Fig.9ROC Curves of four di↵erent classifiers using 8 ⇥ 8 receptive fields with various depths. Circles represent thresholds that optimize sensitivity + specificity.

Fig. 10

 10 Fig. 10 An example of sparse synthetic performance capture data generation. (top) Top and side view of the 10 cameras positioned around a surface. (bottom) Four examples of generated points of view.

Fig. 11 ROC

 11 Fig. 11 ROC Curves of two di↵erent classifiers using planar and volumetric receptive fields, on the sparse synthetic data. Circles represent thresholds that optimize sensitivity + specificity.

Fig. 12 (

 12 Fig. 12 (Left ) 3 input images, (middle) plane based classifier, (right) volumetric classifier. The face is highly occluded in many views (left) making the reconstruction noisy and inaccurate when using a planar support whereas the volume counterpart yields smoother and more accurate details.

Fig. 13 (

 13 Fig. 13 (top row ) Input images of captured subjects. (middle row ) Reconstructions without probability accumulation along rays. (bottom row ) Results with accumulation.

Fig. 14 (

 14 Fig. 14 (top) Input images, (middle) result with [10], (bottom) result with our method. Motion blur and low contrast are visible in the input images . Best viewed magnified.

Fig. 15

 15 Fig. 15 Close up view of the arm region in Figure 1. (left) Results from [10], (right) our reconstruction

Fig. 17

 17 Fig. 17Qualitative comparison with [6]. (Left ) input image with the horizontal section in red, (middle) point cloud with [6], (right-top) point cloud horizontal section with [6] (rightbottom) point cloud horizontal section with our approach.

  Fig. 17Qualitative comparison with [6]. (Left ) input image with the horizontal section in red, (middle) point cloud with [6], (right-top) point cloud horizontal section with [6] (rightbottom) point cloud horizontal section with our approach.

  -middle), plaguing the subsequent surface extraction step. Fig-ure 17-left also shows a horizontal section of the model in a poorly contrasted image region of the dress. The global strategy used in

Fig. 18 (

 18 Fig. 18 (top) Results provided by [8] on the kick 540 sequence. (middle) Poisson Reconstruction of the output point cloud of [8]. (bottom) Our result.

Fig. 19

 19 Fig. 19 Point clouds density comparison between results provided by [8] (left) and our output (right). Best viewed magnified.

Fig. 20

 20 Fig. 20 Two points of view of a subject from [23] (left). (middle) Reconstruction provided by the authors. (right) Results using our learning strategy.

Fig. 21

 21 Fig. 21 Close up of the face of the subject from [23] (left). The reconstruction provided by the authors (middle) is very smooth compared to our result (right).

40 CHAPITRE 4. APPEARANCE ESTIMATION AND REFINEMENTS Appearance map T Warp Blur kernel K Subsampling S Pixel Noise & Perturbations ⌧ Mapping i-th view projection at time t Model M t Appearance projection operator P t i Scene geometry High resolution projected image H t i Warped high hesolution image Imaging F W t i Blurred high resolution Image Noiseless low resolution image Input: noisy low res. image Time t

  Summary of image formation model and problem notation.

	⇡ t i
	FIGURE 4.1 -

Table 1 .

 1 Mean error and standard deviation over the sequence of the Marker dataset.

		method			mean error (mm) standard deviation (mm)
	no coupling, no mean pose [9]		55.11			48.02
		our method				43.22			29.58
		Liu et al. [19]				29.61			25.50
							Mean Error at Markers
		250			without coupling estimation, without mean pose our method
	Mean Error (mm)	100 150 200
		50
		0	0	50	100	150	200	250	300	350	400	450	500
								Frame

  . AN EFFICIENT VOLUMETRIC FRAMEWORK FOR SHAPE TRACKING 83

		∴
	An Efficient Volumetric Framework for Shape Tracking
	Benjamin Allain	Jean-Sébastien Franco
		Inria Grenoble Rhône-Alpes -LJK Grenoble Universities, France
		firstname.lastname@inria.fr
	A.3 AN EFFICIENT VOLUMETRIC FRAMEWORK FOR SHAPE TRACKING
	Benjamin Allain, Jean-Sébastien Franco, Edmond Boyer CVPR 2015 -IEEE
	International Conference on Computer Vision and Pattern Recognition, Jun 2015, Boston, United States. (Oral Presentation, acceptance rate <3%)

Table 1 .

 1 Variation of the estimated volume over the sequence for MARKER and BALLET datasets.

	Allain et al. 2014 [2] our method	.85 4.32 2.24	1.22 1.20 0.95

Table 2 .

 2 Mean and statistics of silhouette reprojection error over BALLET dataset, expressed in percentage of silhouette area.

	method	mean stddev. median max
	Cagniart et al. [8] Allain et al. [2] Ours, no vol. fitting 4.62 5.74 5.81 Ours 4.56	1.88 1.70 1.94 1.21	5.48 5.61 4.28 4.43	15.20 13.77 17.20 11.00
	method		mean std. dev.
	Cagniart et al. [8] Allain et al. [2] Proposed, no surface fitting 42.60 55.11 43.22 Proposed 38.41 Liu et al. [21] 29.61	48.02 29.58 29.32 26.70 25.50

1

Video available at https://hal.inria.fr/hal-01141207

Table 3 .

 3 Mean marker error and std.dev. (mm), MARKER dataset. Note that Liu et al. assume a rigged skeleton is associated to the template, a stronger and more restrictive assumption.

  4. ESTIMATION OF HUMAN BODY SHAPE IN MOTION WITH WIDE CLOTHING 95

	Estimation of Human Body Shape in Motion
	with Wide Clothing
	∴
	Jinlong Yang, Jean-Sébastien Franco, Franck Hétroy-Wheeler, and
	Stefanie Wuhrer
	A.4 ESTIMATION OF HUMAN BODY SHAPE IN MOTION WITH WIDE CLOTHING
	Jinlong Yang, Jean-Sébastien Franco, Franck Hétroy-Wheeler, Stefanie Wuhrer
	European Conference on Computer Vision 2016, Oct 2016, Amsterdam, Netherlands

: hal-01348837 https://hal.inria.fr/hal-01348837

	Submitted on 7 Oct 2016
	HAL is a multi-disciplinary open access	L'archive ouverte pluridisciplinaire HAL, est
	archive for the deposit and dissemination of sci-	destinée au dépôt et à la diffusion de documents
	entific research documents, whether they are pub-	scientifiques de niveau recherche, publiés ou non,
	lished or not. The documents may come from	émanant des établissements d'enseignement et de
	teaching and research institutions in France or	recherche français ou étrangers, des laboratoires
	abroad, or from public or private research centers.	publics ou privés.

  e 0 initializations; while e i < e i-1 do Compute alignment warps:{w k } k∈[[1..N ]] s.t. A ref ≈ I k (x + w k ); Align texture maps: A k = I k (x + w k ); Update alignment error: e i = k A k -A ref 2 ;Set A ref as the texture that gives the medoid of the aligned textures:

TABLE 1

 1 Average silhouette overlap error in pixels 4 sequences at low frame rate. Image resolution: 1920×1080.

	Crane 7746.40 surICP [2] 8295.58 16759.29 Jumping Bouncing Handstand ours 9148.94 6847.72 9279.57 9400.76 11690.61

  Huang received the MSc degree in computer and communication engineering from National Cheng-Kung University in 2010. After one year in Academia Sinica as a research assistant, he started his doctoral study in Technische Universit ät M ünchen in 2012 and obtained the Ph.D. degree in 2016. His research interests include 2D/3D conversion, human motion capture and other 3D-vision related topics. He received Studying Abroad Scholarship from Taiwan government and the best paper award runner-up in 3DV'13. Benjamin Allain received the MSc degree in computer science from Ensimag -Grenoble INP, France, in 2012. Then he joined the Morpheo group at Inria Grenoble Rh ône-Alpes and obtained his PhD degree in computer science from Universit é Grenoble Alpes in 2017. Since November 2016, he has been working as a research scientist at Smart Me Up, France. His research interests include non-rigid motion tracking of 3D shapes from multiview video sequences. Edmond Boyer is a senior research scientist at the INRIA where he leads the MORPHEO research team. He obtained his PhD from the Institut National Polytechnique de Lorraine in 1996. He was research assistant at Cambridge university in 1998 before joining the INRIA. His fields of competence cover computer vision, computational geometry and virtual reality. He has done pioneering work in the area of geometric 3D reconstruction with focus on objects with complex shapes like the humans. Edmond Boyer is co-founder of the 4D View Solutions (http://www.4dviews.com/), one of the leading companies worldwide specialized in multi-view acquisition and processing. His current research interests are on 3D dynamic modeling from images and videos, motion perception and analysis from videos. Jean-Sebastien Franco is assistant professor of computer science at the Ensimag (School of Computer Science and Applied Mathematics, Grenoble Universities), and a researcher at the Inria Grenoble Rh ône-Alpes and LJK lab, France, with the Morpheo team since 2010. He obtained his Ph.D. from the Institut National Polytechnique de Grenoble in 2005 with the Inria MOVI / Perception team. He started his professional career as a postdoctoral research assistant at the University of North Carolinas Computer Vision Group in 2006, and as assistant professor at the University of Bordeaux, with the IPARLA team, INRIA Bordeaux Sud-Ouest. His expertise is in the field of computer vision, with several internationally recognized contributions to dynamic 3D modeling from multiple views and 3D interaction.

  Ilic is currently senior key expert research scientist at Siemens Corporate Technology in Munich, Perlach. He is also a visiting researcher and lecturer at Computer Science Department of TUM and closely works with the CAMP Chair. From 2009 until end of 2013 he was leading the Computer Vision Group of CAMP at TUM, and before that he was a senior researcher at Deutsche Telekom Laboratories in Berlin. In 2005 he obtained his PhD at EPFL in Switzerland under supervision of Pascal Fua. His research interests include: 3D reconstruction, deformable surface modelling and tracking, real-time object detection and tracking, human pose estimation and semantic segmentation.Valentin Gabeur 1,2 Jean-Sébastien Franco 1 Xavier Martin 1 Cordelia Schmid 1,2 Grégory Rogez

	A.7. MOULDING HUMANS : NON-PARAMETRIC 3D HUMAN SHAPE
	ESTIMATION FROM SINGLE IMAGES	149
	∴
	Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images
	A.7 MOULDING HUMANS : NON-PARAMETRIC 3D HUMAN SHAPE ESTIMATION FROM SINGLE IMAGES
	Valentin Gabeur, Jean-Sébastien Franco, Xavier Martin, Cordelia Schmid, Gregory
	Rogez. ICCV 2019 -International Conference on Computer Vision, Oct 2019, Seoul,
	South Korea. pp.1-10

Table 1 .

 1 Generator architecture.

		Layer type	Output shape
	Input	Input	256x256x3
	Conv1	Conv 7x7 stride=2, GroupNorm, Relu	128x128x64
	Layer1	Residual module expanded	128x128x128
	Layer2	Residual module expanded	128x128x256
	Layer3	Residual module	128x128x256
	Hg1	Hourglass, skipped connections = 5	128x128x2
	Hg2	Hourglass, skipped connections = 5	128x128x2

Table 2 .

 2 Discriminator architecture.

	Layer	Layer type	Output shape
	Input	Input	128x128x2
	Conv1	Conv 3x3 stride=1, GroupNorm, Relu	128x128x64
	MP1	MaxPool 2x2	64x64x64
	Conv2	Conv 3x3 stride=1, GroupNorm, Relu	64x64x128
	MP2	MaxPool 2x2	32x32x128
	Conv3	Conv 3x3 stride=1, GroupNorm, Relu	32x32x256
	MP3	MaxPool 2x2	16x16x256
	Conv4	Conv 3x3 stride=1, GroupNorm, Relu	16x16x512
	MP4	MaxPool 2x2	8x8x512
	FC1	Fully connected layer	1024
	FC2	Fully connected layer	512
	FC3	Fully connected layer	1

Table 1

 1 Classifier accuracy (%).

	Camera #	5	20	49
	ZNCC	64.98	65.46	65.58
	Ours Plan.	80.67	77.87	75.92
	Ours Vol.	82.95 84.84 83.45

Table 2

 2 Reconstruction accuracy and completeness.

		Acc.	Compl.
	Measure	Mean	Med.	Mean	Med.
	Tola et al. [49]	0.448 0.205	0.754	0.425
	Furukawa et al. [16]	0.678	0.325	0.597	0.375
	Campbell et al. [50]	1.286	0.532	0.279 0.155
	Ji et al. [6]	0.530	0.260	0.892	0.254
	Ours (fused)	0.490	0.220	0.532	0.296
	Hartmann et al. [39]	1.563	0.496	1.540	0.710
	Ours (depthmap)	0.599 0.272 1.037 0.387

https://hal.archives-ouvertes.fr/hal-01567758/file/1361-supp.mp4

Due to the chronology of this work, at the time we investigated the alignment methods described in this chapter, we were using mostly polyhedral visual hulls obtained with algorithms from my thesis[START_REF] Franco | Exact polyhedral visual hulls[END_REF][START_REF]Efficient polyhedral modeling from silhouettes[END_REF], as these were fast, rugged, and implemented with the platform software suite we were using with Kinovis ; but this doesn't alter the essence of this discussion.

https://hal.inria.fr/hal-01016981/file/Allain_ECCV2014_On_Mean_ Pose_and_Variabiliy.mp4

https://hal.inria.fr/hal-00977755/file/CVPR2014-HR-3D-Shape-Texture-from-Videos. mp4

https://hal.inria.fr/hal-01348837/file/EigenAppearance.mp4

http://hal.inria.fr/hal-01016981

The benchmark can be downloaded at http://dressedhuman.gforge.inria.fr/.

This is the author's version of an article that has been published in this journal. Changes were made to this version by the publisher prior to publication.The final version of record is available at http://dx.doi.org/10.1109/TPAMI.2017.2740308Copyright (c) 2017 IEEE. Personal use is permitted. For any other purposes, permission must be obtained from the IEEE by emailing pubs-permissions@ieee.org.

https://kinovis.inria.fr

Acknowledgement

This work was funded by the Seventh Framework Programme EU project RE@CT (grant agreement no. 288369).

Acknowledgements

This work was funded by the Seventh Framework Programme EU project RE@CT (grant agreement no. 288369). It was also supported in part by the INRIA-JSPS Bilateral Program AYAME 146121400001 and the JSPS WAKATE B 26730089.

Acknowledgements

This work was funded by the Seventh Framework Programme EU project RE@CT (grant agreement no. 288369). We thank Li Wang and Franck Hetroy for providing the CVT computation code.

Acknowledgments

Funded by France National Research grant ANR-14-CE24-0030 ACHMOV. We thank Yannick Marion for help with code to e ciently fit an S-SCAPE model to a single frame, Leonid Pishchulin for helpful discussions, Alexandros Neophytou and Adrian Hilton for providing comparison data, and Mickaël Heudre, Julien Pansiot and volunteer subjects for help acquiring the database. The database was acquired using the Kinovis platform: http://kinovis.inrialpes.fr/.

ACKNOWLEDGMENTS

Several datasets proposed in this paper have been acquired using the Kinovis platform at Inria Grenoble Rh ône-Alpes (http://kinovis.inrialpes.fr).

Acknowledgements. We thank Pau De Jorge and Jinlong Yang for their help in capturing the data employed in this paper. The dataset was acquired using the Kinovis 2 platform. This work was supported in part by ERC advanced grant Allegro.

Acknowledgements

This work was conducted at the INRIA Grenoble. Funded by France National Research grant ANR-14-CE24-0030 ACHMOV. Images 1 -2 -15 -17 of Anja Rubik courtesy of Ezra Petronio and Self Service Magazine. Geometric model in figure 7 courtesy of 3DScanStore [54].

, dedicated to the high definition acquisition of subjects in motion (68 4Megapixels cameras, 100m2 studio, 17 twelve-core PC cluster). -Participant and scientific coordinator for IPARLA team, for the ANR-DALIA project (French Ministry of Research Project Grant), from September 1st, 2007 to July 2010. 50% participation. Telepresence and collaborative 3D interaction, in collaboration with MOAIS and Perception teams (INRIA Rhône-Alpes, France) and PRV (LIFO lab, Orléans, France). RNTL OCETRE, FP6-IST STREP HOLONICS (european commission), ACI Jeune Chercheur Cyber I and ACI Masse de Données Cyber II, (mixed-reality projects) at INRIA Rhône-Alpes, France. 3. Video available at https://hal.inria.fr/hal-01141207 4. https://hal.inria.fr/hal-01141207/file/Allain_CVPR2015_ Volumetric_Tracking.avi

List of publications

-45 international peer reviewed, among which :

-37 international conferences (with 20 ICCV/CVPR/ECCV, high impact computer vision conferences). -7 international peer reviewed journals (PAMI, IJCV, TVCG, IJMDB) -1 book chapter -3 international workshops & demos (VRST, AMDO) -5 French peer reviewed conferences (RFIA) -3 research reports