Chun-Hao Huang

Edmond Boyer
email: edmond.boyer@inria.fr

Bibiana Do

Canto Angonese

Nassir Navab
email: navab@in.tum.de

Slobodan Ilic

Toward User-specific Tracking by Detection of Human Shapes in Multi-Cameras

Human shape tracking consists in fitting a template model to temporal sequences of visual observations. It usually comprises an association step, that finds correspondences between the model and the input data, and a deformation step, that fits the model to the observations given correspondences. Most current approaches find their common ground with the Iterative-Closest-Point (ICP) algorithm, which facilitates the association step with local distance considerations. It fails when large deformations occur, and errors in the association tend to propagate over time. In this paper, we propose a discriminative alternative for the association, that leverages random forests to infer correspondences in one shot. It allows for large deformations and prevents tracking errors from accumulating. The approach is successfully integrated to a surface tracking framework that recovers human shapes and poses jointly. When combined with ICP, this discriminative association proves to yield better accuracy in registration, more stability when tracking over time, and faster convergence. Evaluations on existing datasets demonstrate the benefits with respect to the state-of-the-art.

Introduction

Visual shape tracking is the process of recovering temporal evolutions of a template shape using visual information, such as image silhouettes or 3D points. It finds applications in several domains including computer vision, graphics and medical imaging. In particular, it has recently demonstrated a good success in marker-less human motion capture (mocap). Numerous approaches assume a user-specific reference surface, and the objective is to recover the skeletal poses [START_REF] Vlasic | Articulated mesh animation from multi-view silhouettes[END_REF], surface shapes [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF], or both simultaneously [START_REF] Huang | Human shape and pose tracking using keyframes[END_REF].

Most of these model-based methods [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF][START_REF] Gall | Motion capture using joint skeleton tracking and surface estimation[END_REF][START_REF] Huang | Human shape and pose tracking using keyframes[END_REF][START_REF] Liu | Markerless motion capture of interacting characters using multi-view image segmentation[END_REF][START_REF] Vlasic | Articulated mesh animation from multi-view silhouettes[END_REF]] can be viewed as extensions of Iterative-Closest-Point (ICP) framework [START_REF] Besl | A method for registration of 3-d shapes[END_REF], which attempts to explain newly observed data using the previous outcomes. As long as the initialization is close to the optimum solution, it is able to produce outstanding results. However, they also suffer from inherent weaknesses of generative strategies, e.g., the slow convergence. Moreover, when large deformations or outliers occur, discovering associations between data and models is particularly difficult. Unreliable associations result in ambiguous situations that yield erroneous numerical solutions and, consequently, break the tracking process.

In contrast, discriminative approaches that 'detect' rather than track models have shown better robustness over the past decade, for instance, in human pose estimation with Kinect [START_REF] Shotton | Real-time Human Pose Recognition in Parts from Single Depth Images[END_REF][START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF]. These approaches operate frameindependently, and are generally drift free. In this paper, we explore this direction in order to get robust observationmodel associations, regardless of the results from previous frames. We further present a discriminative 'tracking-bydetection' human mocap framework, as in Fig. 1. Inspired by Taylor et al. [START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF], we apply regression forests to improve the associations. Shape geometries are characterized by volumetric representations, and are fed into user-specific forests to predict correspondences in one shot. Contrary to generative methods, this prediction does not require close initializations from a nearby frame. In addition, it allows a single model to be used as a reference surface for several different sequences, again even if large deformations or outliers exist. We combine this strategy with a generative tracking approach that takes our one-shot associations as input. Experiments demonstrate that this hybrid discriminativegenerative framework leads to better, or comparable results than purely generative approaches, e.g. [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF][START_REF] Corazza | Markerless motion capture through visual hull, articulated icp and subject specific model generation[END_REF], reducing error accumulations and hence increasing the stability. The contribution of this work is two-fold: (1) a one-shot correspondence inference with complete 3D input (rather than 2.5D as in [START_REF] Shotton | Real-time Human Pose Recognition in Parts from Single Depth Images[END_REF][START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF]), which leads to (2) a hybrid subjectspecific human pose and shape capture method that relies little on former results (unlike other ICP-like methods [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF][START_REF] Gall | Motion capture using joint skeleton tracking and surface estimation[END_REF][START_REF] Huang | Human shape and pose tracking using keyframes[END_REF][START_REF] Liu | Markerless motion capture of interacting characters using multi-view image segmentation[END_REF]), holding the ability to recover from drifting.

Related work

Among the vast literature on human pose estimation [START_REF] Moeslund | A survey of advances in vision-based human motion capture and analysis[END_REF], we focus on top-down approaches that assume a 3D model and deform it according to input data, either directly with pixels as in [START_REF] Gall | Motion capture using joint skeleton tracking and surface estimation[END_REF][START_REF] Liu | Markerless motion capture of interacting characters using multi-view image segmentation[END_REF][START_REF] Straka | Simultaneous shape and pose adaption of articulated models using linear optimization[END_REF], or with 3D points as in [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF][START_REF] Corazza | Markerless motion capture through visual hull, articulated icp and subject specific model generation[END_REF][START_REF] Huang | Human shape and pose tracking using keyframes[END_REF]. These methods typically decompose into two main steps: (i) data association, where observations are associated to the model, and (ii) deformation estimation, where deformation parameters are estimated given the associations. Our primary objective in this paper is to improve the first part. Existing approaches for this issue are discussed below.

Generative approaches. Methods in this category do not require any training. They follow the association strategy in ICP while extending the motion model to more general deformations than the one in the original method [START_REF] Besl | A method for registration of 3-d shapes[END_REF]. Associations are addressed by searching for closest points, with various distance measures such as point-to-point [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF], pointto-plane [START_REF] Chen | Object modeling by registration of multiple range images[END_REF], or Mahalanobis distances [START_REF] Straka | Simultaneous shape and pose adaption of articulated models using linear optimization[END_REF]. This strategy heavily relies on the fact that observations in consecutive frames are in vicinity. Klaudiny et al. [START_REF] Klaudiny | Towards optimal non-rigid surface tracking[END_REF] generalize the idea from the previous frame to a certain frame in the considered sequences, finding the best non-sequential order to track, but the proximity assumption remains. Gall et al. [START_REF] Gall | Motion capture using joint skeleton tracking and surface estimation[END_REF] establish 3D-2D correspondences by considering both texture in images, and contours in silhouettes. Later, Liu et al. [START_REF] Liu | Markerless motion capture of interacting characters using multi-view image segmentation[END_REF] include image segmentation information in order to differentiate multiple interacting subjects. These approaches implicitly assume that observations only describe the tracking subjects, which does not necessarily hold in 3D data that often contain fake geometries. Cagniart et al. [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF] introduce an additional outlier class to reject associations with noisy observations. Data is explained by Gaussian Mixture Models (GMM) in an Expectation-Maximization (EM) manner. Huang et al. [START_REF] Huang | Human shape and pose tracking using keyframes[END_REF] follow a similar concept, but aggregate the outlier likelihood over every Gaussian component and offer better robustness. All these generative methods are highly likely to fail in large deformations. Furthermore, they are prone to error accumulations and, as a result of matching several successive frames wrongly (whether sequentially or not), they are prone to drift.

Discriminative approaches. Recently, discriminative approaches have demonstrated their strengths in tracking human poses with depth images [START_REF] Baak | A data-driven approach for real-time full body pose reconstruction from a depth camera[END_REF][START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF]. Taylor et al. [START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF] propose a single-frame, or so called one-shot strategy, which yields decent dense correspondences without iterative refinements. With the help of regression forests, they map each foreground pixel to a weighted point in 3D, and thereby search the closest point within a predefined surface. Later, Pons-Moll et al. [START_REF] Pons-Moll | Metric regression forests for human pose estimation[END_REF] train forests with a new objective in metric space, and couple the one-shot strategy with ICP. In the case of full 3D, Kanaujia et al. [START_REF] Kanaujia | Part segmentation of visual hull for 3d human pose estimation[END_REF] use shape context histograms as descriptors, segment visual hulls into body parts with a pre-trained Support Vector Machine (SVM), and build the skeletons with the method similar to [START_REF] Shotton | Real-time Human Pose Recognition in Parts from Single Depth Images[END_REF]. Starka et al. [START_REF] Starck | Correspondence labelling for wide-timeframe free-form surface matching[END_REF] formulate the matching problem as the inference of Markov random field (MRF). Rodola et al. [START_REF] Rodola | Dense non-rigid shape correspondence using random forests[END_REF] apply forests to learn the parameters of wave kernel signatures [START_REF] Aubry | The wave kernel signature: A quantum mechanical approach to shape analysis[END_REF] during training, and facilitate dense matching between two meshes.

To avoid computation overhead, we develop volumetric normal fields to describe meshes in a discretized volume Ω 3 , and extend the comparison features from 2.5D [START_REF] Shotton | Real-time Human Pose Recognition in Parts from Single Depth Images[END_REF][START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF] to full 3D data. Our method can be viewed as a trackingby-detection approach for human shape tracking.

Preliminaries and method overview

We first state the problem and briefly outline our method. A 3D reference surface is denoted as M = (M, T M ), where M = {x v } Nv v=1 ⊂ R 3 are the locations of vertices v, and T M defines the triangles. Evolving M typically amounts to parameterizing M as a function of deformation parameters Θ, namely, M(Θ). Unlike [START_REF] Gall | Motion capture using joint skeleton tracking and surface estimation[END_REF][START_REF] Liu | Markerless motion capture of interacting characters using multi-view image segmentation[END_REF][START_REF] Taylor | The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation[END_REF], our interest lies in recovering not only poses but also shapes of the subjects. We adopt a surface deformation framework that groups vertices into N p patches [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF], and assign each of them a rigid body motion θ. Thus, Θ is the collection of all θ, Θ = {θ k } Np k=1 , encoding the global shape of the surface. We refer the readers to [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF] for detailed explanations.

Given an observed visual hull Y t = (Y t , T t Y ) 1 , where Y t = {y i } Ny i=1 ⊂ R 3 , the goal is to determine the optimal Θt such that M t = M( Θt ) resembles Y t as much as possible. It typically boils down to two sub-problems: Standard ICP-based approaches [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF][START_REF] Corazza | Markerless motion capture through visual hull, articulated icp and subject specific model generation[END_REF] alternate between these two steps, refining C t and Θ t iteratively. They require M t-1 to be close to Y t , and is usually slow to converge. We develop a different strategy that warps the input data Y to the reference mesh M, denoted as Ỹ = ( Ỹ, T Y ), and visualized as a triangular mesh in Fig. 2. If the warping is perfect, this mesh will look clean and resemble M as much as possible. Incorrect mapping, on the other hand, can be told from huge edges. Vertex positions Ỹ represent the locations of potential matches between Y and M. Thus, C can be built directly by doing nearest neighbor search between Ỹ and M just once, as illustrated in Fig. 2.

Specifically, we consider this R 3 → R 3 mapping as a composite one: R 3 → Ω 3 → R 3 . The former mapping is voxelization (Sec. 4.1), while the latter is regression (Sec. 5). A forest is trained with many voxelized meshes off-line (Sec. 5.1). During runtime, y i is first mapped to a voxel v i , and then regressed to a 3D point ỹi ∈ Ỹ. Given the properly estimated C, poses and shapes are then recovered as in [START_REF] Huang | Robust human body shape and pose tracking[END_REF] (Sec. 5.2). Fig. 2 shows our pipeline.

Normal volume and features

Before entering the forest, we cast our data into a volumetric field N : Ω 3 ⊂ R 3 → R 3 , and design a set of features to describe surface geometries in volumes.

Volumetric normal field (VNF)

Voxelizing a mesh in general consists of two parts: (1) finding voxel positions for every vertex, and (2) testing the overlap between triangles and voxels. The first part can be viewed as a quantization mapping from Euclidean space to a discretized space v : R 3 → Ω 3 . The size of the volume is large enough to include all possible pose variations, and its center is aligned with the mean of the surfaces. The voxel size is chosen to be close to the average edge length of meshes, so that a single voxel is not mapped by too many vertices. To check the intersection of triangles with voxels, we apply separating axis theorem which is known to be efficient for collision detection in graphic community [START_REF] Akenine-Möller | Fast 3d triangle-box overlap testing[END_REF].

After determining voxels occupied by the surface, referred to as v suf , we further identify voxels located inside and outside the surface, denoted respectively as v in and v out . Together they define a truncated normal field:

N(v) =      + if v out n ∈ [-1, +1] 3 , n = 1 if v suf - if v in . (1)
n is the average surface normal from the containing triangles, whereas is an indicator vector ( [2, 2, 2] in our experiments). Given a mesh, our VNF naturally encodes both spatial occupancies (surface locations) and first-order derivatives (normals). It shares a similar spirit with implicit surface representations, e.g. level-set, but does not require any expensive distance transform computations.

Volumetric features

Here, we present the features f (v suf ) for describing the above VNF, which are later used to train the forests. Conventionally, only one feature dimension κ is selected to separate data at each branch node of the forest [START_REF] Criminisi | Decision forests for computer vision and medical image analysis[END_REF]. One does not have to prepare the whole high-dimensional feature vector for predictions, because only a few dimensions are needed. The calculation of f is suggested to be dimensionally independent. We therefore avoid descriptors that requires normalization, like MeshHOG [START_REF] Zaharescu | Keypoints and local descriptors of scalar functions on 2d manifolds[END_REF], or SHOT [START_REF] Tombari | Unique signatures of histograms for local surface description[END_REF], and resort to comparison features used in [START_REF] Criminisi | Regression forests for efficient anatomy detection and localization in ct studies[END_REF][START_REF] Shotton | Real-time Human Pose Recognition in Parts from Single Depth Images[END_REF].

As depictcted in Fig. 3, for each surface voxel v suf (blue), we shoot two offsets (red vectors) ψ = (o 1 , o 2 ) ∈ Ω 3 × Ω 3 , and reach two neighboring voxels (green). To describe the local geometry, we concatenate the dot product of their normals, and the difference of VNF within local cuboids to form the feature vector f . Such a computation is repeated for multiple offsets at v suf . By definition, each dimension of f can be evaluated independently. See Supplementary Material for more details.

Note that f is a function of v suf , but takes an offset pair ψ, a binary variable η (whether using Local Coordinate Frame Local coordinate frame. Constructing a LCF requires 3 orthonormal vectors: [e 1 , e 2 , e 3 ]. They usually come from doing PCA on a local region, which is impractical to repeat for all surface voxels v suf . Instead, we apply the following method, as illustrated in Fig. 4(a). The first eigenvector e 1 is always the normal vector (red arrow). e 3 is simply e 1 ×e 2 if the second eigenvector e 2 is known. First we open a local cuboid around the current voxel (blue). Among neighboring gray surface voxels, we find the one that has least normal changes, and yet located far from the current voxel. In practice, given the notations in Fig. 4(a), we select the one that maximizes ( d + cos ϕ), where ϕ is the angle between normals, and d is the normalized distance. Given this reference voxel (yellow), a reference vector (purple) is decomposed into two components. The component perpendicular to the normal is then chosen as e 2 (green). Fig. 4(b) shows that this approach produces LCFs quasi-covariant to pose changes, and as a result, quasi-pose-invariant features f .

Correspondence inference and tracking

Now we proceed to the second mapping: Ω 3 → R 3 , where the surface voxel v suf traverses a pre-trained regression forest according to the feature responses, and attains a point in R 3 that lies on the surface embedding defined by the vertices of the reference model M.

Random forest

A forest is an ensemble of T binary decision trees, each separating data with split functions at branch nodes and storing statistic models at leaf nodes. The training objectives, split functions, and statistical models vary from task to task. We refer readers to [START_REF] Criminisi | Decision forests for computer vision and medical image analysis[END_REF] for a comprehensive review.

Training. To incorporate abundant training variations, we animate M to a variety of poses with a method similar to [START_REF] Sumner | Deformation transfer for triangle meshes[END_REF]. After voxelizing all the animated meshes, we associate each surface voxel to their locations at rest pose, and obtain a pool of sample-label pairs S = {(v suf , x v )}. Each tree is trained with a random subset of S. Our splitting candidate is φ = (ψ, η, κ, τ ) (offset pairs ψ, binary variables η, testing feature channels κ, and thresholds τ ). Let S N denotes the samples arriving at a certain branch node. The training process is to partition S N recursively into two subsets S L and S R , based on randomly generated φ:

S L (φ) = {v suf ∈ S N |f κ (v suf ; R η (ψ)) ≥ τ } (2a) S R (φ) = {v suf ∈ S N |f κ (v suf ; R η (ψ)) < τ }. ( 2b
)
Here f κ denotes the κ th dimension of the feature f . Whether φ is a good split or not depends on whether it produces more homogeneous subsets. In supervised learning, this is often measured by information gain:

G(φ) = H(S N ) - i∈{L,R} |S i (φ)| |S N | H(S i (φ)), (3)
where H = σ 2 (•) is the entropy, measured by the variance of all x v in the sample set as in [START_REF] Gall | Hough forests for object detection, tracking, and action recognition[END_REF]. The split that maximizes the information gain, φ * = arg max φ G(φ), is stored for the later prediction use. The tree recursively splits samples and grows until one of the following stopping criteria is true: (1) it reaches the maximum depth, or (2) the number of samples |S N | is too small. Once a leaf node is reached, we perform mean-shift clustering to represent the distributions of x v as a set of confidence-weighted modes H = {(h, ω)}. h ∈ R 3 is the mode location and ω is a scalar weight.

Outliers such as false geometries, or un-removed background often exist in visual hulls, drastically deteriorating tracking results. If their models are available, we also include them in the training process, so that forests can reject them online. In this case, the goodness of a split φ should be evaluated in terms of both classification and regression. We follow Fanelli et al. [START_REF] Fanelli | Random forests for real time 3d face analysis[END_REF] and adapt the entropy as:

H(S) = - c p(c|S) log p(c|S) + (1 -e D α )σ 2 (S), (4)
where p(c|S) is the class probability of being foreground or background. Eq. 4 is the weighted sum of the afore-mentioned regression measure σ 2 and the classification entropy measure. The regression part gets increasing emphasis when the current depth D gets larger (i.e., the tree grows deeper), and the steepness is controlled by the parameter α.

Subject coordinate frame. To achieve global rotation invariance, we always rotate meshes into a canonical orientation before the voxelization. In the training phase, since each animated mesh is accompanied with a skeletal pose, we compute two unit-length vectors from the skeleton, and align them to xand z-axis respectively. The first one is the common perpendicular vector of bone RShoulder-Torso and bone LShoulder-Torso, while the second one is the sum of them. See Supplementary Material for illustrations.

Recall that the volume center is aligned with the means of meshes, which brings translation invariance. Together these two steps characterize a subject coordinate system, simulating the PCA process on a whole mesh. During tracking, we approximate the skeletal pose using results of the previous frame, and repeat the same process.

Prediction. In the prediction phase, a voxel lands on T leaves containing different collections of modes, denoted together as G = {H 1 ...H T }. A standard way of aggregation is doing mean-shift and keeping the cluster with largest weight. This is usually done independently for each sample. For instance, in Fig. 5(a), the green vertex aggregates with only green leaves, and black vertices aggregate with their respective black leaves as well. The consequent Ỹ is however, often noisy, where vertices of same triangles are mapped to locations distant from each other, as in Fig. 5(b).

We utilize the triangles T Y to ameliorate this problem. All y i are first mapped to voxels, sent into the forest once, and return in total N y × T predictions. For each vertex i, we consider not only its own predictions, but also those from the neighbors N i , e.g. all green and black leaves in Fig. 5(a). We sort the set G i ∪G Ni descendingly according to their weights ω, and do mean-shift only with the first half of them. The new mode location with highest confidence is the final output ỹi . This strategy respects the mesh connectivity and results in more structured forest predictions. Compare Fig. 5(c) to (b) to see the improvements.

Nearest neighbor search. Given the regression results

ỹi , each vertex i in input gets a closest vertex p in the reference vertex set V M , i.e. p = arg min v∈V M ỹix v 2 . Similar to [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF], we reject the searched correspondences if their normals differ from each other too much. Our advantage over [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF] is that M and its normals are fixed throughout tracking. One does not have to re-compute normals online, and other speed up algorithms like kd-tree are also feasible. Each correspondence pair (i, p) is associated to a weight w ip = exp(-d 2 (i, p)/2l 2 ), where d(•) stands for Euclidean distance, and l is the averaged edge length on M. Since the forest gives relatively good initial guess of the matches. It is much easier to find the right matches in this way.

Energy functions

Recall that our goal is to estimate deformation parameter Θ, whose resulting M(Θ) best explains Y. Given correspondences C from above, a data term is formulated as:

E data (Θ; C) = (i,p)∈C w ip y i -x p (Θ) 2 2 , (5)
which is a standard sum of weighted squared distances. Since evolving a surface with discrete observations (even with a good C) is ambiguous by nature, regularization terms are usually introduced to exert soft constraints. We follow the simultaneous estimation framework [START_REF] Huang | Robust human body shape and pose tracking[END_REF], which applies the following regularization terms:

Preserving local rigidity. Without any prior knowledge of motion, patches are preferred to be distributed uniformly on M. Given a vertex v, the rigidity constraint enforces the predicted positions x v (θ k ) and x v (θ l ) from two adjacent patches P k and P l ∈ N k to be consistent:

E r (Θ) = N P k=1 P l ∈N k v∈P k ∪P l w kl x v (θ k ) -x v (θ l ) 2 2 , ( 6
)
where Θ is implicitly encoded in x v (θ k ) and x v (θ l ).

Pose posteriors. It is noteworthy that the method described so far applies to not only humans but all types of surfaces. Nevertheless, since our application is capturing human motions, skeletal poses are also of our interest. We employ the skeleton binding energy that keeps the relationship between skeletons and surfaces:

E b (Θ, J ) = N P k=1 w k T θ k (β 0 k ) -β k 2 2 . ( 7
)
Here β ∈ R Combining Eq. 5-7, we have the energy:

E(Θ, J ; C) = λE data (Θ; C) + E r (Θ) + E b (Θ, J ). (8)
λ defines the softness of the surface, and is empirically set as 10 throughout our experiments. Given an input Y, the regression forest returns a fixed response Ỹ, and hence a fixed C. We therefore apply standard Gauss-Newton method directly to find the minimizer of Eq. 8. Note anyway that refining C like ICP is always possible. In this case, our method provides better initializations than using last frame results, reducing the number of needed ICP-iterations.

Experiment results

The proposed method was evaluated extensively on 9 sequences, whose profiles are summarized in Table 1. An individual forest is trained for each subject with up to 200 meshes, depending on the number of vertices per mesh. For S1 -3 we train standard regression forest; for S4 & S5 we apply the adaptation in Eq. 4 due to the un-properly segmented chairs and tables in input data (α = 2). Growing T = 20 trees to depth 25 with 15000 testing offset pairs ψ takes about 8 hours. The performance of our method is analyzed in different aspects, both qualitatively and quantitatively. We evaluate shapes with the widely-used silhouette overlap error, and evaluate poses with the 2D distances between ground truths and the projected joint locations.

Descriptiveness. First we briefly demonstrate the discriminability of the VNF-forest combination. The task is finding correspondences on the reference surface of S2. We consider animated meshes as testing data, where groundtruth vertex indices are available. The error metric is the geodesic distance between predicted vertices and groundtruths, and we compare with MeshHOG [START_REF] Zaharescu | Keypoints and local descriptors of scalar functions on 2d manifolds[END_REF] which is the extension of image-based histogram of oriented gradient (HOG) on surface manifolds. Fig. 6 depicts the proportion of vertices whose errors are less than a certain distance. The VNF-forest framework gives overall 27.34% exact matches, while searching the vertex with closest MeshHOG response gives 46.07%. However, when increasing the tolerance of correctness, our method presents a significant improvement, and yet Mesh-HOG gains only a little. With our method, 75.47% of vertices obtain correspondences no farther than three times averaged edge length (green dashed line), and 48.39% for MeshHOG. Moreover, almost half of matches from Mesh-HOG lie outside the range of roughly the length of the lower arm (orange dashed line) from the ground truths. Such a phenomenon is confirmed in Fig. 7. Taking Fig. 7(b) as input, MeshHOG returns results as in Fig. 7(c), whereas our method gives visually smoother matches like Fig. 7(d). The proposed VNF-forest framework certainly provides more ours + ICP [START_REF] Straka | Skeletal graph based human pose estimation in real-time[END_REF] + [START_REF] Straka | Simultaneous shape and pose adaption of articulated models using linear optimization[END_REF] bpSVM [START_REF] Huang | Robust human body shape and pose tracking[END_REF] patchedOL [START_REF] Huang | Human shape and pose tracking using keyframes[END_REF] fixOL [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF] locally consistent associations, which, as shown below, are sufficiently accurate to guide plausible deformations.

Registration. Next we show the capacity of the proposed method on registration tasks. With the source surface and distant target points as in Fig. 8(a), ICP-like approach only produces results like Fig. 8(b), where many discrepancies still remain after 30 iterations. Instead, our VNF-forest framework brings the mesh closer to the target points in one shot as in Fig. 8(c). After 14 ICP-iterations it refines the results as in Fig. 8(d). In the following experiments if reference surfaces are not aligned with the first frames, we also register them before the tracking starts.

Tracking without outliers. For sequences without outliers, we compare with surface-based ICP (surICP) [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF] and articulated ICP (artICP) [START_REF] Corazza | Markerless motion capture through visual hull, articulated icp and subject specific model generation[END_REF], both of which explain data with GMM using the Expectation-Maximization algorithm. We run an additional ICP step to reduce the errors (ours + ICP) for all testing sequences. The averaged silhouettes overlap errors are shown in Fig. 10(a-b). In general, our method performs much better than artICP, and yields comparable results with surICP. However, our method requires less ICPiterations to converge (see Supplementary Material). It follows that, compared with using results of previous frames as initializations, our method is capable of providing better ones. Examples of the estimated correspondences, the deformed surfaces and the skeletons are shown in Fig. 12(a).

We further investigate what happens when there are tracking errors in previous frames. Fig. 9(a) shows the results of frame 91 in sequence bouncing. Note that left/right legs are crossed due to their close interaction with each other. In the next frame, when they are separate in the visual hull, the forest discovers correspondences more correctly, as in Fig. 9(b) (c.f. the colors of M in Fig. 2), and lead us to results in Fig. 9(c) without self-intersections. ICP strategy discover wrong associations and gives results in Fig. 9(d) under the same softness parameter λ. Errors propagate to the next frame, and gradually deteriorate the tracking, unless luckily regularization terms implies the opposite.

Tracking with outliers. Four of our testing sequences, Cutting, WalkChair1, HammerTable, and WalkChair2 contain tables or chairs in observations, which play the roles as static outliers. We compare with other outlier rejection strategies such as, fixed outlier proportion (fixOL) [START_REF] Cagniart | Probabilistic deformable surface tracking from multiple videos[END_REF], removing outliers by body-part classifications with SVM (bpSVM) [START_REF] Huang | Robust human body shape and pose tracking[END_REF], and modeling outlier likelihood dynamically by aggregating over all patches (patchedOL) [START_REF] Huang | Human shape and pose tracking using keyframes[END_REF]. As shown in Fig. 10(c-d), conventional outlier strategy fixOL drifts quickly and soon fail to track (green curves). ICP with robust outlier treatment, patchedOL, is able to sustain noisy input to a certain extent. Once it starts drifting, the error only gets higher due to its ICP nature (yellow curves). When subjects and outliers are sperate components in visual hulls, we cast them into VNF, and feed them separately into the joint classification-regression forest. If they are connected to each other, forests inevitably associate some outliers to vertices on the model, and cause undesirable deformations, as the spike in blue curves in Fig. 10(d). Nonetheless, since we rely less on previous frames for data associations, the results can always get recovered when they are separated again. In average, we still yield low errors throughout the whole sequences, as in Table 2. We remark that such ability to recover is the essence of our discriminative approach, which is the biggest advantages over the existing generative methods.

We also verify the efficacy on pose estimations, and compare to another single-frame approach from Straka et al. [START_REF] Straka | Skeletal graph based human pose estimation in real-time[END_REF][START_REF] Straka | Simultaneous shape and pose adaption of articulated models using linear optimization[END_REF]. The error metric is the discrepancy between estimated joint positions and the annotated ground truths. Fig. 11 plots the results of sequences HammerTable and WalkChair2, where we confirm again considerably better accuracy than bpSVM [START_REF] Huang | Robust human body shape and pose tracking[END_REF] (red curves), Straka et al. [START_REF] Straka | Skeletal graph based human pose estimation in real-time[END_REF][START_REF] Straka | Simultaneous shape and pose adaption of articulated models using linear optimization[END_REF] (green curves), and comparable results with patchedOL [START_REF] Huang | Human shape and pose tracking using keyframes[END_REF]. The recovered shapes and poses are also presented in Fig. 12(c), superimposed on original images.

Conclusion

We present a hybrid human shape tracking approach equipped with an alternative data-model association strategy. With the help of regression forests, we learn from preanimated meshes to discover better correspondences on input visual hulls. The one-shot property effectively prevent tracking errors from accumulating, yielding more stability compared with other generative ICP extensions. It contributes to locally consistent correspondences, which speeds up the convergence of ICP when used as the initializations. The reliability of the proposed method is confirmed by the experiments on numerous public sequences. Future directions include alleviating problems of topological changing and incorporating photometric information.

Figure 1 .

 1 Figure 1. Given a reference surface (a), our method discovers reliable data-model correspondences by random forests, color-coded in (c). Instead of tracking, this strategy detects user-specific shapes in a frame-wise manner, resulting in more sustainability.

Figure 2 .

 2 Figure 2. Pipeline of our framework. Correspondences are visualized in the same color. Black means no correspondence for that data point.

1 .

 1 finding correspondence pairs C = {(i, v)} between the vertex sets of Y and the vertex sets of M 2 , and 2. minimizing an energy E that describes the discrepancies between vertices in C: Θ = arg min Θ E(Θ; C).

Figure 3 .

 3 Figure 3. (a) original offset pair ψ. (b) η = 0 results in ψ without re-orientation, i.e. R = I. (c) η = 1. ψ is re-orientated by a rotation matrix R = [e1, e2, e3] characterized by a LCF.

Figure 4 .

 4 Figure 4. (a) The way we compute LCF. We search neighboring surface voxels to get a reference vector ( purple arrow); see text for details. (b) Our method leads to LCFs covariant to pose variations.

Figure 5 .

 5 Figure 5. (a) Traditional approach aggregates leaves of each vertex (green) independently, while we also take the 1-ring neighbors (black) into account. (b) Forest output Ỹ by aggregating each Gi separately. (c) Forest output Ỹ by aggregating Gi ∪ GN i .

Figure 6 .

 6 Figure 6. Matching error comparison of our method and Mesh-HOG of S2. We yields more locally consistent correspondences.

Figure 7 .

 7 Figure 7. (a) references surface of S2 (vertex indices encoded with colors). (b) test mesh. (c) matching results with MeshHOG. (d) matching results of the proposed VNF-forest framework.

Figure 8 .

 8 Figure 8. (a) the source surface and target points. (b) registration results (green) of EM-ICP [6] after 30 iterations. (c) results of VNF-forest framework (no iterations). (d) results of VNF-forest framework plus 14 iterations of EM-ICP.

Figure 9 .

 9 Figure 9. (a) results of frame 91 in bouncing. (b) visual hulls of frame 92 and correspondences from the forest in colors. (c) results of frame 92 from our approach. (d) results from surICP [6].

Figure 10 .

 10 Figure 10. Pixel overlap error of 4 sequences, averaged over all cameras. Image resolution: (a-b): 1920 × 1080. (c-d): 1000 × 1000

Figure 11 .Figure 12 .

 1112 Figure 11. Averaged joint errors. Image resolution: 1000 × 1000.

Table 1 .

 1 Sequences used for evaluation. We apply two different error measures, depending on the provided ground truths. A: silhouette overlap errors averaged over all views. B: distances to annotated joint positions in pixels.

	Our

3

stands for the β-coordinate proposed by Straka et al.

[START_REF] Straka | Simultaneous shape and pose adaption of articulated models using linear optimization[END_REF]

. Θ is encoded in T θ k ∈ SE(3).

of matches within max allowed distance Max allowed geodesic distance to ground truth

	75.47%
	48.39%
	VNF-forest
	MeshHOG-1NN

Table 2 .

 2 Pixel overlap error of 4 sequences over all frames and all cameras. Image resolution: 1000 × 1000.

		sil. error	sil. error	sil. error	sil. error	sil. error
	WalkChair1		7432		-		fail		8931	24976
	Cutting		4048		-		fail		20385	fail
		sil. error	joint error	sil. error	joint error	sil. error	joint error	sil. error	joint error	sil. error
	HammerTable	4019	13.3 ± 6.9	17285	64.2 ± 53.9	fail	fail	3593	10.1 ± 3.0	fail
	WalkChair2	6144	15.8 ± 6.1	12219	20.6 ± 22.0	18063	24.6 ± 10.7	6803	15.9 ± 6.3	18482

Both the reference model and input data are described as triangular meshes. Although the terms mesh and surface are used interchangeably in the text, we refer only to input data as visual hulls.

Whenever it is clear from the context, we will drop the time dependent variable t, in order to keep notations uncluttered.

Acknowledgment. This work is funded by Taiwan Ministry of Education Scholarship n°1001121US089, and partially funded by Technische Universität München -Department of Informatics -Chair for Computer Aided Medical Procedures & Augmented Reality.