Qianli Ma

Jinlong Yang

Anurag Ranjan

Sergi Pujades

Gerard Pons-Moll

Siyu Tang

Michael J Black
email: black@tue.mpg.desergi.pujades-rocamora@inria.frgpons@mpi-inf.mpg.desiyu.tang@inf.ethz.ch

Learning to Dress 3D People in Generative Clothing

Keywords:

Figure 1: CAPE model for clothed humans. Given a SMPL body shape and pose (a), CAPE adds clothing by randomly sampling from a learned model (b, c), can generate different clothing types -shorts in (b, c) vs. long-pants in (d). The generated clothed humans can generalize to diverse body shapes (e) and body poses (f). Best viewed zoomed-in on screen.

Introduction

Existing generative human models [6, [START_REF] Joo | Total Capture: A 3D deformation model for tracking faces, hands, and bodies[END_REF][START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF][START_REF] Pavlakos | Expressive body capture: 3D hands, face, and body from a single image[END_REF] successfully capture the statistics of human shape and pose deformation, but still miss an important component: clothing. This leads to several problems in various applications. For example, when body models are used to generate synthetic training data [START_REF] Hoffmann | Learning to train with synthetic humans[END_REF][START_REF] Ranjan | Learning multi-human optical flow[END_REF][START_REF] Ranjan | Learning human optical flow[END_REF][START_REF] Varol | Learning from synthetic humans[END_REF], the minimal body geometry results in a significant domain gap between synthetic and real images of humans. Deep learning methods reconstruct human shape from images, based on minimally dressed human models [START_REF] Rıza | Densepose: Dense human pose estimation in the wild[END_REF]23,[START_REF] Kolotouros | Learning to reconstruct 3D human pose and shape via model-fitting in the loop[END_REF][START_REF] Kolotouros | Convolutional mesh regression for single-image human shape reconstruction[END_REF][START_REF] Lassner | Unite the people: Closing the loop between 3D and 2D human representations[END_REF][START_REF] Omran | Neural body fitting: Unifying deep learning and model based human pose and shape estimation[END_REF][START_REF] Pavlakos | Expressive body capture: 3D hands, face, and body from a single image[END_REF][START_REF] Pavlakos | Learning to estimate 3D human pose and shape from a single color image[END_REF]. Although the body pose matches the image observation, the minimal body geometry does not match clothed humans in most cases. These problems motivate the need for a parametric clothed human model.

Our goal is to create a generative model of clothed human bodies that is low-dimensional, easy to pose, differentiable, can represent different clothing types on different body shapes and poses, and produces geometrically plausible results. To achieve this, we extend SMPL [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF] and factorize clothing shape from the undressed body, treating clothing as an additive displacement in the canonical pose (see Fig. 2). The learned clothing layer is compatible with the SMPL body model by design, enabling easy re-posing and animation. The mapping from a given body shape and pose to clothing shape is one-to-many. However, existing regression-based clothing models [16,[START_REF] Yang | Analyzing clothing layer deformation statistics of 3D human motions[END_REF] produce deterministic results that fail to capture the stochastic nature of clothing deformations. In contrast, we formulate clothing modeling as a probabilistic generation task: for a single pose and body shape, multiple clothing deformations can be sampled. Our model, called CAPE for "Clothed Auto Person Encoding", is conditioned on clothing types and body poses, so that it captures different types of clothing, and can generate pose-dependent deformations, which are important for realistically modeling clothing.

We illustrate the key elements of our model in Fig. 1. Given a SMPL body shape, pose and clothing type, CAPE can generate different structures of clothing by sampling a learned latent space. The resulting clothing layer plausibly adapts to different body shapes and poses.

Technical approach. We represent clothing as a displacement layer using a graph that inherits the topology of SMPL. Each node in this graph represents the 3dimensional offset vector from its corresponding vertex on the underlying body. To learn a generative model for such graphs, we build a graph convolutional neural network (Sec. 4), under the framework of a VAE-GAN [START_REF] Bao | CVAE-GAN: fine-grained image generation through asymmetric training[END_REF][START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF], using graph convolutions [11] and mesh sampling [START_REF] Ranjan | Generating 3D faces using convolutional mesh autoencoders[END_REF] as the backbone layers. This addresses the problem with existing generative models designed for 3D meshes of human bodies [START_REF] Litany | Deformable shape completion with graph convolutional autoencoders[END_REF][START_REF] Verma | Feastnet: Featuresteered graph convolutions for 3D shape analysis[END_REF] or faces [START_REF] Ranjan | Generating 3D faces using convolutional mesh autoencoders[END_REF] that tend to produce oversmoothed results; such smoothing is problematic for clothing where local details such as wrinkles matter. Specifically, the GAN [15] module in our system encourages visually plausible wrinkles. We model the GAN using a patch-wise discriminator for mesh-like graphs, and show that it effectively improves the quality of the generated fine structures.

Dataset. We introduce a dataset of 4D captured people performing a variety of pose sequences, in different types of clothing (Sec. 5). Our dataset consists of over 80K frames of 8 male and 3 female subjects captured using a 4D scanner. We use this dataset to train our network, resulting in a parametric generative model of the clothing layer.

Versatility. CAPE is designed to be "plug-and-play" for many applications that already use SMPL. Dressing SMPL with CAPE yields 3D meshes of people in clothing, which can be used for several applications such as generating training data, parametrizing body pose in a deep network, having a clothing "prior", or as part of a generative analysisby-synthesis approach [START_REF] Hoffmann | Learning to train with synthetic humans[END_REF][START_REF] Ranjan | Learning human optical flow[END_REF][START_REF] Varol | Learning from synthetic humans[END_REF]. We demonstrate this on the task of image fitting by extending SMPLify [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF] with our model. We show that using CAPE together with SMPLify can improve the quality of reconstructed human bodies in clothing.

In summary, our key contributions are: [START_REF] Alldieck | Learning to reconstruct people in clothing from a single RGB camera[END_REF] We propose a probabilistic formulation of clothing modeling. (2) Under this formulation, we learn a conditional Mesh-VAE-GAN that captures both global shape and local detail of a mesh, with controlled conditioning based on human pose and clothing types. (3) The learned model can generate pose-dependent deformations of clothing, and generalizes to a variety of garments. (4) We augment the SMPL 3D human body model with our clothing model, and show an application of the enhanced "clothed-SMPL". [START_REF] Rıza | Densepose: Dense human pose estimation in the wild[END_REF] We contribute a dataset of 4D scans of clothed humans performing a variety of motion sequences. Our dataset, code, and trained model are available for research purposes at https://cape.is.tue.mpg.de.

Related Work

The capture, reconstruction and modeling of clothing has been widely studied. Table 1 shows recent methods categorized into two major classes: (1) reconstruction and capture methods, and (2) parametric models, detailed as follows.

Reconstructing 3D humans. Reconstruction of 3D humans from 2D images and videos is a classical computer vision problem. Most approaches [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF][START_REF] Peng Guan | Estimating human shape and pose from a single image[END_REF]23,[START_REF] Kolotouros | Learning to reconstruct 3D human pose and shape via model-fitting in the loop[END_REF][START_REF] Kolotouros | Convolutional mesh regression for single-image human shape reconstruction[END_REF][START_REF] Lassner | Unite the people: Closing the loop between 3D and 2D human representations[END_REF][START_REF] Omran | Neural body fitting: Unifying deep learning and model based human pose and shape estimation[END_REF][START_REF] Pavlakos | Expressive body capture: 3D hands, face, and body from a single image[END_REF][START_REF] Smith | FACSIMILE: Fast and Accurate Scans From an Image in Less Than a Second[END_REF] output 3D body meshes from images, but not clothing. This ignores image evidence that may be useful. To reconstruct clothed bodies, methods use volumetric [START_REF] Natsume | SiCloPe: Silhouette-based clothed people[END_REF][START_REF] Saito | PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization[END_REF][START_REF] Varol | Bodynet: Volumetric inference of 3D human body shapes[END_REF][START_REF] Zheng | DeepHuman: 3D human reconstruction from a single image[END_REF] or bi-planar depth representations [START_REF] Gabeur | Moulding Humans: Non-parametric 3D Human Shape Estimation from Single Images[END_REF] to model the body and garments as a whole. We refer to these as Group 1 in Table 1. While these methods deal with arbitrary clothing topology and preserve a high level of detail, the reconstructed clothed body is not parametric, which means the pose, shape, and clothing of the reconstruction can not be controlled or animated.

Another group of methods are based on SMPL [START_REF] Alldieck | Learning to reconstruct people in clothing from a single RGB camera[END_REF][START_REF] Alldieck | Detailed human avatars from monocular video[END_REF]3,4,[START_REF] Lal Bhatnagar | Multi-Garment Net: Learning to Dress 3D People from Images[END_REF][START_REF] Hao Zhu | Detailed human shape estimation from a single image by hierarchical mesh deformation[END_REF]. They represent clothing as an offset layer from the underlying body as proposed in ClothCap [START_REF] Pons-Moll | Clothcap: Seamless 4D clothing capture and retargeting[END_REF]. We refer to these approaches as Group 2 in Table 1. These methods can change the pose and shape of the reconstruction using the deformation model of SMPL. This assumes clothing deforms like an undressed human body; i.e. that clothing shape and wrinkles do not change as a function of pose. We also use a body-to-cloth offset representation to learn Parametric models for 3D bodies and clothes. Statistical 3D human body models learned from 3D body scans, [6, [START_REF] Joo | Total Capture: A 3D deformation model for tracking faces, hands, and bodies[END_REF][START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF][START_REF] Pavlakos | Expressive body capture: 3D hands, face, and body from a single image[END_REF] capture body shape and pose and are an important building block for multiple applications. Most of the time, however, people are dressed and these models do not represent clothing. In addition, clothes deform as we move, producing changing wrinkles at multiple spatial scales. While clothing models learned from real data exist, few generalize to new poses. For example, Neophytou and Hilton [START_REF] Neophytou | A layered model of human body and garment deformation[END_REF] learn a layered garment model on top of SCAPE [6] from dynamic sequences, but generalization to novel poses is not demonstrated. Yang et al. [START_REF] Yang | Analyzing clothing layer deformation statistics of 3D human motions[END_REF] train a neural network to regress a PCA-based representation of clothing, but show generalization on the same sequence or on the same subject. Lähner et al. [START_REF] Lähner | DeepWrinkles: Accurate and realistic clothing modeling[END_REF] learn a garment-specific pose-deformation model by regressing low-frequency PCA components and high frequency normal maps. While the visual quality is good, the model is garment-specific and does not provide a solution for full-body clothing. Similarly, Alldieck et al. [4] use displacement maps with a UVparametrization to represent surface geometry, but the result is only static. Wang et al. [START_REF] Tuanfeng | Learning a shared shape space for multimodal garment design[END_REF] allow manipulation of clothing with sketches in a static pose. The Adam model [START_REF] Joo | Total Capture: A 3D deformation model for tracking faces, hands, and bodies[END_REF] can be considered clothed but the shape is very smooth and not pose-dependent. Clothing models have been learned from physics simulation of clothing [16,[START_REF] Gundogdu | GarNet: A two-stream network for fast and accurate 3D cloth draping[END_REF][START_REF] Patel | The Virtual Tailor: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style[END_REF][START_REF] Santesteban | Learning-Based Animation of Clothing for Virtual Try-On[END_REF], but visual fidelity is limited by the quality of the simulations. Furthermore, the above methods are regressors that produce single point estimates. In contrast, our model is generative, which allows us to sample clothing.

A conceptually different approach infers the parameters of a physical clothing model from 3D scan sequences [START_REF] Stoll | Video-based reconstruction of animatable human characters[END_REF]. This generalizes to novel poses, but the inference problem is difficult and, unlike our model, the resulting physics simulator is not differentiable with respect to the parameters.

Generative models on 3D meshes. Our model predicts clothing displacements on the graph defined by the SMPL mesh using graph convolutions [START_REF] Bruna | Spectral networks and locally connected networks on graphs[END_REF]. There is an extensive recent literature on methods and applications of graph convolutions [11,[START_REF] Kipf | Semi-supervised classification with graph convolutional networks[END_REF][START_REF] Litany | Deformable shape completion with graph convolutional autoencoders[END_REF][START_REF] Ranjan | Generating 3D faces using convolutional mesh autoencoders[END_REF][START_REF] Verma | Feastnet: Featuresteered graph convolutions for 3D shape analysis[END_REF]. Most relevant here, Ranjan et al. [START_REF] Ranjan | Generating 3D faces using convolutional mesh autoencoders[END_REF] learn a convolutional autoencoder using graph convolutions [11] with mesh down-and up-sampling layers [START_REF] Garland | Surface simplification using quadric error metrics[END_REF]. Although it works well for faces, the mesh sampling layer makes it difficult to capture the local details, which are key in clothing. In our work, we capture local details by extending the PatchGAN [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] architecture to 3D meshes.

Additive Clothed Human Model

To model clothed human bodies, we factorize them into two parts: the minimally-clothed body, and a clothing layer represented as displacements from the body. This enables us to naturally extend SMPL to a class of clothing types by treating clothing as an additional additive shape term. Since SMPL is in wide use, our goal is to extend it in a way that is consistent with current uses, making it effectively a "drop in" replacement for SMPL.

Dressing SMPL

SMPL [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF] is a generative model of human bodies that factors the surface of the body into shape (β) and pose (θ) parameters. As shown in Fig. 2 (a), (b), the architecture of SMPL starts with a triangulated template mesh, T , in rest pose, defined by N = 6890 vertices. Given shape and pose parameters (β, θ), 3D offsets are added to the template, corresponding to shape-dependent deformations (B S (β)) and pose dependent deformations (B P (θ)). The resulting mesh is then posed using the skinning function W . Formally: Let S clo (z, θ, c) be the clothing displacement layer. We extend Eq. ( 1) to a clothed body template in the rest pose:

T (β, θ) = T + B S (β) + B P (θ) (1) M (β, θ) = W (T (β, θ), J(β), θ, W) (2)
T clo (β, θ, c, z) = T (β, θ) + S clo (z, θ, c).
(3)

Note that the clothing displacements, S clo (z, θ, c), are posedependent. The final clothed template is then posed with the SMPL skinning function, Eq. ( 2):

M (β, θ, c, z) = W (T clo (β, θ, c, z), J(β), θ, W). ( 4
)
This differs from simply applying blend skinning with fixed displacements, as done in e.g. [START_REF] Alldieck | Learning to reconstruct people in clothing from a single RGB camera[END_REF][START_REF] Lal Bhatnagar | Multi-Garment Net: Learning to Dress 3D People from Images[END_REF]. Here, we train the model such that pose-dependent clothing displacements in the template pose are correct once posed by blend skinning.

Clothing representation

Vertex displacements are not a physical model for clothing and cannot represent all types of garments, but this approach achieves a balance between expressiveness and simplicity, and has been widely used in deformation modeling [16], 3D clothing capture [START_REF] Pons-Moll | Clothcap: Seamless 4D clothing capture and retargeting[END_REF] and recent work that reconstructs clothed humans from images [START_REF] Alldieck | Learning to reconstruct people in clothing from a single RGB camera[END_REF][START_REF] Lal Bhatnagar | Multi-Garment Net: Learning to Dress 3D People from Images[END_REF][START_REF] Hao Zhu | Detailed human shape estimation from a single image by hierarchical mesh deformation[END_REF].

The displacement layer is a graph

G d = (V d , E d ) that in- herits the SMPL topology: the edges E d = E SMPL . V d ∈ R 3×N
is the set of vertices, and the feature on each vertex is the 3-dimensional offset vector, (d x , d y , d z ), from its corresponding vertex on the underlying body mesh.

We train our model on 3D scans of people in clothing. From data pairs (V clothed , V minimal ) we compute displace-ments, where V clothed stands for the vertices of a clothed human mesh, and V minimal the vertices of a minimally-clothed mesh. Therefore, we first scan subjects in both clothed and minimally-clothed conditions, then use the SMPL model with free deformation [START_REF] Alldieck | Learning to reconstruct people in clothing from a single RGB camera[END_REF][START_REF] Zhang | Detailed, accurate, human shape estimation from clothed 3D scan sequences[END_REF] to register the scans. As a result, we obtain SMPL meshes capturing the geometry of the scans, the corresponding pose parameters, and vertices of the unposed meshes1 . For each (V clothed , V minimal ) pair, the displacements are then calculated as

V d = V clothed -V minimal ,
where the subtraction is performed per-vertex along the feature dimension. Ideally, V d has non-zero values only on body parts covered with clothes.

In summary, CAPE extends the SMPL body model to clothed bodies. Compared to volumetric representations of clothed people [START_REF] Natsume | SiCloPe: Silhouette-based clothed people[END_REF][START_REF] Saito | PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization[END_REF][START_REF] Varol | Bodynet: Volumetric inference of 3D human body shapes[END_REF][START_REF] Zheng | DeepHuman: 3D human reconstruction from a single image[END_REF], our combination of the body model and the garment layer is superior in the ease of reposing and garment retargeting: the former uses the same blend skinning as the body model, while the latter is a simple addition of the displacements to a minimally-clothed body shape. In contrast to similar models that also dress SMPL with offsets [START_REF] Alldieck | Learning to reconstruct people in clothing from a single RGB camera[END_REF][START_REF] Lal Bhatnagar | Multi-Garment Net: Learning to Dress 3D People from Images[END_REF], our garment layer is parametrized, low-dimensional, and pose-dependent.

CAPE

Our clothing term S clo (z, θ, c) in Eq. ( 3) is a function of z, a code in a learned low-dimensional latent space that encodes the shape and structure of clothing, body pose θ, and clothing type c. The function outputs the clothing displacement graph G d as described in Sec. 3.2. We parametrize this function using a graph convolutional neural network (Graph-CNN) as a VAE-GAN framework [15, 24,[START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF].

Network architecture

As shown in Fig. 3, our model consists of a generator G with an encoder-decoder architecture and a discriminator D. We also use auxiliary networks C 1 , C 2 to handle the conditioning. The network is differentiable and is trained end-to-end. For simplicity, we use the following notation in this section. x: the vertices V d of the input displacement graph; x: vertices of the reconstructed graph; θ and c: the pose and clothing type condition vector; z: the latent code.

Graph generator. We build the graph generator following a VAE-GAN framework. During training, an encoder Enc(•) takes in the displacement x, extract its features through multiple graph convolutional layers, and maps it to the lowdimensional latent code z. A decoder is trained to reconstruct the input graph x = Dec(z) from z. Both the encoder and decoder are feed-forward neural networks built with mesh convolutional layers. Linear layers are used at the end of the encoder and the beginning of the decoder. The architecture is shown in the supplemental materials.

Stacking graph convolution layers causes a loss of local features [START_REF] Li | Deeper insights into graph convolutional networks for semi-supervised learning[END_REF] in the deeper layers. This is undesirable for clothing generation because fine details, corresponding to wrinkles, are likely to disappear. Therefore, we improve the standard graph convolution layers with residual connections, which enable the use of low-level features from the layer input if necessary.

At test time, the encoder is not needed. Instead, z is sampled from the Gaussian prior distribution, and the decoder serves as the graph generator: G(z) = Dec(z). We detail different use cases below.

Patchwise discriminator. To further enhance fine details in the reconstructions, we introduce a patchwise discriminator D for graphs, which has shown success in the image domain [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF][START_REF] Zhu | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[END_REF]. Instead of looking at the entire generated graph, the discriminator only classifies whether a graph patch is real or fake based on its local structure. Intuitively this encourages the discriminator to only focus on fine details, and the global shape is taken care of by the reconstruction loss. We implement the graph patchwisediscriminator using four graph convolution-downsampling blocks [START_REF] Ranjan | Generating 3D faces using convolutional mesh autoencoders[END_REF]. We add a discriminative real / fake loss for each of the output vertices. This enables the discriminator to capture a patch of neighboring nodes in the reconstructed graph and classify them as real / fake (see Fig. 3).

Conditional model. We condition the network with body pose θ and clothing type c. The SMPL pose parameters are in axis-angle representation, and are difficult for the neural network to learn [START_REF] Kolotouros | Convolutional mesh regression for single-image human shape reconstruction[END_REF][START_REF] Lassner | Unite the people: Closing the loop between 3D and 2D human representations[END_REF]. Therefore, following previous work [START_REF] Kolotouros | Convolutional mesh regression for single-image human shape reconstruction[END_REF][START_REF] Lassner | Unite the people: Closing the loop between 3D and 2D human representations[END_REF], we transform the pose parameters into rotational matrices using the Rodrigues equation. The clothing types are discrete by nature, and we represent them using one-hot labels. Both conditions are first passed through a small fully-connected embedding network, C 1 (θ), C 2 (c), respectively, so as to balance the dimensionality of learned graph features and of the condition features. We also experiment with different ways of conditioning the mesh generator: concatenation in the latent space; appending the condition features to the graph features at all nodes in the generator; and the combination of the two. We find that the combined strategy works better in terms of network capability and the effect of conditioning.

Losses and learning

For reconstruction, we use an L1 loss over the vertices of the mesh x, because it encourages less smoothing compared to L2, given by

L recon = E x∼p(x),z∼q(z|x) [∥G(z, θ, c) -x∥ 1 ] . (5)
Furthermore, we apply a loss on the mesh edges to encourage the generation of wrinkles instead of smooth surfaces. Let e be an edge in the set of edges, E, of the ground truth graph, and ê the corresponding edge in the generated graph. We penalize the mismatch of all corresponding edges by

L edge = E e∈E, ê∈ Ê [∥e -ê∥ 2 ] . ( 6
)
We also apply a KL divergence loss between the distribution of latent codes and the Gaussian prior

L KL = E x∼p(x) [KL(q(z|x) ∥ N (0, I))] . ( 7
)
Moreover, the generator and the discriminator are trained using an adversarial loss

L GAN = E x∼p(x) [log(D(x, θ, c))] + E z∼q(z|x) [log(1 -D(G(z, θ, c)))] , (8)
where G tries to minimize this loss against the D that aims to maximize it. The overall objective is a weighted sum of these loss terms given by

L = L recon + γ edge L edge + γ kl L KL + γ gan L GAN . (9)
Training details are provided in the supplemental materials.

CAPE Dataset

We build a dataset of 3D clothing by capturing temporal sequences of 3D human body scans with a high-resolution body scanner (3dMD LLC, Atlanta, GA). Approximately 80K 3D scan frames are captured at 60 FPS, and a mesh with SMPL model topology is registered to each scan to obtain surface correspondences. We also scanned the subjects in a minimally-clothed condition to obtain an accurate estimate of their body shape under clothing. We extract the clothing as displacements from the minimally-clothed body as described in Sec. 3.2. Noisy frames and failed registrations are removed through manual inspection.

The dataset consists of 8 male subjects and 3 female subjects, performing a wide range of motions. The subjects gave informed written consent to participate and to release the data for research purposes. "Clothing type" in this work refers to the 4 types of full-body outfits, namely shortlong: short-sleeve upper body clothing and long lower body clothing; and similarly shortshort, longshort, longlong. These outfits comprise 8 types of common garments. We refer to the supplementary material for the list of garments, further details, and examples from the dataset.

Compared to existing datasets of 3D clothed humans, our dataset provides captured data and alignments of SMPL to the scans, separates the clothing from body, and provides accurate, captured ground truth body shape under clothing. For each subject and outfit, our dataset contains large pose variations, which induces a wide variety of wrinkle patterns. Since our dataset of 3D meshes has a consistent topology, it can be used for the quantitative evaluation of different Graph-CNN architectures. The dataset is available for research purposes at https://cape.is.tue.mpg.de.

Experiments

We first show the representation capability of our model and then demonstrate the model's ability to generate new examples by probabilistic sampling. We then show an application to human pose and shape estimation.

Representation power

3D mesh auto-encoding errors. We use the reconstruction accuracy to measure the capability of our VAE-based model for geometry encoding and preserving. We compare with a recent convolutional mesh autoencoder, CoMA [START_REF] Ranjan | Generating 3D faces using convolutional mesh autoencoders[END_REF], and a linear (PCA) model. We compare to both the original CoMA with a 4× downsampling (denoted as "CoMA-4"), and without downsampling (denoted "CoMA-1") to study the effect of downsampling on over-smoothing. We use the same latent space dimension n z = 18 (number of principal components in the case of PCA) and hyper-parameter settings, where applicable, for all models.

Table 2 shows the per-vertex Euclidean error when using our network to reconstruct the clothing displacement graphs from a held-out test set in our CAPE dataset. The model is trained and evaluated on male and female data separately. Body parts such as head, fingers, toes, hands and feet are excluded from the accuracy computation, as they are not covered with clothing.

Our model outperforms the baselines in the autoencoding task; additionally, the reconstructed shape from our model is probabilistic and pose-dependent. Note that, CoMA here is a deterministic auto-encoder with a focus on reconstruction. Although the reconstruction performance of PCA is on par with our method on male data, PCA can not be used directly in the inference phase with a pose parameter as input. Furthermore, PCA assumes a Gaussian distribution of the data, which does not hold for complex clothing deformations. Our method addresses both of these issues. Fig. 4 shows a qualitative comparison of the methods. PCA keeps wrinkles and boundaries, but the rising hem on the left side disappears. CoMA-1 and CoMA-4 are able to capture global correlation, but the wrinkles tend to be smoothed. By incorporating all the key components, our model manages to model both local structures and global correlations more accurately than the other methods.

Ablation study. We remove key components from our model while keeping all the others, and evaluate the model performance; see Table 2. We observe that the discriminator, residual block and edge loss all play important roles in the model performance. Comparing the performance of CoMA-4 and CoMA-1, we find that the mesh the downsampling layer causes a loss of fidelity. However, even without any spatial downsampling, CoMA-1 still underperforms our model. This shows the benefits of adding the discriminator, residual block, and edge loss in our model.

Conditional generation of clothing

As a generative model, CAPE can be sampled and generates new data. The model has three parameters: z, c, θ (see Eq. ( 3)). By sampling one of them while keeping the other two fixed, we show how the conditioning affects the generated clothing shape.

Sampling. Fig. 5 presents the sampled clothing dressed on unseen bodies, in a variety of poses that are not used in training. For each subject, we fix the pose θ and clothing type c, and sample z several times to generate varied clothing shapes. The sampling trick in [START_REF] Ghosh | From variational to deterministic autoencoders[END_REF] is used. Here we only show untextured rendering to highlight the variation in the generated geometry. As CAPE inherits the SMPL topology, the generated clothed body meshes are compatible with all existing SMPL texture maps. See supplemental materials for a comparison between a CAPE sample and a SMPL sample rendered with the same texture.

As shown in the figure, our model manages to capture long-range correlations within a mesh, such as the elevated hem for a subject with raised arms, and the lateral wrinkle on the back with raised arms. The model also synthesizes local details such as wrinkles in the armpit area, and boundaries at cuffs and collars.

Pose-dependent clothing deformation. Another practical use case of CAPE is to animate an existing clothed body. This corresponds to fixing the clothing shape variable z and clothing type c, and reposing the body by changing θ. The challenge here is to have a clothing shape that is consistent across poses, yet deforms plausibly. We demonstrate the pose-dependent effect on a test pose in Fig. 6. The difference of the clothing layer between the two poses is calculated in the canonical pose, and shown with color coding. The result shows that the clothing type is consistent while local deformation changes along with pose. We refer to the supplemental video for a comparison with traditional rigand-skinning methods that use fixed clothing offsets.

User study of generated examples. To test the realism of the generated results from our method, we performed a user study on Amazon Mechanical Turk (AMT). We dress virtual avatars in 3D and render them into front-view images. Following the protocol from [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], raters are presented with a series of "real vs fake" trials. On each trial, the rater is presented with a "real" mesh render (randomly picked from our dataset), and a "fake" render (generated by our model), shown side-by-side. The raters are asked to pick the one that they think is real. Each pair of renderings is evaluated by 10 raters. More strictly than [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF], we present both real and fake renderings simultaneously, do not set a time limit for raters and allow zoom-in for detailed comparison. In this setting, the best score that a method can obtain is 50%, meaning that the real and fake examples are indistinguishable.

We evaluate with two test cases. In test case 1, we fix the clothing type to be "shortlong" (the most common clothing type in training), and generate 300 clothed body meshes with various poses for the evaluation. In test case 2, we fix the pose to be an A-pose (the most frequent pose in training), and sample 100 examples per clothing type for evaluation. On average, in the direct comparison with real data, our synthesized data "fools" participants 35.1% ± 15.7% and 38.7% ± 16.5% of the time respectively (i.e. these participants labeled our result as "real").

Image fitting

CAPE is fully differentiable with respect to the clothing shape variable z, body pose θ and clothing type c. Therefore, it can also be used in optimization frameworks. We show an application of CAPE on the task of reconstructing a body mesh from a single image, by entending the optimization-based method, SMPLify [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF]. Assuming c is known, we dress the minimally-clothed output mesh from SMPLify using CAPE, project it back to the image using a differentiable renderer [START_REF] Henderson | Learning single-image 3D reconstruction by generative modelling of shape, pose and shading[END_REF] and optimize for β, θ, z, with respect to the silhouette discrepancy.

We evaluate our image fitting pipeline on renderings of 120 randomly selected unseen test examples from the CAPE dataset. To compare, we measure the reconstruction error of SMPLify and our results against ground truth meshes using mean square vertex error (MSE). To eliminate the error introduced by the ambiguity of human scale and distance to the camera, we optimize the global scaling and translation of predictions for both methods on each test sample. A mask is applied to exclude error in the non-clothed regions such as head, hands and feet. We report the errors of both methods in Table 3. Our model performs 18% better than SMPLify due to its ability to capture clothing shape.

More details about the objective function, experimental setup and qualitative results of the image fitting experiment

Conclusions, Limitations, Future Work

We have introduced a novel graph-CNN-based generative shape model that enables us to condition, sample, and preserve fine shape detail in 3D meshes. We use this to model clothing deformations from a 3D body mesh and condition the latent space on body pose and clothing type. The training data represents 3D displacements from the SMPL body model for varied clothing and poses. This design means that our generative model is compatible with SMPL in that clothing is an additional additive term applied to the SMPL template mesh. This makes it possible to sample clothing, dress SMPL with it, and then animate the body with pose-dependent clothing wrinkles. A clothed version of SMPL has wide applicability in computer vision. As shown, we can apply it to fitting the body to images of clothed humans. Another application would use the model to generate training data of 3D clothed people to train regression-based pose-estimation methods.

There are a few limitations of our approach that point to future work. First, CAPE inherits the limitation of the offset representation for clothing: (1) Garments such as skirts and open jackets differ from the body topology and cannot be trivially represented by offsets. Consequently, when fitting CAPE to images containing such garments, it could fail to explain the image evidence; see discussions on the skirt example in the supplementary material. (2) Mittens and shoes: they can technically be modeled by the offsets, but their geometry is sufficiently different from fingers and toes, making this impractical. A multi-layer model can potentially overcome these limitations. Second, the level of geometric details that CAPE can achieve is upper-bounded by the mesh resolution of SMPL. To produce finer wrinkles, one can resort to higher resolution meshes or bump maps. Third, while our generated clothing depends on pose, it does not depend on dynamics. This does not cause severe problem for most slow motions but does not generalize to faster motions. Future work will address modeling clothing deformation on temporal sequence and dynamics.

Figure 2 :

 2 Figure 2: Additive clothed human model. Our contribution is highlighted with yellow background. Following SMPL, our model (a) starts from a template mesh, and linearly adds offsets contributed by (b) individual body shape β, and (c) pose θ; note the deformation on the hips and feet caused by the ballet pose. (d) We further add a clothing layer parametrized by pose θ, clothing type c and a clothing shape variable z. (e) The vertices are then posed using the skinning equation of SMPL. where the blend skinning function W (•) rotates the rest pose vertices T around the 3D joints J (computed from β), linearly smoothes them with the blend weights W, and returns the posed vertices M . The pose θ ∈ R 3×(23+1) is represented by a vector of relative 3D rotations of the 23 joints and the global rotation in axis-angle representation.SMPL adds linear deformation layers to an initial body shape. Following this, we define clothing as an extra offset layer from the body and add it on top of the SMPL mesh, Fig.2 (d). In this work, we parametrize the clothing layer by the body pose θ, clothing type c and a low-dimensional latent variable z that encodes clothing shape and structure.Let S clo (z, θ, c) be the clothing displacement layer. We extend Eq. (1) to a clothed body template in the rest pose:

Figure 3 :

 3 Figure 3: Overview of our method. (a) Computing displacements from scan data (Sec. 3.2) by subtracting the minimal body shape from clothed body mesh in the unposed space. (b) Schematic of our network architecture.

Figure 4 :

 4 Figure 4: Example of reconstruction by the baselines, ablated version of our model, and our full model. Pay attention to the hem and wrinkles on upper back. Our model is able to recover both long-range correlations and local details. Best viewed zoomed-in on screen.

Figure 5 :Figure 6 :

 56 Figure 5: Clothing sampled and generated from our CAPE model applied to four unseen body shapes (green colored) with different poses. Each body in green is followed by five blue examples (generated by sampling the latent vector z) showing different versions of the same outfit type.The four groups are wearing outfit types "longshort", "longlong", "shortlong" and "shortshort", respectively. Best viewed zoomed-in on screen.

Table 2 :

 2 Per-vertex auto-encoding error in millimeters. Upper section: comparison with baselines; lower section: ablation study.

		Male		Female
	Methods	Error mean	median	Error mean	median
	PCA	5.65 ± 4.81	4.30	4.82 ± 3.82	3.78
	CoMA-1	6.23 ± 5.45	4.66	4.69 ± 3.85	3.61
	CoMA-4	6.87 ± 5.62	5.29	4.86 ± 3.96	3.75
	Ours	5.54 ± 5.09	4.03	4.21 ± 3.76	3.08
	Ablated Components	Error mean	median	Error mean	median
	Discriminator	5.65 ± 5.18	4.11	4.31 ± 3.78	3.18
	Res-block	5.60 ± 5.21	4.05	4.27 ± 3.76	3.15
	Edge loss	5.93 ± 5.40	4.32	4.32 ± 3.78	3.19

Table 3 :

 3 Vertex MSE of image fitting results measured in meters.

	Method	SMPLify [9]	Ours
	Per-vertex MSE	0.0223	0.0189

We follow SMPL and use the T-pose as the zero-pose. For the mathematical details of registration and unposing, we refer the reader to[START_REF] Zhang | Detailed, accurate, human shape estimation from clothed 3D scan sequences[END_REF].

Acknowledgements: We thank Daniel Scharstein for revisions of the manuscript, Joachim Tesch for the help with Blender rendering, Vassilis Choutas for the help with image fitting experiments, and Pavel Karasik for the help with AMT evaluation. We thank Tsvetelina Alexiadis and Andrea Keller for collecting data. We thank Partha Ghosh, Timo Bolkart and Yan Zhang for useful discussions. Q. Ma and S. Tang acknowledge funding by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) -276693517 SFB 1233. G. Pons-Moll is funded by the Emmy Noether Programme, Deutsche Forschungsgemeinschaft -409792180.

Disclosure: MJB has received research gift funds from Intel, Nvidia, Adobe, Facebook, and Amazon. While MJB is a part-time employee of Amazon, his research was performed solely at, and funded solely by, MPI.

* Work was done when S. Tang was at MPI-IS and


Rajat Prince Thakur

Sergi Pujades Rocamora

Lavika Goel

Rolf Pohmann

Jürgen Machann

Michael J Black

GENTEL : GENerating Training data Efficiently for Learning to segment medical images

Keywords: Segmentation IRM, méthode semi-automatique, données limitées MRI Segmentation, Semi-automatic method, scarce data

Accurately segmenting MRI images is crucial for many clinical applications. However, manually segmenting images with accurate pixel precision is a tedious and time consuming task. In this paper we present a simple, yet effective method to improve the efficiency of the image segmentation process. We propose to transform the image annotation task into a binary choice task. We start by using classical image processing algorithms with different parameter values to generate multiple, different segmentation masks for each input MRI image. Then, the user, instead of segmenting the pixels of the images, she/he only needs to decide if a segmentation is acceptable or not. This method allows us to efficiently obtain high quality segmentations with minor human intervention. With the selected segmentations we train a state-of-the-art neural network model. For the evaluation, we use a second MRI dataset (1.5T Dataset), acquired with a different protocol and containing annotations. We show that the trained network i) is capable to automatically segment cases where none of the classical methods obtained a high quality result ii) generalizes to the second MRI dataset, which was acquired with a different protocol and never seen at training time ; and iii) allows to detect missannotations in this second dataset. Quantitatively, the trained network obtains very good results : DICE score -mean 0.98, median 0.99-and Hausdorff distance (in pixels) -mean 4.7, median 2.0-.

Résumé

La segmentation précise d'images à résonnance magnétiques (IRM) est cruciale pour de nombreuses applications cliniques. Cependant, une segmentation manuelle visant une précision au niveau du pixel est une tâche longue et fastidieuse. Dans cet article, nous proposons une méthode simple pour améliorer l'efficacité de la segmentation d'images. Nous proposons de transformer la tâche d'annotation d'une image en une tâche de choix binaire. D'abord, nous utilisons plusieurs algorithmes classiques de traitement d'image pour générer plusieurs candidats de masques de segmentation. Ensuite, l'utilisat.eur.rice, au lieu de segmenter les pixels des images, décide si une segmentation est acceptable ou non. Cette méthode nous permet d'obtenir efficacement un grand nombre de segmentations de haute qualité avec une intervention humaine limitée. Avec les images et leurs segmentations sélectionnées, nous entrainons un réseau de neurones de l'état de l'art qui prédit les segmentations à partir des images d'entrée. Nous le validons sur un autre jeu de données IRM, acquis avec un protocole different, et qui contient des segmentations. Nous montrons que le réseau entrainé 1) est capable de segmenter automatiquement des cas où aucune des méthodes classiques n'a obtenu un résultat de haute qualité, 2) est capable de segmenter un autre jeu de données IRM, acquis avec un protocole different et jamais vu lors de l'entrainement, et 3) permet de détecter des annotations erronées dans ce jeu de données. Quantitativement, le réseau entrainé obtient de très bons résultats : Score DICE -moyenne 0,98 et médiane 0,99 -et distance de Hausdorff (en pixels) -moyenne 4,7, médiane 2,0.

Introduction

Accurately segmenting clinical images is crucial for many clinical applications, as it allows to focus on a specific part of the human body, such as bone, organs, or different tissues in the human body. However, accurately segmenting clinical images is very challenging and the problem has attracted many research efforts. Segmentation methods can mainly be clustered into automatic and semi-automatic methods, the latter requiring some human intervention in the segmentation process. In the automatic methods, we can also find two main categories, the ones based on empiric mathematical computations or models and the ones based on machine learning [START_REF] Bezdek | Review of MR image segmentation techniques using pattern recognition[END_REF] [START_REF] Sivakumar | A review on image segmentation techniques[END_REF]. Examples of the first ones are algorithms based on image histograms [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF] or image features, such as contours [START_REF] Kass | Snakes : Active contour models[END_REF]. These typically include some post-processing to refine the obtained results. In the second ones, starting with a cohort of segmentation examples, an algorithm is trained to generalize to new unseen data. Many machine learning algorithms exist [START_REF] Bishop | Pattern recognition and machine learning[END_REF], and in the recent years, neural networks [START_REF] Lecun | Deep learning[END_REF] have been the ones obtaining the best performance. As neural networks have a large number of parameters that need to be solved for, one key component for them to achieve a good performance is the amount of available training data. This has guided researches addressing computer vision tasks, such as object recognition, to create large annotated datasets [START_REF] Deng | Imagenet : A large-scale hierarchical image database[END_REF] [7] [START_REF] Kuznetsova | The Open Images Dataset V4 : Unified image classification, object detection, and visual relationship detection at scale[END_REF], which have unveiled the potential of neural networks. However, in some domains, such as the medical domain, the amount of available data for a given task may be limited. Although MICCAI Challenges, among others, are contributing to the creation of better and richer cohorts of segmented images, these cohorts still remain relatively small. For example, the Infant Brain MRI Segmentation [START_REF]Infant Brain MRI Segmentation Challenge[END_REF] has 10 subjects, the Large Scale Vertebrae Segmentation Challenge [START_REF]Large Scale Vertebrae Segmentation Challenge[END_REF] has 120 subjects, the Automatic Structure Segmentation for Radiotherapy Planning Challenge [START_REF]Automatic Structure Segmentation for Radiotherapy Planning Challenge[END_REF] provides 50 cases, the Liver Cancer Segmentation Challenge [START_REF]Liver Cancer Segmentation[END_REF] has 50 cases and the Kidney Tumor Segmentation Challenge [START_REF]Kidney Tumor Segmentation Challenge[END_REF] provides 210 subjects. This relatively low number of cases is mainly due to two facts. First, the amount of raw data may be limited, as the access to clinical subjects presents several constraints. Second, the annotation of the raw data often presents an important challenge. In this paper we focus on how to efficiently obtain accurate segmentations from raw data. Two key insights drive our work. The first is that rater time is scarce and expensive. While this might not apply for general computer vision problems, as for instance, anybody could segment a cat on an RGB image, segmenting clinical images requires medical skills. However, clinical personnel are heavily demanded and their time is precious. Our second key insight is that for a rater, it is fairly easy to distinguish if a segmentation is good or bad. Thus the binary task consisting of deciding if a segmentation is acceptable or not becomes very fast. Based on these insights, we present a simple, yet effective method to improve the efficiency of the image segmentation process, by transforming the pixel annotation task into a binary choice task. We start by using classical image processing algorithms with several different parameter values to generate mul-tiple, different segmentation masks for each input image. Then, the rater, instead of segmenting the pixels of the images, she/he only needs to decide if a segmentation is acceptable or not. With this methodology, we annotate a fairly large dataset with minimal rater intervention. To evaluate our method, we train a neural network with the annotated data and illustrate the results, both qualitatively and quantitatively. Moreover, we show that the trained network i) is capable to automatically segment cases where none of the classical methods obtained a high quality result ii) generalizes to a second MRI dataset, acquired with a different protocol, never seen at training time and iii) allows to detect missannotations in this second dataset. The trained network is made available at http: //gentel.is.tue.mpg.de/. The proposed methodology is generic. We demonstrate it by addressing the specific problem of body segmentation in MRI images. We focus on MRI images as, in contrast to CT-Scans, in which standardized Hounsfield units are used, the absolute values in MRI images do not correspond to specific tissues. This makes their segmentation particularly challenging.

Task and Datasets

Segmentation task. In this paper we are interested in the detection of the human body, i.e. which parts belong to the human and which don't, in 2D MRI images. Our input images are 2D MRI images and the output is a binary image with pixels corresponding to the body being one, and the rest zero. In this work we used two datasets, one acquired with a 1.5T scanner and another with a 3T scanner. We name them 1.5T and 3T datasets in the rest of the paper.

1.5T Dataset -6 subjects. The 1.5T dataset was acquired with a 1.5 T scanner (Magnetom Sonata, Siemens Healthcare). Subjects were lying in a prone position with extended arms reaching up. From feet to hands, axial T1-weighted fast-spin-echo images were recorded. The measurement parameters were : 256x192 matrix, 10mm-thick sections, 10-mm gap between sections, adjusted field of view depending on the extension of the subject (450-530mm) and five sections per acquisition. Each MRI consists of around 110 slices, which varies slightly depending upon the height of the subject. Subjects gave prior written consent. The 1.5T dataset was segmented using an automatic method [START_REF] Würslin | Topography mapping of whole body adipose tissue using A fully automated and standardized procedure[END_REF] which was specifically designed for data acquired with the used protocol.

3T Dataset -33 subjects. The 3T dataset was captured with a 3.0T scanner (PRISMA Fit, Siemens Healthcare) with a bore diameter of 60 cm. The subjects were in supine position with arms lying on the chest. Axial T1-weighted turbo-spin-echo images were recorded from feet to head applying the following measurement parameters : 512x352 matrix, 10-mm-thick sections, 10-mm gap between sections, adjusted field of view depending on the extension of the subject. Five sections per acquisition were acquired within 27s, which is short enough to allow the subject to hold The automatic segmentation method [START_REF] Würslin | Topography mapping of whole body adipose tissue using A fully automated and standardized procedure[END_REF] completely failed on the 3T dataset, mainly due to two reasons : i) the method is not capable to deal with the arms being near the torso as it was designed to handle subjects with the arms reaching up ii) the MRI sequences had a different contrast (due to the TR parameters and the different machines 1.5T vs 3T). Thus, the 3T dataset has no segmentations.

Method

To avoid the cumbersome task of manual pixel annotation, we proceed in a two-step methodology to annotate the 3T MRI dataset. First we automatically generate candidate segmentations for each dataset image. Then the candidate segmentations are presented to a rater to decide if the segmentation is acceptable or not.

Creating segmentation candidates

To segment the body in the MRI images, we used several basic image processing operations to obtain candidate segmentation masks. We developed different methods, in order to handle different cases in the images such as a different number of body parts in the images. After the development of one method, it was run on the 3T dataset, and obvious error cases were used as guidance to create a new method and introduce parameter variants. In this manner we created 3 methods with variants, obtaining 18 candidate segmentations for each input image.

Method 1. We started by blurring the input image with a 5x5 Gaussian kernel, and performed Otsu's binarization [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF]. We tried out other binarization methods also but, Otsu's method worked the best for the 3T dataset images. To fill in the remaining holes in the obtained binary image, we applied a floodfilling algorithm [START_REF]Flood fill algorithm[END_REF] starting from the top left corner of the image. We then extracted the contours from the floodfilled image. We discarded contours with a length shorter than 20 pixels and retained the largest 3 contours. We only consider the 3 largest contours as we don't expect to see any more contours in MRIs of subjects in the supine position. Smaller contours would basically correspond to fingers, that we do not consider, or noise. The remaining contours were convexified using a convex hull. The C largest contours were retained. Method 2. In this case we did not blur the image and performed Otsu's binarization [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF] directly on the input image followed by the flood-filling algorithm [START_REF]Flood fill algorithm[END_REF]. We extracted the connected components and removed the ones with an area smaller than 500 squared pixels. From these, we retained the 3 having the largest area. We then performed morphological closing on the remaining contours (using a circular structuring element of radius 10) and applied the flood-filling algorithm [START_REF]Flood fill algorithm[END_REF]. We then extracted the contours from the flood-filled image. We discarded contours with a length shorter than 20 pixels and retained the largest 3 contours as in Method 1. We did not convexify the contours and retained the C largest contours. Method 3. We started by blurring the input image with a 5x5 Gaussian kernel, and performed Otsu's binarization [START_REF] Otsu | A threshold selection method from gray-level histograms[END_REF]. To fill up the remaining holes, we applied a floodfilling operation [START_REF]Flood fill algorithm[END_REF]. We called this image img init . We extracted the connected components and kept all of them.

After this we performed one IOP operation, which could be IOP = 1 or IOP = 2. In IOP = 1, we performed morphological closing with a circular structuring element of radius 18. In IOP = 2, we performed morphological closing with a circular structuring element of radius 21, and retained only the components having more than half the number of pixels before the morphological closing. We called this image img a . After this, we performed morphological closing on the image max(img init -img a , 0) using the same kernel as above obtaining img b . We then took the union of img a and img b and flood-filled the union image to obtain the final segmentation.

After the operation IOP we extracted the contours from the resulting image. The morphological operation did remove some of the minor details. However, this also provided good contours in noisy image regions where other methods created noisy contours. Performing morphological closing gave an alternative with overall smoother contours.

We discarded contours with a length shorter than 20 pixels and retained the largest 3 contours as in Method 1 and 2. In one variant the remaining contours were convexified using a convex hull and in the other the contours were left untouched. The C largest contours were retained. Generated Candidates. After this step, we had a total of 18 different algorithms generating potentially 18 different segmentation masks. Fig. 1 shows examples illustrating cases where one particular method worked or failed. In practice we observed that in some cases, several variants of the same method, or even different methods, produced the exact same result. For example, if the algorithm only detected one contour in the image, the variants retaining 2 and 3 contours obtained exactly the same result. However, given an MRI slice, we did not know which of the 18 algorithms would produce the most accurate segmentation. Thus we asked an expert rater to select acceptable segmentations.

Choosing acceptable segmentations

Although we had many candidate segmentations for each image, we did not ask the rater to identify the "best segmentation mask". Instead, we built a simple tool (see Sec 3.3), which would randomly select one of the candidate segmentation and display the contour(s) superimposed on the MRI image. The rater was then asked to either discard or keep the candidate segmentation, thus transforming the annotation task into a binary decision task. The rater was not able to refine the segmentation and was invited to discard it if errors were visible. We had 18 candidate segmentations for each MRI image, which might seem a lot, but, for most of the cases several of the 18 possibilities were clear mismatches and were quickly identified. The rating process took around 12h, which accounts to an average of 5 seconds per image. Certain regions of the body required more careful examinations. Af- ter the process, 7146 segmentations were retained. Some input images had multiple segmentations, with slight subtle differences mostly located at the boundary of the segmentation. Also some input images had no retained segmentation, as none of the methods provided a correct segmentation.

Annotation Tool

We used a simple command line tool to annotate the 3T dataset. We used the MRI images and the corresponding segmentation masks obtained with the image processing based algorithms. The tool picks up a segmentation mask and generates contours from it. It then displays the contours superimposed on the MRI images as shown in Fig. 2. The user is then asked if the contours describe an accurate segmentation. The segmentation and the corresponding MRI image are retained if the user presses 'y'. They are discarded in all other cases. This is done for all of the segmentation masks one after the other.

Experiments

Training a segmentation network. With the annotated 3T dataset, we split it into train, test and validation sets to train our neural network based segmentation model. The annotated 3T dataset contains 5154 images in the training set, 847 in the validation set and 1145 in the test set. The 1.5T dataset contains 601 images, that were solely used as a second test set. We preprocess all images so that the min and max values in the MRI image are mapped to 0 and 255, as some of the MRI images in the dataset had low contrast. We used the pix2pix [START_REF] Isola | Imageto-image translation with conditional adversarial networks[END_REF] network, consisting of a generator and a discriminator. The generator is a convolutional autoencoder and the discriminator is a convolutional encoder followed by a fully connected network. We trained the network for 200 epochs using the AdamOptimizer [START_REF] Kingma | Adam : A method for stochastic optimization[END_REF].

Quantitative Results. To evaluate our method, we used the segmentations of the 1.5T Dataset as ground truth and computed DICE and Hausdorff distance (HD) metrics for the trained network and the automatic methods presented in Sec. 3. Let us point out that for 14 images some of our methods detected a contour and the reference had none thus creating an undefined HD value. We observed that these errors were consistent across the methods so we visually inspected these 14 images. Two cases arose : 10 very noisy images and 4 errors in the ground truth annotations (see Fig. 6 first four rows). We decided to exclude these 14 images from the 1.5T dataset only for numerical evaluation. We used the remaining 601 images for evaluation. Table 1 shows the obtained results. The network outper-forms all methods in terms of Hausdorff distance. Only the automatic method 2, variant 3 is marginally better (+0.004) in terms of median DICE scores.

Qualitative Results. We observed that the algorithm M2(3) had comparable, even slightly better, quantitative DICE results compared to the network. However, when visually inspecting the worse performing cases for each method, we observed that the networks' errors were qualitatively more acceptable in terms of body segmentation. Fig. 3 shows two representative cases. In one case the network segments what seems to be fingers in the bottom of the image and thus the HD is very high (157.9). We found several numeric high errors due to these phantom fingers. On the contrary, M2 (3) was not robust to heart beating motion artifacts, and these were segmented as part of the body. The network was robust to these artifacts. We further show in Fig. 4 that the network produced visually better segmentation masks than the rest of the other methods. Let us also highlight, that the network was able to gene- ralize to the 1.5T dataset, which had two significant differences from the 3T dataset. For instance i) subjects had their arms near the head in the 1.5T dataset, while in the 3T dataset all subjects had their hands on their chest ii) subjects where in a prone position in the 1.5T dataset, while in the 3T dataset, the subjects were in a supine position. Fig. 5 illustrates the results for two representative examples where the trained model provides better segmentation masks than all classical segmentation methods.

Application

Next we show an application where we use the proposed method to extract a 3D point-cloud from the individually segmented MRI slices of a subject and fit a body model (SMPL [START_REF] Loper | SMPL : A skinned multi-person linear model[END_REF]) to it. teresting applications, such as the registration of the MRI volumes through the body model parametrization, or the study of the deformations on the body due to gravity and the platform experienced during the MRI scan. We start with the full body MRI images, and we use the proposed method to extract the segmentations masks. Then, with the MRI metadata (pixel-space and slice distance), we transform the segmentations into a 3D pointcloud. We then register the SMPL [START_REF] Loper | SMPL : A skinned multi-person linear model[END_REF] body model to it by optimizing the point to mesh distance between the pointcloud and the SMPL body mesh. We initialized the SMPL mesh with the shape of the subject -obtained with a 3D scanner in a standing position -and optimized the SMPL pose parameters to best match the segmented point-cloud. We illustrate the segmented 3D point-clouds and the corresponding SMPL mesh fits in Fig. 7.

Conclusion

We proposed a simple and effective method to transform the manual pixel annotation task into an easy binary image choice task. By using several basic methods we automatically computed candidate segmentations. These were labeled as correct or wrong by a rater. We demonstrated that this process provided rich annotations which can be leveraged by a state of the art neural network. We showed that the trained network i) generalized well to another MRI dataset (1.5T), acquired with a different protocol and never seen at training time ii) was capable of automatically segmenting cases where none of the basic methods produced a correct result and iii) allowed to find segmentation errors in the 1.5T dataset. The trained network is made available at http://gentel.is.tue.mpg.de/. We also present an application where the segmented images are converted into coherent point-clouds and a body model is fit. Future work will address the segmentation of other anatomic structures, such as bones and the subcutaneous adipose tissue.

7 Disclosure

Michael J. Black has received research gift funds from Intel, Nvidia, Adobe, Facebook, and Amazon. While he is a part time employee of Amazon and has financial interests in Amazon and Meshcapade GmbH, his research was performed solely at, and funded solely by, MPI.

FIGURE 1 -

 1 FIGURE 1 -Obtained segmentation candidates. The contours of the candidate segmentations are shown in red over the input image (contour width was enlarged for display purposes). Different methods are better at different images. The best variant for each method is shown.

3

 3 Variants : We used C = [1, 2, 3].

3

 3 Variants : We retained 3 values for C = [1, 2, 3].

  12 Variants : We retained 3 values for C = [1, 2, 3], 2 values for IOP = [1, 2], and 2 values for doing or not doing the convexification [T rue, F alse].

FIGURE 2 -

 2 FIGURE 2 -Selection tool : the MRI image is displayed with a contour on it. The rater looks at the images sequentially and decides weather the segmentation should be retained or not.

FIGURE 3 -

 3 FIGURE 3 -Comparison of representative errors between the best classic algorithm M2(3) and the network. First row : M2 performs numerically better than the network (HD 8.0 vs 157.9). The network detects the fingers at the bottom of the image. Second row : the network performs numerically better than M2. M2 segments the heart beating artifact as body. The network is robust to the motion noise.

FIGURE 4 -

 4 FIGURE 4 -From left to right : Ground truth mask, segmentation obtained with the automatic methods (best variant), the segmentation mask obtained with the trained network. While none of the automatic methods obtained a good segmentation, the trained network predicted a very good mask.

FIGURE 5 -

 5 FIGURE 5 -Example of segmentations produced by the trained network on the 1.5T dataset. (First triplet) The network had never seen arms near the head, as all subjects in the training set had their hands on their chest. (Right triplet) The network had never seen prone subjects, as all subjects in the training set were lying in supine position. The trained model was able to obtain valid segmentations in these cases.

FIGURE 6 -

 6 FIGURE 6 -Our method allowed to detect several wrong reference segmentations. From left to right : input image, reference segmentation, segmentation mask and network segmentation contours.

FIGURE 7 -

 7 FIGURE 7 -Pairs of obtained point-clouds and SMPL body fit. We initialized the SMPL mesh with the shape of the subject -obtained with a 3D scanner in a standing position -and optimized the SMPL pose parameters to best match the segmented point-clouds.

Acknowledgments

We would like to thank Andrea Keller and Tsvetelina Alexiadis for the 3T MRI data acquisition. This research was partially supported by the French National Agency (ANR) under the JCJC project SEMBA.