Nikolas Hesse
email: nikolas.hesse@iosb.fraunhofer.de

Sergi Pujades
email: sergi.pujades-rocamora@inria.fr.

Michael J Black

Michael Arens

Ulrich G Hofmann

A Sebastian Schroeder

Learning and Tracking the 3D Body Shape of Freely Moving Infants from RGB-D sequences

Keywords: body models, data-driven, RGB-D, infants, motion analysis. !

Statistical models of the human body surface are generally learned from thousands of high-quality 3D scans in predefined poses to cover the wide variety of human body shapes and articulations. Acquisition of such data requires expensive equipment, calibration procedures, and is limited to cooperative subjects who can understand and follow instructions, such as adults. We present a method for learning a statistical 3D Skinned Multi-Infant Linear body model (SMIL) from incomplete, low-quality RGB-D sequences of freely moving infants. Quantitative experiments show that SMIL faithfully represents the RGB-D data and properly factorizes the shape and pose of the infants. To demonstrate the applicability of SMIL, we fit the model to RGB-D sequences of freely moving infants and show, with a case study, that our method captures enough motion detail for General Movements Assessment (GMA), a method used in clinical practice for early detection of neurodevelopmental disorders in infants. SMIL provides a new tool for analyzing infant shape and movement and is a step towards an automated system for GMA.

INTRODUCTION

S TATISTICAL parametric models of the human body, such as SCAPE [START_REF] Anguelov | SCAPE: Shape completion and animation of people[END_REF] or SMPL [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF] describe the geometry of the body surface of an observed population in a lowdimensional space. They are usually learned from dense high quality scans of the surface of the human body. Typically, subjects are instructed to stand in the same pose to simplify the problem of modeling body shape.

Since the pioneering work of Blanz and Vetter [START_REF] Blanz | A morphable model for the synthesis of 3D faces[END_REF] on a morphable face model, parametric shape models have been evolved and have found a wide range of applications in computer vision and computer graphics. For example, the low-dimensional representation of the human body surface has played a key role in enabling i) the precise capture of shape and pose of humans in motion from low quality RGB-D sequences [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF]; ii) the temporal registration of highly dynamic motions of the human body surface [START_REF] Bogo | Dynamic FAUST: Registering human bodies in motion[END_REF]; iii) the prediction of human shape and pose from single RGB images based on deep neural networks [START_REF] Tung | Selfsupervised learning of motion capture[END_REF], [START_REF] Omran | Neural Body Fitting: Unifying deep learning and model based human pose and shape estimation[END_REF], [START_REF] Pavlakos | Learning to estimate 3D human pose and shape from a single color image[END_REF]; and iv) learning detailed avatars from monocular video [START_REF] Alldieck | Detailed human avatars from monocular video[END_REF] .

Human movements contain key information allowing to infer internal properties of the observed subject [START_REF] Lu | Human identity and gender recognition from gait sequences with arbitrary walking directions[END_REF]. In our work, we consider the application of motion analysis, i.e., the acquisition and quantification of poses an observed subject strikes. Human motion analysis is used in medicine Manuscript revised March 18, 2019 for patient monitoring [START_REF] Achilles | Patient MoCap: Human pose estimation under blanket occlusion for hospital monitoring applications[END_REF], quantifying therapy or disease progression [START_REF] Kontschieder | Quantifying progression of multiple sclerosis via classification of depth videos[END_REF], or performance assessment [START_REF] Clark | Validity of the Microsoft Kinect for assessment of postural control[END_REF]. Most interestingly, it can be applied to the early detection of neurodevelopmental disorders like cerebral palsy (CP) in infants at a very early age. The General Movements Assessment (GMA) enables trained experts to detect CP at an age of 2 to 4 months, based on assessing the movement quality of infants from video recordings [START_REF] Prechtl | Qualitative changes of spontaneous movements in fetus and preterm infant are a marker of neurological dysfunction[END_REF]. Infants with abnormal movement quality have very high risk of developing CP or minor neurological dysfunction [START_REF] Hadders-Algra | General movements: a window for early identification of children at high risk for developmental disorders[END_REF]. While GMA is the most accurate clinical tool for early detection of CP, it is dependent on trained experts and is consequently subject to human perceptual variability. GMA experts require regular practice and re-calibration to assure accurate ratings. Automation of this analysis could reduce this variability and dependence on human judgment. To allow GMA automation, a practical system must first demonstrate that it is capable of capturing the relevant information needed for GMA. Moreover, to allow its widespread use, the solution needs to be seamlessly integrated into the clinical routine. Approaches aimed at GMA automation have relied on wearable sensors or vision-based systems for capturing infant motion. For a review of existing methods, we refer the reader to [START_REF] Marcroft | Movement recognition technology as a method of assessing spontaneous general movements in high risk infants[END_REF] and [START_REF] Hesse | Computer vision for medical infant motion analysis: State of the art and rgb-d data set[END_REF].

Inspired by previous work on capturing motion from RGB-D data using a body model [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF], we follow this direction in our work. Two main problems arise on the way to capture infant motion using a body model. The first problem is that there is no realistic infant body model. While parametric body models like SMPL [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF] cover a wide variety of adult body shapes, the shape space does not generalize to the new domain of infant bodies (see Fig. 1a). As the body part dimensions between infants and adults vary significantly, the goal of this work is to learn an infant body model that faithfully captures the shape of infants (see Fig. 1b).

However, most statistical models are learned from high- quality scans, which are expensive, and demand cooperative subjects willing to follow instructions. This is the second problem we face: there is no repository of high quality infant 3D body scans from which we could learn the statistics of infant body shape. Acquiring infant shape data is not straightforward, as one needs to comply with strict ethics rules as well as a an adequate environment for the infants. Therefore, we acquire sequences of moving infants in a children's hospital. We use one RGB-D sensor and a laptop as this provides a low-cost, easy-to-use alternative to bulky and expensive 3D scanners. Our proposed system produces minimal overhead to the standard examination protocol, and does not affect the behavior of the infants. We use the captured RGB-D sequences to learn an infant body model. Infant RGB-D data poses several challenges. We have to deal with incomplete data, i.e., partial views, where large parts of the body are occluded most of the time. The data is of low quality and noisy, and captured subjects are not able to follow instructions and take predefined poses.

Contributions. We present the first work on 3D shape and 3D pose estimation of infants, as well as the first work on learning a statistical 3D body model from low-quality, incomplete RGB-D data of freely moving humans. We contribute (i) a new statistical Skinned Multi-Infant Linear model (SMIL), learned from 37 RGB-D low-quality sequences of freely moving infants, and (ii) a method to register the SMIL model to the RGB-D sequences, capable of handling severe occlusions and fast movements. Quantitative experiments show how SMIL properly factorizes the pose and the shape of the infants, and allows the captured data to be accurately represented in a low-dimensional space. With a case-study involving a high-risk former preterm study population, we demonstrate that the amount of motion detail captured by SMIL is sufficient to enable accurate GMA ratings by humans. Thus, SMIL provides a fundamental tool that can form a component in an automated system for the assessment of GMs. We make SMIL available to the community for research purposes 1 . This article is an extended version of [START_REF] Hesse | Learning an infant body model from RGB-D data for accurate full body motion analysis[END_REF].

1. Model available at http://s.fhg.de/smil

RELATED WORK

We review two main areas: the creation of statistical models of the human body surface and the estimation of shape and pose of humans in movement from RGB-D sequences.

Human Surface Models

Statistical parametric models of the human body surface are usually based on the Morphable Model idea [START_REF] Blanz | A morphable model for the synthesis of 3D faces[END_REF], stating that a single surface representation can morph and explain the different samples in a population. These models can be intuitively viewed as a mathematical function taking shape and pose parameters as input and returning a surface mesh as output. The shape space and the pose prior, i.e., the statistics of most plausible poses, are learned by registering the single surface representation, i.e., a template surface, to real-world data. The shape space and the pose prior allow a compact representation of the human body surface describing the geometry of the human body surface of an observed population in a low-dimensional space. Existing models have been learned from different real-world data. For example, to model the variation of faces De-Carlo et al. [START_REF] Decarlo | An anthropometric face model using variational techniques[END_REF] learn a model from a cohort of anthropometric measurements. Blanz and Vetter [START_REF] Blanz | A morphable model for the synthesis of 3D faces[END_REF], use dense geometry and color data to learn their face model. Allen et al. [START_REF] Allen | The space of human body shapes: Reconstruction and parameterization from range scans[END_REF] use the CAESAR dataset to create the space of human body shapes by using the geometry information as well as sparse landmarks identified on the bodies. Similarly, Seo et Magnenat-Thalmann [START_REF] Seo | An automatic modeling of human bodies from sizing parameters[END_REF] learn a shape space from high quality range data. Angelov et al. propose SCAPE [START_REF] Anguelov | SCAPE: Shape completion and animation of people[END_REF], a statistical body model learned from high quality scan data, which does not only contain the shape space, but also accounts for the pose dependent deformations, i.e., the surface deformations that a body undergoes when different poses are taken. SCAPE and all successive statistical models of the human body surface [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF], [START_REF] Hasler | A statistical model of human pose and body shape[END_REF], [START_REF] Hirshberg | Coregistration: Simultaneous alignment and modeling of articulated 3D shape[END_REF], [START_REF] Chen | Tensor-based human body modeling[END_REF], [START_REF] Pishchulin | Building statistical shape spaces for 3D human modeling[END_REF] or their soft tissue dynamics [START_REF] Pons-Moll | Dyna: A model of dynamic human shape in motion[END_REF], [START_REF] Kim | Data-driven physics for human soft tissue animation[END_REF] have been learned from a relatively large number of high quality range scans of adult subjects. Adults, in contrast to infants, are typically cooperative and can be instructed to strike specific poses during scanning.

Animal shape modeling methods face a similar difficulty as ours: live animals are generally difficult to instruct and their motions make them difficult to scan. Thus, they provide a source of inspiration to create models without a large cohort of high quality scans. Cashman and Fitzgibbon [START_REF] Cashman | What shape are dolphins? Building 3D morphable models from 2D images[END_REF] learn a deformable model of dolphins by using manually annotated 2D images. Kanazawa et al. [START_REF] Kanazawa | Learning 3D deformation of animals from 2D images[END_REF] learn the deformations and the stiffness of the parts of a 3D mesh template of an animal from manually annotated 2D images. To create the animal model SMAL [START_REF] Zuffi | 3D Menagerie: Modeling the 3D shape and pose of animals[END_REF], Zuffi et al. circumvent the difficulty to instruct and scan real animals by using high quality scans of toy figurines. They fit their model to new animals using a set of multi-view images with landmarks and silhouette annotations.

In this work, we learn our statistical model of the shape of infants from low-quality RGB-D sequences. While there are several methods that fit body shape to RGB-D data, we do not know of any that estimates a statistical model from such input. Our method does not rely on manually annotated landmarks and leverages the ones that are automatically extracted from RGB images [START_REF] Wei | Convolutional pose machines[END_REF], [START_REF] Cao | Realtime multi-person 2D pose estimation using part affinity fields[END_REF], [START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF].

Capturing motion from RGB-D using a body model

The existing body models have proven to be successful in capturing the pose and shape of a subject from RGB-D sequences. The model is parameterized with the shape, giving information about the joint locations, and the pose, defining the angles between limbs. Once a model is registered to the input data, one can obtain the desired motion information. As the joint locations depend on the shape, the closer the estimated body shape is to the actual subject's shape, the better, i.e., more accurate, the tracking of motions will be.

Parametric body models generally model the space of human bodies without clothing or hair. Some approaches propose to overcome the discrepancy between such a shape space and the real world by creating personalized avatars from the input data and then registering them to the dynamic sequences by keeping the personalized shape fixed. This creates an additional step and usually requires cooperative subjects to take predefined poses. Personalized avatars have been created from multiple Kinect scans [START_REF] Weiss | Home 3D body scans from noisy image and range data[END_REF], [START_REF] Helten | Personalization and evaluation of a real-time depth-based full body tracker[END_REF], Kinect fusion scans [START_REF] Zhang | Quality dynamic human body modeling using a single low-cost depth camera[END_REF], [START_REF] Chen | Realtime reconstruction of an animating human body from a single depth camera[END_REF] or laser scanners [START_REF] Ye | Performance capture of interacting characters with handheld Kinects[END_REF] and the obtained avatars are then registered to different RGB-D scans of the same person in different poses.

Other methods use a parametric body model to capture the pose without the preliminary step of the personalized shape creation. Ganapathi et al. [START_REF] Ganapathi | Real-time human pose tracking from range data[END_REF] use a simplistic body model for real-time pose tracking from range data. Ye and Yang [START_REF] Ye | Real-time simultaneous pose and shape estimation for articulated objects using a single depth camera[END_REF] introduce a method for real-time shape and pose tracking from depth. They register an articulated deformation model to point clouds within a probabilistic framework. Yu et al. [START_REF] Yu | DoubleFusion: Real-time capture of human performances with inner body shapes from a single depth sensor[END_REF] capture non-rigid surface motion by combining a parametric body model to represent the inner body shape with a freely deformable outer surface layer capturing surface details.

Bogo et al. [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF] fit a multi-resolution body model to Kinect sequences. This work is the closest to ours, which is why we give a brief summary and identify similarities and differences. The authors aim at creating highly realistic textured avatars from RGB-D sequences. They capture shape, pose and appearance of freely moving humans based on a parametric body model, which is learned from 1800 high-quality 3D scans of 60 adults. They create a personalized shape for each sequence by accumulating shape information over the sequence in a "fusion cloud". The captured subjects wear tight clothing and take a predefined pose at the beginning of each sequence, as is common in scanning scenarios. They use their body model at different resolutions in a coarse-tofine manner to increasingly capture more details. They use a displacement map to represent fine details that lie beyond the resolution of the model mesh. The model contains a head-specific shape space for retrieving high-resolution face appearance.

In our work, we also capture shape and pose of sequences containing unconstrained movements. To create a personalized shape, we also merge all temporal information into fusion clouds. In the gradient-based optimization, some of our energy terms are similar to the ones from [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF]. In contrast to their work, an initial infant body model is not available and we must create it. We adapt an existing adult body model to a different domain, namely infants, to use as an initial model for registering our infant sequences.

The fact that infants lie in supine position in our scenario presents two different constraints. First, it means that very few backs are visible and we have to deal with large areas of missing data in our fusion clouds. Second, as infants are in contact with the background, i.e., the examination table, we can not rely on a background shot to segment the relevant pointcloud. When the infants move they wrinkle the towel they are lying on with their hands and feet. However, we can take advantage of the planar geometry to fit a plane to the table data to segment it. Moreover, we can (and do) use the fitted plane as a geometric constraint, as we know the back of the infants can not be inside the examination table. Also, in contrast to [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF], we can not rely on predefined poses for initialization since the infants are too young to strike poses on demand. We contribute a new automatic method for choosing the best poses for initialization. Moreover, the clothing in our setting is not constrained: we have to deal with diapers, onesies and tights. In particular, diapers pose a challenge since their shape largely deviates from the human body. We handle the unconstrained cloth condition by segmenting the points corresponding to clothes, and by introducing different energy terms for clothing and skin parts. Finally, in our work we do not use the appearance of the surface, but rather use the RGB information to extract 2D landmark estimates to have individual constraints on the face and hand rotations.

LEARNING THE SKINNED MULTI-INFANT LINEAR MODEL FROM LOW QUALITY RGB-D DATA

Learning a body model from data is a chicken-and-egg problem. We need a model to register the data to a common topology, and we need registered, or aligned, meshes to learn a model. Since no infant body model is available, we first create an initial infant model by adapting the adult SMPL model [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF] (see Sec. 3.2). We then register this initial model to RGB-D sequences of moving infants (Sec. 3.3). To mitigate the incompleteness of data due to the monocular setup, we accumulate shape information from each sequence into one "personalized shape" (Sec. 3.6). Finally, we learn a new infant shape space from all personalized shapes, as well as a new prior over plausible infant poses from our registrations (Sec. 3.7). An overview of the complete learning pipeline is given in Fig. 2.

Data

There are multiple reasons why no public repository of infant 3D scans exists. Protection of privacy of infants is more strict as compared to adults. The high cost of 3D scanners prevents them from being widespread. Creating a scanning environment that takes into consideration the special care required by infants, like warmth and hygiene, requires additional effort. Finally, infants can not be instructed to strike poses on demand, which is usually required in standard body scanning protocols. RGB-D sensors offer a cheap and lightweight solution for scanning infants, only requiring the sensor and a connected laptop. The data used to learn SMIL was obtained by setting up recording stations at a children's hospital where infants and parents regularly visit for examinations. The acquisition protocol was integrated in the doctor's medical routine in order to minimize overhead.

Align

Personalized shapes PCA shape space

Pose prior Poses

Initial Model SMPL B

Registration

SMIL

Data acquisition and preprocessing

Fig. 2. Skinned Multi-Infant Linear (SMIL) model creation pipeline. We create an initial infant model based on SMPL. We perform background and clothing segmentation of the recorded sequences in a preprocessing step, and estimate body, face, and hand landmarks in RGB images. We register the initial model, SMPL B , to the RGB-D data, and create one personalized shape for each sequence, capturing infant shape details outside the SMPL shape space. We learn a new infant specific shape space by performing PCA on all personalized shapes, with the mean shape forming our base template. We also learn a prior over plausible poses from a sampled subset of all poses.

Preprocessing. We transform depth images to 3D point clouds using the camera calibration. To segment the infant from the scene, we fit a plane to the background table of the 3D point cloud using RANSAC [START_REF] Fischler | Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography[END_REF] and remove all points close to or below the table plane and apply a simple cluster-based filtering. Further processing steps operate on this segmented cloud, in which only points belonging to the infant remain. Plane-based segmentation is not always perfect, e.g., in case of a wrinkled towel very close to the infant body, some noise may remain. However, the registration methods have proven to be robust to outliers of this kind. The estimated table plane will be reused for constraining the infants' backs in the registration stage (Sec. 3.3).

In order to avoid modeling diapers and clothing wrinkles in the infant shape space, we segment the input point clouds into clothing and skin using the color information by adapting the method from Pons-Moll et al. [START_REF] Pons-Moll | ClothCap: Seamless 4D clothing capture and retargeting[END_REF]. The clothing type determines the number of color modes k and the distribution of clothing and skin probabilities on the model surface. We perform unsupervised k-means clustering to obtain the dominant modes in RGB, and one of k labels is assigned to each scan point. We register the initial model, SMPL B (Sec. 3.2), to the scan in order to transfer the scan point labels to the model vertices, and solve a minimization problem on a Markov random field defined by the model topology. This gives us a clean segmentation of the scan points belonging to clothing (or diaper) and the ones belonging to the skin. An example of the segmentation result can be seen in the Data acquisition and preprocessing box of Fig. 2; blue is skin and red is diaper. To avoid registering all scans twice, i.e., a first rough registration to segment the sequence and a second to obtain the final registration, we transfer the clothing labels from the registration at frame t-1 to the point could at frame t. In practice this works well, since most of the clothing is on, or close to the trunk, which does not move much in consecutive frames. In sequences with clothing types longarm onesie and tights we do not observe severe labeling failures due to fast motion.

Scanning of adults typically relies on them striking a simple pose to facilitate model fitting and registrations. The scanned infants can not take a predefined pose to facilitate an initial estimate of model parameters. However, existing approaches on 2D pose estimation from RGB images (for adults) have achieved impressive results. Most interestingly, experiments show that applying these methods to images of infants produces accurate estimates of 2D pose [START_REF] Hesse | Computer vision for medical infant motion analysis: State of the art and rgb-d data set[END_REF]. In order to choose a "good" candidate frame to initialize the model parameters (see Sec. 3.5), we leverage the 2D body landmarks together with their confidence values. From the RGB images we extract body pose [START_REF] Cao | Realtime multi-person 2D pose estimation using part affinity fields[END_REF] as well as face [START_REF] Wei | Convolutional pose machines[END_REF] and hand [START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF] landmarks. We experimentally verify that they provide key information on the head and hand rotations to the registration process, which is complementary to the noisy point clouds.

Initial model

We manually create an initial model, SMPL B , by adapting the Skinned Multi-Person Linear model (SMPL) [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF], which we briefly recap. SMPL is a linear model of shape and pose. It represents shape deformations as a combination of identity specific shape (shape blend shapes) and pose-dependent shape (pose blend shapes). The pose blend shapes are learned from 1786 registered scans of adults in predefined poses, while the shape blend shapes are learned from registered scans of 1700 males and 2100 females. The SMPL shape space is represented by a mean template shape and principal shape directions that are created by performing principal component analysis (PCA) on pose-normalized registered meshes. The shape of a body is described by a vector of linear coefficients, β, that multiply the principal component displacements. The model can be viewed as a mapping from shape and pose parameters to a shaped and posed mesh. Shape and pose blend shapes are modeled as vertex offsets, which are added to the mean template, and the result is then transformed by a standard blend skinning function to form the output mesh. The SMPL template consists of 6890 vertices and 23 body joints. Each body joint has 3 degrees of freedom (DoF) resulting, with 3 DoF for global rotation, in 72 pose parameters, θ. SMPL contains a learned joint regressor for computing joint locations from surface vertices.

Adaptation to infants. We manually create an initial infant mesh using makeHuman, an open source software for creating 3D characters. We wish to register SMPL to this mesh to use this base shape in the SMPL model. Directly registering SMPL to the infant mesh fails due to differences in size and proportions. We make use of the fact that meshes exported from makeHuman share the same topology, independent of shape parameters. We register SMPL to an adult makeHuman mesh, and describe makeHuman vertices as linear combinations of SMPL vertices. This allows us to apply this mapping to the infant mesh and transfer it to the SMPL topology. We then replace the SMPL base adult-shape template with the registered infant mesh.

We further scale the SMPL pose blend shapes, which correct skinning artifacts and pose-dependent shape deformations, to infant size. Specifically, we divided infant height by average adult height and multiply the blend shapes by this factor. We keep the SMPL joint regressor untouched, since it worked well for infants in our experiments.

The initial pose priors, i.e., prior probabilities of plausible poses, are learned from data of adults in upright positions. We used the same pose prior as [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF] which is computed from more than 700 sequences of the CMU MoCap data set. These priors can not be directly transferred to lying infants. We experienced that the (adult) shape space of SMPL B does not properly explain round infant bellies, which is why the spine was bent to create a more infant-like belly. To avoid explaining shape deformations with pose parameters, we penalize the bending of the spine, which is supposed to be relatively straight in supine position. Infant feet are not conditioned on standing poses, which is why we further adjust the prior to to allow the ankle to extend more.

Registration

We register the initial model to the segmented point cloud using gradient-based optimization. The main energy being optimized w.r.t. shape β and pose θ parameters is

E(β, θ) = E data + E lm + E table + E sm + E sc + E β + E θ . (1)
We omit the weight factors λ term associated with E term , term ∈ {data, lm, table, sm, sc, β, θ}, for compactness. In the following, we explain each term of the energy in detail.

Data term. The data term E data consists of two terms:

E data = λ m2s E m2s + E s2m . (2)
E m2s accounts for the distance of the visible model vertices to the scan points and E s2m accounts for the distance of the scan points to the model surface E m2s can be written as

E m2s (M, P ) = m∈vis(M) ρ(min v∈P ||(m, v)||), (3)
where M denotes the model surface and P the scan points. min v∈P ||(m, v)|| is the Euclidean distance between each visible model point and its closest scan point, and ρ is the robust Geman-McClure function [START_REF] Geman | Statistical methods for tomographic image reconstruction[END_REF]. In the preprocessing stage, P is segmented into the scan points belonging to the skin (P skin ) and the ones belonging to clothing (P cloth ). The function vis(M) selects the visible model vertices. The visibility is computed using the Kinect V1 camera calibration and the OpenDR renderer [START_REF] Loper | OpenDR: An approximate differentiable renderer[END_REF]. E s2m consists of two terms, E skin enforces the skin points to be close to the model mesh and E cloth enforces the cloth points to be outside the model mesh. The skin term can be written as

E s2m = λ skin E skin + λ cloth E cloth . (4)
E skin (M, P skin , W ) = vi∈P skin W i ρ(dist(v i , M )), (5)
where dist(v i , M ) gives the minimal distance of vertex v i to the model mesh surface M , and W i are the skin weights associated with v i , which are computed using the same method as in [START_REF] Zhang | Detailed, accurate, human shape estimation from clothed 3D scan sequences[END_REF]. These are used to avoid learning clothing wrinkles in the shape space (cf. Sec. 3.7). We assign low weights to clothing points and high weights to skin points with smooth transition weights in between (see Fig. 3) to avoid discontinuities at the clothing borders. The cloth term is divided into two more terms, depending on cloth points lying inside or outside the model mesh:

E cloth = E outside + E inside , (6)
with E outside applying a Geman-McClure penalty to cloth points lying outside the body. This is to keep the model from matching clothing points that are far away from the body. At the same time, nearby points are closely matched.

E outside (M, P cloth , W ) = vi∈P cloth δ(v i ) out W i ρ(dist(v i , M )), (7)
with δ(v i ) out an indicator function, returning 1 if v i lies outside the model mesh, else 0.

The term E inside "pushes" the model inside the cloth points. We apply a quadratic penalty to cloth points lying inside the body mesh, since clothing is not supposed to penetrate the body at all.

E inside (M, P cloth , W ) = vi∈P cloth δ(v i ) in W i dist(v i , M ) 2 , (8)
with δ(v i ) in an indicator function, returning 1 if v i lies inside the model mesh, else 0. Only using E inside would favor shrinking the model too much.

Landmark term. Due to the low-quality of depth data, depth-only methods can not reliably capture details like head or hand rotations. However, we can estimate 2D landmark positions from the RGB images and use them as additional constraints in the optimization energy. Body landmarks [START_REF] Cao | Realtime multi-person 2D pose estimation using part affinity fields[END_REF] are used for initialization (Sec. 3.5), whereas face [START_REF] Wei | Convolutional pose machines[END_REF] and hand [START_REF] Simon | Hand keypoint detection in single images using multiview bootstrapping[END_REF] landmarks are used in the registration energy of every frame. In cases where the face detection [START_REF] Wei | Convolutional pose machines[END_REF] fails, mostly profile faces, we use the ear and eye landmarks from the body pose estimation method [START_REF] Cao | Realtime multi-person 2D pose estimation using part affinity fields[END_REF]. These help to guide the head rotation in such extreme cases.

The landmark term E lm is similar to Eq. 2 from [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF], where the distances between the 2D landmarks estimated from RGB and the corresponding projections of the 3D model landmarks are measured. Instead of using the body joints, which we use only for guiding the initial fitting (Sec. 3.5), we only use the estimated 2D face landmarks (nose, eyes outlines, mouth outline and ears) as well as the hand landmarks (knuckles). We note the set of all markers as L. The 3D model points corresponding to the above landmarks were manually selected through visual inspection. They are projected into the image domain using the camera calibration matrix in order to compute the final 2D distances to the estimated landmarks. The landmark term is then

E lm = λ lm l∈L c l ρ(||l M -l est ||), (9)
where c l denotes the confidence of an estimated landmark 2D location l est , and l M is the projected model landmark location. All confidences from the different methods are in the interval [0, 1], making them comparable in terms of magnitudes.

Table term. The recorded infants are too young to roll over, which is why the back is rarely seen by the camera. However, the table on which the infants lie, lets us infer shape information of the back. We assume that the body can not be inside the table, and that a large part of the back will be in contact with it. The table energy has two terms: E in prevents the model vertices M from lying inside the table, i.e., behind the estimated table plane, by applying a quadratic error term on points lying inside the table. E close acts as a gravity term, pulling the model vertices M towards the table by applying a robust Geman-McClure penalty function to vertices that are close to the table and facing away from the camera. We write the table energy term as

E table = λ in E in + λ close E close , (10)
with

E in (M) = x∈M δ in (x)dist(x, Π) 2 , (11)
and Other terms. Depth data contains noise, especially around the borders. To avoid jitter in the model caused by that noise, we add a temporal pose smoothness term. It avoids important changes in pose unless one of the other terms has strong evidence. The temporal pose smoothness term E sm is the same as in Eq. 21 in [START_REF] Romero | Embodied Hands: Modeling and capturing hands and bodies together[END_REF] and penalizes large differences between the current pose θ and the pose from the last processed frame θ . The penalty for model self intersections E sc and the shape prior term E β are the same as in Eq. 6 and Eq. 7 in [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF] respectively. Bending the model in unnatural ways might decrease the data term error, which is why the pose prior term keeps the pose parameters in a realistic range. The SMIL pose prior consists of a mean and covariance matrix that were learned from 37,000 sample training poses; these are not used during testing. E θ penalizes the squared Mahalanobis distance between θ and the pose prior, as described in [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF].

E close (M) = x∈M δ close (x)ρ(dist(x, Π)), (12)

Registration Optimization

To compute the registrations of a sequence we start by computing an initial shape using 5 frames. In this first step, we optimize for the shape and pose parameters, β and θ, as well as the global translation t. The average shape parameters from these 5 frames will be kept fixed and used later on as a shape regularizer. Experiments showed that otherwise the shape excessively deforms in order to explain occlusions in the optimization process.

With the initial shape fixed, we compute the poses for all frames in the sequence, i.e., we optimize the following energy w.r.t. pose parameters θ and the global translation t:

E(θ, t) = E data + E lm + E table + E sm + E sc + E θ . (13)
Notice that this energy is equal to Eq. 1 without the shape term E beta , as shape is kept fixed. We denote S f the computed posed shape at frame f .

In the last step, we compute the registration meshes R f and allow the model vertices v ∈ R f to freely deform to best explain the input data. We optimize w.r.t. v the energy

E(v) = E data + E lm + E table + E cpl , (14)
where E cpl is a "coupling" term, used to keep the registration edges close to the edges of the initial shape. We use a similar energy term as Eq. 8 in [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF]:

E cpl (R f , S f ) = λ cpl e∈V ||(AR) e -(AS) e || 2 F , (15)
with V denoting the edges in the model mesh. AR and AS are edge vectors of the triangles of R f and S f , and e indexes the edges. The results of these optimizations are the final registrations. All energies are minimized using a gradient-based dogleg minimization method [START_REF] Nocedal | Numerical optimization[END_REF] with OpenDR [START_REF] Loper | OpenDR: An approximate differentiable renderer[END_REF] and Chumpy [START_REF] Loper | Chumpy[END_REF]. We display registration samples in Fig. 4.

Initialization

In order to find the global minimum, the optimization needs a good initial estimate. In adult settings, subjects are usually asked to take an easy pose, e.g., T-pose (extended arms and legs), at the start of the recording. Infants are not able to strike poses on demand, which is why we can not rely on a predefined pose.

We automatically find an initialization frame containing an "easy" pose by relying on 2D landmark estimates acquired in the preprocessing stage. We make the assumption that a body segment is most visible if it has maximum 2D length over the complete sequence. Perspective projection would decrease 2D body segment length and therefore visibility. The initialization frame is chosen as

f init = argmax f s∈S len(s, f )c(s, f ), ( 16
)
where S is the set of body segments, len(s, f ) is the 2D length of the segment s in frame f , and c(s, f ) is the estimated confidence of joints belonging to s in frame f . For f init , we optimize a simplified version of Eq. 1, i.e., the initialization energy [START_REF] Hesse | Computer vision for medical infant motion analysis: State of the art and rgb-d data set[END_REF] where E j2d is similar to E lm with landmarks being 2D body joint positions. E j2d is needed to guide the initial pose towards the target pose without getting stuck in local minima (due to other error terms). In the sequential registration (Eq. 1), the initial pose is taken as the result from the previous frame and therefore is already close to the target pose, rendering E j2d obsolete. E θ and E β are pose and shape priors as in Eq. 1 with strong weights λ θ and λ β . E a (θ) = i exp(θ i ) is an angle limit term for knees and elbows. The minimum of E init provides a coarse estimation of shape and pose, which is refined afterwards. In contrast to [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF], we omit the self intersection term, and add a scan-tomesh distance term E s2m , containing 3D information, while [START_REF] Bogo | Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image[END_REF] solely relies on 2D information.

E init = λ j2d E j2d + λ θ E θ + λ β E β + λ a E a + λ s2m E s2m

Personalized Shape

To capture the subject specific shape details, we create one personalized shape from each sequence, which we do not restrict to the shape space of the model. We unpose a randomly selected subset of 1000 frames per sequence. The process of unposing changes the model pose to a normalized pose (Tpose) in order to remove variance related to body articulation. For each scan point, we calculate the offset normal to the closest model point. After unposing the model, we add these offsets to create the unposed point cloud for each of the 1000 frames. Since the recorded infants lie on their backs most of the time, the unposed clouds have missing areas on the back side. To take advantage of the table constraint in each frame and sparsly fill the missing areas, we add virtual points, i.e., points from model vertices that belong to faces oriented away from the camera, to the unposed cloud. We retain the clothing segmentation labels for all unposed scan points. We call the union of all unposed point clouds including virtual points the fusion cloud; cf. [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF].

To compute the personalized shape, we uniformly sample 1 million points at random from the fusion cloud and proceed in two stages. First, we optimize E = E data + E β w.r.t. the shape parameters β, and keep the pose θ fixed in the zero pose of the model (T-pose with legs and arms extended). We obtain an initial shape estimate that lies in the shape space of the initial model SMPL B . Second, we allow the model vertices to deviate from the shape space, but tie them to the shape from the first stage with a coupling term.

We optimize E = E data + E cpl w.r.t. the vertices.

The clothing segmentation is also transformed to the unposed cloud and therefore, the fusion cloud is labeled into clothing and skin parts. These are used in the data term to enforce clothing points to lie outside the model surface and to avoid learning clothing artifacts in the shape space.

Learning SMIL shape space and pose prior

We compute the new infant shape space by doing weighted principal component analysis (WPCA) on personalized shapes of all sequences. Despite including the clothing segmentation in the creation of personalized shapes, clothing deformations can not be completely removed and diapers typically tend to produce body shapes with an over-long trunk. The recorded sequences contain infants with longarm onesies, short-arm onesies, tights, diapers and without clothing. These different clothing types cover different parts of the body. We use a weight value of 1 for clothing points P cloth and a weight value of 3 for skin points P skin in the PCA. We compute weighted PCA with an iterative expectation-maximization approach 2 and retain the first 20 shape components.

In Sec. 4, we describe the data set from which SMIL is learned, consisting of more than 200K frames . We create a pose data set by looping over all poses of all sequences and only add poses to the set if the dissimilarity to any pose in the set is larger than a threshold. The final set contains 47K poses and is used to learn the new pose prior. As the Gaussian pose prior can not penalize illegal poses, e.g., unnatural bending of knees, we manually add penalties to avoid such poses. The final SMIL model is composed of the shape space, the pose prior, and the base template, which is the mean of all personalized shapes.

Manual Intervention

In our method we use manual intervention three times: i) to decide which type of clothing the infant is wearing (see Sec 3.1); ii) to generate the initial model SMPL B (see Sec 3.2) and iii) to define illegal poses in the pose prior. The illegal poses are only defined once and the initial model is no longer used once SMIL is learned. However, given a new sequence, one still needs to manually define the type of clothing: short onesie, long onesie, tights, naked or diapers. Each cloth type defines the corresponding number of color modes and priors to be used. We believe that a classifier predicting the clothing type from RGB images could be learned, making our method fully automatic.

Method Parameters

The values of the weights in the energy functions were empirically adjusted to keep the different terms balanced. For optimization of the main energy w.r.t. shape and pose parameters (Eq. 1) and the modified energy w.r.t. pose parameters (Eq. 13) we use the weight values: λ skin = 800, λ cloth = 300, λ m2s = 400, λ lm = 1, λ table = 10000, λ sm = 800, λ sc = 1, λ β = 1 and λ θ = 0.15. For optimization of the energy w.r.t. the model vertices (Eq. 14) we use: λ skin = 1000, λ cloth = 500, λ m2s = 1000, λ lm = 0.03, λ table = 10000 and λ cpl = 1. For the creation of the personalized shape (Sec. 3.6), we use: λ skin = 100, λ cloth = 100 λ β = 0.5 and λ cpl = 0.4. Finally, for the initialization energy (Eq. 17), we use: λ j2d = 6, λ θ = 10, λ a = 30, λ β = 1000, λ s2m = 30000. We keep the chosen weights constant for all experiments.

EXPERIMENTS

As elaborated in the introduction, gathering high quality 3D scans of infants is highly unpractical, which is why we quantitatively evaluate SMIL and our initial model SMPL B on the 37 acquired RGB-D sequences of infants. We record the infants using a Microsoft Kinect V1, which is mounted 1 meter above an examination table, facing downwards. All parents gave written informed consent for their child to participate in this study, which was approved by the ethics committee of Ludwig Maximilian University Munich (LMU). The infants lie in supine position for three to five minutes without external stimulation, i.e., there is no interaction with caregivers or toys. The recorded infants are between 9 and 18 weeks of corrected age (post term), and their size range is 42 to 59 cm, with an average of 53.5 cm. They wear different types of clothing: none, diaper, onesie shortarm / longarm, or tights. All sequences together sum up to roughly 200K frames, and have an overall duration of over two hours. We evaluate SMIL with a 9-fold crossvalidation, using 33 sequences for training the shape space and the pose prior, and 4 for testing. We distribute different clothing styles across all training sets. We measure the distance E s2m (cf. Eq. 4) of the scan to the model mesh by computing the Euclidean distance of each scan point to the mesh surface. For evaluation, we consider all scan points to be labeled as skin, which reduces Eq. 4 to Eq. 5. Note that we do not use the Geman-McClure function ρ here, as we are interested in the actual Euclidean distances.

To compare the SMPL B shape space to the SMIL shape space, we register both models to each of the 37 fusion scans, using different numbers of shape components. Results are displayed in Fig. 6. We plot average error heatmaps for using the first 1, 3, 5 and all 20 shape components for the registrations (Fig. 7). We observe lower error for SMIL for smaller numbers of shape parameters, and a nearly identical error when using all 20 parameters. The small difference between SMPL B 3 and SMIL poses the question if there is a significant difference between the two. A visual comparison of the personalized shape registration with all 20 shape components shows that SMPL B shapes display more adultlike features, e.g., a more pronounced waisting, a more "muscular" chest and shoulder region, as well as a smaller (fore)head area (Fig. 5).

To evaluate how well the computed personalized shapes and poses explain the input sequences, we calculate E s2m for all 200K frames. SMIL achieves an average scan-to-mesh distance of 2.51 mm (SD 0.21 mm), SMPL B has an average E s2m of 2.67 mm (SD 0.22 mm), which is slightly higher than the error of the fusion cloud registrations (cf. Fig. 6). This is in the range of Bogo et al. [START_REF] Bogo | Detailed fullbody reconstructions of moving people from monocular RGB-D sequences[END_REF], who report an average 3. Please note that SMPL B is not the SMPL model [START_REF] Loper | SMPL: A skinned multi-person linear model[END_REF], but our initial infant model, registered to the data using our method. error between 2.45mm (static scans) and 3.23 mm (arbitrary motions) on adults captured with a Kinect V2. Due to the lack of ground truth data for evaluation of infant pose correctness, we perform a manual inspection of all sequences to reveal pose errors. We distinguish between "unnatural poses" and "failure cases". Unnatural poses contain errors in pose, like implausible rotations of a leg (cf. Fig. 8 top row), while the overall registration is plausible, i.e., the 3D joint positions are still at roughly the correct position. Failure cases denote situations in which the optimization gets stuck in a local minimum with a clearly wrong pose, i.e., one model body part registered to a scan part which it does not belong to (cf. Fig. 8 bottom row). We count 16 unnatural leg/foot rotations lasting 41 seconds (= 0.54% of roughly 2 hours) and 18 failure cases (in 7 sequences) lasting 49 seconds (= 0.66% of roughly 2 hours).

To evaluate how well SMIL generalizes to older infants, we register the model to 25 sequences of infants at the age between 21 and 36 weeks, at an average of 26 weeks. The resulting average scan to mesh distance is 2.83 mm (SD: 0.31 mm). With increasing age, infants learn to perform directed movements, like touching their hands, face, or feet, as displayed in Fig. 9. This makes motion capture even more challenging, as standard marker-based methods would not be recommended because of the risk of infants grabbing (and possibly swallowing) markers. Failure cases. The most common failure is a mixup of feet, i.e., left foot of the model registered to the right foot of the scan and vice versa. Despite our energy having the interpenetration penalty E sc , we observe a few cases where the legs interpenetrate, as in the bottom row in Fig. 8. The registration of all sequences is time consuming (between 10 and 30 seconds per frame) and rerunning the full 200K registrations many times to optimize the parameters is not feasible. The energy term weights are manually selected in order to balance the different terms, and by visually inspecting the results. Further manual adjustment of the E sc weight could fix these rare cases. In the example in the top row of Fig. 8, the right knee is twisted in an unnatural way after the right foot was completely occluded. When the foot is visible again, the pose recovers (knee twisted for 5-6 seconds). Similar to the first failure case, a higher weight on the pose prior would prevent such cases, but finding the perfect weight which completely forbids all illegal poses while allowing all legal poses would require a significant engineering effort.

Motion analysis case study. To show that SMIL captures enough motion information for medical assessment we conduct a case study on GMA. Two trained and certified GMA-experts perform GMA in different videos. We use five stimuli: i) the original RGB videos (denoted by V rgb ), and ii) the synthetic registration videos (V reg ). For the next three stimuli we use the acquired poses of infants, but we animate a body with a different shape, namely iii) a randomly selected shape of another infant (V other ), iv) an extreme shape producing a very thick and large baby (V large ), and v) the mean shape (V mean ). We exclude three of the 37 sequences, as two are too short and one has non-nutritive sucking, making it non suitable for GMA. As the number of videos to rate is high (34*5), for iv) and v) we only use 50% of the sequences, resulting in 136 videos. For a finer evaluation, we augment GMA classes definitely abnormal (DA), mildly abnormal (MA), normal suboptimal (NS), and normal optimal (NO) of [START_REF] Hadders-Algra | General movements: a window for early identification of children at high risk for developmental disorders[END_REF] into a one to ten scale. Scores 1-3 correspond to DA, 4-5 to MA, 6-7 to NS, and 8-10 to NO. We consider two ratings with an absolute difference ≤ 1 to agree, and otherwise to disagree.

Rater R 1 is a long-time GMA teacher and has worked on GMA for over [START_REF] Pishchulin | Building statistical shape spaces for 3D human modeling[END_REF] GMA. Average rating score (and standard deviation) for R1 is 4.7 (1.4), for R2 4.0 (1.9). The agreement on original RGB ratings V rgb between R 1 and R 2 is 65%. This further stresses that GMA is challenging and its automation important. In Fig. 10, we present rating differences between synthetic and reference sequences. Each rater is compared to her own V rgb ratings as a reference. R 1 V reg ratings agree on 91% of the reference ratings, whereas R 2 achieves an agreement rate of 79%. The agreement decreases more (R 2 ) or less (R 1 ) when the motions are presented with a different body shape. We intend to conduct further studies to elucidate the biases introduced by the variation of the infants' shape. Generation of realistic data. Human body models have been used to create training data for deep neural networks [START_REF] Varol | Learning from synthetic humans[END_REF]. We used SMIL to create a realistic (but privacy preserving) data set of Moving INfants In RGB-D 4 (MINI-RGBD) [START_REF] Hesse | Computer vision for medical infant motion analysis: State of the art and rgb-d data set[END_REF]. To create the data set, we captured shape and pose of infants from RGB-D sequences as described in Sec. 3.3, but additionally captured texture. We selected random subsets of shapes and textures, and averaged them to create new, synthetic, but realistic shapes and textures. We mapped the real captured poses to the new synthetic infants and extracted ground truth 3D joint positions. We used OpenDR [START_REF] Loper | OpenDR: An approximate differentiable renderer[END_REF] for rendering RGB and depth images to resemble commodity RGB-D sensors. We created the data set with the intention to provide an evaluation set for pose estimation in medical infant motion analysis scenarios. We plan to create a larger data set to provide enough data to train neural networks for infant shape and pose estimation.

DISCUSSION / CONCLUSION

Why does it work? Even though each RGB-D frame is a partial observation of the shape, the motions in the different frames of a sequence reveal previously hidden body parts. Moreover, even though the backs of the infants are rarely visible, we can still faithfully restrict the shape by taking into account the background table constraints. This lets us accumulate shape information over sequences. In addition, we leverage 2D pose estimation from RGB in two ways: i) it allows us to add landmark constraints to guide the model where depth-only approaches fail and ii) allows 4. Data set available at http://s.fhg.de/mini-rgbd us to find a good initialization frame and circumvent the need of predefined poses. Key to handling fast motion is a careful balancing of the different error terms. It has to be ensured that large pose changes are allowed, but at the same time, we have to make sure that body parts are not moved to distant scan points in the event of occlusions of corresponding scan parts or noisy segmentation.

Why has it not been done before? Most previous work has been done on adults, who can be instructed and 3D scanned much more easily. Recording infants poses more challenges. As our setup only relies on a low-cost RGB-D camera and a laptop, we can capture infants with great flexibility, for instance at a children's hospital that parents and children are visiting anyway. The creation of an initial model was not straightforward. Thanks to the flexibility of the SMPL model, we were able to adapt it to infants and get a starting point for capturing real infant shapes.

Limitations. Due to the data resolution, fine structures like finger and toe movements are not captured. Moreover, the model is not yet able to capture facial expressions. In our registration results, we found few cases of unusual neck twists. These occur in frames with side views when there is a mismatch between the computed initial head/neck shape and the head/neck shape in the scan. The model neck tries to match the scan shape by applying a small twist, making the neck look unnatural. By applying a stronger pose prior this could be forbidden, but might at the same time restrict the neck too much in other cases where strong rotations should be allowed. We will rather handle this by creating better initial shapes by sampling from the complete sequence instead of using 5 consecutive frames.

Conclusion. We contribute a method for learning a body model from RGB-D sequences of freely moving infants. We show that our learned Skinned Multi-Infant Linear model factorizes pose and shape and achieves metric accuracy of 2.5 mm. Although the metric difference to our initial model SMPL B is small, SMIL creates visually more realistic shapes. SMPL B is a miniaturized adult model that mimics infant appearance while SMIL is characteristic of a real infant population.

We applied our method to the task of medical infant motion analysis. Two expert GMA raters achieve a scoring agreement of 91% and 79% when comparing the GMA ratings from RGB video and from rendered SMIL registration results of the same sequence. Our method is a step towards a fully automated system for GMA, to help early-detect neurodevelopmental disorders like cerebral palsy.

Future work. We have not learned pose-dependent shape deformations for infants, and we reused the scaled down pose blend shapes of SMPL. While these provide sufficient numerical accuracy, we will learn infant pose blend shapes to further increase the realism of SMIL.

In an ongoing study, we are collecting more RGB-D data by taking advantage of the lightweight recording setup: we capture the infants' motions in their homes to minimize stress and effort for both infants and parents. We will further pursue the clinical goal to create SMIL: the automation of GMA, i.e., learn how to infer GMA ratings from captured motions. We are currently applying the system to infants affected by spinal muscular atrophy, with the goal to quantify the disease progress as well as the impact of therapy.

Fig. 1 .

 1 Fig. 1. (a) Simply scaling the SMPL adult body model and fitting it to an infant does not work as body proportions significantly differ. (b) The proposed SMIL model properly captures the infants' shape and pose.

Fig. 3 .

 3 Fig. 3. a) Original RGB image. b) Skin weights used for weighted PCA. White denotes high weight, red denotes low weight.

Fig. 4 .

 4 Fig. 4. SMIL registration results. From left to right: RGB, point cloud, point cloud (other view), point cloud with registered SMIL, rendered registration.

Fig. 5 .

 5 Fig. 5. Visual comparison of four SMPL B (left sides) and SMIL (right sides) registration samples with all 20 shape components. Most visually significant differences occur at waist, chest, shoulders and head.

Fig. 6 .

 6 Fig.6. Average scan-to-mesh error E s2m in mm w.r.t. the number of shape parameters for the two models registered to all fusion scans.

Fig. 7 .

 7 Fig. 7. Average error heatmaps for SMIL and SMPL B on fusion clouds for different numbers of shape components (sc). Top: SMIL. Bottom: SMPL B . Blue means 0 mm, red means ≥ 10 mm.

Fig. 8 .

 8 Fig. 8. Registration errors. Top: Unnatural pose sample. Bottom: Failure case sample. From left to right: RGB image, 3D point cloud (rotated for improved visibility), overlay of point cloud and registration result, registration result, rendered result from same viewpoint as RGB image.

Fig. 9 .

 9 Fig. 9. Older infant in very challenging pose. (a) RGB image, (b) 3D point cloud (rotated for improved visibility), (c) overlay of point cloud and SMIL registration result, (d) rendered SMIL registration result.

  with Π denoting the table. δ in (x) is an indicator function, returning 1 if x lies inside the table (behind the estimated table plane), or 0 otherwise and δ close (x) is an indicator function, returning 1 if x is close to the table (distance less than 3 cm) and faces away from the camera, or 0 otherwise.To account for soft tissue deformations of the back, which are not modeled, we allow the model to virtually penetrate the table. We effectively enforce this by translating the table plane by 0.5 cm, i.e., pushing the virtual table back.

  years, R 2 has 15 years experience in Fig.10. Results of GMA case study. Percentage of ratings of synthetic sequences, generated using SMIL, that agree with the reference ratings R 1 V rgb (left) and R 2 V rgb (right), respectively. V {reg,other,large,mean} denotes different stimuli.

	100%	Agreement with V RGB
	80%		a V reg
	60%		b V other
	40%		c V large
	20%		d V mean
	0%
		R1	R2

ACKNOWLEDGMENTS

Authors thank Javier Romero for helpful discussions, Mijna Hadders-Algra and Uta Tacke for GMA ratings and medical insights, R. Weinberger for data acquisition and help with evaluation and Paul Soubiran for the video voice over.

Disclosure: MJB has received research gift funds from Intel, Nvidia, Adobe, Facebook, and Amazon. While MJB is a part-time employee of Amazon, his contribution was performed solely at, and funded solely by, MPI.