Yaohui Wang
email: yaohui.wang@inria.fr

Di Yang
email: di.yang@inria.fr

Francois Bremond
email: francois.bremond@inria.fr

Antitza Dantcheva
email: antitza.dantcheva@inria.fr

LATENT IMAGE ANIMATOR: LEARNING TO ANIMATE IMAGES VIA LATENT SPACE NAVIGATION

Due to the remarkable progress of deep generative models, animating images has become increasingly efficient, whereas associated results have become increasingly realistic. Current animation-approaches commonly exploit structure representation extracted from driving videos. Such structure representation is instrumental in transferring motion from driving videos to still images. However, such approaches fail in case the source image and driving video encompass large appearance variation. Moreover, the extraction of structure information requires additional modules that endow the animation-model with increased complexity. Deviating from such models, we here introduce the Latent Image Animator (LIA), a self-supervised autoencoder that evades need for structure representation. LIA is streamlined to animate images by linear navigation in the latent space. Specifically, motion in generated video is constructed by linear displacement of codes in the latent space. Towards this, we learn a set of orthogonal motion directions simultaneously, and use their linear combination, in order to represent any displacement in the latent space. Extensive quantitative and qualitative analysis suggests that our model systematically and significantly outperforms state-of-art methods on VoxCeleb, Taichi and TED-talk datasets w.r.t. generated quality.

 [START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF]onto the still images. LIA is able to successfully animate these two images without relying on any explicit structure representations, such as landmarks and region representations.

INTRODUCTION

In the series of science fiction books Harry Potter [START_REF] Kathleen | Harry Potter and the Cursed Child-Parts One and Two (Special Rehearsal Edition)[END_REF][START_REF] Jk Rowling | The 100 Greatest Literary Characters[END_REF], wizards and witches were able to magically enchant portraits, bringing them to life. Remarkable progress of deep generative models has recently turned this vision into reality. This work examines the scenario where a framework animates a source image by motion representations learned from a driving video. Existing approaches for image animation are classically related to computer graphics [START_REF] Cao | Displaced dynamic expression regression for real-time facial tracking and animation[END_REF][START_REF] Thies | Face2face: Real-time face capture and reenactment of RGB videos[END_REF][START_REF] Jk Rowling | The 100 Greatest Literary Characters[END_REF][START_REF] Zhao | Learning to forecast and refine residual motion for image-to-video generation[END_REF] or exploit motion labels (Wang et al., 2020b) and structure representations such as semantic maps [START_REF] Pan | Video generation from single semantic label map[END_REF]Wang et al., 2018;[START_REF] Jk Rowling | The 100 Greatest Literary Characters[END_REF], human keypoints [START_REF] Jang | Video Prediction with Appearance and Motion Conditions[END_REF][START_REF] Yang | Pose guided human video generation[END_REF][START_REF] Walker | The pose knows: Video forecasting by generating pose futures[END_REF][START_REF] Chan | Everybody dance now[END_REF][START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF]Wang et al., 2019;[START_REF] Siarohin | First order motion model for image animation[END_REF], 3D meshes [START_REF] Liu | Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis[END_REF][START_REF] Hao Chen | Joint generative and contrastive learning for unsupervised person re-identification[END_REF], and optical flows (Li et al., 2018;[START_REF] Ohnishi | Hierarchical video generation from orthogonal information: Optical flow and texture[END_REF]. We note that the ground truth of such structure representations has been computed a-priori for the purpose of supervised training, which poses constraints on applications, where such representations of unseen testing images might be fragmentary or difficult to access.

Self-supervised motion transfer approaches [START_REF] Wiles | X2face: A network for controlling face generation using images, audio, and pose codes[END_REF][START_REF] Siarohin | First order motion model for image animation[END_REF][START_REF] Wang | Learning to Generate Human Videos[END_REF] accept raw videos as input and learn to reconstruct driving images by warping source image with predicted dense optical flow fields. While the need for domain knowledge or labeled ground truth data has been obviated, which improves performance on in-the-wild testing images, such methods entail necessity of explicit structure representations as motion guidance. Prior information such as keypoints [START_REF] Siarohin | First order motion model for image animation[END_REF]Wang et al., 2021a) or regions [START_REF] Siarohin | Motion representations for articulated animation[END_REF] are learned in an end-to-end training manner by additional networks as intermediate features, in order to predict target flow fields. Although online prediction of such representations is less tedious than the acquisition of ground truth labels, it still strains the complexity of networks.

Deviating from such approaches, we here aim to fully eliminate the need of explicit structure representations by directly manipulating the latent space of a deep generative model. To the best of our knowledge, this constitutes a new direction in the context of image animation. Our work is motivated by interpretation of GANs [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF][START_REF] Goetschalckx | Ganalyze: Toward visual definitions of cognitive image properties[END_REF][START_REF] Jahanian | On the "steerability" of generative adversarial networks[END_REF][START_REF] Voynov | Unsupervised discovery of interpretable directions in the gan latent space[END_REF], showcasing that latent spaces of StyleGAN [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]2020b) and BigGAN [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF] contain rich semantically meaningful directions. Given that walking along such directions, basic visual transformations such as zooming and rotation can be induced in generated results. As in image animation, we have that motion between source and driving images can be considered as higher-level transformation, a natural question here arises: can we discover a set of directions in the latent space that induces high-level motion transformations collaboratively?

Towards answering this question, we introduce LIA, a novel Latent Image Animator constituting of an autoencoder for animating still images via latent space navigation. LIA seeks to animate a source image via linearly navigating associated source latent code along a learned path to reach a target latent code, which represents the high-level transformation for animating the source image. We introduce a Linear Motion Decomposition (LMD) approach aiming to represent a latent path via a linear combination of a set of learned motion directions and associated magnitudes. Specifically, we constrain the set as an orthogonal basis, where each vector indicates a basic visual transformation. By describing the whole motion space using such learned basis, LIA eliminates the requirement of explicit structure representations.

In addition, we design LIA to disentangle motion and appearance within a single encoder-generator architecture. Deviating from existing methods using separate networks to learn disentangled features, LIA integrates both, latent motion code, as well as appearance features in a single encoder, which highly reduces the model complexity and simplifies training.

We provide evaluation on multiple datasets including VoxCeleb [START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF], TaichiHD [START_REF] Siarohin | First order motion model for image animation[END_REF] and TED-talk [START_REF] Siarohin | Motion representations for articulated animation[END_REF]. In addition, we show that LIA outperforms the state-of-the-art in preserving the facial structure in generated videos in the setting of one-shot image animation on unseen datasets such as FFHQ [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] and GermanPublicTV [START_REF] Thies | Neural voice puppetry: Audio-driven facial reenactment[END_REF].

RELATED WORK

Video generation GAN-based video generation is aimed at mapping Gaussian noise to video, directly and in the absence of prior information [START_REF] Vondrick | Generating videos with scene dynamics[END_REF][START_REF] Saito | Temporal generative adversarial nets with singular value clipping[END_REF][START_REF] Tulyakov | MoCoGAN: Decomposing motion and content for video generation[END_REF]Wang et al., 2020a;[START_REF] Wang | Learning to Generate Human Videos[END_REF]. Approaches based on deep probabilistic models (Den-Figure 2: General pipeline. Our objective is to transfer motion via latent space navigation. The entire training pipeline consists of two steps. Firstly, we encode a source image x s into a latent code z s→r . By linearly navigating z s→r along a path w r→d , we reach a target latent code z s→d . The latent paths are represented by a linear combination between a set of learned motion directions (e.g., d 1 and d 2 ), which is an orthogonal basis, and associated magnitudes. In the second step, we decode z s→d to a target dense optical flow field φ s→d , which is used to warp x s into the driving image x d . While we train our model using images from the same video sequence, in the testing phase, x s and x d generally pertain to different identities.

ton & Birodkar, 2017; [START_REF] Li | Disentangled sequential autoencoder[END_REF][START_REF] Bhagat | Disentangling multiple features in video sequences using gaussian processes in variational autoencoders[END_REF][START_REF] Xie | Motion-based generator model: Unsupervised disentanglement of appearance, trackable and intrackable motions in dynamic patterns[END_REF] were also proposed to tackle this problem, however only show results on toy datasets with low resolution. Recently, with the progress of GANs in photo-realistic image generation [START_REF] Brock | Large scale GAN training for high fidelity natural image synthesis[END_REF][START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]2020a), a series of works [START_REF] Clark | Adversarial video generation on complex datasets[END_REF]Wang et al., 2021c) explored production of highresolution videos by incorporating the architecture of an image generator into video GANs, trained jointly with RNNs. [START_REF] Tian | A good image generator is what you need for high-resolution video synthesis[END_REF] directly leveraged the knowledge of a pre-trained StyleGAN to produce videos of resolution up to 1024×1024. Unlike these approaches, which generate random videos based on noise vectors in an unconditional manner, in this paper, we focus on conditionally creating novel videos by transferring motion from driving videos to input images.

Latent space editing

In an effort to control generated images, recent works explored the discovery of semantically meaningful directions in the latent space of pre-trained GANs, where linear navigation corresponds to desired image manipulation. Supervised [START_REF] Shen | Interpreting the latent space of gans for semantic face editing[END_REF][START_REF] Jahanian | On the "steerability" of generative adversarial networks[END_REF][START_REF] Goetschalckx | Ganalyze: Toward visual definitions of cognitive image properties[END_REF] and unsupervised [START_REF] Voynov | Unsupervised discovery of interpretable directions in the gan latent space[END_REF][START_REF] Peebles | The hessian penalty: A weak prior for unsupervised disentanglement[END_REF][START_REF] Shen | Closed-form factorization of latent semantics in gans[END_REF] approaches were proposed to edit semantics such as facial attributes, colors and basic visual transformations (e.g., rotation and zooming) in generated or inverted real images [START_REF] Yang | Transmomo: Invariance-driven unsupervised video motion retargeting[END_REF][START_REF] Abdal | Image2stylegan++: How to edit the embedded images[END_REF]. In this work, as opposed to finding directions corresponding to individual visual transformations, we seek to learn a set of directions that cooperatively allows for high-level visual transformations that can be beneficial in image animation.

Image animation Related approaches [START_REF] Chan | Everybody dance now[END_REF]Wang et al., 2018;[START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF]Wang et al., 2019;[START_REF] Yang | Transmomo: Invariance-driven unsupervised video motion retargeting[END_REF] in image animation required strong prior structure labels as motion guidance. In particular, [START_REF] Chan | Everybody dance now[END_REF], [START_REF] Yang | Transmomo: Invariance-driven unsupervised video motion retargeting[END_REF] and Wang et al. (2018) proposed to map representations such as human keypoints and facial landmarks to videos in the setting of image-to-image translation proposed by [START_REF] Isola | Image-to-Image Translation with Conditional Adversarial Networks[END_REF]. However, such approaches were only able to learn an individual model for a single identity. Transferring motion on new appearances requires retraining the entire model from scratch by using videos of target identities. Several recent works [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF]Wang et al., 2019) explored meta learning in fine-tuning models on target identities. While only few images of target identities were required during inference time, it was still compulsory to input pre-computed structure representations in those approaches, which usually are hard to access in many real-world scenarios. Towards addressing this issue, very recent works [START_REF] Siarohin | First order motion model for image animation[END_REF][START_REF] Wang | Learning to Generate Human Videos[END_REF]Wang et al., 2021b;[START_REF] Wiles | X2face: A network for controlling face generation using images, audio, and pose codes[END_REF] proposed to learn image animation in self-supervised manner, only relying on RGB videos for both, training and testing without any priors. They firstly predicted dense flow fields from input images, which were then utilized to warp source images, in order to obtain final generated results. Inference only required one image of a target identity without any fine-tuning step on pre-trained models. While no priors were required, state-of-the-art methods still followed the idea of using explicit structure representations. FOMM [START_REF] Siarohin | First order motion model for image animation[END_REF] proposed a first order motion approach to predict keypoints and local transformations online to generate flow fields. [START_REF] Siarohin | Motion representations for articulated animation[END_REF] developed this idea to model articulated objects by replacing a keypoints predictor by a PCA-based region prediction module. Wang et al. (2021b) extended FOMM by predicting 3D keypoints for view-free generation. We Figure 3: Overview of LIA. LIA is an autoencoder consisting of two networks, an encoder E and a generator G. In the latent space, we apply Linear Motion Decomposition (LMD) towards learning a motion dictionary D m , which is an orthogonal basis where each vector represents a basic visual transformation. LIA takes two frames sampled from the same video sequence as source image x s and driving image x d respectively during training. Firstly, it encodes x s into a source latent code z s→r and x d into a magnitude vector A r→d . Then, it linearly combines A r→d and a trainable D m using LMD to obtain a latent path w r→d , which is used to navigate z s→r to a target code z s→d . Finally, G decodes z s→d into a target dense flow field and warps x s to an output image x s→d . The training objective is to reconstruct x d using x s→d . note though that in all approaches, given that keypoints or regions are inadequately predicted, the quality of generated images drastically decreases. In contrast to such approaches, our method does not require any explicit structure representations. We dive into the latent space of the generator and self-learn to navigate motion codes in certain directions with the goal to reach target codes, which are then decoded to flow fields for warping.

METHOD

Self-supervised image animation aims at learning to transfer motion from a subject of a driving video to a subject in a source image based on training with a large video dataset. In this work, we propose to model such motion transformation via latent space navigation. The general pipeline is illustrated in Fig. 2. Specifically, for training, our model takes in a pair of source and driving images, randomly sampled from one video sequence. These two images are encoded into a latent code which is used to represent motion transformation in the image space. The training objective is to reconstruct the driving image by combining source image with learned motion transformation. For testing, frames of a driving video are sequentially processed with the source image to animate the source subject.

We provide an overview of the proposed model in Fig. 3. Our model is an autoencoder, consisting of two main networks, an encoder E and a generator G. In general, our model requires two steps to transfer motion. In the first step, E encodes source and driving images x s , x d ∼ X ∈ R 3×H×W into latent codes in the latent space. The source code is then navigated into a target code, which is used to represent target motion transformation, along a learned latent path. Based on proposed Linear Motion Decomposition (LMD), we represent such a path as a linear combination of a set of learned motion directions and associated magnitudes, which are learned from x d . In the second step, once the target latent code is obtained, G decodes it as a dense flow field φ s→d ∼ Φ ∈ R 2×H×W and uses φ s→d to warp x s and then to obtain the output image. In the following, we proceed to discuss the two steps in detail.

LATENT MOTION REPRESENTATION

Given a source image x s and a driving image x d , our first step constitutes of learning a latent code z s→d ∼ Z ∈ R N to represent the motion transformation from x s to x d . Due to the uncertainty of two images, directly learning z s→d puts forward a high requirement on the model to capture a complex distribution of motion. Mathematically, it requires modeling directions and norms of the vector z s→d simultaneously, which is challenging. Therefore, instead of modeling motion transformation x s → x d , we assume there exists a reference image x r and motion transfer can be modeled as x s → x r → x d , where z s→d is learned in an indirect manner. We model z s→d as a target point in the latent space, which can be reached by taking linear walks from a starting point z s→r along a linear path w r→d (see Fig. 2), given by z s→d = z s→r + w r→d , (1) where z s→r and w r→d indicate the transformation x s → x r and x r → x d respectively. Both z s→r and w r→d are learned independently and z s→r is obtained by passing x s through E.

We learn w r→d via Linear Motion Decomposition (LMD). Our idea is to learn a set of motion directions D m = {d 1 , ..., d M } to represent any path in the latent space. We constrain D m as an orthogonal basis, where each vector indicates a motion direction d i . We then combine each vector in the basis with a vector A r→d = {a 1 , ..., a M }, where a i represents the magnitude of d i . Hence, any linear path in the latent space can be represented using a linear combination

w r→d = M i=1 a i d i , (2)
where

d i ∈ R N and a i ∈ R
< d i , d j >= 0 i = j 1 i = j. (3)
We implement D m as a learnable matrix and apply the Gram-Schmidt process during each forward pass, in order to meet the requirement of orthogonality. A r→d is obtained by mapping z d→r , which is the output of x d after E, through a 5-layer MLP. The final formulation of latent motion representation for each x s and x d is thus given as

z s→d = z s→r + M i=1 a i d i . (4)

LATENT CODE DRIVEN IMAGE ANIMATION

Once we obtain z s→d , in our second step, we use G to decode a flow field φ s→d and warp x s . Our G consists of two components, a flow field generator G f and a refinement network G r (we provide details in App. A).

Towards learning multi-scale features, G is designed as a residual network containing N models to produce a pyramid of flow fields φ s→d = {φ i } N 1 in different layers of G f . Multi-scale source features x enc s = {x enc i } N 1 are obtained from E and are warped in G f . However, as pointed out by [START_REF] Siarohin | First order motion model for image animation[END_REF], only relying on φ s→d to warp source features is insufficient to precisely reconstruct driving images due to the existing occlusions in some positions of x s . In order to predict pixels in those positions, the network is required to inpaint the warped feature maps. Therefore, we predict multi-scale masks {m i } N 1 along with {φ i } N 1 in G f to mask out the regions required to be inpainted. In each residual module, we have

x i = T (φ i , x enc i ) m i , (5)
where denotes the Hadamard product and T denotes warping operation, whereas x i signifies the masked features. We generate both dense flow fields, as well as masks by letting each residual module output a 3-channel feature map in which the first two channels represent φ i and the last channel m i . Based on an inpainted feature map f (x i ), as well as an upsampled image g(x i-1 ) provided by the previous module in G r , the RGB image from each module is given by

o i = f (x i ) + g(o i-1 ), (6)
where f and g denote the inpainting and upsampling layers, respectively. The output image o N from the last module constitutes the final generated image x s→d = o N .

LEARNING

We train LIA in a self-supervised manner to reconstruct x d using three losses, i.e., a reconstruction loss L recon , a perceptual loss L vgg [START_REF] Johnson | Perceptual losses for real-time style transfer and super-resolution[END_REF] and an adversarial loss L adv . We use L recon to minimize the pixel-wise L 1 distance between x d and x s→d , calculated as

L recon (x s→d , x d ) = E[ x d -x s→d 1 ]. (7)
Towards minimizing the perceptual distance, we apply a VGG19-based L vgg on multi-scale feature maps between real and generated images, written as

L vgg (x s→d , x d ) = E[ N n F n (x d ) -F n (x s→d ) 1 ], (8)
where F n denotes the n th layer in a pre-trained VGG19 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. In practice, towards penalizing real and generated images in multi-scale images, we use a pyramid of four resolutions, namely 256 × 256, 128 × 128, 64 × 64 and 32 × 32 as inputs of VGG19. The final perceptual loss is the addition of perceptual losses in four resolutions.

Further, towards generating photo-realistic results, we incorporate a non-saturating adversarial loss L adv on x s→d , which is calculated as

L adv (x s→d ) = E x s→d ∼prec [-log(D(x s→d )], (9)
where D is a discriminator, aimed at distinguishing reconstructed images from the original ones. Our full loss function is the combination of three losses with λ as a balanced hyperparameter

L(x s→d , x d ) = L recon (x s→d , x d ) + λL vgg (x s→d , x d ) + L adv (x s→d ). (10)

INFERENCE

In inference stage, given a driving video sequence V d = {x t } T 1 , we aim to transfer motion from V d to x s , in order to generate a novel video V d→s = {x t→s } T 1 . If V d and x s stem from the same video sequence, i.e., x s = x 1 , our task comprises of reconstructing the entire original video sequence. Therefore, we construct the latent motion representation of each frame using absolute transfer, which follows the training process, given as

z s→t = z s→r + w r→t , t ∈ {1, ..., T }. (11)
However, in real world applications, interest is rather placed on the scenario, where motion transfer between x s and V d , the latter stemming from different identities, i.e., x s = x 1 . Taking a talking head video as an example, in this setting, beyond identity, x 1 and x s might also differ in pose and expression. Therefore, we propose relative transfer to eliminate the motion impact of w r→1 and involve motion of w r→s in the full generated video sequence. Owing to a linear representation of the latent path, we can easily represent z s→t for each frame as z s→t = (z s→r + w r→s ) + (w r→t -w r→1 ) = z s→s + (w r→t -w r→1 ), t ∈ {1, ..., T }.

The first term in Eq. ( 12), z s→s indicates the reconstruction of x s , while the second term (w r→t -w r→1 ) represents the motion from x 1 to x t . This equation indicates that the original pose is preserved in x s , at the same time motion is transferred from V d . We note that in order to completely replicate the position and pose in V d , it requires x s and x 1 to contain similar poses in relative motion transfer.

EXPERIMENTS

In this section, we firstly describe our experimental setup including implementation details and datasets. Secondly, we qualitatively demonstrate generated results based on testing datasets. Then, we provide quantitative evaluation w.r.t. image quality on (a) same-identity reconstruction, (b) crossvideo motion transfer, presenting (c) a user study. Next, we conduct an ablation study that demonstrates (d) the effectiveness of our proposed motion dictionary, as well as (e) associated size. Finally, we provide an in-depth analysis of our (f) latent codes and (g) motion dictionary to interpret their semantic meanings.

Datasets Our model is trained on the datasets VoxCeleb, TaichiHD and TED-talk. We follow the pre-processing method in [START_REF] Siarohin | First order motion model for image animation[END_REF] to crop frames into 256 × 256 resolution for quantitative evaluation.

Implementation details Our model is implemented in PyTorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF]. All models are trained on four 16G NVIDIA V100 GPUs. The total batch size is 32 with 8 images per GPU. We use a learning rate of 0.002 to train our model with the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF].

The dimension of all latent codes, as well as directions in D m is set to be 512. In our loss function, we use λ = 10 in order to penalize more on the perceptual loss. It takes around 150 hours to fully train our framework. Evaluation metrics We evaluate our model w.r.t. (i) reconstruction faithfulness using L 1 , LPIPS, (ii) generated video quality using video FID, as well as (iii) semantic consistency using average keypoint distance (AKD), missing keypoint rate (MKR) and average euclidean distance (AED). Details are available in App. B.2.

QUALITATIVE RESULTS

Firstly, we evaluate the ability of LIA to generate realistic videos and compare related results with four state-of-the-art methods. For TaichiHD and TED-talk datasets, we conduct an experiment related to cross-video generation. Corresponding results (see Fig. 4) confirm that our method is able to correctly transfer motion on articulated human bodies, in the absence of explicit structure representations. For the VoxCeleb dataset, we conduct a cross-dataset generation-experiment, where we transfer motion from VoxCeleb to images of the FFHQ dataset. We observe that our method outperforms FOMM and MRAA w.r.t. image quality, as both approaches visibly deform the shape of the original faces. This is specifically notable in the case that source and driving images entail large pose variations. At the same time, LIA is able to successfully tackle this challenge and no similar deformations are visible.

COMPARISON WITH STATE-OF-THE-ART METHODS

We quantitatively compare our method with the state-of-the-art approaches X2Face, Monkey-Net, FOMM and MRAA on two tasks, namely (a) same-identity reconstruction and (b) cross-video motion transfer. Additionally, we conduct a (c) user study.

(a) Same-identity reconstruction We here evaluate the reconstruction ability of our method. Specifically, we reconstruct each testing video by using the first frame as x s and the remaining frames as x d . Results on three datasets are reported in Table 1. Focusing on foregroundreconstruction, our method outperforms the other approaches w.r.t. all metrics. More results are presented in App. B.3, discussing background-reconstruction.

(b) Cross-video motion transfer Next, we conduct experiments, where source images and driving videos stem from different video sequences. In this context, we mainly focus on evaluating talking head videos and explore two different cases. In the first case, we generate videos using the VoxCeleb testing set to conduct motion transfer. In the second case, source images are from an unseen dataset, namely the GermanPublicTV dataset, as we conduct cross-dataset motion transfer.

In both experiments, we randomly construct source and driving pairs and transfer motion from driving videos to source images to generate a novel manipulated dataset. Since ground truth data for our generated videos is not available, we use video FID (as initialized by Wang et al. (2020a)) to (c) User study We conduct a user study to evaluate video quality. Towards this, we displayed paired videos and asked 20 human raters 'which clip is more realistic?'. Each video-pair contains a generated video from our method, as well as a video generated from FOMM or MRAA. Results suggest that our results are more realistic in comparison to FOMM and MRAA across all three datasets (see Tab. 3). Hence, the obtained human preference is in accordance with our quantitative evaluation.

Table 4: Ablation study on motion dictionary.

We conduct experiments on three datasets with and without Dm and show reconstruction results.

VoxCeleb TaichiHD TED-talks Method L 1 LPIPS L 1 LPIPS L 1 LPIPS w/o D m 0.049 0.165 0.062 0.186 0.031 0.12 Full 0.041 0.123 0.057 0.180 0.028 0.11

Table 5: Ablation study on Dm size. We conduct experiments on three datasets with 5 different Dm size and show reconstruction results.

VoxCeleb TaichiHD TED-talks M L 1 LPIPS L 1 LPIPS L 1 LPIPS 5 0.051 0.15 0.070 0.22 0.037 0.15 10 0.043 0.13 0.065 0.20 0.036 0.13 20 0.041 0.12 0.057 0.18 0.028 0.11 40 0.042 0.12 0.060 0.19 0.030 0.12 100 0.041 0.12 0.058 0.18 0.028 0.11

ABLATION STUDY

We here analyze our proposed motion dictionary and focus on answering following two questions.

(d) Is the motion dictionary D m beneficial? We here explore the impact of proposed D m , by training our model without D m . Specifically, we output w r→d directly from MLP, without using LMD to learn an orthogonal basis. From the evaluation results reported in Tab. 4 and qualitative results in App. B.5, we observe that in the absence of D m , model fails to generate high-quality images, which proves the effectiveness of D m , consistently on all datasets.

(e) How many directions are required in D m ? Towards finding an effective size of D m , we empirically test three different M , viz. 5, 10, 20, 40 and 100. Quantitative results in Tab. 5 show that when using 20 directions, the model achieves the best reconstruction results.

FURTHER ANALYSIS

(f) Latent code analysis. While our method successfully transfers motion via latent space navigation, we here aim at answering the question -what does x r represent? Towards answering this question, we proceed to visualize x r . We firstly decode z s→r into a dense flow field φ s→r , which is then used to warp x s (we show details in App. B.4). Fig. 5 shows examples of x s and x r . Interestingly, we observe that x r represents the canonical pose of x s , regardless of original poses of the subjects. And for all datasets, reference images resemble each other w.r.t. pose and scale. As such reference images can be considered as a normalized form of x s , learning transformations between x s and x d using x s → x r → x d is considerably more efficient than x s → x d , once x r is fixed. Noteworthy, we found the similar idea of learning a 'reference image' has also been explored by [START_REF] Siarohin | First order motion model for image animation[END_REF] (FOMM) and Wiles et al. (2018) (X2Face). However, deviating from our visualized 'reference image', the 'reference image 'in FOMM refers to a non-visualized and abstract concept. In addition, LIA only requires a latent code z s→r , rather than the 'reference image' for both, training and testing, which is contrast to X2Face.

(g) Motion dictionary interpretation. Towards further interpretation of directions in D m , we conduct linear manipulations on each d i . Images pertained to manipulating four motion directions are depicted in Fig. 6. The results suggest that the directions in D m are semantically meaningful, as they represent basic visual transformations such as head nodding (d 8 ), eye movement (d 6 ) and facial expressions (d 19 and d 7 ). More results can be found on our project webpage2 .

CONCLUSIONS

In this paper, we presented a novel self-supervised autoencoder LIA, aimed at animating images via latent space navigation. By the proposed Linear Motion Decomposition (LMD), we were able to formulate the task of transferring motion from driving videos to source images as learning linear transformations in the latent space. We evaluated proposed method on real-world videos and demonstrated that our approach is able to successfully animate still images, while eliminating the necessity of explicit structure representations. In addition, we showed that the incorporated motion dictionary is interpretable and contains directions pertaining to basic visual transformations. Both quantitative and qualitative evaluations showed that LIA outperforms state-of-art algorithms on all benchmarks. We postulate that LIA opens a new door in design of interpretable generative models for video generation.

ETHIC STATEMENT

In this work, we aim to synthesize high-quality videos by transferring motion on still images. Our approach can be used for movie production, making video games, online education, generating synthetic data for other computer vision tasks, etc. We note that our framework mainly focuses on learning how to model motion distribution rather than directly model appearance, therefore it is not biased towards any specific gender, race, region, or social class. It works equally well irrespective of the difference in subjects.

REPRODUCIBILITY STATEMENT

We assure that all the results shown in the paper and supplemental materials can be reproduced. We intend to open-source our code, as well as trained models.

B.2 EVALUATION METRICS

We use five different metrics to evaluate our experimental results, namely L 1 , LPIPS, AKD, MKR and AED that quantify the reconstructed results. In addition, we compute video FID to evaluate video quality in motion transferring tasks.

L 1 represents the mean absolute pixel difference between reconstructed and real videos.

LPIPS [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF] aims at measuring the perceptual similarity between reconstructed and real images by leveraging the deep features from AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

Video FID is a modified version of the original FID [START_REF] Heusel | Gans trained by a two time-scale update rule converge to a local nash equilibrium[END_REF]. We here follow the same implementation as Wang et al. (2020a) and utilize a pre-trained ResNext101 [START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF] to extract spatio-temporal features to compute the distance between real and generated videos distributions. We take the first 100 frames of each video as input of the feature-extractor to compute the final scores.

Average keypoint distance (AKD) and missing keypoint rate (MKR) evaluate the difference between keypoints of reconstructed and ground truth videos. We extract landmarks using the face alignment approach of [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF] and extract body poses for both TaiChiHD and TED-talks using OpenPose [START_REF] Cao | Openpose: Realtime multi-person 2d pose estimation using part affinity fields[END_REF]. AKD is computed as the average distance between corresponding keypoints, whereas MKR is the proportion of keypoints present in the groundtruth that are missing in a reconstructed video.

Average Euclidean distance (AED) measures the ability of preserving identity in reconstructed video. We use a person re-identification pretrained model [START_REF] Zheng | Vehiclenet: Learning robust visual representation for vehicle re-identification[END_REF] for measuring human bodies (TaichiHD and TED-talk) and OpenFace [START_REF] Amos | Openface: A general-purpose face recognition library with mobile applications[END_REF] for faces to extract identity embeddings from reconstructed and ground truth frame pairs, then we compute MSE of their difference for all pairs.

B.3 COMPARISON WITH FULL MRAA

We show quantitative evaluation results with the full MRAA model in Tab. 6. We observe that our method achieves competitive results in reconstruction and keypoint evaluation. While we do not explicitly predict keypoints, w.r.t. the TaichiHD dataset, interestingly we outperform MRAA in both, AKD and MKR. Such results showcase the effectiveness of our proposed method on modeling articulated human structures. However, reconstruction evaluation cannot provide a completely fair comparison on how well the main subjects (e.g., faces and human bodies) are generated in videos. This is in particular the case for TaichiHD and TED-talk, where backgrounds have large contributions to the final scores. To produce x r , we use G to decode z s→r into the flow field φ s→r . Reference image x r is obtained by warping x s using φ s→r . The entire process is shown in Fig. 9. Fig. 10 illustrates the generated results on transferring motion from VoxCeleb to GermanPublicTV with and without motion dictionary. We observe that without the motion dictionary, appearance information is undesirably transferred from driving videos to generated videos.

B.6 LIMITATIONS

For human body, one limitation of our method is dealing with body occlusion. We observe in Fig. 11 that in taichi videos, in case of occlusion cause by legs and arms, motion is not transferred successfully. In addition, in TED-talks, transferring hand motion is challenging, as hands are of small size, articulated and sometimes occluded by human bodies.

Figure 10: Generated results with and without D m . We observe that the disentanglement of appearance and motion is much better by using D m .

Figure 11: Failure cases. We observed that it is still challenging for LIA to handle arm-leg occlusion (Taichi) and hand motion (TED-talk).

Figure 1 :

 1 Figure 1: LIA animation examples. The two images of Marilyn Monroe and Emmanuel Macron are animated by LIA, which transfers motion of a driving video (smaller images on the top) from VoxCeleb dataset (Chung et al., 2018) onto the still images. LIA is able to successfully animate these two images without relying on any explicit structure representations, such as landmarks and region representations.

Figure 4 :

 4 Figure 4: Qualitative results. Examples for same-dataset absolute motion transfer on TaichiHD (top-right) and TED-talk (bottom-right). On VoxCeleb (left), we demonstrate cross-dataset relative motion transfer. We successfully transfer motion between x 1 and x t from videos in VoxCeleb to x s from FFHQ, the latter not being used for training.

Figure 5 :

 5 Figure 5: Visualization of reference images. Example source (top) and reference images (down) from VoxCeleb, TaichiHD and TED-talk datasets. Our network learns reference images of a consistently frontal pose, systematically for all input images of each dataset.

Figure 6 :

 6 Figure 6: Linear manipulation of four motion directions on the painting of Mona Lisa. Manipulated results indicate that d 6 represents eye movement, d 8 represents head nodding, whereas d 19 and d 7 represent facial expressions.

Figure 8 :

 8 Figure 8: Generator architecture. We show details about architecture of G in (a) and G block in (b).TaiChiHD[START_REF] Siarohin | First order motion model for image animation[END_REF] consists of videos of full human bodies performing Tai Chi actions. We follow the original pre-processing of FOMM[START_REF] Siarohin | First order motion model for image animation[END_REF] and utilize its 256 × 256 version. TaiChiHD contains 1096 training videos and 115 testing videos. TED-talk is a new dataset proposed in MRAA (Siarohin et al., 2021). It comprises a number of TED-talk videos, where the main subjects have been cropped out. We resize the original version into 256 × 256 resolution to train our model. This dataset includes 1124 training videos and 130 testing videos.

  Method L 1 AKD AED LPIPS L 1 (AKD, MKR) AED LPIPS L 1 (AKD, MKR) AED LPIPS MRAA 0.041 1.303 0.135 0.124 0.045 (5.551, 0.025) 0.431 0.178 0.027 (3.107, 0.0093) 0.379 0.11 Ours 0.041 1.353 0.138 0.123 0.057 (4.823, 0.020) 0.431 0.180 0.027 (3.141, 0.0095) 0.399 0.11 B.4 REFERENCE IMAGE GENERATION.

Figure 9 :

 9 Figure 9: Reference image generation.

  for all i ∈ {1, ..., M }. Semantically, each d i should represent a basic visual transformation and a i indicates the required steps to walk in d i towards achieving w r→d . Due to D m entailing an orthogonal basis, any two directions d i , d j follow the constrain

Table 1 :

 1 Same-identity reconstruction. Comparison with state-of-the-art methods on three datasets for same-identity reconstruction.

		VoxCeleb			TaichiHD				TED-talks
	Method	L 1 AKD AED LPIPS	L 1	(AKD, MKR) AED LPIPS	L 1	(AKD, MKR) AED LPIPS
	X2Face	0.078 7.687 0.405	-	0.080 (17.654, 0.109)	-	-	-	-	-	-
	Monkey-Net 0.049 1.878 0.199	-	0.077 (10.798, 0.059)	-	-	-	-	-	-
	FOMM	0.046 1.395 0.141 0.136	0.063 (6.472, 0.032) 0.495 0.191	0.030 (3.759, 0.0090) 0.428 0.13
	MRAA w/o bg 0.043 1.307 0.140 0.127	0.063 (5.626, 0.025) 0.460 0.189	0.029 (3.126, 0.0092) 0.396 0.12
	Ours	0.041 1.353 0.138 0.123	0.057 (4.823, 0.020) 0.431 0.180	0.027 (3.141, 0.0095) 0.399 0.11

Table 2 :

 2 Cross-video generation. We report video FID for both inner-and cross-dataset tasks on VoxCeleb and GermanPublicTV.

		VoxCeleb GermanPublicTV
	FOMM	0.323	0.456
	MRAA	0.308	0.454
	Ours	0.161	0.406

Table 3 :

 3 User study. We ask 20 human raters to conduct a subjective video quality evaluation.compute the distance between generated and real data distributions. As shown in Tab. 2, our method outperforms all other approaches w.r.t. video FID, indicating the best generated video quality.

		VoxCeleb(%) TaichiHD(%) TED-talk(%)
	Ours/FOMM	92.9/7.1	64.5/35.5	71.4/28.6
	Ours/MRAA	89.7/10.3	60.7/39.9	54.8/45.2

Table 6 :

 6 Comparison with full MRAA.

https://wyhsirius.github.io/LIA-project/

https://wyhsirius.github.io/LIA-project/

ACKNOWLEDGEMENTS

This work was granted access to the HPC resources of IDRIS under the allocation AD011011627R1. It was supported by the French Government, by the National Research Agency (ANR) under Grant ANR-18-CE92-0024, project RESPECT and through the 3IA Côte d'Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002.

A DETAILS OF MODEL ARCHITECTURE

We proceed to describe the model architecture in this section. Fig. 7 shows details of our E. In each ResBlock in E, spatial size of input feature maps are downsampled. We take feature maps of spatial sizes from 8 × 8 to 256 × 256 as our appearance features x enc i . We use a 5-layer MLP to predict a magnitude vector A d→r from z d→r . Fig. 8 (a) shows the general architecture of our G, which consists of two components, a flow field generator G f and a refinement network G r . We apply StyleConv (Upsample + Conv3 × 3), which is proposed by StyleGAN2, in G f . StyleConv takes latent representation z s→t as style code and generates flow field φ i and corresponding mask m i . G r uses UpConv (Conv1 × 1 + Upsample) to upsample and refine inpainted feature maps to target resolution. We show details pertaining to G block in Fig. 8 (b). Each G block is used to upsample ×2 the previous resolution. We stack 6 blocks towards producing 256 resolution images.

B EXPERIMENTS

We proceed to introduce details of datasets and evaluation metrics used in our experiments.

B.1 DATASETS

VoxCeleb [START_REF] Nagrani | Voxceleb: Large-scale speaker verification in the wild[END_REF] consists of a large amount of interview videos of different celebrities. Following the process of FOMM [START_REF] Siarohin | First order motion model for image animation[END_REF], we extract frames and crop them into 256 × 256 resolution. In total, VoxCeleb contains a training set of 17928 videos and a test set of 495 videos.


domain: spi chim phys

text:
Thomas Fourest
email: thomas.fourest@onera.fr

Pascal Bouda
email: pascal.bouda@onera.fr

Bertrand Langrand
email: bertrand.langrand@onera.fr

Delphine Notta-Cuvier
email: delphine.notta@univ-valenciennes.fr

Eric Markiewicz
email: eric.markiewicz@uphf.fr

Lloyd Fletcher
email: l.c.fletcher@soton.ac.uk

Fabrice Pierron
email: f.pierron@soton.ac.uk

Éric Markiewicz

Dynamic VFM to Identify Viscoplastic Parameters. Analysis of Impact Tests on Titanium Alloy

Keywords: Titanium alloy, Johnson-Cook, Virtual fields method, Full-field measurement, Viscoplasticity

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Normalized tensile tests performed at different constant strain rates are regularly used to study the strain-rate dependency of the mechanical behavior of metallic materials. The strain-stress responses are successfully used to identify parameters of hardening and viscoplastic models. However, the method is costly due to the large number of specimens to be tested to investigate a wide range of strain rates, mostly because of the required hypothesis of constant stain rate during loading. In addition, a very limited number of loading directions are usually studied with the method. The present work aims at identifying viscoplastic model parameters over a large plastic strain and strain-rate range using the dynamic Virtual Fields Method (dynamic VFM). The dynamic VFM is a very promising method to identify material model parameters based on an inverse approach and full-field measurements of displacements only (no contact measurement). It can be based on a limited number of impact tests thanks to the full exploitation of non-homogeneous strain, strain-rate and stress state generated in the specimen. The complete experimental setup was designed using simulated image-based numerical process (e.g., specimen shape, impact conditions, spatial and temporal regularization parameters). In the present work, the experimental set-up is presented, the impact tests performed on a titanium alloy are described, the results are discussed, and an identification of a viscoplastic model is proposed with the dynamic VFM.

Image-Based Intertial Impact Tests

Image-Based Inertial Impact (IBII) method has been recently proposed [START_REF] Pierron | Beyond Hopkinson's bar[END_REF]2] to take advantage of the dynamic VFM. Among existing inverse methods, the dynamic VFM enables the identification of material parameters with the sole knowledge of the strain and acceleration fields (Eq. 15.1). In particular, the dynamic VFM does not require the knowledge of any external loads, thus avoiding the use of an intrusive sensor (e.g., accelerometer, strain gauge, …) which was proven to be problematic for high strain rate tests. The design of the tests has been performed using finite element analysis (FEA) to generate virtual images that were processed with the same toolchain that will be used to analyse the tests results. This work has been presented in [START_REF] Bouda | SEM 2018 Annual Conference[END_REF]. As nonlinear viscoplastic behavior might need a lot of material parameters to be described, it is necessary to explore more extensively the design space in order to optimize the procedure of identification, in particular for a high yield stress material such as Ti6Al4V (973 MPa [START_REF] Notta-Cuvier | Identification of Johnson-Cook's viscoplastic model parameters using the virtual fields method: application to titanium alloy Ti6Al4V[END_REF]). From this process three different kinds of specimen geometry (i.e., rectangular, with notches, and with a hole) have been selected. The full-field measurements are obtained over the specimen surface using the grid method and pictures recorded with an ultra-high speed camera. The strain rate dependency has been identified. Figure 15.1 shows an example of the plastic strain computed for the last stage recorded during an IBII test on a rectangular specimen.

Viscoplastic Spectra Covered By the IBII Tests

One advantage in inverse methods such as the Virtual Fields Method is that inhomogeneous tests can be performed to generate a wider range of experimental data than the one that could be obtained from "standard" tests. Figure 15.2 illustrates spectra in terms of plastic strain/plastic strain rate typically recorded for each specimen geometry. It can be seen that a large spectrum is covered using only one IBII test. In terms of plastic strain rate it is mainly between 30 s -1 to 2 × 10 3 s -1 for the rectangular specimen. Also, for the same impact speed the specimens including stress concentrating geometries enable to cover a larger spectrum (between 30 s -1 to over 10 4 s -1 for the specimen geometry including a hole for instance).

Conclusion

In this study the strain rate dependency of Ti6Al4V has been successfully identified using the dynamic VFM. Very high strain rate (over 10 4 s -1 ) have been reached in the IBII tests. The next step is to compare the experimental results with those predicted by the numerical approach to better understand the sources of possible differences in the toolchain used.

Fig. 15 . 1

 151 Fig. 15.1 Typical longitudinal plastic strain at the last stage recorded during the IBII test (a) rectangular specimen, (b) specimen with a hole, (c) specimen with two notches


Abdoul-Hadi Konfé
email: ahkonfe@gmail.com

Pasteur Poda

Raphaël Le Bidan

Design Techniques of Spatially Coupled Low-Density Parity-Check Codes: A Review and Tutorial on 5G New Radio

Keywords: 5G, LDPC codes, SC-LDPC codes, tail-biting SC-LDPC codes, braided SC-LDPC codes, tailbiting braided SC-LDPC codes

As active as the research is on the various possible uses of 5G and B5G (beyond 5G), we herein make a tutorial and review on the existing spatial coupling techniques that are used in the protographbased design of Spatially Coupled Low-Density Parity-Check (SC-LDPC) codes. We unroll useful details for the computing of these techniques, implement them in the context of the 5G standard and draw up their performances. As a main result in terms of lesson learnt, a guide is provided to select the most appropriate spatially coupled technique for the three main 5G services and for three of its numerous use cases. Three research tracks are pointed out.

I INTRODUCTION

The 5G, 5th generation of mobile communication standards, is since 2019 to nowadays a reality in several cities around the world. Research on 5G is increasingly active to propose, in connection with its three types of uses (enhanced Mobile Broad Band (eMBB), Ultra-Reliable Low-Latency Communications (URLLC), massive Machine Type Communications (mMTC)), a diversity and multitude of use cases (e.g.: in health, smart societies, education, etc.). At the same time, the B5G (beyond 5G) and the evolution toward the 6G standard are underway. In this dynamic, many works have focused on improving the performance of the Low-Density Parity-Check (LDPC) codes. In the 5G New Radio (5G NR), the air interface of the 5G standard, block LDPC codes have been specified [1]. Indeed block LDPC codes have shown to achieve performance very close to the Shannon limit [START_REF] Chen | Near Shannon Limit Quasi-Cyclic Low-Density Parity-Check Codes[END_REF]. A new family of LDPC codes, originally referred to as LDPC convolutional (LDPC-CC) codes [START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF] and more recently as spatially coupled LDPC (SC-LDPC) codes [START_REF] Mitchell | Spatially Coupled LDPC Codes Constructed from Protographs[END_REF], are built from block LDPC codes by introducing a memory effect [START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF]. SC-LDPC codes have excellent performance promise over a wide variety of channel conditions [START_REF] Mitchell | Spatially Coupled LDPC Codes Constructed from Protographs[END_REF]. Many papers are written in the published literature on the conception of SC-LDPC codes [START_REF] Ma | On tail-biting convolutional codes[END_REF]- [START_REF] Zhang | Braided convolutional codes: a new class of turbo-like codes[END_REF]. However, of the different ways of achieving spatial coupling, there are only five: one is based on an algebraic structure and four are protograph-based.

The basic idea of the first construction technique of SC-LDPC codes was introduced in 1981 through a patent [START_REF] Tanner | Error-correcting coding system[END_REF]. Rediscovered years later [START_REF] Tanner | Convolutional codes from quasi-cyclic codes: A link between the theories of block and convolutional codes[END_REF], it is based on an algebraic structure. The second technique is based on a construction called unwrapping procedure. It uses protograph to generate large graphs [START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF]. The third technique is called tail-biting. It uses protographs to design SC-LDPC with rate-loss mitigation [START_REF] Ma | On tail-biting convolutional codes[END_REF]- [START_REF] Tavares | Tail-biting LDPC convolutional codes based on protographs[END_REF]. The fourth technique is related to braided convolutional codes. It is based on protographs and founded on the braided concatenation of convolutional codes [START_REF] Zhang | Braided convolutional codes: a new class of turbo-like codes[END_REF]. We denoted by braided SC-LDPC codes, the resulting code from this technique. The fifth technique is the tail-biting version of braided SC-LDPC codes [START_REF] Bruno | On low-density parity-check convolutional codes: constructions, analysis and VLSI implementation[END_REF] [START_REF] Tavares | LDPC convolutional codes based on braided convolutional codes[END_REF]. It allows to avoid the rate-loss due to termination saturation. Works involving all these techniques demonstrated good Bit Error Rate (BER) performance over theoretical block LDPC codes [START_REF] Lentmaier | Iterative decoding threshold analysis for LDPC convolutional codes[END_REF]- [START_REF] Lentmaier | Asymptotically good LDPC convolutional codes with AWGN channel thresholds close to the Shannon limit[END_REF]. This performance having been obtained on theoretical LDPC codes, we are interested here in how these same techniques would behave on codes used in practice, in this case the LDPC codes used in the 5G NR. We are also interested in knowing what pratical lessons we could draw from the observed results.

Our objective is to focus on the spatial coupling techniques used in the construction of protographbased SC-LDPC codes in order, for each of these techniques, to make a state of the art, and explain step by step how they can be implemented on 5G LDPC codes. It is also for us to analyze these techniques together with the results that they can yield when applied to 5G codes in order to rank them according to their suitability to each of the three 5G uses.

Our literature search involved the keywords "LDPC-CC", "SC-LDPC codes" and "protographs". Scientific databases of IEEE and Elsevier were used together with other resources like channel coding books and doctoral dissertations and the Google scholar search engine. In the remainder of this paper, Section II deals with the review and tutorial on the different techniques of spatial coupling, Section III is about a comparative analysis together with main conclusions and future research directions, Section IV concludes the paper.

II SC-LDPC CODES DESIGN: A REVIEW AND A TUTORIAL

In this Section, we firstly present the different techniques used to achieve spatial coupling in the design of SC-LDPC codes. Secondly, related works are presented for each of these techniques. Thirdly, we explain step by step how to implement each technique for the construction of a 5G SC-LDPC code. For background on LDPC and SC-LDPC codes, one could refereed to the works in [START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF]- [START_REF] Mitchell | Spatially Coupled LDPC Codes Constructed from Protographs[END_REF] and [START_REF] Gallager | Low-Density Parity-Check Codes[END_REF]- [START_REF] Mitchell | Spatially Coupled LDPC Codes Constructed from Protographs[END_REF]. Details about the specification of the LDPC code for the 5G standard are also given in Annex 1.

Unwrapping technique

The unwrapping technique used to design SC-LDPC codes was presented by Jiménez-Feltström and Zigangirov in 1999 [START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF]. In [START_REF] Solomon | A connection between block and convolutional codes[END_REF], the notions of unterminated and terminated regular LDPC-CC are evoked. Even though the Tanner graph representation of an LDPC-CC code proceeds to infinity in time, practically speaking, it always has a finite start and end time, i.e., the Tanner graph terminates both at the beginning and at the ending. We speak of a completed Tanner graph, and of a terminated LDPC-CC. It is shown in [START_REF] Gallager | Low-Density Parity-Check Codes[END_REF] [25] that for a set of terminated LDPC-CC, the iterative decoding thresholds are better than for the corresponding regular and irregular block LDPC codes. In the papers [21] [34], the authors present the fundamental concepts of coding for lightwave systems, namely the use of spatially coupled codes. In [START_REF] Schmalen | Spatially coupled soft-decision error correction for future lightwave systems[END_REF] and [START_REF] Prasad | A comparative performance study of LDPC and Turbo codes for realistic PLC channels[END_REF], the IEEE 1901 standard for power line communications is studied, where an LDPC-CC with unwrapping technique is specified.

The unwrapping technique is based on a matrix unpacking scheme to get the parity-check matrix of a periodically time varying convolutional code from the block code parity-check matrix. The process performs copy and paste as well as diagonal matrix expansion operations on a block parity-check matrix to generate a bi-infinite convolutional parity-check matrix [START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF]. Details on how to design code using the unwrapping technique can be found in Annex 2.

Tail-Biting technique

LDPC-CC have some advantages over block LDPC codes, especially for the transmission of streaming data [START_REF] Tavares | Tail-biting LDPC convolutional codes[END_REF] or data in packets of various lengths, as the same encoder can be employed to encode data blocks of various lengths. An interesting aspect of LDPC-CC is that the same encoder may be used to generate code sequences of different lengths with reasonably high performance by choosing several termination lengths. However, the insertion of a termination produces a rate-loss, which is particularly noticeable for short frame lengths. The introduction of the tail-biting technique avoids this loss. Tail-biting was originally presented by Solomon and Van Tilborg [START_REF] Solomon | A connection between block and convolutional codes[END_REF] and, separately, by Ma and Wolf [START_REF] Ma | On tail-biting convolutional codes[END_REF] as a technique for terminating a convolutional code with none of the rate-loss due to standard termination. The tail-biting codes that result from have the minimum distance of both convolutional and block codes.

A spatially coupled tail-biting LDPC code can be produced from an SC-LDPC code by combining the control nodes at times t = L, L + 1, ..., L + m s -1 with the matching control nodes of the same design at times t = 0, 1, ..., m s -1, respectively. Therefore, the base matrix corresponding to a tail-biting SC-LDPC code defined in this way has a size equal to b c .L × b v .L. By comparing the two base matrices (terminated SC-LDPC and tail-biting SC-LDPC), one can easily see that the base matrix of tail-biting SC-LDPC can be obtained from the base matrix of terminated SC-LDPC by adding its last b c × m s rows to the first b v × m s rows. As such, tail-biting SC-LDPC codes have the same code rate and degree distribution as the matching protograph codes.

Braided technique

Braided block codes were introduced for the first time in [START_REF]IEEE Standard for Broadband over Power Line Networks: Medium Access Control and Physical Layer Specifications[END_REF], [START_REF] Thorpe | Low-Density Parity-Check (LDPC) Codes Constructed from Protographs[END_REF]. This method of constructing LDPC codes by spatial coupling is analogous to the design of LDPC codes using protographs and constitutes a novel approach to get code sets having robust distance properties, robust iterative thresholds, and low-complexity encoding/decoding from basic codes [START_REF] Bruno | On low-density parity-check convolutional codes: constructions, analysis and VLSI implementation[END_REF]. The codes resulting from this technique fall into the category of LDPC-CC [START_REF] Zhang | Braided convolutional codes: a new class of turbo-like codes[END_REF] [13], called braided LDPC-CC. However, compared to braided block codes (BBC) [START_REF]IEEE Standard for Broadband over Power Line Networks: Medium Access Control and Physical Layer Specifications[END_REF]- [START_REF] Mitchell | AWGN channel analysis of terminated LDPC convolutional codes[END_REF], braided convolutional codes (BCC) [START_REF] Zhang | Braided convolutional codes: a new class of turbo-like codes[END_REF] use low-stress length convolutional codes as candidate codes. BBC and BCC are both similar in their coding procedures. They are built using a bidimensional infinite array composed of a horizontal and a vertical encoder. Both encoders are connected by a return of parity. In this way, parity and systematic symbols are braided at the same time.

Tail-Biting braided technique

As in [START_REF] Ma | On tail-biting convolutional codes[END_REF] and [START_REF] Bruno | On low-density parity-check convolutional codes: constructions, analysis and VLSI implementation[END_REF], it is possible to derive tail-biting versions of braided LDPC-CC codes to eliminate the problem of rate-loss due to termination saturation. The tail-biting periodic LDPC-CC codes are quasi-cyclic codes. Therefore, it is possible to encode them in linear time, by use of circuits relying on shift registers [START_REF] Truhachev | On braided block codes[END_REF]. The methods described in [START_REF] Feltström | Braided block codes[END_REF] can also be used and very successful coding structures can be generated. The implementation of this technique is a combination of the two previous methods, that is, the tail-biting braided SC-LDPC codes are constructed using an array composed of a vertical and a horizontal encoder. Both encoders are connected by a parity return to braid together the parity and systematic symbols. Indeed, introducing a termination always results in a rate-loss. A new code is generated from the braided code by combining the control nodes with corresponding control nodes of the same nature, respectively.

III COMPARATIVE ANALYSIS, MAIN RESULTS AND RESEARCH TRACKS

Starting from the 5G standard block LDPC code, we have developed a redesigned version to evaluate an SC-LDPC code with m s = 1 and L = 10. Let's denote by LDPC-5G this 5G block LDPC code. Let's also denote by : SC-LDPC-5G the modified version of LDPC-5G using the unwrapping technique, SC-LDPC-Tail-Biting-5G the modified version of LDPC-5G using the tail-biting technique, SC-LDPC-Braided-5G the modified version of LDPC-5G using the braided technique and SC-LDPC-Tail-Biting-Braided-5G the modified version of LDPC-5G using tail-biting braided technique. The resulting BER performance of the five codes (LDPC-5G, SC-LDPC-5G, SC-LDPC-Tail-Biting-5G, SC-LDPC-Braided-5G and SC-LDPC-Tail-Biting-Braided-5G) is shown in Figure 1. In Annex 3, useful information about the simulation is provided.

The unwrapping technique code achieves 0.8 dB additional coding gain at a BER of 10 -6 . Good performance is so obtained over the standard 5G code. However, as an insufficiency the introduction of a termination introduces a rate-loss. As about the code designed using the tail-biting technique, 0.9 dB of additional coding gain is obtained for a BER of 10 -6 . The performance is little better than the unwrapping technique with at plus no rate-loss. The evaluation of the code built on the braided technique yields an additional coding gain of 1.1 dB for a BER of 10 -6 . So further better performance is achieved and this third technique outperforms the first two ones. This is in line with the literature based on theoretical codes, as it has been illustrated that the iterative decoding performance is significantly increased with block-braided codes. However, the limitation of this method remains the presence of rate-loss. As about the last technique consisting of a hybridation of the tail-biting technique and the braided technique, a 1.5 dB of additional coding gain at a BER of 10 -6 is achieved. The best performance is so achieved compared to all the previous techniques with the advantages of both the tail-biting and the braiding : a code without rate-loss nor performance degradation. As a conclusion, spatial coupling gives a significant improvement in overall BER performance regardless of the coupling method that is used. From the simulation results discussed above, we derived some main results as lessons learnt in Table 1 and Table 2. Indeed, Table 1 is provided as a guide to better choose the spatial coupling technique according to the three main categories of 5G uses. Table 2 is more focused on three 5G use cases, selected as challenging for many African countries, to show how each of the four spatial coupled techniques can suit to each of these three use cases. The marker "+" means that the technique is appropriate, and the more it is repeated, the more the corresponding technique is appropriate. The choice of one of the spatial coupling techniques for a given service is fundamentally related to reliability, throughput, latency, complexity and number of the connected objects.

Table 1: Spatial coupling techniques and their level of suitability to 5G type of uses.

Table 2: Spatial coupling techniques and their level of suitability to 5G use cases.

URLLC transmissions require, for certain uses, high reliability with very constrained latency times [START_REF] Shirvanimoghaddam | Short Block-Length Codes for Ultra-Reliable Low Latency Communications[END_REF]. The eMBB aims to support the ever-increasing end-user data rate and capacity of the system [START_REF] Popovski | 5G Wireless Network Slicing for eMBB, URLLC, and mMTC: A Communication-Theoretic View[END_REF]. Also the mMTC communications aim for robust and cost-effective connection of billions of devices without overloading the network [START_REF] Popovski | 5G Wireless Network Slicing for eMBB, URLLC, and mMTC: A Communication-Theoretic View[END_REF]. For the improved remote education, high speed, higher capacity and high reliability is required [START_REF] Brittain | [END_REF]. Reliability, mobility and capacity is important for the case of advanced healthcare [START_REF] Konfé | Enhancing 5G Forward Error Correction Codes for URLLC by Spatial Coupling[END_REF]. Cost reduction, resource consumption reduction and high capacity connectivity is imposed for the case of agricultural productivity improvement [START_REF] Brittain | [END_REF].

From this study, we have drawn above practical lessons for the three 5G basic types of uses and for three of its many use cases. As about its implications for research, we can note three tracks for further research. The first track relies on the fact that our study was limited to a spatial coupling memory equal to unity. It seems to us important to investigate how larger values of the coupling memory act on the performance. The second track is to analyze the performance of these techniques in terms of Block Error Rate (BLER). Indeed, for pratical interest, BLER performance analysis is more relevant. The third track is to study the competitiveness of these different techniques with respect to the decoder blocking problem in a sliding window decoding scheme.

IV CONCLUSION

In this paper, we made a review of the different techniques for performing spatial coupling of LDPC codes for 5G. For each technique, we presented the related works. We then made a tutorial exhibition of how to implement each of them. We finally proceeded with a comparative analysis using simulation results, and provided implications for industry and research. In a nutshell, this study could be used as a guide to make a judicious choice of the SC-LDPC code design technique that suits for a target 5G use case and opens a path toward the next wireless communication standard. some memory is introduced into the coding process. An unwrapping of the protograph branches of B is performed to obtain a convolutional protograph (CC-P). For this purpose, an unwrapping constraint is first determined, it is defined by :

ms i=0 B i = B (1)
where B i , 0 ≤ i ≤ m s , are submatrices of B and m s is the memory size of the CC-P. The unwrapping mechanism is equivalent to the subdivision of B such that (1) is satisfied. The terminated CC-P corresponds to a convolutional base matrix (CC-BM) in the case of terminated SC-LDPC codes. The CC-BM matrix is defined by: We get a terminated regular SC-LDPC ensemble, if the graph expansion process (also called graph covering by the Z-factor [START_REF] Mitchell | AWGN channel analysis of terminated LDPC convolutional codes[END_REF]) is applied to the CC-P.

B [0,L-1] =         B 0 . . . . . .
We explain now how we begin from the design of standard codes to produce a novel spatially coupled code. The base matrix B is decomposed into two sub-matrices B 0 and B 1 to generate a memory effect with depth m s . This decomposition is illustrated on an example with a clear explanation of the approach that we followed to obtain the new codes.

To illustrate, we consider a block code protograph with 2 variable nodes and 1 constraint node.

Figure 1 :

 1 Figure 1: BER of various techniques of SC-LDPC 5G codes with N = 128 and R = 4/5.

  L is called the termination factor. The resulting base matrix B [0,L-1] is of dimension (L+m s ).b c ×L.b v and each submatrix B i is of dimension b c × b v . The coding rate of SC-LDPC is then R SC = [L.b v -(L + m s ).b c ]L.b v .

  The base matrix B =[START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF][START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF] defines the parity-check matrix of dimension 1 × 2. Let be a convolutional protograph coupled with the introduction of a memory equal to 2. The unwrapping is given by the relation B = B 0 + B 1 + B 2 and B cc represents the completed convolutional base matrix. The block code is of dimension 1 × 2 with a coding rate 1/2. The convolutional base matrix B SC is of dimension (L + 2) × 2.L with code rate (L -2)/2.L and constraint length 6Z. Now, the graph expansion results in an LDPC convolutional code set with a regular termination[START_REF] Felström | Time-varying periodic convolutional codes with low-density parity-check matrix[END_REF][START_REF] Solomon | A connection between block and convolutional codes[END_REF].

Table 4 :

 4 BG1 and BG2 parameters of LDPC code.

A ANNEX 1 : 5G STANDARD LDPC

We technically present the 5G code and its construction. Quasi-cyclic LDPC (QC-LDPC) codes have been accepted by 3GPP (3rd Generation Partnership Project) as a channel coding scheme for data channels [1]. These codes are developed to support various expansion factors. To correctly adapt to various code rates and block sizes, they also have rate-compatibility properties. Support for multiple expansion factors is handled by these codes and they have rate compatibility properties. The block sizes of the codes and the coding rates required by 3GPP are specified in papers [START_REF]-Remaining Issues on URLLC Data Channel Coding[END_REF] [START_REF] Konfé | Enhancing 5G Forward Error Correction Codes for URLLC by Spatial Coupling[END_REF]. The base matrix is subdivided into sub-matrices (SM1, SM2, SM3, SM4 and SM5) [START_REF] Richardson | Design of Low-Density Parity Check Codes for 5G New Radio[END_REF], for an efficient implementation of the codec (Figure 2). LCE are extension checks and LCN are core checks. 3 and are organized into 8 sets, one for each value of A. Two base graphs (BG1 and BG2) are defined to cover the wide range of data rates and block sizes that must be supported in 5G. The details of the BG1 and BG2 matrices are summarized in Table 4 and [START_REF] Konfé | Enhancing 5G Forward Error Correction Codes for URLLC by Spatial Coupling[END_REF].

B ANNEX 2 : CONSTRUCTION OF UNWRAPPED SC-LDPC CODE

Let (b, c) ∈ N × N . A protograph is a bipartite graph that has c -b constraint nodes and c variable nodes with b representing the size of the coded block [START_REF] Thorpe | Low-Density Parity-Check (LDPC) Codes Constructed from Protographs[END_REF]. A protograph is used to represent the parity-check matrix of a block code of rate R = b/c. The corresponding base matrix is of dimension (c -b) × c with c also representing the block length. To design a base matrix of a spatially coupled code from a base matrix B of an block LDPC code,

C ANNEX 3 : USEFUL INFORMATION FOR

We simulated the performance of the newly designed codes in order to analyze the increase in reliability that can be achieved by introducing spatial coupling according to the different techniques reviewed in Section II. We considered a BPSK (Binary Phase Shift Keying) transmission on an additive white Gaussian noise (AWGN) channel. The belief propagation (BP) decoding approach was employed to do decoding of 5G standard LDPC codes that are based on the parity-check matrices in [START_REF] Liva | Pretty-good-codes[END_REF]. The maximum number of decoding iterations was set to 200 to achieve the best performance. We consider that channel codes with short block length and low code rate are recommended for highly reliable data transmissions [START_REF] Mehrotra | Cellular Radio Performance Engineering[END_REF]. We present BER performance results for 5G codes and make a comparison with the performance of their spatially coupled counterparts that are obtained by the newly developed conceptions. The performance of different LDPC-5G codes with rates 4/5 are considered in our simulations, representing a favorable condition for high-speed transmission. We constructed the SC-LDPC-5G, SC-LDPC-Tail-Biting-5G, SC-LDPC-Braided-5G and SC-LDPC-Tail-Biting-Braided-5G codes with 128 bit block lengths from different SC-LDPC codes. We have constructed modified versions to obtain the SC-LDPC codes with coupling length L = 10 and memory m s = 1.