Sandeep Manandhar
email: sandeep.manandhar@bio.ens.psl.eu

Auguste Genovesio
email: auguste.genovesio@ens.psl.eu

An Invertible Video GAN in a Conditional Setting

Keywords: conditional video generation, temporal style, dynamics transfer

: Our proposed method allows for independent manipulation of content and temporal styles in videos.Top row: Our method can effectively transfer learnt motion to target actors who were not seen performing the act during the training. Bottom row: The facial expression can be gradually changed along the time direction by interpolating the label embedding, while preserving the lip motion.

Introduction

Image synthesis has seen significant advancements with the development of generative models. However, generative models of videos have not been as successful, and controlling the dynamic generation process has been a major challenge. This is largely due to the complex spatio-temporal relationships between content/actors and dynamic/actions, which makes it difficult to synthesize and control the dynamics independently. Several methods have been proposed to address this challenge, each with their own design principles. Broadly speaking, there are two primary classes of video generative models: 3D models that learn from 2D+time volumetric data by employing 3D convolutional neural networks (CNNs), and 2D models that generate a sequence of 2D frames while disentangling the spatio-temporal components of a given video distribution. Many of the earlier methods took the former approach treating each video clip as a point in latent space, thus making the manipulation in such space hardly possible. The latter approach is not only more resource-efficient, but also allows for greater control over the generation process, as demonstrated by [START_REF] Xu | From continuity to editability: Inverting gans with consecutive images[END_REF][START_REF] Zablotskaia | Dwnet: Dense warp-based network for poseguided human video generation[END_REF][START_REF] Zhao | Thin-plate spline motion model for image animation[END_REF]. However, these methods require some pre processing (optical flow, pose information) to manipulate the generated videos.

In their work, [START_REF] Grathwohl | Disentangling space and time in video with hierarchical variational auto-encoders[END_REF] introduced a variational encoder for visual learning, which assumes that higher-level semantic information within a short video clip can be decomposed into two independent sets: static and dynamic. With similar notion, [START_REF] Emily | Unsupervised learning of disentangled representations from video[END_REF] employed two separate encoders to produce content and pose feature representations. Pose features are processed by an LSTM to predict future pose information which is then used along with the current content information to generate the next frame. The idea of treating content and motion information independently has laid a foundation for many works in video generation.

Instead of considering a video as a rigid 3D volume, one can model it as a sequence of 2D video frames x(t) ∈ R 3×H×W , where t is the temporal point, (H, W ) are the height and the width of the video frame. An image generator G(z) can be trained to produce an image x ′ ∼ x(t) from a vector z coming from a latent space Z ∈ R d , where d < H × W . However, the problem at hand is to come up with a sequence of z(t) that can be fed into G(z) to produce a realistic video frame sequence. And, if such z(t) can be obtained, how can we manipulate the video generation process?

The authors of [START_REF] Saito | Temporal generative adversarial nets with singular value clipping[END_REF] proposed to first map a latent vector to a series of latent codes using a temporal generator. An image generator would then use the set of codes to output video frames. MOCOGAN, [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF], on the other hand proposed to decompose the latent space Z into two independent subspaces of content Z c and motion Z m . Z c is modeled by the standard Gaussian distribution, whereas Z m is modeled by a recurrent neural network (RNN). The content code remains the same for a generated video, while motion codes varies for each generated frames. MOCOGAN-HD [START_REF] Tian | A good image generator is what you need for high-resolution video synthesis[END_REF] and StyleVideoGAN [START_REF] Fox | Stylevideogan: A temporal generative model using a pretrained stylegan[END_REF] took advantage of a pretrained Style-GAN2 [START_REF] Karras | Analyzing and improving the image quality of StyleGAN[END_REF] image latent space and proposed to traverse in the latent space using RNNs to produce video frames.

Interestingly, in the context of a pretrained StyleGAN2 network, one can perform GAN inversion [START_REF] Xu | From continuity to editability: Inverting gans with consecutive images[END_REF] on a image sequence to obtain its latent representation. StyleGAN2 produces a continuous and consistent latent space, where close by latent vectors map to similar realistic images. Tak-ing advantage of this property, the latent vector obtained by optimization from the previous frame can be used as the starting point to search for the latent vector of the next frame, thus optimizing for minor changes. Upon simple linear projection (such as PCA) of the latent trajectory of a movie optimized in such manner, we can observe that the higher components are similar to cosine waves (see supplementary material). [START_REF] Hess | Convergence of sampling in protein simulations[END_REF] also made this observation in the context of protein trajectory simulation, where he finds that the cosine content of the principal components are negatively related to the randomness of the simulation. In the case of optimized vectors corresponding to the inverted images, they are correlated. Hence, the waves are obvious and visible. This hints us that sinusoidal bases could naturally facilitate training of a StyleGAN generator to produce image sequences.

To this end, we propose a temporal style generator in order to generate videos using StyleGAN2's sythesis network. Alongside the StyleGAN2's style space, which we treat as the content space, we use a time2vec [START_REF] Seyed Mehran Kazemi | Time2vec: Learning a vector representation of time[END_REF] network to introduce a temporal embedding from where the temporal styles will be generated. time2vec network provides a learnable Fourier bases and carries an additional linear term which prevents the encoding from being cyclic. Main contributions of our work are as follow:

• We integrate a novel temporal latent space in Style-GAN's generator network using a sinusoid-based temporal embedding.

• We evaluate our method against prevalent methods in an unconditional setting, demonstrating a significant enhancement of video quality.

• We propose several approaches to rigorously evaluate conditional video generation through contexts such as talking faces and human activities.

• We demonstrate the benefits of style-based temporal encoding to independently transfer dynamics or content, editing motion mid-sequence or reuse the dynamic extracted from a real video by GAN-inversion of temporal codes.

We trained our model on videos of human facial expression (MEAD [START_REF] Wang | Mead: A large-scale audio-visual dataset for emotional talking-face generation[END_REF]) and human activities (UTD-MHAD [2]). Besides the Fréchet video distance (FVD) [32] metric, we conducted human evaluation focused on the realism of the generated videos using the MEAD dataset. Additionally we proposeed LiA (Lips Area) metric to evaluate the talking face videos from the MEAD dataset. We also benchmarked our results using publicly available method for human action recognition with UTD-MHAD dataset.

Related work

The domain of video synthesis consists of tasks such as future frame prediction [START_REF] Finn | Unsupervised learning for physical interaction through video prediction[END_REF][START_REF] Mathieu | Deep multi-scale video prediction beyond mean square error[END_REF]34,[START_REF] Emily | Unsupervised learning of disentangled representations from video[END_REF], frame interpolation [START_REF] Niklaus | Video frame interpolation via adaptive convolution[END_REF]14,[START_REF] Xiang | Zooming slow-mo: Fast and accurate one-stage space-time video super-resolution[END_REF] and in our context, video generation from scratch [33]. Video generation follows the success of image generative adversarial models which can produce highly controllable images of remarkable quality [START_REF] Goodfellow | Generative adversarial nets[END_REF]. Much focus has been given to temporal extension of such GANs. [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF][START_REF] Saito | Temporal generative adversarial nets with singular value clipping[END_REF][START_REF] Saito | Train sparsely, generate densely: Memoryefficient unsupervised training of high-resolution temporal gan[END_REF][START_REF] Munoz | Temporal shift gan for large scale video generation[END_REF] have adopted the strategy to use content and motion codes by leveraging on 2D image generator. MOCOGAN-HD [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF] used a pretrained StyleGAN2's network [START_REF] Karras | Analyzing and improving the image quality of StyleGAN[END_REF] and trained a RNN model to simply explore along the principal components of the latent space. Recently, [START_REF] Brooks | Generating long videos of dynamic scenes[END_REF] also proposed a style-based temporal encoding for a 3D version of StyleGAN3's synthesis network [START_REF] Karras | Alias-free generative adversarial networks[END_REF] where temporal codes are generated by a noise vector filtered by a fixed set of temporal low pass-filters. [START_REF] Yu | Generating videos with dynamics-aware implicit generative adversarial networks[END_REF] used implicit neural representation (INR) [4,[START_REF] Chen | Learning continuous image representation with local implicit image function[END_REF] to model videos as continuous signal. Concurrently, StyleGAN-V [START_REF] Skorokhodov | Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2[END_REF], relied on training a modified StyleGAN2 generator where they propose a INR-inspired positional embedding for the time-points of the video frames. Both of these methods produce videos with arbitrary frame rates. Our method is related to StyleGAN-V as it uses StyleGAN2 synthesis network. However, their approach differs from ours as they concatenate their temporal codes with the constant input tensor of the synthesis network, whereas we treat our temporal codes as style vectors.

Conditional generative models are another exciting field of research. Besides explicit vector based labels, text, audio and images have been used in conditioning for frame generation. [START_REF] Yaohui | Imaginator: Conditional spatio-temporal gan for video generation[END_REF] proposes a simple and efficient 3D CNN based generator that takes a single image and a conditioning label as an input to generate videos. [START_REF] Wiles | X2face: A network for controlling face generation using images, audio, and pose codes[END_REF] takes a source frame with one human face and generates video that has pose and expression of another face in a driving video. [START_REF] Ting-Chun | Video-tovideo synthesis[END_REF] conditioned their video generation on semantic maps where objects present in the frame are labelled with colors. The network can also take information like optical flow and pose information during the training. [START_REF] Songsri-In | Face video generation from a single image and landmarks[END_REF] generated videos of talking face using sequence of facial landmarks of target face. [START_REF] Zhao | Thin-plate spline motion model for image animation[END_REF] is yet another image-conditioned video generation model, which has dedicated networks for motion prediction and keypoint detection. However, it is not straightforward to generated videos with arbitrary frame rates with image-conditioned models. [START_REF] Ho | Video diffusion models[END_REF] proposed a 3D U-Net based diffusion model for text to video generation. Following this [START_REF] Singer | Make-a-video: Text-to-video generation without text-video data[END_REF] proposed another text-to-video generation method that makes use of efficient 3D convolutions and temporal attention modules. They also added an embedding for specifying frame rates.

Method

Our method contains two main components: (1) a temporal style generator that drives StyleGAN2's synthesis network to produce frames in time-conditioned manner, (2) two discriminators to impose content consistency and temporal consistency. Our generator is further conditioned on actor identity and action classes, though it can be used in unconditional setting.

Generator

Our generator G is a conditional generator conditioned on actor-id and action-id. We equip the generator with appropriate embeddings for both conditions. The two embeddings are summed together to obtain a content style vector w c , which defines the general appearance of the actor along with the nature of action. Now, we define our novel temporal generator F t . It maps a random vector z m , using a 4-layered MLP to a k dimensional motion style vector m t . The vector scales the waves provided by an independant time embedding. The time embedding time2vec maps a time value t to a k dimensional vector v(t). The resulting product is w t m , which we refer to as temporal style vector. To generate a frame at time t, we concatenate both styles as [w c , w t m ] before injecting them to the synthesis blocks. During the training, we generate three consecutive frames for each video element per batch. The triplets share the same vector m t while their time2vec embedding are generated from their respective time points. More formally,

m t = F t (z m ), w t m = m t * v(t), w t+1 m = m t * v(t + 1). (1)
A basic structure of the generator network is shown in Figure 2. To ensure the smooth integration of action-id embeddings, we employ a ramp function [START_REF] Shahbazi | Collapse by conditioning: Training classconditional GANs with limited data[END_REF] that linearly scales the vectors derived from the action-id embedding with a factor ranging from 0 to 1, in a scheduled manner. Time2vec: We used k -1 sinusoidal bases accompanied by a linear term to create our temporal embedding as seen in Eq. 2, where the parameters w j and ϕ j are trainable. By restricting the dynamics to a fixed set of sinusoidal functions, we can avoid overfitting to the training data, since the model has a limited capacity to represent complex dynamics. This makes the model more robust and generalizable to unseen data. Moreover, since sinusoidal functions are periodic, they can naturally capture cyclic patterns in the data (e.g. lip movement, hand waving).

v j (t) = ω j t + ϕ j , if j = 0 sin(ω j t + ϕ j ), if 1 ≤ j ≤ k -1 (2)
The linear term represents the time direction. The time t does not need to be discrete as the time2vec embedding is a continuous domain. This allows us to generate videos with arbitrary frame rates. However, during the training we use integer valued time-points. We note that StyleGAN-V's time representation lacks the linear term, which might hint to why its generation is plagued by unnatural repetitive motion despite its elaborate interpolation scheme. Unlike StyleGAN-V, we have chosen to stay closer to StyleGAN's original principal, which is to allow variations in input only through the style vectors rather than the traditional input tensor. Furthermore, our time embedding fundamentally differs from StyleGAN-V's in its design. StyleGAN-V requires interpolation of multiple noise vectors to compute a single trajectory. Additionally, the wave parameters involved are dependent on the noise samples. In contrast, our wave parameters are independently learned and are fixed during inference. The latent vector m t interacts with the waves only as an amplitude scaling factor. This makes our time representation more compact and simpler. We leverage this representation to perform GAN-inversion for the motion style using off-the-self methods, which is not possible with StyleGAN-V representation. (see )

Discriminators

Shuffle discriminator: Consistency in content over time is a crucial aspect of video generation. Although the time2vec module in G provides temporal bases to guide motion learning, it does not ensure consistency in content across the sequence. In order to address this, we design a 2D-CNN based discriminator D s (see Figure 2) that evaluates whether the frame features are consistent or not. During the training of D s , each batch element consists of two frames. For the fake adversarial example, pairs of frames are shuffled among the batch to contain two different contents. In contrast, for the real example, the pairs are consecutive frames drawn from real videos. The feature maps of the pairs undergo a series of 2D convolutions, are flattened, and then concatenated into a single vector before passing through a fully connected layer. During the training of G, a batch of unshuffled fake pairs is input to D s .

Conditional discriminator: To ensure temporal consistency in the generated videos, we adopt a time-conditioned discriminator, inspired by prior works such as [START_REF] Miyato | cGANs with projection discriminator[END_REF][START_REF] Yu | Generating videos with dynamics-aware implicit generative adversarial networks[END_REF][START_REF] Skorokhodov | Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2[END_REF]. This discriminator, denoted as D t , takes in a batch of video triplets along with their respective time information, and learns to distinguish real videos from fake ones based on their temporal coherence. Then the video frames are processed by a set of 2D CNNs and a linear layer d t (.) to produce frame features. These features are then concatenated following the temporal order. D t is equipped with another time2vec module which enforces learning of a time representation. The temporal encoding for the three input time points are also concatenated. The dot product of these concatenated vectors is then computed, generating a final score [START_REF] Miyato | cGANs with projection discriminator[END_REF].

In conditional setting, actor-id and action-id embeddings are introduced in D t as well. As shown in Figure 2, two additional linear layers (d action (.), d actor (.)) are present at the level of d t (.), which produce actor and action representations. Dot products are computed between the corresponding embedded vector and the feature vector. The final output of the discriminator is the weighted sum of the three dot products. We use the same ramp-up function to scale d action (.) as in the generator [START_REF] Shahbazi | Collapse by conditioning: Training classconditional GANs with limited data[END_REF].

Experimental settings

We perform experiments with our video generator in both conditional and unconditional settings. Though our unconditional video generation performs competitively against the baseline methods, we focus on the conditional generation as it permits for better disentanglement between actions and actors, as well as better control over the generation process.

Datasets

We have used three publicly available video datasets with their labels: MEAD [START_REF] Wang | Mead: A large-scale audio-visual dataset for emotional talking-face generation[END_REF], RAVDESS and UTD-MHAD [2]. Our MEAD training set contains 30 individuals talking while expressing 8 different emotions (18883 videos). We train our network only with the sequences where generic sentences are being recited. We set aside the emotion specific dialogues as unseen test sequences. For the training, we chose 128 × 128 image dimension and between 60 -170 frames as the dataset contains videos of variable length.

The RAVDESS dataset contains 24 talking faces also with 8 different emotions (not same categories as MEAD). To create a test set, we exclude sequences of 7 different emotions for four individuals. Though the dataset set contains only two dialogues, compared to over 20 dialogues in MEAD, RAVDESS contains more variation in head movements of the actors.

We used all the RGB videos provided with UTD-MHAD. It contains 754 videos of 8 individuals performing 27 different actions. The video frame size is 128 × 128 with variable video length (33 -81) as provided in the dataset. We created a test set by excluding videos of each action sequence performed by few selected target actor from the training set. Thus, we train the network to learn motion and content independently.

Baseline Methods

To demonstrate that our generator does not falter in video quality, we choose MOCOGAN-HD [START_REF] Tulyakov | Mocogan: Decomposing motion and content for video generation[END_REF] and StyleGAN-V [START_REF] Skorokhodov | Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2[END_REF] as our baselines in unconditional setting as they both use StyleGAN2's image synthesizer. For the conditional video generation, we choose ImaGINator [START_REF] Yaohui | Imaginator: Conditional spatio-temporal gan for video generation[END_REF]. Though it requires an input image to generate videos, it is free of any Figure 2: Description of the proposed model: a temporal style generator F t equipped with a time2vec module generates the motion code. In a conditional setting the content style generator F u is replaced by F c , which consists of two independent actor and action embeddings. The corresponding embeddings are activated in D t 's final layer. A ramp function [START_REF] Shahbazi | Collapse by conditioning: Training classconditional GANs with limited data[END_REF], which gradually increases from 0 to 1, is used to scale the vectors coming from the action embedding in both F c and D t additional representation like pose or motion maps. For MOCOGAN-HD, we first trained a StyleGAN2 network on MEAD dataset with 256 2 image size with batch size of 16 for upto 150K iterations. Then the MOCOGAN-HD network was trained with the hyper-parameters set as suggested in the author's implementation. For StyleGAN-V, we trained on both datasets with image of dimension 256 2 , with a batch size of 64 and with up to 25000K images according to the author's implementation. We adapted ImaGINator's network to output 128 × 128 × 32 size image (as it was originally 64 × 64 × 32). We trained it on both datasets in a conditional manner for up to 5K epochs.

Training

We trained our method on a single Nvidia's A100 GPU with 80GB VRAM. The training image size was 128 2 with a batch size of 16 triplet frames. The hyperparameters for the generator, discriminators and the optimizers were kept the same as suggested in [START_REF] Karras | Analyzing and improving the image quality of StyleGAN[END_REF]. The transition factor λ of action-id vectors in both generator and discriminator started at 4000 iterations and ended at 6000 iterations, which was set empirically. We trained our model for both datasets for up to 150k iterations which took about 2 weeks.

Results

Video quality is improved

Table 1 reports the FVD scores of the generated videos by all the methods. Our conditional method (Ours(C)) scores the best which is in agreement with the videos provided in the supplementary data. Few frames of the generated video samples are depicted in Figure 3. Motion artifacts are strongly present in MOCOGAN-HD and ImaGINator's output. StyleGAN-V has relatively higher quality videos but suffers from erratic, repeated motion. However, our methods (both conditional Ours(C) and unconditional Ours(UC)) produce far better results. To assess the preservation of the actor's identity, we computed the ArcFace [START_REF] Deng | Arcface: Additive angular margin loss for deep face recognition[END_REF] similarity between the frames of the generated videos. ArcFace computes the cosine similarity between the feature vector of the first frame and the successive ones obtained from a network trained for face recognition. As seen in Table 1, our methods preserve the appearance of the actor throughout the sequence while MOCOGAN-HD is not consistent generating the same face over the sequence. The authors of [START_REF] Skorokhodov | Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2[END_REF] also made this observation.

The FVD score is widely used to evaluate video quality. However, as it is a comparison of distributions of representations in a high dimensional space, it may not accurately characterize the true quality of the video. The same can be said about the ArcFace score. Furthermore, these metrics can be influenced by factors such as spatial resolution, video length, etc. To complement these metrics, a human evaluation was conducted to assess the realism of the generated videos. To conduct the human evaluation, we generated 10 sets of videos, each consisting of 6 videos (1 real video and 5 generated videos using the proposed methods and the baselines). We asked 25 university students and researchers to watch 3 randomly selected sets and rank the 6 videos based on their perceived realism. The ranking distributions of the survey is presented in Figure 4. Notably, videos generated with Ours(C) and Ours(UC) models consistently ranked higher than those generated using the baseline methods. This demonstrates that our method produces more realistic videos compared to existing approaches.

Temporal style encodes temporal semantic

While we demonstrated that the video quality is improved, the aforementioned metric cannot assess the preservation of temporal semantics across different sequences. We then propose a new metric named LiA (for Lips Area) to evaluate our ability to reproduce the semantic of talking-face videos while changing the content such as the actor-id or actionid (emotion). LiA value computes the polygonal area of the lips detected using face landmark detectors [START_REF] King | Dlib-ml: A machine learning toolkit[END_REF]. A LiA signal is then obtained by computing LiA value sequentially for each frames of a generated or a real video. Though there are other factors such as eye brows and head orientation that contribute to the overall dynamics of a talking face, we focus on lip motion as it appears to be the most dynamic part of the face on this dataset. We generated 100 different sequences using different content styles and the same temporal style for the baseline methods. The average correlation coefficient rt of the LiA signals of the generated videos by all the methods are reported in Table 1. We observed that even for the same temporal style, the ImaG-INator produced different motion pattern depending on the

Generation of unseen coupled conditions

We generate videos of unseen actor-action combination only present in the test set. Figure ?? shows a few selected frames of real videos, and generated videos by ImaGINator, and our conditional method. Our method is able to successfully transfer a learnt action to an actor who was never seen performing this action in the training set. To evaluate our method in this dataset, we additionally train an action recognition model using the implementation of [START_REF] Duan | Revisiting skeleton-based action recognition[END_REF]. We train the model on skeletal key points extracted from the video frames of our training set which contains 27 different actions. The trained model was able to achieve (77%, 100%) top-1 and top-3 accuracies on the real test cases. In our generated case, it was able to achieve (68.5%, 93.5%) top-1 and top-3 accuracies. We present the confusion matrices for 27 different classes in the supplementary data. Our model not only generated high-quality videos, as shown in Table 2, but also accurately captured many actions. On the other hand, ImaGINator performed poorly on this dataset, with evidence of mode collapse in the type of motion despite the conditioning during inference. We have included the generated videos in the supplementary data. Table 3: FVD and classification accuracy for UTD-MHAD with three different versions of our model.

Interpolating conditions over time

Because our content and motion space is highly disentangled, it is possible to edit the attributes of the videos over time. We choreograph a sequence where actors change their expression over time by a linear interpolation in the actionid embedding space. The interpolation does not interfere with the general motion of the face. (See supplementary videos)

Motion recovery with GAN inversion

A talking face video can be generated with random temporal style. However, the apparent motion of the mouth may not recite any plausible sentence. We show that with our generator, it is possible to obtain a temporal style, free of any motion computation or landmark point detection for reciting a given sentence by simply using GAN-inversion. In the following experiments, we invert unseen videos from test cases of MEAD and RAVDESS dataset. For the MEAD dataset, we recover the motion from the real video of actor reciting the sentences which were excluded from the training set. We assume that the excluded dialogues carry unseen lip motions and show that our GAN-inversion is capable of recovering them. After fixing the learned actor and emotion labels of the real input sequence, we optimize only for m t (we keep the sinusoidal bases fixed as well). We minimize the LPIPS [START_REF] Zhang | The unreasonable effectiveness of deep features as a perceptual metric[END_REF] and mean squared error losses over the batch of frames. Figure 6a shows an example of LiA signals for real and inverted videos using network trained with k = 63, 127, 255. The more the number of sinusoidal bases is used in the generator, the more faithful the recovered motion is to the real video. We performed the inversion and LiA signal analysis for 39 different emotion specific sentences excluded from the training set (see Appendix for the complete list), and report the average correlation to be 0.6, 0.79 and 0.91 for k = 63, 127, 255 simultaneously, which supports our observation. Furthermore, in Figure 6b, the inversion is able to recover the large movement of head in RAVDESS dataset. The facial structure is further improved using pivotal tuning [START_REF] Roich | Pivotal tuning for latent-based editing of real images[END_REF] where we adjust generator's weight by fixing the previously optimized m t vector. Thus recovered motion in the form of m t can then be transferred to other learned actors of choice. We believe this is a novel way for re-enactment or lip syncing between different individuals and different emotion.

Ablation

In our ablation studies, we investigate the impact of different components on the performance of our model. Using more sinusoidal bases improves the recovered motion with GAN-inversion as discussed in the section before. However, higher number of k leads to minute intermittent motion artefacts of eyes in MEAD dataset. For k = 127, most of the artefacts are unnoticeable. We trained D t using only one time point instead of three time points, which resulted in a decrease in action recognition accuracy from 68% to 57% for unseen conditions. Secondly, we removed D s in the training on the MEAD dataset, which led to an FVD score of 600. We report the affect of tweaking of D t on UTD-MHAD dataset in Table 3. We also examined the effect of using a ramp function to schedule scaling of the action-id vectors. We found that without the ramp function, introducing the action-id at the beginning of the training caused the generator to favor one class over the other, while using the ramp function stabilized the quality of the videos for all classes.

Conclusion

In this study, we proposed a video generation model which produces high quality videos in both conditional and unconditional settings. Through various experiments, we show that the temporal style can independently encode the dynamics of the training data and can be transferred to unseen targets. We demonstrated that it is possible to generate different types of action with high accuracy as seen in UTD-MHAD videos. Our generator produces videos with bet-ter fidelity than the prevalent style-based video generation methods as shown by various metrics as well as human preference score. Though we did not generate high resolution videos, the StyleGAN2's synthesis network allows us to do so as shown by [START_REF] Skorokhodov | Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2[END_REF]. We also provide an example of GANinversion for temporal styles which opens possibilities for GAN-based editing for videos.

Figure 3 :

 3 Figure 3: Top panel shows videos generated by our conditional model for various emotions and actors using the same temporal style. The bottom row depicts few sampled frames from videos generated using StyleGAN-V, MOCOGAN-HD, and ImaGINator.

5 )

 5 Table2: FVD score for the UTD-MHAD dataset. Though we train StyleGAN-V unconditionally, it serves as a good baseline for assessing the video quality. We also report top-(1,3) action recognition accuracies among 27 action classes.

Figure 4 :

 4 Figure 4: Human preference ranking for different videos. Videos generated by our conditional model (Ours(C)) tops the preference over other methods.

Ablation FVD 16

 16 accuracy (top-1/top-3)% D t (1 time-point) 179.73 59.2/79.6 D t (3 time-points) 184.55 68.5/93.5 w/o D t 526.21 42.6/80.5

  (a) Sit-to-Stand sequence (b) Pick-and-throw sequence

Figure 5 :

 5 Figure 5: Dynamics transfer in unseen conditions. Both panels display few frames sampled from videos of real sequence (top), generated sequence by ImaGINator (mid), and our method (bottom). The actors were never seen performing these actions (top) during the training.

  Figure 6: (a) The correlation coefficient (ρ) between LiA signals of real and inverted videos suggest that the network with higher number of sinusoidal bases generate more faithful videos. (b) Pivotal tuning further improves the facial structure even though most of the motion is already recovered in the first step.

Table 1 :

 1 All the scores pertain to the training with MEAD dataset. FVD 16/64 is computed with 16 and 64 (only for StylgeGANV and ours(C)) frames. rt is the average correlation coefficient of the LiA signals. ArcFace is the average of the cosine similarity between the features of the first frame and the successive frames.

	Method	FVD 16/64 ↓	rt ↑	ArcFace ↑
	ImaGINator	319	0.041 0.93±0.03
	MOCOGAN-HD	272	0.52 0.80±0.13
	StyleGANV	191/920	0.77 0.92±0.05
	Ours(UC)	140	0.79 0.97±0.018
	Ours(C)	115/655	0.7	0.96±0.02


Elsa Cazelles
email: elsa.cazelles@u-bordeaux.fr

Jérémie Bigot

Nicolas Papadakis

Regularized Barycenters in the Wasserstein Space

Keywords: Wasserstein space, Fréchet mean, Barycenter of probability measures, Convex regularization, Bregman divergence

 on the convex regularization of Wasserstein barycenters for random measures supported on R d . We discuss the existence and uniqueness of such barycenters for a large class of regularizing functions. A stability result of regularized barycenters in terms of Bregman distance associated to the convex regularization term is also given. Additionally we discuss the convergence of the regularized empirical barycenter of a set of n iid random probability measures towards its population counterpart in the real line case, and we discuss its rate of convergence. This approach is shown to be appropriate for the statistical analysis of discrete or absolutely continuous random measures. In this setting, we propose an efficient minimization algorithm based on accelerated gradient descent for the computation of regularized Wasserstein barycenters.

Introduction

This paper is concerned by the statistical analysis of data sets whose elements may be modeled as random probability measures supported on R d . It is an overview of results that have been obtain in [START_REF] Bigot | Penalized barycenters in the Wasserstein space[END_REF]. In the special case of one dimension (d = 1), we are able to provide refined results on the study of a sequence of discrete measures or probability density functions (e.g. histograms) that can be viewed as random probability measures. Such data sets appear in various research fields. Examples can be found in neuroscience [START_REF] Wu | An information-geometric framework for statistical inferences in the neural spike train space[END_REF], biodemographic and genomics studies [START_REF] Zhang | Functional density synchronization[END_REF], economics [START_REF] Kneip | Inference for density families using functional principal component analysis[END_REF], as well as in biomedical imaging [START_REF] Petersen | Functional data analysis for density functions by transformation to a Hilbert space[END_REF]. In this paper, we focus on first-order statistics methods for the purpose of estimating, from such data, a population mean measure or density function.

The notion of averaging depends on the metric that is chosen to compare elements in a given data set. In this work, we consider the Wasserstein distance W 2 associated to the quadratic cost for the comparison of probability measures.

Let Ω be a subset of R d and P 2 (Ω) be the set of probability measures supported on Ω with finite order second moment. Definition 1. As introduced in [START_REF] Agueh | Barycenters in the Wasserstein space[END_REF], an empirical Wasserstein barycenter νn of a set of n probability measures ν 1 , . . . , ν n (not necessarily random) in P 2 (Ω) is defined as a minimizer of

µ → 1 n n i=1 W 2 2 (µ, ν i ), over µ ∈ P 2 (Ω). (1)
The Wasserstein barycenter corresponds to the notion of empirical Fréchet mean [START_REF] Fréchet | Les éléments aléatoires de nature quelconque dans un espace distancié[END_REF] that is an extension of the usual Euclidean barycenter to nonlinear metric spaces.

However, depending on the data at hand, such a barycenter may be irregular. As an example let us consider a real data set of neural spike trains which is publicly available from the MBI website 1 . During a squared-path task, the spiking activity of a movement-encoded neuron of a monkey has been recorded during 5 seconds over n = 60 repeated trials. Each spike train is then smoothed using a Gaussian kernel (further details on the data collection can be found in [START_REF] Wu | An information-geometric framework for statistical inferences in the neural spike train space[END_REF]). For each trial 1 ≤ i ≤ n, we let ν i be the measure with probability density function (pdf) proportional to the sum of these Gaussian kernels centered at the times of spikes. The resulting data are displayed in Fig. 1(a). For probability measures supported on the real line, computing a Wasserstein barycenter simply amounts to averaging the quantile functions of the ν i 's (see e.g. Section 6.1 in [START_REF] Agueh | Barycenters in the Wasserstein space[END_REF]). The pdf of the Wasserstein barycenter νn is displayed in Fig. 1(b). This approach clearly leads to the estimation of a very irregular mean template density of spiking activity.

In this paper, we thus introduce a convex regularization of the optimization problem (1) for the purpose of obtaining a regularized Wasserstein barycenter. In this way, by choosing an appropriate regularizing function (e.g. the negative entropy in Subsection 2.1), it is of possible to enforce this barycenter to be absolutely continuous with respect to the Lebesgue measure on R d .

Regularization of barycenters

We choose to add a penalty directly into the computation of the Wasserstein barycenter in order to smooth the Fréchet mean and to remove the influence of noise in the data. Definition 2. Let P ν n = 1 n n i=1 δ νi where δ νi is the dirac distribution at ν i . We define a regularized empirical barycenter µ γ P ν n of the discrete measure P ν n as a minimizer of

µ → 1 n n i=1 W 2 2 (µ, ν i ) + γE(µ) over µ ∈ P 2 (Ω), (2)
where P 2 (Ω) is the space of probability measures on Ω with finite second order moment, E : P 2 (Ω) → R + is a smooth convex penalty function, and γ > 0 is a regularization parameter.

In what follows, we present the main properties on the regularized empirical Wasserstein barycenter µ γ P ν n .

1 http://mbi.osu.edu/2012/stwdescription.html

Existence and uniqueness

We consider the wider problem of min µ∈P2(Ω)

J γ P (µ) = W 2 2 (µ, ν)dP(ν) + γE(µ). (3)
Hence, (2) corresponds to the minimization problem (3) where P is discrete ie

P = P n = 1 n n i=1 δ νi .
Theorem 1 (Theorem 3.2 in [START_REF] Bigot | Penalized barycenters in the Wasserstein space[END_REF]) Let E : P 2 (Ω) → R + be a proper, lower semicontinuous and differentiable function that is strictly convex on its domain

D(E) = {µ ∈ P 2 (Ω) such that E(µ) < +∞}.
Then, the functional J γ P define by (3) admits a unique minimizer.

Such assumptions on E are supposed to be always satisfied throughout the paper. A typical example of regularization function satisfying such assumptions is the negative entropy defined as

E(µ) =    R d f (x) log(f (x))dx, if µ admits a density f with respect to
the Lebesgue measure dx on Ω, +∞ otherwise.

Stability

We study the stability of the minimizer of (3) with respect to the discrete distribution P ν n = 1 n n i=1 δ νi on P 2 (Ω). This result is obtained for the symmetric Bregman distance d E (µ, ζ) between two measures µ and ζ. Bregman distances associated to a convex penalty E are known to be appropriate error measures for various regularization methods in inverse problems (see e.g. [START_REF] Burger | Convergence rates of convex variational regularization[END_REF]). This Bregman distance between two probability measures µ and ζ is defined as

d E (µ, ζ) := ∇E(µ) -∇E(ζ), µ -ζ = Ω (∇E(µ)(x)-∇E(ζ)(x))(dµ -dζ)(x),
where ∇E : Ω → R denotes the gradient of E. In the setting where E is the negative entropy and µ = µ f (resp. ζ = ζ g ) admits a density f (resp. g) with respect to the Lebesgue measure, then d E is the symmetrised Kullback-Leibler divergence

d E (µ f , ζ g ) = (f (x) -g(x)) log f (x) g(x) dx.
The stability result of the regularized empirical barycenter can then be stated as follows.

Theorem 2 (Theorem 3.3 in [START_REF] Bigot | Penalized barycenters in the Wasserstein space[END_REF]) Let ν 1 , . . . , ν n and η 1 , . . . , η n be two sequences of probability measures in P 2 (Ω). If we denote by µ γ P ν n and µ γ P η n the regularized empirical barycenter associated to the discrete measures P ν n and P η n , then the symmetric Bregman distance (associated to E) between these two barycenters is bounded as follows

d E µ γ P ν n , µ γ P η n ≤ 2 γn inf σ∈Sn n i=1 W 2 (ν i , η σ(i) ), (4)
where S n denotes the permutation group of the set of indices {1, . . . , n}.

In particular, inequality (4) allows to compare the case of data made of n absolutely continuous probability measures ν 1 , . . . , ν n , with the more realistic setting where we have only access to a dataset of random variables X = (X i,j ) 1≤i≤n; 1≤j≤pi organized in the form of n experimental units, such that X i,1 , . . . , X i,pi are iid observations in R d sampled from the measure ν i for each 1 ≤ i ≤ n. If we denote by ν pi = 1 pi pi j=1 δ Xi,j the usual empirical measure associated to ν i , it follows from inequality (4) that

E d 2 E µ γ P ν n , µ γ X ≤ 4 γ 2 n n i=1 E W 2 2 (ν i , ν pi ) ,
where µ γ X is given by µ γ X = argmin

µ∈P2(Ω) 1 n n i=1 W 2 2 (µ, 1 pi pi j=1 δ Xi,j ) + γE(µ).
This result allows to discuss the rate of convergence (for the symmetric squared Bregman distance) of µ γ X to µ γ P ν n as a function of the rate of convergence (for the squared Wasserstein distance) of the empirical measure ν pi to ν i for each 1 ≤ i ≤ n (in the asymptotic setting where p = min 1≤i≤n p i is let going to infinity). As an illustrative example, in the one-dimensional case (that is d = 1), one may use the work in [START_REF] Bobkov | One-dimensional empirical measures, order statistics and Kantorovich transport distances[END_REF] on a detailed study of the variety of rates of convergence of an empirical measure on the real line toward its population counterpart for the expected squared Wasserstein distance. For example, by Theorem 5.1 in [START_REF] Bobkov | One-dimensional empirical measures, order statistics and Kantorovich transport distances[END_REF], it follows that

E W 2 2 (ν i , ν pi ) ≤ 2 p i + 1 J 2 (ν i ), with J 2 (ν i ) = Ω F i (x)(1 -F i (x)) f i (x) dx,
where f i is the pdf of ν i , and F i denotes its cumulative distribution function. Therefore, provided that J 2 (ν i ) is finite for each 1 ≤ i ≤ n, one obtains the following rate of convergence of µ γ X to µ γ

P ν n (in the case of measures ν i supported on an interval Ω of R) E d 2 E µ γ P ν n , µ γ X ≤ 8 γ 2 n n i=1 J 2 (ν i ) p i + 1 ≤ 8 γ 2 1 n n i=1 J 2 (ν i ) p -1 . (5)
Note that by the results in Appendix A in [START_REF] Bobkov | One-dimensional empirical measures, order statistics and Kantorovich transport distances[END_REF], a necessary condition for J 2 (ν i ) to be finite is to assume that f i is almost everywhere positive on the interval Ω.

Convergence to a population Wasserstein barycenter

Introducing this symmetric Bregman distance also allows to analyze the consistency of the regularized barycenter µ γ Pn as the number of observations n tends to infinity and the parameter γ is let going to zero. When ν 1 , . . . , ν n are supposed to be independent and identically distributed (iid) random measures in P 2 (Ω) sampled from a distribution P, we analyze the convergence of µ γ P ν n with respect to the population Wasserstein barycenter defined as

µ 0 P ∈ argmin µ∈P2(Ω) W 2 2 (µ, ν)dP(ν),
and its regularized version

µ γ P = argmin µ∈P2(Ω) W 2 2 (µ, ν)dP(ν) + γE(f ).
In the case where Ω is a compact of R d and ∇E(µ 0 P ) is bounded, we prove that µ γ P converges to µ 0 P as γ → 0 for the Bregman divergence associated to E. This result corresponds to showing that the bias term (as classically referred to in nonparametric statistics) converges to zero when γ → 0. We also analyze the rate of convergence of the variance term when Ω is a compact of R: Theorem 3 (Theorem 4.5 in [START_REF] Bigot | Penalized barycenters in the Wasserstein space[END_REF]) For Ω compact included in R, there exists a constant C > 0 (not depending on n and γ) such that

E d 2 E µ γ P ν n , µ γ P ≤ C γ 2 n .
Therefore, when ν 1 , . . . , ν n are iid random measures with support included in a compact interval Ω, it follows that if γ = γ n is such that lim n→∞ γ 2 n n = +∞ then lim

n→∞ E(d 2 E µ γ P ν n , µ 0 P ) = 0.

Numerical experiments

We consider a simulated example where the measures ν i are discrete and supported on a small number p i of data points (5 ≤ p i ≤ 10). To this end, for each i = 1, . . . , n, we simulate a sequence (X ij ) 1≤j≤pi of iid random variables sampled from a Gaussian distribution N (µ i , σ 2 i ), and the µ i 's (resp. σ i ) are iid random variables such that -2 ≤ µ i ≤ 2 and 0 ≤ σ i ≤ 1 with E(µ i ) = 0 and E(σ i ) = 1/2. The target measure that we wish to estimate in these simulations is the population (or true) Wasserstein barycenter of the random distribution N (µ 1 , σ 2 1 ) which is N (0, 1/4) thanks to the assumptions E(µ 1 ) = 0 and E(σ 1 ) = 1/2. Then, let ν i = 1 pi pi j=1 δ Xij , where δ x is the Dirac measure at x. In order to compute the regularized barycenter, we solve (3) with an efficient minimization algorithm based on accelerated gradient descent (see [START_REF] Cuturi | A smoothed dual approach for variational wasserstein problems[END_REF]) for the computation of regularized barycenters in 1-D (see Appendix C in [START_REF] Bigot | Penalized barycenters in the Wasserstein space[END_REF]).

To illustrate the benefits of regularizing the Wasserstein barycenter of the ν i 's, we compare our estimator with the one obtained by the following procedure which we refer to as the kernel method. In a preliminary step, each measure ν i is smoothed using a standard kernel density estimator to obtain

f i,hi (x) = 1 p i h i pi j=1 K x -X ij h i , x ∈ Ω,
where K is a Gaussian kernel. The bandwidth h i is chosen by cross-validation. An alternative estimator is then defined as the Wasserstein barycenter of the smoothed measures with density f 1,h1 , . . . , f n,hn . Thanks, to the well-know quantile averaging formula, the quantile function F -1 n of this smoothed Wasserstein barycenter is given by F

-1 n = 1 n n i=1 F -1 f i,h i
where F -1 g denotes the quantile function of a given pdf g. The estimator F -1 n corresponds to the notion of smoothed Wasserstein barycenter of multiple point processes as considered in [START_REF] Panaretos | Amplitude and phase variation of point processes[END_REF]. The density of F -1 n is denoted by f n , and it is displayed in Fig. 2. Hence, it seems that a preliminary smoothing of the ν i followed quantile averaging is not sufficient to recover a satisfactory Gaussian shape when the number p i of observations per unit is small.

Alternatively, we have applied our algorithm directly on the (non-smoothed) discrete measures ν i to obtain the regularized barycenter f γ Pn defined as the minimizer of (2). For the penalty function E, we took either the negative entropy or a Dirichlet regularization. The densities of the penalized Wasserstein barycenters associated to these two choices for E and for different values of γ are displayed as solid curves in warm colors in Fig. 2. For both penalty functions and despite a small number of observations per experimental units, the shape of these densities better reflects the fact that the population Wasserstein barycenter is a Gaussian distribution.

Finally, we provide Monte-Carlo simulations to illustrate the influence of the number n = 100 of observed measures on the convergence of these estimators. For a given 10 ≤ n 0 ≤ n, we randomly draw n 0 measures ν i from the whole sample, and we compute a smoothed barycenter via the kernel method and a regularized barycenter for a chosen γ. For given value of n 0 , this procedure is repeated 200 times, which allows to obtain an approximation of the expected error E (d(μ, µ P )) of each estimator μ, where d is either d E or W 2 . The penalty used is a linear combinaison of Dirichlet and negative entropy functions. The results are displayed in Figure 3. It can be observed that our approach yields better results than the kernel method for both types of error (using either the Bregman or Wasserstein distance). In this paper, we have summarize some of the results of [START_REF] Bigot | Penalized barycenters in the Wasserstein space[END_REF]. We provide a study on regularized barycenters in the Wasserstein space, which is of interest when the data are irregular or for noisy probability measures. Future works will concern the numerical computation and the study of the convergence to a population Wasserstein barycenter for Ω ⊂ R d .

Fig. 1 .

 1 Fig. 1. (a) A subset of 3 smoothed neural spike trains out of n = 60. Each row represents one trial and the pdf obtained by smoothing each spike train with a Gaussian kernel of width 50 milliseconds. (b) Probability density function of the empirical Wasserstein barycenter νn for this data set.

Fig. 2 .

 2 Fig. 2. Simulated data from Gaussian distributions with random means and variances. In all the figures, the black curve is the density of the true Wasserstein barycenter. The blue and dotted curve represents the pdf of the smoothed Wasserstein barycenter obtained by a preliminary kernel smoothing step. Pdf of the regularized Wasserstein barycenter µ γ Pn (a) for 20 ≤ γ ≤ 50 with E(f ) = ||f || 2 (Dirichlet), and (b) for 0.08 ≤ γ ≤ 14 with E(f ) = f log(f ) (negative entropy) .

Fig. 3 .

 3 Fig. 3. Errors in terms of expected Bregman and Wasserstein distances between the population barycenter and the estimated barycenters (kernel method in dashed blue, regularized barycenter in red) for a sample of size n0 = 10, 25, 50 and 75.

Acknowledgment

This work has been carried out with financial support from the French State, managed by the French National Research Agency (ANR) in the frame of the GOTMI project (ANR-16-CE33-0010-01).