Louis Airale

A Comprehensive Multi-scale Approach for Speech and Dynamics Synchrony in Talking Head Generation

Animating still face images with deep generative models using a speech input signal is an active research topic and has seen important recent progress. However, much of the effort has been put into lip syncing and rendering quality while the generation of natural head motion, let alone the audio-visual correlation between head motion and speech, has often been neglected. In this work, we propose a multiscale audio-visual synchrony loss and a multi-scale autoregressive GAN to better handle short and long-term correlation between speech and the dynamics of the head and lips. In particular, we train a stack of syncer models on multimodal input pyramids and use these models as guidance in a multi-scale generator network to produce audioaligned motion unfolding over diverse time scales. Our generator operates in the facial landmark domain, which is a standard low-dimensional head representation. The experiments show significant improvements over the state of the art in head motion dynamics quality and in multi-scale audio-visual synchrony both in the landmark domain and in the image domain. Our code, models and demo will be made available on the project's GitHub page. 1

Introduction

Among the many computer vision tasks that have benefited from the breakthrough of deep learning, talking face generation, that aims to animate still images from a condi-1 https://github.com/LouisBearing/HMo-audio. tioning audio signal, has received considerable attention in the previous years. The advent of potent reenactment systems, as [START_REF] Siarohin | First order motion model for image animation[END_REF] or [START_REF] Ting-Chun | One-shot free-view neural talking-head synthesis for video conferencing[END_REF], and powerful loss functions allowing for a finer correlation between the generated lip motion and the audio input [START_REF] Son | Out of time: automated lip sync in the wild[END_REF] have paved the way for a new state of the art. In both tasks of talking head generation and face reenactment, where lip and head motion are given as a driving video sequence, it is customary to represent face dynamics in a low dimensional space [START_REF] Greenwood | Joint learning of facial expression and head pose from speech[END_REF][START_REF] Chen | Hierarchical cross-modal talking face generation with dynamic pixel-wise loss[END_REF][START_REF] Zhou | Makelttalk: speaker-aware talking-head animation[END_REF][START_REF] Zhang | Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset[END_REF][START_REF] Zakharov | Fast bi-layer neural synthesis of oneshot realistic head avatars[END_REF][START_REF] Ha | Marionette: Few-shot face reenactment preserving identity of unseen targets[END_REF][START_REF] Ting-Chun | One-shot free-view neural talking-head synthesis for video conferencing[END_REF][START_REF] Zhao | Sparse to dense motion transfer for face image animation[END_REF][START_REF] Meshry | Learned spatial representations for few-shot talking-head synthesis[END_REF]. For this reason recent breakthrough in face reenactment has also benefited the talking head synthesis task. The above approach assumes that image texture and face dynamics can be processed independently, and that all necessary cues to handle the dynamics fit on a low dimensional manifold. It is then a reliable strategy to treat audio-conditioned talking face synthesis as a two-step procedure, where the audio-correlated dynamics are first generated in the intermediate space of an off-the-shelf face reenactment model, which is later used to reconstruct photorealistic video samples [START_REF] Wang | Audio2head: Audio-driven one-shot talking-head generation with natural head motion[END_REF][START_REF] Wang | One-shot talking face generation from single-speaker audio-visual correlation learning[END_REF][START_REF] Ji | Eamm: One-shot emotional talking face via audio-based emotion-aware motion model[END_REF]. This allows to focus on improving the audio-visual (AV) correlation between the input speech signal and the produced face and lips movements in a much sparser space than that of real-world images.

Nevertheless, synthesising natural-looking head and lip motion sequences adequately correlated with an input audio signal remains a challenging task. In particular, although it has long been known that speech and head motion are tightly associated [START_REF] Hani C Yehia | Linking facial animation, head motion and speech acoustics[END_REF], only recently has this relation attracted the attention of the computer vision community. A likely reason for the difficulty of producing realistic head motion is the lack of an adequate loss function. So far, the most successful strategy to produce synchronized lip movements has relied on the maximization of the cross-modal correlation between short audio and output motion clips, measured by a pre-trained model [START_REF] Son | Out of time: automated lip sync in the wild[END_REF][START_REF] Hao Zhu | Arbitrary talking face generation via attentional audio-visual coherence learning[END_REF][START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF][START_REF] Se | Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory[END_REF][START_REF] Yin | Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan[END_REF]. This fails, however, to account for lower frequency motion as that of the head which remains quasi-static over the short duration considered, typically of the order of a few hundreds of milliseconds. Surprisingly, there was no attempt to generalize this approach beyond lip synchronization. Neither has possible multi-scale audio-visual correlation been explored in the talking face generation literature. Head motion is often produced through the use of a separate subnetwork trained to match the dynamics of a ground truth sequence, which in practice decouples the animation of head and lips.

We argue that to account for motion that unfolds over longer duration such as the head rhythm, a dedicated loss enforcing the synchrony of AV segments of various lengths is needed. We propose to implement this loss using a pyramid of syncers, replacing the lip-sync expert of [START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF] with a stack of syncer models evaluating the correlation between the audio input and the dynamics of the whole face over different time scales. One advantage of this syncer-pyramid loss function is that it allows to produce head and lip movements together; here one may train a single network endto-end on the dynamics of both head and lips, resulting in overall lighter architecture and training procedure. A natural way to exploit the gradients from the multi-scale AV correlation loss is then to construct a similar hierarchical structure in the generative model itself. The proposed method, hereafter labelled MS-Sync, is implemented in the landmark domain [START_REF] Bulat | How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)[END_REF]: for the reasons previously mentioned it is sufficient to parameterize the speech-correlated facial dynamics, which is the focus of the present work. Our generative model, loss functions and most of the metrics used to measure the quality and synchrony of the produced motion therefore apply in this domain. Although it is out of the scope of this study, several landmark-based real-world face reconstruction methods exist [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF][START_REF] Ha | Marionette: Few-shot face reenactment preserving identity of unseen targets[END_REF][START_REF] Zhao | Sparse to dense motion transfer for face image animation[END_REF][START_REF] Meshry | Learned spatial representations for few-shot talking-head synthesis[END_REF]. Last, in contrast with the current trend in talking face synthesis, we rely on an autoregressive generative network for its inherent ability to model sequential dependencies, and its flexibility to handle sequences of arbitrary length. To do so, we build on the autoregressive Generative Adversarial Network (GAN) baseline of [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF], and show that the conditioning speech signal has a stabilizing effect that hinders error accumulation on a much longer term than in the unconditional setting. In particular, we demonstrate experimentally that the error drift can be mitigated on test sequences more than five times the length of the training sequences. More importantly, we show that the proposed model, coupled with the multi-scale discriminator of [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF], largely outperforms the state of the art in terms of multi-scale audio-visual correla-tion and head dynamics quality.

The contributions of the present work are:

• A multi-scale audio-visual correlation loss based on a pyramid of syncer networks,

• A multi-scale autoregressive GAN for the generation of speech-synchronized head and lip motion in the 2Dlandmarks domain with minimal error accumulation,

• Extensive experiments on three datasets that show that our architecture outperforms previous works on all metrics related to both quality of head dynamics and AV correlation.

Related Work

Talking head generation

The task of animating a human face with a neural network can be either guided, when the head motion comes from a driving sequence, or unguided, in which case the head and lip motion must be inferred by the generative model from other modalities. Compelling results have been achieved over the years to improve the photorealistic rendering of guided methods [START_REF] Zakharov | Few-shot adversarial learning of realistic neural talking head models[END_REF][START_REF] Siarohin | First order motion model for image animation[END_REF][START_REF] Guo | Ad-nerf: Audio driven neural radiance fields for talking head synthesis[END_REF][START_REF] Yurui Ren | Pirenderer: Controllable portrait image generation via semantic neural rendering[END_REF]. Among these, several works rely on low dimensional representations, e.g. facial landmarks [START_REF] Zhao | Sparse to dense motion transfer for face image animation[END_REF][START_REF] Meshry | Learned spatial representations for few-shot talking-head synthesis[END_REF][START_REF] Zakharov | Fast bi-layer neural synthesis of oneshot realistic head avatars[END_REF], learned keypoints [START_REF] Siarohin | First order motion model for image animation[END_REF][START_REF] Ting-Chun | One-shot free-view neural talking-head synthesis for video conferencing[END_REF], or morphable models [START_REF] Zhang | Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset[END_REF] to handle the dynamics, which are later used to warp or normalize the style of the source identity image. On the other hand, the primary focus of audiodriven talking head synthesis has been on syncing output lip movements and input speech signal, either leaving visual reenactment as a separate task, or limiting it to static pose scenarios [START_REF] Taylor | A deep learning approach for generalized speech animation[END_REF][START_REF] Karras | Audio-driven facial animation by joint endto-end learning of pose and emotion[END_REF][START_REF] Suwajanakorn | Synthesizing obama: learning lip sync from audio[END_REF][START_REF] Greenwood | Joint learning of facial expression and head pose from speech[END_REF][START_REF] Song | Talking face generation by conditional recurrent adversarial network[END_REF][START_REF] Hao Zhu | Arbitrary talking face generation via attentional audio-visual coherence learning[END_REF][START_REF] Zhou | Talking face generation by adversarially disentangled audio-visual representation[END_REF][START_REF] Vougioukas | Realistic speech-driven facial animation with gans[END_REF][START_REF] Fan | Faceformer: Speech-driven 3d facial animation with transformers[END_REF]. For this reason, there has been comparatively few endeavors to generate realistic head motion [START_REF] Chen | Talking-head generation with rhythmic head motion[END_REF][START_REF] Zhou | Makelttalk: speaker-aware talking-head animation[END_REF]. As a noticeable improvement over previous research, recent works showed very promising results producing rich head motion in a low-dimensional keypoint space in combination with proficient visual reenactment systems [START_REF] Wang | Audio2head: Audio-driven one-shot talking-head generation with natural head motion[END_REF][START_REF] Wang | One-shot talking face generation from single-speaker audio-visual correlation learning[END_REF]. However there remains a margin of improvement in particular in the diversity of output head motion, and in the time alignment between speech, lips and head motion over different time scales, which has never been addressed before.

Learning to align speech and head dynamics

Two trends coexist regarding the syncing of audio and face dynamics. Originally, learning audio-correlated lip movements was only done with a mean squared error loss to the ground truth sequence [START_REF] Chen | Hierarchical cross-modal talking face generation with dynamic pixel-wise loss[END_REF][START_REF] Das | Speech-driven facial animation using cascaded gans for learning of motion and texture[END_REF][START_REF] Zhou | Makelttalk: speaker-aware talking-head animation[END_REF][START_REF] Yi | Audio-driven talking face video generation with learning-based personalized head pose[END_REF][START_REF] Ji | Eamm: One-shot emotional talking face via audio-based emotion-aware motion model[END_REF][START_REF] Sinha | Emotion-controllable generalized talking face generation[END_REF]. In parallel, following SyncNET [START_REF] Son | Out of time: automated lip sync in the wild[END_REF], the use of contrastive loss variants turned out to be a strong alternative for its effectiveness on cross-modal training tasks [START_REF] Zhou | Pose-controllable talking face generation by implicitly modularized audio-visual representation[END_REF][START_REF] Wang | Progressive disentangled representation learning for fine-grained controllable talking head synthesis[END_REF]. In particular, in [START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF] the authors proposed to train a lip-sync expert network to regress the cross-modal alignment between short audio and video segments. The expert would later be frozen during the generative model training phase, and used as a loss function to enhance output audio-visual alignment. This strategy was later employed successfully in several works [START_REF] Yin | Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan[END_REF][START_REF] Wang | One-shot talking face generation from single-speaker audio-visual correlation learning[END_REF][START_REF] Se | Synctalkface: Talking face generation with precise lip-syncing via audio-lip memory[END_REF]. We argue however that the commonly used segment length of 200 ms is insufficient to properly align lower-frequency movements like that of the head, for which several such syncer networks operating on various segment lengths are required.

Multi-scale data processing

Learning on representations of the input data over multiple scales has become the standard in computer vision tasks such as object detection or semantic segmentation where objects of the same class can have different sizes [START_REF] Lin | Feature pyramid networks for object detection[END_REF][START_REF] Tan | Efficientdet: Scalable and efficient object detection[END_REF]. In the generative models literature, multi-scale approaches may either be implemented in the discriminator network of GANs as a way to improve multi-scale faithfulness of generated data [START_REF] Ting-Chun | High-resolution image synthesis and semantic manipulation with conditional gans[END_REF][START_REF] Lin | Human motion modeling using dvgans[END_REF][START_REF] Kumar | Melgan: Generative adversarial networks for conditional waveform synthesis[END_REF] but also in the generative model itself [START_REF] Emily L Denton | Deep generative image models using a laplacian pyramid of adversarial networks[END_REF][START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF]. Although this was not explored so far in talking head generation, multi-scale feature hierarchies can be readily computed to align speech and dynamics of various motion frequencies.

Method

Given a set of initial landmark coordinates x 0 ∈ R 2L (the 2D coordinates of the L = 68 landmarks) and a conditioning audio signal a 0:T = (a 0 , . . . , a T ) ∈ R d×T (here d = 26) over T time steps, we aim to produce a sequence of landmark positions x 1:T such that the joint distributions over generated and data samples match:

p g (x 0:T , a 0:T ) = p data (x 0:T , a 0:T ), ∀x 0:T , a 0:T . (1)
In this section we describe our procedure to tackle this problem as follows. We start by introducing in 3.1 the multi-scale AV synchrony loss which is the major contribution of the present work. Then in 3.2 we propose a multiscale generator architecture able to exploit appropriately the devised multi-scale AV loss. Finally section 3.3 details our overall training procedure.

Multi-scale audio-visual synchrony loss

The most prominent procedure to align dynamics with speech input relies on the optimization of a correlation score computed on short audio-visual segments of the generated sequence using a pre-trained AV syncer network [START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF]. Several contrastive loss formulations are possible to train the syncer network, that suppose the maximization of the agreement between in-sync AV segments or positive pairs (a t , x t ) versus that of out-of-sync or negative pairs. One particularly interesting formulation is the Info Noise Contrastive Estimation loss, that maximizes the mutual information between its two input modalities [START_REF] Van Den Oord | Representation learning with contrastive predictive coding[END_REF]. Given a set X = (a t , x t , x neg 1 , . . . , x neg N ) containing a positive pair and N negative position segments, this loss writes:

L InfoNCE = -E X e S(at,xt) e S(at,xt) + N n=1 e S(at,x neg n ) , (2)
with S the syncer model score function, which is hereafter implemented as the cosine similarity of the outputs from the audio and position embeddings e a and e x of the syncer network:

S(a t , x t ) = e a (a t ) e x (x t ) ||e a (a t )||.||e x (x t )|| . (3)
Following the usual practice, we take a t and x t respectively as the MFCC spectrogram and position segment of a 200 ms window centered on time step t. Negative pairs can be indifferently misaligned audio and position segments from the same sequence sample, or segments from different samples, and N is hereafter fixed to 48.

Once trained, the weights of e a and e x are frozen and the following term is added to the loss function of the generative model:

L AV = -E at,xt S(a t , x t ), (4)
where a t is now part of the conditioning signal and x t is output by the model. The above procedure is insufficient when one needs to discover AV correlations over different time scales. A straightforward extension consists in building multi-scale representations of the audio-visual inputs and training one syncer network S i for each level i in the resulting pyramid. The training process of the pyramid of syncers is represented in Figure 1 (left). Specifically, the audio and landmark position hierarchies {a i 0:T /2 i-1 } i and {x i 0:T /2 i-1 } i are constructed by successive passes through an average pooling operator that blurs and downscales its input by a factor 2, e.g. for positions:

x i t = 1 2k + 1 k τ =-k x i-1 2t+τ (5)
where we choose k = 3. The objective is to progressively blur out the highest frequency motion when moving upward in the pyramid, forcing the top level syncers to exploit better the rhythm of the head motion. A total of four syncer networks are trained on the input pyramid following (2), input segment duration ranging from the standard 200 ms on the bottom level to 1600 ms at the coarsest scale.

After the training of the pyramid of syncers, all networks S 1 to S 4 are frozen and used to compute the multi-scale audio-visual synchrony loss. The principle of this loss is presented in Figure 1 (right). Similar to the input pyramids used to train the syncer networks, we construct a multi-scale representation of the input speech a 0:T and the generated landmark positions x 0:T . Then for each hierarchy level i one loss term L i AV is computed according to (4) using pretrained syncer S i . Those terms are then averaged to give the overall multi-scale AV synchrony loss L MS AV . To better exploit the effects of this loss, we propose a multi-scale autoregressive generator network that we describe in the following section.

Multi-scale autoregressive generator

Through the multi-scale synchrony loss, the generator receives gradients that push it to produce audio-synced landmark positions over multiple time scales. In this section, we describe the architecture of our generator network, which is itself implemented with a multi-scale structure to better exploit the loss gradients. The overall architecture is described in Figure 2.

The proposed generative model is inspired from SUHMo [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF] which implements an autoregressive model to generate facial landmark velocities. This however requires substantial adaptations to deal with the present multimodal data. Very generally, given landmark positions x 0:t until time step t and next frame audio input a t+1 , the generator G produces instantaneous velocities v t+1 :

v t+1 = G(x 0:t , a t+1 ), (6)
x t+1 = x t + v t+1 . (7)
As depicted in Figure 2, G is constituted of a temporal module operating on a sequence of landmark positions, and of a multi-scale module that takes the output of the temporal module h t , the positions x t and audio a t+1 as input to produce v t+1 . We implement the multi-scale module as the bottom-up path of a Feature Pyramid Network [START_REF] Lin | Feature pyramid networks for object detection[END_REF]. Namely, the input spectrogram is processed by several downsampling convolutional layers, producing feature maps a 1 0:T to a 4 0:T /2 3 of the same resolution as those used to compute the AV loss pyramid. Feature maps 2 to 4 are later interpolated back to the length T of the finest map, such that one vector a i t+1 can be extracted from each pyramid level i to produce the next step velocity. Concretely, each vector a i t+1 is concatenated with x t and h t and is processed by an independent fully connected branch, the rationale being that processing each input resolution separately would allow the model to produce different motion frequencies.

The outputs of the four branches of the multi-scale generator are merged using a learnable soft spatial mask. Each branch i outputs a velocity vector v i ∈ R 2L (note that time index is omitted for the sake of clarity) and a mask vector w i ∈ R 2L such that w i 2k-1 = w i 2k , ∀k ≤ L, responsible for enhancing or weakening the contribution of each landmark on the given branch. This is because we expect facial regions to play different roles depending on the scale: the finest resolution branch might emphasize lip landmarks, while at the coarsest scale, more weight may be put on head contour. The output of the multi-scale module finally writes:

v t+1 = 4 i=1 e w i j e w j v i (8)

Overall architecture and training

In addition to the AV synchrony loss L MS AV , we make use of the discriminator networks proposed in [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF], that proved effective to train an autoregressive generator on landmark sequences. These consist in one frame discriminator D f which computes the realism of static landmarks, and two window-based multi-scale networks D s and D j s on sequences. The difference between those is that D j s processes pairs of samples to help reducing mode collapse [START_REF] Lin | Pacgan: The power of two samples in generative adversarial networks[END_REF]. Although in our audio-conditioned setting mode collapse is at most a minor issue, we found that using this additional Table 1. Training dataset and head motion preprocessing and generation for the considered methods. The preservation of head translation typically implies re-creating the datasets from the original sources following the strategy in [START_REF] Siarohin | First order motion model for image animation[END_REF].

Method

Train Dataset Head transl. in prepro.

Predicted head motion

Wav2Lip [START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF] LRS2 × × PC-AVS [START_REF] Zhou | Pose-controllable talking face generation by implicitly modularized audio-visual representation[END_REF] VoxCeleb2 (I)

× × MakeItTalk [50] VoxCeleb2 (I) × Audio2Head [37] VoxCeleb2 (II)
OSTF [START_REF] Wang | One-shot talking face generation from single-speaker audio-visual correlation learning[END_REF] Obama Weekly Address MS-Sync (ours)

VoxCeleb2 (II)
loss slightly improves the dynamics quality. Adversarial losses are implemented with the geometric GAN formulation of [START_REF] Lim | Geometric gan[END_REF]. Namely, given the generated and ground truth landmark position distributions p g and p data , the generator losses write:

L G f = -E x 0:T ∼pg   1 T t≥1 D f (x t )   , (9)
L Gs = -E x 0:T ∼pg [D s (x 0:T )] , (10)
L G j s = -E x 0:T ∼pg,x 0:T ∼pg D j s (x 0:T , x 0:T ) , (11)
as for the generic discriminator loss:

L D * = E x∼pg [max(0, 1 + D * (x))] + E x∼pdata [max(0, 1 -D * (x))] , (12)
where D * is replaced respectively by D f , D s and D j s and sequences are sampled according to equations 9 to 11. Additionally, we trained with the weak L 2 reconstruction loss in [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF] but found no significant improvement. Overall, training consists in minimizing alternatively the two following terms:

L D = L D f + L Ds + L D j s , (13)
L = λL MS AV + L G f + L Gs + L G j s , (14)
with λ = 8 in all experiments.

Experiments

We conducted three benchmark evaluations to measure the proficiency of our model, assessing respectively head dynamics quality, multi-scale AV synchrony in the landmark domain, and AV synchrony in the image domain.

Experimental protocol

Datasets. Experiments are conducted on two versions of the VoxCeleb2 dataset [START_REF] Chung | Voxceleb2: Deep speaker recognition[END_REF] with different preprocessing. The first version, labelled VoxCeleb2 (I), follows the standard preprocessing that centers the face in every frame. Second, we use the preprocessing strategy in [START_REF] Siarohin | First order motion model for image animation[END_REF] to re-generate subsets of respectively ∼18k and 500 short video clips from the original VoxCeleb2 train and test sets. The interest of this preprocessing method is that it keeps the reference frames fixed, thus preserving head motion. We refer to this second version as VoxCeleb2 (II). HDTF dataset [START_REF] Zhang | Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset[END_REF] contains ∼400 long duration frontal-view videos from political addresses, which despite limited dynamics diversity makes it suitable for AV correlation measurements. Last, we use LRS2 [START_REF] Afouras | Deep audio-visual speech recognition[END_REF], which is preprocessed similarly to VoxCeleb2 (I), to measure the AV synchrony in the image space.

Benchmark models. We compare our method, MS-Sync, with the following prominent speech-driven talking head generation models. Wav2Lip [START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF] uses a pre-trained lip syncer to learn the AV synchrony, and achieved state-ofthe-art performances on the visual dubbing task. However, MakeItTalk [START_REF] Zhou | Makelttalk: speaker-aware talking-head animation[END_REF] was one of the first successful attempts to produce speech-correlated head motion. Its dynamics are learned in the landmark domain on VoxCeleb2 (I), i.e. no head translation was seen at training. Audio2Head [START_REF] Wang | Audio2head: Audio-driven one-shot talking-head generation with natural head motion[END_REF] and its follow-up model OSTF [START_REF] Wang | One-shot talking face generation from single-speaker audio-visual correlation learning[END_REF] propose methods to generate vivid dynamics, learning head motion and AV synchrony in a sparse keypoint space using a two-step training procedure. Audio2Head dynamics module is trained on VoxCeleb2 (II), while OSTF is trained on a single identity, namely using Obama Weekly Address dataset. As a noticeable improvement over Audio2Head, in OSTF AV synchrony is controlled with the contrastive loss of [START_REF] Kr Prajwal | A lip sync expert is all you need for speech to lip generation in the wild[END_REF]. An overview of the different models can be found in Table 1.

Training details. The temporal module introduced in Section 3 is implemented as a 1-layer LSTM with hidden size 256. All convolutions and fully connected layers are implemented as 1D ConvNeXt blocks (with kernel size 1 for dense layers) [START_REF] Liu | A convnet for the 2020s[END_REF]. We trained two versions of our model, varying only the training sequence length from 40 to 80 frames, resulting in MS-Sync-short and MS-Sync-long: the aim is to see how this affects the quality of the produced sequences on various output lengths. Models were trained on VoxCeleb2 (II) for 70k iterations (about 500 epochs) using Adam optimizers with β 1 = 0 and β 2 = 0.999 and learning rates 2 × 10 -5 and 1 × 10 -5 respectively for the generator and the discriminator, after which a decay factor of 0.1 was applied on the learning rates for 5k additional iterations. All audio inputs are sampled at 16 kHz, and to generate the 26dimensional MFCC spectrogram we used a window size of 400 and hop size of 160.

Dynamics quality

Protocol. The quality of the produced dynamics is evaluated on the 500 videos of VoxCeleb2 (II) test set, which preserve head motion. The Fréchet Inception Distance (FID) is used to measure static face realism, while the Fréchet Video Distance (FVD) and temporal FID [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF] metrics measure the distance between the distributions of data and generated motion. The two latter metrics require a fixed sequence length that we set to either 40 or 80 frames (equivalent to 1.6 s and 3.2 s at 25 fps), and we refer to the resulting metrics as FVD 40 (t-FID 40 ) and FVD 80 (t-FID 80 ), respectively. When generating longer sequences, we measure the FVD 40 , t-FID 40 and FID on the last 40 frames. Image-rasterized landmarks are used to compute the metrics (see Figure 4 and [START_REF] Airale | Autoregressive gan for semantic unconditional head motion generation[END_REF]).

Results. The results of the dynamics quality evaluations are reported in Table 2. MS-Sync-short shows similar FVD and t-FID scores to OSTF but significantly better FID, especially on 40 and 80 frame sequences. Since the faces produced by OSTF are also very sharp, we interpret this result as a hint that this model lacks diversity. Audio2Head suffers from the same limitation to an even greater extent: although visually compelling, the movements it produces are stereotypical and therefore penalized by their too small variance in the Fréchet distance calculations. On the other hand, MiT performs well in FID but its dynamics are of noticeable lower quality. Finally, the MS-Sync-long results show that a mere change in training strategy allows to greatly reduce error accumulation over 200 time steps, although it is here at the cost of a slightly lower quality on shorter sequences.

Method |AV-Off 1 | ↓ AV-Conf 1 ↑ |AV-Off 2 | ↓ AV-Conf 2 ↑ |AV-Off 3 | ↓ AV-Conf 3 ↑ |AV-Off 4 | ↓ AV-
Method |AV-Off 1 | ↓ AV-Conf 1 ↑ |AV-Off 2 | ↓ AV-Conf 2 ↑ |AV-Off 3 | ↓ AV-Conf 3 ↑ |AV-Off 4 | ↓ AV-Conf 4 ↑ Static 9.

Landmark-domain multi-scale AV synchrony

Protocol. Ideal multi-scale AV synchrony scores should convey how much a model succeeds in exploiting the audio signal to produce motion over diverse time scales. To that end, we resort to audio-visual datasets which preserve motion dynamics, namely VoxCeleb2 (II) and HDTF [START_REF] Zhang | Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset[END_REF], and carry our evaluations in the landmark domain. We split HDTF into 291 and 51 train and test identities, and further split the test videos into 1058 80-frame clips. Likewise, measurements on VoxCeleb2 (II) are made on sequences of 80 frames. Pyramids of AV syncers dedicated to metrics calculation, equivalent to landmark-domain SyncNETs [START_REF] Son | Out of time: automated lip sync in the wild[END_REF] on full faces, are trained beforehand on both datasets. Note that contrary to Section 3, we use the triplet loss of [START_REF] Son | Out of time: automated lip sync in the wild[END_REF] to train the syncers. The AV synchrony is evaluated using the absolute value of the audio-visual offset (|AV-Off|) and the confidence score (AV-Conf) introduced in [START_REF] Son | Out of time: automated lip sync in the wild[END_REF], measured at four different scales via the syncer pyramids on successively downsampled audio-visual chunks of duration 200 ms, 400 ms, 800 ms and 1600 ms. Hence an offset of 1 at the finest resolution, sampled at 25 fps, amounts to a misalignment of 40 ms between modalities, whereas at the coarsest scale this rises to 320 ms. We report in Figure 3 the performances of the first level syncer on VoxCeleb2 (II) test set. The distributions of ground truth synchrony scores are closer to Gaussian than perfect Dirac. This is partly because the same audio signal may correspond to more than one facial configuration, and the syncers may not fully grasp this diversity. The synchrony scores reported here should therefore be viewed in light of this assumption: rather than an actual finer AV alignment, results that appear "better" than the ground truth instead correspond to a closer match to the modes of the AV distribution discovered by the syncers.

Results. The AV synchrony scores are reported in

Image-domain AV synchrony

Protocol. In a third batch of experiments the synchrony is calculated in the image domain, similar to the classical evaluation protocol. To do so, we first map the landmarks output by our model in the image space using the reenactment system from MakeItTalk (hereafter MiT [START_REF] Zhou | Makelttalk: speaker-aware talking-head animation[END_REF]). Although this leads to blurry results when the pose changes from the initial orientation, we found it sufficient for the sake of AV synchrony measurements. We use two datasets for these experiments: a subset of 2141 videos from the original test set of VoxCeleb2 (I), and LRS2. To cope with the imbalanced duration of VoxCeleb2 videos and keep computation time manageable, we work with the first 40 frames in each clip, while we use the whole LRS2 test set, which contains shorter videos. In addition to the absolute AV offset and confidence score given by SyncNET, we compute the Landmark Distance (LMD [START_REF] Chen | Lip movements generation at a glance[END_REF]), together with a frontalized version LMD front that better accounts for face rotation. For a fair comparison, we do not directly use the landmarks produced by MS-Sync to measure the LMD but extract it back from the reenacted video clips; the same procedure is applied to the ground truth landmarks to help assess the effects of each of the previous steps on the metrics.

Results.

Although not the primary scope of our study, the landmarks produced by MS-Sync and reenacted with MiT behave surprisingly well in the image domain (Table 5). MS-Sync outperforms all other models in terms of AV offset on both VoxCeleb2 (I) and LRS2, simply falling short of Wav2Lip in terms of AV confidence on the two datasets, and of LMD on VoxCeleb2. In particular, MS-Sync performs better than all other models with head motion (and notably MakeItTalk) on all considered metrics. Notice also how the fact that Wav2Lip leaves the whole input face beyond the lips intact seems to bias the calculation of AV confidence in its favor, especially when considering the MiT-reenacted ground truth landmarks. This suggests that SyncNET is sensitive to the image sharpness: the shortcomings of the image reenactment systems probably set limits on the achievable offset and confidence values.

Qualitative results

In Figure 4 we present several sequences output by different models and the corresponding truth sequences over 120 frames. An examination of these examples shows that mouth closing and opening produced by MS-Sync look correctly aligned with the original, but interestingly this also seems to be the case for head motion although the loss only enforces convergence of distributions. Although the motion produced by OSTF is qualitatively good, it is slightly less diverse and tends to frontalize the face disregarding the original orientation. Wav2Lip, on the other hand, only synchronizes the lips.

Ablation study

In this section we explore the roles of the multi-scale AV synchrony loss and of the multi-scale generator on the output results, and in particular in AV confidence at different resolutions. As can be seen in Figure 5 for the evolution of the validation AV confidence along training, almost no difference is visible at the finest resolution between the full MS-Sync model and its single-scale loss, single-scale generator equivalent. However as expected the confidence of the latter model falls significantly below as one moves upward in the feature pyramid as the loss does not explicitly enforce multi-scale synchrony. It is possible to circumvent this effect by enabling the multi-scale AV synchrony loss, however it is clear that if the generator is not itself a multiscale network, it lacks capacity to fully exploit the loss, resulting in average performances on every scale.

Conclusion

The approach proposed in this work is the first attempt to learn and model audio-visual correlations at multiple scales for talking head generation. This is enabled thanks to a pyramid of syncer models that are trained on hierarchical representations of input audio and landmark position sequences, and then used to compute the loss for the training of the generative model. Importantly, we showed that this model should also be built on a multi-scale backbone, implemented here as a feature pyramid network together with individual branches for each pyramid level that are merged using a soft learnable mask. The very encouraging results of MS-Sync let us foresee numerous applications of similar approaches on other audio-visual generation tasks. One research direction could thus consist in replacing the facial landmarks with other quantities, be it low dimensional keypoints or body joints, or real-world images. Another orthogonal direction may lead to extend the focus to additional cross modal relationships, such as audio-visual emotions.

Figure 1 .

 1 Figure 1. Left. A stack of syncer networks S i are trained on multi-scale positive and negative multimodal pairs using contrastive losses. Right. The syncer models are frozen and used to compute the multi-scale audio visual synchrony loss of the generative model.

Figure 2 .

 2 Figure 2. Left. Our network is composed of a temporal module and a multi-scale module. Right. Details of the multi-scale module.

Figure 3 .

 3 Figure 3. Distributions of confidence (AV-Conf1) and offset (AV-Off1, without absolute value) scores measured on VoxCeleb2 (II) test dataset by our metric syncer (equivalent to landmarkdomain SyncNET [8]) at the finest time scale.

Figure 4 .

 4 Figure 4. Qualitative comparison of the results produced by different methods on three VoxCeleb2 (II) test sequences.

Figure 5 .

 5 Figure 5. Evolution of multi-scale audio-visual confidence over training, measured on VoxCeleb2 (II) validation set. Top is the finest scale, bottom the coarsest.

Table 2 .

 2 Dynamics quality of rasterized landmarks on VoxCeleb2 (II) test set. All metrics need to be minimized. Bold indicates best score, underline second best. We additionally report the static face Wav2Lip results as reference scores for the reader. FID t-FID 40 FVD 40 FID t-FID 40 FVD 80 t-FID 80 FVD 40 FID t-FID 40

	Duration (frames)		40				80				200
	Method FVD 40 Methods that predict head motion
	MakeItTalk [50]	236	4.0	107	234	3.2	101	476	133	224	4.2	114
	Audio2Head [37]	406	66.4	109	593	82.5	133	682	149	649	92.5	141
	OSTF [38]	113	12.4	36	164	25.4	50	249	33	225	30.5	65
	MS-Sync-short	105	3.3	52	126	5.5	47	239	41	279	25.2	68
	MS-Sync-long	134	6.6	42	104	6.8	39	257	28	144	8.3	47
	Methods with fixed head pose
	Wav2Lip [27]	263	0.9	167	261	1.0	167	557	188	265	3.2	182
	it only reenacts the lip region and therefore does not pro-
	duce any head motion. Similarly, PC-AVS [49] produces
	speech-synchronized talking head videos using a driving
	head motion sequence, mapping the results directly in im-
	age space without any explicit intermediate representation.

Table 3 .

 3 Landmark domain multi-scale AV synchrony on VoxCeleb2 (II) test set. A separate syncer model inspired from SyncNet is trained in the landmark domain for each corresponding time scale and used to compute the correlation scores.

	Time scale	200 ms (1)	400 ms (2)	800 ms (3)	1600 ms (4)

Table 4 .

 4 Conf 4 ↑ Landmark domain multi-scale AV synchrony on HDTF test set. As above, separate syncer models inspired from SyncNet are trained on each time scale on HDTF dataset and used to compute the correlation scores.

	Static	9.45±4.49	0.63±0.23 10.22±4.17 1.07±0.34	9.01±4.21	1.06±0.35	5.72±4.26	0.99±1.11
	Wav2Lip [27]	0.47±0.50	2.42±0.43	0.03±0.51	2.75±0.62	1.47±3.82	1.76±0.71	3.67±4.22	1.48±1.61
	MakeItTalk [50]	1.55±2.75	1.36±0.59	2.48±4.74	1.71±0.70	4.63±5.81	1.47±0.68	4.02±4.26	1.27±1.42
	Audio2Head [37] 2.58±0.51	1.67±0.48	2.41±3.34	1.88±0.64	3.74±4.89	1.45±0.63	3.43±4.10	1.39±1.43
	OSTF [38]	2.13±2.26	1.38±0.50	3.15±4.53	1.62±0.64	5.47±5.42	1.24±0.50	3.62±4.07	1.40±1.45
	MS-Sync-short	0.0±0.0	3.37±0.41	0.01±0.08	3.62±0.52	0.07±0.67 2.48 ± 0.74 2.60 ± 3.92 2.18 ± 1.85
	MS-Sync-long	0.01 ± 0.08 3.26 ± 0.41	0.01±0.08 3.56 ± 0.51 0.10 ± 0.82 2.49±0.71 1.84 ± 3.38 2.20±1.74
	Ground truth	0.87±0.93	2.41±0.46	0.52±1.40	2.50±0.62	2.39±4.31	1.57±0.67	2.97±3.83	1.69±1.64
	Time scale	200 ms (1)	400 ms (2)	800 ms (3)	1600 ms (4)

Table 5 .

 5 Image domain AV synchrony. † We rescaled the extracted landmarks from PC-AVS that are cropped differently by a factor 0.75 to make it comparable with the original data scale.

	Dataset		VoxCeleb2 (I)			LRS2
	Method	|AV-Off| ↓	AV-Conf ↑	LMD ↓	LMD front ↓	|AV-Off| ↓	AV-Conf ↑	LMD ↓	LMD front ↓
	Ground truth	1.89±1.92	6.29±1.66	0.0	0.0	0.08±0.4	8.36±1.62	0.0	0.0
	GT landmarks + MiT	3.52±4.50	3.55±1.52	1.60±0.30	1.53±0.29	1.72±3.91	4.61±1.70	1.60±0.31	1.54±0.30

Table 3

 3 and 4 for VoxCeleb2 (II) and HDTF, respectively. We did not include PC-AVS in this section because of distinct cropping strategies producing inconsistent results. Although the loss functions are different, the syncer pyramid used to train the model and the one which serves to compute the met-Audio2Head the best |AV-Off 2 | at the second scale, MS-Sync possesses the second best scores and largely outperforms all models in terms of AV confidence. What is more, the gap in favor of our model increases at the two coarsest scales, highlighting the effectiveness of the proposed approach to correlate input speech and generated motion on multiple time scales. Last it is noteworthy that although it produces no head motion, Wav2Lip results remain way above the static, uncorrelated boundary even at the coarsest resolution. This means that the mouth region is still partly informative at the top pyramid level, possibly pleading for a stronger blurring strategy.

rics were both trained on VoxCeleb2 (II): our model almost perfectly learned to optimize this loss, hence the very strong AV correlation scores. The HDTF results expose the generalization abilities of the different methods. Although Wav2Lip presents the best |AV-Off 1 | at the first scale and