Théo Ladune

P Philippe

W Hamidouche

L Zhang

O Déforges

ModeNet: Mode selection network for learned video coding IEEE MLSP 2020

Chapitre 2 : Algorithmes de compression vidéo conventionnels Depuis les années 90, le Moving Picture Experts Group (MPEG) et l'International Telecommunication Union (ITU-T) ont proposé plusieurs standards de compression vidéo. Advanced Video Coding (AVC) [8] a été finalisé en 2003, suivi par High Efficiency Video Coding (HEVC) [9] en 2013 et récemment Versatile Video Coding (VVC) [10] en 2020. Les standards successifs de MPEG/ITU ont continûment amélioré un même paradigme, désigné dans ce manuscrit comme la compression vidéo conventionnelle. viii Commençons par décrire la façon dont les systèmes de compression conventionnels traitent une image seule, appelée une image intra (I). L'image est tout d'abord partitionnée en différents blocs. Chacun des blocs est ensuite prédit d'après les blocs de l'image déjà transmis. Puis, le résidu (erreur de prédiction) est transmis par un processus de codage composé d'une transformée, d'une quantification et d'un codage entropique. Les étapes de prédiction et de transformée présentent deux caractéristiques importantes : • Elles sont linéaires, offrant ainsi une complexité limitée ; • Différentes prédictions et transformées sont disponibles. Les plus adaptées pour le bloc sont choisies par l'encodeur afin d'optimiser le coût débit distorsion. Pour des raisons pratiques, les différentes étapes sont optimisées séparément, ce qui peut entraîner des choix sous-optimaux par rapport à une optimisation globale du système. La dimension temporelle d'une vidéo présente de nombreuses redondances, qui doivent être réduites afin d'abaisser le débit total. Pour ce faire, une prédiction temporelle, implémentée par un algorithme de compensation de mouvement, est mise en place. Un mouvement translationnel est appliqué à un bloc de référence issu d'une autre image, préalablement transmise. Ce mouvement indique les déplacements entre le bloc de référence et le bloc à coder. Comme il doit être transmis au décodeur, sa valeur et sa précision sont déterminées pour optimiser le coût débit distorsion. Une prédiction bi-directionnelle est souvent implémentée afin d'améliorer la precision de la prédiction. Elle repose sur une moyenne de deux compensations de mouvement intermédiaires. Enfin, le résidu issu de la prédiction temporelle est transmis de manière similaire aux images intra.

 présente un exemple de structure de codage appelée Random Access. Une image s'appuyant sur d'autres images déjà reçues, est appelée une image inter. On différencie ensuite les images P (utilisant une seule image de référence), des images B (utilisant deux images de référence). La structure de codage doit répondre à un certain nombres de contraintes pratiques. Par exemple, l'utilisation d'images de références futures introduit une latence dans le processus de décodage qui peut être prohibitive pour certaines applications (par exemple en visioconférence).

Les systèmes de compression vidéo conventionnels se doivent de répondre à un large éventail de contraintes telles que la complexité ou la latence. En conséquence, les principes exposés dans le Chapitre 1 sont implémentés avec des limitations : opérations linéaires,

to me, during which he has provided me with so many advices, recommandations and ideas. Our insightful discussions have been extremely valuable to me and are the source of some of the findings of this thesis.

I am grateful to Jenny Benois-Pineau and Marc Antonini for having accepted to review this manuscript. I also thank the other jury members for accepting to be part of the jury. I would also like to thank all my colleagues from Orange, who played in key role in the accomplishment of this thesis. In particular, I am indebted to Patrick Boissonade and Rémi Rouaud for the design and the maintenance of the GPU cluster. Without them, it would not have been possible to carry out all the work presented in this manuscript. A special thanks goes to Corentin Kervadec, which has made my stay at Orange even more enjoyable. I have greatly benefited from the valuable exchanges I have had with Félix Henri, Gordon Clare, Matthieu Gendrin and Stéphane Pateux about video coding and deep learning. Finally, I would like to express my gratitude to Didier Tattevin for being always available and for the particular attention he has showed to me, particularly during the Covid-19 outbreak. I want to thank my PhD supervisors from the VAADER team, Wassim Hamidouche, Lu Zhang and Olivier Déforges for their assistance, especially during the writing of articles and of this manuscript. Our monthly meetings have always been the opportunity to receive relevant comments and interesting ideas.

My final thanks go to Lisette, to my friends and to my parents.

iii

RÉSUMÉ EN FRANÇAIS

Contexte

L

A quantité toujours croissante de données transmises sur Internet représente une contribution significative au changement climatique. Avant même la pandémie de la Covid-19, la vidéo représentait environ 80 % du trafic internet [1]. De plus, la consommation de vidéos a encore progressé lors de la pandémie, obligeant des plateformes comme Netflix à réduire la qualité des vidéos afin d'éviter un engorgement du réseau [2]. Selon certaines estimations, la diffusion de vidéos sur Internet est d'ores et déjà responsable de 1 % des émissions mondiales de gaz à effet de serre [3], soit l'équivalent d'un pays comme l'Espagne. Réduire la quantité de données nécessaire à la transmission ou au stockage des vidéos pourrait permettre de réduire les émissions de carbone du streaming vidéo, dans la mesure où un effet rebond (paradoxe de Jevons) est évité [START_REF] York | Ecological paradoxes: william stanley jevons and the paperless office[END_REF].

Depuis les années 90, les algorithmes de compression vidéo visent à réduire la taille d'une vidéo tout en garantissant une qualité acceptable pour les utilisateurs. Bien qu'initialement motivés pour répondre aux limitations de débit imposées par les infrastructures existantes, réduire l'impact environnemental du streaming video est une motivation supplémentaire pour concevoir de nouveaux algorithmes de compression. Différents standards de compression vidéo ont été développés par des organismes de standardisation tels que le Moving Picture Experts Group (MPEG) et l'International Telecommunication Union (ITU-T). Chaque génération de standards (AVC en 2003, HEVC en 2013 et VVC en 2020) présente des améliorations incrémentales qui offrent des gains en compression significatifs, permettant de diviser par deux le débit d'une vidéo à qualité égale.

Ces algorithmes de compression conventionnels traitent les vidéos au travers d'opérations successives, généralement linéaires. De plus, les codeurs conventionnels sont conçus de manière incrémentale, occasionnant une optimisation séparée de leurs différentes opérations. Récemment, les réseaux de neurones sont devenus une réponse populaire à de nombreuses problématiques, grâce à leur capacité à modéliser toute fonction au travers d'un processus d'optimisation. Dans le contexte de la compression, cela permet de réaliser des opérations plus riches et non linéaires. Ces différentes opérations peuvent être v conjointement optimisées afin de mieux minimiser la taille de la vidéo et les dégradations visuelles entraînées par sa compression.

Des travaux récents montrent déjà que les schémas de compression d'image basés sur des réseaux de neurones sont compétitifs avec les meilleurs codeurs conventionnels.

Dans ces approches, les réseaux de neurones apprennent les fonctions transformant une image vers une représentation compressée et vice-versa. Cependant, la présence d'une dimension temporelle fait de la compression vidéo une tâche encore difficile pour les approches neuronales. Néanmoins, la réduction de la taille des vidéos est la problématique à laquelle il faut répondre. Dans ce but, cette thèse étudie la conception de codeurs vidéo neuronaux, afin de profiter de leurs capacités prometteuses. Étant donné la nouveauté de ce sujet, le codeur proposé est conçu à partir d'une page blanche et chacun de ces éléments est considéré et motivé.

Partie I -Contexte et état de l'art

La première partie de la thèse présente les notions importantes relatives à la compression d'image et de vidéo. Ces notions sont illustrées à travers leur implémentation par les systèmes de compression conventionnels ainsi que les codeurs d'images neuronaux.

Chapitre 1 : Fondamentaux de compression vidéo

Les algorithmes de compression vidéo visent à réduire le nombre de bits nécessaires à l'envoi de séquences d'images. En effet, les valeurs brutes des pixels d'une séquence vidéo représente une quantité prohibitive de données. Par exemple, une vidéo haute définition est constituée d'une succession de 24 à 60 images par seconde typiquement. Chaque image est composée de 1920 × 1080 pixels, où la couleur de chaque pixel est codée par un triplet de valeurs 8 bits. Ainsi, le débit nécessaire à la transmission des pixels bruts est de :

24 images 1 s × 1920 × 1080 pixels 1 image × 3 × 8 bits 1 pixel ≃ 1.20 Gbit/s. ( 1
)
Une vidéo n'est pas une collection de pixels aux valeurs aléatoires, mais présente au contraire une forte organisation spatio-temporelle. Différentes techniques de compression sans perte exploitent cette structure afin de réduire la taille des vidéos, sans pour autant perdre d'information. Ces techniques sont implémentées par un encodeur, qui encode la vidéo dans un flux binaire, et par un décodeur, qui décode du flux binaire la vidéo initiale.

vi Tout d'abord, les algorithmes de codage entropique exploitent les propriétés statistiques d'une vidéo en assignant moins de bits aux combinaisons de pixels les plus probables.

Supposons que x = (x 1 , . . . , x N ) soit une variable aléatoire discrète suivant une distribution q, représentant un message à transmettre. Par exemple, x représente les N pixels composant une seconde de vidéo. D'après le théorème de Shannon [5], le débit optimal R ⋆ est exprimé au travers de l'entropie H, définie comme :

R ⋆ = H (x) = E x∼q -log 2 q(x) .
(2)

Dans l'équation ci-dessus, chaque message (chaque seconde de vidéo) est représenté par un code binaire de longueurlog 2 q(x). En pratique, la distribution q est souvent inconnue, ce qui nécessite d'estimer q au travers d'une approximation p. L'erreur causée par l'utilisation d'une approximation entraîne un débit R supérieur à R ⋆ :

R -R ⋆ =
Longueur de code estimée E x∼q -log 2 p(x) - Longueur de code optimale E x∼q -log 2 q(x) .

(

Ainsi, la performance du codage entropique dépend de la qualité de l'approximation p.

Néanmoins, la dimension d'une séquence vidéo (plusieurs millions de pixels par seconde) empêche d'estimer une distribution multidimensionnelle des pixels. En conséquence, les différents pixels sont supposés indépendants et transmis successivement :

p(x) = N i=1 p i (x i ). (4)
Bien que nécessaire afin d'implémenter un algorithme de codage entropique, cette hypothèse n'est jamais vérifiée en pratique. Cela entraîne un surcoût de débit, qui augmente avec la dépendence statistique entre les pixels de la vidéo. Pour réduire ce surcoût de débit, deux types d'opérations sont appliqués sur les vidéos.

• Les prédictions visent à réduire les redondances entre les pixels déjà transmis x <i (mémorisés au décodeur) et ceux à transmettre. Les pixels à transmettre sont prédits (à l'encodeur et au décodeur) grâce à x <i , et seule la partie non-prédite est envoyée ;

• Les transformées sont appliquées sur tous les pixels afin de réduire les dépendences statistiques.

Les standards de compression vidéo emploient prédictions et transformées en amont du codage entropique, permettant de réduire le débit d'un facteur trois [6]. D'après l'équation vii (1), le débit est donc abaissé à quelques centaines de Mbit/s, ce qui reste supérieur au débit de quelques Mbit/s utilisés en pratique pour le la transmission de vidéos. Pour atteindre un tel débit la reproduction exacte de la vidéo originale n'est pas possible.

La compression avec pertes permet de considérablement abaisser le débit au prix de la perte d'information. Typiquement, cette perte d'information est réalisée au travers d'une étape de quantification, réduisant la précision des valeurs transmises. Elle entraîne une dégradation potentiellement perceptible (distorsion), mesurée par l'intermédiaire de différentes métriques (erreur quadratique, MS-SSIM [7]). Le compromis entre débit R et distorsion D doit être considéré avec attention en fonction du cas d'usage. Il est souvent formulé comme l'optimisation d'une fonction de coût débit distorsion J exprimée à l'aide un Lagrangien, dont le multiplicateur λ établit le compromis entre débit et distorsion :

min J λ = D + λR. ( 5
)
La Figure 1

Chapitre 3 : Apprendre à compresser des images

Les réseaux de neurones sont un outil mathématique permettant de représenter n'importe quelle fonction [START_REF] Goodfellow | Deep Learning[END_REF]. Ils sont composés d'une succession d'opérations linéaires et non linéaires, dont les paramètres sont appris au cours d'un processus d'optimisation appelé entraînement. Notons y = f (x; θ) la fonction réalisée par un réseau de neurones qui relie une entrée x à une sortie y à l'aide de paramètres θ. Le but du réseau est que y corresponde à la sortie désirée y ob j . Pour ce faire, il est nécessaire de trouver les paramètres optimaux :

θ ⋆ = arg min θ E x J f (x; θ), y ob j , (6)
J est la fonction de coût, qui mesure l'écart entre la sortie attendue et celle observée. Les paramètres θ sont appris au travers d'un algorithme itératif de descente de gradient stochastique. À l'itération numéro n, plusieurs exemples x sont tirés aléatoirement depuis un ensemble d'entraînement. La sortie du réseau ainsi que la fonction de coût sont calculées pour chacune des entrées, puis la fonction de coût est moyennée. Enfin, les paramètres θ n du réseaux sont mis à jours de la manière suivante : θ n+1 = θ n -η∇ θ E x J f (x; θ), y ob j , η ∈ R. (7) x Depuis 2017 et les travaux de Ballé [START_REF] Ballé | End-to-end optimized image compression[END_REF] et Theis [START_REF] Theis | Lossy image compression with compressive autoencoders[END_REF], les autoencodeurs sont la méthode la plus populaires pour implémenter des systèmes de compression basés réseaux de neurones. La Figure 3 présente l'architecture d'un autoencodeur, pourvue de 2 sous-réseaux réalisant respectivement la transformée d'analyse g a et de synthèse g s .

Habituellement, les autoencodeurs sont entraînés pour que leur sortie x soit la plus proche possible de l'entrée x, tout en imposant des contraintes sur la représentation latente y. Cela permet d'apprendre une représentation latente pertinente et expressive, offrant un domaine de représentation mieux adapté que celui du domaine vidéo initial. Dans le contexte de la compression, la contrainte imposée sur y a pour vocation de limiter son entropie, c'est-a-dire son débit. La variable latente y obtenue grâce à la transformée d'analyse g a est quantifiée, puis transmise au décodeur via un codage entropique. Enfin, la transformée de synthèse g s permet d'obtenir l'image décodée x depuis les latentes quantifiées. Afin de mener à bien l'étape de codage entropique, un modèle de probabilité p des variables latentes doit être disponible. C'est le rôle d'un troisième réseau de neurones, entraîné pour représenter la distribution de ŷ. Tous les paramètres du schéma de compression (g a , g s et p) sont optimisés conjointement afin de minimiser une fonction de coût débit distorsion : L λ = E x d(x, x) -λr( ŷ) , [START_REF] Wiegand | Overview of the h.264/avc video coding standard[END_REF] où d mesure la distorsion, r mesure le débit associé aux variables latentes, et λ est la contrainte de débit permettant d'équilibrer débit et distorsion.

Ainsi, un schéma de compression d'image basé apprentissage est composé de trois éléments principaux : une transformée d'analyse, de synthèse et un modèle de probabilité des variables latentes. Une grande partie des travaux menés depuis 2017 vise à améliorer ces trois éléments, à travers des architectures mieux adaptées. Dans le cadre de cette thèse, un nouveau modèle de probabilité a été proposé. Il permet de mieux représenter la distribution des variables latentes et donc d'abaisser leur débit de 6 % [START_REF] Ladune | Binary probability model for learning based image compression[END_REF].

Chapitre 4 : De la compression d'image à la compression vidéo

En 2019, des travaux précurseurs [START_REF] Habibian | Video compression with rate-distortion autoencoders[END_REF] ont proposé d'étendre directement les autoencodeurs de compression d'image pour traiter de façon groupée T images de vidéo . Cependant, ces approches requièrent un nombre élevé de paramètres et présentent des défauts inhérents au traitement par groupe d'images. En effet, il est nécessaire d'attendre d'avoir T images à traiter pour commencer le codage, entraînant une latence de T images. Puisque ces défauts ne sont pas compensés par des performances encourageantes, la grande majorité des travaux ultérieurs (y compris ceux présentés dans ce manuscrit) ont adopté le paradigme conventionnel d'un traitement image par image. Cependant, cela impose une étape de prédiction temporelle pour réduire les redondances. En pratique, elle est effectuée par compensation de mouvement, de manière similaire aux codeurs conventionnels. L'ajout d'une étape de compensation de mouvement amène à un schéma de compression en trois étapes, présenté dans la Fig. Les autoencodeurs sont utilisés ici comme des outils génériques permettant de transmettre les différentes quantités en jeu (informations de mouvement, partie non-prédictible).

La meilleure façon de transmettre x t étant donné sa prédiction xt est étudiée dans le xiii Chapitre 5. Puis, le Chapitre 6 se concentre sur l'estimation et la transmission des informations de mouvement, afin d'obtenir une prédiction temporelle précise. Afin d'évaluer les différentes solutions proposées, une situation réaliste de codage vidéo est choisie : la tâche de compression vidéo du Challenge on Learned Image Compression (CLIC) 2021 [START_REF]Workshop and challenge on learned image compression[END_REF].

L'objectif de ce challenge est de compresser 100 vidéos haute définition à un débit moyen de 1 Mbit/s tout en obtenant la meilleure qualité, mesurée au travers d'une métrique objective nommée MS-SSIM.

Pour le moment, les meilleures implémentations des standards de compression conventionnels (HM pour HEVC et VTM pour VVC) demeurent les solutions de compression vidéo les plus performantes. Ainsi, elles sont utilisées comme ancrage pour attester de la pertinence des résultats proposés. Puisque le challenge ne comporte pas de contrainte sur la structure de codage, une configuration Random Access (voir Fig. 2) issue des conditions de test de VVC est utilisée [START_REF] Bossen | VTM common test conditions and software reference configurations for sdr video[END_REF].

La configuration Random Access nécessite de savoir traiter des images avec zéro, une ou deux références, c'est-à-dire des images I, P et B. De ce fait, nous proposons d'entraîner nos systèmes afin de coder des triplets d'images, comme illustré par la Fig. 7. Enfin, la totalité des paramètres du système est optimisée de manière conjointe au travers de l'optimisation de la fonction de coût suivante, composée de la somme du coût débit distorsion des images successives :

L λ = E x t d(x t , xt ) + λr( xt ) . ( 9
)
Pour répondre au challenge, la métrique de distorsion d est logiquement basée sur le MS-SSIM.

x out = x dec + autoencodeur g s (Q(g a (x enc -x dec résidu ))) . (11)
Une fois transmis, le résidu est combiné avec x dec au travers d'une addition. Effectuer le mélange via une addition et une soustraction présente deux défauts :

• Ce n'est pas nécessairement la combinaison la mieux adaptée ;

• Cela nécessite des quantités compatibles : image et différence d'images.

Pour remédier à ces défauts, nous proposons une nouvelle architecture appelée codage conditionnel [START_REF] Ladune | Modenet: mode selection network for learned video coding[END_REF], présentée Fig. 8. Une transformée d'analyse g a prend en entrée les informations présentes à l'encodeur et au décodeur et calcule une variable latente y xv représentant l'information à transmettre, présente à l'encodeur mais pas au décodeur :

y = g a (x enc , x dec ). ( 12
)
Comme g a est un réseau de neurones, ses opérations successives permettent de réaliser un mélange plus riche qu'une simple soustraction.

En codage résiduel, l'information présente au décodeur x dec est combinée avec la sortie de la transformée de synthèse g s via une addition. Le codage conditionnel utilise une nouvelle transformée de conditionnement g c qui sert à incorporer les informations de x dec à la sortie x out . Pour ce faire, g c extrait une nouvelle variable latente y c :

y c = g c (x dec ) . ( 13
)
Il est important de noter que la variable latente y c issue de la transformée de conditionnement est calculée au décodeur et ne nécessite donc pas de transmission. Enfin, la transformée de synthèse g s prend en entrée les deux variables latentes pour calculer la sortie désirée :

x out = g s (y, y c ) . (14)

Conclusion et perspectives

Cette thèse a reconsidéré entièrement les schémas de compression vidéo existants afin de concevoir un codeur vidéo basé apprentissage. Différentes contributions (architecture de codage conditionnel, ajout d'un mode de codage) ont permis d'obtenir un codeur vidéo basé apprentissage performant, compétitif avec les algorithmes conventionnels (HEVC) sous différentes configurations de codage. Ces résultats démontrent la pertinence des approches basées apprentissage. De plus, la rapidité des progrès de ce type de codeur laisse à penser qu'ils dépasseront bientôt, par un affinement de leurs éléments constituants, les algorithmes conventionnels.

Ainsi, tout au long de cette thèse, différents axes d'amélioration des performances ont été entrevus. Il a été mis en évidence que l'estimation de mouvement par le MNet est un élément sensible du système. Ce composant pourrait être amélioré au travers de l'utilisation de techniques issues de la littérature dédiée à l'estimation de flux optiques (cost volume). De plus, dans ce travail, les systèmes de compression vidéo sont entraînés à l'aide d'une configuration de codage comprenant 3 frames (voir Fig. 7). Adapter le codeur en le ré-entraînant pour sa configuration de codage cible améliorerait certainement les performances.

Les algorithmes conventionnels de compression mettent en place une compétition entre différents modes de codage, ce qui leur permet d'effectuer des opérations adaptées au signal à transmettre. Si certains éléments du codeur proposé vont dans ce sens (Skip mode, précision de quantification variable), l'adaptation au contenu des systèmes basés apprentissage demeure trop limité. Pour autant, la nature même des réseaux de neurones est de s'adapter aux données d'entraînement. Ainsi, entraîner un codeur sur une séquence vidéo particulière permettrait d'obtenir un codeur optimal, même si les paramètres du codeur sont alors à transmettre avec la vidéo. Des travaux récents [START_REF] Dupont | COIN: compression with implicit neural representations[END_REF], [START_REF] Van Rozendaal | Overfitting for fun and profit: instance-adaptive data compression[END_REF] . Furthermore, the pandemic has made video consumption progress even further, obliging some streaming services (Netflix and YouTube) to reduce the video quality in order to avoid network congestions [2]. According to some estimates, video streaming already accounts for 1 % of the overall greenhouse gas emissions [3], resulting in the same impact than a country like Spain. As such, reducing the amount of data required to transmit or store videos could reduce the carbon emissions of video streaming, as long as the so-called rebound effect or Jevons paradox is avoided [START_REF] York | Ecological paradoxes: william stanley jevons and the paperless office[END_REF].

From the 90s onwards, video compression algorithms aim to reduce the size of videos while maintaining an acceptable quality for the users. Even though the primary motivation for compression algorithms was to cope with the limited bandwidth imposed by existing infrastructures, mitigating the environmental impact of online video streaming provides an additional motivation to design better compression algorithms. Video compression standards have been designed by standardization organisms such as the Moving Picture Experts Group (MPEG) and the International Telecommunication Union (ITU-T). Each generation of standards (AVC in 2003, HEVC in 2013 and VVC in 2020) introduced incremental refinements leading to substantial compression gains i.e. halving the rate required for equivalent quality.

These traditional coding algorithms process videos through successive operations, which are often linear. Given the incremental design of traditional coders, their operations are optimized separately. In recent years, deep neural networks have been used to solve a variety of problems, thanks to their ability to model virtually any function through a cost function optimization. In the context of compression, this allows to perform richer, non-linear operations which are jointly optimized to minimize both the size of the video and the visual degradations brought by its compression.

Recent works show that neural-based still image compression can be competitive 1 with state-of-the-art coding standards, by learning the functions mapping back and forth the image to a compressed representation. Due to the additional challenges brought by the temporal dimension, video coding remains a more demanding task for neuralbased approaches. Yet reducing the size of video sequences is the actual issue to tackle.

Consequently, this thesis studies the design of neural-based video compression schemes, to leverage the promising abilities of neural networks. Given the novelty of this field, the proposed neural-based coding scheme starts from a blank page and each element is reconsidered and motivated.

Structure of the thesis Part I -Context and state of the art

The first part of the thesis presents important notions for image and video compression. Then, these notions are illustrated by their implementation in traditional coding algorithms and neural-based image coders.

Chapter 1 introduces the main concepts of lossless and lossy compression from an information theory standpoint. The usage of predictions, transforms and entropy coding is justified. Rate-distortion optimization is presented to balance the size and quality of compressed videos.

Chapter 2 illustrates the habitual implementation of the concepts expounded in the first chapter by traditional compression algorithms e.g. HEVC and VVC.

Chapter 3 starts with a presentation of deep neural networks. Then, they are used to implement a neural-based image coder, which is successively refined to reach state-of-the-art performance.

Part II -Learned video coding

The second part of the thesis offers to design a neural-based video coder from a blank page.

The requirements of the coder and a realistic experimental framework are introduced.

Then, the different components of the coder are thoroughly studied.

Chapter 4 motivates the requirements of the proposed neural-based video coder. A real-life video coding situation and relevant anchors are proposed to assess the performance of the neural coder.

Chapter 5 reconsiders the exploitation of a prediction. Using neural networks abilities, a novel architecture called conditional coding is motivated to better leverage prediction than residual coding.

Chapter 6 adds motion information to the neural coder to compute a more accurate prediction. The optimization of the motion-related components is fostered through the introduction of an additional coding mode. Finally, conditional coding is also implemented for motion information.

Chapter 7 evaluates the resulting neural-based coder through the participation to a video compression challenge. Due to the challenge constraints, the proposed coder is modified to operate under a continuous and variable rate constraint.

Contributions

The work carried out in this thesis contains several contributions to the field of learned image and video compression.

• Enhance entropy coding with a more expressive probability model;

• Reconsider the usage of residual coding to exploit prediction;

• Propose a generic architecture to leverage decoder-side information;

• Introduce an additional coding mode to both improve performance and ease the optimization of motion-related components.

More generally, this thesis presents a method to design a flexible neural-based video coder, competitive with modern video coding standards. CHAPTER 1

FUNDAMENTALS OF VIDEO CODING 1.1 Introduction

V IDEO coding algorithms aim to reduce the number of bits required to transmit or store sequences of images. Indeed, transmitting raw values of all pixels in the video would require a prohibitive amount of bits to be sent. For instance, a high-definition video is made of successive images composed of 1920 × 1080 pixels, where each pixel is usually represented as three 8-bit values indicating its colour, while the number of images per second often varies between 24 to 60. Therefore, conveying the raw pixel values requires to transmit:

24 images 1 s × 1920 × 1080 pixels 1 image × 3 × 8 bits 1 pixel ≃ 1.20 Gbit/s. (1.1)
This bandwidth exceeds the usual capacity of computing devices and highlights the need to reduce the number of bits representing a video. This is made possible by the fact that most videos exhibit a spatio-temporal structure. Different techniques leverage this organization to code videos with fewer bits, without losing any information. First, entropy coding algorithms exploit the statistical properties of videos by assigning less bits to the more probable pixel combinations. Second, invertible mathematical operations such as predictions and transforms are performed to reduce redundancies among video pixels.

These techniques compose the lossless coding framework presented in the first part of this chapter.

In many cases, the size of a losslessly coded video exceeds the available bit budget of practical applications. Lossy coding algorithms allows to reach further data size reduction, at the expense of information loss. Due to this loss, the compressed video exhibits some form of distortion compared to the original one. This distortion is measured and balanced against the rate reduction offered, in a process called rate-distortion optimization. The second part of this chapter introduces the different notions related to lossy coding.

Lossless coding 1.2.1 Optimal rate and information theory

This section lays out the information theory framework for lossless coding. Let the message to code be a discrete random variable s = (s 1 , . . . , s N ) composed of N symbols following a distribution q, such as an N-pixel image. Lossless coding performs a reversible mapping from the message s to a set of K bits b = (b 1 , . . . , b K ). The process of mapping the message into a bitstream is called encoding and is performed by an encoder. Similarly, the decoder performs the inverse mapping, decoding the bitstream back to the initial message. To decrease the number of transmitted bits, the mapping is designed to leverage statistical properties of the distribution q. Indeed, the Shannon's source coding theorem states that the optimal number of bits for b is equal to the information I of the message:

I(s) = -log 2 q(s), with I(s) in bits. (1.2)
Consequently, the most frequent messages are represented with fewer bits. When transmitting many successive messages drawn from q (e.g. successive N-pixel images), the average number of bits for the messages is called the rate. It is expressed in bits per message which translates in practice to bits per pixel, bits per image, bits per second, etc.

Equation 1.2 yields that the minimal rate R ⋆ is expressed through the Shannon entropy

H [5]
, defined as:

R ⋆ = H (s) = E s∼q [I (s)] = E s∼q -log 2 q(s) . (1.3)
In most cases, the underlying data distribution q is unknown. As such, it is not possible to compute I(s) to determine the optimal mapping for each message. This issue is overcome by estimating q through an approximation p. Each message is now mapped to a stream of log 2 p(s) bits. Thus, the rate R needed to code messages drawn from q becomes: R = E s∼q -log 2 p(s) .

(1.4)

The estimate p is an approximation of q, causing a non-optimal mapping. The resulting 1.2. Lossless coding rate overhead is often expressed by D K L , the Kullback-Leibler divergence:

R -R ⋆ = Estimated code length E s∼q -log 2 p(s) - Optimal code length E s∼q -log 2 q(s) (1.5) = E s∼q log 2 q(s) -log 2 p(s) = D K L (q || p) .
The Kullback-Leibler divergence measures the mismatch between two probability distributions. This highlights that achieving efficient lossless coding comes down to performing accurate modelling of the data distribution of s.

Arithmetic coding

Entropy coding algorithms are used to implement the mapping from a message to a stream of bits, based on the estimated distribution p. Among them, Huffman coding [START_REF] Huffman | A method for the construction of minimum-redundancy codes[END_REF] reaches the theoretical rate expressed in Eq. (1.4) only when all the probabilities p (s) are equal to the inverse of a power of two. This constraint on the distribution p often leads to a greater mismatch between q and p, increasing the rate overhead. Arithmetic coding [START_REF] Langdon | An introduction to arithmetic coding[END_REF] is more effective as it handles a probability model not constrained to inverse powers of two. It maps multiple messages into a single number w ∈ [0, 1[, allowing the practical rate to converge to the theoretical one when the number of messages increases.

To illustrate the arithmetic coding process, let us suppose that a message s has a set of possible values Ω s with a finite number of elements denoted as card (Ω s ). Figure 1.1 depicts the arithmetic coding algorithm. Each message is sequentially encoded by subdividing an interval [u, v[, initialized at u = 0 and v = 1. At each step, this interval is divided into card (Ω s ) segments, one for each possible values of s, denoted as s ( j) . The length l j of each segment is proportional to the probability of its corresponding value:

l j = 1 v -u • p s ( j) (1.6)
Finally, the interval boundaries u, v are set to the ones of the interval corresponding to the current message value and the next message is processed. Once the messages are processed, w is set to a value in [u, v[ and converted to b, a set of bits. b must have enough bits so that any base-2 values starting with the bits b belongs to [u, v[.

Part I, Chapter 1 -Fundamentals of Video Coding u = 0 v = 1 0.2 0.5 0.6 s (0) s (1) s (2)
s (3) Step 1 ) , s (1) , s (2) , s (3) , whose respective probability are 2 10 , 3 10 , 1 10 and 4 10 . At the end of the arithmetic coding process, the value w = 0.2421875 represents the three successive messages s (1) , s (0) , s (3) .

u = 0.2 v = 0.5 0.26 0.35 0. 38
s (0) s (1) s (2) s (3) Step 2 u = 0.2 v = 0.26 0.212 0.23 0.236 s (0) s (1) s (2) s (3) Step 3 w ∈ [0.236, 0.

Practical limitations of arithmetic coding

The arithmetic algorithm might be intractable in practice. Indeed, if s represents a 8-bit greyscale image with N pixels, the number of possible values is card (Ω s ) = 256 N . Dividing an interval into that many segments is not practicable. One of the solutions to mitigate this issue is to rely on a factorized probability model, where the symbols composing s are assumed independent, modelled with their own distribution p i :

p(s) = N i=1 p i (s i ).
(1.7)

This allows the arithmetic coding process to be conducted symbol-wise instead of message-wise. A message is now composed of a single scalar symbol, instead of N symbols.

Continuing the example, arithmetic coding now works pixel-by-pixel, dividing the interval into card (Ω s ) = 256 segments at each step, which can be performed in practice. Some real-life arithmetic coders, e.g. the Context Adaptive Binary Arithmetic Coding (CABAC) [START_REF] Wien | Entropy coding[END_REF], makes the additional assumption that the symbol takes binary value, leading to an even smaller number of possible values: card (Ω s ) = 2. This requires a binarization step, which decomposes each symbol into several binary values.

Despite being necessary to perform arithmetic coding, the assumption of symbols independence almost never holds in practice and results in a suboptimal probability model leading to a significant rate overhead. Let us illustrate the rate overhead through the lossless coding of a greyscale N-pixel image x = (x 1 , . . . , x N ). The minimal rate required to transmit the pixels is obtained thanks to Eq. (1.3):

R ⋆ = H (x) = H (x 1 , . . . , x N ) . (1.8)
Yet, the pixels of x cannot be jointly coded due to the practical concerns previously raised. Instead, the pixels are independently coded and their probability models are assumed independent. This leads to the following rate:

R = N i=1 H (x i ) ≥ R ⋆ = H (x 1 , . . . , x N ) .
(1.9)

The rate overhead brought by the assumption of independent pixels can be expressed through the mutual information I [32]:

R -R ⋆ = N i=1 H (x i ) -H (x 1 , . . . , x N ) (1.10) = D K L (p(x 1 , . . . , x N ) || N i=1 p(x i )) = I(x 1 , . . . , x N ) ≥ 0.
As illustrated by the Kullback-Leibler divergence, the mutual information assesses the degree to which random variables are independent. If the variables are independent, there is no mutual information and no rate overhead. However, one pixel in a video sequence is likely correlated to its neighbouring pixels. This results in a non-negligible mutual information, increasing the rate.

To lower the resulting rate, bijective functions are applied to x prior to arithmetic coding. The purpose of these functions is to remove as much statistical redundancy as possible among x components, reducing the mutual information. The next two sections introduce two complementary techniques for redundancy removal, based either on prediction or on transform.

Predictive coding

In practice, a N-pixel video sequence x = (x 1 , . . . , x N ) is often decomposed into smaller sets of pixels, in order to make its processing more convenient. One natural decomposition is to split the video image-by-image and into small blocks of pixels. Yet, this leads to a rate overhead, illustrated on a toy example where the video x is split into two sets of pixels. The first M pixels x dec = (x 1 , . . . , x M ) of the video have already been transmitted (i.e.

available at the decoder) while the others pixels x enc = (x M+1 , . . . , x N ) are still to be sent.

According to Eq. (1.10), transmitting x enc without leveraging the information contained in x dec leads to a rate increase of I (x dec , x enc ).

Predictive coding palliates the effects of independent arithmetic coding for each set of pixels through a two-step process. First, the pixels x enc are predicted thanks to the already transmitted pixels xenc = f (x dec ), so that the decoder is able to reproduce the prediction.

Second, xenc and x enc are combined through a bijective function m to obtain a prediction residual r = m (x enc , xenc ). Usually the function m is a simple difference between the prediction and the symbols to transmit i.e. r = x enc -xenc . With an ideal prediction, the prediction residue r does not longer contain information predictable from x dec , meaning that x dec and r are independent. The residue r is transmitted instead of x enc , yielding an optimal rate without overhead:

R ⋆ = H (x dec ) + H (r) = R ⋆ H (x dec , r) + =0 I (x dec , r) . (1.11)
Since the function m is bijective, no information is lost when transmitting the prediction residue r instead of x enc , that is: H (x dec , x enc ) = H (x dec , r). Following the transmis- sion of x dec and r, the prediction is computed at the decoder side and the original pixel

x enc is retrieved from r and xenc .

Predictive coding aims to remove statistical dependencies between the two sets of pixels, so that sending x dec and x enc separetely does not bring a rate overhead. However, it does not target the statistical dependencies within each pixel blocks. According to Eq.

(1.10), the pixel-by-pixel arithmetic coding of the residue r = (r M+1 , . . . , r N ) results in a rate overhead of I (r M+1 , . . . , r N ) bits. This additional rate is targeted by transform coding, which aims to remove statistical relationship inside each pixel block. As such, the role of the prediction and of the transform is complementary. It is theoretically possible to feed the entire video to a transform (i.e. x enc = x and x dec = ), so that the transform removes all the statistical relationship within the image, making the prediction step irrelevant. It is also possible to rely only on the prediction step, by predicting and transmitting each pixel x i successively, using all the previously sent pixels: x enc = {x i } and x dec = {x 1 , . . . , x i-1 }.

In this case, it is the transform which becomes irrelevant. In practice, using both the prediction and the transform through a relevant partitioning of the video often leads to better performance.

Transform coding

While predictive coding focuses on removing statistical relationships between pixel blocks, transform coding targets the dependencies inside a set of pixels. Let us suppose that the current pixel block is a H ×W greyscale image x = (x 1 , . . . , x HW ). It is losslessly compressed using arithmetic coding. As such, the H ×W symbols are independently processed, yielding a rate overhead derived from Eq. (1.10):

R -R ⋆ = I(x 1 , . . . , x HW ). (1.12)
This issue is addressed by reducing the statistical dependencies among the pixels, leading to a lower mutual information. To this end, a bijective transform g is applied to x.

Without loss of generality, the distribution of x is assumed to be centered, with a covariance matrix denoted as Σ x . Let us make the additional assumption that the transform is linear, such as it is expressed through a matrix A:

y = g (x) = Ax, with A ∈ R HW×HW .
(1.13)

The covariance matrix of y can now be expressed as:

Σ y = E (Ax) (Ax) ⊤ = AE xx ⊤ A ⊤ = AΣ x A ⊤ . (1.14)
Since Σ x is real-valued and symmetric, there exists a matrix A, composed of the eigenvectors of Σ x , which results in Σ y being diagonal. Therefore, the different symbols of y are decorrelated i.e. there is no linear relationship between them. If the image follows a Gaussian distribution, this also translates into independent symbols [5] for y. In this case, the rate overhead expressed by the mutual information in Eq. (1.12) is null. For non-Gaussian data, decorrelating the symbols reduces, but does not remove the statistical dependencies, still lowering the rate overhead.

The decorrelation matrix A is orthogonal i.e. A -1 = A ⊤ , ensuring the bijective nature of the transform and allowing to retrieve the original image at the decoder side, after the arithmetic coding step:

x = g -1 (y) = A ⊤ y. (1.15)
This transform is known as the Karhunen-Loève Transform (KLT) [START_REF] Rao | The Transform and Data Compression Handbook[END_REF]. Although limited to the removal of linear relationships, variations of this transform such as the Discrete Cosine Transform (DCT) are widely used in video coding. It should be noted that non-linear transforms could remove additional statistical relationships. This could be achieved by neural networks, which are known to effectively model non-linear transforms.

Conclusion

Lossless coding performs a reversible mapping from a message to a bitstream. Theoretically, leveraging the statistical properties of the message allows to reach a minimal number of bits, expressed by the message entropy.

In practice, the message is a video, which is usually decomposed into smaller sets of pixels corresponding to successive images, often additionally partitioned into blocks of pixels. These smaller sets of pixels are transmitted one after another as successive independent messages. The mapping from a message to a bitstream is performed through arithmetic coding, which assumes that the different symbols composing the message are independent. Yet, neither the messages nor their symbol are independent, leading to a significant rate overhead compared to the video entropy.

This issue is addressed by performing two complementary operations on the messages: prediction and transform. Predictive coding aims to remove dependencies between the successive sets of pixels, while transform coding fosters the independence between the symbols of a given set of pixels. To ensure the coding process remains lossless, these bijective operations are implemented at the encoder, and their inverses at the decoder.

Figure 1.2 presents a typical example of a lossless encoding and decoding process.

Lossy compression

Modern lossless video coding algorithms offer a rate reduction up to a factor of three [6], often resulting in a rate of a few hundred of Mbit/s. The usual rate target for a video sequence is far smaller, being in the order of a few Mbit/s for a 1080p video. To this end, lossy coding removes information from the video before its transmission. This decreases the symbols entropy, allowing for lossless transmission at a lower rate. Yet, this information loss introduces distortion i.e. differences between the original and the compressed data. The trade-off between rate and distortion needs to be adjusted according to the application.

Discarding information

Discarding information is achieved by using a non-bijective (i.e. non-invertible) surjective function g during the encoding process. Indeed, such function is allowed to map several different inputs to an identical output. This increases the probability of the different possible output values, lowering the resulting entropy:

y = g (x) ⇒ H (y) < H (x)
, for a non-bijectiveg.

(1.16)

The typical example of a function discarding information is the uniform scalar quantization Q (see Fig. 1.3), which maps any number x ∈ R to the closest multiple of ∆ ∈ N:

Q ∆ (x) = ∆ round x ∆ .
(1.17)

The quantization parameter ∆ is called the quantization step. Setting a bigger value of ∆ increases the amount of discarded information.

Besides quantization, others stages of the coding process can also be made lossy by relaxing the constraint on the bijective nature of the operations. For instance, the Principal Component Analysis (PCA) is a transform reducing the data dimensionality. It is similar to the KLT but the PCA matrix is composed of a subset of the most important eigenvectors of the data covariance matrix. The PCA is a lossy transform, discarding the

-2 -1 1 2 -2 -1 1 2 x Q (x)

Distortion metrics

Discarding some information during the encoding process decreases the rate, at the expense of decoding a distorted version of the original data. The error between the original data x and the decoded data x is called the distortion denoted as D (x, x). Among all the existing distortion metrics, two are considered in this manuscript. First, the Mean Squared Error (MSE):

D (x, x) = MSE (x, x) = 1 N N i=1 (x i -xi ) 2 , (1.18)
with N the dimension of x and x. For images and videos, it is common to translate the MSE (the distortion) into the Peak Signal to Noise Ratio (PSNR), which is to be maximized:

PSNR (x, x) = 10 log 10 max 2 MSE (x, x) . (1.19)
The constant max is the maximum possible value for x i.e. 255 for 8-bit images and videos considered in this work. The PSNR translates the quality of the reconstruction x compared to the original data x.

The PSNR only considers pixel-wise error i.e. the energy of noise introduced in compressed images. Yet, the human visual system is not equally sensitive to the presence of noise in an image. Indeed, we are more tolerant when the noise occurs in high-contrast areas, while we are more sensitive to noisy low-contrast areas. This is the reason behind the second quality metric used in this work: the Multi-Scale

Structural Similarity Metric (MS-SSIM) [7]. This metric computes statistics on the entire image, at different scales, to better assess the perceived degradations due to the presence of noise. In short, the MS-SSIM measures whether the overall structure of the image is conserved i.e. a low-contrast area should not become a high-contrast one due to noise presence. The MS-SSIM returns a quality score ranging from 0 (worst quality) to 1 (best quality). Throughout this work, the MS-SSIM is often expressed on a logarithmic scale:

MS-SSIM dB (x, x) = -10 log 10 (1 -MS-SSIM (x, x)) .

(1.20)

Figure 1.5 presents how the repartition of the same amount of noise is qualified by the PSNR and the MS-SSIM. In Fig. 1.5b, the noise is added to a low-contrast area (the girl's face). In Fig. 1.5c, the noise is added to a high-contrast area (the beanie). Since the noise added has approximately the same energy, the PSNR remains the same. However, the MS-SSIM penalizes significantly the introduction of noise in low-contrast areas. Adding the same amount of noise exclusively to high-contrast areas is better tolerated by the MS-SSIM, which is consistent with the human visual system. While some distortion metrics are more advanced than others, there is no perfect one and the choice of the distortion metric remains the subject of many discussions.

Rate-distortion optimization

The objective of lossy coding can be turned into the minimization of the distortion under a rate constraint of R c bits: where D (x, x) is the distortion between the original data x and the compressed one x, and R ( x) is the rate associated to the transmission of x. This optimization task is usually solved through Lagrangian optimization where the distortion term is balanced against the rate term. Using a Lagrangian formulation, the problem becomes:

min D (x, x) subject to R ( x) < R c , (1.21)
min J λ = D (x, x) + λR ( x) , (1.22)
where λ is called the Lagrange multiplier. Thanks to the Lagrangian formulation [START_REF] Sullivan | Rate-distortion optimization for video compression[END_REF],

each solution of Eq. (1.22) for a given Lagrange multiplier λ corresponds to an optimal solution of the original Eq. (1.21) under a particular rate constraint R c .

The value of J λ is called the rate-distortion (RD) cost. Optimization of the RD cost is universally used in video coding. For instance, each coding possibility (choice of the prediction and transform) is rated with its RD cost. Similarly, learning-based approaches presented in this manuscript are trained to minimize a RD cost.

Conclusion

This chapter has introduced the theoretical framework for lossy video compression. Predictions and transforms are performed to obtain a compact representation of the video, more suited for compression. The encoding is then made lossy by selecting the information that has to be kept, through a quantization step or a non-bijective transform. Finally, the information is mapped to a bitstream using a lossless coding algorithm such as arithmetic coding.

Lossy compression algorithms aim to minimize the distortion under a given rate constraint. This is achieved by optimizing a rate-distortion cost, expressed using a Lagrange [START_REF] Wiegand | Overview of the h.264/avc video coding standard[END_REF] was finalized in 2003, followed by High Efficiency Video Coding (HEVC) [START_REF] Sullivan | Overview of the high efficiency video coding (hevc) standard[END_REF] in 2013 and recently Versatile Video Coding (VVC) [START_REF] Bross | Developments in international video coding standardization after avc, with an overview of versatile video coding (vvc)[END_REF] in 2020. These standards address the decoding process (i.e.

from the bitstream to the decoded video). The encoder is required to produce a bitstream compliant with the decoding algorithm, resulting in an encoding process which tends to mirror the behaviour of the decoder.

The successive MPEG/ITU standards have continuously improved a coding algorithm based on the same paradigm, which is refered to as traditional video coding in this work.

This chapters introduces how MPEG/ITU standards practically implement the principle expounded in Chapter 1 for individual images and video sequences.

Hybrid image coding

This section describes how traditional video coders compress a standalone image. In video coding, a standalone image is often called an Intra (I)-frame, since it focuses exclusively on the removal of the statistical dependencies within the image.

Representing the colour: the RGB and YUV colour spaces

A colour image is modelled by expressing each pixel value into a colour space. For instance, the RGB colour space decomposes a colour into three primary colours: red, green and blue.

Video sequences usually use the YUV colour space, due to its better correlation with the human visual system. Indeed, humans are more sensitive to variation of luminance (the quantity of light) than to variation of colours. As such, the YUV colour space dedicates the Y component to the representation of the luminance and leaves the colour representation for the U and V components.

Since the human visual system is less sensitive to the colour, a pre-processing step often reduces the resolution of the UV colour components prior to the compression process.

This allows to reduce the rate with little visual degradations. For example, the YUV 420 format used in this work refers to a Y component of size N × N for a U and V components of size N 2 × N 2 . In some cases, particularly for still images compression, the UV components are not sub-sampled. Such data format is called YUV 444. Since the Y channel represents most of the rate, the rest of this chapter focuses on the Y channel compression. The U and V channels are often independently coded using the same principles than the Y channel.

Overview of the video coding structure

In order to process an image, traditional coders split it into non-overlapping blocks of variable size and shape. Gathering pixels with similar properties within a block allows to perform adapted operations (prediction and transform) on this block.

Intra prediction

For intra frames, the purpose of the (intra) prediction step is to remove the redundancies between the block x and the previously transmitted blocks, which serve as reference. Since the blocks are encoded in a lexical order, the reference pixels available at the decoder x dec to perform the prediction are located on the left and above of x.

Multiple intra prediction modes are available, which correspond to different functions f when computing the prediction x = f (x dec ). Figure 2.3 gives an illustration of the 35 different prediction modes available in HEVC [START_REF] Sullivan | Overview of the high efficiency video coding (hevc) standard[END_REF], illustrated for 8 × 8 blocks. Among them, 33 are directional prediction modes aiming to propagate directional patterns. The 2 others compute either a gradient (Planar mode) or predict the mean value (DC mode). All the intra prediction modes compute linear operations on the reference pixels.

Once the prediction x is available, it is subtracted from the block to code to obtain a residue r = xx. Thanks to the prediction, the residue exhibits less redundancies regarding its spatial or temporal neighbours, reducing the rate. As multiple prediction modes are available, the selected one must be signalled to the decoder.

Transform

While the (intra) prediction step reduces the dependencies between the current block and its neighbours, the transform step aims to reduce the statistical redundancies within the block. This is achieved through a linear transform of the residual block r.

Section 1.2.5 has introduced the KLT, which is the optimal linear transform. The KLT of a N × N residual block (reshaped as a vector of N 2 elements), is an N 2 × N 2 matrix, resulting in N 4 multiplications for one block. For complexity considerations, video coders trade this optimal two-dimensional transform for 2 one-dimensional transforms, applied successively on the rows and columns of the residual block. The transform resulting from these 2 one-dimensional transforms is called a separable transform and requires only 2N 3 multiplications. Yet, lowering the number of computation scomes at the price of sub-optimality. Indeed, separable transforms only remove statistical correlations across the horizontal and vertical axis and do not decorrelate pixels in the other directions (e.g. along the diagonals).

From the legacy image coding standard JPEG [START_REF] Wallace | The jpeg still picture compression standard[END_REF] to the recent VVC, one of the most used transforms is the Discrete Cosine Transform of type II (DCT-II) [START_REF] Ahmed | Discrete cosine transform[END_REF]. It is based on the average statistical properties of natural images, which are known to closely follow a first order auto-regressive model. In such model, a pixel value x n is assumed to be a combination of the neighbouring pixel x n-1 and a noise term ǫ n :

x n = ρx n-1 + ǫ n , (2.1)
with ρ the correlation between x n and x n-1 , ǫ n is an independent noise. When ρ → 1, the KLT yields the DCT-II, a one-dimensional transform, decomposing a signal More recent works focus on the statistical properties of the residual blocks, according to the type of prediction used to obtain them [START_REF] Saxena | Dct/dst-based transform coding for intra prediction in image/video coding[END_REF]. This allows to derive adapted transforms, such as the Discrete Sine Transform of type VII (DST-VII), which are implemented in HEVC and VVC. As for the partitioning and the prediction step, the transform type is selected through the minimization of the rate-distortion cost.

x ∈ R N into N frequential basis B k , k ∈ {0, . . . N -1}: B k (n) = ξ(k) N cos (2n + 1)kπ 2N , (2.2) with n, k = ∈ {0, . . . N -1} and ξ(k) = 1 if k = 0,
Once the residue has been transformed, a scalar quantization step Q takes place to discard information in order to lower the entropy of the residue. The amount of discarded information is parameterized by the quantization step ∆, which is itself set through the quantization parameter (QP): ∆ ∝ 2 QP 12 .

CABAC: Context Adaptive Binary Arithmetic Coding

At the end of the encoding process, all the values required for the decoding process are transmitted using the Context Adaptive Binary Arithmetic Coding (CABAC) algorithm.

This encompasses signal data (i.e. the quantized and transformed residue) and multiple side-information such as the partitioning, the prediction mode or the transform type.

Due to the practical consideration presented in the previous chapter, the arithmetic coder operates only on binary messages, requiring all the values to be binarized beforehand.

The probability model of a binary message is parameterized through a single parameter

P ∈ [0, 1],
corresponding to the probability of being equal to one. One benefit of the CABAC is to adjust P throughout the encoding process by adapting P to the context of the message.

For instance, having many consecutive messages equal to 1 increases P. Since there are different sorts of message (residue, motion vectors, intra prediction mode), several independent contexts are used simultaneously, each one dedicated to a type of message.

From coding still images to video sequences

A video sequence exhibits a significant level of statistical dependencies along the temporal axis. As such, leveraging the relevant information contained in already sent frames allows to lower the amount of data to send for the current frame to code. Frames which depend on previously transmitted frames are called inter frames. Inter frame coding closely follows the processing described for intra frames. Inter frames are partitioned into blocks, which are processed using the prediction-transform-quantization pipeline, followed with arithmetic coding to transmit the selected information. However, an important additional prediction mode is available: the inter prediction.

Inter prediction

While the intra prediction focuses on the removal of spatial relationship between blocks, the inter-prediction targets temporal redundancies. It is achieved through a process called motion compensation. It usually applies a translational motion to a reference block, extracted from one of the already transmitted frames. The motion is parameterized by a motion vector v = v x , v y , indicating the horizontal and vertical displacements between the block to code and its reference block. The motion vector value and accuracy is selected to optimize the rate-distortion cost. It is sent to the decoder to ensure proper decoding.

Modern coders refine the motion compensation process through bi-directional prediction. It consists in feeding two reference blocks (from two already received frames) alongside two motion vectors to the motion compensation process. The final predicted frame is the weighted average of the two motion compensations. This allows to improve the relevance of the prediction by reducing its noise, as illustrated in the following toy example. Let us suppose that a prediction operation w is applied on two reference blocks xp and xf using their respective motion vectors v p and v f . The prediction function allows to retrieve the block to code x t up to noise terms ǫ p and ǫ f :

w( xp ; v p ) = x t + ǫ p ; w( xf ; v f ) = x t + ǫ f . (2.3)
The terms ǫ p and ǫ f denote two independent noises, whose variance σ 2 w is assumed to be identical since they are obtained through the same prediction process w. These two intermediate predictions are combined through a sum weighted by β ∈ [0, 1]. This yields the bi-directional prediction x:

xt = βw( xp ; v p ) + (1 -β)w( xf ; v f ) = x t + βǫ p + (1 -β)ǫ f .
(2.4)

The bi-directional prediction weighting β allows to select the contribution of each reference block. Typically, the value of β is taken from a discrete set of possible values.

Very often, β is set to 1 2 , but other values can be useful in peculiar cases e.g. fading transitions from one scene to another. The purpose of the bi-directional prediction is to decrease the noise in the prediction error r. From Eq. (2.4) it comes that the prediction error is:

r = x t -xt = βǫ p + (1 -β)ǫ f .
(2.5)

Since ǫ p and ǫ f are independent noises with and identical variance σ 2 w , the energy of the prediction error is:

σ 2 r = β 2 σ w + (1 -β) 2 σ w = (2β 2 -2β + 1) σ w ≤ σ w ∀ β ∈ [0, 1] . (2.6)
As such, the energy of the prediction error is smaller when using two reference blocks.

Coding structure

Encoders select the reference blocks from a restrained set of already transmitted frames, called reference frames, which are defined by the coding structure. The coding structure is a fixed repetitive pattern, often called a Group of Pictures (GOP). The number of frames in a GOP indicates the maximal temporal distance available for an inter prediction. Among the inter frames, a distinction is made between B-frames and P-frames. While B-frames have two reference frames available, P-frames have a single one.

Besides the performance aspect, the coding structure has to comply with a significant set of practical constraints. In most video streaming use cases, the Random Access (RA) coding configuration is used. This coding structure is depicted in Fig. 2.5a. The intra period which corresponds to the number of inter frames between two intra frames sets the maximum latency before the first decoded frame is displayed. Indeed, when an user wants to access a video (e.g. on his TV), the content can only be accessed within an I-frame. The common test conditions (CTC) of VVC [START_REF] Bossen | VTM common test conditions and software reference configurations for sdr video[END_REF] conditioned the intra period on the frame-rate in order to approximately have one I-frame per second. For instance, the VVC CTC states that a video at 30 frames per second has an intra period of 32 frames.

Some peculiar use cases such as video conferencing impose strong latency constraints.

As it is not practical to wait for future frames to be decoded, specific coding structures are designed to compress frames without relying on future frames. This reduces the latency at the expense of compression efficiency. Such structures include the low-delay P (LDP) configuration, presented in Fig. 2.5b.

Conclusion

This chapter has introduced traditional video coding, which implements the operations motivated from a source coding perspective in the first chapter: prediction, transforms, quantization and arithmetic coding. The traditional video coding scheme is a particular implementation of the lossy video coding framework. This traditional implementation has to address a large set of constraints (complexity, handcrafted and linear operations), that restricts the compression performance. Moreover, each step of the coding scheme has to be separately designed through potentially misaligned proxy metrics, which can yield to suboptimal results.

The next chapter introduces deep neural networks (DNN) as a novel set of tools for image and video coding. Thanks to their ability to represent non-linear operations designed to optimize any (differentiable) functions, DNNs relax the constraints on the different steps of the compression process. Furthermore, DNNs enable end-to-end optimization of the whole coding scheme. Consequently, DNN-based coding scheme should reach further coding performance.

LEARNING TO COMPRESS IMAGES

Introduction

N

EURAL networks have recently seen a surge in their popularity and are at the heart of state-of-the-art systems for many different applications, such as image classification, natural language processing or generative modelling. This is due to the neural network ability of being an universal function estimator [START_REF] Goodfellow | Deep Learning[END_REF], allowing them to fulfil virtually any tasks. Neural networks consist of a succession of linear and non-linear operations, whose parameters are set to optimize a particular objective function. This allows to design powerful non-linear functions, achieving better performance than the more simple handcrafted (usually linear and optimized through proxy metrics) functions.

In the context of compression, neural networks are used as transforms, predictions and probability models. This chapter introduces the concepts and techniques of learning-based image compression, which will be extended to video coding in the next chapters. The first half of this chapter presents the main operations performed by neural networks as well as their training process. The second half of the chapter discusses neural networks applied to image coding. The overall framework of learned image coding is explained. Then, different implementations are proposed, successively increasing the coding scheme performance.

Basics of neural networks

Building a neural network

A neural network performs a function y = f (x), mapping an input x to an output y. Such mapping can consist in labelling an image into different categories, removing noise in an input or obtaining a compressed representation, suited for arithmetic coding. All the neural networks presented in this manuscript are modelled as a succession of functions f i called layers. The network output is obtained by applying successively all the layers:

y = f (x) = f L • f L-1 • . . . • f 1 (x), (3.1)
where y is the network output and L ∈ N denotes the number of layers. The organization and the operations performed by all the layers are called the network architecture. Each layer f i outputs a hidden state h i , i.e. a vector acting as an internal representation of the input data. The hidden states represent prominent features of the data e.g. edges or textures for images. In most cases, a layer f i is decomposed as a linear operation, represented by a matrix W i and a bias b i , followed by a non-linear operation σ i :

h i = f i (x) = σ i (W i x + b i ) . (3.2)
Some of the most common non-linearities σ such as the Leaky Rectified Linear Unit (ReLU) or the sigmoid functions are illustrated in Fig. 3.1.

-4 -2 2 4 -4 -2 2 4 x LeakyReLU(x) (a) LeakyReLU(x) = max (ǫx, x) with ǫ ≪ 1. -4 -2 2 4 -1 1 x Sigmoid(x) (b) Sigmoid(x) = 1 1+ e -x

Training a neural network

The training objective is to find the optimal set of parameters θ ⋆ to represent the desired function. Ideally, this function maps an input x to an output y = f (x; θ), which has to be close to the desired output y ob j . Moreover, the network must generalize to (i.e. perform well on) all inputs x drawn from a distribution q, such as the natural images distribution.

Basics of neural networks

These requirements yield the following optimization problem:

θ ⋆ = arg min θ L(θ) = arg min θ E x∼q J f (x; θ), y ob j , (3.3)
where L is the loss function to minimize and J is a per-example cost, measuring the network performance on one single example. The minimization problem is solved using an iterative gradient descent algorithm [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF]. After a random initialization of the network parameters θ 0 , the parameters are updated as follows:

θ n+1 = θ n -η∇ θ L(θ), η ∈ R. (3.4)
Here, ∇ θ L(θ) denotes the gradient of the loss function with respect to the network parameters θ. This equation states that at each step, the parameters are updated towards the steepest descent of the average loss function. The learning rate η sets the displacement magnitude. In theory, the gradient descent algorithm converges towards the optimal set of parameters θ ⋆ as long as the loss function is convex [START_REF] Boyd | Convex Optimization[END_REF].

Two issues arise in practice. First, there is no guarantee that the loss function is convex. Second, it is not possible to compute the exact expectation of the per-example cost J over the whole distribution of x. The first issue remains an open problem, sometimes tackled through the adaptation of the network architecture, which affects the convexity of f . The second issue is solved using the Stochastic Gradient Descent (SGD) [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF], which estimates the gradient based on a reduced subset of samples x (i) , i = 1, . . . , B , called a batch. As a result, the average loss function becomes:

L(θ) = 1 B B i=1 J f (x (i) ; θ), y (i) ob j . (3.5)
Estimating the derivative of the loss with respect to each network parameter is achieved using the backpropagation algorithm [START_REF] Goodfellow | Deep Learning[END_REF]. It consists in propagating the gradient backward, from the output to the input layers. This is often called the backward pass.

By analogy, the process of computing the network output y = f (x; θ) is called the forward pass. Let us illustrate the backward pass on the L-layer network described in Eq. (3.1).

For the sake of brevity, the final network output y is supposed to be scalar and each layer f i is assumed to have a single scalar parameter θ i and a scalar hidden state h i . The loss gradient with respect to a parameter θ i is expressed using the chain rule as:

∂L ∂θ i = ∂L ∂ y × ∂ y ∂h L-1 × ∂h L-1 ∂h L-2 × . . . × ∂h i+1 ∂h i × ∂h i ∂θ i . ( 3.6)
The successive multiplications represent the gradient propagation from the output layer to the i-th layer. This equation highlights a major constraint of neural networks training:

all layers as well as the loss function must be (piece-wisely) differentiable with respect to both their input and their parameters.

Convolutional neural networks

A layer composed of a generic matrix multiplication followed by a non-linearity, as de- The number of parameters is imposed by the dependence on the spatial position. Indeed, each output pixel of a fully connected layer is obtained by its own set of parameters, applied on all input pixels. As such, detecting the same feature for all N × N input pixels requires to learn a dedicated set of N × N parameters for each output pixel, even though the task to perform is identical for all pixels. This issue is solved by trading the generic matrix multiplication for a convolution between the input pixels and a small set of parameters called a kernel. A unique kernel is slid over the input, computing each output pixel through a weighted sum of the neighbouring pixels. As such, the convolutional layer is invariant to translation [START_REF] Goodfellow | Deep Learning[END_REF]. That is, the same processing is applied regardless of the position of the input pixels within the image. Insightful convolutional layers visualisations are available in [START_REF] Dumoulin | A guide to convolution arithmetic for deep learning[END_REF].

The output of the convolution with one kernel is called a feature map. It is a filtered version of the input, aiming to extract relevant features (e.g. edge detection). Usually, more than one feature is extracted from the input by applying F different convolutional kernels in parallel. Successive convolutional layers are often applied to refine the extracted features. In order to process a three-dimensional input (F feature maps with a N × N spatial dimension), each kernel is three-dimensional and composed of F × k × k weights.

Learning-based lossy image coding

Autoencoders

Since Ballé [START_REF] Ballé | End-to-end optimized image compression[END_REF] and Theis' work [START_REF] Theis | Lossy image compression with compressive autoencoders[END_REF] in 2017, most learned compression systems [START_REF] Cheng | Learned image compression with discretized gaussian mixture likelihoods and attention modules[END_REF]- [START_REF] Lee | Context-adaptive entropy model for end-to-end optimized image compression[END_REF] operate using autoencoders. Autoencoders are specific neural network architecture,s whose purpose is to find a latent representation y of an input data x. Ideally, y is a meaningful low-dimensional representation of x, which encodes the most important properties of the input. Beside compression, this simplified representation can be used for a wide variety of task e.g. it is a more convenient input for classification or noise removal.

To compute the latent representation, an autoencoder is composed of two sub-networks:

the analysis and the synthesis transforms, presented in Fig. 3.3a. First, the analysis transform g a maps an image-domain input x to a latent domain variable y. Then the synthesis transform g s maps back the latent variable y to the input domain by computing

x. To foster the learning of a meaningful representation, constraints are imposed on the latent variable y [START_REF] Goodfellow | Deep Learning[END_REF]. In the context of compression, the constraint targets the entropy (i.e. the rate) of the latent variable. In order to be efficiently sent to the decoder, the latent variable is losslessly compressed using arithmetic coding. Consequently, the processing pipeline of a compression autoencoder features two additional components: a quantization step and a latent probability model. The quantization step is present since arithmetic coding only operates on discrete data. The latent probability model is required to allocate less bits to the most probable latent variable values. The synthesis and analysis transforms are not constrained to be the inverse of each other. Thus, the quantization step is not the sole lossy step in the coding pipeline: the analysis transform also removes information. The resulting autoencoder-based architecture yields a coding scheme similar to the abstract lossless coding pipeline presented in Fig. 1.4. The analysis g a and synthesis g s transforms represent the transforms T a and T s in Fig. 1.4. So far, no prediction step is used in the learned coding scheme.

Training a learned coding scheme

The analysis, the synthesis and the probability model are implemented with neural networks, whose parameters are learned during a training stage. This enables the joint optimization of the entire coding scheme, in an end-to-end fashion. This is different from traditional approaches, where each step (prediction, transform, arithmetic coding) is separately and incrementally optimized potentially leading to suboptimal systems.

Yet, according to Eq. (3.6), the training process requires each operation of the coding scheme to be differentiable with respect to its input. This is not the case for the arithmetic coding algorithm which operates on discrete data. Additionally, the quantization step derivative is defined almost everywhere, but is always equal to zero (see Fig. • the straight-through estimate [START_REF] Theis | Lossy image compression with compressive autoencoders[END_REF] sets the quantization derivative to 1 during the backward pass while still performing the rounding operation in the forward pass;

• the soft quantization [START_REF] Agustsson | Soft-to-hard vector quantization for end-to-end learning compressible representations[END_REF] approximates the quantization with a differentiable function (sum of sigmoids) for both forward and backward passes;

• the additive noise [START_REF] Ballé | End-to-end optimized image compression[END_REF], [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF] models the quantization by adding an independent uniform noise for the forward and backward passes. At high rate, this is known to mimic the quantization effect [START_REF] Kleijn | A basis for source coding[END_REF].

Once the training stage is done, all three methods use the real quantization to actually compress images. All the systems presented in this manuscript rely on the additive uniform noise model for their training, as it usually offers better performance. Indeed, recent works [START_REF] Guo | Soft then hard: rethinking the quantization in neural image compression[END_REF] hint that it may be due to the regularization effect of the stochastic noise addition, helping the learning of relevant and expressive latent variables. In this setting, the quantized latent variable ŷ is replaced by the relaxed latent variable ỹ:

ỹ = y + u, with the noise u ∼ U - 1 2 , 1 2 . ( 3
p ỹ( ỹ) = p y * p u ( ỹ) = ỹ+ 1 2 ỹ-1 2 p y (a) da (3.9) = ỹ+ 1 2 -∞ p y (a) da - ỹ-1 2 -∞
p y (a) da

= c y ỹ + 1 2 -c y ỹ - 1 2 ,
with p u the PDF of the continuous uniform distribution introduced in Eq. (3.7). The above equation allows to express the PDF of the relaxed latent variable through c y , the cumulative distribution function of the non-quantized latent variable y. Moreover, it highlights that the p ỹ is an interpolation of p ŷ, with equality for all integers:

p ỹ(n) = p ŷ(n), ∀ n ∈ Z.
(3.10)

Consequently, modelling the non-quantized latent variable distribution p y allows to obtain the distribution of the quantized (and relaxed) latent variable, as illustrated in Fig. The coder objective is to minimize both the rate and the distortion. The latter is denoted as a function d, e.g. the mean-squared error or the MS-SSIM. The rate constraint is expressed through a Lagrange multiplier λ, yielding the loss function:

L λ = E x d(x, x) -λ log 2 p ỹ( ỹ)
(3.12)

Initial architecture

Because of their desirable behaviour for image processing, convolutional neural networks are used to implement the analysis transform g a and the synthesis transform g s . A typical architecture of a learning-based image coder [START_REF] Ballé | End-to-end optimized image compression[END_REF], is presented in Fig. 3.5. Varying the number of kernels F in the internal convolutions provides a simple-yet effective-way of trading computational complexity for performance. Since g a and g s are convolutional transforms, the latent variable y = g a (x) consists of multiple two-dimensional feature maps. To foster the reduction of the latent variable entropy, the analysis transform performs a spatial downsampling while the synthesis transform upsamples the latent variable to restore the original input dimension. For an H × W input image, it is common [START_REF] Cheng | Learned image compression with discretized gaussian mixture likelihoods and attention modules[END_REF]- [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF] to have latent feature maps with a spatial dimension of H 16 × W 16 . In practice, the number of latent feature maps F y is set big enough to not constrain the autoencoder (i.e. the constraint is imposed on the entropy of the latent, not their dimensionality).

An additional benefit of using convolutional transforms is the seamless processing of colour images. Traditional approaches consider the different colour channels separately.

Here, the convolutional nature of the learned coding scheme allows to naturally process colour images. Indeed, a colour image (e.g. a RGB image or a YUV video sequence) is seen by the first convolutional layer as a three-feature-maps input.

The non-linearity function implemented in neural image coder is usually a variation of the Generalized Divisive Normalization (GDN) [START_REF] Ballé | Density modeling of images using a generalized normalization transformation[END_REF]. The GDN is designed as a Gaussianizing transform, with the objective of producing a set of marginally independent Gaussian variables. Here, the Gaussian distribution of the variables is not of primary interest but the independence between the different latent feature maps is important, as they are transmitted independently. Let us denote w i (k) the i-th pixel in the k-th feature map produced by a convolutional layer. The purpose of the GDN operation is to reduce the statistical dependencies among all the i-th pixels across the F feature maps i.e. for k = 1, . . . , F. It is achieved through the following transform:

u i (k) = w i (k) β k + F j=1 γ k, j w i ( j) 2 1 2 (3.13)
where u i (k) is the output pixel and β k , γ k, j ∈ R + are learned parameters. Thus, a GDN transform requires F 2 + F parameters for F feature maps. Similarly, the inverse GDN (IGDN) is defined as:

u i (k) = w i (k) β k + F j=1 γ k, j w i ( j) 2 1 2 . (3.14)
Unlike the non-linear functions presented in Fig. 3.1, the (I)GDN is not a point-wise non linearity. As it mixes different inputs with several learnable parameters, the GDN is able to represent richer functions yielding better rate-distortion performance [START_REF] Ballé | End-to-end optimized image compression[END_REF]. It is worth noting that common deep learning techniques such as the batch normalization [START_REF] Ioffe | Batch normalization: accelerating deep network training by reducing internal covariate shift[END_REF] do not bring any improvement on a GDN-based system [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF]. This is due to the normalization effect of the GDN, which is able to center and re-scale the data. For all these reasons, the GDN and its inverse are found in a substantial number of works [START_REF] Hu | Learning end-to-end lossy image compression: a benchmark[END_REF].

In order to perform arithmetic coding or to estimate the rate during the training process, a probability model p of the latent is required. To tackle the dimensionality issue mentioned in Chapter 1, the latent pixels are coded independently:

p y (y) = k,i p y k,i (y k,i ), (3.15)
With y k,i the i-th pixel in the k-th feature map of y. Yet, estimating a probability model p y k,i

for each latent pixel is not desirable. It would require a significant number of parameters and forbid the processing of variable size inputs (as the dimension of the latent feature maps depends on the input dimension). Ballé et al. [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF] propose to learn one single probability model p y k per feature map:

p y (y) = k,i p y k (y k,i ). (3.16)
Each distribution p y k is represented by its cumulative c y k , see Eq. (3.9), parameterized by a fully connected network. In order to represent a cumulative c (indices omitted for clarity), the neural network is designed to fulfil the following constraints:

c (-∞) = 0 ; c (∞) = 1 ; ∂c(x) ∂x ≥ 0.
(3.17)

Training and rate-distortion results

In order to measure their performance, learning . This proves the relevance of using non-linear transforms. Although the non-linear systems outperform JPEG, they underperform compared to HEVC and VVC. The next section details how to improve the system to reach performance competitive with VVC.

Advanced learned coding schemes

Hyperprior to refine the probability model

According to Eq. (1.5), a mismatch between the latent variable probability model p and its actual distribution q brings a rate overhead of D K L (q || p) bits. As such, an efficient way of improving the rate-distortion performance of the system is to reduce this mismatch through a better probability model estimation. The current probability model introduced previously presents two main weaknesses. First, it is assumed to be spatially invariant i.e. The hyperprior mechanism [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF] proposes to condition the probability model p on both the spatial position and the input image. This is achieved by conditioning p on the side information ẑ, called the hyperprior. This hyperprior is sent using an additional autoencoder, with an adapted analysis transform h a and synthesis transform h s . The hyperprior analysis transform computes z using the latent variable:

z = h a (y) . (3.19)
The hyperprior z is then quantized and transmitted using arithmetic coding. Its probability model p z is a feature-wise cumulative function as explained before for the latent variable y. The hyperprior synthesis transform takes the quantized hyperprior as input to compute ψ, a vector of parameters conditioning the latent variable distribution:

ψ = h s ( ẑ) . (3.20)
In Ballé and Minnen's initial work [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF], [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF], the latent distribution parameters consists of a pixel-wise mean value and standard deviation, ψ = µ, σ . For a latent variable y ∈ R F×H×W , ψ ∈ R 2F×H×W which correspond to two parameters µ i and σ i per latent pixel y i . Consequently, each latent pixel y i is modelled with its probability distribution p y i , with a closed form cumulative distribution function, typically a Gaussian or a Laplace distribution. In this thesis, a Laplace distribution L is used:

p y i ∼ L µ i , σ i . (3.21)
The hyperprior mechanism makes the latent probability model p y i vary according to the pixel index i (the spatial dimension). As ψ is computed from the latent variable, it is dependent on the input image and able to match the image statistics. Figure 3.8 shows an implementation of the hyperprior-based system, derived from [START_REF] Ballé | Variational image compression with a scale hyperprior[END_REF]. Since the hyperprior ẑ is sent as side information, its associated rate has to be taken into account in the loss function. Equation (3.12) becomes:

L λ = E x d(x, x) -λ log 2 p ỹ ( ỹ) + log 2 p z ( z) . (3.22)
Hyperprior-based systems (Fig. 3.8) are evaluated against fixed probability systems (Fig. 3.5) with a similar number of internal feature maps F = 192. The rate-distortion results are shown in Fig. 3.9. The hyperprior-based systems achieve a BD-rate of -32 % compared to the fixed probability systems, significantly improving the coding performance.

Besides the performance increase, the relevance of the hyperprior can be visually assessed. As each latent pixel is independently coded, the system must strive to make them independent to avoid a significant rate overhead. Figure 3.10b presents one feature map of y, where each latent pixel exhibits a high spatial correlation with its neighbours.

The Fig. 3.10c shows the inter pixel correlation captured by µ by representing yµ.

Similarly, Fig. 3.10d indicates the structure captured by σ through the visualisation of There is an interesting relationship between modelling the latent variable through an adaptive probability model and performing adaptive operations on the signal:

y i ∼ L(µ i , σ i ) ⇔ y i -µ i ∼ L(0,σ i ). (3.23)
This equation gives a meaningful interpretation to the quantities µ and σ. The expectation µ aims to predict the value of the latent variable y, while the standard deviation σ indicates the incertitude of the prediction. This highlights that µ acts as a prediction of the signal, conditioned on some previously sent values: the hyperprior z. This latent domain prediction is a significant improvement over the naive architecture, presented in Section 3.3.3, which lacks a prediction step.

Unlike the autoencoders previously introduced, the hyperprior autoencoder does not aim to copy its input. Indeed, the hyperprior autoencoder maps the latent variable y to the hyperprior z and then to the probability distribution parameters µ and σ. That is, the input of the (hyperprior) analysis h a and the output of the synthesis h s do not lie in the same domain. Even though this is not a conventional autoencoder, such analysissynthesis architecture will be called autoencoder in this manuscript. Nevertheless, this is an important difference with the conventional autoencoders, where the synthesis output aims to replicate the analysis input e.g. image-domain input and output. One insightful lesson of the hyperprior mechanism is that the autoencoder architecture is able to simultaneously:

1. Compute any quantity (in this case probability parameters and in future systems, motion information, coding mode selection etc.) based on image-domain inputs; 2. Transmit this quantity as a low-entropy latent variable suited for arithmetic coding. This will be used in all further systems to efficiently estimate and transmit any desired quantity that has to be computed at the encoder and available at the decoder.

Auto-regressive probability model

The hyperprior probability model performance proves the relevance of modelling each latent pixel with its own Gaussian or Laplace distribution. Auto-regressive models (ARM) [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF], [START_REF] Lee | Context-adaptive entropy model for end-to-end optimized image compression[END_REF] have been proposed to improve hyperprior-based systems by also conditioning the latent variable distribution on previously transmitted values. Indeed, the i-th quantized latent pixel ŷi is likely correlated with the previously transmitted pixels ŷ<i . ARM mechanisms is shown in Fig. 3.11. The already received latent pixels ŷ<i are fed to the ARM h r , which extracts relevant information. Finally, information coming from the hyperprior synthesis h s and the ARM h r are fused using a fusion module f :

ψ i = f (h s ( ẑ) , h r ( ŷ<i )) . (3.24)
In practice, the ARM is a PixelCNN-like network [START_REF] Van Den Oord | Conditional image generation with PixelCNN decoders[END_REF], [START_REF] Salimans | Pixelcnn++: improving the pixelcnn with discretized logistic mixture likelihood and other modifications[END_REF] i.e. a convolutional neural network with masked kernels processing only causal pixels. Its detailed architecture is available in appendix B.3. The fusion module is also a PixelCNN-like network, whose architecture is depicted in appendix B.3.

Figure 3.12 presents the rate-distortion gain brought by the addition of an ARM.

Compared to hyperprior-based system, using both hyperprior and ARM yields a BD-rate of -7 %. Such results are consistent with the literature [START_REF] Minnen | Joint autoregressive and hierarchical priors for learned image compression[END_REF], [START_REF] Lee | Context-adaptive entropy model for end-to-end optimized image compression[END_REF] where rate gains are claimed to be around 10 %.

Yet, the ARM makes the entire decoding process sequential, as one latent pixel can not be processed while all the previous one have been decoded. While the hyperprior mechanism allows to obtain the probability parameters ψ all at once, the ARM requires to perform one iteration per (dependent) latent pixels, often leading to a number of iterations in the order of H × W. This results in a prohibitive decoding time [START_REF] Mentzer | Practical full resolution learned lossless image compression[END_REF], even though some approaches aims to decrease the number of dependent pixels [START_REF] Minnen | Channel-wise autoregressive entropy models for learned image compression[END_REF]. In this work, we consider that the relatively minor performance gain was not worth the significant increase in complexity.

Attention modules for better networks

Moreover recent approaches [START_REF] Cheng | Learned image compression with discretized gaussian mixture likelihoods and attention modules[END_REF], [START_REF] Guo | Causal contextual prediction for learned image compression[END_REF]- [START_REF] Liu | A unified end-to-end framework for efficient deep image compression[END_REF] propose to improve the rate-distortion performance through more powerful transforms. This is achieved through residual blocks and attention modules inside the different transforms of the coding scheme. More information on these architectures are available in appendix B. 3.13. The attention-based system offers a BD-rate of -31 %, allowing to significantly outperform HEVC. This highlights the relevance of using more advanced architectures.

It is interesting to note that even such rich architectures exhibit slightly worse performance at higher rates. This is a known problem in the literature, which may be due to the intrinsic nature of the operations performed within the coding pipeline. Indeed, the transforms g a and g s are not lossless. Constraining the operations to be invertible (i.e.

Binary Probability Model

The successive introduction of the hyperprior and the ARM indicates that a better probability model for the latent variable is an important source of performance improvement.

Following this trend, some recent work propose to add more parameters to the probability model, allowing to more accurately represent the latent distribution. Usually, this is achieved by using mixtures of Gaussian distributions [START_REF] Liu | A unified end-to-end framework for efficient deep image compression[END_REF], [START_REF] Cheng | Learned lossless image compression with A hyperprior and discretized gaussian mixture likelihoods[END_REF]. One of the early contribution of this thesis is to propose the binary probability model (BPM): an alternative means of refining the latent variable probability model [START_REF] Ladune | Binary probability model for learning based image compression[END_REF].

So far, hyperprior-based systems model each latent pixel with a Laplace distribution.

As such, the centered latent pixel ȳi = y i -µ i follows:

y i ∼ L µ i , σ i ⇔ ȳi ∼ L(0,σ i ) (3.25)
As the most frequent values of ȳi tend to be around zero, it is particularly important to precisely model the centered latent pixel distribution in this range to avoid a significant rate overhead. Following this intuition, the binary probability model (BPM) features additional parameters to explicitly model the probability of ȳi being quantized to the value -1, 0 or 1. This allows to achieve a more accurate probability model.

The BPM is implemented by modifying the signalling process of each latent variable.

Instead of simply being conveyed at once through arithmetic coding based on a Laplace distribution, the latent variable is decomposed into different quantities. First, each latent variable is centered and quantized. For sake of clarity, the latent variable index is omitted and y stands for any y i . The value to convey is:

ŷ = Q ( ȳ) , with ȳ = y -µ. (3.26)
The signalling process of one latent variable is shown in Fig. 3.14. Since ȳ is assumed centered, it focuses on | ŷ| i.e. amplitude of ŷ while the sign is conveyed has an additional bit. The signalization of the amplitude is inspired by the binarization scheme of the CABAC. That is, | ŷ| is decomposed into two binary flags:

G 0 = | ŷ| > 0 and G 1 = | ŷ| > 1. (3.27)
The flag G 0 indicates whether ŷ is greater than 0. Similarly the flag G 1 indicates whether it is greater than one. These two binary flags are transmitted using arithmetic coding, which requires to model their respective probabilities with two parameters P 0 and P 1 . Finally, if the amplitude is greater than one i.e. if | ŷ| = k > 1, then the value k is explicitly conveyed. In this case, the probability used for the arithmetic coding of k is:

p (k) = p (| ŷ| = k | | ŷ| > 1) = 2 k+0.5 k-0.5 p ȳ (a) da 1.5 -1.5 p ȳ (a) da , ( 3.28)
The probability density function of the centered latent variable p ȳ, is modelled with a centered Laplace distribution i.e. p ȳ ∼ L(0,σ). The BPM can thus be thought of as an enhancement of the Laplace model, as it introduces two additional parameters (P 0 and P 1 ) to more accurately model the latent distribution, allowing to decrease the rate. The BPM is implemented through a single modification of hyperprior-based systems, such as those presented in Fig. 3.16a is fed to the coder analysis transform, which computes a latent variable sent to the decoder. The spatial distribution of the latent variable rate is illustrated in Fig. 3.16d. As expected, the high-frequency areas (edges, text) require more rate to be transmitted, while low-frequency area (the blurry background) costs very little rate. Finally, the synthesis transform uses the latent variable to generate the compressed image presented in Fig. 3.16b.

Visualisations

In order to respect a tight rate constraint, the learned coder discards some information.

It mostly concerns the details, as it is measured as less harmful according to the PSNR (the training quality metric). For instance, the heavy grain of the original image presents a heavy grain (visible on the camera) is removed by the compression process to reduce the rate. Similar visual degradations are observed on the hands, where most of the details are lost during the compression. For comparison, the visual result obtained by coding the image with the VTM at a similar rate is presented in Fig. 3.16c, allowing to note some difference concerning the nature of the visual degradations. Due to its convolution operations, the learned coder tends to lose most of the sharpness, resulting in smooth images. On the other hand, the VTM performs a block-based processing which causes more discontinuities in the compressed image.

A word on complexity

Most of the improvements introduced in this chapter come at the cost of additional parameters, increasing both the memory footprint and the complexity. Yet , the number of parameters of a given design can be tuned by modifying the number of internal convolutional kernels. This enables to modulate the complexity without dramatically altering the overall architecture. Applying this technique to all the architecture allows to obtain performance-complexity curves, that exhibits whether a design is intrinsically better at similar complexity (e.g. whether hyperprior models are better than fixed-pdf models) or if this is only due to the presence of additional parameters.

Although the number of parameters is an important factor of the overall complexity, it is not a satisfactory measurement. For instance, a single convolutional layer with lots of input and output feature maps can have the same number of parameters as many small successive layers. On appropriate computing devices, it is possible to efficiently parallelize the big convolutional layer while the successive small ones are inherently sequential, leading to a more important execution time. The encoding and decoding time could be a relevant measurement of the complexity. Yet, it omits an important dimension Consequently, the complexity is measured through the training time. This somehow reflects the execution time, as one iteration and thus the whole training will be shorter if the execution time of the architecture is small. The performance of an architecture is computed by averaging the rate-distortion cost J for different rate-constraint λ i :

J = 1 N N i=1 D + λ i R. ( 3

Conclusion

In this chapter, the main concepts of learned compression have been introduced. The most common techniques of the literature (hyperprior, ARM, attention mechanism) as well as one contribution of this thesis (the binary probability model) have been evaluated to assess their coding performance. Table 3.1 summarises the performance gains brought by the successive improvements and highlights two mains levers of performance improvement. The first one is the enhancement the analysis and synthesis transforms through non-linearities and attention modules. The second is the refinement of the latent probability model using hyperprior, ARM and the binary probability model. The resulting learned coding scheme achieves a BD-rate of -55.6 % compared to the initial non-linear autoencoder i.e. it requires 55.6 % less rate to obtain an equivalent quality.

End-to-end training of a coding scheme implementing all these components allows to reach image coding performance competitive with VVC, achieving a BD-rate of +1.4 % when compared to VVC. These are compelling results, as VVC represent the state-ofthe-art of the standardized compression algorithms, which uses a conventional approach.

Furthermore, learned approaches are still quickly improving. In a span of a few years, their performance has gone from JPEG to VVC. Yet, the performance of learned systems comes at the expense of an important complexity and memory footprint. For instance, the best performing approaches presented in this chapter require 20 million parameters to Finally, the different designs presented in this chapter turn out to be generic architecture that can be applied to efficiently transmit any two-dimensional signal. As such, these architectures are the building blocks of the learned video schemes designed in the next chapters of this manuscript.

et al. [START_REF] Habibian | Video compression with rate-distortion autoencoders[END_REF] proposes to straightforwardly extend the learned image coding schemes to video coding. That is, all the T video frames are fed to the analysis transform g a which computes a single latent variable y representing all the stacked frames. The latent variable undergoes a quantization step Q and it is sent to the decoder using arithmetic coding. In the end, the synthesis transform g s reconstructs the T frames in its entirety: x1 , . . . , xT = g s ( ŷ) , with ŷ = Q (g a (x 1 , . . . , x T )) . ( 4.1)

In practice, the analysis and synthesis transforms are implemented using 3D convolutions [START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF]. We recall that the usual 2D convolutions are fed with

F × H ×W input data, that is F two-dimensional feature maps of size H × W. 2D convolutions use several F × k H × k W
kernels to compute the output feature maps. On the other hand, 3D convolutions process 3D convolutions present two main weaknesses. First, compared to the successive processing of T frames with 2D convolutions, 3D convolutions require k T times more parameters and multiplications, with k T often greater than 3. Second, the processing of T frames together prevent to tune the coding structure according to the application needs. For instance, the coding latency is by design T frames. This can be troublesome in some context (e.g. in video-conferencing setup), where the latency has to be as low as possible. In such case, video coders resorts to low-delay coding configurations, featuring a coding latency of a single frame. These two limitations are particularly concerning as the 3D convolution-based approaches have not been shown to bring compelling performance.

F × T × H × W
Consequently, this thesis follows most of the publications from the literature and discards the 3D convolution-based approaches in favour of 2D convolution-based ones, which process each frame separately. This reduces the complexity and gives more flexibility on the coding configuration.

Yet, naively processing each frame separately leads to a rate overhead. As motivated in the first chapter (see Section 1.2.4), a prediction step is introduced to lower the mutual information between the frames, reducing the rate overhead. In practice, it is achieved through a motion compensation process, similarly to traditional coders. [START_REF] Lu | DVC: an end-to-end deep video compression framework[END_REF] SpyNet Yilmaz, ICIP 20 [START_REF] Yilmaz | End-to-end rate-distortion optimization for bidirectional learned video compression[END_REF] SpyNet Hu, ECCV 20 [START_REF] Hu | Improving deep video compression by resolution-adaptive flow coding[END_REF] SpyNet Lu, ECCV 20 [START_REF] Lu | Content adaptive and error propagation aware deep video compression[END_REF] SpyNet Yang, CVPR 20 [START_REF] Yang | Learning for video compression with hierarchical quality and recurrent enhancement[END_REF] SpyNet Golinski, ACCV 20 [START_REF] Golinski | Feedback recurrent autoencoder for video compression[END_REF] AE †

⋆

Liu, AAAI 20 [START_REF] Liu | Learned video compression via joint spatial-temporal correlation exploration[END_REF] AE † Agustsson, CVPR 20 [START_REF] Agustsson | Scale-space flow for end-to-end optimized video compression[END_REF] AE Yang, STSP 21 [START_REF] Yang | Learning for video compression with recurrent auto-encoder and recurrent probability model[END_REF] SpyNet Hu, 21 [START_REF] Hu | FVC: A new framework towards deep video compression in feature space[END_REF] ? latent residual

⋆ Not explicitely computed at the encoder-side, but the decoder features residue + prediction.

† Requires an explicit loss term to converge.

Motion estimation, transmission and compensation

Most learned video coders from the literature use a single reference frame, restraining their performance. In order to obtain a more accurate prediction, bi-directional prediction is preferred:

xt = βw( xp ; v p ) + (1 -β)w( xf ; v f ), with β ∈ [0, 1] . (4.2)
Here, w denotes the warping function used to apply a motion v to a reference frame

x. The two intermediate warpings are combined through a sum weighted by β, the bidirectional prediction weighting. For the vast majority of works as well as for the work presented in this thesis, w is a linear warping i.e. pixels of the motion compensated image are sampled from a linear interpolation of the original image.

In order to compute a relevant prediction, accurate motion information must be available at the decoder. In most existing schemes, a two-step process takes place.

1. Motion is estimated at the encoder using a pre-trained network such as SpyNet [START_REF] Ranjan | Optical flow estimation using a spatial pyramid network[END_REF] or PWCNet [START_REF] Sun | Pwc-net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF];

2. An autoencoder is used to convey the motion information to the decoder.

An other approach is to straightforwardly estimate and transmit the motion at once with a single autoencoder. This last method offers the benefit of a simpler design and will be chosen in this manuscript. Once the motion information is available at the decoder, it must be applied to the reference frames through the motion compensation function. This is usually achieved with a bilinear warping. Chapter 6 details how neural networks are used to estimate, transmit and apply the motion information.

Transmitting the unpredicted part

Once a temporal prediction xt is available, only the part of x t missing from xt is conveyed.

To this end, xt is combined with the frame to code x t through a function m. This function aims to remove the predictable part from x t , to avoid conveying redundant information.

To this end, traditional video coders, as well as most learned ones rely on residual coding.

That is, m is a simple subtraction in the spatial (image) domain:

m (x t , xt ) = Residual x t -xt . (4.3)
Then, the decoder recovers xt as a sum between the prediction xt and the transmitted residual. We will argue in Chapter 5 that performing a simple difference between the frame to code and its prediction can be improved from a more advanced combination of these two quantities. Indeed, the representational power of neural networks will be leveraged to design different mixing operations m. Different designs of m are compared, to determine the most effective way of leveraging a temporal prediction.

Experimental conditions

CLIC 21 video track

The next two chapters carefully consider the design of the motion estimation, transmission and compensation as well as the more effective means of leveraging the temporal prediction (i.e. improving the legacy residual coding). In order to accurately assess the different designs introduced, a real-life video coding task is used: the video track of the Challenge on Learned Image Compression (CLIC) 2021 [START_REF]Workshop and challenge on learned image compression[END_REF]. The objective of this challenge is to compress 100 videos at an average rate of 1 Mbit/s, while getting the highest MS-SSIM.

All videos have 60 frames with a resolution of 720 lines with 30 frames per second.

Anchors

For the next two chapters, the different design choices are assessed recent modern video coders: HEVC and VVC. This decision is motivated by their performance and their flexibility. So far, the vast majority of the learned systems in the literature are assessed against a fast encoder implementation of HEVC of HEVC e.g. x265 using the very fast preset.

Consequently, the performance achieved by the best implementations of modern video coders (such as the HM for HEVC or VTM for VVC) remains an upper bound for learned approaches [START_REF] Ladune | Coding standards as anchors for the CVPR clic video track[END_REF].

Then, most learned coding schemes tend to implement restrained coding configurations.

Indeed, they are often still limited to P-frames and target higher rates [START_REF] Yang | Learning for video compression with recurrent auto-encoder and recurrent probability model[END_REF], [START_REF] Hu | FVC: A new framework towards deep video compression in feature space[END_REF], with a frequent intra period. This makes difficult to achieve proper comparison between the approaches presented in this manuscript, which targets real-life video coding and the literature.

Coding configuration

The coding structure is of primary importance to obtain the best rate-distortion trade-off.

The bi-directional prediction, motivated in section 2.3.1, offers a more accurate prediction which usually results in better coding efficiency. Since, the bi-directional prediction is only available for B-frames, the coding structure must feature as many B-frames as as possible.

For fair comparisons between the proposed learned coders and existing coders, the coding structure from the VVC Common Test Conditions [START_REF] Bossen | VTM common test conditions and software reference configurations for sdr video[END_REF] are selected. As such, the Intra Period is set to 32 frames (approximately one intra frame per second at 30 fps).

Since there is no constraint on the latency, the best performing coding configuration is chosen. This configuration is called the random access configuration and is presented in Fig. 4.2 (with a GOP size of 8). It presents B-frames wherever it is possible, leading to better compression results. To further maximise the number of B-frames, a GOP size of 32 is used.

End-to-end training

Random access coding features I-frames, P-frames and B-frames. As such, one need to process a frame with zero, one or two reference frames with the same set of parameters.

Indeed, at first sight, it is not desirable to have a dedicated intra coder as most existing systems do in the literature. To this end, the training stage has to prepare the coder In order to achieve the highest coding performance, the different components of the coder are jointly optimized according to the actual rate-distortion objective. To this end, all the frames within a GOP are successively coded and the following loss is minimized by the learning algorithm:

L λ = E x t d(x t , xt ) + λr( xt ) . (4.4)
To comply with the experimental conditions of the CLIC video track, the distortion metric d is based on the MS-SSIM. The rate metric r measures the rate required to reconstruct the frame xt at the decoder and the rate constraint λ sets the rate-distortion trade-off.

Framing the optimization task at a GOP level by minimizing the loss function on a whole GOP allows to more accurately represent the propagation of the compression noise among frames. This enables the model to learn the impact of having compressed (i.e. noisy) reference frames, as well as preparing the system to work with compressed references. In the end, it should result in better performance.

Following the common data format in video compression, each frame is represented in YUV 420. For convenience, all three channels are converted to the same spatial resolution.

Several methods are explored in the literature [START_REF] Egilmez | Jvet-u0079: a DNN architecture for intra-frame coding in yuv 4:2:0 format with cross-component prediction[END_REF]. Unlike [START_REF] Egilmez | Jvet-u0079: a DNN architecture for intra-frame coding in yuv 4:2:0 format with cross-component prediction[END_REF], we choose the simple yet efficient nearest neighbour upsampling of the U and V channels. Although this method has a slightly superior computational cost, it is more convenient as it allows to align the spatial dimensions of U and V with those of the Y channel. As a result, each frame is presented as a tensor of dimension 3 × H × W, that is three channels with a spatial

EXPLOITATION OF A PREDICTION

Introduction

V IDEO coders rely on temporal prediction to identify the statistical relationship within the data. Once the prediction of a frame is available, it is subtracted to the current frame to code to remove some of the redundancies. This process is called residual coding and it is implemented by most traditional and learned coders. Yet, neural networks are able to learn how to exploit the prediction beyond a simple difference.

This chapter compares different mixing functions between the frame to code and its prediction. Existing approaches from the literature are compared and a novel architecture called conditional coding is proposed to enhance the compression efficiency. Finally, we propose to condition operations in the coding scheme based on the type of the current frame (I, P or B), increasing the rate-distortion performance.

Baselines and experimental conditions

Naive prediction

This chapter aims to investigate the most effective means to exploit a temporal prediction.

That is, we are interested in the design of the last step presented in Fig. 4.1 which consists in transmitting the unpredicted part of the frame. In this context, let us suppose that a bi-directional prediction process is used to obtain xt , a temporal prediction of x t :

xt = βw( xp ; v p ) + (1 -β)w( xf ; v f ), with β ∈ [0, 1] . (5.1)
The function w applies the motion information v p (resp. v f ) to the reference frame xp (resp. xf ) and β is the bi-directional prediction weighting. For fair comparison between different methods, the temporal prediction must remain identical. To this effect, a naive prediction without any motion information is used. That is, the prediction is simply a combination of the raw reference frames:

xt = β xp + (1 -β) xf = 1 2 xp + xf for B-frames, β = 1 2 xp for P-frames, β = 1 . (5.2)
For I-frames, xt = 0 since they do not have any reference frame available.

Learned and traditional baselines

As presented in Table 4.1, residual coding in the image (i.e. spatial) domain is the usual method to exploit the temporal prediction. This is also the case in all the ITU/MPEG video coding standards. Figure 5.1 shows the autoencoder-based residual coding scheme.

It first mixes the current frame and its prediction through a simple subtraction. Then, only the prediction error x t -xt is sent. Its architecture is based on the components known to perform well for learned image coders (hyperprior, attention modules), with F = 192 internal features. Their exact implementation are available in appendix D.1. To better appreciate the performance of the learned residual coder, a pure image coder is also implemented. It relies on exactly the same architecture than the residual coder, except that it transmits the frame x t without reference.

Different configurations and implementations of traditional coding algorithms are used as anchors to verify the performance of the learned approaches. First, the All Intra (image coding only) configuration of HEVC is tested through the HM (HEVC test Model) 16.22. This serves as a lower bound for the required performances of learned approaches.

Indeed, even the learned image coding scheme should outperform HEVC All Intra (as in Chapter 3). Conversely, we used a fully fledged configuration of HEVC, with an actual motion compensation, to represent the performance of a modern video coder. This is achieved using the HM 16.22 and x265 (medium preset). While the former is arguably the best implementation available, the latter is very often used in the learned compression literature [START_REF] Lu | DVC: an end-to-end deep video compression framework[END_REF]- [START_REF] Yang | Learning for video compression with recurrent auto-encoder and recurrent probability model[END_REF].

Training and testing the baselines

For the learning-based residual and image coding scheme, the training process follows the principles described in the learned image coding chapter. Namely, the quantization is approximated by an additive noise and the rate is measured through the entropy of At the beginning of the training, the compressed frames are not relevant enough to be used as references. Relying on them would result in an irrelevant prediction that would not allow the optimization algorithm to converge. This is avoided through the usage of non-compressed reference frames during the first 20 training epochs out of 50.

After training, the systems are evaluated on the CLIC21 validation set, using a Random Access coding structure with a GOP size of 32 and an intra period of 32. The rate-distortion results are shown in Fig. 5.2. The learned image coder outperforms the HM in All Intra configuration. This is coherent with the image coding result presented in Chapter 3. Since the residual coder has access to a (naive) temporal prediction, it logically outperforms the image coder. The rest of the chapter aims to investigate ways of better leveraging the prediction, without changing the way it is computed, defined in Eq. (5.2).

Conditional coding

Motivations

Let us generalize the task of inter frame coding. Information from an encoder-side quantity

x enc is sent, while a variable x dec containing relevant information is already available at the decoder-side (e.g. a prediction or a reference frame). An ideal inter frame coding scheme is able to leverage all the decoder information. As such, it avoids conveying redundant information i.e. information from x enc which is already available at the decoder, in x dec .

The main idea of residual coding is that the decoder information can be removed from x enc by simply subtracting x dec to it. The results of this subtraction is called the residue. It is conveyed by an autoencoder, through the usual analysis-quantization-synthesis pipeline.

In the end, x dec is added back to obtain the output x out , as in Fig. 5.1:

x out = x dec + autoencoder g s (Q(g a (x enc -x dec residue ))) . (5.3)
This equation highlights two weaknesses of residual coding. First, the mixture between

x enc and x dec is limited to a simple subtraction. Second, the fact that x out results from a final addition between x dec and the autoencoder output requires these quantities to have compatible nature such as image and image difference. Here, we propose to design a novel architecture to overcome these two issues. Its purpose is to condition the transmission of x enc on the decoder information x dec , to transmit as few bits as possible. We call this architecture conditional coding. It must comply with two design constraints:

1. Learn the optimal combination of x enc and x dec , to identify the information unavailable at the decoder, i.e. which has to be transmitted; require any transmission. Through the training process, the conditioning transform learns to compute the conditioning latent variable y c in a form that suits the synthesis transform g s . Indeed, the latent variables of both the analysis and the conditioning transforms are concatenated before being fed to the synthesis transform, which learns to combine the encoder and decoder information and computes the desired output x out :

x out = g s (y, y c ) .

(5.6)

Implementation and rate-distortion performance

The conditional coding mechanism is implemented to replace residual coding. Therefore, in the context of inter frame coding:

      
x enc = x t the frame to code,

x dec = xt the temporal prediction,

x out = xt the decoded frame.

(

Compared to residual coding, the analysis g a and synthesis g s transforms remain unchanged, except for the number of input features. For the analysis, residual coding has a 3-feature input (residual for the Y, U and V channels). Here x t and xt are concatenated, resulting in a 6-feature input (two Y, two U and two V channels). Similarly, the synthesis transform is now fed with the concatenation of two latent variables, y and y c . The conditioning transform replicates the architecture of the analysis, with a 3-feature input (only xt ). Similarly to the residual coder, all transforms use F = 192 internal feature maps.

The training and the inference are performed as for the residual coder. Figure 5.4

shows the performance of conditional coding compared to residual coding. It significantly improves the performance, reaching equivalent quality at a rate 30 % lower. This proves the relevance of performing a richer mixture than a simple difference. Yet, this enhancement comes at the cost of the additional conditioning transform. In this case, the residual coder has 15 million parameters, while the conditional one has 21 million parameters.

Visualisation

This section aims to detail the inner behaviour of the conditional coding. To this effect, the output of the conditional coder is computed while disabling some latent variables (by setting them to zero). Namely, the analysis and conditioning latent variables are alternatively set to zero, as explained in Table 5.1. The visual results are presented in Fig.

5.5, in a B-frame coding context.

For now, the prediction xt is a simple average of the reference frames. As such, it presents some artifacts around moving objects e.g. the girl in Fig. 5.5b. This temporal prediction is fed to to the conditioning transform to obtain the conditioning latent y c = g c ( xt ). Conversely, Fig. 5.5d presents the synthesis transform g s output when computed only from the conditioning latent variable, allowing to represent the information contained in y c . In residual coding, the prediction would be directly added to the synthesis transform output to obtain the reconstructed frame. Thus, any artifacts present in the temporal prediction would be found in the output frame. Here, the conditioning transform allows to remove some of the artifacts. This is particularly visible around the girl's face (Fig. 5.5d),

where most of the visual degradations visible in the temporal prediction are absent from the conditioning-only synthesis. This mechanism is particularly useful at low-rate as it Table 5.1: Different configurations for the conditional coding ablation.

Latent variables Synthesis Analysis Conditioning operation

Normal

g s (y, y c ) Conditioning only g s (0, y c ) Analysis only g s (y, 0) avoids sending corrective terms through the analysis transform.

Figure 5.5e presents the synthesis of the analysis latent variable y only. This latent variable is obtained by applying the analysis transform on both the frame to code and its prediction: y = g a (x t , xt ). As such, it represents the unpredicted part of the frame i.e.

the correction that has to be sent. In this examples, the unpredicted areas concern mostly the girl and the ball. The analysis latent is the only quantity requiring to be sent to the decoder. Consequently, the spatial distribution of the rate (see Fig. 5.5f) shows that bits are spent only for the areas requiring information from the analysis transform. Finally, Fig. 5.5c depicts the usual output of conditional coding i.e. when the latent variables from both the analysis and the conditioning transforms are fed to the synthesis transform.

This illustrates that the combination of the encoder-side and decoder-side information is properly achieved by conditional coding, without noticeable artifacts (except the obvious loss in quality due to lossy compression).

Those illustration shows that the decoder-side information xt is well exploited by the proposed conditional coding. Both for lower rates (Fig. 5.5e) and for higher rates (Fig. 5.6e), the coder conveys only what is not present at the decoder-side.

The set of Figures 5.6 presents the same visualisations as the previous one except they are obtained from a higher-rate network. When the rate-constraint is decreased, the analysis latent variable y are allowed to carry more information. This is clearly visible in Fig. 5.6e, which exhibits more details and is active in more areas than its low-rate counterpart. This translates into a more important rate, as illustrated through the spatial rate distribution, in Fig. 5.6f. As the higher-rate network relies more on the analysis transform, the conditioning transform tends to be less important. For instance, Figure 5.6d shows that it removes less artifacts from the temporal prediction.

Latent domain residual coding

The main idea of conditional coding is the combination x enc and x dec in the latent domain, allowing to perform a richer mixture of these quantities. Likewise, the latent-domain residual approach [START_REF] Djelouah | Neural inter-frame compression for video coding[END_REF] proposes a similar but not identical method, summarised in Fig. 5.7. In this coding scheme, the analysis transform g a is applied successively on x enc and x dec , yielding two latent variables: Since the latent variable y dec is computed based solely on x dec , it is available at the decoder without transmission, enabling to compute a latent-domain residue:

y enc = g a (x enc ) ; y dec = g a (x dec ) . (5.8)
r = y enc -y dec . (5.9)
This residue is quantized and sent to the decoder. At the decoder side, y dec is added back to the quantized residue to retrieve a noisy version of the encoder latent ŷenc , fed to the synthesis transform g s to compute the desired output:

x out = g s ( ŷdec ) .

(5.10)

As with conditional coding, the encoder and decoder information is mixed in the latent domain. Like conditional coding, this approach requires additional decoder-side computation (in order to obtain y dec = g a (x dec )).

Figure 5.8 presents the latent-domain residual coding performance, which offer significantly better results than spatial-domain residual coding. Yet, latent-domain residual coding slightly underperforms compared to conditional coding. Two reasons explain why conditional coding is a more interesting architecture than latent-domain residual coding.

First, the encoder and decoder quantities are still combined through an addition, even though it takes place in the latent domain. Compared with conditional coding which simply concatenates the quantities, it may hinder the expressiveness of the combination.

Second, the same analysis transform g a is used at the decoder-side, instead of learning a specialized conditioning transform g c . Recall that the transform g a aims to reduce the entropy of its input (due to the rate constraint). When performed at the decoder to achieve latent residual coding, it purposely removes information from the temporal prediction.

This might lead to a suboptimal mixture. Besides the performance aspect, latent-domain residual coding presents a second drawback. As x enc and x dec are fed to the same transform g a , with the same parameters, they must have the same nature and dimension. In the current setup (i.e. x enc = x t and

x dec = xt ), this constraint is acceptable. As we will see in the next chapter, it will not be the case for coding of the motion information. In the end, conditional coding is the most effective and the most flexible architecture to exploit decoder-side information. It will be implemented in most of the proposed video coding scheme to exploit decoder-side information as much as possible.

Do we need a dedicated intra frame coder?

Most existing learned video coders feature a dedicated intra frame coder (see the literature review in Table 4.1). This is due to the usage of a image-domain residual coding, where the nature of the conveyed data varies significantly between intra and inter frames. For inter frames, the autoencoder is fed with a difference image, while intra frames require to convey image data. Thus, it looks sensible to learn one dedicated coder for each nature of content. To assess whether a separate intra coder is required, the different systems previously learned are evaluated in an All Intra configuration. For this experiment, the image coder constitutes the upper bound for the performance. Indeed, it is only able of coding intra frames and is thus trained for this task.

Variable quantization steps

Motivation and implementation

During the processing of a video sequence, the role of the analysis transform g a varies significantly. For intra frames, there is no contribution from the conditioning transform g c and the frame is transmitted through the latent variable y = g a (x). In the case of an inter frame, a prediction x is available and the conditioning transform is used. As such, far less information are sent through the analysis latent variable y, as most of the relevant information is already available from the conditioning transform.

Since the importance of the analysis latent variable y depends on the frame type, we propose to apply a linear operation on y, conditioned on the frame type. The proposed method is inspired by the one introduced by Guo et al. for variable rate coders [START_REF] Guo | Variable rate image compression with content adaptive optimization[END_REF] and is presented in Fig. 5.10. For each frame type f ∈ {I, P, B}, a feature-wise pair of quantization gains {Γ enc f , Γ dec f } is learnt. Each quantization gain Γ ∈ R F y with F y the number of feature maps composing the analysis latent variable y. The feature-wise quantization gains are applied prior to quantization at the encoder-side and at the decoder-side after arithmetic decoding. Seen from the synthesis transform, the received analysis latent is:

ŷ = Γ dec f ⊙ Q(Γ enc f ⊙ y). (5.11)
where ⊙ denotes the feature-wise multiplication and Q the quantization function. Conceptually, these frame-type dependent quantization gain can be compared to the ∆ QP of traditional coders, which sets different quantization accuracies depending on the frame type and position in a GOP. Yet, the proposed quantization gains offers more flexibility than the usual ∆ QP :

• A different gain is learned per latent feature maps, compared to a single scalar gain;

• There is no constraint imposed such as

Γ dec f = 1 Γ enc f ;
• The quantization gains can be learned to operate as a switch, enabling or disabling particular latent feature maps according to the frame type;

• All the quantization gains are set through an end-to-end optimization of the entire video coding schemes.

Experimental results

The quantization gains are implemented on the conditional coding scheme. While the conditional coder has 25 million parameters, the three pairs of feature-wise quantization gains represent a few parameters:

Encoder & Decoder × I, P & B × F y = Number of parameters 2 × 3 × 256 = 1 536 (5.12)
The training stage of the system is carried out similarly to the other coding schemes and its results are presented in Fig. 5.11a. For a negligible increase in complexity, the introduction of quantization gains conditioned on the frame type allows to save 9.6 % rate.

To better understand the behaviour of the quantization gains, Fig. 5.11b shows the normalized values of Γ enc for one system. As expected, the quantization gain i.e. the quantization accuracy is different for the intra frames (I) and the inter frames (P & B).

For a given feature map, its associated quantization accuracy is almost always smaller for an inter frame than for an intra frame. Together with the rate-distortion results, this proves the relevance of learning dedicated quantization accuracy according to the type of the frames. Similar visualisations for all rate targets are presented in Appendix D.3, where the behaviour of the quantization gains remain similar regardless the rate target.

Conclusion

In this chapter, several ways of leveraging a temporal prediction have been investigated and the conditional coding architecture has been introduced. Table 5.2. summarises the rate saving brought by each element. Conditional coding couplet with quantization gains conditioned on the frame type, achieves a BD-rate of -35.9 % over the usual spatial residual coding i.e. it obtains equivalent quality at a rate 35.9 % lower. This demonstrates the relevance of the different techniques introduced in this Chapter. Besides its compelling results, conditional coding allows to process intra frames and inter frames with the same parameters, preventing the need for a dedicated intra coder.

Thanks to its performance and flexibility, the conditional coding mechanism has been used throughout the work carried out in this thesis. Conditional coding was introduced in a conference paper presented at the IEEE International Workshop on Machine Learning for Signal Processing (MLSP) 2020 [START_REF] Ladune | Modenet: mode selection network for learned video coding[END_REF]. Yet, the naive temporal prediction of the coding scheme hinders its performance, which explains the significant gap with HEVC (BD-rate of +42.9 % when compared to the HM).

In order to further improve the compression efficiency, a better temporal prediction has to be available. To this end, the next chapter focuses on the design of a proper motion compensation process, which implies to accurately estimate and efficiently transmits motion information. based on the previously received frames. In order to be accurate, the motion compensation step relies on motion information which needs to be properly estimated and efficiently transmitted so that the prediction improves the rate-distortion cost of the codec.

This chapters builds upon the coding scheme designed in the previous chapter based on a naive prediction. In the first half of the chapter, motion information is integrated into the existing coding scheme, improving the compression efficiency. Then, the conditional coding architecture is implemented for the motion information to further increase the performance. Lastly, the resulting coding scheme is evaluated in details to assess its strength and weakness.

Coding scheme featuring a prediction step 6.2.1 Bidirectional prediction

This chapter aims to enhance the coding scheme introduced in the previous chapter.

Let us call CNet, the conditional coder which implements the operation c (x t , xt ) i.e. it conveys the frame x t while leveraging a temporal prediction xt available at the decoder.

In order to improve the compression performance, we propose to move forward the naive

The first component, MNet, estimates and conveys the motion information. Then, Motion

Comp. implements the motion compensation described in Eq. 6.1. Finally, CNet performs the conditional coding of x t conditioned on xt , as presented in the previous Chapter.

Figure 6.2: Video coding scheme with an actual motion compensation step. The first one is a two-step method, with a separate motion estimation network located at the encoder which computes encoder-side motion maps m p and m f . Then, these motion maps are fed to a motion autoencoder which performs lossy coding and retrieves decoder-side motion information v p and v f . The bidirectional weighting β is obtained as an additional output of the motion autoencoder, as in [START_REF] Djelouah | Neural inter-frame compression for video coding[END_REF].

Motion information at the decoder side

The second method consists of a single step, based on an autoencoder fed with the current frame and the reference frames. The autoencoder both estimates the motion and perform lossy coding i.e. finding a low-entropy representation suited for transmission allowing to retrieve motion information at the decoder. In the literature, the single-step approach is only found for P-frame coding [START_REF] Golinski | Feedback recurrent autoencoder for video compression[END_REF]- [START_REF] Agustsson | Scale-space flow for end-to-end optimized video compression[END_REF]. Since we strive to design a simple video coder, the single-step approach is chosen and extended to B-frame coding by adding a supplementary output for β, see Fig. 6.3b. Thanks to this approach, we avoid relying on a dedicated and pre-trained motion estimation network, which can often results in a cumbersome training process and heavier systems.

Regardless of the method used to obtain motion information, the training of the overall coding scheme is not trivial. In particular, obtaining proper motion information at the decoder is notoriously difficult [START_REF] Golinski | Feedback recurrent autoencoder for video compression[END_REF]. In the case of the two-step method, this is tackled by using an off-the-shelf network for the motion estimation such as SpyNet [START_REF] Ranjan | Optical flow estimation using a spatial pyramid network[END_REF] or PWCNet [71]. We argue that integrating pre-trained motion estimation networks is not the ideal solution for several reasons:

• It brings about additional parameters (e.g. 8 million weights for PWCNet);

• Such networks are mostly trained on synthetic datasets, for which the underlying motion is known (e.g. MPI-Sintel [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF]). They are not necessarily tuned for natural video content;

• They aim to predict the most accurate motion information as they are often trained through the minimization of the L 2 error of the motion. Yet, the objective of motion estimation in an actual video coding scheme is not to be the most accurate. Its goal is to obtain motion information leading to the best rate-distortion cost, once integrated in the whole coding chain.

Since pre-trained motion estimation networks add supplementary parameters and require a retraining to be adapted to the video coding objective and content, they are not used in our coding scheme. On the other hand, the single-step method very often requires some sort of pre-training or a dedicated loss term to foster convergence [START_REF] Golinski | Feedback recurrent autoencoder for video compression[END_REF], [START_REF] Liu | Learned video compression via joint spatial-temporal correlation exploration[END_REF].

Yet, adding dedicated loss terms to the rate-distortion objective leads to an optimization inaccurate temporal prediction. Consequently, this prediction can not be copied from the Skip mode and these areas rely on CNet to be transmitted (red areas on the mode selection map). The other areas are well predicted and use mostly the Skip mode (blue areas). In order to be available at the decoder, α must be transmitted. Since it is applied on CNet inputs, its transmission must occur prior to CNet. To this end, an additional output is added to the motion autoencoder MNet to retrieve α at the decoder side.

Training

The entire coding scheme is trained in an end-to-end fashion to minimize the actual rate-distortion loss. This is particularly important to note that there is no dedicated loss term for α, β, v p or v f . The MNet autoencoder learns through the rate-distortion cost to efficiently transmit a relevant and low-rate coding mode selection and motion information.

As such, a simple rate-distortion training allows to perform a relevant partitioning between Skip mode and CNet. Furthermore, the learning step also enables to automatically share the rate budget between MNet and CNet.

Yet, there is a subtlety at the beginning of the training. During the first training iterations, CNet and Skip mode are not ready to compete. Indeed, CNet has to be trained, while the Skip mode is already able to copy the prediction. For this reason, α is set to zero on one half of the image and to one on the other half. The zeroed half relies on the Skip mode, fostering the learning of a relevant motion compensation provided by MNet. The other half uses only CNet, preparing CNet for the competition with the Skip mode.

Behaviour

The new quantities introduced in this Chapter are illustrated through a comprehensive visual example of B-frame coding. The first step is the coding pipeline is the MNet, whose inputs and outputs are shown in Figure 6.7. MNet inputs are the frame to code x t (Fig. The bidirectional weighting β exhibits an interesting behaviour in Fig. 6.7g. In this example, most of the frame is composed of motionless background. For these areas, the value of β is of limited importance, as xp and xf are almost identical and both can serve as reference. Here the system uses β ≃ 0.75, which corresponds to most of the medium blue areas. All the people in the frame are moving. For these areas, the system relies on β ≃ 0.5 (in white), decreasing the prediction error by averaging the intermediate predictions.

The thin dark blue and dark red areas around the girl at the foreground allows the system to elegantly handle disocclusions. Indeed, the girl moves from the left to the right.

As such, the area located at its left side is only visible in the future reference frame xf .

Similarly, the area at the right of the girl is only visible in the past reference xp . Using β ≃ 0 (dark blue) or β ≃ 1 (dark red) enables the usage of a unique reference frame for the motion compensation process, allowing to properly handle disocclusions. This is an interesting behaviour, which is not achieved by the weighted prediction of traditional video coders. More detailed examples of disocclusions are presented in appendix E.3. Once a temporal prediction is available, the coding mode selection α (Fig. 6.7h) arbitrates each pixel between the two coding modes: CNet and Skip. Figure 6.9d shows the areas selected for the Skip mode. Since the videos are in the YUV colour space, zeroed areas appear in green. Thanks to the accuracy of the temporal prediction, most of the reconstructed frame is a simple copy of the prediction, without any transmission. Only some areas difficult to predict (the ball and the girl) require to be conveyed through CNet (Fig. 6.9e). Finally, the contributions of both CNet and the Skip mode are summed to obtain the final reconstructed frame (Fig. 6.9f). As expected, CNet rate distribution (Fig. To improve the performance of the proposed coding scheme, especially at low rate, the conditional coding mechanism is implemented for the MNet in the next section.

Conditional coding for the MNet

Principles

One of the important requirements in the design of the conditional coding mechanism was its flexibility. Namely, being able to use any decoder-side information to lower the rate required to transmit any quantity from the encoder to the decoder. Here, we propose to implement conditional coding for MNet, with the intuition that the reference frames are capable to infer most of the motion information at the decoder-side. Indeed, having one past and one future reference allows to interpolate the motion information and thus to compute relevant optical flows.

Training and rate-distortion results

The complete system, featuring conditional coding for CNet and MNet alongside the Skip mode is trained end-to-end. This yields the rate-distortion results presented in Fig. 6.12a. Adding the conditional coding mechanism to the MNet significantly enhances the compression performance at lower rate, i.e. lower than 2 Mbit/s. Thanks to conditional coding, MNet is able to compute relevant motion information at very low rate: even when MNet rate is around 10 kbit/s, it still significantly increases the rate-distortion results (see Fig. 6.12b). Comparatively, systems without conditional coding for MNet are not able to improve their performance at low-rate, due to the rate constraint being too tight for MNet.

Additionally, it seems that adding conditional coding to MNet, leads its rate to exhibit a more regular behaviour (Fig. 6.12b). This may hint that conditional coding makes the convergence of the system easier, which results in a more regular behaviour during the training stage.

However, conditional coding for MNet slightly decreases the performance at higher rate.

In theory, this should not happen: if conditional coding becomes counter-productive, the training process should learn to disable the conditioning transform by always computing null conditioning latent variable. It is not the case in practice, leading to degraded compression performance at higher rate. Yet, we believe that conditional coding is a relevant technique to improve low-rate performance, while the higher-rate performance issue can be alleviated through a more suited or longer training policy.

Visualisations

In order to illustrate the interest of conditional coding for MNet, a low-rate system (circled in Fig. 6.12a) compresses a video sequence. To understand the behaviour of MNet conditional coding, its output is computed while setting either the analysis or the last column (Fig. 6.13c and 6.13f) shows these quantities when synthesized from the analysis latent variable. The conditioning-only outputs are almost identical to the normal one, while the analysis-only outputs are virtually empty. This highlights that most of the relevant information is contained in xp and xf and at the decoder-side without any transmission. Since no information is conveyed, the MNet rate is extremely low, see Fig.

6.

12b. Yet, the motion information as well as α and β remain accurate enough, allowing to improve the rate-distortion cost even at low rate.

Comprehensive evaluation of the final system 6.5.1 Different coding configurations

Throughout this chapter, the proposed video coding scheme has undergone successive enhancements, improving the compression performance and leading to the final proposed system. It features two conditional coders, MNet and CNet, and an additional Skip mode.

While most design choices have been evaluated on a Random Access configuration (Fig.

2.5a

), the training stage of the systems prepares them to code I, P and B-frames. In order to thoroughly examine the performance of our learned coding scheme, it is evaluated against HEVC (HM 16.22) under three established coding configurations [START_REF] Bossen | VTM common test conditions and software reference configurations for sdr video[END_REF]:

• All Intra (AI): only I-frames are used;

• Low-delay P (LDP): P-frames only, except for the first frame;

• Random Access (RA): usual random access configuration with a gop size of 32 and an intra period of 32.

Figure 6.14 presents the rate-distortion results of our learned coding scheme, compared to those of the HM. For all evaluated coding configurations and across the entire rate range, the learned coding scheme achieves performance on par with the best implementation of HEVC, a modern video coder. These are compelling results, proving the flexibility and the relevance of the proposed coder. It is interesting to remark that due to the approximately symmetrical architecture of the coder, the encoding and decoding speed are comparable.

For the 720p video sequences of the CLIC dataset, both the encoding and decoding run at a speed of a few frames per second. Access configuration with a GOP size of 2. While it prepares the system for the processing of I, P and B-frames, it also specializes the coder for this peculiar coding configuration.

Performing a training closer to the desired coding configuration would allow to improve the compression performance. However, this causes computational issues, as the training time is proportional to the GOP sizes.

Despite the overall encouraging performance of the learned coder, these results highlights some limitations of our coding scheme. On one hand, it significantly outperforms traditional methods for I-frame coding. One the other hand, when the coding configuration is also composed of P or B-frames, it simply achieves results on par with traditional methods. This means that the learned coder has a significant advantage on I-frame coding, but lose some of this benefit on P and B-frame coding. It might be due to weaknesses in the motion estimation and transmission process.

MNet limitations

This section aims to evaluate the ability of MNet to estimate and convey relevant motion information, in order to discover its potential limitations. To this effect, a synthetic video sequence is introduced and shown in Fig. A poorly estimated motion leads to an irrelevant temporal prediction (Fig. 6.16a). The effect of a not accurate prediction is two-fold.

• Skip mode can not be used since the direct copy of a degraded prediction does not result in acceptable quality. This is illustrated by the coding mode α in Fig. 6.16b, and by the Skip mode contribution in Fig. 6.16c. Consequently most of the frame is transmitted through CNet (see Fig. 6.16d), leading to an increased rate.

• The conditional coder CNet can not rely too much on the conditioning transform, as the conditioning input (i.e. the prediction) is not accurate enough. As such, most of CNet output comes from the analysis latent variable, that requires an actual transmission (see Fig. 6.16e), further increasing the rate. This can be thought of as intra coding: no relevant information is available in the prediction, resulting in the complete transmission of the image.

Finally, Figure 6.16f presents the spatial distribution of the rate. The right side of the image (i.e. the areas with bigger motion) appears brighter, highlighting the relation between an inaccurate temporal prediction and an increased rate.

Conclusion

In this chapter, we have enhanced the initial coding scheme composed of a conditional coder CNet, through the introduction of several motion-based components. An additional autoencoder MNet is added to estimate and convey motion information, allowing to achieve a relevant temporal prediction. In order to both foster the convergence and increase the performance, an additional coding mode called Skip mode is proposed. Finally, compression efficiency is improved by implementing MNet with the conditional coding architecture. Table 6.2 summarises the rate spared through the introduction of these different components. The coding scheme featuring motion information and Skip mode through MNet with the conditional coding architecture achieves a BD-rate of -33.9 % compared to the initial motionless system. That is, it obtains equivalent quality requires 33.9 % less rate. Furthermore, the proposed learned video coder achieves a BD-rate of +3.1 % against the HM i.e. the best implementation of HEVC available. In the end, we have designed a learned video coder, competitive with HEVC under different coding configurations. This demonstrates the relevance of the design choices made for our learned coder.

Despite these compelling results, some elements of the coding scheme could be improved. Based on the performance gap between all intra coding and the other coding configurations, we believe that MNet is the main weakness of the coding scheme. Obtain- ing better motion information, by using more suited architecture inspired from the motion estimation literature, would likely improve the overall coding performance.

Skip mode was initially introduced for performance purpose within a motionless coding scheme in a conference paper presented at the IEEE International Workshop on Machine Learning for Signal Processing (MLSP) 2020 [START_REF] Ladune | Modenet: mode selection network for learned video coding[END_REF]. A following paper, further justified skip mode to allow for an easier learning of motion information. This paper was presented at the IEEE International Workshop on Multimedia Signal Processing (MMSP) 2020 [START_REF] Ladune | Optical flow and mode selection for learning-based video coding[END_REF].

Finally, the implementation of MNet with conditional coding as well as the flexibility of the overall coder for different coding configuration was presented at the Neural Compression Workshop, International Conference on Learning Representations (ICLR) 2021 [START_REF] Ladune | Conditional coding for flexible learned video compression[END_REF].

rate target (i.e. different λ). There are two different means of varying its coding strategy:

• Tuning the coding structure i.e. the organization of the I, P and B frames. Since the systems are prepared to process I, P and B-frames, they can be arranged in any desired order, enabling the compression of the video using any frames structure;

• Tuning the rate of the compressed video, by choosing one coder-decoder pair among the different systems available.

So far, a learned coder is mono-rate i.e. it is only able to target one specific rate per sequence. Indeed, it always performs the same operations for a given input, leading to the same quantization accuracy and the same data to be transmitted. Therefore, the competition between N different rates requires to store the parameters of N different mono-rate decoders, resulting in a overall coding scheme with N times more parameters.

Rate competition 7.3.1 With mono-rate coders

There is a great diversity of video sequences within the CLIC dataset. Some are almost motionless, some exhibit few textured areas while others feature important motions or many high-frequency contents. This results in a wide variety of rate-distortion costs, presented in Fig. 7.1. The important rate range advocates for rate competition, as a monorate coder can not be optimal for all the sequences. It is possible that the sequences far from the average rate-distortion point of the coder would benefit from being compressed with an other coder (i.e. at an other rate-distortion point) in order to improve their quality (for low-rate sequences) or reduce their rate (for high-rate sequences).

Rate competition is implemented through the coding of all the sequences using all the previously trained coders, shown in Fig. 7.2a. Seven different mono-rate coders are available, leading to seven coding choices for each sequence. The best coding strategy (i.e.

the best rate target out of the seven available) is chosen sequence-wise according to Eq. While rate competition does bring some performance improvement, this benefit is

R 7 R 5 R 3 R 1 R 2 R 4
(multi-rate) M + quantization gains N × 3 × 256 ≪ M × N (mono-rate) (7.7)
Each additional rate constraint adds only 3 × 256 = 756 parameters. In the end, addressing N different rates is achieved with virtually N times less parameters thanks to multi-rate systems.

Training

With multi-rate coders, the training objective is to prepare the coder to compress I, P and B frames, under different N rate constraints {λ 1 , . . . , λ N }. As for mono-rate systems, the optimization is performed according to a rate-distortion criterion and similarly, each training example is a 3-frame videos compressed using an IBP configurations. In order to learn the different quantization gains, each training iteration is composed of the following steps:

1. Randomly draw a rate constraint index i ∈ {1, . . . , N};

2. Process example with the corresponding quantization gains Γ enc f ,i , Γ dec f ,i (and the other parameters common to all rate constraints); 3. Gradient descent to minimize the corresponding rate-distortion cost: D + λ i R. Similarly to mono-rate coders, the encoder gains dedicated to intra frames are more important which translates into an improved quantization accuracy for the intra frames.

As expected, when the rate constraint becomes tighter, the encoder quantization gains decreases leading to a less accurate quantization, see Fig. 7.4b. Moreover, once the training stage is achieved, the resulting coder is able to compress a video sequence at any continuous rate target, which conveniently allows to freely set the rate of a sequence. This is achieved through a geometric averaging of the quantization gains. Let us suppose that one wants to address the continuous rate constraint r ∈ [1, N].

In such case, the encoder and decoder quantization gains are expressed as follows:

Γ f ,r = Γ f ,⌊r⌋ 1-l ⊙ Γ f ,⌈r⌉ l , with l = r -⌊r⌋. ( 7

Rate-distortion results

To evaluate the benefits of multi-rate systems, the CLIC dataset is compressed, with and without rate competition. The results are presented in Fig. 7.5a. The dashed curves indicate the performance of the systems without rate competition. While the blue one is obtained through the training of multiple mono-rate coders, the orange curve result from a single multi-rate system. While the multi-rate system does not cover a rate range as wide as the mono-rate ones, its performance remains competitive around the rate target of 1 Mbit/s. Activating rate competition, the multi-rate system reaches performance competitive with the mono-rate ones, while having a significantly smaller decoder. Finally, Fig. 7.5b presents the quality at the challenge rate target R T , for mono-rate and multi-rate systems with competition. As explained before, rate competition with mono-rate system is not a relevant choice as the size of the additional decoders exceeds the benefit of rate competition (the blue curve on the graph). This is not the case for a multi-rate system, where the additional parameters is negligible, making possible to actually benefit from rate competition. Rate competition using multi-rate system improves the quality obtained at the challenge rate by +0.26 dB.

Participation to the CLIC21 video track 7.4.1 Proposed systems

Our submission to the CLIC challenge was a multi-rate system, with rate competition to achieve better compression performance. By design, this system is able to process I, P and 0.5 0. B-frames. As such, it is possible to organize these types of frame as desired to achieve any coding structure. This is relevant, due to the wide variety of motion exhibited by the CLIC videos. Indeed some videos with important motions do not benefit from being coded using distant reference frames. For such sequences, using a coding structure with a smaller GOP size mitigates the prediction issues and improves the compression performance.

Coding structure competition is performed sequence-wise and the optimal structure is chosen according to Eq. (7.4). Since the encoding and decoding speed of the learned coder remains reasonable (a few frames per second for a 720p video sequence), it is possible to test a great variety of coding configurations and rates. To assess the performance of the proposed system, traditional coders are evaluated. In addition to HEVC implementations (x265 medium and HM), VTM (test model of VVC) is also tested. For fair comparison, the traditional anchors also performs sequence-wise rate and coding structure competition.

Figure 7.6a shows the different rate-distortion results and Fig. 7.6b presents the MS-SSIM obtained by each system at the challenge rate target.

The interest of competition is clearly illustrated in these rate-distortion results. Each additional competition (rate, coding structure) leads to further performance improvement, yielding an overall increase of +0.47 dB in quality. The resulting learned coder, with full competition, achieves performances better than x265 medium but worse than the HM.

In Chapter 6, our learned coder has been shown to be on par with the HM, which is no 0.5 0. longer the case with the challenge results presented here. This can be explained by two main factors.

• Traditional coders (HM, VTM) feature an important number of different coding strategies: different rate constraints, coding configurations, dedicated tools for certain contents. This makes them particularly suited for sequence-wise competition.

• The size of traditional decoder is significantly smaller than the learned one, requiring around 1 MByte of storage (against 100 MBytes for learned decoders). As such, the decoder rate is negligible, leaving R data ≃ 1 Mbit/s for the video sequences (against R data = 0.92 Mbit/s for learned decoders).

Challenge results

Due to its compelling performance, we submitted the VTM alongside the learned coder to the CLIC video track. The objective of the VTM submission is to represent the state-ofthe-art in traditional video coding.

In order to ensure cross-platform arithmetic encoding and decoding, it is not straightforwardly possible to use the hyperprior mechanism to describe the latent variable probability distribution. At that time, the proposed coder had to be slightly degraded by using a fixed probability distribution instead of the hyperprior mechanism. Additionally, decreasing the number of internal convolutional feature maps from 192 to 128 resulted in a better trade-off between the model size and the compression performance. This leads to the following proposals the CLIC video track, summarised in the Table 7 The challenge results gives several insightful information regarding learned video coding. First of all, submissions based on traditional coders significantly outperform learning-based ones. Indeed, the winner of the challenge is the VTM-based approach we submitted, without learned component. Among the end-to-end category, our submission obtains the best performance, even though its performance is altered due to the removal of the hyperprior mechanism. The actual results of our best learned coder are shown in green, even though it was not submitted to the challenge. The design choices of this thesis are thus further justified, since it leads to state-of-the-art a learned video coder.

Conclusion

This chapter has strived to address a more practical video coding situation through the participation of a compression challenge. Due to wide variety of video sequences, we have proposed to resort to coding strategies competition, namely rate and coding structure competition. Furthermore the decoder size constraint has brought us to come up with multi-rate systems in order to benefit from rate competition while avoiding an increase in the decoder size. Thanks to its enhanced flexibility, the proposed coder obtains compelling results, being ranked first among all the learned systems submitted to the challenge. More importantly, this chapter has shown than the learned coder can be configured to achieve different coding strategies. This is an important step towards a more practical use case of video coding, where real-life constraints (latency, intra period, rate) are more prominent. This challenge is a snapshot of the video coding landscape in 2021. As all systems are evaluated using an identical method, results are more comparable than self-reported performance under different test methods, as it is often the case in the literature. While learned coding performance is continuously and quickly improving, traditional coders such as the VTM remains more competitive. So far, traditional coding systems achieve better performance and present more flexibility due to their many coding strategies available.

For instance, the VTM is able to achieve state-of-the-art performance on a wide-range of rate. This remains a challenge for multi-rate learned systems. Through the limited scope of this challenge, the learned multi-rate system meets most of our expectation, as it allows to use rate competition. In a more realistic setup, it is desirable to address a larger range of rate using a multi-rate system. Empirically, the quantization gains conditioned on the rate constraints does not seem to be powerful enough to operate at widely different rate targets. We believe that other elements of our system would have to be conditioned on the rate constraint in order to obtain a satisfactory multi-rate system. In traditional video coding, the quantization gains is one of the many operations (partitioning, prediction mode etc.) conditioned on the rate target.

The handling of the temporal dimension is discussed in Chapter 4, leading to a two-step coder. A temporal prediction of the current frame is first computed through a motion compensation process, based on previously received frames. Then, the prediction is exploited to only convey the unpredicted part of the current frame. In both traditional and learned coders from the literature, this is often achieved through residual coding.

Chapter 5 introduces conditional coding to better leverage the prediction. It is designed as a generic technique to use any decoder-side information available. Conditional coding is implemented based on components from the learned image coding literature and reduces the rate by 30 % compared to residual coding. Unlike most learned approaches from the literature, conditional coding prevents the need for a dedicated intra frame coder, leading to a more factorized system. Motion compensation is used to compute a relevant temporal prediction. As such, the motion between successive video frames has to be accurately estimated and efficiently transmitted. We propose to perform this two tasks through a single autoencoder. To foster the convergence of this additional encoder a coding mode is introduced: the Skip mode which consists in a direct copy of the temporal prediction. Unlike most existing coding schemes, the addition of the Skip mode allows to learn the overall coding scheme through an end-to-end rate-distortion training. Finally, conditional coding is implemented to reduce the motion information rate. The resulting video coder achieves performance competitive with the best implementation of HEVC under various coding configurations and rate constraints.

Finally, the proposed coder is assessed through a participation to the video track of the CLIC 2021 challenge. The coder flexibility is demonstrated by performing sequencewise competition of coding structures and rate targets, in order to improve the average performance on the entire test set. Due to a specific challenge rule on the decoder size, a multi-rate system is introduced. It allows to target different rates for a video sequence at the cost of few additional parameters. The resulting coder was submitted to the challenge and obtained the best score among the end-to-end learned coders.

Additionally, we also proposed a traditional solution for the CLIC 2021, which won the challenge. This shows that in 2021, traditional coders remains more performant than learning-based ones when evaluated in a realistic video coding situation. Nevertheless, our proposed coder is competitive with HEVC, demonstrating the practicability of learned video coding. Regarding the speed of improvements for learning-based coding schemes, there is no doubt that these approaches will soon outperform traditional ones.

B.4 Residual blocks and attention modules

When training a deep neural network, one may encounter vanishing gradient issues. Indeed, it often happens that the gradient of each layer with respect to its input becomes small. This causes the product of the successive gradients to be very close to zero. Consequently the backpropagation algorithm only perform tiny adjustments to the network parameters and the optimization process yields a suboptimal network. Figure B.3a introduces the architecture of a residual block, which has been shown to ease convergence, allowing to obtain more performant systems [START_REF] He | Deep residual learning for image recognition[END_REF]. This is due to the fact that a residual block learns a residual mapping:

ResBlock(x) = x + f (x). (B.2)
Consequently the derivative of the residual block with respect to its input is:

∂ResBlock(x) ∂x = 1 + ∂ f (x) ∂x . (B.3)
As a result, even if the term

∂ f (x)
∂x becomes too small, the overall residual block derivative is close to 1, avoiding the vanishing gradient issue.

138

The attention mechanism presented in Fig. B.3b allows the system to focus on the more relevant extracted features. The middle branch, called the trunk, is responsible for extracting features from the input, as an usual convolutional networks would do. Yet, the bottom branch computes attention maps based on the inputs, with values in [0, 1] thanks to the sigmoid function. These attention maps are applied through a pixel-wise multiplication of the trunk feature maps, allowing to outline the more important features.

Finally, a residual connexion is added to ease the training of the overall architecture. Abstract: The ever-growing amount of images and videos conveyed over the Internet has a substantial impact on climate change. This impact can be mitigated through the design of better compression algorithms, which aims to reduce the size of videos while maintaining an acceptable quality for the user. Since the 90s, video coding standards have been devised to reduce the video size through a succession of linear operations. Video coding standards are incrementally designed, leading to a separate optimization of the different operations.

Recently, neural networks have emerged as a relevant solution for a wide variety of issues, due to their ability to learn non-linear functions through an end-to-end optimization process. While learned approaches already achieve state-of-the-art performance for still image compression, video coding remains a more challenging task. This thesis proposes to design a learned video coder, to leverage the promising abilities of neural networks. Due to the novelty of this field, the design of the proposed learned coding scheme starts from a blank page. The different elements of the coder are thoroughly considered. In the end, it is shown that the proposed learned coder is able to achieve performance equivalent to modern video coding standards in a realistic video coding setup.

  montre l'utilisation de techniques de compression sans perte (prédiction P, transformée T et codage entropique AE et AD), couplée avec une quantification Q. Ces principes constituent la base des algorithmes de compression. Ils sont repris aussi bien par les standards de compression conventionnels, présentés dans le Chapitre 2, que par les schémas de compression basés apprentissage développés dans le reste de ce manuscrit.

FIGURE 1 :

 1 FIGURE 1 : Compression avec pertes d'un message x avec un modèle de probabilité p.

FIGURE 2 :

 2 FIGURE 2 : Configuration de codage Random Access.

FIGURE 3 :

 3 FIGURE 3 : Architecture d'un autoencodeur.

  xi

FIGURE 4 :FIGURE 5 :

 45 FIGURE 4 : Schéma de compression d'image basé sur un autoencodeur. Q dénote la quantification, AE et AD le codage arithmétique avec p le modèle de probabilité.

6 .

 6 Supposons que x t soit une image à coder et que deux images de références xp , xf soient disponibles. Tout d'abord, il est nécessaire d'estimer le mouvement entre x t et les références puis de le transmettre au décodeur. Ensuite, la compensation de mouvement calcule la prédiction temporelle xt . Finalement, seule la partie non-prédictible est transmise pour obtenir l'image reconstruite xt .

FIGURE 6 :

 6 FIGURE 6 : Compression d'une image inter.

FIGURE 7 :Chapitre 5 :

 75 FIGURE 7 : Configuration de codage pour l'entraînement.

FIGURE 8 :xFIGURE 9 :Chapitre 6 :

 896 FIGURE 8 : L'architecture du codage conditionnel.

FIGURE 10 :FIGURE 11 :FIGURE 12 :Chapitre 7 :

 1011127 FIGURE 10 : Schéma de codage vidéo présentant une compensation de mouvement.

  ont montré que cette idée semble prometteuse. Enfin, les codeurs présentés dans cette thèse ont été conçu avec peu de contraintes de complexité et comportent à ce titre plusieurs dizaines de millions de paramètres. Sur du matériel dédié, il est tout de même possible d'encoder et de décoder plusieurs images xxiii 720p par seconde. Néanmoins, les architectures proposées sont symétriques et requièrent donc autant de puissance de calcul pour l'encodeur que pour le décodeur. En pratique, le décodeur est souvent embarqué dans de nombreux appareils à faible consommation d'énergie (téléphone, set-top boxes). Mettre en place de nouvelles architectures asymétriques serait un pas important vers une utilisation pratique des schémas de codage basés apprentissage. xxiv

  Figure 1.1: Arithmetic coding of 3 successive messages into a single number c. Each message has 4 possible values: Ω s = s(0) , s(1) , s(2) , s(3) , whose respective probability are

Figure 1 . 2 :

 12 Figure 1.2: Block diagram for the lossless coding of a message x with a probability model p. The function P corresponds to prediction and function T corresponds to transform. AE and AD stand for arithmetic encoding and decoding.

Figure 1 . 3 :

 13 Figure 1.3: A uniform scalar quantizer Q with a quantization step ∆ = 1.less relevant dimensions of the signal. Figure1.4 presents a lossy coding scheme using both a quantization step and a non-invertible transform step. Since the transform is no longer constrained to be bijective, T -1 is not guaranteed to exist. As such, the encoder and decoder transforms are respectively denoted T a and T s .

Figure 1 . 4 :

 14 Figure 1.4: Lossy coding of a message x with a probability model p.

  Figure 1.5: Influence of two different noise repartitions evaluated with two quality metrics: PSNR and MS-SSIM dB (higher means better). Example image is ad249bba099568403dc6b97bc37f8d74 from the test set of the Challenge on Learned Image Compression 2020 [16].

  Figure 2.1 illustrates the partitioning of an I-frame: some large blocks gather low-frequency areas (background) while smaller ones isolate the edges of the players. Each block is processed with a different prediction and transform, yet all blocks follow a coding pipeline similar to the one introduced in Chapter 1.

Figure 2

 2 Figure 2.1: Partitioning of an intra frame in HEVC. Reproduced from [35].

Figure 2 .

 2 Figure 2.2 presents the coding pipeline for a block x. It consists of a prediction step (P) to reduce statistical dependencies between the current block and previously coded

Figure 2

 2 Figure 2.2: Block diagram of the coding of a block x using the reference pixels x dec to compute the prediction P. T is an invertible linear transform, Q a quantization operation. The arithmetic encoding and decoding (AE and AD) rely on a probability model p.

  (a) P lanar and DC modes (b) Angular Modes

Figure 2 . 3 :

 23 Figure 2.3: The 35 intra prediction modes of HEVC, illustrated on the Y channel of a 8 × 8 block. Reproduced from [35].

  2 otherwise. Applying the DCT-II successively on the rows and columns of a block gives an equivalent two-dimensional transform, whose basis are shown in Fig. 2.4 for N = 8.

Figure 2 . 4 :

 24 Figure 2.4: The 64 two-dimensional frequential basis of the DCT-II for a 8 × 8 signal.

  (a) Random access coding configuration. The GOP size is 8 and the intra period is 16. (b) Low-delay P coding configuration.

Figure 2 . 5 :

 25 Figure 2.5: Random access and Low-delay P coding configuration.

Figure 3 .

 3 Figure 3.1: The LeakyReLU and Sigmoid non-linearity functions.

  scribed in Eq. (3.2) is called a fully connected layer. Similarly, a neural network made of fully connected layers is a Fully Connected Network (FCN). It is the most generalist network type and can theoretically learn to perform any function, with sufficent number of parameters and layers. However, the FCN universality suffers from an increased complexity. Let us consider the case where the network input is a N × N greyscale image. Suppose that the first layer of the network is a fully connected layer without dimensional reduction i.e. represented by a matrix W ∈ R N 2 ×N 2 . Then, each row of W contains N 2 parameters, used only once to weight each input pixels in order to compute a single output pixel. This results in a prohibitive number of parameters e.g. more than 4 billion parameters for a single layer applied on a 256 × 256 input image.

Fig. 3 .

 3 Fig. 3.2 presents different feature maps extracted from an input image.

Figure 3 . 2 :

 32 Figure 3.2: Different feature maps extracted from an input image. Red and blue denote respectively positive and negative values.

  (a) A vanilla autoencoder. (b) Autoencoder-based image coding scheme. Q denotes a uniform scalar quantization function, AE and AD stand for arithmetic encoding and decoding with p the latent probability model.

Figure 3 . 3 :

 33 Figure 3.3: The autoencoder architecture.

  1.3). Using it as such would prevent the gradient propagation to the other layers, as the chain rule equation (3.6) would feature multiplications by zero. In the literature, three main methods are proposed to circumvent the quantization differentiability issue during the training:

3. 4 .Figure 3 . 4 :

 434 Figure 3.4: Distribution of the relaxed (p ỹ) and quantized (p ŷ) latent variables.

Figure 3 . 5 :

 35 Figure 3.5: Architecture of a typical learning-based image coder. Conv F × 5 × 5 denotes a convolutional layer with F output feature maps and a 5 × 5 kernel. Q is a scalar uniform quantizer, AE and AD respectively denote arithmetic encoding and decoding with a probability model p.

  -based coding schemes have to undergo a training stage. As such, the rate-distortion results achieved by a given architecture vary significantly according to the training parameters. The appendix B.1 provides additional information regarding the training parameters. Unless specified otherwise, all the results presented in this manuscrit result from a personal training by the author in order to enable consistent results. The learning-based coding scheme presented in Fig. 3.5 is trained end-to-end, to minimize the rate-distortion cost presented in Eq. (3.12) with the Mean Squared Error (MSE) as the distortion metric. In order obtain rate-distortion curves with N = 4 points, N individual systems are learnt under N different rate constraints λ. The quality of the reconstructed image is measured through the PSNR, and the rate is expressed in bits per pixel (bpp), computed as: R b p p = -images log 2 p( ŷ) Number of pixels . (3.18)The learned image coder is evaluated on the CLIC 2020 validation set, against existing image/video coding standards. More details on the evaluation and the anchors are available in the appendix B.2.

Figure 3 .

 3 Figure 3.6 presents the rate-distortion performance of the learning-based coding scheme with F = 192 internal convolution features. The interest of non linear transform is illustrated by training the exact same system without (I)GDN layers. The Bjøntegaard rate delta (BD-rate) [54] accurately compares different systems. It measures the variation of rate required to achieve equivalent quality between two codecs. Compared to a linear autoencoder, the non-linear behaviour brought by the GDN allows to have a BD-rate of -40 %, i.e. the rate can be decreased by 40 % while maintaining the same quality (same PSNR). This proves the relevance of using non-linear transforms. Although the non-linear

Figure 3 . 6 :

 36 Figure 3.6: Rate-distortion comparison between a GDN-based and a linear system.

Figure 3 .

 3 Figure 3.7 presents the same latent feature map for two different input images. It is clear that the statistics (mean value and variance) of the latent variable vary both according to the spatial dimension and the input image.

Figure 3 . 7 :

 37 Figure 3.7: The 116 th feature map for two different pictures.

Figure 3 . 8 :

 38 Figure 3.8: Image coding scheme with a hyperprior. The cumulative-based hyperprior probability model is denoted as p z .

Figure 3 . 9 :

 39 Figure 3.9: Performance of the hyperprior-based coding scheme.

Figure 3 . 10 :

 310 Figure 3.10: Visualisations of the hyperprior.

Figure 3 . 11 :

 311 Figure 3.11: Image coding scheme featuring both hyperprior and ARM.

4 .Figure 3 . 12 :

 4312 Figure 3.12: Performance brought by adding an ARM to an hyperprior-based system.

Figure 3 . 13 :

 313 Figure 3.13: Baseline architecture vs. advanced one (see appendix B.5).

Figure 3 . 14 :

 314 Figure 3.14: Signalling process of a centered latent variable ŷ.

Figure 3 . 15 :

 315 Figure 3.15: Laplace distribution vs. binary probability model (BPM).

Figure 3 .

 3 Figure 3.16 presents a visual example of a learned image coder. Detailed visual comparisons for different images and rate constraints are available in Appendix B.6. The original image, shown in Fig. 3.16a is fed to the coder analysis transform, which computes

3. 5

 5 . Visualisations (a) Original image x. (b) Reconstructed image x. PSNR = 33.47 dB, Rate = 0.050 bpp. (c) Image compressed by the VTM. PSNR = 34.08 dB, Rate = 0.053 bpp. (d) Spatial distribution of the rate.

Figure 3 . 16 :

 316 Figure 3.16: Visual comparison between a learned coder and the VTM. The test image is 400984b87394ada6d9627ed918908986 from the CLIC20 dataset. 53

  of learning-based systems: the training stage. Ideally, one would like to compare systems by measuring their performance and execution time once the training stage has properly converged. In practice, some systems still improves after weeks of training, resulting in impractical convergence time.

  inputs, i.e. F three-dimensional feature maps of size T × H × W, with T the number of frames. They use several F × k T × k H × k W kernels to compute the output feature maps. The additional dimension of their kernel, makes 3D convolutions able to process the supplementary temporal dimension.

Figure 4 .

 4 Figure 4.2: Random Access coding configuration, with a GOP size of 8.

Figure 4 . 3 :

 43 Figure 4.3: The training coding configuration.

  dimension of H × W pixels. During training, the frames are crops of size 256 × 256. The appendix C details the composition of the training dataset.

Figure 5 .

 5 Figure 5.1: Autoencoder-based residual coding scheme. The latent variable probability model is omitted, Q stands for quantization, AE and AD for arithmetic encoding and decoding.

Figure 5 . 2 :

 52 Figure 5.2: Different baselines comparison. MS-SSIM dB = -10 log 10 (1 -MS-SSIM) is the quality (higher is better). RA stands for Random Access, AI for All Intra.

Figure 5 . 4 :

 54 Figure 5.4: Conditional coding performance.

  (a) Original frame to code x t . (b) The temporal prediction xt = 1 2 xp + xf . (c) Reconstructed frame xt = g s (y, y c ). (d) Conditioning transform only: g s (0, y c ).(e) Analysis transform only: g s (y, 0). (f) Spatial distribution of the rate.

Figure 5 . 5 :

 55 Figure 5.5: Low-rate conditional coding illustrated on a B-frame from the CLIC sequence Sports_1080P-6710. This frame costs 2 481 bits and has a MS-SSIM of 13.35 dB.

  (a) Original frame to code x t (Identical to Fig. 5.5a). (b) The temporal prediction xt (Identical to Fig. 5.5b). (c) Reconstructed frame xt = g s (y, y c ). (d) Conditioning transform only: g s (0, y c ).(e) Analysis transform only: g s (y, 0). (f) Spatial distribution of the rate.

Figure 5 . 6 :

 56 Figure 5.6: High-rate conditional coding illustrated on a B-frame from the CLIC sequence Sports_1080P-6710. This frame costs 33 712 bits and has a MS-SSIM of 21.48 dB.

Figure 5 . 7 :

 57 Figure 5.7: Latent-domain residual.

Figure 5 . 8 :

 58 Figure 5.8: Latent residual coding performance.

Fig. 5 .Figure 5 . 9 :

 559 Fig. 5.9 shows the All Intra coding results of the different architectures. While the image-domain residual coder struggles to properly code images, both latent-domain residual coding and coding coding achieve All Intra performance on par with the pure image coder. This demonstrates that conditional coding avoids the need of a dedicated intra coder.Similar conclusions are drawn by Djelouah et al.[START_REF] Djelouah | Neural inter-frame compression for video coding[END_REF], who states that performing the combination of encoder and decoder information in the latent domain prevents the need of designing a dedicated intra coder.

Figure 5 . 10 :

 510 Figure 5.10: Feature-wise quantization gains.

  Encoder quantization gains Γ enc f for f a I, P or B-frames.

Figure 5 . 11 :

 511 Figure 5.11: Performance and analysis of the quantization gains.

  a prediction of frames in order to only convey the unpredicted content. Consequently, achieving a more accurate temporal prediction allows to reduce the amount of unpredicted content, reducing the overall rate. The successive video frames usually present similar content with small displacements among frames. As such, it is possible to compute a relevant prediction by performing a motion compensation,

Figure 6 .

 6 Figure 6.3 presents two means to obtain relevant motion information at the decoder side. The first one is a two-step method, with a separate motion estimation network

  (a) Two-step MNet, with MEst a dedicated motion estimation network. (b) Single step MNet.

Figure 6 . 3 :

 63 Figure 6.3: Two main trends for the MNet architecture. MEnc and MDec denotes a learned encoder-decoder pair.

  (a) Past reference xp . (b) Current frame x t . (c) Future reference xf . (d) Mode selection α. blue: skip, red: CNet.

Figure 6 . 4 :

 64 Figure 6.4: Coding mode selection α for a B-frame from the CLIC sequence TelevisionClip_1080P-4c94.

Figure 6 . 5 presents

 65 the block diagram of a coding scheme featuring the Skip mode. CNet implements the conditional coding mechanism presented in Section 5.3. On the other hand, the MNet implements the usual analysis-synthesis architecture, shown in Fig. 6.6. The MNet transforms do not share the same parameters than those of the CNet and their architecture can be slightly different. To highlight this fact, the MNet transforms are denoted with a superscript m.

Figure 6 . 5 :

 65 Figure 6.5: Block diagram of a coding scheme featuring the Skip mode.

Figure 6 . 6 :

 66 Figure 6.6: The MNet architecture.

6 .

 6 7b) and two reference frames xp and xf (Fig 6.7a and 6.7c). MNet uses these inputs to compute a latent variable, conveyed to the decoder, where the MNet synthesis computes different outputs: two optical flows v p , v f (Fig. 6.7d and 6.7f), the bidirectional prediction weighting β (Fig. 6.7g) and the coding mode selection α (Fig. 6.7h).(a) Past reference xp . (b) Original frame to code x t . (c) Future reference xf . (d) Optical flow v p : motion from x t to xp . (e) Optical flow representation. Motion is expressed in pixel. (f) Optical flow v f : motion from x t to xf . (g) Bidirectional prediction weighting β.blue: warp from xf , red: warp from xp . (h) Coding mode selection α. blue: skip, red: CNet.

Figure 6 . 7 :

 67 Figure 6.7: The inputs and outputs of MNet.

  (a) Error before compensation x t -xp . (b) Error before compensation x t -xf . (c) Prediction error x t -xt . (d) Intermediate motion comp. w( xp ; v p ). PSNR = 28.39 dB. (e) Intermediate motion comp. w( xf ; v f ). PSNR = 27.91 dB. (f) Prediction xt , computed using Eq. (6.1). PSNR = 30.25 dB.

Figure 6 . 8 :

 68 Figure 6.8: Motion compensation through a bidirectional prediction process.

Figure 6 . 10 :

 610 Figure 6.10: Rate-distortion results for coding schemes featuring motion information.

Figure 6 .

 6 Figure 6.11 presents the new MNet architecture, featuring the additional conditioning transform g m c . As for CNet, the conditioning transform g m c replicates the architecture of the analysis transform g m a with different number of input and output features. MNet conditioning transform aims to extract, from the two reference frames, relevant information regarding the motion, as well as α and β. Consequently, the MNet analysis g m a only transmits the information missing at the decoder, decreasing the rate. The addition of conditional coding for MNet increases the overall number of parameters from 36 million to 42 million.

Figure 6 . 11 :

 611 Figure 6.11: The conditional coding MNet architecture.

  rate [Mbit/s] MNet rate [Mbit/s] MNet rate vs. overall rate. Motion & Skip & CC Motion & Skip (b) MNet rate for different total rates.

Figure 6 . 12 :

 612 Figure 6.12: Conditional coding (CC) for MNet, rate-distortion results.

  Motion of x t relatively to xp . (b) Optical flow v p . (c) Legend for the optical flow representation. Motion is expressed in pixel.

Figure 6 . 15 :

 615 Figure 6.15: Example of an inaccurate optical flow.

Figure 6 .

 6 Figure 6.15b presents the optical flow obtained at the decoder i.e. after estimation and transmission through the MNet. For the 8-pixel motions, the resulting optical flow is accurate, regardless the motion direction. When motion amplitudes become more important, e.g. a 16-pixel motion, the optical flow is properly estimated for horizontal motions, while vertical ones are no longer captured by the MNet. The system is not able to compute any optical flow for motion of 24 pixels or more. Finally, the edges of the optical flows are less accurate when the motion magnitude becomes more important. This weakness regarding the estimation and transmission of important optical may be caused by several factors. First, important motions are not often found in video sequences and therefore in the training set. The lack of relevant example prevents the system from learning to identify important motion. Then, the proposed architecture for MNet is mostly derived from an autoencoder. Integrating components from the optical flow estimation literature such as cost volume elements could lead to better results.

Figure 6 . 16 :

 616 Figure 6.16: Consequences of an inaccurate prediction. This frame costs 80 954 bits, for a quality of MS-SSIM dB = 10.65 dB. The PSNR is equal to 20.80 dB

( 7 . 4 )Figure 7

 747 Figure 7.1: Sequence-wise rate-distortion for a low-rate learned encoder-decoder pair.

  dB @ Rate target R T [dB] Rate competition with N rate constraints R dec = N × 0.08 Mbit/s (b) Influence of the model size on the rate target.

Figure 7 .

 7 Figure 7.2: Sequence-wise rate competition with mono-rate coders.

Figure 7 .

 7 Figure 7.4 presents the encoder quantization gains obtained for two rate constraints.

  Rate constraint λ 0 (higher rate).

  Rate constraint λ 1 (lower rate).

Figure 7 . 4 :

 74 Figure 7.4: Encoder quantization gains for two rate constraints.

Figure 7 . 5 :

 75 Figure 7.5: Sequence-wise competition for a multi-rate system. Cpt stands for competition.

Figure 7 . 6 :

 76 Figure 7.6: Performance of different systems with a focus on the quality at the challenge target rate. GOP stands for coding structure competition.

Figure 7 . 7 :

 77 Figure 7.7: Challenge test set leaderboard. The two named systems are our submissions.

  (a) Image IMG_0426 from the user-generated dataset. (b) Image nomao-saeki-33553 from the professional dataset.

Figure B. 1 :

 1 Figure B.1: Examples from the professional and user-generated datasets.

  137

  Figure B.2: Architecture of the Auto-Regressive Module and the hyperprior/ARM fusion component. MConv F × 5 × 5 denotes a masked convolutional layer with F output feature maps and kernels of size 5 × 5. The kernel of masked convolutions have null parameters for anti-causal pixels [79] i.e. they only process past (already received) values.

  (a) Residual block. (b) Residual attention module.

Figure B. 3 :

 3 Figure B.3: Residual block and a residual attention module.

[ 44 ] 3 R 4 Figure B. 7 :

 44347 Figure B.4: The analysis and synthesis transforms g a and g s . The number of internal features F is set to 192, the number of latent features F y is set to 256.

3 R 4 Figure B. 8 :

 348 Figure B.8: Visual comparison for the image 08052112d0151f7c9ac4879f838d5a0c.

3 R 4 Figure B. 9 :

 349 Figure B.9: Visual comparison for the image 400984b87394ada6d9627ed918908986.

3 R 4 Figure B. 10 :

 3410 Figure B.10: Visual comparison for the image 72e19343f46a447bea2206c368a9692a.

Figure D. 1 :

 1 Figure D.1: The analysis and synthesis transforms g a and g s . The number of internal features F is set to 192, the number of latent features F y is set to 256.

Figure D. 2 :Figure D. 3 :

 23 Figure D.2: The hyperprior analysis and synthesis transforms h a and h s . The number of internal features F is set to 192, the number of hyperprior features F z is set to 64.

Figure E. 1 :

 1 Figure E.1: MNet analysis and synthesis transforms g a and g s . The number of internal features F is set to 192, the number of latent features F y is set to 64.

Figure E. 2 :

 2 Figure E.2: MNet hyperprior analysis and synthesis transforms h a and h s . The number of internal features F is set to 192, the number of hyperprior features F z is set to 16.

Figure E. 3 :E. 3

 33 Figure E.3: Additional examples of optical flows from the sequence Sports_1080P-08e1, extracted from the CLIC21 validation set.

  (a) Frame to predict x t . (b) Bidirectional prediction weighting β. (c) Intermediate motion comp. w( xp ; v p ). (d) Intermediate motion comp. w( xf ; v f ). (e) Weighted motion comp. β ⊙ w( xp ; v p ). (f) Weighted motion comp. 1 -β ⊙ w( xf ; v f ). (g) Prediction xt , computed using Eq. (6.1). (h) Prediction error x t -xt .

Figure E. 4 :

 4 Figure E.4: Example of β handling a disocclusion.

  (a) Frame to predict x t . (b) Bidirectional prediction weighting β. (c) Intermediate motion comp. w( xp ; v p ). (d) Intermediate motion comp. w( xf ; v f ). (e) Weighted motion comp. β ⊙ w( xp ; v p ). (f) Weighted motion comp. 1 -β ⊙ w( xf ; v f ). (g) Prediction xt , computed using Eq. (6.1). (h) Prediction error x t -xt .

Figure E. 5 :

 5 Figure E.5: Additional examples of β performing dissoclusion.

  ). Intérêt de la compétition. E2E désigne les systèmes basés apprentissage.Il est intéressant de constater que les résultats présentés dans le Chapitre 6 (Fig.12) montrent que le codeur basé apprentissage est a égalité avec le HM, ce qui n'est plus le cas ici. La différence majeure est qu'à présent, la compétition de stratégies de codage xxi est intégrée dans les résultats. La grande variété de choix de codage disponibles dans le HM et le VTM permet d'effectuer une compétition accrue, qui augmente les performances de manière importantes. Le codeur basé apprentissage ne bénéficie pas autant de la compétition, car il présente un nombre plus faible de stratégies de codage disponibles. Enfin, la taille d'un décodeur conventionnel est environ 100 fois plus faible que celle d'un décodeur basé apprentissage, ce qui laisse d'autant plus de place pour la représentation compressée des vidéos, permettant une meilleure qualité. Après quelques ajustements pour permettre un encodage et décodage cross-platform, le codeur vidéo basé apprentissage a été proposé comme candidat au challenge. Étant donné les très bonnes performances du VTM, ce dernier a également été soumis comme candidat par nos soins. L'intégralité des résultats du challenge sont présentés Fig. 14. Onze codeurs ont été soumis au challenge, dont 5 systèmes entièrement basés apprentissage. Les autres sont basés sur des codeurs vidéos conventionnels avec éventuellement des modules de post-traitement basés apprentissage. Parmi la catégorie des codeurs basés entièrement apprentissage, le codeur E2E_T_OL conçu dans le cadre de cette thèse obtient les meilleures performances. Cela légitime les différents choix techniques motivés dans cette thèse.

	E2E 16.88 Qualité au débit cible du challenge, ensemble de validation x265 E2E E2E HM VTM 16.92 18.10 19.13 17.14 17.35 Ancrage Pas de compétition Compétition débit 15.07 Résultats définitifs du challenge CLIC 2021 18.37 18.81 18.94 19.19 19.30 14.27 14.51 15 16 17 18 19 20 12.83 12.99 Qualité en MS-SSIM dB [dB] 15.48 17.31 Basé codeurs conventionnels Codeur E2E Codeur E2E non soumis Compétition débit et structure E2E non soumis ANC_T_OL FIGURE 14 : Les entrées nommées correspondent à nos soumissions. E2E désigne les 10 12 14 16 18 20 dB [dB] Qualité en MS-SSIM FIGURE 13 : E2E_T_OL systèmes basés apprentissage
	Les résultats du challenge fournissent plusieurs enseignements. Tout d'abord, les
	approches conventionnelles sont actuellement les plus performantes. C'est même notre

soumission ANC_T_OL basée VTM, sans aucun module basé apprentissage, qui remporte xxii le challenge.

TABLE OF CONTENTS

 OF

Introduction 1 Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Structure of the thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

I Context

and State of the Art 5 1 Fundamentals of Video Coding 7

  Overview of the video coding structure . . . . . . . . . . . . . . . . . . 2.2.3 Intra prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Transform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.5 CABAC: Context Adaptive Binary Arithmetic Coding . . . . . . . . . . 2.3 From coding still images to video sequences . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	6.1 6.2 Coding scheme featuring a prediction step . . . . . . . . . . . . . . . . . . . .
	2.2.1 Representing the colour: the RGB and YUV colour spaces . . . . . . . 6.2.1 Bidirectional prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	6.2.2 Motion information at the decoder side . . . . . . . . . . . . . . . . . . 6.3 Skip mode: an additional coding mode . . . . . . . . . . . . . . . . . . . . . . . 6.3.1 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.2 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.3 Behaviour . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3.4 Rate-distortion results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Conditional coding for the MNet . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.1 Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4.2 Training and rate-distortion results . . . . . . . . . . . . . . . . . . . . 6.4.3 Visualisations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Comprehensive evaluation of the final system . . . . . . . . . . . . . . . . . . 6.5.1 Different coding configurations . . . . . . . . . . . . . . . . . . . . . . . 6.5.2 MNet limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 7 Towards Practical Video Coding 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 2.2.2 3.3.1 Autoencoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 CLIC21 video track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.3.2 Training a learned coding scheme . . . . . . . . . . . . . . . . . . . . . 7.2.1 Challenge presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
	3.3.3 Initial architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2.2 Sequence-wise competition . . . . . . . . . . . . . . . . . . . . . . . . . . 106
	3.3.4 Training and rate-distortion results . . . . . . . . . . . . . . . . . . . . 7.3 Rate competition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
	3.4 Advanced learned coding schemes . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 With mono-rate coders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
	3.4.1 Hyperprior to refine the probability model . . . . . . . . . . . . . . . . 7.3.2 Design of a multi-rate coder . . . . . . . . . . . . . . . . . . . . . . . . . 109
	3.4.2 Auto-regressive probability model . . . . . . . . . . . . . . . . . . . . . 7.3.3 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
	3.4.3 Attention modules for better networks . . . . . . . . . . . . . . . . . . . 7.3.4 Rate-distortion results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
	3.4.4 Binary Probability Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Participation to the CLIC21 video track . . . . . . . . . . . . . . . . . . . . . . 111
	3.5 Visualisations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 Proposed systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
	3.6 A word on complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Challenge results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
	3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5
	xxvii

1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 Lossless coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.2.1 Optimal rate and information theory . . . . . . . . . . . . . . . . . . . . 8 1.2.2 Arithmetic coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.2.3 Practical limitations of arithmetic coding . . . . . . . . . . . . . . . . . 10 1.2.4 Predictive coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.2.5 Transform coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3 Lossy compression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3.1 Discarding information . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.2 Distortion metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.3.3 Rate-distortion optimization . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 xxvi 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Hybrid image coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Inter prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3.2 Coding structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Learning to Compress Images 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Basics of neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Building a neural network . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Training a neural network . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Convolutional neural networks . . . . . . . . . . . . . . . . . . . . . . . 3.3 Learning-based lossy image coding . . . . . . . . . . . . . . . . . . . . . . . . .

115 xxix Conclusions and Future Work 117 Thesis

  objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	Future works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Bibliography 121 A List of Publications 131 B Additional Details on Learned Image Coding 135

  Additional visual examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	B.1 Training recipe . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	B.1.1 Training dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	B.1.2 Details on the training stage . . . . . . . . . . . . . . . . . . . . . . . . .
	B.2 Additional information regarding the test stage . . . . . . . . . . . . . . . . .
	B.2.1 Inference dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	B.2.2 Coding standards as anchors . . . . . . . . . . . . . . . . . . . . . . . .
	B.3 Detailed ARM architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	B.4 Residual blocks and attention modules . . . . . . . . . . . . . . . . . . . . . . .
	B.5 Comprehensive architecture of a state-of-the-art image coder . . . . . . . . .
	B.6

C Training dataset for learned video coding 147

	C.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	C.2 Dataset composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D Comprehensive Experimental Details for CNet 149

  The autoencoder architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Distributions of the latent variable. . . . . . . . . . . . . . . . . . . . . . . . . . CNet and Skip mode contributions. This frame costs 15 037 bits, for a quality of MS-SSIM dB = 22.60 dB. The PSNR is equal to 38.46 dB . . . . . . . . . . . 93 6.10 Rate-distortion results for coding schemes featuring motion information. . . 94 6.11 The conditional coding MNet architecture. . . . . . . . . . . . . . . . . . . . . 95 6.12 Conditional coding (CC) for MNet, rate-distortion results. . . . . . . . . . . . 96 6.13 Low-rate MNet conditional coding . . . . . . . . . . . . . . . . . . . . . . . . . 97 6.14 Rate-distortion results for different coding configurations. . . . . . . . . . . . 99 6.15 Example of an inaccurate optical flow. . . . . . . . . . . . . . . . . . . . . . . . 6.16 Consequences of an inaccurate prediction . . . . . . . . . . . . . . . . . . . . . 7.1 CLIC sequence-wise rate-distortion . . . . . . . . . . . . . . . . . . . . . . . . .

	LIST OF FIGURES 6.3 Two main trends for the MNet architecture. . . . . . . . . . . . . . . . . . . . 86 6.4 Coding mode selection α for a B-frame from the CLIC sequence TelevisionClip_1080P-3.3 3.5 Architecture of a typical learning-based image coder. . . . . . . . . . . . . . . 4c94. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
	3.6 Rate-distortion comparison between a GDN-based and a linear system. . . . 6.5 Block diagram of a coding scheme featuring the Skip mode. . . . . . . . . . . 88
	3.7 The 116 th feature map for two different pictures. . . . . . . . . . . . . . . . . . 6.6 The MNet architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
	1 3.8 Image coding scheme with a hyperprior. . . . . . . . . . . . . . . . . . . . . . . Schéma bloc d'une compression avec pertes. . . . . . . . . . . . . . . . . . . . . viii 6.7 The inputs and outputs of MNet. . . . . . . . . . . . . . . . . . . . . . . . . . . 90
	2 3.9 Performance of the hyperprior-based coding scheme. . . . . . . . . . . . . . . Configuration de codage Random Access. . . . . . . . . . . . . . . . . . . . . . x 6.8 Motion compensation through a bidirectional prediction process. . . . . . . . 92
	3 4 5 6 7 8 9 10 Schéma de codage vidéo présentant une compensation de mouvement. . . . xviii Architecture d'un autoencodeur. . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Schéma de compression d'image basé sur un autoencodeur. . . . . . . . . . . xii Performance d'un système de compression d'image basé autoencodeur (AE). xii Compression d'une image inter. . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Configuration de codage pour l'entraînement. . . . . . . . . . . . . . . . . . . . xiv L'architecture du codage conditionnel. . . . . . . . . . . . . . . . . . . . . . . . xvi Performances du codage conditionnel. . . . . . . . . . . . . . . . . . . . . . . . xvii 11 Schéma de codage vidéo avec le Skip mode. . . . . . . . . . . . . . . . . . . . . xix 12 Performances apportée par l'ajout d'information de mouvement. . . . . . . . xx 13 Intérêt de la compétition. E2E désigne les systèmes basés apprentissage. . . xxi 3.10 Visualisations of the hyperprior. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.11 Image coding scheme featuring both hyperprior and ARM. . . . . . . . . . . . 3.12 Performance brought by adding an ARM to an hyperprior-based system. . . 3.13 Rate-distortion results of a more advanced architecture . . . . . . . . . . . . 3.14 Signalling process of a centered latent variable ŷ. . . . . . . . . . . . . . . . . 3.15 Performance of the binary probability model . . . . . . . . . . . . . . . . . . . 3.16 Visual comparison between a learned coder and the VTM. . . . . . . . . . . . 3.17 Average rate-distortion cost as a function of the training time for different architectures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Conceptual processing pipeline of an inter frame. . . . . . . . . . . . . . . . . 6.9 7.2 Sequence-wise rate competition with mono-rate coders. . . . . . . . . . . . . . 4.2 Random Access coding configuration, with a GOP size of 8. . . . . . . . . . . 14 Les entrées nommées correspondent à nos soumissions. E2E désigne les 4.3 The training coding configuration. . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Principles of CNet multi-rate quantization gains. . . . . . . . . . . . . . . . . systèmes basés apprentissage . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii 7.4 Encoder quantization gains for two rate constraints. . . . . . . . . . . . . . .
	D.1 Residual autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D.2 Quantization gains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . E Comprehensive Experimental Details for MNet 5.1 Autoencoder-based residual coding scheme. . . . . . . . . . . . . . . . . . . . . 7.5 Sequence-wise competition for a multi-rate system. Cpt stands for competition.112 153 1.1 Arithmetic coding of 3 successive messages into a single number c. . . . . . 10 5.6 High-rate conditional coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Influence of noise repartitions for PSNR and MS-SSIM. . . . . . . . . . . . . 18 5.5 Low-rate conditional coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Lossy coding block diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 5.4 Conditional coding performance . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 A uniform scalar quantizer Q with a quantization step ∆ = 1. . . . . . . . . . 16 5.3 The conditional coding architecture. . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Lossless coding block diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 5.2 Comparison of the different baselines . . . . . . . . . . . . . . . . . . . . . . . 7.6
	E.1 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1 Partitioning of an intra frame in HEVC. Reproduced from [35]. . . . . . . . . 22 5.7 Latent-domain residual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	E.2 Additional examples of optical flows . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Block diagram of the coding of a block. . . . . . . . . . . . . . . . . . . . . . . . 23 5.8 Latent residual coding performance. . . . . . . . . . . . . . . . . . . . . . . . .
	E.3 Bidirectional prediction weighting and disocclusions . . . . . . . . . . . . . . 2.3 The 35 intra prediction modes of HEVC. . . . . . . . . . . . . . . . . . . . . . . 24 5.9 All Intra coding performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	5.10 Feature-wise quantization gains. . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 The 64 two-dimensional frequential basis of the DCT-II for a 8 × 8 signal. . . 25 2.5 Random access and Low-delay P coding configuration. . . . . . . . . . . . . . 28 5.11 Performance and analysis of the quantization gains. . . . . . . . . . . . . . .
	3.1 The LeakyReLU and Sigmoid non-linearity functions. . . . . . . . . . . . . . 32 6.1 Bi-directional temporal prediction. . . . . . . . . . . . . . . . . . . . . . . . . .
	3.2 Different feature maps extracted from an input image. . . . . . . . . . . . . . 35 6.2 Video coding scheme with an actual motion compensation step. . . . . . . . .
	xxx xxxii	xxxi

Performance of different systems with a focus on the quality at the challenge target rate

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Leaderboard on the challenge test set. . . . . . . . . . . . . . . . . . . . . . . . B.1 Examples from the professional and user-generated datasets. . . . . . . . . . B.2 Architecture of the Auto-Regressive Module and the hyperprior/ARM fusion component. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.3 Residual block and a residual attention module. . . . . . . . . . . . . . . . . . B.4 The analysis and synthesis transforms g a and g s . . . . . . . . . . . . . . . . . The analysis and synthesis transforms g a and g s . . . . . . . . . . . . . . . . . D.2 The hyperprior analysis and synthesis transforms h a and h s . . . . . . . . . . MNet analysis and synthesis transforms g a and g s . . . . . . . . . . . . . . . . E.2 MNet hyperprior analysis and synthesis transforms h a and h s . . . . . . . . . E.3 Additional examples of optical flows from the sequence Sports_1080P-08e1, extracted from the CLIC21 validation set. . . . . . . . . . . . . . . . . . . . . . E.4 Example of β handling a disocclusion. . . . . . . . . . . . . . . . . . . . . . . . E.5 Additional examples of β performing dissoclusion. . . . . . . . . . . . . . . . .

	B.9 Visual comparison for the image 400984b87394ada6d9627ed918908986. . . B.10 Visual comparison for the image 72e19343f46a447bea2206c368a9692a. . . . INTRODUCTION
	B.5 The hyperprior analysis and synthesis transforms h a and h s . . . . . . . . . . B.6 Visual comparison for the image 46c1831600829ae8b30c6b06557424ef. . . . B.7 Visual comparison for the image ad249bba099568403dc6b97bc37f8d74. . . B.8 Visual comparison for the image 08052112d0151f7c9ac4879f838d5a0c. . . . xxxiii D.1 D.3 Encoder quantization gains Γ enc for different rate targets. . . . . . . . . . . . f Context T HE ever-growing quantity of data conveyed over the Internet has a substantial impact on climate change. Even before the Covid-19 outbreak, video data represented around E.1 xxxiv 80 % of the internet traffic [1]

  .7) Arithmetic coding operates exclusively on discrete data and is thus non-differentiable.Since it is a lossless step within the coding pipeline, it is removed during the training stage, without altering the value of the decoded image. As explained in Section 1.2.2, arithmetic coding rate can be estimated by the cross entropy: R = E ŷ∼q ŷ -log 2 p ŷ( ŷ) .

(3.8)

Yet, there is no quantized latent variable ŷ during the training stage due to the additive noise model. Instead, we have the relaxed latent variable ỹ, whose probability density function (PDF) is:

  Average rate-distortion cost (lower is better) as a function of the training time for different architectures. Architectures are detailed in Table3.1.

					Linear AE
	Rate-distortion cost J				Non-linear AE Hyperprior Attention BPM
	0	3	6	9	12	15	18
			Training time (days)
	Figure 3.17:
							.29)
	Figure 3.17 presents loss-training time curves for all the proposed designs, except
	the autoregressive-based ones which has already be discarded due to complexity issue.
	This graph shows that even at equivalent complexity (equivalent training time), more
	advanced approaches (hyperprior, attention, BPM) are consistently better. This proves
	that the successive improvements presented in this Chapter perform better due to their
	intrinsic quality, and not because they feature more parameters and more training time.

Table 3 .

 3 1: BD-rates of the different coding schemes. AE stands for autoencoder, ARM for autoregressive module and BPM for binary probability model.

	Codec	Non-linear HP ARM Attention BPM	BD-rate
	Linear AE Non-linear AE Hyperprior ARM Attention BPM JPEG HEVC VVC		+63.7 % Reference +132.5 % +285.9 % -32.1 % +56.7 % -36.7 % +45.0 % -52.9 % +7.4 % -55.6 % +1.4 % +46.2 % +236.6 % -44.8 % +28.1 % -56.9 % Reference

achieve performance on par with VVC. So far, such convolutional networks still requires high-end computing devices to achieve realistic encoding/decoding time.

Table 4 .

 4 1: Summary of the different learned video coders in the literature. AE stands for autoencoder, Bi-pred for bi-directional prediction.

		Motion	Image	Same intra
		Estimation Bi-pred.	residual	and inter coder
	Habibian, ICCV 19 [15]	3D convolutions
	Djelouah, ICCV 19 [66]	PWCNet	latent residual
	Lu, CVPR 19

Table 5 .

 5 2: BD-rates of the different coding schemes in Random Access configuration.

	Residual Image Latent	Conditional Coding	Q. gains	BD-rate
	HM (All Intra) ⋆ Image Spatial res. Latent res. Cond. Cond., Γ x265 medium † (motion) HM † (motion)	/ / /		+158.2 % +580.3 % +36.7 % +184.0 % Reference +129.0 % -21.2 % +71.9 % -29.3 % +60.8 % -35.9 % +42.9 % -46.0 % +22.8 % -56.3 % Reference
	⋆ : evaluated at QP 36, 41, 46, 51.

† : evaluated at

QP 22,[START_REF] Dupont | COIN: compression with implicit neural representations[END_REF][START_REF] Kleijn | A basis for source coding[END_REF][START_REF] Ahmed | Discrete cosine transform[END_REF]

Table 6 .

 6 2: BD-rates of the different coding schemes in Random Access coding configuration. CC denotes conditional coding.

		MNet Skip mode MNet CC	BD-rate
	No motion Motion Motion & Skip Motion & Skip & Cond. x265 medium † HM †	/ /	Reference +42.9 % -7.6 % +24.4 % -17.5 % +14.7 % -33.9 % +3.1 % -14.5 % +22.8 % -37.2 % Reference

† : evaluated at

QP 22,[START_REF] Dupont | COIN: compression with implicit neural representations[END_REF][START_REF] Kleijn | A basis for source coding[END_REF][START_REF] Ahmed | Discrete cosine transform[END_REF]

  Influence of the decoder size on the rate target.

		19 RD curves, CLIC 2021 valid. set, RA config	T [dB]	18 Rate competition with N rate constraints
	MS-SSIM dB [dB]	15 16 17 18	75	1 Mono-rate no Cpt 1.25 1.5 Mono-rate Cpt Multi-rate no Cpt Multi-rate Cpt	dB @ Rate target R MS-SSIM	14 16 17 15	0 1 2 3 4 5 6 7 8 Mono-rate Multi-rate
				Rate [Mbit/s]			Number rate constraints N
		(a) Performance of rate competition.	(b)

  Challenge results. Lnd stands for learned coder.

	MS-SSIM dB [dB]	15 16 17 18 19 20	75 Rate [Mbit/s] 1 1.25 V T M H M x 2 6 5 No competition 1.5 Rate Rate, GOP RD curves, CLIC 2021 valid. set	MS-SSIM dB @ target rate (dB)	15 16 17 18 19 20	Lnd x265 Lnd Lnd HM VTM Challenge results of different systems 16.92 18.10 19.13 16.88 17.14 17.35 Anchors No competition Rate Rate, GOP
	(a) Different coders with competition.	(b)

Table 7 .

 7 .1 below. 1: CLIC21 leaderboard results validation set. ft. stands for feature maps. Coders from 11 different participants were proposed to the challenge. Among them, 6 are based on traditional video coders, with some additional learning-based enhancements (mostly post-processing filters). The other 5 systems are end-to-end learned coders. To prevent overfitting from the learned systems, a new dataset is used to determine the challenge winner.Figure 7.7 presents the definitive results of the CLIC 2021 video track.

	System	Submission name	MS-SSIM dB Decoder size [dB] [MBytes]
	VTM	ANC_T_OL	19.13	1
	Learned (128 ft., fixed PDF)	E2E_T_OL	15.53	43
	Learned (192 ft., HP)	Not submitted	17.35	108

  Table C.1: Composition of the training dataset. UGC stands for user-generated content and Prof. for professional content.

	Dataset	Nature UGC Prof. Natural Synthetic	Resolution	Average FPS	Number examples
	KoNVid_1k [81] CLIC2021 [16] YouTube-NT [82] YUV_4K [83] Total		540p 720p 1080p 2160p	28 30 25 58	0.9 × 1.4 × 2.5 × 0.4 × 5.2 ×

(a) Original image.

ACKNOWLEDGEMENTS

First and foremost, I would particularly like to thank Pierrick Philippe, my PhD supervisor from Orange. I am extremely grateful for the considerable amount of time he has dedicated Acknowledgements

PART II

Learned Video Coding

T HE presence of the temporal dimension makes video coding a more challenging task than still image coding, due to the supplementary statistical redundancies. On one hand, the well-established traditional video coders remove these redundancies using a temporal prediction performed through a motion compensation process. On the other hand, learned video coding is a brand new field allowing to reconsider some historical design choices of traditional coders. Since the initial work published in 2019, different strategies of temporal redundancies removal have been considered in the literature.

The first part of this chapter consists of a brief review of the literature to motivate the usage of motion compensation to remove temporal redundancies. Consequently, the learned approaches proposed in this thesis are composed of three successive stages: estimating and transmitting the motion information, computing a temporal prediction based on the motion information and finally conveying only the unpredicted part. These three stages are implemented by neural networks. In order to carefully compare among different possible network architectures, strict experimental conditions are used. The second part of the chapter presents these experimental conditions, implemented to evaluate the different proposed systems.

Temporal dependencies reduction

Need of an explicit motion compensation

Let us model a video sequence V as set of T frames: V = {x 1 , . . . , x T }. Each video frame x t is represented as a tensor of dimension 3 × H × W. Early work from 2019 by Habibian

inter frame coding with neural networks

The introduction of a motion compensation process leads to the decomposition of the learned coding schemes in three main steps, presented in Fig. 4.1. The first step is to estimate and efficiently convey the motion between the frame to code and its references.

Then, the motion compensation takes place to compute a temporal prediction through the application of the motion information to the reference frames. Lastly, only the remaining unpredicted part of the frame needs transmission. This unpredicted part exists for two reasons: either the content is simply unpredictable (a new object appearing on the image for instance), or the motion compensation step is not accurate enough to achieve a satisfactory temporal prediction. As presented in the learned image coding chapter, autoencoders are generic tools to efficiently transmit a quantity from the encoder to the decoder. Thus, they are used to transmit the motion information as well as the unpredicted part. The other steps are implemented in several ways in the literature. Table 4.1 summarises the main differences among existing learned video coders. The motion estimation column indicates how the motion information is obtained. The Bi-pred column details whether a bi-directional prediction is available. If not, the coders only relies on P-frames, leading to reduced compression efficiency. Then, the Image residual entry details how the non-predictable information is computed and sent to the decoder. The last column indicates whether the same coder (i.e. the same network with the same parameters) is used for intra and inter frames, or if a dedicated intra coder is required.

Information presented in Table 4.1 are discussed in the next two sections. The objective is to identify the limitations of existing learned video coding schemes in order to design a better learned coder.

Conclusion

In this introductory chapter, a thorough review of the learned video coding literature has been carried out. Based on this literature review, a coding strategy has been chosen to comply with requirements regarding the complexity, the performance and the flexibility of the coder. The most significant design choice is to perform a frame-by-frame processing, which in turns implies a motion compensation step. As such, the processing of a single frame is split into three successive steps: motion estimation and transmission, motion compensation and sending the remaining unpredicted part.

Throughout the literature review, it has been noted that most existing learned coding schemes perform these steps using usual tools such as residual coding or pre-trained motion estimation networks. The two next chapters reconsider these design choices, in the light of the possibilities offered by neural networks. All the neural-based designs presented in the following chapters will have to be accurately evaluated. To this end, a real-life coding task is selected: the CLIC21 video track. In order to ensure the relevance of the learned coding schemes, state-of-the-art anchors are proposed. To this day, traditional approaches (HEVC, VVC) remains the best performing systems. For fair comparison with these anchors, the random access coding configuration of the CTC [START_REF] Bossen | VTM common test conditions and software reference configurations for sdr video[END_REF] is chosen. Finally, a training process is devised to prepare the systems for the target coding configuration.

2. Be resilient to any nature of data for x enc , x dec and x out .

Conditional coding principles

The proposed conditional coding architecture is depicted in Fig. 5.3. At the encoder-side, the analysis transform g a is fed with the concatenation of the feature maps from x enc and x dec . It computes latent variable y, representing the information to be sent i.e. the information present at the encoder but missing at the decoder:

(5.4)

The analysis transform g a is a convolutional network. As such its successive layers intertwine the encoder and decoder quantities in a more general way than a simple subtraction. This allows to learn an adapted mixture through the training process. Since the conditioning latent variable is computed at the decoder-side, it does not prediction of the previous chapter by achieving an actual motion compensation process.

To this end, the coding scheme now transmits some motion information, conditioning the motion compensation process. In the learned video compression literature, bidirectional predictions remain less common than simple single-reference prediction. Indeed, only 3 out of the 12 coding schemes presented in Table 4.1 feature a bidirectional prediction. Yet, Section 2.3.1 and Eq. (2.6) remind that a bidirectional prediction step allows to reduce the prediction noise, i.e. improving its accuracy and thus the entire coding scheme. As such, we propose to implement a bidirectional prediction within the coding scheme. Fig.

6

.1 details the bi-directional motion compensation step. x t to one of its reference frames. The two optical flows are applied separately on the two reference frames through w, a warping function which performs motion compensation.

Here, sub-pixel motions are achieved through a bilinear interpolation. As such, w is called a bilinear warping function. Finally, the two intermediate motion compensated references are combined through a sum weighted by β ∈ [0, 1] H×W , a pixel-wise bidirectional weighting (see Section 2.3.1). This yields the bidirectional prediction equation:

The temporal prediction has to be present at the decoder so that only the unpredicted content is sent by the encoder. To this end, the two optical flows and the bidirectional weighting must be conveyed to the decoder. Figure 6.2 presents the overall coding scheme.

objective which deviates from the targeted rate-distortion optimization. This might lead to suboptimal systems. For both convenience and performance-related motivations, we advocate for an actual end-to-end rate-distortion training, with no pre-trained component or supplementary loss terms. To this end, an additional coding mode called the Skip mode is introduced to promote the learning of relevant motion information.

Skip mode: an additional coding mode 6.3.1 Principles

In this thesis, the coding scheme is enhanced through a supplementary coding mode: the Skip mode. It is inspired to an existing Skip mode in conventional video coding, where the temporal prediction is used without the residual. Here, the proposed Skip mode consists in a pixel-wise copy of the temporal prediction. Offering the possibility of copying the prediction acts as a strong incentive to learn relevant motion information. Indeed, in a system without the Skip mode, the motion information is learned to obtain a proper prediction which performs well when fed to the CNet. Thus, the training of the MNet happens through CNet, making it more difficult to converge. Here, the Skip mode is a shortcut: an accurate prediction can now be exploited directly (i.e. by being copied) instead of going through the CNet. This considerably stabilizes the training process, allowing to train the whole coding scheme from scratch using the rate-distortion loss.

The reconstructed frame xt is computed by adding the contributions of both the CNet and Skip mode. This sum is arbitrated by the continuous pixel-wise coding mode selection α ∈ [0, 1] H×W , with H and W the spatial dimension of x t :

2)

The coding mode selection applied on the CNet input (at the encoder) to select the areas of x t that have to be conveyed through the conditional coder. Its role is to zero areas from x t before coding them through CNet, allowing to save their associated rate.

Conversely, α applied at the decoder side to obtain the Skip mode contribution. The coding mode selection is continuous, allowing blending between the two coding modes.

Rate-distortion results

The performance brought by the addition of motion information and the Skip mode is presented in Fig. 6.10a. Additionally, performance of systems featuring motion information but no Skip mode are presented. Recall that the absence of the Skip mode prevents an actual rate-distortion end-to-end training from scratch. In order to obtain skipless systems, pre-trained systems with motion information and Skip mode are used as starting points for the training process. Then, the Skip mode is disabled by forcing α = 1 (CNet only) and the resulting skipless system is finetuned by extending the training phase.

Throughout the evaluated rate range, systems with a Skip mode are equivalent or better than systems without one. This further justifies the presence of the Skip mode, which not only helps the convergence of motion information, but also improves the coding performance. This improvement is mainly explained by the fact that Skip mode takes the majority of the frame area, as shown in the preceding visual examples. This leaves CNet with only a small fraction of the frame, allowing it to be specialized for this smaller set of data through the learning of more suited parameters, which results in better performance.

At high rate, the presence of motion information significantly improves the compression conditioning latent variable to zero. This ablation is identical to the one proposed in Chapter 5 for CNet, except that it now concerns MNet. The different configurations of this ablation are summarised in Table 6.1 and the results are presented in Fig. 6.13 in a B-frame coding context. Table 6.1: Different configurations for the MNet conditional coding ablation. The first column presents two quantities: the optical flow v f (Fig. 6.13a) and the coding mode selection α (Fig. 6.13d). These quantities are obtained through the complete synthesis transform i.e. by feeding it with both the conditioning latent and the analysis latent variables. On the second column (Fig. 6.13b and 6.13e), the same quantities are presented when synthesized from the conditioning latent variable only. Conversely, the CHAPTER 7

Latent variables Synthesis Analysis Conditioning operation

TOWARDS PRACTICAL VIDEO CODING was submitted to the challenge to assess its performance.

Besides the evaluation of our system against other learned coders, this challenge highlights the importance of having a flexible coder. Indeed, due to the challenge rules, using the optimal coding strategy for each video sequence improves the overall performance. To this end, this chapter proposes different dimensions of coding strategies competition.

CLIC21 video track

Challenge presentation

The objective of this challenge is to obtain the highest quality (measured with the MS-SSIM) while compressing 100 high-resolution (720p) video sequences, with a frame rate of

Sequence-wise competition

The challenge rules do not impose sequence-wise constraints concerning the rate or quality of each individual sequences. Thus, it is possible to improve the overall results through a relevant sequence-wise quality selection. For instance, sparing a few bits on a sequence might lead to a minimal quality loss while investing these bits on some other sequence dramatically improves its quality. This rationale justifies the competition between different coding strategies for each sequence, in order to optimize an overall rate-distortion cost.

To formalize this problem, let us denote J λ (v i ; C) the rate-distortion cost of the video sequence v i compressed using the coding strategy C:

Here D and R denote the distortion and the rate associated to the current video and coding strategy. For each video sequence v i , the objective is to find the optimal coding strategy C ⋆ i , yielding the minimal rate-distortion cost:

Searching the best coding strategy for all the N video sequences of the dataset allows to obtain the optimal average rate R and distortion D:

The value of the rate control parameter λ is set empirically so that the resulting rate R respects the desired rate target.

We propose to perform coding strategies competition using the final system designed in the previous chapter (i.e. with Skip mode and conditional coding for MNet and CNet).

Recall that several instances of the proposed learned coder have been trained for different cancelled by the significant amount of additional parameters required. To fully benefit from rate competition, the number of additional has to be mitigated in order to be competitive according to the challenge rules.

Design of a multi-rate coder

In order to benefit from rate competition between N coders, the size of those systems must remain small. We propose to transform a mono-rate system into a multi-rate one, with a negligible increase in the number of parameters. For all rates addressed by the multi-rate system, the main components of the coder remains the same. That is, MNet and CNet transforms (analysis, synthesis, conditioning and hyperpriors) remain identical, regardless of the rate target. In the proposed multi-rate system, the single operation conditioned on the rate target is the CNet quantization gains. In Section 5.6, quantization gains were used to achieve different quantization accuracies according to the type of the frame (I, P and B). To obtain multi-rate systems, CNet quantization gains are now also conditioned on the rate constraint, as proposed in [START_REF] Guo | Variable rate image compression with content adaptive optimization[END_REF]. In order to address N rate constraints, 3 × N pairs of encoder-decoder quantization gains are learned: Let us denote M ≃ 27 × 10 6 the number of parameters for the decoder of a mono-rate system. The number of parameters required to address N = 6 different rates becomes:

CONCLUSIONS AND FUTURE WORK Thesis objectives

T HE ever-increasing consumption of images and videos has to be met with improved compression efficiency to mitigate the environmental impact of the infrastructures required to convey them. To this end, this thesis proposes to study the design of end-to-end learned video coding schemes. While the tools and techniques required to obtain state-ofthe-art learned image coders are known and presented in Chapter 3, learned video coding remains a more challenging task. So far, existing coders from the literature underperform compared to traditional coders such as HEVC and VVC. Furthermore, learned coders often exhibit less flexibility compared to traditional coders. For instance, they feature a constrained frame organization or they require a dedicated intra frame coder.

Given the novelty of learned video coding, this thesis has reconsidered the entire design of a learned video coder. For instance, the computation of a temporal prediction and the exploitation of this prediction are thoroughly investigated to incrementally build a learned coder. The design process is carried out with the objective of obtaining a practical video coder, competitive with traditional approaches when evaluated under different coding configurations and rate constraints.

Learned compression implements the usual compression techniques (entropy coding, prediction, transform) with neural networks. In this context, the benefits of neural networks are two-fold. First, they are able to perform non-linear operations achieving better results than the linear ones of traditional coders. Second, the different steps of the compression process can be jointly optimized through an end-to-end learning process, allowing to further improve the performance.

Learned image coding is tackled in Chapter 3 as the first step towards the design of learned video coders. The different elements of the compression pipeline are motivated from an information theory standpoint (Chapter 1) and from their usage in traditional coders (Chapter 2). The learned image coder proposed in this thesis achieves performance competitive with the image coding configuration of VVC, the latest video coding standard.

Future works

Further performance improvement

This thesis presents evidence of the neural networks ability to outclass handmade operations of traditional coders. For instance, this is the case in image coding where learned approaches already achieve state-of-the-art results, or for conditional coding which outperforms residual coding. Yet, learned approaches seem to struggle for one key step of the coding pipeline: computing a temporal prediction. Indeed, Section 6.5.2 has shown that MNet fails to capture important motions. We believe that this step could be significantly enhanced. So far, the motion autoencoder MNet shares the same architecture than the signal autoencoder CNet. Using known techniques from the optical flow estimation literature, such as the addition of a cost volume component, might yields important improvements.

All the video coding trainings done in this thesis have been performed using a proxy coding configuration, composed of one I-frame, one P-frame and one B-frame. The rationale behind this choice is to prepare the model to process all three types of frame. However, finetuning the system using the target frame organization would allow to obtain a better coder, specialized for the desired coding structure.

More practical learned coding

The models presented in this thesis are designed with few complexity constraints, resulting in coders with dozens of million of parameters. In video coding, the complexity constraint is often tighter on the decoder than on the encoder, since the decoder has to be embedded in a variety of low-power devices (smartphones, set-top boxes). Yet, learned coder architectures are often symmetrical. For instance, both the encoding and decoding speed of the proposed system are around a few frame per second for a 720p video. This is different from traditional video coding, where the encoding time significantly exceeds the decoding time. Investigating different balance of complexity between the encoder and decoder would be a important step towards a real-life usage of learned compression.

Likewise, most of the work carried out on learned compression relies on mono-rate coders. This requires to store multiple coders to adjust the rate of the video sequences regarding the available bandwidth. The basic multi-rate coder presented in Chapter 7 offers a solution to this issue. Yet, it is not able to reach competitive performance on a range of rates as wide as traditional coders. Conditioning more operations and coding modes on the rate constraint would allow to obtain better multi-rate coders.

Content adaptation

In traditional coding, the purpose of the encoding stage is to find the tools which are the most suited to the current video, within the set of tools provided by the standard. These tools have been designed beforehand and target different types of signal: low-frequency signals, screen content signals etc. On the other hand, the nature of neural networks is to adapt to their training data. As such, performing the training on the data to compress would allow to obtain an optimal coder for this particular video sequence. Even though the parameters would have to be sent alongside the video, several recent works have hinted this idea as a promising means of improving compression efficiency [START_REF] Dupont | COIN: compression with implicit neural representations[END_REF], [START_REF] Van Rozendaal | Overfitting for fun and profit: instance-adaptive data compression[END_REF].

Performing operations adapted to the signal is one key factor for improvement and might be one of the explanation regarding the better performance of traditional coders.

Throughout this thesis, several means of adapting the operations to the signal have been introduced:

• The hyperprior mechanism allows to use an adapted probability model for each latent pixel;

• The Skip mode offers the choice between two coding modes;

• The quantization gains conditioned on the frame type enable to accurately set the quantization accuracy.

We believe that offering even more possibilities of adaptation to the signal to compress would yield an important increase in performance. The richer probability model results in a better entropy coding leading to lower rate.

LIST OF PUBLICATIONS

Experiments under the Challenge on Learned Image Compression test conditions demonstrate that this method achieves 18 % rate saving compared to Gaussian or Laplace models. [START_REF]Workshop and challenge on learned image compression[END_REF], the DIV2K dataset [START_REF] Agustsson | Ntire 2017 challenge on single image super-resolution: dataset and study[END_REF] and the JPEG-AI dataset [START_REF] Dataset | [END_REF]. These datasets are known to feature miscellaneous contents with diverse resolutions, allowing to learn a generic image coder.

Patent applications

B.1.2 Details on the training stage

The expectation of the loss function is estimated by computing the average loss when As such, the learning rate is successively decreased to η = 2 • 10 -5 and η = 4 • 10 -6 .

The purpose of the training is to optimize a rate-distortion loss, parameterized through a rate constraint λ:

Empirically, replacing the rate-constraint λ with a smaller one λ ′ < λ during the first iterations of the training often results in better performance.

B.2 Additional information regarding the test stage B.2.1 Inference dataset

The performance of the system is evaluated on the CLIC 2020 validation set [START_REF]Workshop and challenge on learned image compression[END_REF]. It is composed of 102 natural images with a resolution ranging from 512 × 384 to 2021 × 1518.

To provide additional variety, two different categories of images are present in this dataset.

Half is made of photos from professional photographers, featuring sharp and in-focus edges as well as bokeh effects. The other half is composed of user generated content, which often exhibits more noise and less sharpness.

B.2.2 Coding standards as anchors

In order to assess the performance of the learned coding schemes, several existing standards are used as anchors. The JPEG image coding standard [START_REF] Wallace | The jpeg still picture compression standard[END_REF] is evaluated with the following image magick command, relying on libjpeg 6.2:

Varying the quality parameter Q allows to obtain a rate-distortion curve.

Despite being very popular, JPEG is not a state-of-the-art still image coder. The image coding configuration of modern video coding standards (HEVC and VVC) achieves better rate-distortion performance. To ensure a fair comparison, the results of HEVC and VVC are obtained through one of their best implementation, the HM 16.22 (HEVC Test Model) and the VTM 10 (VVC Test Model), using the intra coding configuration for RGB 444.

B.3 Detailed ARM architecture

This section details the architecture of the Auto-Regressive Module (ARM) and the fusion module whose results are presented in Section 3.4.2. objective is to obtain a generic coder. In particular, the system should perform well on:

B.6 Additional visual examples

• Different natures of video sequences (user-generated contents, professional videos and synthetic sequences);

• Different resolutions, from 540p to 2160p;

• Different frame rates, from 24 to 60 frames per second.

To this end, the training dataset is constructed by gathering different existing video datasets. the subtraction between the frame to code and its prediction. For image coding, the input is the frame to code x t . of the intra frames is always more important than those of the inter frames. Moreover, this example allows to illustrate that the system learns to disable some latent feature maps when the rate constraint becomes too tight. For instance, compare the highest and the lowest rate systems, respectively the systems a and g. For the high-rate system, the As the zoom out is centered on the man, the pixels on the borders of the images move faster than those in the center. This can be seen through the darker shade of the colours at the border of the optical flows. While the man's body remains still, its hands are moving vertically. This is well captured in the optical flows. Lastly, the text remains at the same place throughout the sequence. This explains the white areas at the bottom right of the optical flows. This example proves that the system is able to learn to estimate and transmits complex and multiple motions, as well as properly segmenting between different areas (background, the man's hands, the text).

C.2 Dataset composition

D.2 Quantization gains

E.2 Additional examples of optical flows