Etienne Meunier

Thèse Présentée

Nicolas Thome Professeur

Patrick Perez

Elisa Fromont

Patrick Bouthemy

Renaud Fraisse

Unsupervised learning for motion segmentation and motion saliency in videos

Defence&Space

Un grand merci aux membres de l'équipe Sairpico et Odyssey à l'Inria Rennes avec qui j'ai pu avoir des échanges enrichissants, qui m'ont aidé à surmonter certains de mes problèmes et à développer un intérêt pour de nouveaux sujets fascinants.

Je souhaite exprimer ma reconnaissance envers ma famille et mes amis qui m'ont encouragé, soutenu et avec qui j'ai partagé des moments qui me sont chers. Ils ont, par leur présence, rendu possible la rédaction de cette thèse.

Enfin, je tiens à mentionner le potager collectif situé au croisement de la rue des Plantes et de la rue Mirabeau ainsi que les volontaires qui en prennent soin. Tout au long de mon travail de thèse, c'est ici que j'ai pu trouver le calme nécessaire pour écrire et porter un regard nouveau sur mon travail. J'ai eu la chance extraordinaire de réaliser ma thèse dans ce cadre si encourageant, et je suis fier des trois années que j'ai consacrées à l'étude du mouvement.

Résumé en Français

Motion Segmentation

Clustering trajectories

Several works achieve motion segmentation in an image sequence by analysing trajectories. The method presented in [51] is representative of the classical approach to this problem. In this paper, the authors compute trajectories along the video sequence and build an affinity metric function that evaluates the distance between two trajectories. Then, they apply a clustering algorithm on the pairwise affinity matrix to group trajectories together, leading to a sparse motion segmentation. Finally, they adopt a variational approach that exploits image gradients to convert the sparse labels into a dense segmentation.

This method underlines the three key elements of the trajectory clustering methods :

-A representation of trajectories along the sequence and an affinity metric between trajectories, -A clustering algorithm to group trajectories together, -A method to go from the sparse labeling to a dense segmentation.

. The background motion model along the sequence is defined by a set of homographies H k , k ∈ {1 • • • T }, where H i is the homography representing the global motion between I i and I i+1 . In this method, they exploit a CRF to obtain a dense segmentation from the sparse labeling. One advantage of this approach is that it is more robust than classical clustering. However it assumes that the background occupies most of the image, and, because of the use of homographies to represent global motion, it can only accommodate the motion of a plan in 3D space.

Recently, another approach [53] revisited the trajectory clustering methodology using deep learning, focusing in particular the transition from sparse to dense labeling. The authors cluster the trajectories, then, train a U-net to provide a dense segmentation from a frame of the video using the sparse annotation as ground-truth labels. This method allows to improve the dense segmentation results by relying on the convolutional network ability 1.

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • THÈSE DE DOCTORAT DE L'UNIVERSITÉ DE RENNES ÉCOLE DOCTORALE N O 601
Mathématiques, Télécommunications, Informatique, Signal, Systèmes, Électronique Spécialité : AST (Signal, Image, Vision)

RÉSUMÉ EN FRANÇAIS

Contexte et motivation

Contexte de l'analyse du mouvement

La façon dont nous percevons et interprétons le mouvement nous fournit des informations essentielles sur notre environnement. D'un point de vue fonctionnel, parce qu'un objet en mouvement peut représenter quelque chose de particulièrement digne d'attention, le mouvement nous aide à nous concentrer sur ce qui est important autour de nous. Le mouvement intègre également le fait que le monde qui nous entoure est en train de changer et que nous devons prêter attention à cette version actualisée de notre environnement. Tout comme l'apparence, le mouvement des objets ou des entités qui nous entourent peut faire l'objet d'une interprétation et nous fournir des indications utiles sur la manière d'interagir avec eux.

Dans le contexte de la recherche visant à développer des systèmes de vision par ordinateur et d'intelligence artificielle pour assister les humains, il est crucial de construire des mécanismes capables de percevoir et de représenter le mouvement environnant de manière significative. Toutefois, cette tâche est difficile à reproduire car le système visuel humain est étonnamment doué pour comprendre les mouvements. Par exemple, lorsque nous marchons dans un endroit bondé où se produisent de nombreux mouvements différents, nous sommes capables de remarquer rapidement un seul d'entre eux et de nous concentrer dessus. De même, lorsque nous courons dans une forêt, nous sommes capables d'ignorer la parallaxe, due à notre propre déplacement, de mouvement apparent des arbres qui nous entourent, afin de nous concentrer sur la présence potentielle d'objets se déplaçant indépendamment les uns des autres. Un examen de cette question et quelques explications sur la manière dont le cerveau humain gère cette tâche sont donnés dans des travaux antérieurs [1].

Motivations pour l'analyse du mouvement

L'analyse du mouvement est l'une des principales tâches de la vision par ordinateur et a de nombreuses applications dans une grande variété de domaines. Tout d'abord, la segmentation d'objets en mouvement a des applications dans le montage vidéo, où nous pouvons sélectionner des objets en fonction de leur mouvement, mais aussi en robotique ou en conduite autonome, où le mouvement des objets qui nous entourent peut fournir des informations sur la partie de la scène sur laquelle nous devons nous concentrer, ou sur les objets que nous devons éviter.

Deuxièmement, la détection des mouvements saillants a des applications en matière de sécurité, où elle peut être utilisée, par exemple, pour détecter des mouvements anormaux dans un groupe de personnes ou de voitures dans une vidéo aérienne d'une ville. Dans ces cas, le mouvement joue un rôle critique car l'apparence ne révèle pas la situation anormale. En biologie, la détection d'objets qui ont un mouvement singulier est également intéressante car elle peut permettre de détecter des réactions anormales.

Troisièmement, développer des descripteurs pour représenter le mouvement est également un objectif intéressant car il est à la base de nombreuses tâches en aval. Par exemple, certaines recherches cliniques exploitent les descripteurs de mouvement pour aider à diagnostiquer la maladie de Parkinson à partir de vidéos de mouvements de la main [2]. Nous pouvons également utiliser les descripteurs de mouvement pour améliorer la reconnaissance d'activités ou aider à construire de meilleures interfaces personne-machine. Développer de bons descripteurs de mouvement permet de "capturer" un mouvement à partir d'une vidéo et de le faire correspondre à un mouvement observé dans d'autres vidéos.

Enfin, une autre motivation, qui est au coeur de plusieurs travaux récents [3]- [5] est d'apprendre des descripteurs d'apparence à partir de la segmentation du mouvement. L'idée repose sur le principe de Gestalt et sur le fait que des points se déplaçant solidairement sont suceptibles d'appartenir au même objet. Par conséquent, en faisant coïncider la segmentation basée sur l'apparence avec la segmentation basée sur le mouvement, nous pouvons apprendre les descripteurs d'apparence des objets sans dépendre des annotations humaines, qui sont coûteuses à obtenir. Cela conduit à la perspective passionnante que nous pourrions apprendre à segmenter des objets dans des images statiques simplement en regardant des vidéos.

Développement de descripteurs de mouvement

. Le domaine de l'apprentissage profond a récemment développé de très bonnes représentations pour les images [6], le son [7], le texte [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF] et montrent qu'il permet des applications impressionnantes et facilitent l'apprentissage des tâches en aval. Des descripteurs de vidéo informatifs ont également été développées pour décrire les vidéos avec par exemple des travaux sur la reconnaissance d'activités [START_REF] Vahdani | Deep learning-based action detection in untrimmed videos: A survey[END_REF] où le réseau est entraîné à classer les vidéos dans un ensemble fini d'activités prédéterminées et doit donc apprendre à représenter le clip vidéo de manière compacte. La plupart de ces travaux s'appliquent au niveau de la vidéo et prennent en compte l'apparence, qui peut être un discriminant fort dans certaines activités. En outre, comme ils sont élaborés sur un ensemble forcément restreint de vidéos, il y a une limite en termes de généralisation. Le développement de descripteurs de mouvement généraux pouvant être utilisés dans n'importe quel contexte constitue un objectif intéressant pour faciliter l'analyse des mouvements. Cela doit permettre de comparer deux mouvements différents ou de faire correspondre un mouvement observé à une base de données de mouvements connus.

L'élaboration de descripteurs à partir de mouvement n'est pas nouvelle et n'est pas non plus propre à l'apprentissage profond. Par exemple, la notation Laban illustée à la Fig. 1 a été développée en 1928 pour représenter le mouvement des corps humains dans les chorégraphies de danse. Ce système utilise des symboles pour décrire la direction du mouvement dans l'espace, la partie du corps qui exécute le mouvement (chaque partie du corps humain est étiquetée avec un symbole différent), la durée du mouvement (chaque ligne représente un pas de temps, et les mouvements simultanés sont représentés sur la même ligne) et la dynamique du mouvement. Bien qu'il existe de nombreuses notations de mouvement comme celle-ci, l'apprentissage profond pourrait aider à construire des représentations plus générales qui pourraient s'appliquer non seulement au mouvement humain, mais aussi à tous les mouvements qui nous entourent. Comme nous l'avons vu précédemment, cette notation pourrait être utile dans plusieurs types de tâches.

Principaux objectifs de l'analyse du mouvement

Le flot optique est défini comme le champ de déplacement 2D qui décrit le mouvement apparent des pixels entre deux images successives [START_REF] Horn | Determining optical flow[END_REF]. Les approches classiques s'appuient généralement sur l'hypothèse de conservation de l'intensité (Brightness Con-Figure 1 -Exemple de partition utilisant le système de notation Laban. -Loren Foster, France Culture -"Danse : comment s'écrit le mouvement ?" stancy Equation -BCA-en anglais), qui suppose que l'intensité d'un pixel en mouvement reste constante dans le temps. Plus récemment, les approches par apprentissage profond ont atteint les meilleurs résultats dans l'estimation du flot optique. La plupart de ces méthodes sont entraînées dans un cadre supervisé sur une base de données synthétiques. Toutefois, certaines méthodes sont entraînées dans un cadre non supervisé en utilisant des fonctions de pertes s'appuyant sur les fonctionnelles précédemment définies dans un cadre variationnel.

Les techniques de segmentation du mouvement visent à distinguer et séparer les mouvements 2D au sein d'une vidéo, quelle que soit la source de chaque mouvement individuel. Aujourd'hui, il est possible de réaliser la segmentation du mouvement directement à partir d'un champ de flot optique, et de développer des approches d'apprentissage profond non supervisées, qui apprennent la segmentation du mouvement sans annotations humaines. Nous proposons une taxonomie des techniques de segmentation du mouvement à la figure 1.1 du chapitre 1, où nous soulignons que nous pouvons diviser les méthodes en correspondance visuelles intra-video, en regroupement de trajectoires et en segmentation du mouvement instantané. Cette dernière catégorie est au coeur de notre travail.

La segmentation des objets mobiles dans les vidéo (Video Object Segmentation -VOSen anglais) est l'une des principales tâches en analyse du mouvement. L'objectif est de segmenter dans la vidéo l'objet principal en mouvement au premier plan de la scène. Les méthodes de VOS sont évaluées sur des ensembles de données tels que DAVIS2016. Dans la figure 2, nous montrons deux séquences de DAVIS2016 où nous pouvons voir que l'objet segmenté est l'objet principal en mouvement dans l'ensemble des images de la séquence. La plupart des méthodes de VOS combinent des informations d'apparence et des informations de mouvement obtenues via le champ de flot optique. En outre, nous pouvons également mentionner des méthodes s'attaquant à la segmentation amodale, une tâche connexe où l'objectif est de segmenter les objets dans leur ensemble même lorsqu'ils sont partiellement occultés ou lorsqu'ils s'arrêtent. Cette tâche est plus difficile car elle nécessite de modéliser la forme de l'objet pour la prédire lorsqu'elle est occultée. Un autre intérêt de l'étude des méthodes VOS est qu'elles donnent une idée de ce que les méthodes non supervisées peuvent réaliser si nous trouvons les fonctions de perte et les procédures d'apprentissage adéquates.

Enfin, nous pouvons mettre en avant la saillance des mouvements (Motion Saliency -MS-en anglais), qui consiste à mettre en évidence les mouvements locaux qui se distinguent de leur contexte environnant et tendent ainsi à révéler un événement important.

Motivations

De nombreux travaux ont été réalisés dans le domaine de la segmentation du mouvements et les progrès ont été rapides ces dernières années. Cependant, il reste encore des défis à relever dans cette tâche. Nous soulignons ci-dessous les motivations qui ont guidé notre travail et notre méthodologie en analyse du mouvement. Dans de nombreux travaux sur l'analyse de vidéo, l'apparence et le mouvement sont combinés. Plusieurs travaux montrent que cette fusion permet d'obtenir des améliorations significatives sur les benchmarks classiques de segmentation VOS, ce qui justifie l'adoption de caractéristiques d'images. Cependant, il existe un biais important dans ces ensembles de données : l'objet à détecter n'est pas seulement saillant par son mouvement par rapport à l'arrière-plan, mais aussi par son apparence et son emplacement, l'objet étant généralement placé au centre des images (ce qui est un indice explicitement exploité par plusieurs approches [START_REF] Zhang | Video saliency prediction based on spatial-temporal twostream network[END_REF], [START_REF] Kim | Spatiotemporal saliency detection using textural contrast and its applications[END_REF]).

Le fait que la plupart des travaux récents sur l'analyse de vidéo combinent les deux modalités pourrait être un problème, car il limite l'utilisation de ces algorithmes aux vidéos pour lesquelles l'apparence de l'objet d'intérêt peut être facilement représentée par un réseau neuronal. Par exemple, de nombreuses méthodes reposent sur l'utilisation de caractéristiques DINO qui ont été entraînées sur un vaste ensemble de vidéos naturelles provenant d'Internet, ce qui ne peut pas être appliqué à d'autres types de vidéos représentant des objets pour lesquels nous ne disposons pas d'une telle base de données, comme les images satellites, les images biologiques, les vidéos sous-marines ou industrielles. Par exemple, les résultats expérimentaux en apprentissage profond montrent que les modèles entraînés sur des flots optiques, y compris synthétiques, ont une bien meilleure capacité de généralisation que ceux entraînés sur les images. En outre, le fait de se concentrer uniquement sur le mouvement permet des transferts intéressants qui modifient le contexte des vidéos. On pourrait imaginer transférer un algorithme de détection de saillance du mouvement entraîné sur des foules à l'analyse du mouvement de groupes de cellules.

De même, l'entraînement supervisé limite l'applicabilité des méthodes. Premièrement, elle nécessite une grande quantité de données annotées par des humains, ce qui est coûteux à obtenir. Deuxièmement, il contraint la tâche et peut introduire des biais. Par exemple, si nous prenons la tâche de segmentation du mouvement, toutes les bases de données segmentent l'objet principal (ou les quelques objets principaux pour FBMS59 et DAVIS2017) dans la scène. Cependant, dans la pratique, lorsque nous regardons ces vidéos, nous pouvons y trouver d'autres mouvements à segmenter qui ne sont pas recensés dans la vérité terrain. Tout d'abord, nous avons des mouvements indépendants d'objets d'arrière-plan comme dans les séquences breakdance ou scooter-black de DAVIS2016, des mouvements dus à l'eau comme dans la séquence blackswan de DAVIS2016. Dans un environnement naturel, on pourrait également évoquer le mouvement des arbres ou des nuages lorsqu'il y a du vent. Deuxièmement, nous avons la parallaxe du mouvement liée aux objets statiques d'avant plan et causée par le mouvement de la caméra et les différences de profondeur. Elle peut entraîner de fortes différences de flot, comme dans les séquences de parkour ou de libby dans DAVIS2016. Troisièmement, nous avons les mouvements articulés, qui pourraient être traités de manière plus complète. Tous ces mouvements nous donnent des informations précieuses sur le monde qui nous entoure et sont écartés dans les approches supervisées, car il serait impossible de tous les segmenter manuellement. Par conséquent, l'apprentissage de la segmentation des mouvements de manière supervisée ne résout qu'une partie du problème. Nous pensons que pour aborder la segmentation du mouvement, la bonne approche est l'apprentissage non supervisé. C'est la seule qui puisse nous conduire à une compréhension complète du mouvement de la scène.

Principales contributions

Pour les raisons susmentionnées, nous avons choisi d'établir un cadre bien formalisé pour ce travail. Nous essayons d'identifier des objets mobiles saillants ou de segmenter des objets mobiles indépendants en nous basant uniquement sur leur mouvement. Cela implique un certain nombre de décisions méthodologiques. Tout d'abord, nous ne nous appuyons pas sur l'apparence ou sur des informations sémantiques. Deuxièmement, nous n'avons pas besoins des informations d'égomotion ou des paramètres intrinsèques de la caméra. Enfin, notre procédure d'apprentissage ne repose pas sur l'annotation manuelle. Cela nous conduit au défi stimulant d'exploiter les seuls champs de flot optique qui indiquent seulement le mouvement des points individuels d'une paire d'images.

Les contributions de cette thèse s'articulent autour de deux grands axes d'analyse du mouvement.

Le premier concerne les approches d'apprentissage profond pour la segmentation non supervisée du mouvement. Dans cet axe, nous construisons un cadre pour entraîner un réseau à segmenter le mouvement à partir du champ de flot optique en utilisant une fonction de perte inspirée de l'algorithme EM. Il décompose le flot en segments de mouvement cohérents, chacun représenté par un modèle de mouvement paramétrique. Nous étendons ensuite progressivement ce cadre à des séquences vidéo plus longues. Dans un premier travail, nous prenons en entrée des triplets de champs de flot optique. Nous introduisons la dimension temporelle à court terme avec un terme de la fonction de perte qui assure la cohérence des étiquettes au sein des triplets. Nous l'étendons à une cohérence temporelle à long terme par un post-traitement qui relie les triplets pour former une segmentation longue et temporellement cohérente. Dans un second travail, nous introduisons une représentation du mouvement basée sur les splines qui est capable de représenter l'évolution d'un champ de mouvement paramétrique dans le temps. Nous considérons sur un réseau neuronal incluant des transformers pour permettre des interactions entre les caractéristiques de la séquence complète. Il en résulte une segmentation du mouvement temporellement cohérente sur l'ensemble de la séquence, sans avoir recours à un quelconque post-traitement. Toutes ces méthodes ont été testées sur des benchmarks VOS classiques (DAVIS2016, FBMS59, SegTrackV2) et fournissent des résultats compétitifs tout en étant efficaces au moment du test.

Le deuxième axe est la localisation des mouvements saillants à partir des champ de flot optique. Dans cette partie, nous articulons notre travail autour de l'idée que les zones saillantes sont celles qui influencent la sortie d'un réseau pré-entraîné qui prédit si un champ de flot optique comprend des mouvements saillants. Dans cette définition, la saillance est spécifique à la tâche que le réseau tente de résoudre. Dans un premier travail, nous utilisons une méthode d'interprétation de réseau par gradient pour localiser les zones saillantes. Nous y adjoignons une méthode de segmentation du mouvement.

Dans un second travail, nous optons pour une approche adverse afin de localiser les zones saillantes. Nous substituons le flot optique par inpainting dans le but de modifier la sortie du réseau de classification. Nous avons considéré deux tâches différentes de saillance de mouvement, l'une impliquant la détection de mouvements indépendants dans une scène observée par une caméra en mouvement, et l'autre impliquant la détection de mouvements anormaux de piétons dans une foule.

Organisation du manuscrit

Ce manuscrit comporte deux parties principales après le chapitre sur la description de l'état de l'art sur la segmentation du mouvement.

La première partie, composée des chapitres 2, 3 et 4, se concentre sur la segmentation du flot optique non supervisée à l'aide de réseaux profonds. Dans le chapitre 2, nous décrivons notre méthode de segmentation de mouvement du flot optique calculé entre deux images consécutives. Le chapitre 3 détaille notre extension à la segmentation de petits volumes de flot optique, fournissant une segmentation temporellement cohérente et une robustesse aux valeurs aberrantes. Le chapitre 4 décrit l'introduction de splines pour les modèles de mouvement et les transformers pour une segmentation temporellement cohérente de volumes de flot optique plus conséquents.

La deuxième partie, composée des chapitres 5 et 6, se concentre sur la détection faiblement supervisée des mouvements saillants dans les champs de flot optiques. Dans le chapitre 5, nous décrivons notre méthode de localisation des mouvements saillants à l'aide d'une interprétation par le gradient d'un réseau de classification. Dans le chapitre 6, nous présentons une méthode d'entraînement d'un réseau de segmentation pour localiser les zones saillantes à l'aide d'une approche adverse où nous masquons sélectivement une partie du champ de flot optique d'entrée.

Nous donnons plus d'informations sur chaque chapitre dans ce qui suit.

Chapitre 1

Dans le chapitre 1, nous présentons les travaux connexes principaux, en nous concentrant sur la segmentation du mouvement, mais aussi sur la segmentation des objets moviles dans des vidéos (VOS) et la détection de mouvements saillants. Nous présentons d'abord les méthodes traitant de la segmentation VOS. Ensuite, nous nous concentrons sur les méthodes liées à la segmentation du mouvement, en décrivant une taxonomie des méthodes. Enfin, nous présentons les méthodes liées à la saillance du mouvement, en traitant également les méthodes de saillance vidéo basées sur les caractéristiques d'apparence et de mouvement et les méthodes de saillance basées sur l'eye-tracking.

Chapitre 2

Dans le chapitre 2, nous définissons une méthode par apprentissage profond entièrement non supervisée pour segmenter les champs de flot optique en régions de mouvement cohérent. Nous supposons que le flot optique d'entrée peut être représenté comme un ensemble de modèles polynomiaux, typiquement des modèles de mouvement affine ou quadratique, chacun caractérisant un segment de mouvement (un segment est soit une région, c'est-à-dire une composante connectée, soit une couche qui n'est pas nécessairement connectée). L'idée centrale de notre travail est de se fonder sur l'approche EM (Expectation-Maximisation). Cela nous permet de concevoir de manière mathématiquement bien fondée la fonction de perte et la procédure d'entraînement de notre réseau neuronal de segmentation du mouvement. Contrairement à la méthode EM classique, notre réseau, une fois entraîné, peut fournir une segmentation pour n'importe quel champ de flot optique non vu en une seule étape d'inférence, sans dépendre de l'initialisation et sans estimer de modèles de mouvement paramétriques. Différentes fonctions de perte ont été étudiées, y compris des fonctions robustes. Nous avons également défini un nouveau schéma d'augmentation des données dédié au flot optique, qui a un impact notable sur la performance du réseau. Notre réseau de segmentation du mouvement a été testé sur quatre benchmarks, DAVIS2016, SegTrackV2, FBMS59, et MoCA, et a obtenu de très bons résultats tout en étant rapide au moment du test. Le logiciel EM-Flow qui met en oeuvre cette méthode est disponible sur Github (https://github.com/Etienne-Meunier-Inria/EM-Flow-Segmentation).

Chapitre 3

Dans le chapitre 3, nous définissons une méthode de segmentation pour les séquences de champs de flot optique, en introduisant la cohérence temporelle à notre travail développé précédemment. La cohérence temporelle est une caractéristique clé dans la segmentation de mouvement, mais elle est souvent négligée dans les méthodes non supervisées de segmentation de flot optique. Nous avons proposé un cadre spatio-temporel non supervisé original pour la segmentation du mouvement par flot optique qui explore pleinement la dimension temporelle du problème. Plus précisément, nous avons défini un réseau 3D pour la segmentation de mouvements multiples qui prend en entrée un sous-volume de flots optiques successifs et renvoie un sous-volume correspondant de cartes de segmentation cohérentes. Notre réseau est entraîné de manière totalement non supervisée. La fonction de perte combine un terme de reconstruction de flot qui incorpore des modèles de mouvement paramétriques spatio-temporels, et un terme de régularisation qui renforce la cohérence temporelle des masques. Nous avons spécifié un post-traitement simple pour la liaison temporelle à long terme des segments prédits, ce qui permet d'obtenir une segmentation cohérente sur de longues séquences tout en ne donnant que des triplets au réseau de segmentation. En outre, nous avons conçu une méthode flexible et efficace pour encoder les réseaux Unets. Nous avons réalisé des expériences sur plusieurs datasets avec des résultats quantitatifs convaincants. L'implémentation logicielle appelée ST-Segmentation est disponible sur github (https://github.com/Etienne-Meunier-Inria/ST-Segmentation).

Chapitre 4

Dans le chapitre 4, nous présentons les modèles de mouvement s'appuyant sur des splines pour une représentation du mouvement adaptée à une longue période de temps.

Chapitre 5

Dans le chapitre 5, nous introduisons un nouveau paradigme pour calculer le mouvement saillant dans les images vidéo à partir uniquement du flot optique. La saillance du mouvement (MS) est une question importante dans l'analyse de scènes dynamiques, quel que soit le domaine d'application (par exemple, l'observation au microscope de cellules vivantes, la télédétection ou la navigation autonome). La saillance des mouvements vise à mettre en évidence les mouvements locaux qui s'écartent du contexte environnant, ce qui tend à révéler des événements dynamiques significatifs. Nous avons formulé la saillance du mouvement comme une méta-tâche qui peut être instanciée pour différentes tâches habituellement traitées de manière indépendante. Pour étayer cette affirmation, nous avons abordé deux problèmes généraux importants : la segmentation d'objets mobiles dans une scène observée avec une caméra en mouvement et la détection de mouvements anormaux dans des foules.

Nous estimons la saillance du mouvement à partir de l'interprétation d'un réseau de classification de la saillance du mouvement prenant des flots optiques en entrée. Notre paradigme peut s'adapter à toute forme de saillance du mouvement en entraînant simplement le réseau de classification sur la tâche appropriée. Notre estimation de la saillance des mouvements est non supervisée. Toutefois la classification initiale est faiblement supervisée. Nous avons développé un schéma original d'interprétation du réseau en deux étapes exploitant la méthode LRP. Nous avons besoin de lui adjoindre une technique de segmentation de flot optique. Il fournit une segmentation binaire des mouvements saillants. Enfin, nous pouvons récupérer la carte de saillance de mouvement valuée par inpainting de flot paramétrique. Nous testons notre méthode sur deux cas de saillance différents. Le premier est le mouvement saillant produit par des objets se déplaçant indépendamment les uns des autres dans une scène observée par une caméra en mouvement. Le second cas est le mouvement saillant résultant d'un mouvement distinctif au sein d'un ensemble de mouvements cohérents. Les résultats expérimentaux sur des vidéos réelles et la comparaison avec les méthodes existantes démontrent la performance de notre méthode.

Chapitre 6

Dans le chapitre 6, nous présentons une première étude d'une approche adverse pour apprendre à segmenter les régions de mouvement saillant dans les champs de flot optique, à partir de la prédiction d'un réseau de classification. Dans ce travail, nous fusionnons les étapes de segmentation et d'interprétation pour produire directement la segmentation du mouvement saillant à partir du champ de flot optique. Au lieu de s'appuyer sur des techniques d'interprétation du flot, telles que la LRP, nous formons un réseau de segmentation pour localiser les zones qui, lorsqu'elles sont modifiées par inpainting paramétrique, changent la prédiction du réseau de classification. Nous montrons également que nous pouvons le formuler comme une tâche d'optimisation sous contrainte et utiliser une approche basée sur le lagrangien pour calculer automatiquement un hyperparamètre critique, la pondération entre les termes de la fonction de perte de notre réseau. A ce stade, nous montrerons des résultats préliminaires essentiellement sur la tâche de détection du mouvement saillant correspondant à un mouvement distinctif au sein d'un ensemble de mouvements cohérents.

GENERAL INTRODUCTION

Context and Motivations

Context of motion analysis

The way we perceive and interpret motion provides us with critical information about our environment. From a functional perspective, because a moving object may represent something particularly worthy of attention, motion helps us focus on what is important around us. Motion also embeds the fact that the world around us is changing and that we need to pay attention to this updated version of our environment. Much like appearance, the motion of the objects or entities around us can be subject to interpretation and provide us with useful cues about how to interact with them.

In the context of research trying to develop AI systems to assist humans, it is crucial to build mechanisms that are able to perceive and represent the surrounding motion in a meaningful way. However, this task is complicated to reproduce because the human visual system is surprisingly good at understanding motion. For example, when walking in a crowded place with many different motions, we are able to quickly notice a single one among them and focus on it. Furthermore, when running in a forest where we are able to ignore the motion parallax of the trees around us in order to focus on the potential presence of independently moving objects. A review of this question and some explanations on how the human brain handles this task is given in previous works [1].

Practical motivations for motion analysis :

Motion analysis is one of the main tasks in computer vision and has many applications in a wide variety of fields. First, segmentation of moving objects has applications in video editing, where we can select objects based on their motion, but also in robotics or autonomous driving, where the motion of objects around us can provide information about which part of the scene we should focus on, or about objects we have to avoid.

Second, the detection of salient motion has security applications, where it can be used, for example, to detect abnormal motion in a group of people or of cars in an aerial video of a city. In these cases, motion plays a critical role because the appearance does not reveal the abnormal situation. In biology, the detection of objects that have a singular motion is also of interest because it can allow to detect abnormal reactions.

Third, developing features to represent motion is also an interesting task as it is the foundation of many downstream tasks. For example, some projects exploit motion descriptors to help to diagnose Parkinson's disease from hand movement videos [2]. We could also use motion descriptors to improve activity recognition or help to build better human-machine interfaces. Developing good motion description features could allow us to "capture" a motion from a video and match it with motion observed in other videos. Finally, another motivation, which is at the core of several recent works [3]- [5] is to learn appearance descriptors from motion segmentation. The idea is based on the Gestalt principle and the fact that points that move together belong to the same object. Therefore, by making appearance-based segmentation coincide with motion-based segmentation, we can learn object features without relying on human annotations, which are expensive to obtain. This leads to the exciting perspective that we could learn to segment objects in static images just by watching videos.

Development of general motion features

Deep learning (DL) area recently developed very good feature embeddings for images [6], sound [7], text [START_REF] Pennington | GloVe: Global vectors for word representation[END_REF] and show that it allows impressive applications and facilitate learning downstream tasks. Informative video features have also been developed to describe videos with for example works on activity recognition [START_REF] Vahdani | Deep learning-based action detection in untrimmed videos: A survey[END_REF] where the network is trained to classify videos into a finite set of predetermined activities and thus must learn to represent the video clip in a compact way. Although most of these works proceed at the video level and take into account the appearance, which can be a strong discriminator in certain activities, furthermore as they are trained on a limited set of videos, there is a limit in terms of generalization. The development of general motion descriptors that can be used in any context seems to be an interesting possibility to facilitate downstream motion analysis. It would allow one to compare two different motions or to match an observed motion against a database of known motions.

Developing descriptors from motion is not new, nor is it unique to deep learning. For example, the Laban notation was developed in 1928 to represent the motion of human bodies in dance choreography (see Fig. 3). This system uses symbols to describe the direction of the movement in space, the part of the body performing the movement (each Figure 3 -Example of a partition using the Laban notation system. -Loren Foster, France Culture -"Danse : comment s'écrit le mouvement ?" part of the human body is labeled with a different symbol), the duration of the movement (each line represents a time step, and simultaneous movements are represented on the same line), and the dynamics of the movement. Although many motion notations like this exist, deep learning could help build more general representations that could apply not only to human motion, but to all motion around us. As we discussed earlier, this notation could help in several downstream tasks.

Main objectives of motion analysis

Optical flow is defined as the 2D displacement field that describes the apparent motion of pixels between two successive images [START_REF] Horn | Determining optical flow[END_REF]. Classical approaches typically rely on the Brightness Constancy Assumption (BCA), which assumes that the brightness of a moving pixel remains constant over time. More recently, deep learning approaches have shown the best results in estimating optical flow. Most of these methods are trained in a supervised setting on a synthetic database, and some methods are trained in an unsupervised setting using losses based on image warping.

Motion segmentation techniques aim to partition 2D motion along a video, regardless of the source of each individual motion. Nowadays, one can rely on optimizing the segmentation directly on a given optical flow field, and develop unsupervised deep learning approaches, that learn to segment unseen optical flow fields without relying on human annotation. We propose a taxonomy of motion segmentation techniques in Figure 1.1 of Chapter 1, where we highlight that we can divide methods into intra-video correspondence, trajectory clustering, and instantaneous motion segmentation. This latter category is central to our work.

Video Object Segmentation (VOS) is one of the prominent task in motion analysis. The objective is to segment throughout the video the primary moving object in the foreground of the scene. The VOS methods are evaluated on datasets such as DAVIS2016. In Fig. 4, we show two sequences from DAVIS2016 where we can see that the segmented object is the principal moving object common to all frames in the sequence. Most of the VOS methods combine appearance information and motion information obtained via the optical flow field. Furthermore, we can also mention methods tackling amodal segmentation, a related task where the goal is to segment objects as a whole even when they are partially occluded or when they stop. This task is more challenging as it requires modeling the object shape to predict it when it is occluded. Another interest in studying these methods is that they give an idea of what unsupervised methods can achieve if we find the right loss functions and training procedures.

Finally, we can put forward motion saliency (MS), which is the task of highlighting local motions that deviate from their surrounding context and thus tend to reveal a significant event.

Motivations

Many works have been carried out in the field of motion segmentation and the progress has been really fast these last years. However, there are still challenges to overcome in this task. In this subsection, we highlight the rationale that guided our work and methodology In many works on video analysis, appearance and motion are combined. Several works show that this fusion yields significant improvement on classical video object segmentation benchmarks, justifying the adoption of image features. However, there is an important bias in these datasets: the object to be detected is not only salient by its motion compared to the background, but also by its appearance and its location, the object being generally placed in the center of the images (which is a cue explicitly exploited by several approaches [START_REF] Zhang | Video saliency prediction based on spatial-temporal twostream network[END_REF], [START_REF] Kim | Spatiotemporal saliency detection using textural contrast and its applications[END_REF]).

The fact that most recent works on video analysis combine the two modalities could be a problem, as it limits the use of these algorithms to videos for which the appearance of the object of interest can be easily represented by a neural network. For example, many methods rely on the use of DINO features that have been trained on a large set of natural videos from the Internet, which cannot be applied to other types of videos that represent objects for which we do not have such a database, such as satellite images, biological images, underwater or industrial videos. For example, experimental results in deep learning show that models trained on optical flow have a much better generalization ability than those trained on images. Moreover, focusing only on motion allows interesting transfers that change the context of the videos. We could imagine transferring a motion saliency detection algorithm trained on crowds to the analysis of the motion of groups of cells.

Similarly, supervised training limits the applicability of the methods. First, it requires a large amount of human annotated data, which is expensive to obtain. Second, it constrains the task and can introduce biases. For example, if we take the task of motion segmentation, all databases segment the main object (or the few main objects for FBMS59 and DAVIS2017) in the scene. However, in practice, when we look at a video, we may have many motions to segment that are not segmented in the ground truth. First, we have independent motions of background objects as in the breakdance or scooter-black videos in DAVIS2016, motion due to water as in the blackswann video in DAVIS2016. In a natural environment, we could also mention the movement of trees or clouds when there is wind. Second, we have motion parallax attached to static objects, which is caused by camera movement and depth differences. It can lead to strong flow differences as in the parkour or libby videos in DAVIS2016. Third, we have articulated motion, which could be handled in a more comprehensive way. All these motions give us valuable information about the world around us and are discarded in supervised approaches, because it would be impossible to manually segment them all. Therefore, training motion segmentation in a supervised way only solves part of the problem. We believe that to address motion segmentation, the good approach is unsupervised learning. It is the only one that can lead us to a holistic understanding of the scene motion.

Main Contributions

For the above reasons, we choose to set up a strict framework for this work. We try to identify salient moving objects or segment independently moving objects based only on their motion. This involves a number of methodological decisions. First, we do not rely on appearance or semantic information. Second, we do not take in account given egomotion information or camera intrinsic parameters. Third, our training procedure does not rely on human annotation. This leaves us with the interesting challenge of analyzing optical flow fields that only indicate the motion of individual points of a pair of images.

The contributions of this thesis are organized around two axes of motion analysis.

The first one is concerned with deep learning approaches for unsupervised motion segmentation. In this axis, we build a framework to train a network to segment motion from an optical flow field using a loss function based on the EM algorithm. It decomposes the flow into coherent motion segments, each represented by a parametric motion model. We then gradually extend this framework to longer video sequences. In a first work, we take as input triplets of optical flow fields, representing a total of four frames in the input video. It introduces short-term temporal consistency with a loss term that enforces that labels within the triplets are consistent, and long-term temporal consistency with a post-processing that links triplets together to form a long, temporally consistent segmentation. In a second work, we introduce a spline-based motion representation that is able to represent the evolution of a parametric motion field over time, and we rely on a transformer-based neural network to allow interactions between features of the full sequence. This results in a temporally consistent motion segmentation over the entire sequence in one shot without relying on any post-processing. All these methods have been tested on classical VOS benchmarks (DAVIS2016, FBMS59, SegTrackV2, DAVIS2017), and provide competitive results while being very efficient at test time.

The second axis is the localization of salient motions from the optic flow field. In this part, we articulate our work around the idea that salients areas are those that influence the output of a pre-trained network predicting whether an optic flow field includes salient motions. In this definition, saliency is specific to the task the network is trying to solve. In a first work, we use a gradient-based network interpretation method to localize salient areas, and use the flow to improve the provided segmentation. In a second work, we aim to design an adversarial framework to localize salient areas and inpaint them into the input optical flow, with the goal of altering the output of the classification network. We have considered two different motion saliency tasks, one involving the detection of independent motions in a scene observed by a moving camera, and the other involving the detection of anomalous pedestrian motions in a crowd.

Organisation of the manuscript

There are two main parts to this manuscript after the related work chapter.

The first part, consisting of Chapters 2, 3 and 4, focuses on unsupervised optical flow segmentation using deep networks. In Chapter 2, we describe our method for motion segmentation of the optical flow computed between two consecutive frames. Chapter 3 details our extension to the segmentation of small optical flow volumes, providing temporally consistent segmentation and robustness to outliers. Chapter 4 details the introduction of splines and transformer-based models for temporally consistent segmentation of longer optical flow volumes.

The second part, consisting of Chapters 5 and 6, focuses on the weakly supervised detection of salient motion in optical flow fields. In Chapter 5, we describe our method to localize salient motion using a gradient-based interpretation of a classification network.

In Chapter 6, we present a method for training a segmentation network to localize salient areas using an adversarial approach where we selectively mask part of the input optical flow field.

Finally, we have also investigated the optical flow estimation issue. More specifically, we report in Appendix 7.4 an additional work on the estimation of optical flow for small moving objects in large-scale images, which raises specific problems for optical flow methods based on deep learning.

We give more information on each chapter in the following.

Chapter 1

In Chapter 1. we present related work, focusing primarily on motion segmentation, but also on video object segmentation and motion saliency detection. First, we present methods dealing with video object segmentation. Then, we focus on methods related to motion segmentation, describing a taxonomy of methods where we distinguish methods based on trajectory methods and one based directly on instantaneous motion segmentation. Finally, we introduce methods related to motion saliency, treating both video saliency methods based on appearance and motion features and eye-tracking based saliency methods that learn to imitate the human gaze.

Chapter 2

In Chapter 2, we define a fully unsupervised CNN-based method for segmenting optical flow fields into coherent motion regions. We assume that the input optical flow can be represented as a piecewise set of polynomial models, typically affine or quadratic motion models, each of which characterizing a motion segment (a segment is either a region, i.e., a connected component, or a layer that is not necessarily connected). The core idea of our work is to use the Expectation-Maximization (EM) framework. This allows us to design the loss function and training procedure of our motion segmentation neural network in a well-founded way. In contrast to classical EM, our network, once trained, can provide segmentation for any unseen optic flow field in a single inference step, without dependence on initialization and without estimating any parametric motion models. Different loss functions have been investigated, including robust ones. We have also defined a novel data augmentation scheme dedicated to optical flow, which has a noticeable impact on the network performance. Our motion segmentation network has General Introduction been tested on four benchmarks, DAVIS2016, SegTrackV2, FBMS59, and MoCA, and performed very well while being fast at test time. The EM-Flow software that implements this method is publicly available on Github (https://github.com/Etienne-Meunier-Inria/EM-Flow-Segmentation).

Chapter 3

In Chapter 3, we define a segmentation method for sequences of optical flow fields, introducing temporal consistency to our previously developed work. Temporal consistency is a key feature in motion segmentation, but it is often neglected in unsupervised optical flow segmentation methods. We have proposed an original unsupervised spatio-temporal framework for optical flow motion segmentation that fully explores the temporal dimension of the problem. More specifically, we have defined a 3D network for multiple motion segmentation that takes as input a sub-volume of successive optical flows and returns a corresponding sub-volume of coherent segmentation maps. Our network is trained in a fully unsupervised manner. The loss function combines a flow reconstruction term that incorporates spatio-temporal parametric motion models, and a regularization term that enforces temporal consistency on the masks. We have specified a simple post-processing for long-term temporal linking of predicted segments allowing to have consistent segmentation along long sequences while giving only triplets to the segmentation network. Furthermore, we have designed a flexible and efficient way to encode U-nets. We have performed experiments on several benchmarks with convincing quantitative results. The software implementation called ST-Segmentation is available on github (https://github.com/Etienne-Meunier-Inria/ST-Segmentation).

Chapter 4

In Chapter 4, we introduce spline-based motion models as a representation of motion over a long period of time. We show that the advantage of these motion models is that they can represent a smooth temporal evolution of the parametric motion models. We also introduce a maskformer type architecture [START_REF] Cheng | Per-pixel classification is not all you need for semantic segmentation[END_REF] to segment the optical flow and a temporal attention module that enable interaction between features over the entire sequence. We show that the combination of these two contributions allows us to train a network that can segment long optical flow sequences in an unsupervised manner. Our motion segmentation network has been tested on four benchmarks, DAVIS2016, DAVIS2017, SegTrackV2, FBMS59, and shows competitive results, providing accurate segmentation while maintaining long-term label consistency. It is also very fast, able to process sequences of several hundred frames in one pass at an average of 210 fps. The software implementation called LT-Segmentation is available on github (https://github.com/Etienne-Meunier-Inria/LT-Segmentation).

Chapter 5

In Chapter 5, we introduce a new paradigm for computing salient motion in video frames based solely on optical flow. Motion saliency (MS) is an important issue in dynamic scene analysis, regardless of the application domain (e.g., live cell microscopy, remote sensing, or autonomous navigation). MS aims at highlighting local motions that deviate from their surrounding context, thus tending to reveal significant dynamic events. We have formulated MS as a meta-task that can be instantiated for different tasks that are usually handled independently. To support this claim, we addressed two important general problems with our MS paradigm: the segmentation of independently moving objects and the detection of anomalous motion in videos captured by a mobile camera.

We estimate MS from the interpretation of a frame-based motion saliency classification network with optical flow as input. Our paradigm can accommodate any form of motion saliency by simply training the frame-based classification network on the appropriate task. Our MS estimation is unsupervised as it does not require ground-truth saliency maps for training. We have developed an original two-step network interpretation scheme using the LRP method. It provides the binary salient motion segmentation. Finally, we can recover the valued motion saliency map using parametric flow inpainting. We test our method on two different saliency cases. The first is salient motion produced by independently moving objects in a scene observed by a moving camera. The second case is the salient motion resulting from a distinctive motion within a coherently moving set. Experimental results on real videos and comparison with existing methods demonstrate the performance of our method.

Chapter 6

In Chapter 6, we present a first investigation of an adversarial approach for learning to segment regions of salient motion in optic flow fields based on the prediction of a classification network. The overall goal is to fuse the segmentation and interpretation steps to directly output the salient motion segmentation from the optic flow field. Instead of relying on gradient-based interpretation techniques such as LRP, we train a segmentation network to locate areas that, when modified (or perturbed), change the prediction of the classification network. We also show that we can formulate it as a constrained optimization task and use a Lagrangian-based approach to automatically compute a critical hyperparameter, the weight between the terms of our network loss. We report preliminary results, mainly on the second motion saliency task, e.g., salient motion resulting from a distinctive motion within a coherently moving set.

Chapter 1

RELATED WORK

In this chapter, we review work related to motion segmentation (in a broad sense) in videos. The first part of the chapter focuses on Video Object Segmentation (VOS) that aims to detect the main moving object in videos. The second section introduces more general motion segmentation methods that deal with the partitioning of the 2D motion along a video independently of the source of each motion. We will give a more important focus to this section as it is the most related to our own research. The third section describes methods that deal with the detection of salient motion in videos, including both methods based on motion characteristics and methods that train a neural network to predict human attention from eye tracking data.

Video Object Segmentation (VOS)

Combining motion and appearance

The focus of Video Object Segmentation (VOS) is on segmenting primary objects (typically a single one) that move in the foreground of a scene and are usually followed by the camera. VOS provides a binary segmentation, primary object versus background [START_REF] Zhou | A survey on deep learning technique for video segmentation[END_REF]. However, the background may also contain moving objects, as in some videos from the DAVIS2016 dataset [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF]. The availability of large annotated VOS datasets makes it possible to apply supervised deep learning techniques to VOS.

Part , Chapter 1 -Related Work

In the context of several VOS benchmarks, the term "unsupervised" is not used in the same way as in the deep learning community. For these benchmarks, it refers to the fact that no annotation is given at test time, on the contrary with the "semi-supervised" category, where the first frame of each test sequence is annotated with the primary moving object. For the deep learning community, "unsupervised" rather means that no annotated data were used for training, and "supervised" means that annotations were used for training, although in both cases they do not imply the use of annotations for test sequences. In this section, we only mention methods that would be qualified as "unsupervised" in the sense of VOS. Thus, we use the deep learning vocabulary to distinguish between methods trained on annotated data ("supervised") and the other methods ("unsupervised").

In [START_REF] Tokmakov | Learning motion patterns in videos[END_REF], the authors trained a convolutional network to segment optical flow fields. The network is trained in a supervised manner on synthetic data and provides segmentation from the motion field only. This segmentation is later refined using pre-trained appearance features to account for parallax motion and errors in the optical flow field.

Subsequently, several works have shown that the joint use of object appearance and motion improves performance in VOS. In [START_REF] Cheng | SegFlow: Joint learning for video object segmentation and optical flow[END_REF], the authors train a two-branch network to predict an optical flow field estimation together with a segmentation from a pair of frames. They show that this setting yields better results than treating the two problems independently as learned features, and that it benefits both tasks. In the same vein, several works combine a pre-trained optical flow with an image to perform VOS segmentation. In [START_REF] Jain | FusionSeg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos[END_REF], the authors train a module that fuses deep features of two convolutional networks, each taking as input one of the modalities, using a custom fusion module that takes into account the confidence of the two different streams. In another paper, MATNet [START_REF] Zhou | MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation[END_REF] offers an alternative to this fusion process, using an attention-based architecture that allows a better interaction between these two modalities. Finally, in [START_REF] Dave | Towards segmenting anything that moves[END_REF], the authors combine the two modalities to build the feature map of a mask R-CNN model used for both segmentation and bounding box prediction.

Video Object Segmentation (VOS)

Recent works have revisited the way of coupling appearance and motion. The AMD method [5] includes two pathways, the appearance one and the motion one, that are then combined to predict segment flow and separate moving objects from background. It does not use optical flow as input and brings out the objectness concept. However, it relies on a coarse motion representation. The RCF method described in [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF] involves learnable motion models and is structured in two stages: first, a motion-supervised object discovery stage, then, a refinement stage with residual motion prediction and high-level appearance supervision. However, the method does not impose temporal consistency and cannot distinguish objects undergoing different motions. In a different approach [3], the prediction of probable motion patterns is used at the training stage as a cue to learn objectness from videos.

Exploiting the temporal dimension

It can be beneficial to exploit the intrinsic temporal nature of a video sequences to improve segmentation. In [START_REF] Tokmakov | Learning motion patterns in videos[END_REF], the authors combine the features extracted from convolutional networks applied to the frames and the optical flow fields along a sequence using a recurrent module. The features provided by this recurrent module are then used directly for segmentation. A similar recurrent module is used in [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF] which, unlike this previous work, only takes the optical flows as input to focus on motion cues for their segmentation. In [START_REF] Song | Pyramid dilated deeper ConvLSTM for video salient object detection[END_REF], the authors provide a framework for segmenting a sequence of images, combining a convolutional network and a recurrent network to take into account features from longer sequences. They focus on the architectural aspect of the network by introducing a multi-scale feature extraction module based on extended convolutions. Another work, [START_REF] Lu | See more, know more: Unsupervised video object segmentation with co-attention siamese networks[END_REF], trains a co-segmentation module to segment the object common to a pair of frames using a correlation volume. At test time, they use this co-segmentation module to obtain the segmentation for the whole sequence from a set of reference frames, thus, introducing a temporal dimension to their method.

Amodal segmentation

Several papers have introduced the use of supervised training and temporal networks for amodal segmentation, which according to [START_REF] Lamdouar | Segmenting invisible moving objects[END_REF] "refers to the task of segmenting the object as whole, including the portions that are partially occluded". In the case of motion segmentation, this may involve cases where an object is occluded or static for a short Part , Chapter 1 -Related Work period of time. In [START_REF] Lamdouar | Segmenting invisible moving objects[END_REF], the authors use a synthetic database to train a transformer model that takes a sequence of optical flow fields as input and returns both amodal and modal segmentation. Another paper [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] uses a similar protocol, although they only predict the amodal segmentation and the ordering between layers, and combine these to form the modal segmentation, taking into account occlusions. Both of these works include a temporal dimension, as they segment a sequence of optical flow fields in one go.

Unsupervised methods

Unsupervised VOS methods have also been developed. In [START_REF] Koh | Primary object segmentation in videos based on region augmentation and reduction[END_REF], the authors exploit the recurrence property of the primary moving objects to select regions from a set of candidates extracted using colours and motion edges. This recurrence property is based on the fact that, in the case of VOS, the region of interest appears many times throughout the sequence, unlike background motion, which only appears for a small number of frames. In another work [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF], each segment of the sequence is represented by a set of binary masks, an appearance model in the form of a stripe, and a spatio-temporal transformation that captures the deformation of the appearance along the sequence. These three elements are then optimised for a set of segments in order to reconstruct the input video. The obtained masks can be viewed as object segmentation, taking into account both appearance and motion. Both methods require an optimisation on the input sequences at inference time, which takes time compared to the previously mentioned approach where the network is trained beforehand.

Motion Segmentation

In this section, we focus on motion segmentation, which is the task of partitioning 2D motion along a video, regardless of the source of each individual motion. It is then a "pure" multi-label segmentation problem based on the estimated image motion.

Motion segmentation has received considerable attention over several decades. Seminal works are based on clustering framework or Markov Random Fields (MRF) to perform image motion segmentation into layers [START_REF] Ayer | Layered representation of motion video using robust maximum-likelihood estimation of mixture models and MDL encoding[END_REF], [START_REF] Wang | Representing moving images with layers[END_REF] as well as into connected regions [START_REF] Bouthemy | Motion segmentation and qualitative dynamic scene analysis from an image sequence[END_REF], [START_REF] Odobez | MRF-Based motion segmentation exploiting a 2D motion model robust estimation[END_REF]. The advent of deep learning and the availability of efficient optical flow methods has recently led to new categories of methods. Most of these methods take as input the optical flow computed between two successive images and train neural networks to produce 1.2. Motion Segmentation a segmentation. In addition, some methods couple the input optical flow with image data to provide more reliable segmentation, while others exploit temporal relationships through trajectories or recurrent structures.

The motion segmentation task is related to the previously described Video Object Segmentation (VOS) task, but differs in that they do not focus on extracting a dominant "foreground object" that occupies a central place in the video. Thus, these techniques focus primarily on the motion information rather than the image or semantic characteristics of the frames that compose the sequence.

In addition, it is difficult to build a large labelled training database for motion segmentation, as it would require precise annotation of the different motions in the sequence, which is not trivial in some cases (e.g., articulated motion). Deep learning techniques focused on this task have to come up with new ideas to train a network in an unsupervised way, leading to a wide variety of approaches. In the DiVA paper [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF], it is qualified as a good task to "explore the limits of fully unsupervised object detection".

Given the large body of work in this area, we will not attempt an exhaustive review. Instead, we focus on a handful of methods that we consider representative of the field and that have most influenced our research. We have constructed the taxonomy shown in table 1.1, which highlights the different approaches investigated in the litterature, and we will follow the structure of this taxonomy for this section.

We deliberately do not base the taxonomy on the obvious distinction between learning and optimisation-based approaches, as this would have the side-effect of separating past and recent methods. In contrast, we intend to highlight the methodological similarities or differences between all methods, and show how the seminal approaches inspired the current approaches to formulate motion segmentation.

Instantaneous motion segmentation

Our first focus is on methods that rely on the instantaneous motion between two successive frames of the video. Most of these methods belong to the category of motion decomposition as they produce a partition of this instantaneous motion into coherent segments. They use as input the pair of frames or a pre-computed optical flow field that represents the dense point-to-point correspondance field between the successive frames. In this category we distinguish three types of approaches.

✓ ✓ ✓ [4] ✓ ✓ [37] ✓ [3] ✓ ✓ ✓ [5] ✓ ✓ [29] ✓ Compensation of camera motion [38] ✓ [39] ✓ [40] ✓ [41] ✓ [42] ✓ ✓ [43] ✓ [44]
Non parametric approaches [START_REF] Weiss | Smoothness in layers: Motion segmentation using nonparametric mixture estimation[END_REF] Deep generative models

[46] ✓ ✓ [47] ✓ ✓ [48] ✓ ✓ ✓ [49] ✓ ✓ [21] ✓ ✓ ✓ Trajectory Clustering [50] ✓ ✓ [51] ✓ [52] ✓ [53] ✓ ✓ ✓ [54] ✓ Table 1.
1 -Taxonomy of Motion Segmentation Methods. "OF" denotes methods using a precomputed optical flow field as input, "AF" denotes methods using pretrained appearance features, "Te" represents methods taking into account long temporal context, "DL" methods using deep learning models.

Use of 2D parametric motion models

In this first category, the methods represent the motion within each segment with a parametric motion model, and usually with a polynomial motion model that is related to the projection of the 3D motion in the viewed scene onto the 2D image [START_REF] Bouthemy | Motion segmentation and qualitative dynamic scene analysis from an image sequence[END_REF].

Following the rigid-body dynamics, we can define the motion of any point P on the surface of a rigid body wit P = (X, Y, Z) T in the coordinate sytem attached to the camera and centered in 0, given the instantaneous translation vector ⃗ T = (U, V, W ) T and the rotation vector ⃗ R = (A, B, C) T specifying the 3D rigid motion as :

⃗ V = ⃗ T + ⃗ R × ⃗ OP (1.1)
Using this definition and a perspective projection model providing x = f X Z and y = f Y Z , we can retrieve the 2D velocity vector w(p) = (u(p), v(p)) T of the projection p = (x, y) T , into the image plane, of the 3D point P as :

u(p) ≜ dx dt = f U Z -x W Z -A xy f + B( x 2 f + f ) -Cy, v(p) ≜ dy dt = f V Z -y W Z -A( y 2 f + f ) + B xy f + Cx. (1.2)
Those equations link the 3D motion of a point to the 2D motion of its projection in the 2D image plane. It involves the 3D rigid motion between the object and the camera and the depth of the point. In practice, Z can be different for each point. The difference in the apparent motion of static objects in the foreground of the scene due to the camera motion and the difference in depth is called "motion parallax". Those equations inspired the use of parametric motion models describing the 2D motion in the image sequence. A specific 2D motion model is the 8-parameter quadratic model corresponding to the projection of a rigid motion of a planar surface :

u θ (p) = θ 1 + θ 2 x + θ 3 y + θ 7 x 2 + θ 8 xy, v θ (p) = θ 4 + θ 5 x + θ 6 y + θ 8 xy + θ 7 y 2 . (1.3)

Parametric Motion Models

Part , Chapter 1 -Related Work Some recent methods [3], [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF] use a full 12-parameter quadratic motion model where the two components of the motion model are independent. It can better fit complex motion and account for non planar (but still continuous) surface. The affine (6-parameter) motion model corresponding to a polynomial of degree 1 for both components is also often used.

The authors in [START_REF] Wang | Representing moving images with layers[END_REF] introduced a clustering approach that splits the motion between two images using affine motion models. They first estimate the optical flow field between frames using a local variational approach, and then build an iterative algorithm that alternates between estimating an affine motion model segment by segment and assigning pixels to one of the computed segments. This work inspired several recent deep learning efforts to train a network to perform layered motion segmentation based on parametric motion models.

A pioneering work [4] on the use of loss based on parametric motion models to train a neural network for segmentation was, ironically, not carried out in the objective of segmenting a video, but in the very active field of research of self-supervised feature learning for image segmentation. In fact, in this work, the authors relyed on the Gestalt principle of common destiny to advocate that the pixels moving in a coherent way are likely to belong to the same object. They have pre-trained the feature extractor of a static image segmentation module using unlabelled videos and a criterion that evaluates the segmentation based on the fitting error of an affine motion model to the flow of each given segment. They manage to show that the features obtained are a good pre-training step for segmentation. However, since they use a least-square computation for the motion model fitting, their method is sensitive to noise and outliers in the optical flow estimation, which affects the quality of the result obtained. It leads them to use a more basic loss based on the entropy of the histogram of the flow magnitude within each segment. Several recent works have followed this lead. In [5], the authors train a network to segment an image from a video frame with a loss function based on the decomposition of the motion into piecewise constant motion models. They also learn to predict the optical flow field for each segment from a reconstruction loss. Later, in [3], the authors go further in this direction, using a quadratic motion model, a pre-trained optical flow from RAFT [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF], and a pre-trained appearance feature extractor from DINO [6]. It greatly improves the segmentation results, but complicates the evaluation of the impact of self-monitoring to train feature extraction networks. The authors also presented a probabilistic approach to their work in [START_REF] Karazija | Unsupervised multi-object segmentation by predicting probable motion patterns[END_REF], using affine motion models and testing on the MoVi [START_REF] Ghorbani | MoVi: A Large Multipurpose Motion and Video Dataset[END_REF] and KITTI [START_REF] Geiger | Are we ready for autonomous driving? The KITTI vision benchmark suite[END_REF] datasets.

Motion Segmentation

Using the same criterion for learning, several methods directly segment the motion between two frames. Our motion segmentation method [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF] that we will detail in Chapter 2, belongs to this category as it trains a network to segment optical flow fields using a loss function based on parametric motion models. In [START_REF] Shrestha | Learning To Segment Dominant Object Motion From Watching Videos[END_REF] the authors set up an unsupervised approach to segment motion into dominant and background motion using affine motion models from a pair of frames. Like [5], their network takes two consecutive frames as input and predicts both the motion segmentation and affine flow models for each layer based on image warping and a photometric loss function. The main differences with [5] are that their segmentation is based on both frames instead of just one of the frames and that they use affine motion models instead of constant piecewise models. This approach is interesting as the authors do not rely on precomputed optic flow for training, but they only provide results on binary segmentation of simple translational motion (moving car dataset). Therefore, it is hard to evaluate if the feature encoder is able to represent more complex appearance patterns.

Explicit spatial prior

Related methods combine the use of parametric models with prior conditioning of the output segmentation. In [START_REF] Bouthemy | Motion segmentation and qualitative dynamic scene analysis from an image sequence[END_REF]- [START_REF] Weiss | A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models[END_REF], the authors present methods for segmenting the motion between two successive images based on parametric motion models and a spatial constraint that forces neighbouring pixels to have the same labels using Markov Random Fields. The segmentation involves an iterative algorithm that alternates between estimating the parametric motion models and performing motion segmentation.

Part , Chapter 1 -Related Work

In the Bayesian framework, the probability of a segmentation is decomposed into its likelihood (i.e., data fitting term) and prior terms. The data fitting term U quantifies how well the motion models attached to the segmentation x align with the observed input data y. The prior V c assumes that x is a 2D Markov Random Field (MRF). Due to the equivalence between MRF and Gibbs distribution, the prior can be defined locally on cliques c ∈ C that consists of spatially inter-connected pixels. It allows for example to promote smoothness in the output segmentation. A common prior for segmentation is the Potts model that forces neighbouring sites to have the same label. A common criterion is the Maximum a posteriori (MAP), leading to the minimization of the negative loglikelihood of the segmentation probability, which is expressed using the Bayes rule as :

-log p(x|y) = -log p(y|x) -log p(x) + cst. = i∈Ω U (x i , y i ) + c∈C V c (x c ) + cst. (1.4)
Ω is the set of sites in the image grid. Authors then employ optimisation procedures (e.g., Simulated annealing, Iterative Conditional Modes, Graph-Cut) to minimise this negative log-likelihood and obtain a segmentation for a given input data y.

Markov Random Field and Bayesian Approach

In [START_REF] Perez-Rua | Hierarchical motion decomposition for dynamic scene parsing[END_REF] the authors introduce a richer prior with the idea of a hierarchical segmentation of motion, linking each segment to a parent motion, each child node has a parametric motion model that fits only the residual motion. This leads to a finer representation of motion, which can be useful in applications that link parts moving together. While these priors on segmentation have shown to be an asset in traditional approaches, it is still a challenge to apply these constraints in deep learning methods. Indeed, difficulty comes up with a differentiable version of the compactness constraints. To our knowledge, there are no methods that include them explicitly during the training procedure.

Another way to include a spatial constraint on the prediction is to predict the bound-

Motion Segmentation

aries of the regions, thus forcing them to be enclosed and with a regular shape. In [START_REF] Cremers | Motion competition: A variational approach to piecewise parametric motion segmentation[END_REF], the authors introduce a variational approach to segmenting instantaneous motion from a pair of frames by predicting the boundary between regions. Their framework combines a likelihood probability that evaluates the coherence of a segmentation using constant or affine motion models, and a prior that minimizes the length of the boundaries between motion segments. Their likelihood function is based on a criterion derived from the brightness constancy equation and the representation of motion in terms of parametric models, which is expressed as :

p(w|∇ 3 I) ∝ p(w) p∈Ω p(∇ 3 I(p)|w(p)) ∝ e -γL(C) p∈Ω e -cos 2 (α(p)) , ( 1.5)
where Ω ⊂ R 2 denotes the image grid, w(p) is the motion vector at point p, here in homogeneous coordinates, ∇ 3 I(p) is a vector stacking the spatial and temporal intensity gradients at point p. The likelihood term includes cos(α(p)), which is the angle between ∇ 3 I and w at point p and represents the errors in the optical flow constraint at that point. The prior term includes L(C), the length of the boundary C between the computed regions weighted by parameter γ.

By representing the flow within each region using a parametric motion model, they further decompose the likelihood as :

p∈Ω p(∇ 3 I(p)|w(p)) = n i=1 p∈Ω i p(∇ 3 I(p)|w i ), (1.6)
where w i is the motion model (affine or constant in this case) representing the motion within region i and Ω i is the set of sites belonging to that region, and n the number of regions.

The authors represent the boundaries between regions using either splines or level sets, allowing them to minimize L(C) using gradient descent.

One of the advantages of this method is that it produces regular shaped regions, since it directly models the boundaries. Also, the fact that they combine motion estimation with segmentation alleviates the need to use pre-computed optical flow fields. Some limitations of the method are related to the use of the brightness constancy equation, which restricts the method to the segmentation of small motions and to images where the brightness of each point is constant over time. Other limitations are related to the initialization of the region boundaries, as it can affect results. [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF] are shown on the bottom right. From left to right: two flows from our outdoor videos, one flow from a parkour sequence in Davis. In the left sequence, the parallax motion is the apparent motion of the tree trunk, in the center sequence the apparent motion of the pole, and in the right sequence, it is the motion of the fence.

In [START_REF] Vázquez | Joint multiregion segmentation and parametric estimation of image motion by basis function representation and level set evolution[END_REF], the authors use a framework similar to [START_REF] Cremers | Motion competition: A variational approach to piecewise parametric motion segmentation[END_REF] in that they rely on level sets and parametric motion models to represent the motion within each segment. In addition, they introduce a term that enforces that the motion boundaries coincide with spatial flow gradients.

Compensation of camera motion

Another perspective is to focus on the ability to detect independent moving objects in the scene. As we explained earlier, in the case of a mobile camera, the apparent motion of objects depends on their depth, so that static objects close to the camera can have an apparent motion that is different from the background motion. Consequently, these objects are segmented as individual moving objects using the methods above in subsection 1.2.1.

Several methods tried to address this problem and segment only objects that are really moving in the scene. A reference method in this field is [START_REF] Ranjan | Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation[END_REF], where the authors jointly trained four convolutional networks that take as input a set of images and independently provide estimates of the depth map of the scene, the camera motion between frames, the dense optical flow field, and the segmentation of the moving objects. Then, by combining the depth map and the estimated camera motion they compute the scene flow, representing the apparent motion of the whole static scene. The gap between the static scene flow and the dense optical flow should be high in the regions corresponding to independent moving objects. The four networks are trained using a process called "competitive collaboration" and give good results on the KITTI dataset [START_REF] Geiger | Are we ready for autonomous driving? The KITTI vision benchmark suite[END_REF]. However, in addition to the availability of the camera intrinsic parameters, this method requires an accurate estimation of the scene depth map and the camera pose at the same time, which can be difficult to obtain Figure 1.2 -Framework of the competitive collaboration process presented in [START_REF] Ranjan | Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation[END_REF]. Four networks taking either a frame or a sequence of frames as input are jointly trained. The "E" step represents the combination of the different modalities to form a reconstructed optical flow which is evaluated on the input sequence using photometric loss. Reproduced from [START_REF] Ranjan | Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation[END_REF]. [START_REF] Zhao | Monocular Depth Estimation Based On Deep Learning: An Overview[END_REF]. To our knowledge, there is no application of this methods in natural scenes shot with a handheld camera such as in the DAVIS2016 dataset. It makes it uncertain whether this method could adapt to different intrinsic camera parameters and more complicated camera movements than translation. Furthermore, there are many hyperparameters in this method and no simple approach to find them.

Compensation of the camera rotation

Another interesting approach is presented in [START_REF] Narayana | Coherent motion segmentation in moving camera videos using optical flow orientations[END_REF]- [START_REF] Bideau | The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields[END_REF]. The aim is to ignore parallax motion without having to estimate the full depth map. To do this, they compute the 3D rotational motion of the camera using a robust estimation method. Once the camera rotation estimated, they show that the angular field of the translational motion is independent of the depth of the object, and then, they can segment this flow field without having to worry about parallax motion.

Part , Chapter 1 -Related Work

We can decompose the 2D flow field ⃗ w as the sum of a translational flow field ⃗ w t and a rotational flow field ⃗ w r such that :

⃗ w = 1 Z   f U -xW f V -yW   ⃗ wt +   -A xy f + B( x 2 f + f ) -Cy -A( y 2 f + f ) + B xy f + Cx   ⃗ wr (1.7)
Thus, we can observe that the orientation κ t of the translational field is independent of Z :

κ t = arctan( f V -yW f U -xW ). (1.8)
Several works (e.g., [START_REF] Narayana | Coherent motion segmentation in moving camera videos using optical flow orientations[END_REF], [START_REF] Bideau | It's moving! A probabilistic model for causal motion segmentation in moving camera videos[END_REF]) compensate the rotational component of the flow and segment the orientation field of the resulting translational flow to get a segmentation not affected by parallax motion.

Compensate Rotational Motion

They investigated this perspective first using classical probabilistic models for segmentation and a robust approach for camera rotation estimation [START_REF] Narayana | Coherent motion segmentation in moving camera videos using optical flow orientations[END_REF], [START_REF] Bideau | It's moving! A probabilistic model for causal motion segmentation in moving camera videos[END_REF]. Then, they introduced neural networks into the method. A pre-trained network allows them to segment the rotation-compensated flow, then, they train a network to perform segmentation using supervised training on synthesised sequences with different motions [START_REF] Bideau | The best of both worlds: Combining CNNs and geometric constraints for hierarchical motion segmentation[END_REF]. Finally in [START_REF] Bideau | The Right Spin: Learning Object Motion from Rotation-Compensated Flow Fields[END_REF], they combine rotation estimation with a supervised convolutional network for segmentation and show results on real data. This line of research shows the usefulness of estimating camera motion to improve segmentation, although they still need to have the intrinsic parameters of the camera for the projection model, to apply this technique in practice. Furthermore, it seems intuitive that rotation compensation is a critical step for unsupervised segmentation, as it is hard to find objective criteria allowing to distinguish motion parallax from independent motion. However, in the context of supervised learning, where independent motions are clearly annotated in the ground truth, one could wonder why the network would need a rotation-compensated flow field, since it could learn to achieve the rotation compensation internally.

3D-consistency

It is also possible to detect moving objects in 3D space using a rigidity constraint based on projective geometry. In [START_REF] Irani | Parallax geometry of pairs of points for 3D scene analysis[END_REF], the authors present several approaches for detecting moving objects based on plane and parallax decomposition. The novelty is that they derive a geometric relationship between pairs of points in several images, which makes it possible to check whether these points are moving like rigid objects, even if they are at different depths, thus allowing to detect areas that are moving independently without being affected by parallax motion. In another approach [START_REF] Csurka | Direct identification of moving objects and background from 2D motion models[END_REF], the authors first decompose the motion into segments and then use rigidity constraints and compatibility tests based on the motion characterization of each segment to distinguish foreground from background motion. While these approaches provide valuable insights into the detection of independently moving objects, they have certain limitations. Like previous works that compensate for camera rotation, they rely on a good estimate of camera extrinsics, which can be difficult to compute in cases where there are many moving objects. It also requires knowledge of the camera intrinsic parameters.

Deep generative motion model

A drawback of the parametric approach is the limited representativeness of these models. We have presented techniques above that address this issue by representing motion associated with variation in the depth field, but other types of motion, such as articulated motion, remain a challenge. In MoSeg [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], the authors argue that this lack of flexibility of classical motion models affects the performance of segmentation methods based on them. In this context, they build a layered segmentation method similar to the one presented above, in the sense that they try to reconstruct the input flow using a layered representation, but with the crucial difference that the flow specific to each layer is freely generated by a deep learning decoder, instead of being induced by a set of motion model parameters. They focus on the binary segmentation of the optical flow, which consists in labelling the pixels as foreground moving objects or background, and demonstrate good results on classical VOS datasets.

However, this class of region-based approaches faces a structural drawback: the tradeoff between model expressiveness and segmentation quality. Indeed, the more expressive the generative model, the less relevant the segmentation. In the case of an extremely rich generative model, the entire input stream could be represented in a single class, making segmentation unnecessary and thus defeating the training strategy. Conversely, an overly simple model can only model the flow within small segments, leading to very fragmented Figure 1.3 -Framework of MoSeg [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. Slot attention module is applied on the feature extracted from the input flow field. Extracted slots are then given as input to a decoder network to produce the flow reconstruction and the segmentation masks. Reproduced from [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF] segmentation.

An alternative to this problem has been proposed under the name of Contextual Information Separation (CIS) [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF]. The authors build an adversarial network where the generative network tries to predict the optical flow inside the mask using only information outside it. This way, even with a rich class of generative model, it is impossible to achieve it if the movements inside and outside the mask are completely unrelated.

Motion Segmentation

In [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], the authors make a distinction between region-based segmentation criteria used in the classical literature, where one tries to reconstruct the original flow using decomposition and a parametric model for each segment [START_REF] Chan | Active contours without edges[END_REF], [START_REF] Mumford | Optimal approximations by piecewise smooth functions and associated variational problems[END_REF]. The aim of their approach is the prediction of the flow field inside each region from the flow field outside it. The authors introduce two mathematical formulations as follows:

Classical region-based segmentation

Ω f |w in (p) -ϕ(p, w in )| 2 dp + Ω b |w out (p) -ϕ(p, w out )| 2 dp, (1.9) Contextual information separation Ω f |w in (p) -ϕ(p, w out )| 2 dp + Ω b |w out (p) -ϕ(p, w in )| 2 dp. (1.10)
Here, Ω f represents the foreground region, w in represents the flow within that region (Ω b and w out for the background, respectively). The generative model is ϕ, which can take various form. Within the region-based framework, ϕ is a constant value of each segment in [START_REF] Chan | Active contours without edges[END_REF], it is a smooth function in the Mumford-Shah functional [START_REF] Mumford | Optimal approximations by piecewise smooth functions and associated variational problems[END_REF], a parametric motion model in [3], [START_REF] Karazija | Unsupervised multi-object segmentation by predicting probable motion patterns[END_REF], [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF] and the output of a generative network in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. Within the contextual information separation framework, it is a convolutional network in [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] and a generative network in [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF].

Contextual Information Separation

Let us note that a limitation of this approach is the difficulty to hide part of the optical flow. This is because a probabilistic segmentation mask is involved during training. The CIS method showed good results on binary segmentation of the optical flow field on VOS benchmarks. It was then extended first to the DyStab [START_REF] Yang | DyStaB: Unsupervised object segmentation via dynamic-static bootstrapping[END_REF] method, which in addition uses appearance for segmentation, and the DiVA method [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF], dealing with multiple motion segmentation and relying on the slot attention architecture introduced in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. Another recent work [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF] explores a hybrid approach, combining a very simple (piecewise constant) parametric model with an estimation of the residual flow for each segment, [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] with two cases of execution. Reproduced from [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF].

which represents the reconstruction error between the parametric and the dense optical flow. In this model, they do not control the complexity of the residual flow, but it is just restrict its values to be within [-λ, λ], thus avoiding the residual flow to directly output the target optical flow field. However, the setting of λ is critical and depends on the input data. The authors do not provide a way to easily tune this parameter. Their approach additionally relies on image appearance information with an appearance refinement stage. In addition, the network is trained to predict a cleaner segmentation based on a Conditional Random Field (CRF) and a semantic constraint loss using pre-trained appearance features.

Non-parametric approaches

Non-parametric approaches to motion segmentation base their segmentation on criteria that are not described by parametric motion models. In [START_REF] Weiss | Smoothness in layers: Motion segmentation using nonparametric mixture estimation[END_REF], the authors construct a functional that evaluates the smoothness of the motion in each slice, as measured by the optical flow gradients within the slice. The functional is minimized using an EM algorithm to compute the segmentation along with the flow field while satisfying the smoothness constraint. Green's functions are used to make the computation of the maximization steps efficient.

to learn an appearance representation from an image and to regularise a segmentation by filling the gap between sparse labels. They also introduce a Siamese GRU network to compute the distance between trajectories. The disadvantage is that the network is trained independently on each sequence, which makes the method very slow (30 min/sequence on a GPU). They show that both the GRU and the U-Net improve the segmentation results, demonstrating the benefit of training the modules of this approach. Another method [START_REF] Xu | Learning Clustering for Motion Segmentation[END_REF] sets up a dense network to learn the trajectory representation in the embedding space, where the motion segmentation problem is solved at inference by applying K-means to the output embeddings. By training a loss that predicts the distance between two trajectories, in contrast to directly predicting the segmentation map, they open the door to predicting a variable number of clusters during the K-means step. However, since the ground truth does not involve hierarchical information, one cannot directly expect the network to learn by itself to produce a coherent motion partition with a variable number of segments.

Comparison between learning-based and optimisation-based approaches

along the training. An important drawback of learning-based methods is the difficulty of generalising to data outside the training distribution. In practice, a trained network is often unable to correctly process inputs that are too different from the training data, or to deal with challenges that are not present in the training set. This has a direct impact on the unsupervised image segmentation methods presented above, as they often struggle to adapt to unseen objects, relying on pre-trained image features or test time adaptation to achieve good results on the test set.

Motion Saliency

Motion saliency (MS) aims to highlight local motions departing from their surrounding context, thus prone to reveal a significant event. MS has many applications in computer vision. It can be useful in the navigation of autonomous vehicles to anticipate moving obstacles, or for public safety to trigger alert in abnormal situations. It can facilitate the subsequent analysis of videos where motion may play the key discriminative role.

MS estimation can be formulated as a meta-task that can encompass different tasks.

Video saliency

Video saliency is by definition object-oriented and therefore mixes appearance and motion features.

Previous approaches exploited the fact that salient regions have different low-level appearance and motion features (also known as cues) from neighbouring regions. The authors of [START_REF] Kim | Spatiotemporal saliency detection using textural contrast and its applications[END_REF] first divide the image into regions using hierarchical segmentation, and then rely on descriptive histograms for each region by combining motion and appearance cues. Regions with a large difference from their neighbours are labelled as salient. The pipeline of this method is presented in Fig. 1.5. In [START_REF] Mahapatra | Coherency based spatio-temporal saliency detection for video object segmentation[END_REF], the authors exploit a criterion to measure the coherence of regions using the entropy of the histogram of gradients (HoG) of the luminance channel and the entropy of the histogram of the flow magnitude. In this work, a region with coherent motion, low spatial luminance gradient entropy and greater motion than its neighbours is termed salient.

Other works introduce graphs with distance measures based on appearance and motion indices to perform segmentation. In [START_REF] Liu | Superpixel-based spatiotemporal saliency detection[END_REF], the authors decompose the video into temporal superpixels and then construct a graph using metrics based on appearance and motion cues. Then, they apply a spectral foreground detection method to this graph in order to identify the salient region. In [START_REF] Hu | Unsupervised video object segmentation using motion saliency-guided spatio-temporal propagation[END_REF], the authors first identify the initial salient region under the assumption that the background region is connected to the image boundary. Then, they use a graph to diffuse saliency labels based on pre-calculated image edges, an inter-frame term based on optical flow and a term based on visual similarity between superpixels.

In [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF], the authors assume that salient objects have different low-level appearance and motion features (orientation and magnitude for flow vectors) than the background. They use a Tukey-inspired measure to detect outlier pixels in the images and label them as belonging to moving objects. They also introduced methods based on pre-trained semantic features that improve segmentation.

The methods mentioned above rely on the extraction of meaningful motion features and on assumptions inherent to the data set. For example, they rely on the fact that motion saliency is often correlated with appearance saliency, or that the salient object often has a central position in the image. These assumptions are true for the VOS dataset, but could be not verified for other types of data. For example, the MoCA [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF] dataset presents sequences where the labeled objects exhibit salient motion but have an appearance similar to the background.

Motion Saliency

In [START_REF] Georgescu | Anomaly detection in video via self-supervised and multi-task learning[END_REF], the authors also rely on appearance and motion cues for motion saliency. In this paper, they formulate the problem as anomaly detection and train a 3D convolutional network on image sequences. The network is trained in a self-supervised manner using pretext tasks with the goal of developing discriminative anomaly-specific features. The first two tasks deal with the consistency of the temporal evolution of the video. The first one consists in guessing if the sequence is played forward or backward, and the second one is to detect if there is an irregularity in the motion of the sequence. The third task is to evaluate if the network is able to reconstruct the central frame of the sequence. Finally, a knowledge distillation task based on pre-trained image features allows them to include appearance in the training process. At inference time, the salient frames are defined as those where the network fails to perform these tasks. Several methods related to the used of deep learning for anomaly detection are described in [START_REF] Chalapathy | Deep learning for anomaly detection: A survey[END_REF].

Eye-tracking based saliency

Another way to measure saliency is to use eye-tracking data. By measuring the positions where a human gazes while watching a video, one can retrieve a region of interest that is likely to contain the salient object. This metric was originally used for evaluation, but with the advent of deep learning, several works introduced methods to train neural networks to predict these gaze patterns and thus localize salient regions. We will have a look at some of these approaches here, a more complete overview is available in [START_REF] Borji | Saliency prediction in the deep learning era: Successes and limitations[END_REF].

In [START_REF] Jiang | Predicting video saliency with object-to-motion CNN and two-layer convolutional LSTM[END_REF], the authors introduce a large eye tracking database with videos that present a variety of content. Using this database, they train a deep learning approach combining a convolutional network and a recurrent network (convLSTM) to learn to predict saliency maps from input images in a supervised manner using eye-tracking data as ground-truth labels. In [START_REF] Zhang | Video saliency prediction based on spatial-temporal twostream network[END_REF], the authors also leverage eye-tracking data for training, but they introduce a two-stream network that relies on 3D convolutions to detect salient areas on a sequence of images. By exploiting this two-stream architecture, they have a lighter network that requires less video gaze data to train. In addition, they incorporate a positional prior to the convolutional network for segmentation, pushing the output to be concentrated around the center of the image using Convolutional Gaussian Priors layers. Inspired by previous work on saliency prediction, they use a combination of three different saliency metrics including the ground truth density map and the ground truth binary fixation map for the training loss.

Other works show that it is advantageous to train the gaze prediction network and Figure 1.6 -Overall framework of [START_REF] Fan | Shifting more attention to video salient object detection[END_REF], the image are first processed using a convolutional network with pyramid dilated convolution, then with a LSTM network. On the right, the segmentation and human eye-tracking annotation corresponding to the input images are presented. The figure highlights the presence of saliency shifts where the salient object changes along the sequence. Reproduced from [START_REF] Fan | Shifting more attention to video salient object detection[END_REF]. perform segmentation at the same time. In [START_REF] Fan | Shifting more attention to video salient object detection[END_REF], the authors present a large dataset of videos annotated with both instance segmentations and eye tracking. They train the model combininb convolutional and recurrent network in a supervised manner on both modalities, trying to predict both segmentation and human attention. The overall framework of this method is presented in Fig. 1.6. In [START_REF] Wang | Learning unsupervised video object segmentation through visual attention[END_REF], both modalities are also predicted using an alternative architecture exploiting attention in the segmentation phase.

Pure motion saliency and Trajectory-based saliency

By pure motion saliency, we mean that motion is the discriminative feature, and consequently, only motion information is exploited as input of the saliency detection method. This issue may be also referred to as anomalous motion detection. In that category, we can mention the use of social forces computed from the flow field [START_REF] Mehran | Abnormal crowd behavior detection using social force model[END_REF], and of the linear approximation of the dynamical system defined by the optical flow [START_REF] Solmaz | Identifying behaviors in crowd scenes using stability analysis for dynamical systems[END_REF].

A classical type of motion saliency to detect is salient motion resulting from a distinctive motion within a coherently moving set. The set can typically be a crowd of people, a herd of animals, a flock of birds, vehicle traffic, or a set of cells. In [START_REF] Pérez-Rúa | Detection and Localization of Anomalous Motion in Video Sequences from Local Histograms of Labeled Affine Flows[END_REF], the authors use local affine motion histograms that characterize the motion around a point at different scales, to detect regions that have different motion than their neighbors, and show that Figure 1.7 -Training scheme of the method presenting the processing steps of batch of trajectories along with the two associated losses. Reproduced from [START_REF] Maczyta | Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder[END_REF]. they can use it to detect anomalous motion in a crowd. We refer the reader to [START_REF] Li | Crowded scene analysis: A survey[END_REF] for a more complete description of the methods related to this task. Some methods use the similarity of trajectories along the sequence to group points and detect salient motion. In [START_REF] Zhou | Coherent filtering: Detecting coherent motions from crowd clutters[END_REF], the authors compute a coherence metric between trajectories over the sequence based on motion similarity. They group points with high similarity. They use it to segment trajectories of points in 3D and groups in crowds. In [START_REF] Shao | Scene-independent group profiling in crowd[END_REF], groups in crowd scenes are detected using an iterative process based on Markov chains to characterize the motion in each group. A set of features is then used to describe the relative motion of each group.

In [START_REF] Maczyta | Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder[END_REF], the authors train a salient trajectory detection method using an auto-encoder with a consistency loss that ensures that the latent codes for non-salient trajectories are similar. They achieve this by adding a regularization term that penalizes the latent codes that are far from the median latent code computed on the batch. This facilitates the use of the latent code for later classification, since the salient trajectories should have a very different representation from the non-salient ones. At inference time, they use a statistical criterion based on the distance to the median latent code to detect salient trajectories in an unsupervised setting. They apply this work to pedestrian trajectories in a train station and show that they can successfully extract salient trajectories. The training scheme of this method is presented in Fig. 1.7.

Part , Chapter 1 -Related Work

Partial Conclusion

In this chapter, we have introduced work on motion analysis with papers on motion segmentation, video object segmentation, and motion saliency. Although each of these tasks presents its own set of methods, they share similarities in that they all process image sequences. Across the set of presented works, we can identify three key components.

First, the choice of input data. Methods such as [START_REF] Cheng | SegFlow: Joint learning for video object segmentation and optical flow[END_REF], [START_REF] Lu | See more, know more: Unsupervised video object segmentation with co-attention siamese networks[END_REF], [START_REF] Odobez | MRF-Based motion segmentation exploiting a 2D motion model robust estimation[END_REF], [START_REF] Shrestha | Learning To Segment Dominant Object Motion From Watching Videos[END_REF], [START_REF] Ranjan | Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation[END_REF], [START_REF] Weiss | Smoothness in layers: Motion segmentation using nonparametric mixture estimation[END_REF] work directly on image pairs to extract moving objects. Others [START_REF] Tokmakov | Learning motion patterns in videos[END_REF], [START_REF] Zhou | MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation[END_REF], [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF], [START_REF] Narayana | Coherent motion segmentation in moving camera videos using optical flow orientations[END_REF], [START_REF] Bideau | It's moving! A probabilistic model for causal motion segmentation in moving camera videos[END_REF], [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], use a precomputed flow field. Methods such as [3], [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF] take only images as input to extract salient objects from appearance cues. They use motion as a signal to guide the process. Each choice of input data presents its own challenges. Working from image pairs requires extracting similarities between images. Appearance-based methods depend on the extraction of relevant image features. Optical flow field-based methods must deal with errors in motion estimation.

Second, the category of approach. We highlighted the importance of classical approaches in all tasks [START_REF] Koh | Primary object segmentation in videos based on region augmentation and reduction[END_REF], [START_REF] Wang | Representing moving images with layers[END_REF], [START_REF] Perez-Rua | Hierarchical motion decomposition for dynamic scene parsing[END_REF], [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF], [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF], [START_REF] Pérez-Rúa | Detection and Localization of Anomalous Motion in Video Sequences from Local Histograms of Labeled Affine Flows[END_REF], which provided a solid foundation for motion analysis. With the development of deep learning and the availability of large annotated datasets [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF], [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF], several supervised learning methods have been proposed for VOS and motion saliency such as [START_REF] Tokmakov | Learning motion patterns in videos[END_REF], [START_REF] Cheng | SegFlow: Joint learning for video object segmentation and optical flow[END_REF], [START_REF] Zhou | MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation[END_REF], [START_REF] Dave | Towards segmenting anything that moves[END_REF], [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF], [START_REF] Song | Pyramid dilated deeper ConvLSTM for video salient object detection[END_REF]. We also presented more recent unsupervised learning techniques [3], [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] designed to deal with tasks where there is no available ground truth. Unsupervised approaches share similar challenges with classical approaches, as they require a good modeling of the problem to design the loss function.

Third, how methods model temporal dimension. We have presented methods that consider only instantaneous motion between two frames, others that consider a sequence of frames, and finally methods [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF]- [START_REF] Kardoost | Self-supervised sparse to dense motion segmentation[END_REF], [START_REF] Maczyta | Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder[END_REF] that consider long time spans with sparse spatial representation (e.g., trajectories).This point has important implications for several aspects of the methods. For classical methods, it has an impact on the modeling assumptions, and for deep learning methods, it has an impact on the loss function and the choice of network architectures. Despite these advances, several challenges remain. First, the current unsupervised learning approach still lags behind the supervised approach for VOS tasks. Learning to group dependent motions (e.g. articulated motions or swarn motions) in a fully unsupervised manner is an interesting challenge, and could reduce the gap between the two

Partial Conclusion

categories of approaches on VOS. Second, we can see that recent image-based approaches [3], [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF] struggle to generalize to unseen image distributions. Thus, they often rely on the use of externally pre-trained features or on fine-tuning the method on the test data (i.e., test-time adaptation). Adressing the generalization challenge could improve the applicability of these methods to different input data modalities. Finally, a still open problem is the integration of a rich temporal dimension. Most of the methods we have presented process only a few frames or use trajectories, but incorporating a longer context could make it possible to address complicated motion analysis. Developing network architectures that effectively account for long-term context and designing losses that account for temporal coherence are necessary steps to address this challenge.

In our work, we focus on the first and third challenges as they are the most relevant to motion analysis. We have introduced methods to improve unsupervised motion segmentation and to integrate the temporal dimension into our networks (Chapters 2, 3, and 4). We have attempted to tackle unsupervised motion saliency estimation using task-specific interpretation information from a classification network (Chapters 5 and 6). In addition, we have developed a DL-based optical flow method dedicated to small moving objects in large images (Appendix 7.4). This appendix contains also a brief survey of recent optical flow methods based on deep learning, along with a detailed presentation of the RAFT method [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF] and its architecture.

Chapter 2

EM-DRIVEN UNSUPERVISED LEARNING FOR MOTION SEGMENTATION

In this chapter, we present a CNN-based fully unsupervised method for motion segmentation from optical flow. We assume that the input optical flow can be represented as a piecewise set of parametric motion models, typically, affine or quadratic motion models. The core idea of our work is to leverage the Expectation-Maximization (EM) framework in order to design in a well-founded manner a loss function and a training procedure of our motion segmentation neural network that does not require either ground-truth or manual annotation. However, in contrast to the classical iterative EM, once the network is trained, we can provide a segmentation for any unseen optical flow field in a single inference step and without estimating any motion models. We will investigate different loss functions including robust ones and propose a novel efficient data augmentation technique on the optical flow field, applicable to any network taking optical flow as input. In addition, our method is able by design to segment multiple motions. We have tested our motion segmentation network on four benchmarks, DAVIS2016, SegTrackV2, FBMS59, and MoCA.

Problem Statement

Motion segmentation is among the main computer vision tasks. Its goal is to divide a frame into motion-related coherent segments. Motion coherence must be understood with respect to a given property expressed by motion features, parametric motion models, or even higher-level motion information. Depending on the formulation of the problem or the need of the application, segments can be layers, i.e., non necessarily connected subsets of points, or regions, i.e., connected segments, forming a partition of the image grid. Motion segmentation is a relevant step for many applications covering video interpretation, biomedical imaging, robot vision, and autonomous navigation, to name a few.

Part , Chapter 2 -EM-driven Unsupervised Learning for Motion Segmentation

Motion segmentation is a complex problem that has been investigated for decades as described in Chapter 1, but there are still open questions. Indeed, it combines topology and information aspects in an intricate way. By topology, we mean the partition of the frame constituting the output of the segmentation. By information, the type of features or motion models it relies on. This all results in a chicken-and-egg problem: estimating easily and correctly the involved motion models requires an available partition, getting an accurate and reliable partition implies available motion models driving the segmentation.

As we want to address general-purpose image motion segmentation without anticipating any given application, we cope with optical flow segmentation (OFS). Indeed, the optical flow carries all the information related to motion between two successive frames of a video. Segmenting optical flow enables specific computer vision goals, as for instance, segmenting independent moving objects in the scene. Of course, the image motion depends on one hand on the relative motion between scene objects and the camera, and on the other hand on the object depth. As a consequence, any significant difference in depth results in a distinctive image motion. A static object in the foreground of the scene produces a specific flow pattern compared to the static background, and its image motion is supposed to form a separate segment. As done in Chapter 2, we call this situation "motion parallax".

We assume that the input optical flow can be represented as a piecewise set of parametric motion models, typically affine or quadratic ones, each of them characterizing motion in one segment. Thus, we formulate OFS as a piecewise linear regression problem, where finding supports (segments) and estimating the motion models are intertwined issues. This problem can be addressed with latent variables, which usually imposes an alternate optimisation strategy. The Expectation-Maximization (EM) algorithm is certainly the flagship solution for a statistical approach of this problem [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. Several extensions to the original EM were proposed as the Classification EM (CEM) introduced in [START_REF] Celeux | A classification EM algorithm for clustering and two stochastic versions[END_REF], where emphasis is put on the clustering issue of the problem beyond the mixture model one. However, classical EM relies on handcrafted features, and leads to time-consuming iterative algorithms. On the other hand, deep learning, and more precisely, convolutional neural networks (CNN), have now become the most effective key solution for image and motion segmentation [START_REF] Ranjan | Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation[END_REF], [START_REF] Badrinarayanan | SegNet: A deep convolutional encoder-decoder architecture for image segmentation[END_REF]- [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. Nevertheless, the training step remains an important issue. Supervised learning provides high accuracy, but manual annotation of optical-flow segmentation maps as ground truth is very cumbersome, and almost unreachable at a large scale. Unsupervised training is thus preferable but more challenging to optimize, in 60

Motion Segmentation as an EM Problem

particular to formulate the appropriate loss leading to the intended outcomes. Moreover, an unsupervised method is certainly the best way to deal with videos, and specifically optical flows, unseen during the training phase, thus ensuring better generalisation.

In this chapter, we aim to bring the two, EM and CNN, together in order to design a principled and efficient unsupervised motion segmentation method. By unsupervised, we mean that we do not resort to any ground-truth and manual annotation, both for the training stage in the loss function, for the selection of the optimal trained network model and for the choice of network hyperparameters. On one hand, the parametric motion models will carry the coherence for each motion segment. However, the key point is to confine their estimation to the training stage. On the other hand, we take EM as the well-founded basis for the design of the loss function and consequently the training stage of the motion segmentation network. In the related framework of mixture density networks [START_REF] Bishop | Mixture density networks[END_REF], a neural network is combined with a mixture density model. However, it does not rely on EM. Once trained, our network segments each flow of the video without any iteration and any motion model estimation.

Thus, the main purpose of our EM-driven network is to uncover motion coherence. Taken alone, it can segment motion within videos. It can also be incorporated in a larger framework to solve any video segmentation or understanding problem that could benefit from a motion coherence cue. This could be achieved either by using our network as a module inside a bigger pipeline, for motion saliency detection for instance, or by using our proposed loss as a regularization term for the training of a neural network.

We will evaluate and compare our method to existing methods on four datasets: DAVIS2016 [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF], FBMS59 [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF], SegTrackV2 [START_REF] Li | Video segmentation by tracking many figure-ground segments[END_REF] and MoCA [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF].

The chapter is organized as follows. We present in Section 2.3 how we leverage the EM algorithm to design the network loss function, its architecture and the training procedure. Section 4.4 reports extensive experiments with comparison to other existing methods on four benchmarks. In Section 2.5, we discuss several issues related to the main components of our approach including a comparison with the choices made by other methods.

Motion Segmentation as an EM Problem

The core idea of our work is to leverage the Expectation-Maximisation framework to design in a well-founded manner the loss function and the training procedure of our OFS neural network. In this section, we first describe one way to use the EM algorithm for optical flow segmentation.

Optical flow f ∈ R 2×W ×H is a vector field defined over an image grid Ω of size I = W × H. We denote by f i ∈ R 2 the motion vector associated to each site i ∈ Ω of this grid. We make the assumption that any optical flow field can be decomposed in a set of K segments or layers, each one grouping a (possibly non connected) part of the image grid and exhibiting a coherent motion. In order to enforce coherence, we choose to represent the motion field within each segment k with a parametric model defined by parameters θ k . We denote θ = {θ k , k = 1, . . . , K}. In practice, we use polynomial motion models, typically affine (first-degree polynomial) or quadratic (second-degree polynomial) models. Their interest lies in both an easy physical interpretation and an efficient estimation. For instance, a specific 8-parameter quadratic motion model corresponds to the projection, into the image plane, of the rigid motion of a 3D planar surface. Thus, we can account for any slanted almost planar surface that exhibits a smooth depth variation, not only frontoparallel ones. By almost planar, we mean a negligible depth variation of the object surface compared to its distance to the camera. Our method could nevertheless accommodate other types of parametric models.

We now consider the likelihood of the optical flow field f given the set of parameters θ, denoted by p(f |θ). In order to make explicit the partition of f into k segments and the associated individual θ k 's, we introduce latent variables z i such that p(z i = k|f i , θ k ) represents the probability that site i belongs to layer k. Assuming conditional independence, the logarithm of the likelihood can be written as below in step 1 of eq.(2.1). Then, we introduce the z i variables (steps 2 and 3 of eq.(2.1)), and we finally straightforwardly make any positive distribution q appear (step 4 of eq.(2.1)). We have:

log(p(f |θ)) = log i p(f i |θ) = log i k p(f i , z k i |θ k ) = i log k p(f i , z k i |θ k ) = i log k q(z k i ) p(f i , z k i |θ k ) q(z k i ) , ( 2.1)
where

z k i ≜ [z i = k]. Maximizing log(p(f |θ)) w.r.t.
θ is obviously complicated, even if it boils down to k maximizations w.r.t. the θ k 's. Indeed, variables z i are hidden. To maximize eq.(2.1) w.r.t. θ k , variables z i must be available. Accordingly, we need to maximize also w.r.t. the z i 's.

However, we can use the Jensen's inequality (h(E[x]) ≥ E[h(x)]

for any concave function h), as done in classical EM [START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF], in order to build a lower bound ll(θ) of the loglikelihood log(p(f |θ)). We get log(p(f |θ)) ≥ ll(θ) with:

ll(θ) = i k q(z k i ) log p(f i , z k i |θ k ) q(z k i ) = i k q(z k i ) log p(f i , z k i |θ k ) - i k q(z k i ) log q(z k i ), (2.2)
where the first term of eq.( 2.2) is the expectation over q(z i ) of log p(f i , z i |θ) and the second term is the entropy that we note H. The resulting expression of the lower bound is:

ll(θ) = i E q(z i ) [log p(f i , z i |θ)] + i H(q(z i )). (2.3)
In the classical EM algorithm, one usually takes q(z k i ) ≜ p(z k i |f i , θ k ), i.e., the posterior distribution. Then, one alternates between an expectation step where q(z k i ), ∀i, k, is estimated, and a maximization step where ll(θ) is maximized w.r.t. the θ k 's. This procedure monotonically increases the log-likelihood until it reaches a local optimum [START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF].

CNN-based Motion Segmentation

In our case, we adopt a neural network model g ϕ (f ), taking as input the optical flow f and parameterized by ϕ, to produce the image motion segmentation that is our primary goal. This network has a softmax activation in order to output a valid probability distribution over layers for each site.

The motivation is that, by doing so, we can access a large family of distributions. Most importantly, after the training stage, our network is able to infer the motion segmentation without iterating and without involving motion models, in contrast to the classical EM algorithm. In addition, it is much faster. At the training stage, we deal with two sets of parameters: the parameters of the motion models θ and the parameters of the network model ϕ. At the inference stage, we are only concerned by the network parameters ϕ. The overall flowchart of our method, including training and Training step: First, we segment the optical flow field f with the neural network g ϕ . Then, we get the optimal parametric motion models {f θ * k } k=1,...,K within each probabilistic segmentation masks {g ϕ (f ) k } k=1,...,K using (2.16). Finally, we update the parameters ϕ of the neural network using (2.17), where the loss function is defined in (2.12). This training step is performed iteratively over each batch B (of size 1 in this illustration). Inference step: We directly apply the trained network g ϕ * to any new unseen optical flow field f to obtain the probabilistic segmentation masks {g ϕ * (f ) k } k=1,...,K . There is no estimation of the motion models {f θ * k } k=1,...,K in the inference step in contrast to the training step. For the sake of visualization, optical flows and polynomial motion models are represented with the HSV color code, but actually, the flow field f used as input of the neural network is taken as a 2D vector field. We have a two-channel input. Optical flow coding: correspondence between the arrow visualization of the optical flow field and the HSV color map. inference stages, is given in Fig. 2.1.

EM-driven network specification

Coming back to eq.(2.2) and following the choice expressed above, we take g ϕ (f ) k i as q(z k i ), where g ϕ (f ) k i is the probability (prediction) given by the network for site i to belong to segment k given the input optical flow f . The lower bound now depends on two sets of parameters, θ and ϕ, and writes:

ll(θ, ϕ) = i k g ϕ (f ) k i (log p(f i , z k i |θ k ) -log g ϕ (f ) k i ) = i E g ϕ (f ) i [log p(f i , z i |θ)] + i H(g ϕ (f ) i ). (2.4)
We alternatively optimize ll(θ, ϕ) with respect to θ and ϕ for the training stage as follows:

θ * = arg max θ i k g ϕ (f ) k i log(p(f i , z k i |θ k )) (2.5) ϕ * = arg max ϕ i k g ϕ (f ) k i log(p(f i , z k i |θ * k )) + i H(g ϕ (f ) i ). (2.6)
As previously described in [START_REF] Hathaway | Another interpretation of the EM algorithm for mixture distributions[END_REF], the entropy of the predicted segmentation at each site i, H(g ϕ (f ) i ), naturally arises in eq.( 2.4), and then, in eq.(2.6). Entropy measures statistical uncertainty and is maximised for

g ϕ (f ) k i = 1 K , ∀i, k.
It acts as a regularization term balancing the likelihood term to avoid falling too quickly into (inappropriate) local optima.

Regarding the optimisation on θ, we can reach a local optimum using an off-theshelf iterative algorithm. However, we can only perform a gradient descent step for the optimisation with respect to the network weights ϕ.

In order to gain intuition on how the network is learning to produce the motion segmentation, following [START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF], we rewrite the lower bound as:

ll(θ, ϕ) = i k g ϕ (f ) k i log p(f i , z k i |θ k ) g ϕ (f ) k i = i k g ϕ (f ) k i log p(z k i |f i , θ k ) g ϕ (f ) k i + i log p(f i |θ) k g ϕ (f ) k i = - i KL[g ϕ (f ) i ||p(z i |f i , θ)] + log(p(f |θ)). (2.7)
Consequently, the optimisation step over the network weights is defined by:

ϕ * = arg min ϕ i KL[g ϕ (f ) i ||p(z i |f i , θ * )], (2.8)
where we minimize the KL-divergence between the segmentation produced by the network and the segmentation linked to the optimal parameters θ * . Thus, the network is trained to produce a segmentation for a given set of parameters. As the quality of the network segmentation improves, so does the quality of the estimated θ * , pushing the network weights to produce better segmentation in turn.

Flow likelihood and loss function

In the previous section, we described the overall training process. In this section, we address the definition of the different terms of the loss function.

First, we decompose the joint probability in eq.( 2.4) into a likelihood and a prior:

p(f i , z k i |θ k ) = p(f i |z k i , θ k )p(z k i ).
(2.9)

The likelihood p(f i |z k i , θ k ) assesses how the estimated parametric motion model in a given region fits the observed flow in this region. In this work, we use a uniform prior for p(z k i ). Nevertheless, we could adopt a more complex prior, if we wanted to influence the size of each region for instance.

An important point of our design is to specify the form of the likelihood p(f i |z k i , θ k ) that is used to compare the input optical flow with the parametric flow for a given set of parameters θ. In practice, since our parametric motion models are dependent on the position of the points on the 2D space, we introduce a deterministic function c(i) that maps the site i to a polynomial expansion involving its coordinates. More specifically, for a 6parameter affine motion model, we have c(i) = 1, x i , y i ; for a full 12-parameter quadratic model, we have

c(i) = 1, x i , y i , x 2 i , x i y i , y 2 i .
The likelihood evaluates the distance between the input flow vectors f and the parametric flow vectors

f θ k i ≜ θ T k • c(i), ∀k, i.
Its general form is given by:

p(f i |z k i , θ k ) = 1 Z exp(- 1 α δ(f i , θ T k • c(i))), (2.10)
where δ : R 2 * 2 → R is a distance function to define, and α is a free parameter related to the uncertainty in the flow measure and the resulting adequacy of the parametric motion model. If δ is a translationally invariant function, which we verified for all tested distance functions, then Z is only dependent on the function δ and hyperparameter α and not on the input. The proof can be found in Appendix 2.4.3. This allows us to perform optimisation without explicitly computing Z.

The choice of the distance function δ is central in our approach, as it is used both for the estimation of the parametric motion models and for the training of the network (see eq.(2.5) and eq.(2.6)). Robust loss functions can be beneficial as thoroughly investigated in [START_REF] Barron | A general and adaptive robust loss function[END_REF]. We consider the following distance functions:

-Squared L 2 : δ(f i , θ T k • c(i)) = ||f i -θ T k • c(i)|| 2 2 -L 2 norm : δ(f i , θ T k • c(i)) = ||f i -θ T k • c(i)|| 2 -L 1 norm : δ(f i , θ T k • c(i)) = ||f i -θ T k • c(i)|| 1
L 1 (due to the absolute function involved) and L 2 (due to the square root of the sum involved) norms bring robustness to outliers in the flow field, in contrast to the squared

L 2 .
We define the loss of our model as:

L(f, θ, ϕ) = -ll(θ, ϕ), ( 2.11)
where ll(θ, ϕ) is given by eq.(2.4). Taking into account eq.( 2.10), we can formulate the loss function as:

L(f, θ, ϕ) = 1 α i k g ϕ (f ) k i δ(f i , θ T k • c(i)) + i k g ϕ (f ) k i log g ϕ (f ) k i + I log(K Z), (2.12)
where Z is the normalization term in eq.(2.10) and α allows us to balance the likelihood, prior and entropy parts of the loss. We use α = 10 -2 in all our experiments, but in practice the network model is fairly robust to the choice of this hyperparameter.

Part , Chapter 2 -EM-driven Unsupervised Learning for Motion Segmentation

Herefater, we give more details on the derivation of eq.(2.12). 10)

L(f, θ, ϕ) = -ll(θ, ϕ) (2.13) = - i k g ϕ (f ) k i log p(f i , z k i |θ k ) + i k g ϕ (f ) k i log g ϕ (f ) k i using (4) = - i k g ϕ (f ) k i log p(f i |z k i , θ k ) - i k g ϕ (f ) k i log p(z k i ) using (9) + i k g ϕ (f ) k i log g ϕ (f ) k i = log Z i k g ϕ (f ) k i + 1 α i k g ϕ (f ) k i δ(f i , θ T k • c(i)) using (
- i k g ϕ (f ) k i log p(z k i ) + i k g ϕ (f ) k i log g ϕ (f ) k i .
As aforementioned, we use a uniform prior for p(z k i ), i.e., p(z k i ) = 1 K . We can thus rewrite the loss L as:

L(f, θ, ϕ) = log(K Z) i k g ϕ (f ) k i + 1 α i k g ϕ (f ) k i δ(f i , θ T k • c(i)) (2.14) + i k g ϕ (f ) k i log g ϕ (f ) k i . As k g ϕ (f ) k i = 1, ∀i, we have i k g ϕ (f ) k i = I,
where I is the number of sites i in the frame. We finally obtain:

L(f, θ, ϕ) = I log(K Z) + 1 α i k g ϕ (f ) k i δ(f i , θ T k • c(i)) (2.15) + i k g ϕ (f ) k i log g ϕ (f ) k i .
We describe in Appendix 7.1 an alternative approach to end up with the same loss function, based on the variational inference framework involving the Evidence Lower Bound (ELBO).

Network training

During the training, we minimize the loss function L over a dataset of optical flow fields. For each input flow field f of the training dataset, we minimize L(f, θ, ϕ) with respect to each parameter. This alternate optimisation is performed over every batch B as follows:

θ * = arg min θ f ∈B L(f, θ, ϕ t ) (2.16
)

ϕ t+1 = ϕ t -γ∇ ϕ f ∈B L(f, θ * , ϕ), (2.17)
where t is the iteration number and γ the learning rate.

In practice, we use an optimizer to get θ * and an automatic differentiation to compute the gradients with respect to ϕ. As described in subsection 2.3.1 and in our computation graph presented in Fig. 2.2, we consider θ * as fixed in the gradient step with respect to ϕ, making ∇ ϕ L(f, θ * , ϕ) trivial to compute using automatic differentiation. Practical details are provided in subsection 2.4.1.

Figure 2.2 -Illustration of the computation graph for the training of our network. m ≜ g ϕ (f ) denotes the set of arrays (as many as masks) collecting the probability for each site of the input flow field to belong to each mask. θ is the set of the motion model parameters, and ϕ the set of the network parameters. In our method, we are alternatively optimising w.r.t. θ (optimisation 1) and ϕ (optimisation 2).

Data augmentation

As proven beneficial in many computer vision problems, we proceed to data augmentation to train the motion segmentation network. However, the input data are not images but optical flows in our case, which led us to define an original data augmentation procedure. The goal is to make the network as invariant as possible to the global motion field, since we identify it as an important clue for generalisation.

Part , Chapter 2 -EM-driven Unsupervised Learning for Motion Segmentation

Therefore, we add to each optical flow field of the dataset a parametric motion model whose parameters are drawn at random. For the sake of consistency, we take quadratic motion models. This mimics a large variety of camera movements. This procedure has the advantage of multiplying the flow configurations, while keeping the same flow structure as in the initial sample. In other words, the network is trained with a diversity of flows, while having the same target segmentation to predict. In Appendix 7.1, we give a formal proof that our loss function is invariant to the added parametric global motion. As shown in Section 2.4.3, this data augmentation scheme contributes to improve the overall performance of the network. Let us emphasize that it could also be used to train any network taking optical flow as input with the benefit of such invariance.

Related work on EM and deep learning

To conclude this section, we explain how our approach differs from other works that somehow make use of EM within a neural network framework. In [START_REF] Li | Expectation-maximization attention networks for semantic segmentation[END_REF], the EM paradigm is involved in the network designed for semantic segmentation. However, the purpose is quite different since they exploit the EM algorithm in the attention mechanism. It allows them to iteratively estimate a compact set of bases used to compute the attention maps.

The EM framework plays also a role in [START_REF] Greff | Tagger: Deep unsupervised perceptual grouping[END_REF], [START_REF] Greff | Neural expectation maximization[END_REF], and just recently in [START_REF] Yu | Unsupervised foreground extraction via deep region competition[END_REF], in the design of the neural network architecture concerned with perceptual grouping tasks. However, those approaches differ in several ways. First, they remain iterative. One starts from parameters specifying each latent space component (as our motion parameters θ), but the network iteratively refines them in the inference stage, so that this network remains dependent on the initialization. Another big difference with our method lies in the network architecture itself, which includes generative branches to produce an approximation of the input data from the latent space components and an iterative branch (recurrent network in [START_REF] Greff | Tagger: Deep unsupervised perceptual grouping[END_REF], [START_REF] Greff | Neural expectation maximization[END_REF] and sampling in [START_REF] Yu | Unsupervised foreground extraction via deep region competition[END_REF]) to update the latent space components themselves. Since those methods rely on an iterative inference process implementing EM, they rather belong to the class of algorithm unfolding techniques [START_REF] Monga | Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing[END_REF].

Experimental Results

Implementation details

Optical flow fields are computed on the original video frames using the RAFT method [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF]. A detailed description of the RAFT method and its architecture can be found in Appendix 7.4. More specifically, we use the RAFT implementation 1 with network weights fine-tuned on the MPI Sintel dataset [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF]. At the time we started this work, the RAFT method was among the top methods on the MPI Sintel benchmark 2 , whose code was available. For efficiency, we downsample the computed optical flow fields to obtain 128 × 224 vector fields provided as input to the network, as also done in the MoSeg method [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. The resulting segmentation is subsequently upsampled to the original frame size for evaluation w.r.t. the ground truth. It allows us to perform much more efficient training and inference stages. In all experiments, we use the L 1 norm loss function, unless otherwise specified.

One may wonder about the impact of the chosen optical flow method on the motion segmentation results. Obviously, the network output is likely to depend on its input, but to what extent? Early experiments carried out at the beginning of this work showed that the motion segmentation results were barely downgraded when using the PWC-Net method [START_REF] Sun | PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF]. A more challenging issue is to use a non-supervised optical flow method as the ARFlow method [START_REF] Liu | Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation[END_REF] that is necessarily far less accurate than supervised optical flow methods as RAFT and PWC-Net. We used the official PyTorch implementation of ARFlow 3 with the network weights provided in the file "checkpoints/Sintel/pwclitear.tar". In that case, the performance of our motion segmentation method clearly decreases, but interestingly, it is less affected than the MoSeg method as shown in Table 2.1.

We take the full quadratic motion model with 12 parameters to represent the optical flow within each segment k:

f θ k (x, y) = (θ k 1 + θ k 2 x + θ k 3 y + θ k 4 x 2 + θ k 5 xy + θ k 6 y 2 , θ k 7 + θ k 8 x + θ k 9 y + θ k 10 x 2 + θ k 11 xy + θ k 12 y 2 ) T , (2.18)
where point (x, y) belongs to segment k. We take this parametric motion model since it ARFlow* 57.9 -Table 2.1 -Impact of the optical flow method on the accuracy of the resulting motion segmentation, and comparison with the behaviour of the MoSeg method whose scores are drawn from [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. Experiments were carried out on the DAVIS2016 dataset and the Jaccard index (intersection over union) is used for the evaluation score. *In this experiment, ARFlow is only used at inference. can better fit complex motion. Indeed, it is likely to better encompass the background motion when the camera motion includes both translation and rotation with a static background involving objects at slightly different depths, and for articulated motions as well.

Our method is fully unsupervised. We do not resort to any manual annotation for training nor for model selection. Indeed, in all experiments, we selected the stopping epoch from the loss function evaluated always on the same validation set: the official training split of DAVIS2016 [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF].

We consider generalisation to unseen datasets as an essential property of any motion segmentation method. To apply this idea, we trained once and for all our model on a single dataset, namely Flying Things 3D (FT3D) [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF], for all experiments. In addition, the use of a synthetic dataset alleviates the problem of choosing the relevant real training data for a given experiment. For completeness, we also performed training experiments on real datasets, and obtained similar or slightly better performance depending on the chosen real training set.

Let us recall that the optimisation on θ does not occur at inference stage. The network prediction for each site of the flow field to belong to each segment k is directly used to yield the motion segmentation map. We simply select for each site segment k with the highest probability. No postprocessing is performed on the resulting segmentation. As shown later, the obtained segments are generally smooth, certainly due to the implicit regularization capacity of the network.

We take as input the optical flow in its vector field representation f ∈ R W ×H×2 . Thus, we have a two-channel input for the network. Our loss function and our training procedure could be adapted to any neural network designed for segmentation. We choose the well-known convolutional architecture U-net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF] for g ϕ . We use a slightly modified implementation of the one available under PyTorch Lightning [START_REF] Falcon | A framework for contrastive self-supervised learning and designing a new approach[END_REF]. We take seven downsampling layers and start with a feature depth of 64. The selection of the structure is again unsupervised using the validation loss on FT3D dataset. As did in [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF], we use InstanceNorm between convolutional blocks in order to tackle variations in optical flow magnitude over the dataset.

We use Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] optimizer with a learning rate of 10 -4 to train the network. The optimisation on θ is done with Pytorch implementation of L-BGFS [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF]. Batches comprise flow fields randomly sampled from the training dataset.

Our network 4 is time efficient, being a simple convolutional network, with an average computation time of 0.008s per 128×224 input flow field on a Tesla-V100, if we consider a batch of size 32. Without any parallelisation (batch size of 1), it can run at 36fps making it usable for real-time applications. In particular, it is faster than the fastest method introduced in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], because we do not use an iterative attention module, thus reducing our computational complexity. In contrast to methods involving self attention, there is a linear relationship between the complexity of our network and the size of the optical flow field, which allows us to readily process input of large dimensions.

Comparative evaluation

We want to objectively evaluate the performance of our method for segmenting optical flow fields. Due to the lack of benchmarks dedicated to OFS, we compare our method on four VOS datasets described below.

DAVIS2016 5 [15] includes 50 videos (3455 frames) split in 30 train and 20 validation videos with diverse objects. Only the principal moving object is annotated in the ground truth. We use the official criteria for evaluation on this dataset: the Jaccard score and the contour accuracy score.

SegTrackV2 6 [88] and FBMS59 [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF] respectively comprise 14 videos (1066 annotated frames) and 59 videos (720 annotated frames), each involving one or several moving objects. For FBMS59 we use the 29 sequences of the validation set for evaluation. In cases where there are several moving objects, we group them into a single foreground mask for evaluation, as done in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF], [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF] and [START_REF] Yang | DyStaB: Unsupervised object segmentation via dynamic-static bootstrapping[END_REF]). J is the Jaccard index (region similarity) and F accounts for contour accuracy. The higher the value, the better the performance. For further explanation on the evaluation metrics, we refer the reader to the DAVIS2016 website. Following the evaluation protocols, reported scores are the average of scores over all samples in the corresponding dataset, except for DAVIS2016 where it is the average of each sequence average score. Our network is trained once and for all on the synthetic dataset FT3D whatever the benchmark.

Moving Camouflaged Animals (MoCA) [START_REF] Lamdouar | Betrayed by motion: Camouflaged object discovery via motion segmentation[END_REF] presents camouflaged animals in natural scenes. For a fair comparison, we use the subset released by [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF] with 88 videos and 4803 frames. Ground-truth bounding boxes are provided instead of masks for evaluation. Accordingly, we convert our output to a bounding box around the largest connected region of our output mask as done in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF].

Apart from a few videos in SegTrackV2 and FBMS59, those datasets focus on one primary moving object. Indeed, videos depict one single independently moving object in the foreground. Consequently, the ground truth comprises only two segments: foreground primary moving object versus background. To be coherent with this status, we apply our method with two masks, i.e., K = 2. To choose the foreground mask, we rely on a simple heuristic where we designate the biggest mask as the background one.

We compare our method with several other supervised and unsupervised methods: MoSeg [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], TIS (two versions) [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF], CIS [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], FTS [START_REF] Papazoglou | Fast object segmentation in unconstrained video[END_REF], DyStab [START_REF] Yang | DyStaB: Unsupervised object segmentation via dynamic-static bootstrapping[END_REF], ARP [START_REF] Koh | Primary object segmentation in videos based on region augmentation and reduction[END_REF], MATNet [START_REF] Zhou | MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation[END_REF] and COSNet [START_REF] Lu | See more, know more: Unsupervised video object segmentation with co-attention siamese networks[END_REF]. All these methods were described in Chapter 1 (Related Work).

Results are collected in Table 2.2. For a fair comparison, we underline several points in this table. First, we differentiate methods trained on ground-truth segmentation masks such as COSNet [START_REF] Lu | See more, know more: Unsupervised video object segmentation with co-attention siamese networks[END_REF] or MATnet [START_REF] Zhou | MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation[END_REF], and unsupervised methods. We also indicate methods that use features trained in a supervised way, as it provides a strong advantage, and then do not fit in an unsupervised scenario. Indeed, DyStab [START_REF] Yang | DyStaB: Unsupervised object segmentation via dynamic-static bootstrapping[END_REF] resorts to supervised classification features to initialise its networks, and ARP [START_REF] Koh | Primary object segmentation in videos based on region augmentation and reduction[END_REF] requires a motion boundary detection algorithm previously trained in a supervised way. Secondly, as we evaluate here motion segmentation based on optical flow, we dissociate methods that take RGB images as input from methods using only optical flow like ours. Some works like [START_REF] Yang | DyStaB: Unsupervised object segmentation via dynamic-static bootstrapping[END_REF], [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF] propose versions with (TIS s , DyStab-Stat&Dyn) and without (TIS 0 , DyStab-Dyn) RGB frames taken as input, illustrating the influence of this input modality on the final results. As in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], we distinguish the CIS version involving a strong CRF-based postprocessing, "CIS-With Post", and one without postprocessing, "CIS-No Post", since postprocessing drastically increases runtime making the method unusable for practical applications. It is measured in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF] that CIS [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] has a runtime of 11s/frame with postprocessing against 0.1s without. Table 2.2 shows that our method outperforms all comparable methods on DAVIS2016 and FBMS59. Our method is the second best for the two other datasets. It is even close to the best one and far ahead the other comparable methods for MoCA. By comparable methods, we mean unsupervised methods without unusable postprocessing. Let us stress that we train our method on an external dataset, unlike CIS that is trained on test data as well. Scores obtained by our method for every sequence of the four datasets are collected in Appendix 7.1.

In Fig. 2.3, we report visual results to figure out how our method behaves on different typical examples from the VOS datasets. We can observe four examples of failure cases with respect to the DAVIS2016 ground truth (for scooter-black, kite-surf, blackswan and parkour videos), although the extra parts segmented by our method make sense w.r.t. OFS task. Let us recall that the VOS task takes into account by construction only the primary moving object and not all moving objects in the scene. In scooter-black, the car segmented in the background is moving; in parkour, the fence is segmented as it exhibits an important motion parallax; in kite-surf and blackswan, the ripples on the water are segmented too. This type of complex examples will be more appropriately handled with the multiple motion segmentation described in subsection 2.4.5. Third row: the segmentation produced by our method superimposed in green on the corresponding image.

Part , Chapter 2 -EM-driven Unsupervised Learning for Motion Segmentation

Ablation study

In order to identify the contribution of the different components of our method, we performed an ablation study. We investigated the following four changes:

-DA: removal of the data augmentation on the optical flow described in subsection 2. We performed each change on our full method one by one.

As shown in Table 2.3, the quadratic motion model plays a pivotal role in the performance of the network. The other components still play an important role in an equal measure. We can observe that the L 2 norm provides better performance than the squared L 2 . As mentioned before, we train the network on the FT3D dataset where estimated optical flow fields exhibit low noise, which mitigates the impact of a robust loss for training. In previous experiments, the performance gap between a model trained with L 1 loss and squared L 2 loss on a real dataset was even more significant.

Further analysis and comparison with the classical EM

In this subsection, a quantitative comparison, in term of speed and accuracy, between classical EM and our OFS network is given along with an experimental study on the First, we elaborate on the iterative parametric motion segmentation based on the classical EM algorithm. More precisely, we study the influence of the initialisation of θ, parameters of the motion models, on the classical EM algorithm performance. We perform this experiment still using quadratic motion models. In doing so, we also highlight the impact of the neural network alone on the performance of our method.

Even if EM algorithm is guaranteed to converge to a local optimum [START_REF] Murphy | Machine Learning: A Probabilistic Perspective[END_REF], the quality of the optimum reached is highly dependent on the initialisation of θ. We take as conditional likelihood p(f i , z k i |θ k ) the Gaussian distribution, which leads to the squared L 2 once applying the log function. We evaluate the classical EM on the DAVIS2016 benchmark. Let us recall that the performance criterion is the Jaccard index computed on the primary moving object, not the value of the likelihood being maximised.

To emphasize the dependence of the classical EM on the initialization of θ, we study the evolution of the Jaccard index over the validation set with respect to the number of available random initializations. For each point of the curve, we try available initialisations, and select for each frame the one that maximizes the likelihood. We repeat a second time this experiment, but now using the Jaccard index as selection criterion. Of course, this optimal Jaccard index remains speculative in practice, since we need the ground truth to find the best Jaccard score for each sample. Let us stress that the computational cost involved is enormous in both cases.

We summarize the results obtained on the DAVIS2016 validation set in Fig. 2.4. We observe that the optimal Jaccard index for the classical EM algorithm increases with the number of initialisations until reaching a plateau where testing additional initialisation does not improve the results. To ensure this behaviour, we extended this experiment up to a huge number of 3000 initialisations. The optimal (speculative) Jaccard index for the classical EM goes up to 72.3. The Jaccard index for the classical EM when selecting the initialisation with the maximum likelihood, the only means available in practice without ground-truth, tops out at 63.7 . This demonstrates the impact of the initialisation step on the performance of the classical EM algorithm. As mentioned above, the optimal Jaccard index is only speculative. Thus, we consider this score of 72.3 as a gold standard. Importantly, Fig. 2.4 shows that the performance of our network is relatively close to the gold standard given by the speculative optimal Jaccard index of the classical EM algorithm. Besides, the latter needs a number of 41 initialisations to exceed the score of our network. This is already a large number, since, even with our efficient GPU In Fig. 2.4, we also observe that our network outperforms by a large margin the classical EM algorithm when the initializations are selected according to the highest likelihood, even for many tested initialisations, the only means available in practice without groundtruth. In addition, the network does not need motion models, and of course, does not involve any initialisation on motion parameters θ, when segmenting the optical flow. Moreover, we can easily adopt robust loss functions for the network training instead of the squared L 2 loss.

In Table 2.4, we report a quantitative comparison based on several aspects between our method and the classical EM. Accuracy and inference time were computed on the DAVIS2016 validation dataset. As we mentioned above, results using the classical EM strongly depend on the initialisation of the motion parameters. To assess the accuracy of the classical EM, we report first the average score obtained using randomly chosen motion parameters, and secondly, the score obtained by selecting the best initialization from a set of 500 random initialisations. We used 500 here since this is the number of initialisations where the Jaccard score starts to plateau at its optimal value, as shown in Fig. 2.4. By design, our method involves a segmentation network with a very large number of parameters to train and store. However, as demonstrated in Table 2.4, our method exhibits strong advantages compared to the classical EM regarding inference time and outperforms it by a large margin in term of accuracy. Figure 2.5 -Results obtained with our method for four masks (K = 4) regarding multiple motion segmentation. First row: one image of the video. Second row: input optical flow displayed with the HSV color code that is depicted in Fig. 2.1. Third row: motion segmentation maps with four masks, one colour per mask (the four masks may be not present if not necessary). We adopt the same color code for all segmentation maps (dark blue: mask 1, light blue: mask 2, green: mask 3, yellow: mask 4). Examples are drawn from DAVIS2016, FBMS59 and SegTrackV2 datasets. Videos in lexicographic order: mallard-fly, hockey, libby, swing, hummingbird, cars5, cars4, people2. 2.5 -Results on DAVIS2016 validation dataset, SegTrackV2, FBMS59 validation dataset and MoCA obtained with our method for several numbers of masks. Quantitative evaluation is performed here using ground truth for selecting the right masks as described in the main text.

Multiple motion segmentation

Davis2016 (J ) SegTrack V2 (J ) FBMS59 (J ) MoCA (J ) K = 3
Our method can handle by design the segmentation of multiple motions. Indeed, it has been defined from the start with K masks. In Section 2.4.2, we took into account only two masks for the evaluation on VOS datasets, because the challenge and the ground-truth have been defined in this way. In this subsection, we report additional experiments with several masks (K ∈ {3, 4, 5, 6, 7}) in Table 2.5. Visual results are also displayed in Fig. 4.7 for K = 4. They were obtained on videos from DAVIS2016, FBMS59 and SegTrack-V2.

We observe that our method can deal with multiple motions in the video and correctly segment them. This figure includes several examples of articulated motion (e.g., mallard-fly, hockey), and independently moving objects (e.g., cars5, cars4, people2). The figure also contains examples corresponding to "virtual" failure cases encountered in the twomask VOS challenge reported in Section 2.4.2. We mean segments that really correspond to moving objects but are not included in the ground truth that is focused on the primary moving object. These results demonstrate that, when involving more than two masks, we can correctly deal with interfering motions such as motion parallax (e.g., libby, swing) or additional moving objects (e.g., cars5). In addition, our multimask parametric motion segmentation could deal with objects comprising several significant planar surfaces of distinct orientation, each of them corresponding then to different optical flow segments.

We evaluated our multi-mask segmentation on the VOS datasets presented in Section 4.4. As we generate now several masks, segment selection is an issue for computing the J score that is based on two masks (foreground moving object versus background). Just to be able to compute this score and to figure out the performance we could reach (at least an upper bound), we declare as foreground all layers that overlap for their most part the ground-truth foreground mask. Scores are collected in Table 2.5. As expected, performance increases with the number of masks up to a certain point. We observe that we get very high scores on the four benchmarks, and, for K > 3, we outperform all the other comparable methods by a relatively large margin (see Table 3.2). We are even close to the performance of supervised methods.

Of course, this would not be a valid procedure to extract the primary moving object in practice, since we need the ground-truth masks. Here, it is just used to consider a quantitative evaluation on the benchmarks. It gives an insight into the quality of our multiple motion segmentation, since, so doing, we are not modifying our segmentation masks, but only selecting them. It also highlights the potential of our multiple motion segmentation method, if paired with an effective mask selection procedure dedicated to the considered downstream task.

Based on visual assessment of our results, we also observed that our method implicitly ensures temporal consistency to some extent. Indeed, motion segments are consistently segmented over time. Moreover, they generally correspond to the same mask number within the same video. This seems to be particularly true for the background when looking at the segmentation results. We illustrate this behaviour in Fig. 2.6. This is an appealing property, since the output of the network could be easily exploiting for tracking or for higher-level dynamic scene understanding.

Discussion

Our work takes its origin in the classical methods of image motion segmentation. The main idea is to group elements of similar nature under a common set of parameters. The novelty of CNN approaches is to train a model that extracts groups without relying on an iterative process at inference. In addition, they engage by design an implicit consistency making the network robust to corrupted observations. As discussed in [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], region (or layer)-based motion segmentation approaches face the important challenge of specifying a proper model (or regularization) representing the flow within each segment. A too weak regularization may make the flow reconstruction trivial (i.e., the reconstruction is just a copy of the input flow). Conversely, a too strongly regularized model may make the flow reconstruction coarse, since it is likely to be not expressive enough.

Hereafter, we compare our design choices with the ones of the two competing unsupervised optical-flow segmentation methods: CIS [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] and MoSeg [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF]. They rely on a different view of the problem. CIS circumvents the regularization problem by inpainting the flow of the foreground region from the flow of the background one and vice versa. In this setting, any model, even unconstrained, could not correctly inpaint the flow in the event of a perfect segmentation, thus alleviating the problem of regularization. MoSeg addresses the regularization problem by employing a representational bottleneck linked to the architectural structure of the network. Indeed, they force the flow to be reconstructed only from two latent representations of reduced size, one for the foreground and the other one for the background.

On our side, we address the regularization problem by using polynomial motion models. Of course, parametric motion models may lack expressiveness. However, we claim that it is a well-funded choice for the task of motion segmentation. Indeed, there are clear relationships between 2D parametric motion models in the image and 3D motion in the viewed scene [START_REF] Bouthemy | Motion segmentation and qualitative dynamic scene analysis from an image sequence[END_REF]. For instance, the 8-parameter quadratic motion model corresponds to the projection into the image of the rigid motion of a planar surface. Relying on those simple parametric motion models, our method achieves excellent performance as demonstrated in Table 3.2.

Compared to CIS, our method bypasses the difficulty to train a complicated inpainting generative model. Compared to MoSeg, it is not attached to a dedicated architectural structure and enables light and efficient architectures for motion segmentation. In addition, the use of parametric motion models for flow representation and of a simple network structure makes the training easier. We advocate that this choice allows for a better generalisation. We consider this point as a critical feature in real-world applications and highlighted it in our evaluation protocol (Section 2.4.2).

Additionally, CIS and MoSeg methods are devoted to a two-mask configuration (moving object vs. background), whereas our method can deal with multiple motions by design. Indeed, the use of parametric motion models allows us to perform multilayer motion segmentation (Fig. 4.7, Table 2.5), with for example the segmentation of the background moving objects and of articulated motions.

Partial Conclusion

We have defined an original, real-time, unsupervised method for motion segmentation taking optical flow as input. We leveraged the EM paradigm to define a mathematically well-founded loss function and the training stage of our neural network. No manual annotation is required. We have also designed an effective and simple data augmentation scheme adapted to optical flow fields. In contrast to the classical EM algorithm, our Part , Chapter 2 -EM-driven Unsupervised Learning for Motion Segmentation method is not iterative at the inference stage and does not need to estimate parametric motion models at test time. In addition, our OFS method can handle by design the segmentation of multiple motions.

Our method outperforms state-of-the-art comparable unsupervised methods on the DAVIS2016 and FBMS59 benchmarks, and is the second best for SegTrackV2 and MoCA, yielding the best overall performance. Additionally, the version with more than two masks opens a promising perspective on this point. Interestingly, our OFS method implicitly provides rather time-consistent segments.

In the two next chapters, we will explicitly and deeply handle the temporal dimension of the motion segmentation problem. For the sake of brevity, we will call from now our EM-driven method the EM method.

Chapter 3

TEMPORALLY-CONSISTENT SEGMENTATION OF MULTIPLE MOTIONS

Temporal consistency is a key feature of motion segmentation, but it is often neglected. In this chapter, we propose an original unsupervised spatio-temporal framework for motion segmentation from optical flow that fully investigates the temporal dimension of the problem. More specifically, we have defined a 3D network for multiple motion segmentation that takes as input a sub-volume of successive optical flows and delivers accordingly a sub-volume of coherent segmentation maps. In this chapter, we start with a short-term temporally-consistent motion segementation, meaning that the input sub-volume remains limited (typically, three optical flow fields). The next chapter, Chapter 4, will deal with long-term temporally-consistent motion segmentation.

Our network is still trained in a fully unsupervised way, and the loss function combines a flow reconstruction term involving spatio-temporal parametric motion models, and a regularization term enforcing temporal consistency on the masks. We have specified an easy temporal linkage of the predicted segments. Besides, we have proposed a flexible and efficient way of coding U-nets.

Temporal Dimension of the Problem

Clearly, the motion segmentation problem has a strong temporal dimension, as motion is generally consistent throughout the video, at least in part of it within video shots. The use of one optical flow field at each given time instant may be sufficient to get the segmentation at frame t of the video. However, extending the temporal processing window can be beneficial. Introducing temporal consistency in the motion segmentation framework is certainly useful from an algorithmic perspective: it may allow to correct local errors or to predict the segmentation map at the next time instant. Beyond that, temporal consistency is an intrinsic property of motion that is essential to involve in the In this chapter, we propose an original method for multiple motion segmentation from optical flow, exhibiting temporal consistency, while ensuring accuracy and robustness. To the best of our knowledge, our optical flow segmentation (OFS) method is the first one to involve short-and long-term temporal consistency. We are considering a fully unsupervised method, which overcomes tedious or even unfeasible manual annotation and provides a better generalization power to any type of video sequences.

The main contributions of our work are as follows. We adopt an explicit space-time approach. More specifically, our network takes as input a sub-volume of successive optical flows and delivers accordingly a sub-volume of coherent segmentation maps. Our network is trained in a completely unsupervised manner, without any manual annotation or ground truth data of any kind. The loss function combines a flow reconstruction term involving spatio-temporal parametric motion models defined over the flow sub-volume, and a regularization term enforcing temporal consistency on the masks of the sub-volume. Our method also introduces a latent represention of each segment motion and enables an easy temporal linkage between predictions. In addition, we have designed a flexible and efficient coding of U-nets.

The rest of the chapter is organized as follows. Section 3.2 is devoted to related work specifically concerned with the temporal dimension. In Section 4.2, we describe our unsupervised 3D network for multiple motion segmentation embedding temporal consistency. Section 4.3 collects details on our implementation. In Section 4.4, we report results on several VOS benchmarks with a comparison to several existing methods.

Additional Related Work

Related work on motion segmentation was globally addressed in Chapter 1. Here, we add comments on references specifically concerned with the temporal dimension.

The temporal dimension of the motion segmentation problem has been somewhat considered in various ways. First, regarding classical approaches, in [START_REF] Odobez | MRF-Based motion segmentation exploiting a 2D motion model robust estimation[END_REF] the motion partition at time t was predicted from the one obtained at time t -1 using the affine motion models estimated between images t -1 and t for each segment, within a robust MRF-based method. The authors in [START_REF] Sun | Layered segmentation and optical flow estimation over time[END_REF] showed that it was beneficial to introduce temporal layer constancy over several frames to perform motion segmentation in a MRFbased and graph-cut optimization framework. In [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF], large time windows were taken into

Additional Related Work

account, allowing the use of point trajectories within a spectral clustering method but resulting in sparse displacement fields.

More recently, regarding deep learning approaches, segmentation at a given time instant t is enforced during the training phase of the network designed in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], by considering several time pairs involving instants before and after t and their corresponding optical flow fields. In [START_REF] Lamdouar | Segmenting invisible moving objects[END_REF], the scope is a bit different, since the authors deal with amodal segmentation, i.e., the recovery of the whole object even in case of occlusion or temporary static state. To this end, they introduce a multi-frame analysis comprising a transformer encoder, while using synthetic ground truth for training involving human annotation. In the same vein, multiple object segmentation is addressed in [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] with the addition of depth-ordered layer representation. A self-supervised model for VOS has been very recently proposed in [START_REF] Ding | Motion-inductive self-supervised object discovery in videos[END_REF], taking several consecutive RGB frames as input. Optical flow is computed at training time. Furthermore, a temporal consistency term is added in the loss function. However, the temporal consistency is not applied to two consecutive segmentation maps, but for different pairings between frame t and another (more or less distant) frame. In [START_REF] Duke | SSTVOS: Sparse spatiotemporal transformers for video object segmentation[END_REF], temporal feature propagation is an important component of the framework of spatio-temporal transformers designed for video object segmentation.

Another way to integrate the evolution of the video is to involve memory modules, as in [START_REF] Tokmakov | Learning motion patterns in videos[END_REF] where a two-stream neural network, encompassing spatial and temporal features, is equipped with an explicit memory designed with convolutional gated recurrent units. Memory networks that can be trained end-to-end are leveraged in [START_REF] Oh | Video object segmentation using space-time memory networks[END_REF] for semi-supervised VOS. The memory is fed by frames with object masks and can be dynamically updated. Memory is introduced through a recurrent network for zero-shot VOS in [START_REF] Ventura | RVOS: End-to-end recurrent network for video object segmentation[END_REF]. It is fully end-to-end trainable and involves both the spatial and temporal domains.

Video prediction is a topic of growing importance surveyed in [START_REF] Oprea | A review on deep learning techniques for video prediction[END_REF], which we can also mention in this discussion. The objective is to forecast future frames of a video sequence from several past frames. The authors of [START_REF] Van Amersfoort | Transformation-based models of video sequences[END_REF] concentrate on the prediction of the transformations between successive images, represented by affine models, to generate the next frame of the sequence. In [START_REF] Gao | Disentangling propagation and generation for video prediction[END_REF], the proposed framework handles in different ways predictable moving regions and disoccluded regions, from a confidence factor evaluated after warping. The latter regions are predicted by a dedicated inpainting network. Motion related to actions is predicted from previous frames in [START_REF] Finn | Unsupervised learning for physical interaction through video prediction[END_REF] in the context of robot interactions, which enables to be partially invariant to object appearance. In [START_REF] Luc | Predicting future instance segmentation by forecasting convolutional features[END_REF], convolutional features of the Mask R-CNN instance segmentation model are predicted to produce the segmentation of future frames. Other works proposed LSTM networks for semantic segmentation [START_REF] Nabavi | Future semantic segmentation with convolutional LSTM[END_REF], and local frequency domain transformer networks [START_REF] Farazi | Local frequency domain transformer networks for video prediction[END_REF]. In a different perspective, the prediction of probable motion patterns is used at the training stage in [3], as a cue to learn objectness from videos.

Our fully unsupervised approach differs from all these previous works in several respects. We rely only on optical flow and take a sub-volume of OF fields as input, providing a sub-volume of consistent segmentation maps. Moreover, we introduce space-time parametric motion models and temporal consistency between consecutive masks.

Temporally-consistent Motion Segmentation Method

We have designed a 3D convolutional network for multiple motion segmentation from optical flow. The network takes a sub-volume of several successive optical flow fields as input. Temporal consistency is expressed in two main ways at the training stage. Firstly, we introduce space-time parametric motion models to represent the flow in each segment over the space-time sub-volume. Secondly, we define a regularization term in the loss function enforcing stable labeling of the motion segments over the sub-volume. We call ST-MS this new motion segmentation method.

Spatio-temporal parametric motion model

For each motion segment k from a set of K segments, we define a spatio-temporal parametric motion model fθ k ,α k in the (x, y, t) volume of the sequence, introducing a temporal variation of each spatial parameter of the model. For instance, if we consider an affine spatio-temporal motion model, it is given by:

f θ k ,α k (x, y, t) = (θ k 1 + α k 1 t + (θ k 2 + α k 2 t)x + (θ k 3 + α k 3 t)y, θ k 4 + α k 4 t + (θ k 5 + α k 5 t)x + (θ k 6 + α k 6 t)y) T , ( 3.1)
where θ k = (θ k 1 , .., θ k 6 ) corresponds to the spatial part of the motion model, and α k = (α k 1 , .., α k 6 ) its temporal extension to account for possible variations of the flow over time.

By developing eq.(3.1), an equivalent formulation is:

f θ k ,α k (x, y, t) = (θ k 1 + θ k 2 x + θ k 3 y + α k 1 t + α k 2 xt + α k 3 yt, θ k 4 + θ k 5 x + θ k 6 y + α k 4 t + α k 5 xt + α k 6 yt) T . (3.2)
Similar expressions can be straightforwardly defined for any spatio-temporal quadratic motion model. For the full quadratic motion model with 12 parameters, we get:

fθ k ,α k (x, y, t) = (θ k 1 + α k 1 t + (θ k 2 + α k 2 t)x + (θ k 3 + α k 3 t)y + (θ k 4 + α k 4 t)x 2 + (θ k 5 + α k 5 t)xy + (θ k 6 + α k 6 t)y 2 , θ k 7 + α k 7 t + (θ k 8 + α k 8 t)x + (θ k 9 + α k 9 t)y + (θ k 10 + α k 10 t)x 2 + (θ k 11 + α k 11 t)xy + (θ k 12 + α k 12 t)y 2 ) T . (3.3)
The spatio-temporal motion model encompasses the successive locations of segment k in the sub-volume. In practice, we take a sub-volume of three flow fields sampled every τ time instants, i.e., at t -τ, t, and t + τ . A typical value for τ is 1, that is, a triplet of three consecutive flows, but, other values can be chosen. If τ is positive, respectively negative, it means that we proceed forward, respectively backward, in time. We can cope with a single τ value, or with several ones jointly, all the triplets being centered on t. Other types of sub-volumes could be handled as well.

Network architecture

The overall principle of our unsupervised 3D multiple motion segmentation network is illustrated in Fig. 3.1. It is based on the U-net architecture [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. The network takes a sub-volume of flow fields as input around time t and jointly predicts the sub-volume of segmentation maps, while enforcing temporal consistency on them. One possibility is to keep only the segmentation map m t , and the process is performed again at the next time instant t + 1. However, alternatives can be considered, as keeping the three maps.

Loss function

The loss function of our 3D motion segmentation network is composed of two terms: a segment-wise flow reconstruction term and a temporal consistency one on the predictions. The first term, denoted L r , expresses how the estimated parametric motion models fit the input optical flow within each segment. It writes:

L r = 1 3 s∈{-τ,0,τ } i∈I K k=1 m k (i, t + s)∥f (i, t + s) -fθ k ,α k (i, t + s)∥ 1 i∈I ∥f (i, t + s)∥ 1 , ( 3.4)
where i = (x, y) is a site of the image grid I, K is the number of motion layers or segments, f (i, t) is the flow vector at site i and time instant t, and m k (i, t) denotes the probability of site i to belong to motion segment k at time t, that is, the prediction (or output) of the motion segmentation network. f t will designate the optical flow field at time t, f t = {f (i, t), i ∈ I}. We use the robust norm L 1 to overcome the presence of outliers in the motion segment, especially at the beginning of the training when segments are not yet well extracted, and to mitigate possibly wrong flow vectors, around motion discontinuities in particular.

The second term, denoted L c , enforces temporal consistency of the motion segments. To do this, the probability of site i to belong to segment k is assumed to be stable over time within the considered triplet. We have:

L c = 1 2K|I| i∈I K k=1 (|m k (i, t -τ ) -m k (i, t)| + |m k (i, t) -m k (i, t + τ )|), (3.5)
where |I| designates the number of sites over the image grid I. For the sake of simplicity, we have adopted an Eulerian standpoint [START_REF] Batchelor | An Introduction to Fluid Dynamics[END_REF], that is, we compare segment labels over time at any given site i of the image grid. In fact, every point is likely to move and a Lagrangian standpoint [START_REF] Batchelor | An Introduction to Fluid Dynamics[END_REF] would be more appropriate. It would require the use of the optical flow vectors to track every point over time. However, the computed flow field may be imprecise or even erroneous at some points, and besides, interpolation operations would be necessary since the flow components take on real values. The Eulerian temporal constraint does not mean that the site should be static within the triplets, but it works as long as the site lies on the overlap of the successive positions of the moving parts. However, it does not make sense on occluded or disoccluded parts. This justifies the use of the L 1 norm to deal with the latter configuration.

We further prevent from enforcing the temporal consistency over occlusion areas by ignoring sites i in the summation over I in eq.(4.6) that exhibit a large temporal flow difference. More precisely, we set a threshold λ so that a quantile η of sites i is discarded as follows:

p(∥f (i, t + τ ) -f (i, t)∥ 1 ≥ λ) ≤ η. ( 3.6)
In practice, we take η = 1%. In doing so, we make an implicit assumption on the overall surface of the occlusion areas, but it seems reasonable for the datasets we deal with. The loss function is the sum of the two terms:

L = L r + βL c . (3.7)
We simply set β = 1, since the two terms of the loss function are properly normalized.

With our approach, we can easily infer a temporal linkage between successive predictions, as explained below.

Principle of segment selection for evaluation

To evaluate our method and compare it with similar unsupervised methods, we use VOS benchmarks as a substitute for optical flow segmentation (OFS) benchmarks, as no such benchmarks are available. The two tasks are close. However, the VOS one is attached to the notion of a primary object of interest moving in the foreground (sometimes, a couple of objects). As a consequence, we have to select the right segments to cope with the binary ground truth of the VOS benchmarks, as described below.

First, we link throughout the video the K segments obtained at each instant t. The fact that we proceed with sub-volumes imposing common segment labels within the subvolume, and that consecutive sub-volumes share two masks, helps us establish the temporal linkage. We link segments from one time instant to the next one, using the IoU measure (intersection over union) as linkage criterion. This is achieved throughout the video by sub-sequences, and then, the comparison of the K segments with the ground truth is done at this sub-sequence level, enforcing the temporal dimension of our approach. A detailed description on how to link predictions and select the segments for evaluation is provided in the next subsection.

This procedure is applied on the three datasets DAVIS2016, SegTrackV2 and FBMS59. In practice, we take sub-sequences of 10 frames. The segment association, required to compute the Jaccard score, leverages the ground truth, but with the notable fact that it is done only once at the sub-sequence level, which shows the ability of our method to supply long-term stability.

Let us mention that we are able to infer another information related to motion. We can generate a latent representation of the segment motion. More specifically, this latent representation χ(S i ) of segment S i is defined as the average value of the latent vectors, normalized in mean and variance, of the segment sites. In future work, we could establish a motion similarity measure between two segments S i and S j , given by the dot product of their respective latent representations χ j and χ j . This motion similarity could be used in the temporal linkage of the segments in addition to IoU, or to merge segments, especially if we take a large value for the mask number K. More details on the different items presented in this subsection are provided in Appendix 7.2.

Linking predictions and selecting segments

We detail the procedure for linking predictions in a subsequence and for selecting optimal segments for evaluation. Subsequences are composed of T + 1 consecutive flow fields. Let us define:

mt (i) = arg max k=1,..,K m k (i, t), ( 3.8)
where m k (i, t) is the probability of site i to belong to segment k at time t. Let mt be the segmentation map at time t encompassing the (up to) K segments predicted by the network, mt = { mt (i), i ∈ I}. In other words, mt is the label array representing the set of segments extracted at time t, segments being (non necessary connected) layers. We will also use the term mask to designate mt , when no confusion can occur. The prediction of the network is given for a triplet of input flow fields (f t-1 ; f t ; f t+1 ) as a triplet of masks ( mt-1 ; mt ; mt+1 ) for τ = 1. All those triplet predictions are produced in parallel and the triplet output are independent as illustrated in Fig. 3.2.

Linking predictions

The masks in the same triplet are sharing common labels but not necessary across triplets. By label, we mean mask number. We have to link the labels by finding correspondences between triplets. Since we have three versions of the same mask mt , it is Part , Chapter 3 -Temporally-consistent Segmentation of Multiple Motions straightforward to achieve it.

First, we need to introduce an additional notation mt ′ t as defined below. The segmentation mask mt ′ t (t ′ ∈ {t -1, t, t + 1}) of width W and height H, consisting of K non-overlapping classes ( mt ′ t ∈ {1, .., K} W ×H ) and corresponding to flow f t , is the one predicted by the network when it takes a triplet centered around t ′ as input. We have to find the best label association between instances mt ′ t of the same mask. To do that, we compute a label reassignment table that will be applied to the two masks mt t and mt t-1 . The reassigned label l * for each label l ∈ {1, .., K} is given by:

l * = arg max k∈{1,..,K} J(k t-1 t , l t t ) + J(k t-1 t-1 , l t t-1 ), (3.9)
where J is the IoU score between two segments. The binary array k t ′ t is defined as follows:

k t ′ t (i) =    1 if mt ′ t (i) = k 0 otherwise (3.10) l t ′
t is defined in a similar way. We proceed by pairs, since two consecutive triplets share two masks as illustrated in Fig. 3.3 (the pairs of arrows). Starting from t = 0, we propagate the labels to the whole subsequence using the criterion of eq.(3.9). After the label reassignment propagation, we rename each mt ′ t as mt ′ t .

Figure 3.3 -Propagation of the labels (i.e., masks numbers) over the subsequence.

Select optimal labels

We select an optimal set of segments for evaluation, based on the ground truth, for the entire subsequence. The goal is to show that, since we have coherent labels within the subsequence, we can select the optimal segments at the subsequence level. First, we unroll our sequence by only keeping the central prediction for each step from t = 1 to t = T -1, and just retrieving the single instance produced for the first (t = 0) and last (t = T ) time steps as depicted in Fig. 3.4. Figure 3.4 -Formation of the mask series { mt , t = 0, .., T } over the subsequence for evaluation , once the label propogation step is achieved.

Then, for the entire subsequence, we select the subset S * of labels (that is, of segments) to constitute the predicted foreground, using the binary ground-truth g t . The selected label subset S * is given by:

S * = arg max S⊂P({1,..,K}) T t=0 J( l∈S lt , g t ), (3.11)
where P({1, .., K}) is the partition of the labels in the subsequence, and the binary masks lt correspond to the label mask mt . Once we have selected the subset S * of labels, we can use it to build our binary segmentation {s t , t = 0, .., T } on the whole subsequence for evaluation, as illustrated in Fig. 3.5.

Impact of subsequence length

In the temporal segment linkage and the segment selection process described above, we take into account a subsequence of length T + 1. For the results reported below, we use a subsequence length of 10 frames. However, we can vary the subsequence length to evaluate the robustness of our method to this parameter. All the evaluations regarding the impact of the subsequence length, are produced from the same trained network and initial segmentation. They are plotted in Fig. 3.6. Longer subsequences are more challenging since they require a stronger temporal consistency. Also, they can be more impacted by occlusions or flow estimation errors, which can break label propagation.

We can see that our method is robust to the choice of the subsequence length, and that it is performing well on long subsequences as well.

Implementation

Network coding

In this work, we explored different ways of dealing with the input optical flow volume and especially different interactions between time steps. We also wanted to use a multi-resolution structure to segment large inputs while preserving fine-grain details. In order to easily handle these different options, we developed an original and flexible implementation 1 of a U-net based on the decomposition of its structure into five modules This framework makes it straightforward to implement a multidimensional U-net, to incorporate various transformations in the transit layer (e.g., transformer as in [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] or recurrent CNN as in [START_REF] Nabavi | Future semantic segmentation with convolutional LSTM[END_REF]), or to change the sampling or upsampling steps, keeping the same general skeleton between all these different architectures. For example, in our work, we implemented a version of downsampling and upsampling that is applied only on the spatial dimension, while double convolutions are applied on both the spatial and temporal dimensions. The proposed implementation encompasses many of the solutions described in Section 3.2, and allows new ideas to be tested quickly and in a standardized way. It is also applicable beyond motion segmentation. We have make our code available in an open source repository2 . Figure 3.7 -Diagram of the prototype layer in our implementation of the U-net. "Double-Conv Left" and "Double-Conv Right" are applying several transformations of the feature maps after downscaling and before upscaling respectively. In classical U-net architecture, it is a 2D convolution kernel applied on the spatial dimension followed with a batch norm. "Down" is a downscaling layer that reduces the spatial dimension of the feature map. In classical U-net architecture, it is implemented using max pooling 2D. "Up" is a block that increases the dimension of the feature map usually implemented with bilinear interpolation or transposed convolution. "Transit" is the connection between the down path and the up path. In classical U-net architecture, it is a skip connection.

Implementation details

As in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF], we adopt the RAFT method [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF] to compute the optical flow fields. More specifically, we use the RAFT implementation3 with network weights fine-tuned on the MPI Sintel dataset [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF]. We downsample them to feed the network with 128×224 vector fields as input. Thus, we achieve much more efficient training and inference stages. The output segmentation maps are then upsampled to the original frame size for evaluation w.r.t. the ground truth.

Regarding the estimation of the spatio-temporal parametric motion model, since the x and y coordinates are normalized within [0, 1], we apply a similar normalization for the t 3.4. Implementation coordinate. For instance, if we set τ = 1, we get as normalized time values: t -1 = -0.33, t = 0, and t+1 = 0.33. We use the full quadratic motion model, with 12 spatial parameters and 12 temporal ones, in all the reported experiments. This type of motion model enables to account for complex depth surfaces and movements.

We use only the prediction m k (i, t) of the network obtained when considering the flow triplet (f t-τ , f t , f t+τ ) to decide to which segment k site i belongs to at time instant t. More precisely, we select for each point i the segment k with the highest probability. In all the experiments reported in Section 4.4, we simply use a single value for τ , τ = 1. We refer the reader to Appendix 7.2 for possible alternatives. Let us recall that negative values of τ mean that we proceed backward in time. A combinaison of several τ values could also be used .

Data augmentation and network training

We perform two types of data augmentation. The first one consists in adding a global flow to the input flow as done in [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF]. The global flow is given by a full spatio-temporal motion model whose parameters are chosen at random. We just make sure that the added global flow is roughly equivalent in magnitude to the initial flow. The same global flow is added to the three flow fields of the input sub-volume. This type of data augmentation allows us to mimic different camera motions, enforcing that the motion segments are independent of it. For the second type of data augmentation, we corrupt one input flow out of the three ones. The idea is to simulate a poorly estimated flow field and to compel the temporal consistency to compensate.

Our motion segmentation method is entirely unsupervised. We do not perform any manual annotation in all the experiments. We train the 3D motion segmentation network on the training set of DAVIS2016, once for all. Moreover, the stopping epoch is selected from the loss function evaluated on the DAVIS2016 validation set. We use Adam optimizer with a learning rate of 10 -4 to train the 3D network. The estimation of the parameters {θ k , α k , k = 1, K} of the motion models is achieved with the Pytorch implementation of L-BGFS [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF]. Let us recall that we estimate the parametric motion models only at training time.

Our method is very efficient at test time. For the model (small 3D U-net), the computational time amounts on average to 114 fps on a GPU P100. The impact is negligible regarding the number K of masks used since only the final layer is modified, and it is proportional to the frame size |I|.

Part , Chapter 3 -Temporally-consistent Segmentation of Multiple Motions

Network

No

Experimental Results

Datasets

We have carried out experiments on three VOS datasets: DAVIS2016 4 [15], Seg-TrackV2 5 [START_REF] Li | Video segmentation by tracking many figure-ground segments[END_REF], and FBMS59 [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF].

DAVIS2016 consists of 50 videos (and 3455 frames) that are split in a training set of 30 videos and a validation set of 20 videos. They contain diverse moving objects. Only the primary moving object is annotated in the ground truth. The criteria for evaluation on this dataset are the Jaccard score (denoted J ), and the contour accuracy score (denoted F).

SegTrackV2 includes 14 videos (with a total of 1066 annotated frames), and FBMS59 contains 59 videos (720 annotated frames), both involving one moving object but sometimes a couple of moving objects. For FBMS59, we use the 30 sequences of the validation set for evaluation. As done in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], if there are several moving objects, we group them into a single foreground mask for evaluation.

Ablation study

We have conducted an ablation study to assess three main components of our method, two of them being related to the temporal dimension. We proceeded by modifying only one component at a time. The three network components concerned are: i) no use of any data augmentation, ii) use of a spatial quadratic motion model per frame instead of the spatio-temporal one, iii) specification of the loss function without the consistency term 4. https://davischallenge.org/index.html 5. https://paperswithcode.com/dataset/segtrack-v2-1 100 Table 3.2 -Results obtained with our method (K = 4) on DAVIS2016, SegTrackV2, and FBMS59, including comparison with unsupervised and supervised methods (scores from cited articles). The Jaccard index J expresses the correct overlap (intersection over union) between the extracted segments and the ground truth, while F focuses on segment boundary accuracy (the higher the better). Performance is assessed by the average score over all samples, for all datasets but DAVIS2016. For the latter, the overall score is given by the average of sequence scores. *OCLR is not a truly unsupervised method since it relies on human-annotated sprites to get realistic shapes in the synthetic data used in the training.

Experimental Results

Method

L c . All the ablation experiments were run on the DAVIS2016 validation set, SegTrackV2 and FBMS59 datasets. Results are collected in Table 7.12. We can observe that the spatio-temporal motion model improves the performance of the method, by taking into account the possible motion evolution within the sub-volume. Above all, the introduction of the temporal consistency term L c in the loss function is drastically beneficial. Its impact appears far weaker for FBMS59, but this is due to the evaluation procedure, since the evaluation is performed only every ten frames. The ablation study demonstrates the pivotal role of the two components acting at two levels of temporal consistency in the flow segmentation Figure 3.8 -Results obtained with our method using four masks (K = 4), but the network may not necessarily use all the four masks. Two groups of results. For each group, the first row depicts one image of the video, the second row contains the optical flow input represented with the usual HSV color code, the third row displays motion segments (given by layers that are not necessarily connected) with one colour per segment. Samples are drawn from the different datasets.

Quantitative and comparative evaluation

We report in Table 3.2 the results obtained by our method on the three datasets DAVIS2016, SegTrackV2 and FBMS59, along with those obtained by other existing methods. Since our method is fully unsupervised and only uses optical flow as input, we focus on similar methods for a fair comparison. We consider the method categories that we proposed in [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF] regarding input and training, by the way very close to other propositions. Our previous motion segmentation method, presented in Chapter 2, will be called EM method from now. We have added a category w.r.t. the network input for two very recent methods, [3], [START_REF] Ding | Motion-inductive self-supervised object discovery in videos[END_REF], that only use RGB images as imput at test time, the optical flow being only involved in the loss function. Additionally, the OCLR method [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] exploits humanannotated sprites to generate realistic shapes in the synthetic data used in the training. We consider the OCLR version taking only optical flow as input. The post-processing added to the CIS method [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], based on Conditional Random Fields (CRF), is an heavy one, which leads most authors to retain only the version without post-processing for a fair comparison.

Overall, our method shows convincing performance w.r.t. comparable methods, namely, Figure 3.9 -Impact of temporal consistency on several situations with our method (K = 4). Two groups of results from top to bottom. First group: two cases of amodal segmentation, in fact corresponding to a repeated image in the video file mimicking a stop (first row contains optical flow input, second row displays motion segments). Second group: one case exemplifying the action of the temporal consistency term L c of the loss function (short term) and of the temporal linkage (long term) to maintain the same mask labels over time (first row contains optical flow input, second row displays motion segments obtained without L c , third row includes motion segments obtained with L c and temporal linkage).

unsupervised methods taking optical flow as input. Temporal consistency was properly handled by our method and gave quite satisfying results. More specifically, our method shows an excellent performance on DAVIS2016 and a very good performance on FBMS59. Regarding SegTrackV2, this dataset includes sequences filmed with a poorly controlled handheld camera, which leads to unstable sequences where the contribution of our method cannot be as significant. In addition, information is provided in Appendix 7.3 on the repeatability issue.

Qualitative visual evaluation

Fig. 4.7 contains several visual results to demonstrate how our method behaves on different situations. We display result samples obtained on different videos of the benchmarks. We can observe that the segmentation are globally accurate. Since our method can involve K masks, we can properly handle articulated motions, or the presence of several moving objects in the scene, as illustrated in Fig. 4.7. We must keep in mind that our actual target is the OFS task, even if we evaluate our method on VOS benchmarks. Since the VOS benchmarks mainly deal with the segmentation of one primary object moving in the foreground, it may occur some discrepancies with OFS. For instance, the segmentation of additional parts w.r.t. VOS ground truth makes nonetheless sense from the OFS standpoint. Let us mention the cases of a moving car in the background, two animals running, ripples on the water, motion parallax due to static objects in the foreground, as illustrated in several examples of Fig. 4.7. It can affect the overall scores reported in Table 3.2.

We gather in Fig. 3.9 several result samples that demonstrate the benefit of the shortterm and long-term temporal consistency provided by our method, with respectively the L c term of the loss function defined in eq.(4.6) and the temporal linkage described in Subsection 4.2.4. Our method is able to recover the moving object segment when the object is temporarily static, showing its ability to segment amodally without any dedicated training, as shown in the first group of Fig. 3.9. The second group highlights how our method can maintain the same mask labels in the video.

Results on Davis2017-motion

In addition to the datasets (DAVIS2016, FBMS59, SegTrackV2), we have evaluated our method on the DAVIS2017-motion dataset. DAVIS2017 [START_REF] Pont-Tuset | The 2017 DAVIS Challenge on Video Object Segmentation[END_REF] is an extension of DAVIS2016 dataset that includes additional videos with multi-object contents, resulting in multiple-segment annotations for the ground truth. It contains a total of 90 videos, split into 60 for training and 30 for validation. DAVIS2017-motion is a curated version of the DAVIS2017 dataset performed by the authors of [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] for a fair evaluation of motion segmentation based on flow information only, where connected objects sharing common motion are merged in the ground truth of the validation test. Table 3.3 -Comparative evaluation on the DAVIS2017-motion validation set. The Jaccard index J expresses the correct overlap (intersection over union) between the extracted segments and the ground truth, while F focuses on segment boundary accuracy (the higher the better). J &F is the mean of the two. Evaluation is performed on the video as a whole, and reported scores are the average of the individual video scores. We evaluated our method on the validation set using the official DAVIS-2017 evaluation algorithm that involves a Hungarian matching process over the sequence. Results are collected in Table 3.3. On this dataset, our method has a better J &F score than MoSeg [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF] (42.0 vs 35.8), while OCLR flow-only [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] outperforms both (55.1), but OCLR is trained using synthetic data whose generation involves human annotation.

Method / Scores

Differentiation of motion patterns

Network output may be limited to three segments with K = 4, not due to the model itself but to the train set. Indeed, DAVIS2016 train set comprises too few videos with several moving objects. We have noticed in other applications that the network, trained on a dataset involving many moving objects, is producing a number of segments equal to the specified K. Besides, we are not dealing with instance segmentation, but with motion segmentation into layers. Accordingly, objects with the same motion are prone to belong to the same mask (layer), but the decomposition into connected segments could be an easy postprocessing. On the other hand, our method manages to separate objects with different motions, e.g., two cars or two animals in Fig. 4.7. In addition, results (with K = 4) on the goats1 video of FBMS59 dataset are reported in Fig. 3.10.

Partial Conclusion

We have designed an original unsupervised method 6 for the segmentation of multiple motions in a video. It fully leverages the temporal dimension of the motion segmentation problem. To the best of our knowledge, our method is the first unsupervised networkbased OFS method involving short-and long-term temporal consistency, which leads to Part , Chapter 3 -Temporally-consistent Segmentation of Multiple Motions stable OF segmentation along the video. It introduces at training time spatio-temporal parametric motion models in sub-volumes, and a loss term expressing temporal consistency over consecutive masks while taking care of occlusions. In addition, the method allows for an easy temporal linkage of the motion segments throughout the video.

Our 3D network is flexible by design. It can straightforwardly handle different choices of mask number for the multiple motion segmentation. Different flow sub-volumes can be envisaged as input, including forward and backward in time. Besides, we have proposed an efficient way to code U-nets, which can be easily generalized beyond motion segmentation. Experimental results on several datasets demonstrate its efficiency and its accuracy by providing competitive results.

Nevertheless, our temporally-consistent motion segmentation method, if efficient for relatively short input sequences, may be limited for longer input sequences. Indeed, the linear temporal term of the spatio-temporal polynomial motion model may be too simple for longer time intervals likely to involve a different temporal evolution, and other network models as Transformers can handle more distant interactions. These extensions will be investigated in the next chapter.

Chapter 4

LONG-TERM MOTION SEGMENTATION

Human beings have the ability to continuously analyze a video and immediately extract the main motion components. Computer vision methods usually proceed frame by frame. We want to go beyond this classical paradigm and directly segment a video sequence as a whole. In the preceding chapter (Chapter 3), we have investigated a short-term temporally-consistent motion segmentation approach. In this chapter, we extend it to a long-term perspective.

As stressed in Chapter 3, the motion segmentation problem has a strong temporal dimension, as motion is generally consistent throughout the video (or within each video shot for long videos). The optical flow field at time t may be sufficient to get the segmentation at frame t of the video, leading to a motion-based video segmentation frame by frame. However, extending the time window can be beneficial. Beyond that, considering the video clip as a whole and directly providing the segmentation for the overall video clip is more in line with human vision, and should ensure a more reliable and efficient process.

Main Characteristics of the Proposed Method

We propose a novel long-term spatio-temporal model operating in a totally unsupervised way. We use the consecutive optical flow (OF) fields as input to decompose a video sequence into segments of coherent motion. More specifically, we have defined a transformer-based network for multiple motion segmentation that delivers temporallycoherent segmentation maps over the video sequence. It is trained in a completely unsupervised manner, without any manual annotation or ground truth data of any kind. The loss function combines a flow reconstruction term involving spatio-temporal parametric motion models, and a regularization term enforcing temporal consistency between successive masks. We model with B-splines the long-term temporal evolution of the motion parameters, which brings more flexibility than the linear temporal term (first-order Part , Chapter 4 -Long-Term Motion Segmentation polynomial) of the motion model defined for the ST-MS method.

The use of a transformer decoder on the latent space of the 3D Unet allows interactions between inputs separated by arbitrarily temporal distances compared to the simple 3D convolutions that have a fixed receptive field. This is a key aspect of our design as we want to be able to segment long video sequences at test time. The impact of this choice is shown in Table 7.12. Furthermore, the decision to use only a transformer decoder, rather than a combination of encoder and decoder, is also important since it allows us to keep the computational cost and inference time reasonably low. Finally, on the conceptual side, the transformer decoder produces queries that represent each segment along with the segmentation, which is interesting because we might want to use them for downstream tasks (e.g., classification, motion clustering).

Our method also involves a latent represention of the segment motion augmented with positional embedding. In addition, the temporal linkage of the consecutive motion segmentations over a video sequence was designed as a relevant but ad hoc post-processing of the ST-MS method. Now, this post-processing is removed, and all is integrated in a long-term model that is end-to-end trained in a unsupervisedly way.

We will report experiments on four VOS benchmarks (DAVIS2016, DAVIS2017-motion, SegTrackV2, FBMS59). We also highlight through visual results the key contributions on temporal consistency brought by our method. In the following, we will call it LT-MS method.

Long-term Motion Segmentation Method

Recent years have seen the advent of transformer-based networks involving attention mechanisms [START_REF] Han | A survey on vision transformer[END_REF]. Let us recall that, in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], the authors used a transformer module, more specifically, the slot attention mechanism introduced in [START_REF] Locatello | Object-centric learning with slot attention[END_REF]. Divided attention is promoted in [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF]. The resulting DivA method is based on the same principle as in [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] that motion segments are mutually uninformative. However, it is not limited to binary segmentation (dominant moving object vs background), but can segment a variable number of moving objects taking one optical flow field as input, and without the number of moving objects being specified by the user. It leverages a slot attention mechanism guided by the image content through a cross-modal conditional slot decoder.

We have designed a transformer-based network for multiple motion segmentation from optical flow. It is inspired from the MaskFormer architecture [START_REF] Cheng | Per-pixel classification is not all you need for semantic segmentation[END_REF], but it only comprises one head corresponding to the mask prediction, as described in Fig. 4.1. The network takes as input a volume, of flexible temporal length, comprising several consecutive optical flow fields. Temporal consistency is expressed in two main ways at the training stage. Firstly, we associate a space-time motion model with each segment to characterize its motion along with the evolution of the latter over time. Secondly, we define a regularization term in the loss function enforcing stable labeling of the motion segments over the volume.

Spatio-temporal parametric motion model

We consider the (x, y, t)-space formed by a video. The set of τ consecutive frames will be designated as a space-time volume (or volume, to make it short). For short videos, the volume could be the whole video sequence. Our space-time motion model is defined by B-spline functions [START_REF] Unser | Splines: A perfect fit for signal and image processing[END_REF]. We have K motion segments. We assign a spatiotemporal parametric motion model fθ k to each motion segment k, k = 1, .., K. θ k specifies the motion model f for segment k. The motion model involves J parameters and each parameter θ k j , j = 1, .., J, of the model results from a B-spline function of order n in the variable t over the space-time volume. In practice, we take n = 3. The number L of control points is given by L = 2 + ⌊ τ -2 ν ⌋, where ν allows us to set the temporal frequency of control points. We put a control point at both ends of the volume (at t = 1 and t = τ ), and the other control points are equidistantly placed between the two. Most often, the Part , Chapter 4 -Long-Term Motion Segmentation control points are not located at time points of the video sequence.

The space-time spline-based motion model is illustrated in Fig. 4.2. Just for this illustration, the motion models are computed within the two segments, foreground moving object and background, provided by the ground truth. The estimated motion model for the foreground moving object is able to capture the periodic nature of the swing motion as demonstrated by the plots of the computed motion model parameters. Also, the motion model computed in the background segment perfectly fits the camera motion. Let us note that the articulated motion, specifically the woman's legs, should require multiple-motion segmentation to be correctly handled. Any parametric motion model could be considered. We use the 12-parameter quadratic motion model to be able to account for continuously varying depth surface of the objects in the scene, especially for the whole background, and for complex object or camera motions. In contrast, the affine and the 8-parameter quadratic motion models assume a planar object surface. Indeed, the latter exactly corresponds to the projection in the image of the 3D rigid motion of a planar surface. It is equivalent for velocity fields to the homography between planes. However, in presence of severe depth effects (strong depth discontinuities) and camera motion, the backgound cannot be represented by a single motion segment due to motion parallax corresponding to static objects located in the foreground.

The 2D flow vector yielded by the full quadratic motion model at any point (x, y) is given by:

f θ (x, y) = (θ 1 + θ 2 x + θ 3 y + θ 7 x 2 + θ 8 xy + θ 9 y 2 , θ 4 + θ 5 x + θ 6 y + θ 10 x 2 + θ 11 xy + θ 12 y 2 ) T . (4.1)
By arranging the motion model parameters as a 2x6 matrix:

θ = ( θ 1 θ 2 θ 3 θ 7 θ 8 θ 9 θ 4 θ 5 θ 6 θ 10 θ 11 θ 12 ), (4.2)
we can rewrite eq.( 4.1) in a more compact way as a matrix product:

fθ (i) = θ c(i), ( 4.3)
with c(i) = (1, x, y, x 2 , xy, y 2 ) T being the position expansion for site i = (x, y).

Loss function

The loss function of our motion segmentation network is composed of two terms as in the ST-MS method: a segment-wise flow reconstruction term and a temporal consistency one on the predictions. The first term, denoted L r , expresses how the estimated parametric motion models fit the input optical flow within each segment. It writes:

L r = 1 τ τ s=1 i∈I K k=1 m k (i, t + s)∥f (i, t + s) -fSn(θ k ) (i, t + s)∥ 1 i∈I ∥f (i, t + s)∥ 1 , ( 4.4)
where i is a site of the image grid I, K is the number of motion layers or segments, f (i, t) is the flow vector at site i and time instant t, and m k (i, t) denotes the probability of site i to belong to motion segment k at time t, that is, the prediction (or output) of the motion segmentation network. f t will designate the optical flow field at time t, f t = {f (i, t), i ∈ I}. We normalize by the norm of the input flows to hinder the impact of the flow magnitude. By denoting S n (θ k ) the subscript of f , we want to emphasize that the motion model parameters for each segment k are estimated through the B-spline functions. More specifically, we have:

fSn(θ k ) (i, t) = L l=1 fθ k,l (i, t)B n,l (t), (4.5)
where B n,l is the l th B-spline and θ k,l corresponds to the l th control point of the spline function.

We use the robust norm L 1 in eq.( 4.4) to overcome the presence of outliers in the motion segment, especially at the beginning of the training when segments are not yet well extracted, and to mitigate possibly wrong flow vectors, around motion discontinuities in particular.

The second term, denoted L c , enforces temporal consistency of the motion segments. To do this, the probability of site i to belong to segment k is assumed to be stable over time. We have:

L c = 1 2K|I| i∈I K k=1 τ -1 t=1 |m k (i, t) -m k (i, t + 1)|, (4.6)
where |I| designates the number of sites over the image grid I. For the sake of simplicity, we have adopted an Eulerian standpoint [START_REF] Batchelor | An Introduction to Fluid Dynamics[END_REF], that is, we compare segment labels over time at any given site i of the image grid. In fact, every point is likely to move and a Lagrangian standpoint [START_REF] Batchelor | An Introduction to Fluid Dynamics[END_REF] would be more appropriate. It would require the use of the optical flow vectors to track every point over time. However, the computed flow field may be imprecise or even erroneous at some points, and besides, interpolation operations would be necessary since the flow components take on real values. The Eulerian standpoint works on the overlap of the successive positions of the moving parts, but not on occluded or disoccluded parts. This justifies the use of the L 1 norm to deal with the latter configuration.

As in the ST-MS method, we further prevent from enforcing the temporal consistency over occlusion areas by ignoring sites i in the summation over I in rel.4.6 that exhibit a large temporal flow difference. More precisely, we set a threshold λ so that a quantile η of sites i is discarded as follows:

p(∥f (i, t + τ ) -f (i, t)∥ 1 ≥ λ) ≤ η. (4.7)
In practice, we take η = 1%. In doing so, we make an implicit assumption on the overall surface of the occlusion areas, but it seems reasonable for the datasets we deal with. The loss function is the sum of the two terms:

L = L r + βL c . (4.8)
We simply set β = 1, since the two terms of the loss function are properly normalized.

Let us stress that we impose both short-term and long-term temporal consistency. The loss term L c (4.6) primarily expresses a short-term consistency effect by constraining the mask probability m k (i, t) at a given site i between two consecutive time instants t and t + 1. Nevertheless, it also induces a long-term effect by propagation from near to near. The motion model leads to long-term consistency effect by construction, since it is defined over the volume with the spline function. However, it also involves a short-term effect around the control points.

Network architecture

The overall architecture of our unsupervised multiple motion segmentation framework is illustrated in Fig. 4.1. It includes two main modules. The first one, taking the flow subvolume as input, is a 3D U-net [START_REF] Ronneberger | U-net: Convolutional networks for biomedical image segmentation[END_REF]. The latent code augmented with embedding positions is directed to the transformer decoder. Then, by cross-attention, the output formed by the volume of segmentation masks is produced. The training of the overall architecture is based on the minimization of the loss function defined below, while the motion model parameters of the segments are given by the estimation of the B-spline functions. Temporal consistency is provided both by the loss function and the space-time motion models.

Segment selection for evaluation

As we already mentioned in the two preceding chapters, to evaluate our method and compare it with similar unsupervised methods, we use VOS benchmarks as a substitute for optical flow segmentation (OFS) benchmarks, since no such benchmarks are available.

The two tasks are close. However, the VOS one is attached to the notion of a primary object of interest moving in the foreground (sometimes, a couple of objects). As a consequence, we have to select the right segments to cope with the binary ground truth of the VOS benchmarks, as usually done for the DAVIS2016, SegTrackV2 and FBMS59 datsets.

Since we deal with multiple-motion segmentation, i.e., K segments, we have to group them into two clusters corresponding to the foreground moving object on one side and the background on the other side. We proceed as we did in Chapter 3 for our ST-MS method (details given in Annex 7.2, eq. (3.11). However, this time, we can directly evaluate our LT-MS method in one shot over the predicted segmentation sequence, since there is no temporal linking postprocessing for LT-MS.

Regarding the evaluation on the DAVIS2017-motion dataset whose ground truth is not binary, we also achieved it as we did for our ST-MS metod. We use the Hungarian matching method to associate each predicted segment k with the ground-truth annotations, knowing that we have to consider K = 3 masks for this experimentation.

Implementation

Implementation details

Following [3], [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF], [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF], we adopt the RAFT method [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF] to compute the optical flow fields. More specifically, we use on our side the RAFT version trained on the MPI Sintel dataset [START_REF] Butler | A naturalistic open source movie for optical flow evaluation[END_REF]. We downsample the computed flow fields to provide the network with 128×224 vector fields as input. The output segmentation maps are consequently upsampled to the original frame size for evaluation w.r.t. the ground truth. Thus, we achieve much more efficient training and inference stages. We typically use subvolumes of temporal length τ = 9 in the training stage, but, at test time, we can process subvolumes of longer length.

Regarding the estimation of the spatio-temporal parametric motion model, since the x and y coordinates are normalized within [-1, 1], we apply a similar normalization for the t coordinate. In all the reported experiments, we use the full quadratic motion model with 12 parameters, and we set ν = 3 for the frequency factor in the B-spline function.

We use the prediction m k (i, t) of the network to decide to which segment k site i belongs to at time instant t. More precisely, we select for each site i the segment k with the highest probability.

Our LT-MS method is very efficient at test time. The computational time amounts on average to 210 fps on a GPU P100, that is, twice as fast as the preceding ST-MS method. It is certainly due to the long flow sequence given as input to the network, which allows for parallelisation of some heavy computations. In addition, our LT-MS architecture remains light since it combines only three Unet layers and a transformer decoder on the downsampled feature space.

Data augmentation and network training

We perform two types of data augmentation. The first one consists in adding a global flow to the input flow as we did previously for the EM and ST-MS methods. The global flow is now given by a full spline-based spatio-temporal motion model whose parameters are chosen at random. We just make sure that the added global flow is roughly equivalent in magnitude to the initial flow. The same global flow is added to the flow fields of the input volume. This type of data augmentation allows to mimic different camera motions, enforcing that the motion segments are independent of it. For the second type of data augmentation, we corrupt a few input flow out of the nine ones. The idea is to simulate a poorly estimated flow field and to compel the temporal consistency to compensate.

Our motion segmentation method is entirely unsupervised. We do not perform any manual annotation in all the experiments. We train the overall motion segmentation network on the FlyingThings3D (FT3D) dataset [START_REF] Mayer | A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation[END_REF] once for all. This will ensure that our network generalizes well to any unseen datasets. Moreover, the stopping epoch is selected from the loss function evaluated on the DAVIS2016 training set.

We use Adam optimizer with the following strategy on the learning rate to train the network (3D Unet and transformer). We inspired from the warmup-decay strategy in [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] for the learning rate. We linearly increase it from 0 to 1e -4 for 20 epochs then divide it by two every 40 epochs.

The estimation of the motion model parameters through the B-spline function is achieved with the Pytorch implementation of L-BGFS [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF]. Let us stress again that we estimate the parametric motion models only at training time.

Part , Chapter 4 -Long-Term Motion Segmentation

Experimental Results

Datasets

We have carried out experiments again on the four datasets: DAVIS20161 [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF], Seg-TrackV22 [START_REF] Li | Video segmentation by tracking many figure-ground segments[END_REF], FBMS59 [START_REF] Ochs | Segmentation of moving objects by long term video analysis[END_REF], and DAVIS2017-motion [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF].

Let us recall that DAVIS2016 consists of 50 videos (and 3455 frames) that are split in a training set of 30 videos and a validation set of 20 videos. They contain diverse moving objects. Only the primary moving object is annotated in the ground truth. The criteria for evaluation on this dataset are the Jaccard score (denoted by J ), i.e., intersectionover-union, and the contour accuracy score (denoted by F).

SegTrackV2 includes 14 videos (with a total of 1066 annotated frames), and FBMS59 contains 59 videos (720 annotated frames), both involving one moving object but sometimes a couple of moving objects. For FBMS59, we use the 30 sequences of the validation set. Although annotations may comprise multiple objects, the community usually exploits SegTrackV2 and FBMS59 in the same way as DAVIS2016, i.e., benchmarks with a binary ground-truth, by grouping objects in the foreground. We will follow this practice for SegTrackV2 and FBMS59.

As explained in Chapter 3, DAVIS2017 [START_REF] Pont-Tuset | The 2017 DAVIS Challenge on Video Object Segmentation[END_REF] contains a total of 90 videos, split into 60 for training and 30 for evaluation. Let us recall that DAVIS2017-motion is a curated version of the DAVIS2017 dataset performed by the authors of [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] for a fair evaluation of motion segmentation based on flow information only. Connected objects sharing common motion are merged in the ground truth of the validation test. We evaluate our method on the validation set as in [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF], using the official evaluation algorithm that involves a Hungarian matching process.

Ablation study

We have conducted an ablation study to assess three main components of our method LT-MS with four masks (K = 4), in particular related to the temporal dimension of the problem. We have changed one component at a time as specified below:

-use of the polynomial space-time quadratic motion model of ST-MS instead of the space-time motion model based on B-splines over the input sequence, for the background, or the moving objects. For instance, in the fourth example, the foreground moving car is perfectly segmented throughout the sequence along with its wheels, and (small) moving cars in the background. Fig. 4.5 highlights the contribution of the spline-based motion model on the dog video, and its obvious ability to handle motions that do not vary uniformly, as the erratic movement of the dog.

Quantitative and comparative evaluation

We report in Table 4.2 the results obtained by two versions of our LT-MS method on the three datasets DAVIS2016, SegTrackV2, and FBMS59. LT-MS-K2 performs segmentation with only two masks (K = 2) as our EM method, while LT-MS-K4 involves four masks (K = 4) as done for our ST-MS method. Table 4.2 also collects results obtained by our two preceding methods, EM and ST-MS, and other existing methods when available. Since our method is fully unsupervised and only uses optical flow as input, our comments will focus on similar methods for a fair comparison. We follow the categorization we initially proposed in Chapter 2 or in [START_REF] Meunier | EM-Driven unsupervised learning for efficient motion segmentation[END_REF], regarding input and training, by the way very close to other propositions. However, we have added a category w.r.t. the network input for four recent methods, [3], [5], [START_REF] Lian | Bootstrapping objectness from videos by relaxed common fate and visual grouping[END_REF], [START_REF] Ding | Motion-inductive self-supervised object discovery in videos[END_REF], that only use RGB images as imput at test time, the optical flow being only involved in the loss function. Evaluation is performed on the binary ground truth (foreground moving object vs background) for the three datasets.

We still put the OCLR method [START_REF] Xie | Segmenting moving objects via an objectcentric layered representation[END_REF] in the category of unsupervised methods, whereas For each group, the first row contains sample images with the segmentation ground-truth, when available at that frame, overlaid in yellow, the second row displays the input flows, the third and fourth rows show the predicted motion segmentations, respectively without and with the temporal consistency loss term. Clearly, this model component allows us to get far more consistent segments over time. as our EM method, and LT-MS-K4 involves four masks (K = 4) as done for our ST-MS method. We also include comparison with our two preceding methods, EM and ST-MS, and unsupervised and supervised methods (scores from cited articles). All scores regarding DAVIS2016, SegTrackV2, and FBMS59 corresponds to the evaluation on the binary ground-truth. For LT-MS-K4 and LT-MS-K2, we report results obtained with a cut size of 10. The Jaccard index J expresses the correct overlap (intersection over union) between the extracted segments and the ground truth, while F focuses on segment boundary accuracy (the higher the better for both). J &F averages the two. Performance is assessed by the average score over all samples, for all datasets but DAVIS2016. For the latter, the overall score is given by the average of sequence scores. *Actually, OCLR is not a truly unsupervised method, and the authors of DivA do not include OCLR among the unsupervised methods. Indeed, it relies on human-annotated sprites to include realistic shapes of moving objects in the synthetic data used at training time. †DivA also uses images since its decoder network leverages input images (conditional decoder). The authors of [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF] provide results on FBMS59 only for the multi-object setting.

the authors of the DivA method [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF] did not. Indeed, OCLR is not fully unsupervised, since it relies on human-annotated sprites to include realistic shapes in the computergenerated data used at training time. We consider the OCLR version taking only optical flow as input. The post-processing possibly added to the CIS method [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF], based on Conditional Random Fields (CRF), is an heavy one, which leads most authors to retain only the version without post-processing for a fair comparison.

Our LT-MS method (both LM-MS-K4 and LT-MS-K2) delivers very convincing results compared to unsupervised methods taking optical flow as input, especially for the DAVIS2016 dataset that is the main VOS benchmark. 0ur method also performs very well on the two other datasets. It is the second best for FBMS59, outperformed by OCLR. OCLR and DivA demonstrate better performance than LT-MS-K4 on the SegTrackV2 dataset. However, as aforementioned, OCLR is not a fully unsupervised method, while DivA leverages RGB images at some point. Indeed, in DivA, the decoder network leverages input images (conditional decoder). In addition, DivA, along with MoSeg and CIS methods, takes in turn four flow fields as input for a given time instant t, i.e., flows between t and t + 1, t and t + 2, t and t -1, t and t -2, and then, averages the four corresponding predictions to get the final result. Finally, SegTrackV2 includes sequences acquired with a poorly controlled handheld camera, which leads to unstable sequences where the contribution of our method is therefore less likely to be emphasized. We have also tested how our LT-MS-K4 method behaves when varying the length of the input flow sequence. Results are plotted in Fig. 4.6 for three datasets, DAVIS2016, FBMS59 and SegTrackV2. As expected, the best performance is obtained for the smallest length (equal to 10) of the input flow sequence. However, performance decreases slowly when the length increases, and remains stable for larger ones. It demonstrates the intrinsic ability of the LT-MS method to achieve accurate and consistent motion segmentation over long periods of the video, which is a unique property. We also did it for LT-MS-K2 as reported in Table 4.3. We even get slightly better results for DAVIS2016 and SegTrackV2 when the video sequence is processed in one go, i.e., with an infinite cutsize. 4.4 -Comparative evaluation on the DAVIS2017-motion validation set for three masks (K = 3). The Jaccard index J expresses the correct overlap (intersection over union) between the extracted segments and the ground truth, while F focuses on segment boundary accuracy (the higher, the better). J &F is the mean of the two. Evaluation is performed on the video as a whole, and reported scores are the average of the individual video scores. *Actually, OCLR is not a truly unsupervised method, and the authors of DivA do not include OCLR among the unsupervised methods. Indeed, it relies on humanannotated sprites to include realistic shapes of moving objects in the synthetic data used at training time.

Dataset

We have also performed the evaluation of multiple motion segmentation for a multisegment setting. Since multiple-motion segmentation is harder than the binary motion segmentation (moving foreground vs background), accuracy scores are expected to decrease for all methods. In Table 4.4, we report comparative results on the DAVIS2017motion dataset. As for the other methods, we performed segmentation with three masks.

Part , Chapter 4 -Long-Term Motion Segmentation

Method / Scores bIoU Linear Assignment DivA [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF] 42.0 -Ours K=3 58.3 47.2 Ours K=4 56.8 44.6

Table 4.5 -Results on the FBMS59 dataset for multi-object detection using two different evaluation metrics. Bootstrap IoU (bIoU) is a metric described in [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF] where each ground truth mask is mapped to the most likely predicted segment. This mapping is performed at the frame level, and thus, this evaluation actually does not take into account temporal consistency. Linear assignment corresponds to a bilinear mapping between the groundtruth labels and the predicted segments at the sequence level. This metric is similar to the one used in the official DAVIS 2017 evaluation [START_REF] Pont-Tuset | The 2017 DAVIS Challenge on Video Object Segmentation[END_REF].

To this end, we finetuned the LT-MS-K4 network on the DAVIS2016 training set with now three masks (K = 3). The resulting performance is slightly better than ST-MS. In Table 4.5, we report multimask segmentation results for the FBMS59 dataset with two different metrics. Results with K = 4 are obtained with the same network LT-MS-K4 as the one delivering the results reported in Table 4.2. Results with LT-MS-K3 (K = 3) correspond to the model introduced in Table 4.4. Our LT-MS method outperforms the DivA method.

Overall, temporal consistency is properly handled over long temporal periods by our LT-MS-K4 method which delivers excellent segmentation performance. Beyond segmentation performance, we want to stress that our method is the only one providing by design a coherent segmentation over the sequence, which is a significant added value. Thus, we can claim that we have not only segmented the moving objects throughout the sequence, but also achieved some kind of tracking.

Qualitative visual evaluation

Fig. 4.7 contains several visual results to demonstrate how our method behaves on different situations. We display six result samples obtained on different videos of the benchmarks. From top to bottom of Fig. 4.7, we have the motocross-jump video from DAVIS2016, the monkey and hummingbird videos from SegTrackV2, the people2 and goats01 videos of FBMS59, and finally, the libby video from DAVIS2016.

We can observe that the segmentation are globally accurate. Let us note that the ground-truth is not necessarily available for all the video frames depending on the datasets. Since our method can involve several masks, we can properly handle articulated motions Figure 4.7 -Results obtained with our LT-MS-K4 method, i.e., using four masks (K = 4). Six groups of results are displayed. For each group, we have three rows. The first row samples successive flow fields corresponding to the processed video, the optical flow fields being represented using the usual HSV color code. The second row contains the corresponding images of the video, where the ground-truth segment corresponding to the moving object is overlaid in yellow when available at that frame. The third row shows the motion segments provided by our LT-MS-K4 method (given by layers that are not necessarily connected) with one colour per segment. For all the results, we adopt the same color set for the three masks corresponding to the moving objects (blue, red and orange), and we let the background image for the background mask for the sake of clarity. These examples are drawn from the different datasets.

Part , Chapter 4 -Long-Term Motion Segmentation

(hummingbird, monkey, people2 ), deal with the presence of several moving objects in the scene (goats01, hummingbird, people2 ), separate the rider from the motorbike (motocrossjump), or accomodate motion parallax (libby). We must keep in mind that our actual target is the optical-flow segmentation (OFS) task, even if we evaluate our method on VOS benchmarks. Since the VOS benchmarks mainly deal with the segmentation of one primary object moving in the foreground, it may occur some discrepancies with OFS, which negatively impacts the evaluation scores. The segmentation of additional parts, which appears wrong w.r.t. VOS ground truth, on the contrary makes sense from the OFS standpoint.

In Fig. 4.8, we collect additional visual results on three videos of the SegTrackV2 dataset, from top to bottom, the birdfall, bird-of-paradise, and bmx videos. Results demontrate the ability of our long-term segmentation method to recover, to some extent, correct segmentation even from flow fields wrongly computed at some time instants.

Partial Conclusion

We have designed an original transformer-based unsupervised method for the segmentation of multiple motions in a video. It fully acknowledges the temporal dimension of the motion segmentation problem. Indeed, to the best of our knowledge, our method is the first unsupervised network-based OFS method explicitly leading to a stable and consistent OF segmentation throughout long video sequences. It introduces at training time, on one hand, B-splines spatio-temporal parametric motion models over space-time volumes, and on the other hand, a loss term expressing temporal consistency over successive masks while taking care of occlusions. Experimental results on several datasets demonstrate its efficiency and its accuracy by providing competitive results. In addition, our LT-MS method is very fast at test time.

Our transformer-based network is flexible by design in several ways. At test time, it can be applied to input volumes of any time length, in any case longer than the one used at training time. It can accomodate different choices of mask number made by the user before training for multiple motion segmentation. However, changing the number of masks requires to retrain the model. More flexibility could be obtained by introducing a slot attention mechanism, which should allow us to modify the number of masks at test time without retraining the model. We report in Annex 7.3 an extension of the LT-MS model involving slot attention. However, results are not that convincing so far.

GRADIENT-BASED INTERPRETATION OF A FRAME CLASSIFICATION NETWORK FOR MOTION SALIENCY

In this chapter, we propose a new paradigm for the unsupervised computation of motion saliency maps. We estimate these maps from the interpretation of a frame-based motion saliency classification network with optical flow (OF) as only input.

Motion saliency (MS) is an important issue in dynamic scene analysis. We formulate MS as a meta-task that can be instantiated for different tasks usually handled independently. To support this claim, we have addressed two important computer-vision problems with this MS paradigm: independent motion segmentation and anomalous motion detection in videos.

Our paradigm can accommodate a given form of motion saliency by simply training the frame-based classification network on the corresponding task. Moreover, our MS estimation is unsupervised, as it does not require any ground-truth saliency maps for training. We have designed an original two-step network interpretation method that supplies the binary salient motion segmentation. Finally, we recover the valued motion saliency map using a parametric flow inpainting method. Experimental results on real videos and comparison with existing methods will assess the performance of our method.

Introduction and Case studies

Motion saliency (MS) aims to highlight local motions departing from their surrounding context, thus prone to reveal a significant event. MS has many applications in computer vision. It can be useful in the navigation of autonomous vehicles to anticipate moving obstacles, or for public safety to trigger alert in abnormal situations. It can facilitate the subsequent analysis of videos where motion may play the key discriminative role.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

In this chapter, we are concerned with the computation of salient motion maps from the optical flow only. Motion saliency (MS) cannot be formalized as an absolute notion. A motion is not salient in itself, but only with respect to its context, i.e., its surrounding motions or sometimes a predefined normal motion, and to a given application. A motion salient in a given context might not be salient in other contexts. In addition, we do not want to leverage the object appearance in the image. Indeed, the moving entity may not necessarily be a specific object in a scene, and its appearance may not be distinctive from the background content, e.g., someone in a crowd, a cell among others, a cloud among others.

We aim to design a general and versatile method that does not require any supervised training with ground-truth salient motion maps. Its main ingredient is to leverage the interpretation a frame-based motion saliency classification network. As aforementioned, we will handle two different cases of salient motion. The first one is the salient motion produced by independently moving objects in a scene observed by a moving camera.

Since we will rely on the optical flow only, a specific problem arises with a mobile camera: motion parallax attached to static objects being at the forefront of the scene and image motion of independently moving objects in the viewed scene both generate distinguishing patterns in the computed optical flow. Indeed, image motion depends on both the relative 3D motion between scene objects and camera, and on the object depth. We will solve the problem of motion parallax with unknown scene geometry and unknown camera motion.

The second case that we will investigate is salient motion issued from a distinctive motion within a coherently moving set. The set can be typically human crowd, animal herd, bird flock, vehicle traffic, or cell set. An example is the detection of crowd anomaly, where motion saliency comes from a person moving differently than the surrounding ones [START_REF] Georgescu | Anomaly detection in video via self-supervised and multi-task learning[END_REF], [START_REF] Pérez-Rúa | Detection and Localization of Anomalous Motion in Video Sequences from Local Histograms of Labeled Affine Flows[END_REF], [START_REF] Li | Crowded Scene Analysis: A Survey[END_REF].

Our main contributions are the following. We estimate MS from the interpretation of a frame-based saliency classification network that takes optical flow (OF) as input. Our paradigm can accommodate a given form of motion saliency by simply training the framebased classification network on the corresponding task. Moreover, our MS estimation is unsupervised, since it does not require any ground-truth saliency maps for training. In addition, we have designed an original two-step network interpretation method, which supplies the binary salient motion segmentation. Finally, we compute the motion saliency map using a parametric flow inpainting method.

The rest of the chapter is organized as follows. In Section 5.2, we will present our original framework involving several stages. We report experimental results with comparison to recent methods in Section 5.3. Section 5.5 will contain discussion and concluding remarks for this chapter.

Method Description

We proceed in two main stages to estimate motion saliency. First, we segment the salient motions from the two-step interpretation of the classification network that predicts the presence or the absence of salient motion in every frame of a video. Then, we compute the salient motion maps using a parametric flow inpainting technique. In the Results section, we will designate our method by the acronym NIMS (Network Interpretation for Motion Saliency).

Frame-based motion-saliency classification

We describe our frame-based classification network inspired from [START_REF] Maczyta | CNN-Based temporal detection of motion saliency in videos[END_REF]. Basically, it involves the same layers comprising three convolutional blocks and a fully connected layer. However, it takes as input the optical flow field computed between two successive frames, and not a residual flow as in [START_REF] Maczyta | CNN-Based temporal detection of motion saliency in videos[END_REF]. Thus, it is not necessary to compute any dominant motion resulting from the camera motion, which may introduce bias in case of complex background.

The classification network being shallow, the training is fast. For the first studied case of salient motion, it is trained to predict the presence of independent motion in a frame. It is thus implicitly able to discard motion parallax. In practice, we fine-tuned the network of [START_REF] Maczyta | CNN-Based temporal detection of motion saliency in videos[END_REF] using the same training set as in [START_REF] Maczyta | CNN-Based temporal detection of motion saliency in videos[END_REF]. For the second studied case, our network is trained from scratch to detect the presence of distinctive motion in crowds. We used the crowd synthetic dataset introduced in Section 5.3.3.

Our method is versatile in the sense that we just need to re-train the classification network on the task under study to deal with the salient motion of interest. The classification itself is supervised but the annotation process is extremely light. It boils down to labeling at the frame level, and in practice, it can be almost automatically completed, since the training dataset is composed of videos that were either fully dynamically salient or fully non salient.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

We further modify the classification network for the interpretation. Following advice on applying interpretation techniques to deep neural networks [START_REF] Guillemot | Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-Wise Relevance Propagation[END_REF], we modify the ordering of the inner layers as explained in Fig. 5.1, which results in a functionally identical network but able to provide more operable interpretation maps. A mathematical justification of this modification can be formulated as follows. Considering X : {x 1 , x 2 , • • • x n ; x i ∈ R} a local neighborhood of the input, the results of the computation using the network training order (Max Pool, Batch Norm, Relu) is :

x ≜ relu(γ max x i ∈X (x i ) -E r √ V r + β), (5.1)
where relu(x) = max(x, 0). E r and V r are representing respectively the running estimate of the expectation and the variance accumulated during training phase and used as a fixed value during test phase, and γ, β are the scaling and shift parameters of the batch norm layer learned during training. Thus, if the condition γ ≥ 0 holds, which we verified in all our experiments, we can write :

x = max

x i ∈X (relu(γ x i -E r √ V r + β)). ( 5.2)
This second computation is corresponding to the "Interpretation order" (Batch Norm, Relu, Max Pool) described in the paper. In cases where the running estimate of the expectation and the variance are not computed, we can proceed a first forward step through the training network to retrieve the expectation and variance value for a batch, and then, use those values in the interpretation network. Note that this whole manipulation is not necessary if in the original training order the normalisation layer is already adjacent to the foregoing convolutional layer. In this case, we can directly proceed to the interpretation step. After this step, the justification of the fusion between the Convolution and Normalisation layers is given in [START_REF] Guillemot | Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-Wise Relevance Propagation[END_REF].

Inference of interpretation maps for salient motion segmentation

Since the classification network predicts correctly the dynamically salient frames and the non salient ones (with an average precision of about 90% in our experiments), we can deduce that it has learned to distinguish salient moving entities. Then, we can exploit this knowledge to locate the salient motions in the frame. Specifically, interpretation techniques allow one to generate attribution maps from the trained neural network. Those maps outline which parts of the input really contribute to the prediction. However, we are not dealing with images as usual but with 2D optical flow fields. Our goal is to determine for segmentation purpose which vectors of the optical flow field induce the prediction as dynamically salient frame.

Several techniques exist for network interpretation [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF]. However, to the best of our knowledge, such a network interpretation has not yet been investigated for an optical flow input. An important issue is to extract interpretation information linked to the inner workings of the trained network, rather than to extract instance-specific structures information [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF]. Several works such as [START_REF] Adebayo | Sanity checks for saliency maps[END_REF] showed that in some cases, interpretation maps produced with a trained model could be visually similar to the ones generated by a random model. In our case, we did not observe such a behaviour, as shown by experiments reported at the end of Section 5.3 of this chapter. Integrated Gradients method [START_REF] Sundararajan | Axiomatic attribution for deep networks[END_REF] is a technique where maps are generated by cumulating gradients along a linear interpolation between a chosen baseline and a given input. Even though this technique delivers promising results on multiple datasets, results vary widely depending on the chosen baseline [START_REF] Sturmfels | Visualizing the Impact of Feature Attribution Baselines[END_REF]. In the case of images, the default baseline is a black image that represents the absence of features. However, regarding optical flow, there is no obvious choice to represent the absence of salient motion. In our experiments, applying the Integrated Gradients method on optical flow fields mainly led to highlighting areas with large flow magnitude.

First step: point-wise computation of attribution maps

Instead, we have adopted the layer-wise relevance propagation (LRP) method [START_REF] Montavon | Layer-Wise Relevance Propagation: An Overview[END_REF]. It supplies less noisy attribution maps and removes the need to choose a baseline. This technique propagates a relevance score from the output layer to the input of the convolutional network. It relies on purposely designed propagation rules [START_REF] Montavon | Layer-Wise Relevance Propagation: An Overview[END_REF].

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

We have followed the structure presented in [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF], applying the z + rule to all convolutional and fully-connected layers except for the first convolutional layer were we apply the z β rule. The z + rule only takes into account positive weights during the propagation of relevance. The z β rule propagates positive relevance through layers that take as input negative values, by using an additive term for the extremal admissible values of the input space. The expression of these rules are given below.

LRP rules. Following description in [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF], [START_REF] Montavon | Layer-Wise Relevance Propagation: An Overview[END_REF], we compute the importance score from the prediction output to the input layer L = 0, and use the attribution scores (R 0 ) at this layer as the LRP attribution map. Z + rule, given in [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF], [START_REF] Montavon | Layer-Wise Relevance Propagation: An Overview[END_REF], is applied for all linear and convolutional layers except the first :

z + -rule: R L i = j x i w + ij i ′ x i ′ w + i ′ j R L+1 j . ( 5.3)
Z β rule, given in [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF], [START_REF] Montavon | Layer-Wise Relevance Propagation: An Overview[END_REF], is applied for the first convolutional layer to deal with negative values in input flow :

z β -rule: R L i = j x i w ij -lw + ij -hw - ij i ′ x i ′ w i ′ j -lw + i ′ j -hw - i ′ j R L+1 j , ( 5.4)
where x i is the input value at layer L and R L ∈ R I + the relevance score associated to this input map. Weights in the layer L are denoted w ij . We define

w + ij = w ij * Id{w ij > 0} and w - ij = w ij * Id{w ij < 0}.
In contrast to image intensities, the components of the flow vectors are not restricted to a predefined bounded range. Thus, we have to adopt a different normalisation technique. For each input flow x, we compute l and h as the minimal and maximal values of the channel corresponding to each flow component.

Max-pool layers are handled with a winner-takes-all strategy. In our case, we do not need to deal with normalisation layers as we merged them with convolutional layers.

We come out with attribution maps computed on a point-wise basis. However, these maps may be noisy, fragmented or even incomplete. To get an output of the network interpretation usable as salient motion segmentation maps, we need to introduce a second step in the network interpretation process.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

We still will have to select the salient moving regions among the candidate regions, whether it is for the light version or the elaborated one. This will be driven again by the attribution values as follows. As suggested in [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF], the network interpretation must be primarily driven by the strength of attribution values rather than their spatial pattern. Therefore, we precisely take into account the attribution values to assign an attribution score to each candidate regions (i.e., segments of the optical flow). It is given by the ratio between the sum of the attribution values in the segment, and the size of the segment. Afterwards, we compute an adaptive threshold on the attribution score to filter out segments with the lowest scores considered as not salient. In practice, we use the score of the biggest region, assumed to correspond to the background, as threshold. This procedure allows us to select several salient segments if needed. Finally, we get a set R sal of salient moving regions R j .

Estimation of the motion saliency maps

We want now to estimate the motion saliency map. To this end, we will adopt a parametric flow inpainting. More specifically, we will extend within each segmented region of the set R sal the optical flow surrounding this region, and compare it to the initially computed optical flow inside the region. To achieve it, it is easier to follow a parametric approach. First, we estimate an affine motion model fitting the external optical flow, i.e., the flow outside R sal . We use again the robust estimation of eq.(5.5), but here, it is applied to Ω \ R sal where Ω is the whole image. Then, we leverage this estimated affine motion model to inpaint the flow within all the regions R j of R sal . The inpainted flow within each R j is given by: ∀p = (x, y) ∈ R j , (u imp p , v imp p ) = (â sal 1 + âsal 2 x + âsal 3 y, âsal 4 + âsal 5 x + âsal 6 y), where âsal i designates each of the parameters of the affine motion model estimated outside

R sal .
The parametric inpainted flow allows us to infer the motion saliency map ϕ inside R j from the flow gap w imp p -w p , where w p is the flow vector initially computed at p:

for each R j , ∀p ∈ R j , ϕ(p) = 1 -exp(-λ||w imp p -w p || 2 ), (5.6)
where ϕ(p) values lie within [0, 1] and λ > 0 modulates the visualization of the motion saliency map.ϕ(p) is set to 0 for all the pixels that do not belong to the set R sal = {R j }.

We use λ = 0.15 for all presented visualisations.

Experimental Results

Experimental Results

Implementation details

Optical flow fields are computed using the RAFT method [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF]. For LRP, we adopted a modified version of the implementation described in [START_REF] Gu | Understanding individual decisions of CNNs via contrastive backpropagation[END_REF]. We used Captum to test Integrated Gradients and Scikit Learn to implement the Huber robust regression. For the first case study, we transferred the model and weights from [START_REF] Maczyta | CNN-Based temporal detection of motion saliency in videos[END_REF] onto Pytorch framework, and fine-tuned the last fully connected layer using the original training dataset. We trained for 2 epochs, requiring approximately 20 min on a GPU GeForce MX150. For the second case study, we trained the network form scratch, which took 4.5 hours on a GPU GeForce RTX2080Ti. The average runtime per frame is 0.70s on a CPU 1.90GHz.

First case study: Independent scene motion

First, we want to objectively evaluate the accuracy of our method regarding the salient motion case of independent scene motion. Due to the lack of dedicated benchmarks, we make do with the VOS DAVIS2016 dataset1 as is also done by VS methods. The DAVIS2016 videos involve one single independently moving object in the foreground.

Let us stress that the DAVIS2016 training set has not been used for training our classification network. In this experiment, we use the more elaborated second step of the interpretation process described in subsection 5.2.2. The bitmap of the salient moving regions R j extracted by our method is compared to the ground truth. Results obtained with our method and other unsupervised methods are collected in Table 1. Our method outperforms comparable unsupervised methods on the VOS benchmark as BGM [START_REF] Wehrwein | Video segmentation with background motion models[END_REF], TIS 0 [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF], and FTS [START_REF] Papazoglou | Fast object segmentation in unconstrained video[END_REF]. On our side, we do not resort to any postprocessing step and do not rely on appearance contrary to the CIS and TIS s methods. Indeed, the CIS method involves an important post-processing stage and its performance without the postprocessing, evaluated with the publicly available code, drops to J Mean = 59.7 on the complete DAVIS2016 dataset and to 59.2 on the validation set, that is, both lower than our scores. Besides, our method is penalized by the ground truth of DAVIS2016 focused on the primary object. For instance, the ripples on the water in the flamingo sequence and the foam under the kytesurfer (Fig. 5.4) are recognized as salient motions by our method, which is correct.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

CIS [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] TIS s [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF] TIS 0 [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF] BGM [START_REF] Wehrwein | Video segmentation with background motion models[END_REF] Table 5.1 -Results on the complete DAVIS2016 dataset for several unsupervised methods (scores taken from [START_REF] Griffin | Tukey-inspired video object segmentation[END_REF] and [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF]). In brackets on the DAVIS2016 validation set only. J is the Jaccard index (region similarity) and F accounts for contour accuracy. For further explanation on the evaluation metrics, we refer the reader to the DAVIS2016 website. We want to further demonstrate the ability of our method to distinguish between independent motion and motion parallax. The latter being rarely present in DAVIS2016, we built a dataset of videos taken with a hand-held moving camera.Those videos are taken in an urban environment with complex backgrounds, where at the same time static objects in foreground induce motion parallax and other objects undergo independent motions. Examples of those videos are given in Figs.5.2-5.3-5.4 and in the supplementary material.

First, we assess the performance of the classification network in itself by performing an ablation test. We draw a mask around an area depicting distinguishable motion pattern and inpaint the flow within the mask as done in Section 5.2.3, hiding this prominent flow from the network. The procedure is illustrated in Fig. 5.2. We can observe that removing only moving objects lowers the saliency score, and makes the network predict the frame as "non salient". Conversely, when we remove the static object causing motion parallax, the prediction of the network remains "salient frame" due to the presence of moving objects. It shows that the frame-based classification network truly bases its prediction on the presence of independent motion. Secondly, we visually evaluate the localisation of the salient moving regions. Fig. 5.3 contains a comparison between the motion saliency Figure 5.3 -Extracted salient masks, superimposed on original images, on three examples from our dataset and one from DAVIS2016 (parkour video). Top row : results from [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] using the available code and model (without the postprocessing step). Bottom row : corresponding results using our method (NIMS). CIS [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] detects the foreground static object due to parallax motion, while our method succesfully discards it. segmentation performed by our method, NIMS, and by the CIS method presented in [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF]. We can note that our method is able to correctly extract the person moving as a salient moving region and not the pole, the tree trunk or the fence in the foreground. In contrast, the method [START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] also segments them as independently moving objects.

Finally, we display samples of motion saliency maps in Fig. 5.4. They were extracted from several datasets, respectively, our video dataset, the DAVIS2016 dataset and the Complex dataset [START_REF] Narayana | Coherent motion segmentation in moving camera videos using optical flow orientations[END_REF]. The closer to yellow, the higher the motion saliency degree given by ϕ(p) in eq. (5.6). Let us stress that the static foreground objects are not found salient, whereas their optical flow is prominent since it conveys parallax motion. The computed salient motion maps are representative of the underlying motions.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

Second case study: Distinctive motion in a crowd

In order to train our network and to evaluate the ability of our method to detect distinctive motion within a crowd, we built a dataset made of computer-generated sequences from 3D scenes depicting crowds in motion, using the public software ChAOS2 . The resulting dataset is not trivial, since the salient moving persons are small and not always easily discernible. It is composed of 100 videos of about 250 frames each, taken by a camera in motion. In each video, the crowd of people is walking in a common, randomly chosen direction. In order to simulate distinctive motion, one to three people walking in a different direction are randomly added to some videos. Frames including a distinctive motion are dynamically salient.

The first step is to train the frame-based saliency classification network on this dataset for the new task. This randomly initialized network follows the structure defined in subsection 5.2.1, except that we used InstanceNorm layers instead of BatchNorm to tackle changes in input flow magnitudes. After training, the network is able to recognize frames containing a distinctive motion with 90.6% of accuracy on the test set. We apply the LRP interpretation to the classification network and extract attribution maps. The latter exhibit a low level of noise and mostly highlight salient motions. Therefore, we use the light version of the second step of the network interpretation process described in subsection 5.2.2. Figure 5.5 displays samples of attribution and saliency maps computed with our method on videos of the crowd dataset. We can observe that these maps correctly delineate the salient motion in the crowd despite the small size of the salient area and the global camera motion. We have also processed several real videos of a similar kind. Visual results are again collected in Fig. 5.5. We used the classification network trained on the synthetic dataset without any additional fine-tuning. We have processed the Wrong Way video that depicts a man walking against a crowd. As shown in the video frames displayed in the supplementary material, the salient motion is correctly segmented in the vast majority of them. The method fails in case of poorly visible salient motion. On the other hand, when a woman moves aside to avoid the opposite individual, she produces the main salient motion. We also processed videos from the UCSD Anomaly Detection dataset 3 fitting this second task. Again, correct salient motion masks are recovered, whether it be the vehicle or the pedestrian walking in the opposite direction. Let us note that in the Figure 5.5 -Results obtained with our method (NIMS). Top to bottom row : one image of the video with the salient-motion ground-truth superimposed in yellow, optical flow displayed with the HSV color code, LRP attribution map, computed salient motion map. Left to right: four examples from the synthetic dataset (4th column: the outlier pedestrian is strongly occluded and only the foot is consequently recovered), two samples from the Wrong Way video at two distant time instants, two examples from the UCSD Anomaly Detection dataset, and an example from an outdoor scene acquired for the experiment. later video, the anomaly regarding our task is precisely this pedestrian and not the cyclist as defined in the UCSD dataset benchmark.

In addition, we compared our method with a segmentation U-net network trained in a supervised way using the same synthetic dataset and ground-truth masks corresponding to the salient moving objects. The objective is to build a gold standard, and to achieve a quantitative evaluation on the synthetic dataset. We use again as evaluation metric the Jaccard index J Mean. Let us note that high evaluation scores cannot be expected, because salient moving objects are frequently occluded. For a fair but strict evaluation, we compute it only on salient frames correctly classified as dynamically salient (since for non-salient frames the Jaccard index is 1, the ground-truth being empty). We get J Mean = 67.1 for the supervised network, and J Mean = 53.6 for our unsupervised method, which demonstrates that our method performs well, given the relatively small gap between the two scores.

Model randomization test

Interpretation methods can be sensitive to the structure of the network being analyzed and elements in the input images such as edges [START_REF] Adebayo | Sanity checks for saliency maps[END_REF]. We want to ensure that our attribution (or interpretation) maps depend on the learned parameters of the network and not of intrinsic characteristics of our input flow maps only. Thus, we followed the recommendation in [START_REF] Adebayo | Sanity checks for saliency maps[END_REF] and performed a model parameter randomization test. This test consists in comparing attribution maps obtained using our trained network and a random network. By visual inspection of Fig. 5.6, we can observe that the attribution maps exhibit important differences. While attribution maps of the trained network are focusing only on points that exhibit salient motion, attribution maps of the random network highlight all points with an apparent motion indistinctively of its real saliency. This test confirms the pivotal role of training the network on a classification task to obtain meaningful attribution maps.

Combining LRP Segment Selection with Deep

Learning Motion Segmentation

In this section, we combine our LRP-based method for motion saliency presented in this chapter and the temporally-consistent motion segmentation method described in Table 5.2 -Comparaison of quantitative results for all the sequences of the DAVIS2016 validation dataset, obtained with three different combinations of motion segmentation and segment selection. Each row represents the average Jaccard score for the sequence, and the bottom row is the overall average Jaccard score for all the sequences. The first column includes the original results reported in Section 5.3, where we used Huber segmentation along with the LRP-based salient segment selection. The second column collects the results obtained using the deep-learning motion-segmentation method described in Chapter 3 and called ST-MS, with a simple heuristic where we select for evaluation all segments except the largest one as foreground. The last column reports the results obtained by combining the ST-MS method of Chapter 3 with this time the LRP-based salient segment selection. Red rows indicate sequences with a strong parallax motion, grey rows sequences with background parasite motion (e.g., water ripples), and the green rows sequences with complex motions to segment (e.g., articulated motions). Chapter 3 and called ST-MS. As a matter of fact, we worked on the LRP-based method before studying motion segmentation based on deep learning. Therefore, in the initial design of our LRP-based motion saliency method, we used an iterative flow segmentation algorithm based on the estimation of parametric motion models through a robust regression. This motion segmentation method worked well on simple optical flow fields, but struggled to adapt to more complex scenarios. Here, we replace this motion segmentation algorithm with the ST-MS network presented in Chapter 3, while keeping the rest of the motion-saliency approach identical to measure the gain we can obtain with the ST-MS method.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

Quantitative results on the DAVIS2016 validation dataset reported in Fig. 5.2, show the clear gain (70.6 vs. 60.2) obtained by using our elaborated ST-MS method over the one previously used in section 5.3. Conversely, comparing the second and third columns of Fig. 5.2, we can also measure the performance improvement brought by applying the LRP-based network interpretation to select the right segments obtained with the ST-MS method. This is particularly important in scenarios where it is necessary to distinguish segments corresponding to motion parallax from those exhibiting independent motion (e.g., in parkour and libby sequences). In these cases, the "except biggest" segment selection algorithm indiscriminately selects all moving segments except the largest, which is considered background, whereas the LRP-based motion saliency method allows us to select only trully independent motion. We also provide a qualitative comparison of the results obtained by these three approaches in Fig. 5.7. We can observe that the Huber segmentation struggles especially on challenging optical flow fields representing articulated motion such as in the bear, breakdance, and camel sequences, while it performs well when there is an important difference between the object motion and the background motion (libby, parkour sequences). We can also see that the LRP-based segment selection process allows us to discard the motion parallax effect: in the parkour sequence, clear distinction between the apparent motion of the barrier from that of the running man, and in the libby sequence, between the apparent motion of the tree from that of the dog.

Partial Conclusion

We have defined an original, versatile and efficient method for the estimation of salient motion maps based on the interpretation of a simple frame-based classification network.

Part , Chapter 5 -Gradient-based Interpretation of a Frame Classification Network for Motion Saliency

It only requires the optical flow as input. The computation of the salient motion maps is unsupervised as it does not require any ground-truth on the segmentation maps. To tackle a given motion-saliency task related to a given application, it is sufficient to retrain the classification network, the overall framework remaining unchanged. We have demonstrated the performance of our method on two different case studies. Our method was thus able to disentangle parallax motion and independent scene motion without any 3D information. It can also handle distinctive motion in a crowd. Experimental results demonstrated that we are able to compute accurate and reliable salient motion maps which convey rich, readily available information on the nature of the salient motion, through the flow gap of the inpainting step, and not only the degree of saliency. Finally, combining the LRP-based motion saliency method and the ST-MS method of Chapter 3 provides the best results.

Nevertheless, this motion saliency approach requires an external motion segmentation stage to delimit regions exhibiting salient motion. In the next chapter, we will propose an integrated approach for salient motion segmentation based on adversarial learning.

Chapter 6

ADVERSARIAL LEARNING TO LOCATE SALIENT MOTIONS

Adversarial Approach for Saliency Segmentation

In this chapter, we aim to build a method for learning to segment regions of salient motion in optical flow fields, based on the predictions of a classification network. This work is an initial study to design an alternative to the method presented in Chapter 5, where we want to fuse the segmentation and interpretation steps to directly output the salient motion segmentation from the optic flow field. Instead of relying on gradient-based interpretation techniques such as LRP, we train a segmentation network to locate areas that, when modified (or perturbed), change the prediction of the classification network. This approach has several advantages. First, compared to the gradient-based interpretation approach, it is not sensitive to biases induced by the classification network or the input data. Secondly, after training, the segmentation network produces the motion saliency mask directly from the optic flow field, and is thus independent of the classification network at the inference step. Finally, unlike the gradient-based method presented in Chapter 5, it does not depend on post-processing techniques to extract the segments of salient motion.

In the first part of this chapter, we provide a brief overview of related work using perturbation-based approach for neural network interpretation. In a second part, we provide a description of our framework and our loss function based on adversarial learning. Finally, we present preliminary experimental results, mainly on the second motion saliency task presented in Chapter 5, i.e., discriminating singular motion within a coherently moving set.

Part , Chapter 6 -Adversarial Learning to Locate Salient Motions

Related work on perturbation-based network interpretation

In Chapter 5, we presented network interpretation methods based on gradients to determine the image areas significantly contributing to the prediction of the network classification. In this chapter, we focus on methods that perturbate a region of the input of a neural network to change the prediction of the classification network. In [START_REF] Dabkowski | Real time image saliency for black box classifiers[END_REF], the authors train a network to segment salient areas in an image based on a differentiable classifier. Their loss function compares the prediction of the classifier when only the salient area is perturbed (using either a blurred version of the image or a random colour image with gaussian noise) to the prediction when everything except the salient area is pertubed. They also use total variation (TV) regularization to produce smooth segmentation masks and introduce a loss term on the mask size. Using a similar approach, the authors in [START_REF] Zhmoginov | Information-bottleneck approach to salient region discovery[END_REF] propose a method for training a segmentation network that localizes salient regions. Instead of using a loss term on the size of the masked region, they design a loss term based on information bottleneck to control the information complexity of the masked region. In practice, they compute it from the reconstruction loss of a variational autoencoder [START_REF] Kingma | Auto-encoding variational bayes[END_REF], taking as input the binary mask and the masked image.

The problem with perturbing regions of the input image using classical techniques, such as blurring or adding Gaussian noise, is that it can produce distribution images that are poorly handled by the classification network. To perturb input images in a more realistic way, [START_REF] Yi | Information-Theoretic Visual Explanation for Black-Box Classifiers[END_REF] introduce a method where they train a generator network that can generate a realistic patch from its neighborhood. Using this method, they can compare the classifier output for an image after patch removal and obtain a feature importance map. Their method is highly dependent on the quality of the patch generator network, which can be difficult to train. In addition, they evaluate the importance of each patch sequentially, which may fail to detect interdependencies between patches. In some cases, removing a single patch may not affect the network prediction, but removing a set of patches may have a strong impact.

To our knowledge, there is no method that trains a motion saliency segmentation network using optical flow as input from a classification network. The advantage of our approach is that it is easier to obtain a realistic perturbation of the optical flow field, since it contains less details than RGB images. We mean that optical flow is globally smoother. In our method, we simply resort to parametric optical flow inpainting for this step.

Adversarial Network for Motion Saliency Segmentation 6.3.1 Overall framework

The core idea is to train a network to segment areas that are determinant for the prediction of a frame-based motion saliency classification network. The overall framework is illustrated in Fig. 6.1. In order to describe this framework in details, we first introduce its core components :

f ∈ R W ×H×2 is the input optical flow field. A salient flow field contains at least one salient motion and a non-salient flow field contains no salient motion. ϕ : R W ×H×2 → [0, 1] is a pretrained classification network, it takes as input an optical flow field and outputs its probability of being salient: p ϕ (salient|f ). g γ : R W ×H×2 → [0, 1] W ×H is a segmentation network parametrized by γ, it takes as input an optical flow field and returns a probabilistic mask. We denote the predicted mask by m ≜ g γ (f ). ω : R W ×H×2 × [0, 1] W ×H → R W ×H×2 is an inpainting operation. It takes a input a mask m and an optical flow field f and replaces the flow inside the mask using a parametric flow field computed on the flow field outside the mask. Its aims to produce a natural-looking flow while removing the masked area. We call this newly produced flow field ω(m, f ) the "perturbed flow".

Our goal is to train the weights γ of a segmentation network g to produce a mask g γ (f ) that after inpainting step results in a perturbed flow field ω(g γ (f ), f ) classified as non-salient by ϕ.

Loss function

The objective described above can be specified by the loss function L for an optical flow field f as:

L(f, γ) = p ϕ (salient|ω(g γ (f ), f )), (6.1)
with p ϕ (salient|ω(g γ (f ), f )) = ϕ(ω(g γ (f ), f )). Indeed, minimizing this loss means that the probability of the pertubed flow being salient tends to 0, that is, we have correctly removed the area with the salient motion.

Figure 6.1 -Overall framework of our adversarial saliency segmentation method. g γ represents the segmentation network that outputs the saliency mask, ω the inpainting operation that fills the flow inside the mask using flow information outside of it, and ϕ is the classification network that predicts if the inpainted flow is salient.

However, there are two important issues with the loss function (6.1). First, a trivial solution to minimize L is to predict g γ (f ) = 1 W ×H (which means that all the points are predicted as salient) for all input optical flow fields f , which results in perturbed flow fields exhibiting no motion and thus classified as non salient. In order to avoid that, we add a term S(f, γ) = ||g γ (f )|| 1 that penalizes masking large areas of the input. Secondly, the loss function (6.1) leads to directly minimize ϕ(ω(g γ (f ), f )), although we are only concerned with whether the perturbed flow is classified as non salient. Therefore, the constraint ϕ(ω(γ(f ), f )) < 0.5 is sufficient, as the classification network has been previously evaluated on its global accuracy and not its ability to predict the correct saliency probability values for each input.

Taking into account those issues, we can reformulate our objective as to find the smallest region (possibly consisting of disconnected components) in the input flow that, when perturbed, changes the prediction of the classifier such that the flow becomes classified as non-salient. We can formulate this objective as follows:

min γ S(f, γ) with ϕ(ω(g γ (f ), f )) ≤ 0.5. (6.2)
Using the method of Lagrange multipliers [START_REF] Platt | Constrained Differential Optimization[END_REF], we can reformulate this constrained optimisation problem into an unconstrained problem:

L(f, γ, λ) = S(f, γ) + λ σ(ϕ(ω(g γ (f ), f )), 0.5), (6.3)
where σ(x, κ) = x when x ≥ κ and σ(x, κ) = 0 when x ≤ κ. The parameter λ is the

Adversarial Network for Motion Saliency Segmentation

Lagrange multiplier enforcing the constraint.

As we want to train the segmentation network g γ , we need to find a gradient-based optimisation method to solve the above problem. Authors in [START_REF] Platt | Constrained Differential Optimization[END_REF] propose to solve this optimisation problem using the "Basic Differential Multiplier Method" (BDMM), where one performs a gradient descent on the parameters γ and a gradient ascent on λ, resulting in updates of the form:

γ t+1 = γ t -α∇ γ L(f, γ, λ t ), (6.4
)

λ t+1 = λ t + α∇ λ L(f, γ t , λ), ( 6.5)
where α is the learning rate. We train the network on a dataset D of optical flow fields. Thus, the overall loss function is:

L = f ∈D
L(f, γ, λ). (6.6)

Flow inpainting

The goal of the flow inpainting step ω(m, f ) is to fill the areas within the mask m = g γ (f ) using the information of the optical flow field f . We use parametric motion models as the ones described in Section 2.4.1. Let us assume at this point that we have only two types of motions at most in the flow field: background motion and possibly salient motion. Nevertheless, salient motion may be found in several areas of the frame, meaning that the mask corresponding to salient motion may comprise disconnected components. Let us add that these salient areas do not necessarily undergo the same motion, but their motions are all salient with respect to the surrounding (background) motion.

First, we compute the parameters θ of the parametric motion model from the area outside the mask according to:

θ inp = arg min θ p∈Ω (1 -m(p))||f (p) -fθ (p)|| 1 , (6.7)
where Ω is the image grid, and m(p), f (p) and fθ (p) respectively the saliency prediction, the optical flow vector, and the flow vector of the parametric motion model, at point p.

Then, we inpaint the flow within the masked area as :

ω(m, f ) = (1 -m) ⊙ f + m ⊙ fθ inp , ( 6.8)
where the operator ⊙ denotes the Hadamard product between the two matrices.

Multi-mask extension

The issue with the inpainting technique presented in Section 6.3.3 is that it assumes the presence of at most two types of motions: background motion and optional salient motion. In the case where there is a greater number of motion types in the optical flow field, it would be impossible to compute correct motion model parameters θ inp , since the background area would involve several motions. This situation occurs for instance in the Chaos dataset presented in Chapter 5. We have the background motion, which is null when the camera is static, the (coherent) crowd motion, and the salient motion of one or a couple of pedestrians walking differently than the crowd. This motivated the introduction of a multi-mask extension of our technique where g γ is not only predicting a salient mask, but also, a background mask and a set of subsidiary masks. In this case, the segmentation network no longer outputs one mask, but a set of

K probabilistic masks such that ∀ p ∈ Ω, K k=1 g k γ (f )(p) = 1.
The layers representing the background and the salient masks are hardcoded as being respectively the first and the second ones of the segmentation network output. In the rest of this chapter, we refer to the salient mask as m s ≜ g 0 γ (f ), the background mask as m b ≜ g 1 γ (f ). Let us stress that at this point we are talking about the functional background, not the physical background, that is, the area used to compute the inpainting parametric motion model. We can also refer to each mask using its index m k ≜ g k γ (f ).

Modified flow inpainting

In this setting, the inpainting step takes a different form:

θ inp = arg min θ p∈Ω m b (p)||f (p) -fθ (p)|| 1 , (6.9) ω(m, f ) = (1 -m s ) ⊙ f + m s ⊙ fθ inp . (6.10)

Coherence loss term

In this multi-mask extension, we can introduce the segment-wise flow reconstruction loss term presented in Section 2.12. This has the advantage of training g γ to output motion coherent masks that we can use downstream for other applications. It is also a pretext task we can leverage to improve our network training. Let us recall its expression: For the choice of K, we can take any number greater than the maximum number of motions we expect to observe in the video sequence.

L r (m, f ) = p∈Ω K k=1 m k (p)∥f (p) -fθ k (p)∥ 1 p∈Ω ∥f (p)∥ 1 , ( 6.11

Inference step

In the inference step, we provide an optical flow field to the network, and it outputs the segmentation with the four masks (background, salient objects, other motions). In the next Section on experimental results, we will display only the salient mask, since it is the one of interest for our task. Contrary to the training time, it is not necessary to use the classification network to compute parametric motion models at test time, which makes the network efficient.

Part , Chapter 6 -Adversarial Learning to Locate Salient Motions

Experimental Results

Implementation details

For the segmentation network, we use the same U-net as described in Chapter 2. We output four motion segments, the first one is by default dedicated to the background mask (support used to compute the parameters of the background motion model) and the second one is devoted to the salient motion. The remaining two masks are dedicated to representing non-salient motion segments that are different from the background. To implement the gradient ascent in BDMM, we use the Gradient Reversal Layer introduced in [START_REF] Ganin | Unsupervised domain adaptation by backpropagation[END_REF], which allows us to perform both optimization steps described in 6.4, référence à inclure at the same time. The computation of parametric motion models during training is also the same as presented in Chapter 2.

We use the Adam optimizer with a learning rate of 10 -6 and a linear decay schedule starting at epoch 100. Since the loss depends on λ, which is increasing, it is not necessarily decreasing. For the network model selection, for now, we just choose the model of the last training epoch and use it for the evaluation. We keep the choice of the criterion for a stopping criterion on the validation set for future work.

Regarding the compute time at inference time, we can expect the same results as in Chapter 2, since we use the same network. At training time, the classification network introduces a small overhead.

Detecting anomalous motion

At this stage, we report preliminary results.

Regarding the first motion saliency task, i.e., detecting independent motions in a scene observed by a mobile camera, results are not that convincing so far. Indeed, as illustrated in Fig. 6.3, if the adverse network is able to detect the salient moving objects, it tends to extract only the boundaries of the independent moving objects. Further investigation will be needed.

Therefore, we will focus in this section on the second motion saliency task concerned with discriminating singular motion within a coherently moving set.

Results on Chaos dataset

As we can see, the results on the Chaos sequences are quite good. The network detects most of the abnormal motions, and delivers a blank mask when there is no salient motion. Since the segmentation used for this task in Chapter 5 worked well, the results are on par with the previous ones, although in this case the network does both motion segmentation and salient moving object localization at the same time. We show examples of results in Fig. 6.4.

Results on real sequences

To process real sequences, we use our network trained on Chaos dataset without any finetuning. Results on the Wrong way sequence are less convincing. It seems that the network is not able to segment objects well in these sequences, which is understandable since it was only trained on simulated optical flows. The results on videos from the UCSD datset, are better. In this case, the network manages to segment the salient motions well. To further improve these results, it would be interesting to train the network, or at least the U-net segmentation backbone, on real data. We show examples of results in Fig. 6.5.

Partial Conclusion

In this chapter, we have presented an adversarial method for training a salient motion segmentation network from a pre-trained classification network. Our loss function is based on treating the problem as a constrained optimization and the Lagrange multipliers method to formulate the loss function. We have reported encouraging results on simulated data, validating the intuition behind our paradigm. Yet, the network does not perform well enough on real data.

Several modifications could help improve the results on real-world data. First, it would be interesting to have a variable number of masks to represent salient motion. In this first implementation, we have only one mask representing salient motion. Since this mask is affected by the coherence motion loss term, it restricts the prediction of salient motion to a single motion model. Experiments in Chapters 2, 3, and 4 demonstrated that the aggregation of multiple motion masks is key to represent salient motions on the VOS dataset. An interesting first perspective would be to allow multiple salient motion masks and aggregate them to extract the whole salient motion.

Second, it might be interesting to decouple the segmentation and the motion saliency classification network. Indeed, the motion segmentation network takes much longer to train and can be trained on a wider variety of optical flows, since it does not require a frame classification network. This can affect the results on hard to segment sequences such as Wrong way, where the flows to segment at test time are very different from the flows encountered in the training set. This improvement is hard to implement with convnets, but much easier to implement with the Maskformer architecture.

Third, we could introduce the temporal dimension in our segmentation, as it could improve the distinction between salient and non-salient motions. The optimal setting would be to introduce the temporal dimension in both the frame-based classifier and the segmentation network. Segmenting a volume of frames enforcing temporal consistency and then doing the classification for each frame independently could be a relevant first improvement. motion model that represents the optical flow within each segment. Second, we moved from 3D convolutional networks to an architecture based on transformers to account for longer temporal context and introduce feature interactions across the sequence. This allowed us to have a network capable of segmenting an entire sequence of optical flow fields in a single pass, providing coherent labels across time. It achieves competitive results on video object segmentation benchmarks and is fast at test time.

Localization of salient motions

We have designed methods for localizing salient areas in optical flow fields. We define salient areas as those that affect the prediction of a saliency classification network, which identifies whether an input optical flow field has salient motion.

In a first work, we developed a fast and versatile technique based on the interpretation of a simple frame-based classification network. We use the LRP method to compute saliency, which is then combined with external motion segmentation to obtain saliency maps. Our results demonstrate the adaptability of this method to various saliency scenarios, only requiring a retraining of the classification module to adapt to a given task. We evaluated this method on two different motion saliency scenarios. The first scenario involved disentangling motion parallax from independent scene motions in videos recorded with a moving camera, while the second scenario focused on the localization of distinctive motions in a crowd.

In a subsequent study, we maintained the goal of predicting important areas for the prediction of a frame-based saliency detection network. However, we adopt a perturbationbased approach instead of the gradient-based method used in our prior work. This approach involved training an adversarial convolutional network to segment salient areas in the input optical flow field. A loss function was used to minimize the output of the classification network. This alternate method is appealing since it merges saliency detection and motion segmentation. However, further development is needed to improve results on real video sequences.

Possible improvements on unsupervised motion segmentation Train on longer sequences

One possible improvement would be to enhance the training procedure for long-term motion segmentation by including longer sequences at training time. This could improve the ability of the network to produce temporally consistent segments and help develop features that represent long-term motion. Although this is challenging as it would require modifying the training procedure to deal with the memory overload induced by training on long sequences, we could also modify our spatio-temporal attention operations to be less computationally expensive.

Variable number of segments

In the approaches we have designed so far, the number of segments produced by the network must be fixed at training time and cannot change during inference. An improvement of our method would be to allow the user to choose the number of segments to use at test time. We have done some experiments replacing our transformer attention with a slot attention [START_REF] Locatello | Object-centric learning with slot attention[END_REF] mechanism, although we encountered difficulties to get reproducible results and to ensure the convergence of the network. A somewhat related idea would be to train the network to do hierarchical segmentation, as done in [START_REF] Perez-Rua | Hierarchical motion decomposition for dynamic scene parsing[END_REF], where related segments are grouped together. This would allow the users to choose which level of precision to use for their task.

Include appearance

Including appearance could allow to improve segmentation on parts of an object that are currently not moving (e.g. the moving objects stops at some point). However, as mentioned in the introduction, using appearance may also bias the segmentation towards visually salient objects if they predominate in the dataset. One option would be to simultaneously train a branch that takes object appearance as input and one that takes the optical flow field as input using our motion coherence criteria, and add a constraint that makes the two predictions match. At inference time, we could combine the predictions from the two modalities to ensure that salient static objects are not segmented. Another option would be to learn the appearance of the moving object in a video and use it to refine our motion segmentation.

Remove dependency on optical flow computation

So far, we provide the optical flow field as input to our segmentation network. This makes our network dependent on the quality of the flow field. We could improve our method to make it independent of the optical flow field method. A first improvement would be to replace the optical flow field with a sequence of RGB frames as input to the network. This would require changing the structure of the network to implicitly compute correlation maps, as optical flow methods do, which would increase the computational load of our network. This would remove the dependence on the optical flow field at test time. However, since we need optical flow fields to compute the loss function, it would be dependent on the optical flow computation method at train time. A second improvement would be to modify our loss function with a reconstruction term that measures the warping error induced by the parametric flow field. Together with the input change mentioned above, this would make our segmentation method independent of optical flow field computation at both training and test time allowing us to learn motion segmentation without ever having to compute explicitly an optical flow field.

Possible Improvements on the localization of salient motions Temporal Consistency

A salient object can be expected to remain salient for a few frames. To incorporate temporal consistency into salient object detection and segment a salient object over multiple frames, our method can be expanded to include video sequences. Extending our method to video sequences could also enable us to detect more complex forms of motion saliency that are not visible with a single instantaneous motion but are observable in an abnormal sequence of events.

Anomaly detection

In our previous work, we applied our method to obtain the interpretation of a saliency classification network that was trained in a supervised manner on a database of salient and non-salient optical flows. An interesting extension would be to apply our technique to an anomaly detection network trained in an unsupervised manner (e.g. one class SVM, teacher student approach as in [START_REF] Wang | Self-Trained Video Anomaly Detection Based on Teacher-Student Model[END_REF]). This would eliminate the need for explicit supervision by presenting only non-salient images to the network.

Larger Research Perspectives

Self-supervised learning of appearance features from motion Learning appearance representation from videos is an idea that has been around for a long time [3], [4], [START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF]- [START_REF] Tangemann | Unsupervised Object Learning via Common Fate[END_REF], and has shown interesting results as a pre-training task for supervised learning. This is an attractive alternative to other self-supervised appearance representation methods, such as [6], as it could learn the representation directly from videos without requiring an object-centric dataset. It would thus allow to deal with a greater variety of objects or image modalities. With the recent improvement of optical flow methods [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF], [START_REF] Huang | FlowFormer: A transformer architecture for optical flow[END_REF] and the development of novel network architectures [START_REF] Locatello | Object-centric learning with slot attention[END_REF], [START_REF] Kipf | Conditional object-centric learning from video[END_REF], it seems that motion-based self-supervised representation learning could be a promising research direction in the future. An interesting perspective related to our work would be to learn appearance features at the sequence level, exploiting the temporal consistency of motion segmentation, as we did in Chapter 4. This could lead to the development of appearance features that are robust to temporal evolution (exposure changes, deformations, different camera angles). For example, a motivating idea would be to "capture" the object appearance during the first frames of the video and try to retrieve the object mask later in the video, verifying the quality of the prediction using a motion consistency metric.

Learning representations for complex motion

Action recognition is central to a number of tasks in computer vision and has an important impact because it is used in a wide range of applications. As mentioned in the introduction, existing action recognition methods exploit both motion and appearance cues. However, using images as input to action recognition models can raise a number of issues. First, technical issues, such as the difficulty of performing domain adaptation from simulated data in the image modality compared to performing it in the motion modality, the greater variability depending on context in images compared to motion representations. Second, social issues such as the occurrence of appearance-based bias in deep learning models based on images [START_REF] Steed | Image representations learned with unsupervised pretraining contain human-like biases[END_REF], [START_REF] Pastaltzidis | Data augmentation for fairness-aware machine learning: Preventing algorithmic bias in law enforcement systems[END_REF]. Furthermore, recent camera developments such as event-based cameras [START_REF] Zhu | EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras[END_REF] or on-sensor acceleration [START_REF] Kühne | A fast and accurate optical flow camera for resource-constrained edge applications[END_REF] make optical flow computation computationally cheaper. This context motivates the development of action recognition methods based solely on motion, and especially unsupervised methods that could learn to handle a large range of complex motion without expensive human annotation. However, the whole difficulty of unsupervised action recognition would be to learn motion features that are complex enough to describe high-level motions and generalizable enough to adapt to natural variations in motions (i.e., not everyone claps their hands in exactly the same way, but our brain is able to recognize this action in a wide range of contexts). Another challenge is to group hierarchical motions together to associate them with the same cause (e.g., articulated motion).

In our work on long-term motion segmentation, we learn the representation of motion groups over a sequence as "refined queries" in the maskformer architecture. These queries allow us to build motion segmentation over a sequence using a cross product with the feature map. Thus, it seems that they contain global information about the motion of each group along the sequence. An interesting investigation would be to measure if these queries are informative about the high-level representation of motion, in which case it would make unsupervised long-term motion segmentation a great pre-training task for action recognition or clustering. Future work could focus on defining novel tasks or training procedures to improve the expressiveness of our motion representations, and finding ways to measure this expressiveness.

Dance movements and notations like the one we introduced in Fig. 3 seem to be a fascinating benchmark for these tasks, as they describe a precise and complex movement that will be reproduced by a wide range of interpreters. The Laban notation is quite restrictive, but other notations [START_REF] Kleinman | Movement notation systems: An introduction[END_REF], [START_REF] Galili | Gaga: moving beyond technique with ohad naharin in the twentyfirst century[END_REF] offer more freedom to the dancers. A motion representation of videos that naturally clusters a set of dance moves would be, in my opinion, an impressive first step towards unsupervised action recognition.

Lower Bound (ELBO):

log(D) = f ∈D log p(f ) ≥ f ∈D (E q(z,ϑ|f ) [log p(f |z, ϑ)] -KL(q(z, ϑ|f )||p(z, ϑ))).
(7.3)

Likelihood

Following the assumption stated above, we can write our likelihood as:

p(f |z, ϑ) = i∈Ω K k=1 p(f (i)|ϑ k , z) ∝ i∈Ω K k=1 exp(- 1 α δ(f (i), fϑ k (i))) z k i , ( 7.4)
where fϑ k (i) is the flow vector given by the parametric motion model of parameters ϑ k at point i. Here δ(., .) denotes a distance between the two terms. From this likelihood, we can infer the reconstruction loss :

L r = -E q(z,ϑ|f ) [log p(f |z, ϑ)] = - i∈Ω k E q [log p(f (i)|ϑ k , z)] = - i∈Ω k E q [z k i ] E q [- 1 α (δ(f (i), fϑ k (i))] = i∈Ω k 1 α g ϕ (f ) k i δ(f (i), fθ k ) (7.5)

Prior

The term KL(q(z, ϑ|f )||p(z, ϑ))) allows us to define a prior over the segmentation. In our case, we enforce the prior that every mask have an equal probability, thus p(z k i ) = 1 K . We do not employ a prior on ϑ.

KL(q(z)||p(z)) = i∈Ω KL(q(z i )||p(z i )) (7.6) = i∈Ω K k=1 [q(z i = k) log(q(z i = k)) -q(z i = k) log(K)] (7.7) = i∈Ω K k=1 [q(z i = k) log(q(z i = k))] -|Ω| log(K) (7.8) = i∈Ω K k=1 g ϕ (f ) k i log g ϕ (f ) k i + cst. (7.9)

Training

The loss function is thus defined by:

L = f ∈D -E q(z,ϑ|f ) [log p(f |z, ϑ)] + KL(q(z, ϑ|f )||p(z, ϑ)) (7.10) = f ∈D 1 α i∈Ω k g ϕ (f ) k i δ(f (i), fθ k (i)) + g ϕ (f ) k i log g ϕ (f ) k i (7.11)
We get the same loss as in eq.2.15.

Data augmentation

In this section, we formally analyze the impact of the data augmentation on the motion segmentation.

Theorem 1. Considering a function ll such as: Remark. In this theorem, ll and c(i) are the lower bound and the polynomial terms, respectively, both of which are described in Chapter 2. For the lower bound, we take m instead of g ϕ (f ), since we are not specifically considering the network here. The goal here is to show that the lower bound we optimize in Chapter 2 is invariant to the added parametric global flow with respect to the segmentation. Thus, a segmentation will correspond to the same loss value, regardless of the perturbation of the input flow through the data augmentation, which encourages the network to produce global-motion-invariant segmentations. We can see an illustration of this property in Fig. 7.1. Furthermore, a direct consequence of this property is that the optimal m * segmentation is the same for Predicted mask with our network trained using data augmentation; Predicted mask with our network trained without data augmentation; Ground-truth segmentation mask.

ll(θ, m, f ) = i k m k i (log p(f i , z k i |θ k ) -log m k i ), (7.12
both the original and augmented flows:

m * = arg max m [max θ ll(θ, m, f )] = arg max m [max θ ll(θ, m, f + f ζ )]. (7.14)
Proof. Starting from eq.4 and using eq.10 for the likelihood definition in Chapter 2, we can write:

ll(θ, m, f ) = i k m k i (log p(f i , z k i |θ k ) -log m k i ) (7.15) = i k m k i log p(f i , z k i |θ k ) -m k i log m k i (7.16) = i k m k i log p(f i |z k i , θ k ) + m k i log p(z k i ) p(m k i ) (7.17) = - i k m k i δ(f i , θ T k • c(i)) + κ, (7.18) where κ ≜ m k i log p(z k i ) p(m k i )Z
is independent of f and θ. See Theorem 2 justifying this statement for the normalisation factor Z. Consequently, we have:

ll(θ, m, f + f ζ ) = - i k m k i δ(f i + f ζ,i , θ T k • c(i)) + κ. (7.19)
Using translation invariance of our distance function δ and the additivity property of our parametric motion models (that are linear with respect to the parameters), we can write:

δ(f i + f ζ,i , θ T k • c(i)) = δ(f i , θ T k • c(i) -f ζ,i ) = δ(f i , θ T k • c(i) -θ T ζ • c(i)) (7.20) = δ(f i , (θ k -θ ζ ) T • c(i)) = δ(f i , θT k • c(i)). (7.21)
Thus, we have:

max θ ll( θ, m, f + f ζ ) = max θ - i k m k i δ(f i , θT k • c(i)) + κ. (7.22)
With a change of variable, we get: Then, the normalisation factor Z is independent of θ.

max θ ll(θ, m, f ) = max θ ll(θ, m, f + f ζ ). ( 7
In particular, this result is true for the p-norm defined by:

δ(a, b) = ||a -b|| p = i |a i -b i | p 1/p . ( 7

.25)

Proof. The normalisation factor is defined by:

Z ≜ R D exp(-δ(y, θ T x))dy. (7.26)
Taking b ≜ y -θ T x, we get: We can observe that the results collected in Table 7.5 are stable, whereas the process described above (segment linkage and segment selection) could generate variability.

Z = R D exp(-δ(b + θ T x, θ T x))db = R D exp(-δ(b, 0))db. ( 7

Additional experiments Training without data augmentation

To extend our ablation study, we have also carried out the evaluation of the case when our network is trained without any data augmentation. Results are collected in Table 7

Impact of the temporal interval τ

Regarding the τ parameter (i.e., the temporal interval between the flows of the input triplet), we trained our network with the full configuration and randomly sampled τ values at training time. Let us remind that the flow f t-τ (respectively, f t+τ ) is computed between image frames at time instants t -τ and t -τ + 1 (respectively, t + τ and t + τ + 1). We uniformly sample τ among a set of values during training. At inference, we still use only τ = 1 for the sake of efficiency. Thus, this experiment could also be perceived as a type of data augmentation. As we can observe in Table 7.7, it has no sensible impact on results (even a slight performance decrease). In future work, we plan to investigate the combination of different τ values, including negative values, at test time.

Latent motion representation

We give a few highlights on the latent motion representation issued from the trained network as mentioned in Chapter 3. We carried out a preliminary experiment directly based on the normalized latent vectors of all the sites of a subsequence. The latent vectors are of dimension 32. We applied a PCA procedure to all the latent vectors over the whole subsequence of length T and taking into account the triplets at each time instant. These latent vectors are stacked as an array of dimensions (3 × H × W × T, [START_REF] Weiss | A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models[END_REF], where H and W are respectively the height and the width of each frame of the subsequence.

Then, we compute the softmax of the projections onto the three first components of the PCA output. Interestingly, after thresholding the softmax values (threshold value of 0.7), we observe that the resulting map is likely to provide a binary segment close to the ground truth of the primary moving object, as illustrated in Fig. 7.4. It shows that our latent motion representation is not only informative in its own, but more importantly, is coherent over the subsequence since the PCA is computed once over the subsequence. In future work, we will investigate further this possibility to provide a binary segmentation The bottom row displays the linear relationship between validation loss and evaluation score. The model whose results are reported in the main text is represented as an orange dot.

Addition of Slot Attention

Several works on video object segmentation [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF] include the slot attention mechanism [START_REF] Locatello | Object-centric learning with slot attention[END_REF]. This attention mechanism takes place in the transformer decoder and starts with a random set of queries that are then iteratively refined depending on the input data, similar to a classical soft k-means algorithm, we illustrate this process with Figure 7.7. This is an interesting addition, as it theoretically allows you to vary the number of output masks without having to retrain the network, as highlighted in [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF]. In practice, this is only true if the initial query vector is sampled randomly, as in the original paper, which is done in [START_REF] Lao | Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots[END_REF]. However, in [START_REF] Yang | Self-supervised video object segmentation by motion grouping[END_REF], the authors mention that they were unable to train a model with Gaussian initial query vectors, and so they used learnable queries, which makes their method not adaptable to a variable number of outputs, despite the fact that they use slot attention.

To compare with the classical transformer decoder and to develop a method with a variable number of masks at inference time, we tried to train our model after replacing the transformer decoder with a slot attention decoder. We call this new model LT-MS-SA. We were not able to train the network from scratch because the training diverged, we also

DAVIS2016

Video J (M) J (O) J (D) F (M) F (O) F (

Related work

General framework

The first end-to-end CNN-based flow estimation method dates back to FlowNet [START_REF] Dosovitskiy | FlowNet: Learning optical flow with convolutional networks[END_REF], a supervised neural network that either computes convolutions directly on the stacked images (FlowNetSimple) or produces meaningful representations of both images separately before comparing them at a higher level (FlowNetCorr). Once the feature extraction is done, the resulting coarse feature maps need to be upsampled back to the original resolution using strided convolutions (sometimes called deconvolution). This refinement then yields the final dense flow estimation. This CNN-based optical flow algorithm was trained in a supervised way on the synthetic dataset Flying Chairs that was created by the same authors.

The general framework of most existing deep flow methods can be summarized in these four fundamental steps:

1. Feature maps of both images at different resolutions are extracted; 2. Correlation is then computed in order to find correspondences between the two images;

3. Intermediate optical flow is predicted based on the correlation volume and the previous optical flow estimation;

4. Finally, the previous step (or two previous steps depending on the method) is repeated in an iterative loop.

The iterative refinement is either achieved according to a coarse-to-fine warping-based approach [START_REF] Sun | PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF] or to a single high-resolution flow update approach [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF], [START_REF] Ranjan | Optical flow estimation using a spatial pyramid network[END_REF], [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF].

Stacking multiple networks

FlowNet2.0 [START_REF] Ilg | FlowNet 2.0: Evolution of optical flow estimation with deep networks[END_REF] proposed to improve the accuracy of optical flow estimates by stacking multiple encoder-decoder networks (FlowNetS networks) into a large model, each taking the source image and the target image warped with the previously estimated flow. Compared to the original FlowNet, FlowNet2.0 decreases the estimation error by more than 50%. However, the model is very large (160M parameters) and therefore more prone to overfitting.

Multi-resolution approach

SpyNet [START_REF] Ranjan | Optical flow estimation using a spatial pyramid network[END_REF] and PWC-Net [START_REF] Sun | PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF] embedded the traditional coarse-to-fine approach into the learning framework. This classical concept consists in estimating optical flow using a multi-scale image pyramid, from the coarsest to the finest level. SpyNet proposed a lightweight architecture with only 1.2M parameters. It showed the potential of combining CNNs with classical principles, namely pyramidal processing and warping. However, it failed to fully exploit these principles and underperformed FlowNet2.0 as well as classical energy-based methods. Another compact CNN model called PWCNet tried to solve the trade-off between model size and accuracy by using classical concepts more effectively. Instead of constructing a fixed image pyramid like SpyNet, PWC-Net designed a learnable feature pyramid by using CNNs. Besides the two classical principles listed above, PWCNet also incorporated another one: the use of cost volumes. A cost volume stores the data matching costs for the potential correspondences of each pixel. Feature pyramids from both images are built prior to the partial cost volume computation. To generate features F l 0 and F l 1 at the l-th level, features from the lower level l -1 are downsampled by a factor of two using convolutional networks. It is illustrated in Fig. 7.10.

Single-resolution approach

The coarse-to-fine approach fails to address the well-known challenge of small, fastmoving objects. Indeed, when building the feature pyramid using successive downsampling, objects whose size is smaller than their displacement tend to disappear at the level where their flow is small enough to be estimated [START_REF] Brox | Large displacement optical flow[END_REF]. Furthermore, the warping process used by both PWC-Net and SpyNet can propagate early errors from higher levels, making the final optical flow estimate unreliable. RAFT overcomes these limitations by operating at a single resolution flow field. Figure 7.11 shows that features are extracted at a single resolution. At each iteration k, the flow estimate is updated by adding the generated residual flow ∆w k to the current flow estimate w k : w k+1 = w k + ∆w k . RAFT and other related methods extract feature representations of input images using a CNN-based feature encoder. The feature encoder f θ maps I 0 and I 1 to their abstract feature representations F 0 and F 1 at a lower 1/2 s resolution:

f θ : R H×W ×3 -→ R H/2 s ×W/2 s ×C I -→ F,
where θ represents the parameters of the network, C is the number of feature channels, and H and W are respectively the height and width of the images. A context encoder is generally added to extract features from the source image. The authors of RAFT showed that injecting the context into the update block improves optical flow results. The intuition behind this is that it aggregates spatial information with motion boundaries.

Computing visual similarity

Once the features are extracted, the next task is to build the correlation volume that stores the visual similarities between pixels in the source and target frames. There are various ways to formulate the computation of the cost volume.

The first one is to adopt a local correlation volume. To perform the matching process, FlowNetCorr introduced a local correlation volume that can be expressed as follows:

CV = F 0 (p) • F 1 (p + d) | (p, d) ∈ X × D (7.29) = 1 √ C C-1 c=0 F 0,c (p)F 1,c (p + d) | (p, d) ∈ X × D , ( 7.30)
where F 0 and F 1 ∈ R H×W ×C are the source and target feature maps, X = [0,H( × [0,W( ∩ N 2 refers to the location of a pixel in the source image and D = [-R max , R max ] 2 ∩ Z 2 corresponds to the displacement range. R max represents the search radius of the square neighborhood and consequently the maximum displacement along x and y directions. This local correlation volume contains N (2R + 1) 2 costs, where N = HW is the number of pixels in the images. It may result in an inaccurate optical flow due to the limited search space. Throughout this report, D = 2R max + 1 will refer to the side length of the square search region.

Global correlation volume

RAFT proposed an all-pairs correlation volume that achieved state-of-the-art performance. This full correlation volume is formed by computing the normalized inner product between pairwise feature vectors. Mathematically, it can be formulated as follows:

CV = F 0 (p 0 ) • F 1 (p 1 ) | (p 0 , p 1 ) ∈ X 2 , ( 7.31)
where F 0 and F 1 ∈ R H×W ×C are the source and target feature maps and X = [0,H( × [0,W( ∩ N 2 refers to the source and target image domain. In order to combine information about both large and small displacements, RAFT constructed a 4-level pyramid by pooling the last two dimensions of the correlation volume. Therefore, for each pyramid level l, the correlation volume CV l has dimensions H × W × H/2 l × W/2 l where 2 l is the pooled kernel size (see Fig. 7.12). The global correlation volume contains N 2 costs, where N = HW is the number of pixels in the images. For high-resolution images where the height and the width are large, the global approach leads to large memory consumption and high computational cost.

Iterative updates

The recurrent update operator introduced in Fig. 7.11 iteratively regresses a residual flow ∆w to refine the flow estimate: w k+1 = w k + ∆w k . It is usually achieved using ConvGRU blocks (Convolutional Gated Recurrent Unit). Its principle is illustrated in Figure 7.13. The flow is generally initialized at w 0 = 0 (except for the DIP method [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF] described below, where the flow is randomly initialized). At each iteration k, the current hidden state h k , the current flow w k , and the context are injected into the update operator, which further outputs a residual flow ∆w k . In RAFT, the correlation feature map CF k is also fed into the update operator. This map is constructed after every correlation lookup. This operator, which precedes the update operator, is illustrated in Figure 7.14.

Extensions on the cost volume.

In order to overcome the drawbacks of the global correlation volume involved in RAFT, other methods proposed to improve the processing of cost information. Instead of directly operating on the 4D correlation volume, FlowFormer [START_REF] Huang | FlowFormer: A transformer architecture for optical flow[END_REF] uses the attention mechanism to encode the entire H × W × H × W correlation volume into a compact H × W × K cost memory (K ≪ N ), which better captures information across pixels. Although this method effectively addresses cost information, it still requires to build the entire correlation volume and is therefore heavily memory consuming.

Sparse global correlation volume

The method described in [START_REF] Jiang | Learning optical flow from a few matches[END_REF] introduces a sparse global correlation strategy to reduce the memory consumption of dense correlation computation, while maintaining a global search for correspondences. The Sparse Correlation Volume (SCV) is constructed from k nearest matches in the target feature map for each feature vector in the source feature map. These scores are then stored in a sparse data structure that is further converted into a 2D dense motion tensor via a multi-scale displacement encoder. When building correlation volumes with high-resolution feature maps, this strategy results in significant memory savings in comparison to RAFT (the complexity of correlation computation is reduced from O(N 2 ) to O(N k)). However, it fails in blurry and featureless regions where top-k correlations might no be sufficient to include the correct match.

PatchMatch-based correlation volume Instead of using the global sparse strategy that suffers from a loss of accuracy, the DIP method [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF] adopts a PatchMatch-based correlation volume. The PatchMatch method was proposed by Barnes et al. [START_REF] Barnes | PatchMatch: A randomized correspondence algorithm for structural image editing[END_REF] to compute patch correspondences in a pair of images. It leverages that neighboring pixels usually have coherent matches. DIP consists of two modules: an inverse propagation module and a local search module. The first module propagates information from the neighbors once a good match is found. The second module prevents local minima by performing a local search after each propagation. The correlation volume is then defined by: CV = F 0 (p) • W(F 2 , S(w, ∆s))(p) | (p, ∆s) ∈ X × S , (7.32) where S = (1,1), (-1,-1), (0,0), (+1,-1), (-1,+1) refers to the shift direction, S(w, ∆s) is the flow shifted to ∆s, and W warps the target feature map with the shifted flow. The correlation volume has dimensions H ×W ×|S|, where |S| is the number of flow candidates (5 in this specific case). DIP accomplished a good trade-off between performance and cost.

It also achieved SOTA performances on KITTI-15 dataset and Sintel Clean dataset

The classical propagation stage requires the shift to be run each time the flow is updated. This in turn increases memory operations, especially when working with highresolution images. That is why DIP proposes an inverse propagation module that shifts the target feature maps in advance instead of shifting the flow after every iteration.

Supervised learning

Let θ be the set of learnable parameters in a given network. These parameters typically include feature extractor and update operator parameters. Let D be the set of image sequences {(I i ) T i=1 }, where T is the number of images in a sequence. We learn the parameters θ by minimizing a loss function L: θ * = arg min θ L(D, θ).

When ground truth is available, the commonly used error measure is the Average End-Point Error (AEPE) between the predicted flow vector w and the ground-truth vector w GT , averaged over all pixels (N pixels): In multi-scale approaches as in [START_REF] Sun | PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume[END_REF], the training loss can be written as follows:

L(θ) = 1 N L-1 l=l 0 α l p ∥w l GT (p) -w l (p)∥ 1 , (7.34)
where w l is the flow at the l-th pyramid level, α l are increasing weights since more importance is placed on the finest resolution l = L -1. The model starts to predict the flow at the coarsest-resolution l 0 and uses bilinear interpolation to obtain the fullresolution optical flow at level L -1.

The previously described single-resolution approaches [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF], [START_REF] Huang | FlowFormer: A transformer architecture for optical flow[END_REF], [START_REF] Ranjan | Optical flow estimation using a spatial pyramid network[END_REF], [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF] output a sequence of flow estimates {w 1 , . . ., w M }, where M is the number of iterations in the flow refinement. Their training loss can be expressed as:

L(θ) = 1 N M k=1 α k p ∥w GT,k (p) -w k (p)∥ 1 , (7.35)
where α k is computed from Eq.(7.36) with γ = 0.8 (exponentially increasing weights since more importance is placed on the final optical flow estimate).

α k = γ N -k (7.36)

Unsupervised learning

When working with real-world sequences, it is difficult to obtain ground-truth labels for dense optical flow. That is why supervised techniques usually rely on synthetic data for training. Therefore, generalization remains challenging due to the potential large gap between the synthetic data they were trained on and the real-world images of a given application. In order to overcome this issue, other works proposed to learn optical flow in an unsupervised way.

Unsupervised methods often combine components from supervised learning regarding the network architecture and classical (variational) optical flow methods regarding the loss function. UnFlow [START_REF] Meister | UnFlow: Unsupervised learning of optical flow with a bidirectional census loss[END_REF] proposes an unsupervised version of FlowNetCorr. ARFlow [START_REF] Liu | Learning by analogy: Reliable supervision from transformations for unsupervised optical flow estimation[END_REF] follows the pipeline of PWC-Net while reducing its number of parameters. This lightweight extension of PWC-Net integrates self-supervision from augmentations into the learning framework in order to generate challenging scenes. These augmentations include spatial, occlusion and appearance transformations. SMURF [START_REF] Stone | SMURF: Self-teaching multi-frame unsupervised RAFT with full-image warping[END_REF] extends the RAFT architecture to an unsupervised setting by performing self-supervision in a sequence-aware manner. It handles out-of-frame motion and occlusions thanks to novel ideas such as full-image warping and multi-frame flow refinement. A recent method [START_REF] Guen | Complementing brightness constancy with deep networks for optical flow prediction[END_REF] improves supervised flow estimation by coupling classical supervised training with an unsupervised criterion based on the brightness constancy equation. Their method allows to do semisupervised training where only a part of the data is annotated, which is an asset for unsupervised domain adaptation.

The key difference between classical optical flow methods and unsupervised methods is that the former performs an optimization per image pairs, while the latter optimizes the loss function across the whole training set. This type of optimization allows information to be shared between image pairs, which can improve the performance of a model.

Data term

Like many classical methods [START_REF] Horn | Determining optical flow[END_REF], unsupervised methods rely on the brightness constancy assumption (BCA), which states that the pixel intensity does not change as it moves. Thus, the primary component of the loss function is photometric consistency. Its goal is to minimize the difference between the source image I 0 and the flow-warped target image I 1 . It can be expressed as follows:

L sim (D, θ) = 1 N p ϕ f D (I 0 (p), I 1 (p ′ )) , (7.37)
where ϕ is a penalty function (e.g., the L2 norm, the L1 norm, the Huber function), f D is the displaced frame difference (f D (I 0 (p), I 1 (p ′ )) = I 1 (p ′ ) -I 0 (p)), and p ′ = p + w(p).

The BCA may be violated in several configurations. As a result, some works have proposed to replace the f D measure with more reliable ones such as the Census loss, which is invariant to additive and multiplicative illumination changes. The Structural Similarity Measure (SSIM) could also be used.

The similarity loss is violated when pixels are occluded or moved out of view between two consecutive frames. To enforce photometric consitency only for non-occluded pixels we have to design an occlusion-aware photometric loss:

L sim (D, θ) = 1 N p O(p) • ϕ f D (I 0 (p), I 1 (p ′ )) , (7.38)
where O is a binary occlusion mask that disables photometric consistency for occluded pixels.

Regularization term

In unsupervised learning, the loss function must involve a form of regularization. Indeed, photometric loss is ambiguous in textureless regions. Similarly to classical methods, it is possible to incorporate an edge-aware first order smoothness term into the overall unsupervised loss function: -if α smoo → 0, smoothness loss does not take into account image edges, -if α smoo → +∞, the smoothness term tends to zero and is therefore neglected in the regularization term, -hence, α smoo must be high enough to acknowledge edge sensitivity, while not exceeding a certain threshold so that it allows for smoothness regularization. This threshold depends on the data we are dealing with.

L smoo = 1 N u∈{x,

Limitations of state-of-the-art deep OF methods

In subsection 7.4.2, we presented RAFT, a supervised deep network architecture for optical flow that achieved state-of-the art performance on major datasets such as KITTI and MPI Sintel, when it was first released. Although its dense global correlation volume is effective for accurate optical flow estimation, it leads to large memory consumption and heavy computation, especially for high-resolution large-scale images. Supervised methods like DIP [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF] and SCV [START_REF] Jiang | Learning optical flow from a few matches[END_REF] have proposed alternatives to the global correlation volume, focusing on finding a good compromise between accuracy on one hand and low memory consumption and computational cost on the other hand. However, these methods still fail to capture the motion of fine structures and small moving objects in satellite images, which we will show in Sec.7.4.5. We believe that this is due to the fact that all of these methods use low-resolution feature maps. In RAFT, the all-pair correlation volume comes at a cost. It stores the matching costs within the entire displacement range, and therefore, requires the feature maps to be downsampled by 1 8 . DIP and SCV reduce the resolution of feature maps by 1 4 . In addition, we need to develop an unsupervised OF method, since no OF ground truth is available for the satellite images.

SoFlow: Unsupervised optical flow method for small moving objects

We propose SoFlow, an unsupervised method for estimating the optical flow of small moving objects in high-resolution images. First, we will describe the network architecture and the loss function that we have defined. Then, we will show how our contributions enable the deployment of the model for satellite imagery.

SoFlow overall framework

Network architecture As mentioned in subsection 7.4.2, unsupervised methods usually start from networks proposed by supervised methods. Since RAFT inspired several deep learning networks for optical flow estimation such as DIP or SMURF, we found it relevant to customize RAFT architecture for an effective motion estimation of small moving objects. Then, our network consists of three main components:

1. Feature and context encoder: We replace the 1 8 downsampling induced by the CNNbased encoder with a 1 2 downsampling by modifying the stride parameter in RAFT original encoder. This choice is motivated by the size of the observed vehicles in the satellite images.

Correlation block:

Starting from the observation that the displacement range of moving objects is constrained in satellite images, we replace the global correlation volume with a local one. The former computes an all-pair visual similarity, while the latter computes the visual similarity between each pixel p in I 0 within a radius R in I 1 . By restricting the search space for optical flow, we reduce the memory consumption and computational cost of correlation computation from O(N 2 ) to O(N (2R + 1) 2 ). We also construct a correlation pyramid by pooling the last two dimensions of the correlation volume. The first level of this pyramid captures the smallest displacements, while the highest level gives information about larger displacements.

3. Iterative update: Similarly to RAFT, this stage includes three main tasks, namely correlation lookup (L), GRU-based residual flow estimation, and optical flow upsampling (U). At each iteration, the half-size predicted flow is upsampled to match the resolution of the ground truth. which is a weighted combination of three terms, a similarity loss term L sim , an edge-aware smoothness term L smoo , and a sparsity term L spar . As a matter of fact, the sparsity term was a late addition to the loss function. Therefore, the experimental results reported below were obrained without the sparsity term.

The similarity and smoothness loss terms are similar to the ones we presented in Eq. (7.38) and Eq.(7.39) respectively, except we discard occlusion reasoning. The third term is formulated as follows:

L

Local correlation volume with manual backward implementation

In order to reduce the memory consumption induced by the correlation block and to improve its performance, we propose an efficient way to compute the local cost volume and the gradient of the loss function with respect to the feature maps. Let us recall the Listing 1 -Per-pixel computation Listing 1 introduces the straightforward approach of computing the local correlation volume. For each pixel p in I 0 , we extract the neighbourhood of its corresponding pixel p' in I 1 and compute the correlation between p and all the pixels within the local neighbourhood of p'. This computation involves the torch.nn.Unfold class, an operation that extracts sliding local blocks from the batched input tensor fmap1. For an input tensor of shape B × C × H × W and a D × D kernel, the operation outputs a tensor of shape B ×(CD 2 )×L, where D 2 is the total number of values within each block and L is the total number of such blocks (HW in our case). The main drawback of the Unfold operation is that it extracts the values in the local blocks by copying them from the large input tensor. This considerably increases the memory usage, and makes this approach unusable on high-resolution images. An alternative to the per_pixel computation is the per_shift computation presented in listing 2. Listing 2 -Per-shift computation A per-shift computation is less memory consuming, since it directly operates on the whole feature maps, and does not require to store the features of the neighbors for every pixel in the source image. Also, instead of building the tensor unfolded_fmap1 of shape B × C × (2R + 1) 2 × (HW ), we construct a smaller tensor padded_fmap1 of shape B × C × (H + 2R) × (W + 2R). Following this approach, we need to pad the target feature on all sides with a "pad" value of R, and compute the matching costs for each shifting position.

Manual backward implementation

The graph structure of our local correlation block can slow down the computations performed by the autograd engine (see Fig. 7.16). Preliminary experiments on the correlation volume also showed that its backward is very slow. In order to improve performance during training, we implement our own customized backpropagation function. This manual backward requires us to express the derivatives of the local correlation volume with respect to its input, namely the source and target feature maps. Therefore, given the output gradient ∇ CV L, we want to compute the input gradients ∇ F 0 L and ∇ F 1 L. Replacing (i, j) and (x, y) by q and p to keep the notation uncluttered, we have:

grad_input p k = ∂L ∂F p k = D 2 -1 d=0 q ∂L ∂C q d ∂CV q d ∂F p k . ( 7

.48)

The correlation volume can also be expressed as follows:

CV q d = F 0 (q) • F 1 (q + shift d ), (7.49) where shift d = g -1 map (d) is the shifting coordinates that correspond to integer d.

-Gradient of the loss w.r.t the source feature map F 0 . For a given shift d, we have:

∂C q d ∂F p 0 =     
F 1 (q + shift d ) if q = p 0 otherwise.

Technical details

We train our model using the loss function defined in subsection 7.4.3. For the similarity loss term, we use SSIM, and the hyperparameters are set as follows: ω sim = 1, ω smoo = 1 and α smoo = 20. So far, we have not investigated the influence of sparsity on the performance of our model, which we keep for future work. The network is trained on a subset of 40 satellite sequences. We keep other sequences for the test and for qualitative (visual) evaluation in order to assess the model generalization ability. In order to make a consistent comparison, we use an absolute normalization of the flow before outputting its HSV color representation. For each sequence, we normalized the flow by an approximation of its maximum flow magnitude. Figure 7.17 -Optical flow computed on PlacaTarraco (between frames 8 and 9) for the three methods. From top to bottom, one image of the sequence, our method SoFlow, SCV method [START_REF] Jiang | Learning optical flow from a few matches[END_REF] and DIP method [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF]. SCV method [START_REF] Jiang | Learning optical flow from a few matches[END_REF] and DIP method [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF].

The visual comparison between the OF results of the three methods, SoFlow, SCV and DIP, reported in Fig. 7.17 and 7.18 confirms that our method outperforms the supervised SCV and DIP methods, while being unsupervised. It is able to distinctly capture the motion of the small moving objects in the high-resolution satellite images. In return, results are a bit noisy in the background. In Fig. 7.17, SCV omits most of the small moving vehicles, while DIP achieves better performance than SCV for some fine structure areas. However, DIP blends the optical flow of adjacent vehicles as shown in the red circled regions in Figure 7.17(d). This blurred OF estimation for DIP and SCV is even more obvious in Fig. 7.18.

As for RAFT, we believe that this tendency is due to the 1 4 downsampling in the feature extraction module. By building higher-resolution feature maps, our method alleviates this problem, since we can clearly distinguish close moving vehicles, as highlighted in the green circled areas in Fig. 7.17(b), or even more clearly in Fig. 7.18. Nevertheless, one weakness of our approach is the handling of the (stabilized) background region. Indeed, inference on both PlacaTarraco and A6C15-Glories sequences shows that the model often predicts a non-zero motion in the stabilized background. One assumption is that our loss function does not handle well enough featureless regions. Indeed, the features of such regions may have a large number of apparently correct matches due to the inherent ambiguity, and it is therefore difficult to quantify how much the flow-warped target image is deviated from the source image. The sparsity term should allow us to correct this, by penalizing non-zero flows in the stabilized background.

Partial Conclusion

We have presented several contributions to improve the computation of optical flow in high-resolution satellite imagery. We have designed the SoFlow method, an unsupervised model that integrates a novel local correlation volume with efficient computational cost and reduced memory consumption. SoFlow is able to correctly handle small moving objects in large-scale satellite image sequences, and outperforms supervised optical flow methods. This confirms that local correlation blocks are suitable for the computation of the optical flow field on satellite images. Furthermore, the fact that we obtain better results with unsupervised learning on real images than with supervised learning on synthetic images suggests that training the model on satellite images helps to obtain more accurate flow fields, as it could be expected.

In future work, we will include the sparsity term in the loss function as described in subsection 7.4.3. A more advanced approach would be to learn an optical flow prior, as done in [START_REF] Yang | Conditional prior networks for optical flow[END_REF], which would help to regularize the flow prediction depending on the satellite images seen at training time. Another interesting perspective would be to introduce an attention mechanism that makes the network focus on meaningful areas (e.g., cars in satellite images), and allows to reduce the computational load on uniform areas that are unlikely to move, while maintaining good accuracy on small moving objects. Résumé : Les contributions de cette thèse sont de deux ordres. Premièrement, nous avons développé une approche non supervisée d'apprentissage profond pour la segmentation du mouvement à partir du flot optique. Nous avons construit à partir de l'algorithme EM une fonction de perte qui implique des modèles de mouvement paramétriques. Nous avons progressivement ajouté de la cohérence temporelle à cette méthode. Avec un triplet de flots en entrée, nous ajoutons un terme de perte imposant des étiquettes cohérentes au sein du triplet. Ensuite, avec des séquences de flot plus longues en entrée, nous définissons une représentation plus fexible du mouvement par splines, et nous nous appuyons sur un transformer pour appréhen-der des interactions à long terme entre les caractéristiques. Ces méthodes fournissent des résultats compétitifs sur les benchmarks, tout en étant très efficaces en inférence. La deuxième contribution porte sur la localisation des mouvements saillants à partir du flot optique. Nous supposons que les zones saillantes sont celles qui influencent la prédiction d'un réseau pré-entraîné de classification de saillance. Nous exploitons une méthode d'interprétation du réseau de type gradient pour localiser les zones saillantes. Nous avons également conçu une approche alternative par réseau adverse. Nous avons appliqué ces deux méthodes à deux tâches de saillance du mouvement. Title: Unsupervised learning for motion segmentation and motion saliency in videos Keywords: motion segmentation, unsupervised deep learning, optical flow, motion saliency Abstract: The contributions of this thesis are two-fold. First, we deal with deep learning approaches for fully unsupervised motion segmentation from an optical flow field. We leverage a loss function based on the EM algorithm and involving parametric motion models. We then gradually extend this framework to longer sequences of input flows. With a triplet of input flows, we introduce a loss term enforcing consistent labels within the triplet, and we add long-term temporal consistency with a specific post-processing. Then, we take longer flow sequences as input, and define a spline-based motion representation to handle the evolution of parametric motion over a long time period. In addition, we rely on a transformer decoder to allow interactions between features of the full sequence. These methods provide competitive results on benchmarks, while being very efficient at test time. The second contribution is the localization of salient motions from the optic flow field. In this part, we assume that salients areas are those that influence the output of a pre-trained saliency classification network. We use a gradientbased network interpretation method to localize salient areas. We also design an alternative adversarial approach. We apply both methods on two motion saliency tasks.

Figure 2 -

 2 Figure 2 -Deux séquences du dataset DAVIS2016 avec la vérité-terrain surimposée en jaune (objet mobile en avant-plan).

Figure 4 -

 4 Figure 4 -Two sequences of DAVIS2016 dataset with ground-truth segmentation mask overlayed in yellow.

Figure 1 . 1 -

 11 Figure 1.1 -Examples of parallax motion in natural scenes. Optical flow fields colored with the HSV color code [59] are shown on the bottom right. From left to right: two flows from our outdoor videos, one flow from a parkour sequence in Davis. In the left sequence, the parallax motion is the apparent motion of the tree trunk, in the center sequence the apparent motion of the pole, and in the right sequence, it is the motion of the fence.

Part , Chapter 1 -

 1 Related Work

Figure 1 . 4 -

 14 Figure 1.4 -Framework of CIS[START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF] with two cases of execution. Reproduced from[START_REF] Yang | Unsupervised moving object detection via contextual information separation[END_REF].

Part , Chapter 1 -

 1 Related Work

Figure 1 . 5 -

 15 Figure 1.5 -Pipeline of the dynamic saliency method. Reproduced from [12].

Part , Chapter 2 -Figure 2 . 1 -

 221 Figure 2.1 -Flowchart of the proposed CNN method for the training (top) and inference (bottom) steps.Training step: First, we segment the optical flow field f with the neural network g ϕ . Then, we get the optimal parametric motion models {f θ * k } k=1,...,K within each probabilistic segmentation masks {g ϕ (f ) k } k=1,...,K using (2.16). Finally, we update the parameters ϕ of the neural network using (2.17), where the loss function is defined in (2.12). This training step is performed iteratively over each batch B (of size 1 in this illustration). Inference step: We directly apply the trained network g ϕ * to any new unseen optical flow field f to obtain the probabilistic segmentation masks {g ϕ * (f ) k } k=1,...,K . There is no estimation of the motion models {f θ * k } k=1,...,K in the inference step in contrast to the training step. For the sake of visualization, optical flows and polynomial motion models are represented with the HSV color code, but actually, the flow field f used as input of the neural network is taken as a 2D vector field. We have a two-channel input. Optical flow coding: correspondence between the arrow visualization of the optical flow field and the HSV color map.

Figure 2 . 3 -

 23 Figure 2.3 -Examples of motion segmentation results obtained with our method with two masks, on the videos bmx-trees, breakdance-flare, scooter-black, kite-surf, blackswan, parkour, dogs02 and cuttlefish1, from the DAVIS2016, FBMS59 and MoCA datasets. First row: a frame of the video with the ground-truth superimposed in yellow. Second row: the input flow field displayed with the HSV color code [59] that is depicted in Fig.2.1. Third row: the segmentation produced by our method superimposed in green on the corresponding image.

  3.4. -Quad. Model: replacing the full quadratic motion model with the affine one. -L 1 norm (a): substitution of the robust loss function (L 1 norm) by its non-robust counterpart, the squared L 2 given in subsection 2.3.2. -L 1 norm (b): substitution of the robust loss function (L 1 norm) by the L 2 norm given in subsection 2.3.2.

Table 2 . 3 -

 23 without DA Quad. Model L 1 norm (a) L 1 norm (b) full method J Mean ↑ 58.Ablation study for different components of our method. Each time, we suppress or modify only one component, respectively, removal of the data augmentation, substitution of the quadratic motion model by the affine motion model, substitution (a) of the L 1 norm by the squared L 2 , substitution (b) of the L 1 norm by the L 2 norm. J is given as the average of the average frame Jaccard index obtained for each of the four datasets presented in Section 2.4.2.

Figure 2 . 4 -

 24 Figure2.4 -Sensitivity with respect to the initialization of the classical EM algorithm in contrast to our trained model. First, we plot the evolution of the (speculative) optimal Jaccard index of the classical EM algorithm of Section 3 of the main text (green curve) with respect to the number of available random initialisations of the parametric motion parameters θ on the DAVIS2016 validation set. For all videos, the optimal (speculative) Jaccard index is obtained by selecting the best segmentation for each input flow among all available initialisations. Secondly, the yellow curve plots the evolution of the Jaccard index of the classical EM algorithm over the available initialisations, when selecting the initialisation with the highest likelihood, the only means available in practice without ground-truth. The dashed red horizontal line corresponds to the value of the Jaccard index obtained with our network on the same validation set. The dashed blue vertical line indicates when the optimal Jaccard index of the classical EM algorithm exceeds the score of our network. The figure on the right involves a logarithmic scale to ease the visualisation of the first part of the graph. In further experiments, we launched 6 training runs of our neural network and we evaluated them on DAVIS2016; we obtained an average score of 68.7 and a variance of 0.41, showing the high stability of our method.

Part , Chapter 2 -

 2 EM-driven Unsupervised Learning for Motion Segmentationdependency of the classical EM to the initialization of the motion model parameters.

Figure 2 . 6 -

 26 Figure 2.6 -Illustration of the implicit temporal consistency ensured (to some extent) by our method with four masks. For each group, first row: input optical flow fields displayed with the HSV color code, second row: OFS maps. The color code is the same for all segmentation maps (dark blue: mask 1, light blue: mask 2, green: mask 3, yellow: mask 4). Examples are drawn from DAVIS2016 (from top to bottom): libby, dance-twirl, and car-roundabout videos.

Part , Chapter 3 -

 3 Temporally-consistent Segmentation of Multiple Motionsformulation of the motion segmentation problem.

Figure 3 . 1 -

 31 Figure 3.1 -Principle diagram of our space-time multiple motion segmentation network ψ, taking as input a space-time sub-volume composed of three flow fields sampled every τ time instants and delivering a sub-volume of three coherent segmentation maps.

Figure 3 . 2 -

 32 Figure 3.2 -Output of the network by triplet (for τ = 1) within a subsequence covering the time interval [0, T ]. The lower index t of each mask mt ′ t represents the time instant of the corresponding flow field f t , and the upper one t ′ corresponds to the time instant when it is produced, that is, the one of the reference (central) flow field of the triplet. Segmentation masks with the same lower index correspond to different segmentation instances of the same flow field.

Part , Chapter 3 -Figure 3 . 5 -

 335 Figure 3.5 -Comparison of the selected segments with the ground truth for evaluation.

Figure 3 . 6 -

 36 Figure 3.6 -Evaluation of our temporal segment linkage and segment selection process for different subsequence lengths on the three datasets.

Figure 3 . 10 -

 310 Figure 3.10 -Different instants of goats1 video of FBMS59. First row: input optical flow (HSV color code). Second row: motion segments predicted by the network (before applying the global temporal linkage), and superimposed on the video frame (except background mask); when static, goats are merged with backgound.

Figure 4 . 1 -

 41 Figure 4.1 -Overall architecture of our multiple motion segmentation method ensuring temporal consistency with the consistency loss term L c and the B-splines space-time motion models θ k (for k = 1, .., K). It takes as input a volume composed of τ flow fields. It comprises a 3D U-net (e and d boxes) and a transformer decoder (t box). It also involves positional encoding. A cross-attention product yields the K segmentation masks corresponding to the input volume. For the sake of clarity, the block diagram is represented for three motion segments (K = 3). L r is the flow-reconstruction loss term.

Figure 4 . 2 -

 42 Figure 4.2 -Illustration of the space-time spline-based motion model. Top row: the input flows displayed with the HSV code for the swing video of DAVIS2016 dataset, the binary segmentation ground truth, the flows generated by the estimated spline-based motion models for the two segments. Middle-row: the plot of the temporal evolution of the six estimated model parameters corresponding to the flow u-coordinate for the foreground moving object. Bottom-row: the plot of the temporal evolution of the six estimated model parameters corresponding to the flow v-coordinate for the foreground moving object.

Figure 4 . 3 -

 43 Figure 4.3 -Influence of the length of the input flow sequence (also referred to as cut size on the horizontal axis of the plot) on the three modified versions of our LT-MS-K4 method with comparison with the full model. Left plot: for the DAVIS2016 dataset. Middle plot: for the FBMS59 dataset. Right plot: for the SegTrackV2 dataset. Overall, they exhibit comparable behaviour at a certain distance from the full model.

Figure 4 . 4 -

 44 Figure 4.4 -Four groups of qualitative results regarding the ablation of the temporalconsistency loss term. They respectively correspond to the worm video of SegTrackV2, the dogs01 video of FBMS59, and the kite-surf and car-roundabout videos of DAVIS2016.For each group, the first row contains sample images with the segmentation ground-truth, when available at that frame, overlaid in yellow, the second row displays the input flows, the third and fourth rows show the predicted motion segmentations, respectively without and with the temporal consistency loss term. Clearly, this model component allows us to get far more consistent segments over time.

Figure 4 . 5 -

 45 Figure 4.5 -Visual assessment of the contribution of the spline-based motion model on the dog video of DAVIS2016 dataset (with K = 4). From top to bottom: input flows of the sequence; corresponding predicted motion segmentation maps; flows reconstructed with the space-time polynomial motion model; plots over time of the mean-value of the u-components of the flows provided per segment by the space-time polynomial motion models; plots over time of the mean-value of the v-components of the flows provided per segment by the space-time polynomial motion models; flows reconstructed with the spacetime spline-based motion model; plots over time of the mean-value of the u-components of the flows provided per segment by the spline-based motion models; plots over time of the mean-value of the v-components of the flows provided per segment by the spline-based motion models. Clearly, the space-time polynomial model fails to handle non-uniformly varying motion, whereas the spline-based model does.

120

 120

4

 4

Figure 4 . 6 -

 46 Figure 4.6 -Influence of the length of the input flow sequence (also referred to as cut size on the horizontal axis of the plot) on the LT-MS-K4 method for three datasets, DAVIS2016, FBMS59 and SegTrackV2. As expected, the best performance is obtained for the smallest length (cut size equal to 10), but performance decreases slowly when the cut size increases and remains stable for larger ones. On the right, the plot is shown with the full range of values on the ordinate [0, 100]. On the left, a zoomed-in version of the plot.

Figure 4 . 8 -

 48 Figure 4.8 -Illustration of the temporal consistency provided by our LT-MS-K4 method for three examples, the birdfall, bird-of-paradise, and bmx videos of SegTrackV2. For each group, the first row contains the video images with the ground truth overlaid in yellow when available; the second row depicts the corresponding flow fields represented with the HSV code while normalized independently from each other; the third row provides the predicted segmentation.

Figure 5 . 1 -

 51 Figure 5.1 -Modification of the ordering of the network inner layers. Left: Order for training and test. Right: Order for interpretation. Weights of each layer remain unchanged. For interpretation, Convolution and Batch Norm layers are merged into a functionally equivalent convolutional layer.

FTS [ 104 ]

 104 NIMS (Ours) J Mean ↑ -(71.5) 67.6 (62.6) 58.6 (56.2) 62.5 (-) 57.5 (55.8) 63.0 (60.2) F Mean ↑ -(70.5) 63.9 (59.6) 47.5 (45.6) 59.3 (-) 53.6 (51.1) 68.2 (65.8) J Recall ↑ -(86.5

Figure 5 . 2 -

 52 Figure 5.2 -Starting from the optical flow (left), we replace the flow in the selected regions by inpainting. When the tree-trunk region in the foreground is ablated, the frame is still classified as dynamically salient (middle). In contrast, ablating regions of the two moving people switches the frame as non salient (right). See the prediction score above each frame.

Figure 5 . 4 -

 54 Figure 5.4 -Results obtained with our method (NIMS). From left to right, two outdoor videos we acquired, five examples from the DAVIS2016 dataset, respectively, parkour, libby, dance-twirl, flamingo and kite-surf videos, and the drive video of the Complex dataset. Top row: one image of the video. Middle row: Optical flow in HSV color code; Bottom row: Motion saliency maps. The closer to yellow, the higher the motion saliency degree.

Figure 5 . 6 -

 56 Figure 5.6 -From top to bottom : Input optical flow displayed with the classical HSV color code, LRP attribution map from the random classification network and LRP attribution maps obtained using our trained classification network.

Figure 5 . 7 -

 57 Figure 5.7 -Qualitative results on several sequences from the DAVIS2016 validation dataset. From top to bottom : RGB frame with GT segmentation mask overlayed in yellow, optical flow displayed with the HSV color code introduced in section 2.3, LRP attribution map, results obtained with Huber segmentation, results obtained with our ST-MS deep-learning method using the "except biggest" segment selection heuristic, and results obtained by combining the ST-MS method with LRP-based segment selection. All sequences are taken from the DAVIS2016 dataset, from left to right: bear, breakdance, camel, parkour (2), libby and dog.

Figure 6 . 2 -

 62 Figure 6.2 -Inference step of the adversarial network. From left to right: the input optical flow field in HSV color code, the segmentation network, the segmentation with all masks (background in yellow, salient in blue, other motions in green), the salient mask (extraction of the blue mask).

  ) with θ k = arg min θ p∈Ω m k (p)||f (p) -fθ (p)|| 1 .(6.12)

Figure 6 . 3 -

 63 Figure 6.3 -Results regarding the task of independent motion detection on DAVIS2016 dataset. From left to right: breakdance, car-roundabout, parkour, camel, dog videos. From top to bottom: RGB frame, optical flow field (HSV color code), predicted salient motion mask.

Part , Chapter 6 -Figure 6 . 4 -

 664 Figure 6.4 -Results on simulated sequences from Chaos dataset. Top to bottom : RGB frame, optical flow field (HSV color code), predicted salient motion mask.

Figure 6 . 5 -

 65 Figure 6.5 -Results from left to right on Wrong Way sequence (at two different time instants frames), on real sequences of UCSD dataset (two examples), and on an outdoor scene acquired for the experiment (one example). From top to bottom: RGB frame, optical flow field (HSV color code), predicted salient motion mask.

)

  where θ ∈ R12 are the parameters of a parametric (quadratic) motion model, m ∈ [0, 1] W ×H×K is a segmentation mask and f ∈ R W ×H×2 is the input optical flow. We define f ζ ∈ R W ×H×2 as a parametric flow field generated using parameters θ ζ and positions c(i) as f ζ,i = θ ζ • c(i). We show that for all possible segmentation m and parameters θ ζ we have : max θ ll(θ, m, f ) = max θ ll(θ, m, f + f ζ ). (7.13)

Figure 7 . 1 -

 71 Figure 7.1 -Data augmentation adding a quadratic global motion on the optical flow field. From top to bottom : Optical flow field displayed with the usual HSV code [59];Predicted mask with our network trained using data augmentation; Predicted mask with our network trained without data augmentation; Ground-truth segmentation mask.

Theorem 2 .

 2 Let a translationally invariant function δ : R D × R D → R such that, ∀ a, e, c ∈ R D × R D , δ(a + c, e + c) = δ(a, e), and a probability density defined as: p(y|x; θ) = 1 Z exp(-δ(y, θ T x)). (7.24)

. 27 )Figure 7 . 3 -

 2773 Figure 7.3 -Examples of motion segmentation results obtained with our method with two masks, on the bmx, birds of paradise and drift videos of the SegTrackV2 dataset. First row: a frame of the video with the ground-truth superimposed in yellow. Second row: the input flow field displayed with the HSV color code. Third row: the segmentation produced by our method superimposed in green on the corresponding image.

Figure 7 . 5 -

 75 Figure 7.5 -Model scores depending on initialization. Each dot represents a trained model. The x-axis represents the validation loss on the DAVIS 2016 train dataset, and the y-axis the performance on the evaluation dataset. The top row includes all models and the bottom row excludes models that diverged during training (validation loss > 0.25). The bottom row displays the linear relationship between validation loss and evaluation score. The model whose results are reported in the main text is represented as an orange dot.

Figure 7 . 6 - 5 Figure 7 . 7 -

 76577 Figure 7.6 -Evolution of the model performance as a function of the initialization budget. Left : the validation loss associated to the best network of each subset. Right : average performance on the test data set associated to this network for each different budget. The filled area correspond to +/-the standard deviation for each curve.

Figure 7 . 8 -

 78 Figure 7.8 -Qualitative results obtained with LT-MS-SA (including the slot attention decoder) with a variable number of masks at inference time. The first row of the two groups of results is the optical flow field (HSV color code), then, from top to bottom, we display the motion segmentation from 6 to 2 masks using the same model. In order to push the model to keep coherent labels across levels, we keep the same set of initial queries and give only the requested number for each level as input. The two sequences are breakdance-flare and camel from the DAVIS2016 dataset.

Figure 7 . 9 -

 79 Figure 7.9 -Comparing RAFT (middle row) and Brox's method (bottom row) predictions on two sequences CenterCo (top left) and PlacaTarraco (bottom right).

Figure 7 . 10 -

 710 Figure 7.10 -Iterative refinement of the flow using the coarse-to-fine warping-based approach.

Figure 7 . 11 -

 711 Figure 7.11 -Iterative refinement of the flow using a single high-resolution flow update

Figure 7 .

 7 Figure 7.12 -A 3-layer correlation pyramid

  (a) GRU-based update operator (b) Inside a ConvGRU block (source: towardsdatascience.com -Illustrated guide to LSTMs and GRUs)

Figure 7 .

 7 Figure 7.13 -Principle of the recurrent optical flow update

Figure 7 . 14 -

 714 Figure 7.14 -Correlation lookup (r=1)

2 + w GT,y -w y 2 .

 22 GT (p) -w(p)∥ 2 , (7.33)where ∥w GT -w∥ 2 = w GT,x -w x

Figure 7 .

 7 Figure 7.15 -SoFlow architecture

F. 44 )

 44 mathematical expression of the local correlation volume (CV): CV = F 0 (p) • F 1 (p + d) | (p, d) ∈ X × D (0,c (p)F 1,c (p + d) | (p, d) ∈ X × D , (7Efficient computation of the correlation volumeTo effectively build the cost volume, we adopt a per-shift computation rather than a per-pixel computation:-Per-pixel computation. For each pixel p in I 0 , one extract and store the values of its neighbors in I 1 before computing their dot product. This approach considers the 4D local correlation volumeCV ∈ R D 2 ×H×W as a stack of H × W correlation maps of dimension D 2 . -Per-shift computation. For each neighboring position d in D = [-R, R] 2 ∩ Z 2 ,we compute the correlation between pixels p in I 0 and their neighbors at position p+d in I 1 . These neighbors are obtained by shifting the target feature map by d. This approach allows us to formulate the 4D correlation volume CV as a stack of D 2 correlation maps CV d ′ of dimensions H × W , each representing the correlation between the source feature map and the d-shifted target feature map. The integer d ′ refers to a pointer that scans the grid D in lexicographic order.In the following, each shift vector d will be mapped by g map to its corresponding index given by the integer value d ′ :g map : D -→ {0, . . . , D 2 -1} (7.45) d = (d x , d y ) -→ d ′ (7.46)This mapping is designed such that the first cost volume CV 0 stores the similarity costs between pixels p in I 0 and their top-left neighbor p+d = (x -R, y -R) in I 1 , while the last cost volume CV D 2 -1 stores the correlations between pixels p in I 0 and their bottom-right neighbor p+d = (x + R, y + R) in I 1 . Also, when computing the similarity between a pixel p in I 0 and its corresponding pixel at the same position in I 1 , there is no need to shift the target feature map, which implies thatd ′ = D 2 -1import torch from shapecheck.ShapeCheck import ShapeCheck def _compute_local_corr_volume(fmap0: torch.Tensor, fmap1: torch.Tensor, radius: int): b, c, h, w = fmap0.shape D = 2 * radius + 1 norm = torch.sqrt(torch.tensor(c)) sc = ShapeCheck(fmap0.shape, 'b c h w') sc.update([D, D],'pH pW') unfold = torch.nn.Unfold(kernel_size=D, padding=(radius,radius)) unfolded_fmap1 = sc.rearrange(unfold(fmap1), 'b (c pH pW) (h w) -> b c (pH pW) (h w)') flat_fmap0 = sc.rearrange(fmap0, 'b c h w -> b c (h w)') corr = torch.einsum('Bcm,Bcpm', flat_fmap0, unfolded_fmap1)/norm corr = sc.rearrange(corr, 'b (pH pW) (h w) -> b h w pH pW') return corr

  ShapeCheck import torchvision.transforms as T def _compute_lazy_local_corr_volume(fmap0: torch.Tensor, fmap1: torch.Tensor, radius: int): b, c, h, w = fmap0.shape D = 2 * radius + 1 norm = torch.sqrt(torch.tensor(c)) sc = ShapeCheck(fmap1.shape, 'b c h w') sc.update([D, D],'pH pW') padded_fmap1 = T.Pad(padding=radius)(fmap1) corr = torch.zeros(b,D**2,h,w, device = fmap0.device) for d in range(0,D**2) : # loop over shift dx, dy = d//D,d%D corr[:,d,...] = ((fmap0 * padded_fmap1[:,:,dx:h+dx,dy:w+dy]).sum(axis=1))/norm corr = sc.rearrange(corr, 'b (pH pW) h w -> b h w pH pW') return corr

Figure 7 .

 7 Figure 7.16 -Computation graph of the correlation block

(7. 50 )-F 0 53 )

 50053 Back to Eq.(7.48), we can conclude that: Gradient of the loss w.r.t the target feature map F 1 . For a given shift d: (q) if q = pshift d Like the architecture itself, the backward is implemented in Pytorch. The implementation details are shown in Fig.7.16.

Figures 7 .

 7 Figures 7.17and 7.18 depict a visual comparison of optical flow estimates on the test sequence PlacaTarraco and the training sequence A6C15-Glories, respectively.In order to make a consistent comparison, we use an absolute normalization of the flow before outputting its HSV color representation. For each sequence, we normalized the flow by an approximation of its maximum flow magnitude.

17

 17 Figures 7.17and 7.18 depict a visual comparison of optical flow estimates on the test sequence PlacaTarraco and the training sequence A6C15-Glories, respectively.In order to make a consistent comparison, we use an absolute normalization of the flow before outputting its HSV color representation. For each sequence, we normalized the flow by an approximation of its maximum flow magnitude.

  Figures 7.17and 7.18 depict a visual comparison of optical flow estimates on the test sequence PlacaTarraco and the training sequence A6C15-Glories, respectively.In order to make a consistent comparison, we use an absolute normalization of the flow before outputting its HSV color representation. For each sequence, we normalized the flow by an approximation of its maximum flow magnitude.

Figure 7 . 18 -

 718 Figure7.18 -Optical flow computed on A6C15-Glories (between frames 7 and 8) for the three methods. From top to bottom, one image of the sequence, our method SoFlow, SCV method[START_REF] Jiang | Learning optical flow from a few matches[END_REF] and DIP method[START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF].

Titre:

  Apprentissage non supervisé pour la segmentation et la saillance du mouvement dans des vidéos Mot clés : segmentation, mouvement, apprentissage non supervisé, flot optique, saillance

Table 2 .

 2 Part , Chapter 2 -EM-driven Unsupervised Learning for Motion Segmentation

	Method	Training	Input	DAVIS2016 SegTrack V2 FBMS59 MoCA
				J	F	J	J	J
	Ours			69.3 70.7	55.5	57.8	61.8
	MoSeg [46]		Flow	68.3 66.1	58.6	53.1	63.4
	FTS [104]	Fully Unsupervised		55.8		47.8	47.7	-
	TIS 0 [68]			56.2 45.6	-	-	-
	TIS s [68]			62.6 59.6	-	-	-
	CIS -No Post [47]		Image & Flow	59.2		45.6	36.8	49.4
	CIS -With Post [47]			71.5		62.0	63.6	54.1
	DyStab -Dyn [48]		Flow	62.4		40.0	49.1	-
	DyStab -Stat&Dyn [48]	Supervised Features	Image & Flow 80.0		73.2	74.2	-
	ARP [27]		Image & Flow 76.2 70.6	57.2	59.8	-
	MATNet [19]	Supervised	Flow	82.4 80.7		64.2
	COSNet [24]		Image	80.5 79.5	-	75.6	50.7

2 -Results on DAVIS2016 validation dataset, SegTrackV2, FBMS59 validation dataset and MoCA for several unsupervised and supervised methods (scores taken from

Table 2 .

 2 4 -Quantitative comparaison between classical EM and our trained network on several aspects. Inference time is computed on Tesla-V100 in all cases. Accuracy (average Jaccard) is evaluated on DAVIS2016 validation set. The size of the input flow is noted (W,H) and the number of layers K. Chapitre3/figures for the classical EM are given for two configurations: 1) just applying it once for a random initialisation, but for the accuracy score and a fair comparison, we prefer to give the mean and standard deviation of the average Jaccard index estimated over 3000 initialisations; 2) selecting the best initialisation on every frame among 500 initialisations.

	Classical EM	Our network

implementation, it takes 25 minutes to run the 41 initialisations per sample over the DAVIS2016 validation set on a Tesla P100.

Table 4 . 2

 42 .4. Experimental Results

	Method	Training	Input	DAVIS2016 SegTrack V2 FBMS59
				J ↑ F ↑	J ↑	J ↑
	LT-MS-K4			74.8 72.2	61.3	61.0
	LT-MS-K2			70.3 68.5	58.6	55.3
	ST-MS (Chapter 3) [122]		Flow	73.2 70.3	55.0	59.4
	EM (Chapter 2) [55]	Unsupervised		69.3 70.7	55.5	57.8
	MoSeg [46]			68.3 66.1	58.6	53.1
	FTS [104]			55.8	-	47.8	47.7
	TIS 0 [68]			56.2 45.6	-	-
	OCLR* [26] (flow only)			72.1	-	67.6	65.4
	GWM [3]		RGB (Flow in loss)	79.5	-	78.3	77.4
	RCF [21]			80.9	-	76.7	69.9
	AMD [5]			57.8	-	57.0	47.5
	MOD [106]			73.9	-	62.2	61.3
	DivA(4) † [49]			72.4	-	64.6	-
	TIS s [68]		RGB & Flow	62.6 59.6	-	-
	CIS -No Post [47]			59.2	-	45.6	36.8
	CIS -With Post [47]			71.5	-	62.0	63.6
	DyStab -Dyn [48]		Flow	62.4	-	40.0	49.1
	DyStab -Stat&Dyn [48]	Supervised Features	RGB & Flow	80.0	-	73.2	74.2
	ARP [27]			76.2 70.6	57.2	59.8
	MATNet [19]	Supervised	RGB & Flow	82.4 80.7	50.4	76.1
	COSNet [24]		RGB	80.5 79.5	49.7	75.6

-Results obtained with two versions of our LT-MS method on DAVIS2016, SegTrackV2, FBMS59. LT-MS-K2 performs segmentation with only two masks (K = 2)

Table 4 .

 4 3 -Results obtained on the three datasets DAVIS2016, FBMS59 and SegTrackV2 with LT-MS-K2 for respectively a cut size of 10 and no cut (the video sequence is processed in one go).

	DAVIS2016 FBMS59 SegTrackV2
	Cut Size 10	∞	10	∞	10	∞
	LT-MS-K2 70.3 70.7 55.3 48.7 58.6 59.3
	Method / Scores	J &F ↑ J ↑ F ↑
	LT-MS-K3		42.2		39.3 45.0
	ST-MS (Chapter 3) 42.0		38.8 45.2
	MoSeg [46]		35.8		38.4 33.2
	OCLR* [26]		55.1		54.5 55.7
	Table

Table 7 .

 7 1 -Results given for every sequence of DAVIS2016 dataset. Reported score is the average Jaccard score over frames in the sequence. Last row is the average over sequences scores. J is the Jaccard index and F is the Countour Accuracy. The Mean (M ) is the average of the score, the Recall (O) is the fraction of frames with a score higher than 0.5 and the Decay (D) is the degradation of the score over time in the sequence. More details in[START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF].

	.608	1	-0.022

Table 7 .

 7 2 -Results given for every sequence of SegTrackV2 dataset. Reported score is the average Jaccard score over annotated frames in the sequence.

	FBMS59
	Sequence	Jacc (J )
	camel01	65.0
	cars1	87.1
	cars10	33.2
	cars4	86.6
	cars5	83.7
	cats01	68.7
	cats03	78.3
	cats06	39.4
	dogs01	72.9
	dogs02	69.8
	farm01	81.0
	giraffes01	34.2
	goats01	43.7
	horses02	75.9
	horses04	67.8
	horses05	43.0
	lion01	56.1
	marple12	50.0
	marple2	70.4
	marple4	80.6
	marple6	35.5
	marple7	62.4
	marple9	31.0
	people03	53.3
	people1	78.7
	people2	86.1
	rabbits02	44.6
	rabbits03	40.2
	rabbits04	42.9
	tennis	72.3
	Seq. Avg.	61.1
	Frames. Avg.	57.8

Table 7 .

 7 3 -Results given for every sequence of FBMS59 dataset. Reported score is the average Jaccard score over annotated frames in the sequence.

	MoCA

Table 7 .

 7 4 -Results given for every sequence of MoCA dataset. Reported score is the average Jaccard score over frames in the sequence computed using bounding box annotation as described in Section 5.2. of the main text.

	Sequence	Jacc (J )
	arabian horn viper	71.0
	arctic fox	37.8
	arctic fox 1	85.5
	arctic wolf 0	63.4
	arctic wolf 1	50.8
	bear	61.3
	black cat 0	53.5
	black cat 1	8.6
	crab	68.1
	crab 1	29.3
	cuttlefish 0	23.5
	cuttlefish 1	20.0
	cuttlefish 4	64.5
	cuttlefish 5	72.1
	dead leaf butterfly 1	75.7
	desert fox	27.0
	devil scorpionfish	90.3
	devil scorpionfish 1	92.5
	devil scorpionfish 2	85.7
	egyptian nightjar	72.4
	elephant	80.4
	flatfish 0	59.9
	flatfish 1	64.8
	flatfish 2	82.1
	flatfish 4	22.1
	flounder	89.3
	flounder 3	21.4

Table 7 .

 7

5 -Results (Jaccard index) of five experiments, involving different initialisations of the network, on the three datasets. Reported results in Chapter 3 correspond to experiment 1.

  .6. Clearly, as expected, the data augmentation improves performance.

	Data Augmentation Davis Val SegTrackV2 FBMS
	With	73.2	55.0	59.4
	Without	70.1	52.3	50.4
	Table 7.6 -Impact of the data augmentation. Scores (Jaccard index) obtained on the
	three datasets.

Table 7 .

 7 7 -Impact of the use, at training time, of different values for the time interval τ in the input flow triplet. Scores (Jaccard index) obtained on the three datasets.

	τ while training	Davis Val SegTrackV2 FBMS
	{1}	73.2	55.0	59.4
	{1,2,3,4,5,9,10,12} 73.0	54.3	57.9

Table 7 .

 7 [START_REF] Vahdani | Deep learning-based action detection in untrimmed videos: A survey[END_REF] -Results given for every video of SegTrackV2 dataset. Each reported score is the average Jaccard score over annotated frames in the video. The very last row is the average over all the frames and over all the videos.

	SegTrackV2

Table 7 .

 7 10 -Results given for every video of FBMS59 dataset. Each reported score is the average Jaccard score over annotated frames in the video. The very last row is the average over all the annotated frames and over all the videos.

	DAVIS2017-motion
	Sequence	J-Mean F-Mean
	bike-packing_1	0.072	0.370
	bike-packing_2	0.276	0.393
	blackswan_1	0.577	0.593
	bmx-trees_1	0.520	0.766
	breakdance_1	0.365	0.558
	camel_1	0.716	0.683
	car-roundabout_1	0.900	0.814
	car-shadow_1	0.870	0.804
	cows_1	0.778	0.675
	dance-twirl_1	0.441	0.641
	dog_1	0.481	0.456
	dogs-jump_1	0.433	0.506
	dogs-jump_2	0.018	0.174
	dogs-jump_3	0.190	0.251
	drift-chicane_1	0.381	0.562
	drift-straight_1	0.811	0.739
	goat_1	0.275	0.303
	gold-fish_1	0.175	0.270
	gold-fish_2	0.027	0.325
	gold-fish_3	0.168	0.209
	gold-fish_4	0.000	0.000
	gold-fish_5	0.000	0.000
	horsejump-high_1	0.562	0.783
	india_1	0.057	0.066
	india_2	0.047	0.110
	india_3	0.143	0.186
	judo_1	0.247	0.357
	judo_2	0.417	0.522
	kite-surf_1	0.422	0.418
	lab-coat_1	0.337	0.313
	libby_1	0.730	0.847
	loading_1	0.166	0.226
	loading_2	0.068	0.210
	loading_3	0.407	0.416
	mbike-trick_1	0.644	0.683
	motocross-jump_1	0.509	0.481
	paragliding-launch_1 0.339	0.237
	parkour_1	0.682	0.753
	pigs_1	0.253	0.394
	pigs_2	0.019	0.313
	pigs_3	0.369	0.432
	scooter-black_1	0.856	0.750
	shooting_1	0.575	0.500
	soapbox_1	0.726	0.779
	J&FMean JMean JRecall JDecay FMean FRecall FDecay
	0.420	0.388	0.365	0.006	0.452	0.454	0.039

Table 7 .

 7 [START_REF] Zhang | Video saliency prediction based on spatial-temporal twostream network[END_REF] -Results given for every video of DAVIS2017-motion dataset. The very last row is the average score over all the videos for the different criteria.

Table 7 .

 7 [START_REF] Zhou | A survey on deep learning technique for video segmentation[END_REF] -Results given for every video of DAVIS2016 dataset. Reported scores per video are the average Jaccard score over frames in the video. The very last row is the average over videos scores. J is the Jaccard index and F is the Countour Accuracy. The Mean (M ) is the average of the scores, the Recall (O) is the fraction of frames per video with a score higher than 0.5, and the Decay (D) is the degradation of the score over time in the video.

	D)

Table 7 .

 7 [START_REF] Perazzi | A benchmark dataset and evaluation methodology for video object segmentation[END_REF] -Results given for every video of SegTrackV2 dataset. Each reported score is the average Jaccard score over annotated frames in the video. The very last row is the average over all the frames and over all the videos.

	FBMS59
	Video	Jacc (J )
	camel01	0.313
	cars1	0.878
	cars10	0.344
	cars4	0.713
	cars5	0.856
	cats01	0.705
	cats03	0.72
	cats06	0.365
	dogs01	0.73
	dogs02	0.727
	farm01	0.817
	giraffes01	0.367
	goats01	0.448
	horses02	0.637
	horses04	0.717
	horses05	0.467
	lion01	0.564
	marple12	0.604
	marple2	0.822
	marple4	0.865
	marple6	0.539
	marple7	0.721
	marple9	0.723
	people03	0.587
	people1	0.766
	people2	0.881
	rabbits02	0.498
	rabbits03	0.409
	rabbits04	0.483
	tennis	0.714

Table 7 .

 7

	DAVIS2017-motion
	Sequence	J-Mean F-Mean
	bike-packing 1	0.078	0.327
	bike-packing 2	0.254	0.267
	blackswan 1	0.477	0.575
	bmx-trees 1	0.608	0.804
	breakdance 1	0.450	0.526
	camel 1	0.772	0.726
	car-roundabout 1	0.913	0.810
	car-shadow 1	0.896	0.845
	cows 1	0.821	0.728
	dance-twirl 1	0.444	0.576
	dog 1	0.654	0.571
	dogs-jump 1	0.019	0.147
	dogs-jump 2	0.254	0.323
	dogs-jump 3	0.303	0.351
	drift-chicane 1	0.557	0.627
	drift-straight 1	0.886	0.829
	goat 1	0.220	0.300
	gold-fish 1	0.018	0.240
	gold-fish 2	0.282	0.336
	gold-fish 3	0.357	0.356
	gold-fish 4	0.000	0.000
	gold-fish 5	0.000	0.000
	horsejump-high 1	0.721	0.794
	india 1	0.066	0.085
	india 2	0.163	0.172
	india 3	0.072	0.108
	judo 1	0.225	0.397
	judo 2	0.286	0.414
	kite-surf 1	0.449	0.464
	lab-coat 1	0.332	0.350
	libby 1	0.746	0.853
	loading 1	0.057	0.246
	loading 2	0.128	0.259
	loading 3	0.427	0.514
	mbike-trick 1	0.444	0.436
	motocross-jump 1	0.446	0.435
	paragliding-launch 1 0.577	0.286
	parkour 1	0.531	0.646
	pigs 1	0.105	0.433
	pigs 2	0.437	0.431
	pigs 3	0.059	0.217
	scooter-black 1	0.843	0.724
	shooting 1	0.434	0.568
	soapbox 1	0.487	0.696
	J&FMean JMean JRecall JDecay FMean FRecall FDecay
	0.422		0.393	0.387	0.004	0.450	0.437	0.024

[START_REF] Tokmakov | Learning motion patterns in videos[END_REF]

-Results given for every video of FBMS59 dataset. Each reported score is the average Jaccard score over annotated frames in the video. The very last row is the average over all the annotated frames and over all the videos.

Table 7 .

 7 [START_REF] Cheng | SegFlow: Joint learning for video object segmentation and optical flow[END_REF] -Results given for every video of DAVIS2017-motion dataset. The very last row is the average score over all the videos for the different criteria.

  Intensity edge sensitivity in Eq.(7.39) is weighted by α smoo > 0:

			y} p	∇ u w(p)	1	.e -αsmoo|∇uI 0 (p)| ,	(7.39)
	where ∇ u w(p)	1	= ∂wx ∂x (p) + ∂wx ∂y (p) + ∂wy ∂x (p) + ∂wy ∂y (p) .

Nous montrons que l'avantage de ces modèles de mouvement est qu'ils peuvent représenter une évolution temporelle significative des modèles de mouvement paramétriques. Nous introduisons également une architecture de type maskformer[START_REF] Cheng | Per-pixel classification is not all you need for semantic segmentation[END_REF] pour segmenter le flot optique et un module d'attention temporelle qui permet l'interaction entre les caractéristiques sur l'ensemble de la séquence. Nous montrons que la combinaison de ces deux contributions permet d'entraîner un réseau capable de segmenter de longues séquences de flot optique de manière non supervisée. Notre réseau de segmentation de mouvement a été testé sur quatre benchmarks, DAVIS2016, DAVIS2017, SegTrackV2, FBMS59, et montre des résultats compétitifs, fournissant une segmentation précise tout en maintenant la cohérence de l'étiquette à long terme. Il est également très rapide, capable de traiter des séquences de plusieurs centaines d'images en un seul passage à une moyenne de 210 fps. L'implémentation logicielle appelée LT-Segmentation est disponible sur github (https://github.com/Etienne-Meunier-Inria/LT-Segmentation).

1.2. Motion Segmentation

As we mentioned at the beginning of this section, an important distinction is the difference between learning-based and optimisation-based approaches. We call "learning based" methods that train neural networks to perform the task and then perform inference only at test time. The fact that learning based methods do not require iterative computation at test time makes them really fast and efficient for practical applications, but also removes the need for initialisation, which is often a hindrance for classical approaches. The ability to implicitly learn shape priors or appearance conditioning from the data is another advantage of learning-based methods. Empirical results show that, even without a loss term that explicitly introduces this regularisation, a neural network performing segmentation is more likely to produce compact and regular shaped output at inference time, if it has been trained on this type of target.On the other hand, learning-based approaches have drawbacks, the most commonly cited is that it is long and computationally expensive to train. Another issue is that the results at inference can be hard to explain and quantifying the uncertainty of the network prediction is still an open question. More specifically to our application, it can also be more difficult to integrate priors on segmentation, as one needs to find a loss function that is differentiable and works with the evolution of the segmentation masks

https://github.com/Etienne-Meunier/EM-Flow-Segmentation

https://davischallenge.org/index.html

https://paperswithcode.com/dataset/segtrack-v2-1

https://github.com/Etienne-Meunier-Inria/GeneraUnet

https://github.com/princeton-vl/RAFT

https://github.com/Etienne-Meunier-Inria/ST-Space-Time-Flow-Segmentation

https://davischallenge.org/index.html

https://paperswithcode.com/dataset/segtrack-v2-1

https://davischallenge.org/index.html

Crowd Animation Open Software : https://project.inria.fr/crowdscience/download/

http://www.svcl.ucsd.edu/projects/anomaly/dataset.html

and d = (0, 0).

(a) Input image (b) Ours -SoFlow (c) SCV (d) DIP

ACKNOWLEDGEMENTS

Dans un premier temps, je tiens à remercier mon directeur de thèse, Patrick Bouthemy, pour son accompagnement tout au long de ce travail. C'est grâce à sa bienveillance et à nos échanges que j'ai pu entreprendre les travaux présentés dans ce manuscrit.

Je tiens également à exprimer ma gratitude envers mes rapporteurs, Nicolas Thome et Patrick Perez, pour leur relecture attentive du manuscrit et leurs retours pertinents qui m'ont permis d'améliorer ce travail. Mes remerciements vont également à Elisa Fromont, Daniel Cremers, Ronan Fablet et Renaud Fraisse pour leur participation à mon jury de thèse et leurs retours sur mon travail.

//github.com/Etienne-Meunier

Experimental results

We report sample results of our SoFlow method (without the sparsity term) on satellite images, and compare them with two supervised deep learning methods, DIP [START_REF] Zheng | DIP: Deep inverse patchmatch for high-resolution optical flow[END_REF] and SCV [START_REF] Jiang | Learning optical flow from a few matches[END_REF]. As a matter of fact, these images, provided by Airbus Defence&Space, are aerial images acquired in such a way that they simulate the resolution of the (future) satellite images. In addition, they are stabilized.

Appendix

We have also added several appendices. The three first ones contain supplementary material respectively related to Chapters 2, 3 and 4. The last one corresponds to an additional work on the estimation of optical flow for small moving objects in large-scale images along with some preliminary results. 4.1 -Ablation study for three main components of our method LT-MS (K = 4) on the three datasets DAVIS2016, FBMS59 and SegTrackV2. Only one model component is modified at a time. The performance scores are given by the Jaccard index J ; the higher J , the better. We report ablation results with two input-flow sequence length (or cut size), respectively, by dividing the video into pieces of ten successive frames or by considering 120 successive frames (in practice, the whole video for the DAVIS2016 dataset).

-loss function without the consistency term L c , -just the convnet without the transformer decoder.

All the ablation experiments were run on the three datasets, DAVIS2016, FBMS59, Seg-TrackV2. Results are collected in Table 7.12. In addition, we performed them for two input sequence configurations, respectively input sequences of ten flows, and input sequence of 120 flows (in practice, the whole video for the DAVIS2016 dataset).

We can observe that the three ablations have almost the same impact on the performance. The three corresponding model components, i.e., the spline-based motion model, the temporal-consistency loss term, and the transformer decoder are thus all beneficial in similar proportions. They are able to handle the temporal dimension of the problem and the temporal motion evolution along the sequence in a compelling way. In addition, Fig. 4.3 contains the performance scores for the three ablations when the length of the input flow sequence varies from 10 to 120. Overall, they exhibit comparable behaviour at a certain distance from the full model.

Admittedly, the contributions of these three components are more significant for the FBMS59 and the SegTrackV2 datasets. However, the dynamic content of the majority of the DAVIS2016 videos, and then, the overall performance score, cannot allow us to fully appreciate the contributions of these three model components. Yet, they can be acknowledged by visualizing results obtained on some videos of DAVIS2016, as shown in Visual results reported in Fig. 4.4 clearly demonstrate that the addition of the temporal consistency loss term L c allows us to get far more consistent segments over time, whether

Second step: transforming attribution maps into salient motion segments

Depending on the quality of the attribution maps and possibly the size of the salient moving objects, this second step may imply a light or a strong use of an additional representative flow information, but it is still driven by the attribution values. As described below, this additional flow information is straightforwardly derived from the input optical flow, and it corresponds to respectively a weighted mean flow vector or a hierarchy of affine motion models.

We proceed as follows for the light version of this second interpretation step. Using the square of the attribution values as weights, we compute over the whole frame the weighted average w of the flow vectors, which is likely to adequately represent the salient motion in the frame. In case of several types of salient motion, several modes should be sought for. Then, for every point p = (x, y) and its flow vector w p = (u p , v p ), p belongs to a candidate region if α 1 < (w p . w/|| w|| 2 2 ) -1 < α 2 . In practice, we take α 1 = -0.01 and α 2 = 4. We set asymmetric threshold values because flow orientation and magnitude are both involved making the problem a bit tangled.

The other more elaborate version of the second interpretation step relies on a hierarchical parametric motion segmentation. We take the optical flow as input, and we segment it by iteratively estimating affine motion models fitting the optical flow field, using a robust regression involving the Huber function H. More precisely, we minimize the function:

where Ω k is the image part intervening at iteration k, and the a k i 's the six parameters of the affine motion model computed at iteration k. At each iteration k, inliers are selected to form a layer. Inliers are pixels satisfying the constraint 1 2 where (u k p , v k p ) are the coordinates of w k p , the flow vector given at p by the affine motion model estimated at iteration k. γ k u and γ k v are the standard deviations over the differences of the u and v components at iteration k. Those factors were introduced to avoid being affected by diverse motion magnitudes. In addition, it allows us to set the threshold β once and for all, and we take β = 1.6. Outliers are kept for the next iterations. We continue this procedure until only a small number of outliers remain (in practice 50 points), and we group them together to form the last layer. Following this iterative decomposition, each vector in the initial optical flow field belongs to one unique layer. Finally, we split each layer into connected components to form the candidate regions.

GENERAL CONCLUSION

Contributions

In this thesis we investigated two axes for motion analysis. The first one is concerned with unsupervised motion segmentation, where we have designed and trained a network to decompose an input flow field into coherent segments. The second one deals with the localization of salient motions, where we have developed techniques to identify regions of the input flow field that are critical for a downstream classification task.

Unsupervised motion segmentation

In this part, we have developed a series of works to partition an input optical flow field into coherent motion segments.

In a first work, we introduced a framework to train a neural network to partition an optical flow field into coherent segments without relying on annotated data. Our loss function is based on the EM algorithm and the use of parametric motion models to represent the flow within each segment. We succeeded in training a network capable of segmenting unseen optical flow fields into a given number of coherent parts. Our method gives good results on several video object segmentation datasets (DAVIS2016, FBMS59, SegTrackV2, MoCA), while being fast at test time.

In a second work, we extended this framework to handle a volume of three optical flow fields segmented with coherent labels. To enforce temporal consistency, we coupled the loss from our first work on motion segmentation with a temporal consistency loss that enforces that the labels are consistent across the output segmentation volume. We showed that this work improves the robustness of the segmentation to noise in the input optical flow field, and that with a simple post-processing step we can produce a segmentation with consistent labels over a long sequence.

In a third work, we deepened our idea of segmenting long optical flow sequences by extending our previous method and defining a ent-to-end model to segment a full sequence. First, we introduce the use of splines to represent the temporal evolution of the parametric Chapter 7

APPENDIX

Appendix A: EM-driven Motion Segmentation

Variational inference

In this subsection, we describe an alternative mathematical approach to lead to our loss function, based on the variational inference framework [START_REF] Kingma | An introduction to variational autoencoders[END_REF].

We consider a volume of optical flow fields f ∈ R 2×W ×H , the spatial grid Ω ∈ R W ×H . We denote f (i) ∈ R 2 the flow vector at site i ∈ Ω. We assume that we can decompose the flow field into a set of K segments, each one exhibiting a coherent motion. Flow vectors within a given segment k are represented by a smooth parametric motion model parametrized by ϑ k . Variable z i conveys the motion segmentation: z k i = 1 if site i belongs to segment k, z k i = 0 otherwise. z and ϑ are latent variables, and f is the observed data. Following reasoning developed in [START_REF] Kingma | An introduction to variational autoencoders[END_REF], we introduce an approximate distribution over segmentation and motion model parameters:

where q(ϑ) ≜ δ(θ k ), δ being the Dirac distribution, and q(z k i |f ) = g ϕ (f ) k i . g ϕ is our network model taking as input the optical flow volume and returning a probabilistic motion segmentation volume.

We can also write the data-likelihood over a dataset D of optical flow volumes f as: Code available at : https://github.com/Etienne-Meunier/EM-Flow-Segmentation

Failure cases on SegTrackV2

We observed that the optical flow is not well estimated by the RAFT method in several frames of SegTrackV2 dataset, causing our algorithm to fail on those frames. These failure cases are reported in Fig. 7.3.

Detailed results per sequences of the datasets

Hereafter, we report detailed results through tables collecting the evaluation scores obtained by our EM method for every sequence of the four datasets, DAVIS2016, Seg-TrackV2, FBMS59 and MoCA. directly oriented to the VOS evaluation, when our network is trained for multiple motion segmentation with K masks.

Detailed results per videos of the datasets

Hereafter, we report detailed results through tables collecting the evaluation scores obtained by our ST-MS method for every video of the four datasets, DAVIS2016, Seg-TrackV2, FBMS59, and DAVIS2017-motion. Let us recall that the official evaluation algorithm is not the same for DAVIS2016 and DAVIS2017. The evaluation is done in one go on the whole video for DAVIS2017, while it is achieved frame by frame of the video for DAVIS2016.

DAVIS2016

Appendix C: Long-term Motion Segmentation

Additional ablation study

Training without data augmentation

We also conducted an ablation study regarding data augmentation. Results are collected in Table 7.12. A similar conclusion can be established as the one drawn for the ablation study applied to the three main components of our LT-MS method in Chapter 4, yet with a larger margin in this case. 7.12 -Ablation study for the data augmentation component of our method LT-MS (K = 4) on the three datasets DAVIS2016, FBMS59 and SegTrackV2. The performance scores are given by the Jaccard index J ; the higher J , the better. We report ablation results with two input-flow sequence lengths (or cut size), respectively, by dividing the video into pieces of ten successive frames or by considering 120 successive frames (in practice, the whole video for the DAVIS2016 dataset).

Dataset

Repeatability

With the introduction of the transformer decoder, we experimentally found that the convergence of the network depends on the weights initialization, and that the same network and loss configuration can yield different results at test time. In Fig. 7.5, we show that our unsupervised loss on a held-out validation set is a good indicator of the network performance at test time. This is a critical point for model and hyperparameter selection, since we do not have access to the ground truth at training time (with our fully unsupervised scenario), and thus we cannot evaluate model performance. Fig. 7.6 plots the model performance after training as a function of the initialization budget.

In our evaluation experiments, we account for this randomness by training five models with the same set of seeds for all ablations and by reporting the score of the model with the lowest validation loss for each model. masks. However, the evaluation with K = 2 masks gives poor results compared to the model fine-tuned on two masks. A possible improvement could be to start with a LT-MS-SA model trained with K = 3, as the step in terms of number of masks would be smaller. We can also observe these performances in the qualitative results in Fig. 7.8, where the segmentation is quite good from 3 to 6 masks and then drops when K = 2.

Dataset

Detailed results per videos of the datasets

Hereafter, we report detailed results through tables collecting the evaluation scores obtained by our LT-MS-K4 method for every video of the three datasets, DAVIS2016, Seg-TrackV2 and FBMS59, and by our LT-MS-K3 model for every video of the DAVIS2017motion dataset. Let us recall that the official evaluation algorithm is not the same for DAVIS2016 and DAVIS2017-motion. The evaluation is done in one go on the whole video for DAVIS2017-motion and is multi-segment, while it is binary and performed frame by frame of the video for DAVIS2016.

Introduction and motivation

Appendix D describes joint work done with Sarra Khairi during her internship at Inria Rennes (March-August2023), which I co-supervised regarding methodology, programming and report writing. The internship took place in the collaboration with Airbus Defence&Space supported by the LiChIE contract (Bpifrance). The content of this section is mainly derived from her internship report that describes the work done in detail. An extended summary of the contributions and a subset of results will be given in this Appendix section.

Motion analysis in satellite image sequences leads to focus on small moving objects in large-scale images. Before attempting to segment these small moving objects, we have to make sure that we can get optical flow fields reliable and accurate enough for the segmentation task. Yet, it turned out that the RAFT method was not efficient in this specific context, and we had to look for an alternative. Anyway, developing an optical flow (OF) method for satellite images has great potential. A similar configuration can be also encountered in live cell microscopy. A dedicated optical flow method can facilitate the interpretation of these image modalities, and helps one to build algorithms for dynamic content analysis or event detection for these applications. Several classical optical flow methods have been applied to these modalities. We could expect an improvement of the measured flow field with the introduction of deep learning, although to the best of our knowledge, no methods have been specifically developed with these modalities in mind. We will first show that state-of-the-art deep learning methods for flow estimation fail on these modalities, and then, present a new deep learning estimation method that achieves good results on satellite imagery.

RAFT [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF] is one of the most popular method for optical flow computation. It has proven to deliver competitive results on many datasets. However, this method comes with several drawbacks, especially when transferred to specific applications (e.g., satellite or microscopic videos). First, this method depends on the data it was trained on. Therefore, the accuracy of flow estimations may decrease on newly seen videos whose distribution differs from the training data. Furthermore, it constructs an all-pairs 4D correlation volume, which leads to memory problems when predicting large-scale high-resolution optical flows. Lastly, the downsampling caused by the feature extraction module misleads the flow estimation, especially for thin and small moving structures. These limitations arise specifically when estimating optical flow on the aerial sequences provided by Airbus. These stabilized aerial image sequences mimic (future) satellite images. Indeed, making predictions on these high-resolution images using RAFT requires large memory, and the results tend to miss small moving vehicles. Also, it is not possible to finetune existing supervised methods like RAFT on aerial or satellite sequences due to the lack of video annotation and relevant synthetic data.

To illustrate our remarks, we show results of optical flow field estimation on satellite images using two existing methods in Fig. 7.9. Brox method [START_REF] Brox | High Accuracy Optical Flow Estimation Based on a Theory for Warping[END_REF] is a variational method for flow estimation that appear to work well on satellite images. RAFT [START_REF] Teed | RAFT: Recurrent all-pairs field transforms for optical flow[END_REF] is a commonly used deep learning method for flow estimation on classical images. As we can see, it works poorly on satellite images missing a large part of the small moving objects or merging other ones in blobs. a