Dans ce contexte, l'animation de personnages a une grande importance pour la qualité globale des oeuvres d'art. La façon dont les humains bougent est très variée et transmet beaucoup d'informations sur l'activité réalisée, l'état d'esprit du personnage (intentions, humeurs ou émotions), son identité (caractéristiques morphologiques et biologiques), etc. En outre, la deuxième loi de Newton fait intrinsèquement du mouvement humain un processus dynamique, tandis que notre compréhension de la biomécanique est loin d'être complète. Pour ces raisons, l'animation de personnages humains reste un véritable défi, même si d'énormes progrès ont été réalisés récemment. Il faut donc faire des compromis pour équilibrer les coûts de production, le réalisme, la quantité de travail manuel et le niveau d'expertise des animateurs, en fonction de la qualité attendue. Par exemple, l'industrie du jeu s'appuie déjà sur l'animation de personnages assistée par ordinateur, mais les studios d'animation ont encore recours à une grande quantité de travail manuel, par exemple dans la post-production de films à gros budget.

Le domaine de recherche de l'animation de personnages est actif depuis des décennies pour atténuer ces compromis et rendre l'animation plus accessible, en commençant avec des travaux pionniers tels que l'édition et la déformation de mouvements (p.ex. Witkin et al. [START_REF] Witkin | Motion Warping[END_REF]), la squeletto-morphose de mouvements sur de nouveaux personnages (par exemple Gleicher [START_REF] Gleicher | Retargetting Motion to New Characters[END_REF]), le contrôle de personnages à l'aide de graphes de mouvements (p.ex. Kovar et al., Min et al. [START_REF] Kovar | Motion Graphs[END_REF][START_REF] Min | Motion graphs++: a compact generative model for semantic motion analysis and synthesis[END_REF]), etc. l'émergence de l'apprentissage profond Au cours de la dernière décennie, l'apprentissage profond s'est imposé comme un moyen puissant d'améliorer les performances et les capacités de l'animation de personnages. Il a démontré une capacité sans précédent à traiter des tâches complexes dans une grande variété de domaines non limités à l'animation, tels que la vision par ordinateur, le traitement du langage naturel et bien d'autres encore. Les humains sont surpassés par les algorithmes d'intelligence artificielle dans un nombre croissant de tâches telles que la classification d'images [START_REF] He | Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification[END_REF] ou le jeu de Go [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF]. Cela s'explique notamment par le fait que les réseaux de neurones profonds sont de puissants approximateurs de fonctions, capables d'apprendre des modèles sophistiqués dans des phénomènes complexes du monde réel à partir de données, qui sont de plus en plus facilement capturées, stockées et traitées en amont. En effet, bon nombre des grandes avancées récentes telles que les réseaux antagonistes génératifs ou les transformeurs se sont accompagnées d'une augmentation du volume de données nécessaire pour entraîner ces modèles de pointe. Pourtant, une fois la phase d'entraînement terminée, les données d'entraînement sont rejetées, laissant des modèles compacts capables de répondre aux exigences de performance des applications en temps réel ou de passer à l'échelle des systèmes embarqués.

Dans le domaine de l'animation, les approches basées sur l'apprentissage profond tentent de gérer la complexité des mouvements humains et offrent des perspectives prometteuses pour des techniques d'animation moins coûteuses et plus rapides, avec une fidélité et des capacités créatives de plus en plus grandes. Cependant, ce type d'approches peine à atteindre la qualité de travail obtenue par des animateurs qualifiés et n'est pas massivement déployé dans les studios d'animation qui dépendent encore largement d'animations réalisées à la main. Pour imiter les récentes percées de l'apprentissage profond dans d'autres domaines, l'animation profonde pourrait avoir besoin de davantage de données de mouvement de haute qualité pour alimenter les récents modèles gourmands en données comme les transformeurs. En effet, les grandes bases de données publiques unifiées de données de mouvement sont beaucoup plus rares que dans d'autres domaines tels que les bases de données d'images en vision par ordinateur, et moins faciles à agréger. Souvent, les systèmes de capture de mouvements de haute qualité sont coûteux, nécessitent un environnement contrôlé empêchant les captures dans la nature, et diffèrent quant aux articulations ou aux marqueurs corporels capturés. La recherche en apprentissage profond pourrait avoir un rôle à jouer, soit en améliorant les moyens d'unifier les bases de données existantes sur les mouvements humains, soit en facilitant l'acquisition de données de haute qualité sur les mouvements, par exemple grâce à l'estimation de pose. Pour ces raisons, l'apprentissage profond est aujourd'hui au coeur de la recherche en animation de personnages, ainsi que de cette thèse, de la collecte de grandes bases de données de mouvements jusqu'à la conception et l'apprentissage de modèles puissants.

animation du squelette de personnages virtuels L'animation traditionnelle de personnages virtuels 3D est généralement réalisée en animant un squelette auquel on attache un maillage qui représente la surface du corps, ce qui permet d'offrir un bon compromis entre qualité et complexité. La mise en correspondance du squelette et du maillage offre des moyens pratiques pour animer le personnage en 3D comme le font les fils pour une marionnette, et le maillage qui est coloré et texturé permet de contrôler l'identité visuelle du personnage virtuel animés. L'animation du squelette sous-jacent, qui est le sujet d'intérêt de cette thèse, permet de donner vie au personnage virtuel. Ce sujet peut être divisé en trois catégories principales :

-la synthèse de mouvements, pour générer de nouvelles séquences de mouvement avec les caractéristiques souhaitées.

-le contrôle de personnages, pour piloter et actionner en continu des personnages virtuels à partir de contrôles spécifiques (généralement pilotées par l'utilisateur).

-l'édition de mouvements, pour combiner, transférer, enrichir, améliorer, nettoyer, etc. des séquences de mouvements existantes.

La synthèse de mouvements s'efforce de créer de nouvelles séquences de mouvement plausibles sur le plan perceptif, généralement hors ligne, à partir de paramètres bas niveau tels que des variables latentes ou des paramètres de plus haut niveau tels que le style souhaité, les variations affectives ou un ensemble de poses clés éparses à interpoler ou à extrapoler. En revanche, le contrôle des personnages se réfère par définition au contrôle et à l'actionnement en ligne de personnages virtuels répondant aux flux d'entrée de l'utilisateur. Parmi les cas d'utilisation typiques du contrôle des personnages, on peut citer les jeux vidéo, où des contrôles haut niveau de l'utilisateur pilotent un personnage virtuel censé se déplacer de manière fluide et naturelle dans l'environnement du jeu et s'y adapter, tout en effectuant différentes actions telles que la saisie et la manipulation d'objets. Enfin, l'édition de mouvements se concentre sur le traitement de séquences de mouvements existantes plutôt que sur la production de nouvelles séquences de mouvements via le contrôle ou la synthèse. Il regroupe différentes tâches, y compris la manipulation de caractéristiques de haut niveau, telles que le transfert du style d'une séquence de mouvements à une autre, ou le transfert des mouvements d'un personnage à un autre avec une morphologie ou une structure différente. L'édition de mouvements comprend également toutes sortes d'approches visant à améliorer des séquences de mouvements existantes, par exemple en supprimant des artefacts tels que du bruit.

Il y a une dizaine d'années, des méthodes reposant sur l'apprentissage profond sont apparues et n'ont cessé de se développer dès lors dans tous ces domaines. Ces dernières années, ces approches ont ponctuellement commencé à atteindre la qualité des techniques traditionnelles d'animation. Les prochaines étapes consisteront alors à surpasser complètement les flux de travail de l'animation traditionnelles, en améliorant tous les composants des systèmes d'animation pour synthétiser, contrôler et traiter le mouvement des personnages virtuels, dans le but d'atteindre, et éventuellement de surpasser, la qualité du travail manuel des animateurs qualifiés, ce qui serait particulièrement précieux sur des tâches manuelles peu créatives et fastidieuses (par exemple, les personnages d'arrière-plan).

objectifs Les objectifs de cette thèse sont doubles : premièrement, identifier et surmonter les obstacles actuels dans l'animation de personnages humains squelettiques basée sur l'apprentissage profond, afin d'éviter d'atteindre la qualité de travail des animateurs qualifiés. Deuxièmement, explorer les méthodes de pointe prometteuses en apprentissage profond pour surmonter ces obstacles et apporter de nouveaux outils à la communauté de l'animation du squelettique de personnages virtuels afin d'améliorer le traitement des séquences de mouvements.

En particulier, nous avons d'abord exploré le goulot d'étranglement que constitue la disponibilité limitée des données de mouvement sous deux angles différents : tout d'abord, l'acquisition de données de mouvement pour augmenter le volume et la diversité des données de mouvement de haute qualité dans la nature. À cette fin, nous avons envisagé l'utilisation de modèles génératifs profonds pour améliorer et enrichir les séquences de mouvements estimées à partir de vidéos avec des connaissances a priori. Deuxièmement, l'unification des bases de données de mouvement existantes, par exemple pour tirer parti à la fois des aspects spécifiques de petites bases de données (par exemple les étiquettes de style) et de la diversité et du volume des bases de données plus importantes. Afin de faciliter l'utilisation conjointe de données de mouvement provenant de différentes sources (qui est difficle principalement en raison des différences structurelles et morphologiques du squelette des personnages), nous avons également exploré une nouvelle architecture polyvalente permettant de séparer des caractéristiques abstraites du mouvement et de caractéristiques liées à la structure et et à la morphologie du squelette. Enfin, nous nous sommes également penchés sur l'amélioration des séquences de mouvement existantes, en particulier pour résoudre les artéfacts de footskate, un problème de longue date en animation. En bref, le footskate est une famille d'artefacts présents dans les séquences de mouvement lorsque les mouvements des pieds ne sont pas cohérents par rapport au sol et apparaissent facilement dans les sorties des réseaux de neurones artificiels. Ces artéfacts nuisent fortement à la qualité d'un certain nombre d'approches dans tous les domaines de l'animation du squelette de personnages virtuels et empêchent de s'appuyer sur de telles méthodes imparfaites. structure Toutz d'abord, nous faisons un aperçu approfondi pertinante pour cette thèse dans le chapitre 2. En nous concentrant sur l'animation du squelette de personnages virtuels basée sur l'apprentissage profond, nous commençons par une vue d'ensemble des problèmes bas niveau rencontrés lors du traitement des données de mouvement humain avec des réseaux de neurones profonds, incluant les représentations du mouvement et les bases de données, ainsi que l'apprentissage de caractéristiques spatiales et temporelles du mouvement. Ensuite, nous couvrons les méthodes de pointe en synthèse de mouvement, en contrôle de personnages et en édition de mouvement dans des sections distinctes. D'autres travaux connexes spécifiques, comme les modèles profonds de pointe importants pour certaines parties de cette thèse, sont traités dans chaque chapitre correspondant.

Ensuite, nous présentons dans le chapitre 3 une nouvelle approche visant à améliorer la qualité et le niveau de détail de l'estimation de pose humaine, qui s'efforce de capturer le corps humain avec un équipement limité (souvent une seule caméra monoculaire). Étant donné que de nombreuses approches de l'estimation de pose humaine en 3D décomposent le problème en estimation 2D suivie puis essayent d'inférer la profondeur, nous proposons une étape intermédiaire supplémentaire pour compléter et suréchantillonner les articulations des séquences de pose estimées en 2D, dans l'espoir d'améliorer, d'enrichir et de désambiguïser les séquences de poses en 2D pour faciliter l'étape d'inférence de la profondeur. À cette fin, nous apprenons une représentation profonde des séquences de poses en 2D, que nous exploitons ensuite en y projetant des séquences de pose incomplètes avec un nombre inférieur d'articula-tions, ce qui nous permet de reconstruire finalement les séquences de Finalement, dans le chapitre 5, nous nous attaquons au nettoyage du footskate, un problème de longue date en animation. Les artefacts de footskate consistent typiquement en des glissements de pieds par rapport au sol qui sont facilement introduits lors de toute étape de traitement de séquences de mouvements. Ces artéfacts nuisent fortement au réalisme perçu des animations [START_REF] Pražák | Perceptual Evaluation of Footskate Cleanup[END_REF] et requièrent la connaissances des phases de contacts des pieds pour être nettoyés. Comme les approches traditionnelles reposent généralement sur une annotation manuelle des contacts de pieds, notre première contribution ici est une base de données unique en son genre de mouvements synchronisées avec des données sur la pression des pieds exercées sur le sol, servant de vérité terrain physique pour les contacts de pieds. Ensuite, nous nous appuyons sur cette base de données pour entraîner un réseau de neurones profond à détecter les contacts de pieds, une étape nécessaire pour la résolution du footskate. La principale nouveauté est l'utilisation des forces de réaction du sol -qui sont estimées à partir des séquences de mouvements -comme représentation alternative aux contacts binaires pour mieux capturer les interactions physiques entre les pieds et le sol, à partir desquelles les contacts entre les pieds sont calculés. Nous démontrons que notre approche est plus performante que les solutions heuristiques traditionnellement utilisées. Enfin, nous proposons une approche pour supprimer les artefacts liés au footskate. Conformément aux approches récentes, le footskate est nettoyé en imposant des contraintes sur les pieds pendant les phases de contact (détectées au préalable), à l'aide d'un algorithme de cinématique inverse qui optimise itérativement les angles des articulations pour satisfaire les contraintes. Nous nous appuyons en outre sur notre modèle profond pour préserver les forces de réaction du sol pendant l'optimisation et maintenir la dynamique du mouvement globalement cohérente. perspectives Dans cette thèse, nous avons exploré des approches dans le but général d'améliorer les systèmes basés sur l'apprentissage profond pour l'animation du squelette de personnages virtuels. Dans ce qui suit, nous présentons des perspectives de recherche possibles pour la suite de notre travail. Tout d'abord, nous proposons des perspectives à court terme, des améliorations possibles et des expériences qui découlent naturellement du travail présenté dans cette thèse. Ensuite, nous suggérons un ensemble de directions possibles pour étendre notre travail en s'inspirant de la littérature existante dans le domaine de l'animation de personnages et au-delà. Enfin, nous ouvrons la discussion sur des perspectives à long terme qui nécessitent de résoudre plusieurs défis et de repenser une partie des paradigmes actuels de l'animation de personnages.

Perspectives à court terme

Dans les chapitres 3 et 4, nous avons présenté deux approches reposant sur des représentations profondes du mouvement. Dans la seconde, nous avons proposé une représentation abstraite du mouvement indépendante de la topologie et de la morphologie du squelette. Cependant, comme indiqué dans la section 4.5, notre modèle présente certaines limites en ce qui concerne les variations morphologiques, y compris une diminution des performances sur les morphologies non humaines.

Une première perspective serait d'étudier si le modèle proposé est capable ou non d'encoder avec précision les morphologies humaines et humanoïdes dans une seule représentation unifiée du mouvement. En d'autres termes, est-il possible d'unifier le mouvement de personnages humanoïdes (avec deux jambes, deux bras et une tête, mais des proportions corporelles éventuellement exagérées) dans une représentation profonde apprise selon notre approche ? En outre, les personnages non humanoïdes tels que les quadrupèdes pourraient également être étudiés dans le cadre de travaux futurs. Une approche simple et efficace consisterait à diversifier l'ensemble des données d'apprentissage au-delà des séquences de mouvements humains. Cependant, d'autres changements plus complexes pourraient être nécessaires pour traiter les personnages humanoïdes ou d'autres types de personnages, et sont présentés dans les perspectives à moyen terme ci-après.

Une autre limite de notre travail est que des artefacts de footskate sont susceptibles d'être introduits dans certains cas. À l'heure actuelle, ce problème concerne la plupart, sinon la totalité, des modèles de mouvement profond utilisés pour produire ou traiter des données de mouvement. Une approche possible pour améliorer la situation serait d'introduire des connaissances sur les interactions entre les pieds et le sol. Par exemple, cela pourrait être possible en introduisant une certaine supervision en ce qui concerne le contact des pieds ou les forces de réaction du sol au cours de l'apprentissage. Cela peut être directement mis en oeuvre en utilisant la base de données de mouvement proposée dans le chapitre 5 dans laquelle des données de mouvement sont synchronisées avec des données de semelles de pression. Il existe plusieurs façons d'ajouter de la supervision et qui pourraient être combinées. Par exemple : les contacts des pieds avec le sol pourraient être utilisés pour mesurer et minimiser le footskate ; les forces de réaction du sol ou les contacts pourraient être estimés à partir de la représentation latente de séquences de mouvement ; une fonction de perte sur la cohérence des forces de réaction du sol ou des contacts pourrait être appliquée sur les forces ou contacts estimés à partir de séquences de mouvements reconstruites ou retargetées. En outre, la représentation unifiée du mouvement proposée dans le chapitre 4 et le modèle de détection des contacts avec les pieds proposé dans le chapitre 5 pourraient être directement combinés pour nettoyer automatiquement les artefacts de footskate. Le premier modèle transférerait les séquences de mouvement vers la topologie de squelette du second pour ensuite obtenir les contacts des pieds avec le sol nécessaires au nettoyage du footskate.

Enfin, la quantité d'artefacts de footskate introduits est rarement évaluée. Les raisons à cela sont la rareté des données de mouvement annotées avec des contacts de pieds précis, et la difficulté d'obtenir des contacts de pieds de vérité au sol précis. La base de données et l'approche de détection des contacts avec les pieds proposées dans le chapitre 5 pourraient être exploitées pour définir une procédure d'évaluation standard du footskate.

Perspectives à moyen terme

Un défi croissant dans l'animation de personnages est de proposer des méthodes suffisamment polyvalentes pour être utilisables sur des personnages présentant de grandes variations topologiques et morphologiques, et éventuellement généralisables au-delà des personnages humanoïdes. L'extension de certaines des représentations profondes du mouvement proposées dans cette thèse pour gérer de telles variations pourrait donc nécessiter de repenser le mécanisme de conditionnement que nous avons proposé. En particulier, nous avons actuellement proposé d'utiliser des poses neutres comme modèles de squelette, représentées par la position des articulations, ce qui peut être limitant pour capturer pleinement les caractéristiques morphologiques et topologiques. Bien que ce problème puisse être résolu par une conception plus réfléchie, nous pensons qu'une solution élégante consisterait à apprendre en plus une représentation profonde des topologies et morphologies du squelette ainsi qu'une représentation profonde du mouvement. Une telle approche aurait l'avantage d'éviter la prédéfinition manuelle des modèles de squelettes tout en atténuant les limites de notre approche actuelle. Pour élargir encore le champ d'application de notre approche, il faudrait peut-être aussi envisager de personnaliser les modèles de squelette (prédéfinis ou appris) pour chaque sujet, c'est-à-dire de régler plus finement notre mécanisme de conditionnement pour mieux saisir les variations morphologiques fines.

Au-delà des modèles de squelette, de nouveaux modèles d'apprentissage profond peuvent être exploités pour améliorer l'apprentissage. Par exemple, les modèles de diffusion [START_REF] Sohl-Dickstein | Deep Unsupervised Learning Using Nonequilibrium Thermodynamics[END_REF] sont très attrayants pour construire des représentations profondes utiles pour de multiples tâches. En effet, ces modèles surpassent les réseaux antagonistes génératifs sur la synthèse d'images, et ont également été prometteurs dans une variété de domaines [START_REF] Yang | Diffusion Models: A Comprehensive Survey of Methods and Applications[END_REF]. Enfin, leur nature progressive pourrait être utile à de nombreuses applications qui peuvent être formulées comme des transitions progressives. Il s'agit notamment de la plupart des tâches de complétion, telles que la complétion de mouvement, l'in-betweening ou le suréchantillonnage des articulations, mais aussi des tâches de nettoyage telles que le débruitage ou le nettoyage du footskate. La nature progressive des modèles de diffusion pourrait même étendre la portée des problèmes actuels. Par exemple, les modèles de diffusion de mouvement à mouvement (par analogie avec les modèles de diffusion d'image à image [START_REF] Saharia | Palette: Image-to-Image Diffusion Models[END_REF]) pourraient apprendre à retargeter progressivement une séquence de mouvement d'un personnage à un autre, permettant notamment la morphose de personnages dans les séquences de mouvement, c'est-à-dire le passage progressif d'un personnage à un autre au fil du temps.

Il existe également de multiples perspectives en matière de nettoyage et de prévention des artefacts dans les séquences de mouvement de sortie. Dans le contexte du nettoyage du footskate en particulier, l'une des limitations provient de la représentation binaire des contacts entre les pieds, qui sont ensuite généralement utilisés pour appliquer les contraintes liées aux pieds à l'aide d'algorithmes de cinématique inverse. En effet, les interactions entre les pieds et le sol ne sont pas triviales et les modéliser comme étant en contact ou hors contact est trop simplifié et conduit à ajuster manuellement les algorithmes de cinématique inverse pour des situations spécifiques afin d'éviter de remplacer le footskate par d'autres types d'artefacts. Pour résoudre ce problème, il est possible d'utiliser une représentation plus fine des interactions entre les pieds et le sol, telle que les forces de réaction du sol, et de concevoir un algorithme de cinématique inverse ciblant spécifiquement le nettoyage du footskate, par exemple en imposant des profils cibles des forces de réaction du sol au lieu de phases binaires de contact. C'est un aspect que nous n'avons que très peu exploré dans cette thèse, mais qui, selon nous, présente un potentiel prometteur pour de nouvelles approches. Une autre direction ambitieuse pour prévenir les artefacts de footskate serait de modéliser explicitement les interactions entre les pieds et le sol en même temps que le mouvement, par exemple en construisant un plongement commun des forces de réaction du sol et des séquences de mouvement, de manière similaire aux plongements sémantiques visuels en vision par ordinateur.

Perspectives à long terme

À long terme, nous pensons que l'animation de personnages pourrait devoir changer de paradigme pour rendre la synthèse et l'édition d'animation plus accessibles.

Cela pourrait inclure le remplacement des interfaces homme-machine actuelles dans les systèmes d'animation par des commandes sémantiques plus intuitives pour les concepteurs et les artistes. Dans cette direction, les plongements sémantiques sont prometteurs. En effet, ils ont récemment démontré des capacités impressionnantes à l'intersection du traitement du langage naturel et de la vision par ordinateur, comme le nouveau modèle de génération d'images d'OpenAI DALL•E 2 [START_REF] Ramesh | Hierarchical Text-Conditional Image Generation with CLIP Latents[END_REF] qui s'appuie sur une version de GPT-3 [START_REF] Tom | Language Models are Few-Shot Learners[END_REF] modifiée pour la synthèse d'images. En particulier, les plongements sémantiques basés sur le langage naturel semblent être très efficaces pour produire des représentations puissantes et capturer les informations nécessaires pour les applications impliquant des interactions homme-machine. Dans le domaine de l'animation de personnages, de nombreuses applications incluent des interactions homme-machine à un moment ou à un autre, comme les systèmes de synthèse et d'édition de mouvements pilotés par des artistes et des animateurs, ou les personnages de jeux-vidéo contrôlés par les joueurs. Par exemple, par analogie avec les capacités des modèles actuels de synthèse d'images, nous pourrions imaginer produire rapidement des ébauches de séquences de mouvement à partir de descriptions textuelles, telles que "un personnage de grande taille qui court pendant 4 secondes puis accélère en sautant au-dessus d'un obstacle". Cependant, la construction d'un plongement sémantique pour l'animation de personnages basée sur le langage naturel nécessiterait de résoudre plusieurs problèmes difficiles. Ces défis comprennent la nécessité de gros volume de données pour l'apprentissage des modèles récents, tels que GPT-3. Dans le cas des images associées à des descripteurs textuels, des milliards d'exemples sont disponibles sur internet. En revanche, les données de mouvement sont actuellement beaucoup plus rares que les images. En outre, les séquences de mouvement ne sont pas souvent annotées avec des descripteurs textuels dans les bases de données existantes, ce qui est nécessaire pour construire un tel plongements sémantiques du langage naturel et du mouvement de personnages. Un autre défi important est que des modèles de langage naturel dédiés peuvent être nécessaires, comme pour DALL•E 2. Même si cela peut nécessiter moins de complexité que des modèles de langage naturel généraux comme GPT-3, peu de travaux ont été réalisés dans cette direction. Une tentative récente est le contrôleur proposé par Juravsky et al. [START_REF] Juravsky | PADL: Language-Directed Physics-Based Character Control[END_REF].

Nous pensons que les contributions présentées dans cette thèse constituent un premier pas dans ces directions et qu'elles ouvriront la voie à de nouvelles approches pour l'animation de personnages virtuels basées sur des approches d'apprentissage profond. to right: Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF], MPI-INF-3DHP [START_REF] Mehta | Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision[END_REF], AMASS [START_REF] Mahmood | AMASS: Archive of Motion Capture As Surface Shapes[END_REF],

PSU-TMM100 [START_REF] Scott | From Image to Stability: Learning Dynamics from Human Pose[END_REF] and U nder P ressure [125]). Motion sequence categories in U nder P ressure. . . . . . . . . . . Table 5.2 F 1 score on foot contact labels detection of our method, its variants for ablative study purposes, and the optimal thresholds (OT) baseline. Bold and underline respectively indicate per-column best and second best. Our method outperforms the OT baseline and its linear generalisations, and the proposed architecture seems relevant with respect to the 3-layer multilayer perceptron (MLP). In that context, character animation has a great importance for the overall quality of the artworks. The way humans move is very diverse and conveys a lot of information about the activity performed, the character's state of mind, such as intentions, moods or emotions, the character's identity, such as morphological and biological characteristics, etc., which all have to be consistent together with other visual aspects. In addition, Newton's second law of motion inherently makes human motion a dynamic process, while our understanding of biomechanics is far from being comprehensive. For these However, it suffers from some limitations related to the structure of human motions.

For instance, joint positions do not encode the information of bone orientations around themselves which is often needed for concrete applications in animation, in order to display more natural mesh deformations. Positional representations also do not explicitly constrain bone lengths to remain constant over time, therefore requiring reliance on additional constraints to ensure that the skeleton does not break apart. For these reasons, communities closer to animation, such as computer graphics, rarely use purely positional pose representations, while on the opposite, communities more commonly involved in deep learning, such as computer vision, are more prone to employ these representations.

Angular Pose Representations

Angular representations have been widely used in animation mainly because their hierarchical nature allows straightforward orientation of any joint together with all of its descendants, while keeping bone lengths constant. Indeed, the position of each joint is described with respect to its parent as a 3D rigid transformation, often decomposed into a variable rotation and a fixed translation, corresponding to the joint orientation and the bone dimensions respectively. Main differences among angular pose representations are determined by the parameterisation of the rotations, however there are also representations working at the level of rigid transformation parameterisations.

Formally, the set of all rotations of R 3 equipped with the composition is the 3D [START_REF] Grassia | Practical Parameterization of Rotations Using the Exponential Map[END_REF]. Moreover, they do not work well for interpolations since the space of orientations is highly nonlinear [START_REF] Grassia | Practical Parameterization of Rotations Using the Exponential Map[END_REF]. Finally, multiple conventions exist for the axes considered, including their order, which requires to define them formally in each application to avoid any ambiguity. For these reasons, this representation is inappropriate for a lot of applications.

lie algebras.

A popular angular pose representation in skeletal character animation consists in representing each joint rotation as an element of so(3), where the direction and magnitude of the vector correspond to the axis and angle of the rotation [START_REF] Grassia | Practical Parameterization of Rotations Using the Exponential Map[END_REF], respectively. Since such a vector is an element of so(3), the parameterisation is defined by the exponential map from so(3) to SO(3) which can be efficiently computed with the Rodrigues' formula [START_REF] Rodrigues | Des lois géométriques qui régissent les déplacements d'un système solide dans l'espace, et de la variation des coordonnées provenant de ces déplacements considérés indépendants des causes qui peuvent les produire[END_REF]. This pose representation is often called exponential map, and is sometimes confused with the so-called axis-angle representation that is equivalent but separates the vector into a unit vector and a scalar describing the axis and the magnitude of the rotation.

Since Lie algebras are locally linearised versions of their Lie group, so(3) is a compelling space to work with elements of SO(3). However, as all parameterisations of SO(3) in R 3 , the exponential map representation has singularities [START_REF] Grassia | Practical Parameterization of Rotations Using the Exponential Map[END_REF] leading to losing a DOF in some parts of the representation space, even though these are located on the spheres of radius 2kπ for k ∈ N + , since a rotation of 2π about any axis is equivalent to no rotation. Therefore, this representation is often well-suited in animation since control and simulation deal with small time steps and thus with small rotations that stay inside the sphere of radius 2π, far from the singularities. It has been employed in early works in deep animation (e.g. [START_REF] Taylor | Factored Conditional Restricted Boltzmann Machines for Modeling Motion Style[END_REF][START_REF] Taylor | Modeling Human Motion Using Binary Latent Variables[END_REF]), and broadly exploited for motion synthesis (e.g. [START_REF] Alemi | Affect-Expressive Movement Generation with Factored Conditional Restricted Boltzmann Machines[END_REF][START_REF] Chiu | A style controller for generating virtual human behaviors[END_REF]), as well as in other topics (e.g. [START_REF] Alemi | WalkNet: A Neural-Network-Based Interactive Walking Controller[END_REF][START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Jang | Constructing Human Motion Manifold with Sequential Networks[END_REF][START_REF] Mason | Few-shot Learning of Homogeneous Human Locomotion Styles[END_REF]). However, as quality needs increase, long-term correlations are more and more important and inevitably imply larger rotations, getting close to the singularities of the parameterisation.

Similarly to the exponential map representation which uses so(3) to represent joint orientations with respect to their parents, Liu et al. [START_REF] Liu | Towards Natural and Accurate Future Motion Prediction of Humans and Animals[END_REF] proposed a pose representation using se(3) to represent rigid transformations of each joint with respect to its parent.

The main motivation for choosing such a representation is to explicitly encode both geometric constraints (i.e. bone lengths) and actual DOFs (i.e. joint orientations)

together. Nevertheless, it still has the same singularities as the exponential map representation. As we will see in the following paragraphs, other parameterisations in higher-dimensional spaces than R 3 , i.e. over-parameterised representations, are able to prevent such singularities.

rotation matrices.

In computer graphics, rotation matrices are widely used to represent 3D rotations. The corresponding parameterisation is the identity since elements of SO(3) are 3 × 3 matrices. Rotation matrices have no singularity and can be integrated together with joint translations into a 4 × 4 homogeneous matrix, which is elegant and effective when involved in computations like composition or inverse.

However, such a representation is particularly difficult to work with when its parameters must be estimated since the representation is over-parameterised. Indeed, not all 3 × 3 matrices belong to SO (3). By definition, a matrix R ∈ R 3×3 must satisfy R ⊤ R = I and det(R) = 1 to be a valid 3D rotation. For instance, predicting the orientation of a joint in matrix representation would require to solve a constrained optimisation problem to ensure the validity of the rotation, which can be tedious.

unit quaternions.

A more compact representation than rotation matrices are unit quaternions. Lying in R 4 , they are free of singularities, suitable for interpolation [START_REF] Grassia | Practical Parameterization of Rotations Using the Exponential Map[END_REF], numerically stable and computationally efficient [START_REF] Pavllo | QuaterNet: A Quaternionbased Recurrent Model for Human Motion[END_REF]. Like so(3), the space of unit quaternions has the same local geometry and topology as SO(3) [START_REF] Grassia | Practical Parameterization of Rotations Using the Exponential Map[END_REF]. However, unit quaternions are also over-parameterised, but have only four parameters (in comparison to nine parameters for rotation matrices). Thus, they must be constrained to remain on the unit 4-sphere. This angular representation has been popularised in DL-based animation with QuaterNet, a quaternion-based framework for human motion prediction proposed by Pavllo et al. [START_REF] Pavllo | QuaterNet: A Quaternionbased Recurrent Model for Human Motion[END_REF][START_REF] Pavllo | Modeling Human Motion with Quaternion-Based Neural Networks[END_REF]. In that framework, Pavllo et al. introduced a penalty term in the loss function for all quaternions predicted by the network that minimises their divergence from the unit length. It encourages the network to predict valid rotations and leads to better training stability. Moreover, the predicted quaternions are also normalised after computing the penalty to enforce their validity.

According to the authors, the distribution of predicted quaternion norms converges to a Gaussian with mean 1 during the training, suggesting that the model actually learns to represent valid rotations. Since Pavllo et al. [START_REF] Pavllo | QuaterNet: A Quaternionbased Recurrent Model for Human Motion[END_REF] showed promising results using quaternions, their use is gaining popularity.

gram-schmidt-like. Zhou et al. [START_REF] Zhou | On the Continuity of Rotation Representations in Neural Networks[END_REF] recently pointed out that all representations in R n with n ≤ 4 have discontinuities which can be unfavourable for DNNs training.

They therefore introduced a continuous representation of n-dimensional rotations SO(n) with n 2 -n dimensions. The mapping from SO(n) to the representation space simply drops the last column vector of the input n × n matrix. The inverse mapping back to SO(n), called Gram-Schmidt-like (GSL) process, is a Gram-Schmidt process over the n -1 column vectors followed by the computation of the last column vector by a generalisation to n dimensions of the cross product. In the case of SO(3), this GSL representation gives a 6D representation. Zhou et al. [START_REF] Zhou | On the Continuity of Rotation Representations in Neural Networks[END_REF] also provided a method to further reduce the dimensionality from 6D to 5D while still keeping a continuous representation using a stereographic projection combined with normalisation. However, they empirically found that nonlinearities introduced by the projection can make the learning process more difficult. A few of the most recent methods in skeletal character animation [START_REF] Li | GANimator: Neural Motion Synthesis from a Single Sequence[END_REF][START_REF] Oreshkin | Motion In-betweening via Deep Delta-Interpolator[END_REF][START_REF] Qin | Motion In-Betweening via Two-Stage Transformers[END_REF]] make use of the 6D GSL pose representation, which tends to confirm the interest of the aforementioned appealing properties.

hierarchical representation limitations.

In skeletal character animation, a hierarchical modelling approach is used in conjunction with angular pose representations, i.e. the representation of the joint orientations relative to their parents. In that context, positional errors over proximal joints (e.g. the shoulder) are propagated and accumulated down the kinematic chains. This is problematic in optimisation-based frameworks such as deep learning since equally distributed joint orientation errors will result in growing joint position errors along the kinematic chains as depicted in To solve this problem, Pavllo et al. [START_REF] Pavllo | QuaterNet: A Quaternionbased Recurrent Model for Human Motion[END_REF][START_REF] Pavllo | Modeling Human Motion with Quaternion-Based Neural Networks[END_REF] performed forward kinematics (FK) to convert quaternion-based poses predicted by their DNN into 3D joint positions, and then penalised absolute position errors instead of angular errors. Since FK is a differentiable operation with respect to joint orientations, they can train their network end-to-end using a positional loss.

Hybrid Representations

As mentioned above, both angular and positional approaches for representing human poses have advantages and drawbacks that sometimes depend on the application or viewpoint, dividing researcher communities. For this reason, several works have proposed hybrid representations with the goal of mitigating drawbacks while keeping benefits of both types of representations.

Aberman et al. [3] presented a novel data-driven approach for retargeting motions between homeomorphic skeletons (see Section 2.4.2) along with an interesting and elegant representation of human motion illustrated in Figure 2.2. In this work, both Aberman et al. [3] penalised errors in the positional space after performing FK, while also penalising errors in the quaternion space. Following this work, Shi et al. [START_REF] Shi | MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency[END_REF] addressed the reconstruction of 3D kinematic skeletons from 2D key points estimated from monocular video while dividing the motion representation into static bone lengths and dynamic joint orientations. To this end, a DNN called MotioNet learns to map 2D key points to a symmetric static skeleton, represented by its bone lengths and a dynamic sequence of joint rotations (quaternions) which are then combined through FK to get a full kinematic skeleton.

Finally, the success of multiple recent methods in skeletal character animation mixing different pose representations suggests that DNNs benefit from redundant pose information. In addition to joint orientations, researchers often feed their models with joint positions [START_REF] Lee | Interactive Character Animation by Learning Multi-Objective Control[END_REF], joint positions and velocities [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Yu | Character Controllers using Motion VAEs[END_REF][START_REF] Mason | Few-shot Learning of Homogeneous Human Locomotion Styles[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF] or even joint positions and linear and angular velocities [START_REF] Holden | Learned Motion Matching[END_REF].

Human Motion Datasets

Another crucial aspect of data-driven approaches is the choice of dataset, from which a DL-based model will learn a deep representation of human motion. In particular, large amounts of high-quality motion data are necessary to constitute so-called benchmark datasets and to provide robust assessment procedures. In this section we introduce a selection of human motion datasets that are the most relevant for skeleton character animation. Although many datasets have been proposed, only a small number of them have been repeatedly exploited and even fewer can be considered as standard bench- Size & Framerate Data marks. Table 2.1 provides relevant information about these datasets most commonly used in the works presented in this chapter.

The two most widely used databases in skeleton-based deep human animation are CMU [START_REF]CMU Graphics Lab Motion Capture Database[END_REF] and Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF]. Both are standard large-scale human motion datasets for learning and evaluation. Despite the fact that the CMU dataset was released about a decade earlier than Human3.6M, they have a comparable size (see Table 2.1). The main advantage of Human3.6M over CMU is the presence of RGB+D videos synchronised with human pose sequences, making it sometimes more suitable for tasks closer to computer vision such as motion prediction, even though CMU is also often leveraged.

Beyond these two standard human motion databases, a few others are noticeable, e.g. for the types of motion they contain, for the annotations that are included, or even for the environment in which they were captured. A few years after the release of CMU, Müller et al. [START_REF] Müller | Documentation Mocap Database HDM05[END_REF] It contains challenging sequences including walking in the city, going up-stairs, or taking the bus for a total of more than 51000 frames.

A special need of data occurs from the use-case of retargeting (see Section 2.4.2), which consists in transferring the movements of a character to another one with a different morphology, i.e. motion data with diversity among morphologies. Unfortunately, most human motion datasets feature only a very limited number of subjects with minor morphological differences. As a result the so-called Mixamo Dataset [5] is often used to train or evaluate models for retargeting. Indeed Mixamo is a company developing services for 3D character animation including downloadable animation sequences performed by numerous 3D character models with varied morphologies.

These animations were created using motion capture and cleaned up by key frame animators1 .

Finally, existing datasets were also gathered to build larger human motion databases.

Holden et al. [START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF] constructed a dataset by collecting CMU [START_REF]CMU Graphics Lab Motion Capture Database[END_REF], HDM05 [START_REF] Müller | Documentation Mocap Database HDM05[END_REF],

MHAD [START_REF] Ofli | Berkeley MHAD: A Comprehensive Multimodal Human Action Database[END_REF] and Xia et al. [START_REF] Xia | Realtime Style Transfer for Unlabeled Heterogeneous Human Motion[END_REF] in addition to internal motion capture sequences. This data was retargeted to a uniform skeleton structure and resampled to 120 frames per second. More recently, Mahmood et al. [START_REF] Mahmood | AMASS: Archive of Motion Capture As Surface Shapes[END_REF] proposed AMASS, which unifies 15 different optical marker-based motion capture datasets (including CMU [START_REF]CMU Graphics Lab Motion Capture Database[END_REF], HDM05 [START_REF] Müller | Documentation Mocap Database HDM05[END_REF],

SFU [START_REF] Fraser | SFU Motion Capture Database[END_REF], HumanEva [START_REF] Sigal | HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion[END_REF]). The size of AMASS, initially around 42 hours of data, is still increasing. Motion sequences in AMASS are parameterised using the Skinned Multi-Person Linear (SMPL) model [START_REF] Loper | SMPL: A Skinned Multi-Person Linear Model[END_REF], a learnt model of human body shape and pose that provides a parameter space from which the skeleton, the joint orientations and the body mesh can be computed.

Learning Spatial Features

Besides the pose representation and the dataset, the network architecture can also have a significant impact on the deep representation learned. In this section, we present different types of architectures leveraged to learn spatial correlations in motion data.

While most DNNs designed to address tasks related to skeletal character animation do not exploit the prior knowledge we have about geometric and structural aspects of the human skeleton, e.g. its symmetry or its hierarchical structure, a few methods proposed various clever architectures to benefit from this prior knowledge. We present them in the following sub-sections, divided into three categories: spatially-structured architectures, convolutional neural networks (CNNs) and graph convolutional networks (GCNs).

Spatially-Structured Architectures

A first group of approaches to help DNNs learn spatial correlations rely on network architectures structured around the human skeleton, such that the function computed by the network intrinsically encodes human skeleton characteristics. These approaches split the skeleton into body parts and process the corresponding data either in parallel network branches or hierarchically.

In parallel approaches, the main difference is usually related to the targeted task, which conditions the architecture of individual branches. Wang and Neff [START_REF] Wang | Deep Signatures for Indexing and Retrieval in Large Motion Databases[END_REF] extracted deep motion signatures with an independent autoencoder (AE) for each branch (i.e. limb or torso) and concatenated the outputs. Guo and Choi [START_REF] Guo | Human Motion Prediction via Learning Local Structure Representations and Temporal Dependencies[END_REF] relied instead on fully-connected (FC) layers for each branch, whose outputs are merged using a shared layer to predict the next frame. Nakada et al. [START_REF] Nakada | Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation[END_REF] similarly divided the body into separate modules responsible for controlling muscle activations of body parts from preprocessed common visual information. Both Goel et al. [START_REF] Goel | Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding[END_REF] and Men et al. [START_REF] Men | GAN-based reactive motion synthesis with class-aware discriminators for human-human interaction[END_REF] fed long short-term memories (LSTMs) with human body parts in their adversarial motion synthesis framework. Finally, Jain et al. [START_REF] Jain | Structural-RNN: Deep Learning on Spatio-Temporal Graphs[END_REF] predicted future poses with one recurrent neural network (RNN) per body part, whose inputs are the predictions of neighbouring RNN at the previous timestamps as well as their own previous predictions.

Hierarchical approaches can be either top-down or bottom-up. Wang et al. [START_REF] Wang | Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling[END_REF] proposed a spatial encoder that separates the human pose into five body parts, then encodes and merges them two by two recursively. Both Li et al. [START_REF] Li | Efficient convolutional hierarchical autoencoder for human motion prediction[END_REF] and Bütepage et al. [START_REF] Bütepage | Deep representation learning for human motion prediction and classification[END_REF] used a similar top-down approach with a finer human pose split at the input.

In contrast, Aksan et al. [START_REF] Aksan | Structured Prediction Helps 3D Human Motion Modelling[END_REF] proposed a bottom-up scheme where the human pose is predicted step by step from the root joint (e.g. pelvis) to the end effectors, i.e. the root is first predicted and then the other joints are recursively predicted using neighbouring predictions as additional inputs.

Convolutional Neural Networks

Another type of architecture sometimes employed to model spatial correlations relies on 2D convolutions, i.e. in the spatial and temporal domains. To this purpose, the skeleton graph is flattened along the spatial dimension. CNNs are particularly efficient at learning spatial correlations in data whose structure is regular such as images. However, learning the spatio-temporal dynamics of human joints remains challenging with CNNs because the graph structure of the human skeleton cannot be meaningfully flattened along a single dimension. To capture the spatial correlations of joints from different limbs, Li et al. [START_REF] Li | Convolutional Sequence to Sequence Model for Human Dynamics[END_REF] proposed to enlarge the convolutional kernels in the spatial domain. More recently, Zang et al. [START_REF] Zang | Few-shot Human Motion Prediction via Learning Novel Motion Dynamics[END_REF] proposed to adaptively model the spatial correlations with deformable convolutional kernels whose relative positions of the entries are learned. The problem of learning patterns in irregular data structures like human poses with CNNs can be overcome by extending convolutions to graph-structured data, as we will see next.

Graph Convolutional Networks

To leverage CNNs while properly handling the graph-structure typically used in animation to represent skeletons, GCNs, an extension of CNNs, have been recently considered in different frameworks working on human motion data. GCNs come in two different flavours [START_REF] Bronstein | Geometric Deep Learning: Going beyond Euclidean data[END_REF]. Spatial approaches map neighbourhoods of each node in the graph to Euclidean patches on which a convolution is applied. Spectral approaches operate in the Fourier domain of the feature signals sampled on the graph, which depends on the graph Laplacian operator [START_REF] David | The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains[END_REF]. By analogy with the convolution theorem, convolution filters are defined as spectral coefficients that are multiplied by the Fourier transforms of the signals.

Aberman et al. Other methods leveraging GCNs relied on the spectral approach proposed by Kipf and Welling [START_REF] Kipf | Semi-Supervised Classification with Graph Convolutional Networks[END_REF]. Here, the output F l+1 of the graph convolutional layer l fed with a feature signal F l is formulated as F l+1 = σ( ÂF l W l ) where W l is the tensor of learnable convolution filter weights, Â depends only on the graph adjacency matrix and σ is a non-linear activation function. Most of the time, the weights of the adjacency matrix are learnt in addition to the convolution filter. Mao et al. [START_REF] Mao | Learning Trajectory Dependencies for Human Motion Prediction[END_REF] built their adjacency matrix from a fully connected graph of joints and thereby simultaneously learnt the motion correlations between joints that are physically connected and joints that are far apart but whose motion are dependent, e.g. hands and feet during walking. Cui et al. [START_REF] Cui | Learning Dynamic Relationships for 3D Human Motion Prediction[END_REF] objected that this scheme may result in unstable training and separately learnt two graph adjacency matrices, one in which the weights of non-connected joints in the skeleton are forced to zero and another with full joint connectivity.

Learning Temporal Features

The temporal dimension of motion data is informative of the nature of the action being performed as well as the way it is performed. In the following sub-sections we review the approaches taken to model human dynamics, most of which rely either on RNNs or CNNs.

Recurrent Neural Networks

RNNs are neural networks designed to process each timestep of time series one after another, and can thereby handle variable length sequences. They maintain an internal state that captures the temporal context of the signal. RNNs are most of the time based on LSTM or gated recurrent unit (GRU).

long short-term memory.

A common LSTM is composed of a memory cell that remembers values over arbitrary time intervals, and three gates -an input gate, an output gate and a forget gate -to regulate the flow of information into and out of the cell and to avoid a common problem with RNNs known as the vanishing (or exploding) gradient problem. In general, the problem is that the gradients used to update the network weights can become extremely small (or large), either preventing the network from further learning or making the network diverge, respectively. In the case of RNNs, the backpropagation through time heavily relies on the chain rule to compute gradients which exponentially decrease (vanishing problem) or increase (exploding problem) if any weight is greater or smaller than one, respectively.

The memory cell remembers values over arbitrary time intervals, making LSTM effective at capturing both short-term and long-term temporal dependencies. Indeed,

LSTMs have proven to be powerful for learning temporal dependencies by achieving state-of-the-art performance in key applications e.g. natural language processing and machine translation. In skeletal character animation, LSTMs have been broadly

employed, e.g. in motion synthesis [START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Henter | MoGlow: Probabilistic and controllable motion synthesis using normalising flows[END_REF][START_REF] Wang | Adversarial learning for modeling human motion[END_REF][START_REF] Wang | Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions[END_REF], in character control [START_REF] Lee | Interactive Character Animation by Learning Multi-Objective Control[END_REF][START_REF] Wang | Encoder-decoder recurrent network model for interactive character animation generation[END_REF][START_REF] Wang | Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control[END_REF],

as well as in motion editing [START_REF] Wang | Transferring Style in Motion Capture Sequences with Adversarial Learning[END_REF].

gated recurrent unit.
As an alternative to LSTMs, GRUs have also been widely used, such as in motion synthesis [START_REF] Sadegh Aliakbarian | A Stochastic Conditioning Scheme for Diverse Human Motion Prediction[END_REF][START_REF] Barsoum | HP-GAN: Probabilistic 3D human motion prediction via GAN[END_REF][START_REF] Yuan | DLow: Diversifying Latent Flows for Diverse Human Motion Prediction[END_REF] or in motion editing [START_REF] Jang | Constructing Human Motion Manifold with Sequential Networks[END_REF][START_REF] Villegas | Neural Kinematic Networks for Unsupervised Motion Retargetting[END_REF]. GRUs rely on a gating mechanism similar to LSTMs in order to avoid the vanishing gradient problem but have only two gates, a reset gate and an update gate. As a result, GRUs use fewer parameters and therefore less memory, are computationally less expensive [START_REF] Guo | Human Motion Prediction via Learning Local Structure Representations and Temporal Dependencies[END_REF] and thus train faster than LSTMs. They can process entire motion datasets [START_REF] Martinez | On human motion prediction using recurrent neural networks[END_REF] instead of training action-specific models. However, as shown by Weiss et al. [START_REF] Weiss | On the Practical Computational Power of Finite Precision RNNs for Language Recognition[END_REF], LSTMs are strictly stronger than GRUs as they can easily perform unbounded counting, while GRUs cannot. Thus, LSTMs seem more accurate than GRUs on longer sequences. In summary, the choice between LSTMs and GRUs depends on the processed data and the considered application.

Besides pure LSTMs or GRUs, extensions [START_REF] Tang | Long-Term Human Motion Prediction by Modeling Motion Context and Enhancing Motion Dynamics[END_REF] or combinations of both [START_REF] Wang | Imitation Learning for Human Pose Prediction[END_REF] have been used to model human dynamics. Bidirectional LSTMs (BiLSTMs) stack two LSTMs running forward and backward, respectively. As a result, temporal information is processed and preserved in both directions, i.e. past and future, which is helpful on certain tasks. For instance, BiLSTMs have been leveraged to synthesise motion sequences [START_REF] Goel | Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding[END_REF][START_REF] Xu | Hierarchical Style-based Networks for Motion Synthesis[END_REF] or even to refine 3D motion data [START_REF] Li | A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network[END_REF][START_REF] Li | Bidirectional recurrent autoencoder for 3D skeleton motion data refinement[END_REF].

Transformer

One of the limitations of RNNs is the difficulty to learn to model correlations in time series occurring between arbitrarily distant pieces of information. Indeed, during training, their recurrent nature imposes to gradients to flow through very long computational paths to model distant correlations. This results in eventually falling back to the vanishing gradient problem initially intended to be mitigated by recurrent connections. In natural language processing, researchers identified this issue and introduced attention mechanisms in recurrent architectures to face the vanishing gradient problem in a first step. In contrast with classical RNNs, attention-based RNNs additionally model correlations between all intermediate input and output hidden states instead of only passing the final hidden states. These skip connections effectively create shortcuts for gradients. In a second time, researchers found even more efficient to solely rely on attention mechanisms and dispense recurrent connections [START_REF] Vaswani | Attention is All You Need[END_REF], resulting in the transformer architecture, recently becoming very popular.

A typical transformer network is composed of a stack of multi-head attention blocks.

Each block starts with a self-attention layer which first maps a sequence of input tokens to three values usually referred to as query Q, keys K and values V, in analogy with retrieval systems. Then attention scores are computed by matrix multiplication of Q and K and used to weight V. In a multi-head attention layer, this process is done several time in parallel with different set of learnable parameters, and the corresponding results are summed up. Then, attention is followed by a token-wise feed-forward layer to better fit the input for the next attention layer. Finally, both attention and feed-forward layers are residual and layer normalisation [START_REF] Lei Ba | Layer Normalization[END_REF] is applied on their outputs. Moreover, since transformers contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, information is injected about the relative or absolute position of the tokens in the sequence, e.g. with position encoding.

In skeletal character animation, transformer-based models started to gather attention from researchers essentially in the last three years. In particular, this kind of models have been explored towards motion in-betweening [START_REF] Duan | Single-Shot Motion Completion with Transformer[END_REF][START_REF] Oreshkin | Motion In-betweening via Deep Delta-Interpolator[END_REF][START_REF] Qin | Motion In-Betweening via Two-Stage Transformers[END_REF] (see Section 2.3.1) and upper-body gestures style transfer [START_REF] Kuriyama | Context-based Style Transfer of Tokenized Gestures[END_REF] (see Section 2.4.3).

Temporal Convolutions

Convolutional neural networks (CNNs) also constitute an alternative to RNNs for learning temporal patterns in motion data. They can be either 1D along the temporal dimension, or use 2D spatio-temporal convolutions. Stacking several convolutional layers can efficiently capture both short and long range temporal patterns since lower and higher layers will capture dependencies between nearby and distant frames, respectively. CNN-based approaches are more computationally efficient than RNN-based ones because they process whole motion segments at once rather than frame by frame, allowing greater parallelism. However, CNN-based architectures often contain elements that do not allow variable length inputs (e.g. a few FC layers after convolutions) limiting their use in practice. As a result, they are more prone to be used in tasks where fixed-length motion sequences are suitable, e.g. motion editing [3,4,[START_REF] Dong | Adult2child: Motion Style Transfer using CycleGANs[END_REF][START_REF] Holden | Fast Neural Style Transfer for Motion Data[END_REF][START_REF] Kim | Motion Retargetting based on Dilated Convolutions and Skeleton-specific Loss Functions[END_REF][START_REF] Lim | PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting[END_REF][START_REF] Lohit | Recovering Trajectories of Unmarked Joints in 3D Human Actions Using Latent Space Optimization[END_REF][START_REF] Shi | MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency[END_REF][START_REF] Shimada | PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time[END_REF][START_REF] Zou | Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints[END_REF] (see Section 2.4).

Miscellaneous

Other approaches to better model the temporal flow of human motions include motion phase representation, spectral decomposition of motion data and spatio-temporal attention. For instance, several authors investigated the representation and learning of the phase of movements in the context of kinematic character control, which is detailed in Section 2.3.2. In the character controller network proposed by Holden et al. [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF], the network weights are computed as a spline function of the phase, whose control points, representing network weights configurations during the human locomotion cycle, are learned. Other works [START_REF] Yu | Character Controllers using Motion VAEs[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF] use a gating network instead of the cyclic phase to blend expert weights, resulting in a mixture-of-experts scheme.

motion synthesis and character control

Generating new motion sequences is of strong practical interest to the media and entertainment industry. Besides creating realistic and diverse sequences, a key challenge is to be able to control various aspects of the motion with high-level parameters. In this section we distinguish motion synthesis and character control. The former focuses on synthesising new motions both consistent with the distribution of samples in a reference training dataset and sufficiently diverse to capture its variations, optionally conditioned on semantic cues typically pertaining to the character trajectory or motion style. Here we restrict ourself to general-purpose approaches and exclude works targeting application-specific contexts such as the synthesis of gestures from speech or dance animations from music. The latter also strives to generate motions but in an online fashion, dynamically responding to user inputs while satisfying environmental constraints. The needs of responsiveness and the fact that user inputs are not known in advance, unlike conditioning parameters in motion synthesis, make character control approaches very different from motion synthesis.

Motion Synthesis

We define motion synthesis as the process of creating perceptually plausible motion sequences with a desired style or expressed emotion for instance. Motion synthesis models are thus capable of generating different motions depending on inputs, e.g. low-level parameters such as latent variables or high-level parameters such as trajectory.

In the following sub-sections, we group approaches to motion synthesis based on the framework they build on, all but one being deep generative models. Indeed, in the scheme proposed by Holden et al. [START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF] the synthesis process is purely deterministic and driven by high-level cues. Finally, we conclude this section with an additional subsection treating of a particular case of motion synthesis known as in-betweening, where synthesised motions ought to be interpolated between given sparse poses referred to as keyframes. We summarise the methods presented in this section in Table 2.2.

Early Works

Pioneer works [START_REF] Alemi | Affect-Expressive Movement Generation with Factored Conditional Restricted Boltzmann Machines[END_REF][START_REF] Chiu | A style controller for generating virtual human behaviors[END_REF][START_REF] Taylor | Factored Conditional Restricted Boltzmann Machines for Modeling Motion Style[END_REF][START_REF] Taylor | Modeling Human Motion Using Binary Latent Variables[END_REF] in motion synthesis based on deep learning notably relied on restricted Boltzmann machines (RBMs). Initially proposed by Smolensky [START_REF] Smolensky | Information processing in dynamical systems: Foundations of harmony theory[END_REF],

RBMs build on a parametric expression of the probability density function of the data distribution from which new samples are to be generated. These early works are now outperformed by the more recent approaches described below e.g. variational autoencoders (VAEs) and generative adversarial networks (GANs), and have bee abandoned. Structured as an encoder followed by a decoder, an autoencoder is a DNN whose input and output represent the same data. The encoder output provides an intermediate latent code with a dimension often lower than the input data. Thus, autoencoders provide a scheme for non-linear dimensionality reduction. As illustrated in Figure 2.3, the distribution of the latent codes in a variational autoencoder (VAE) [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF] Gram matrix in the embedding space, following prior work in motion style transfer [START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF] (see Section 2.4.3). To synthesise a motion sequence, a path is determined in the motion graph based on user-defined trajectory controls, and motion primitives are generated along this path using the VAEs.

Variational Autoencoders

Yan et al. [START_REF] Yan | MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics[END_REF] and Aliakbarian et al. [START_REF] Sadegh Aliakbarian | A Stochastic Conditioning Scheme for Diverse Human Motion Prediction[END_REF] relied on similar network architectures for stochastic motion prediction. Pose sequences are mapped to lower-dimensional features by an encoder and transformed back to motion data by a decoder. The VAE operates on the encoded features, its output is fed to the decoder to synthesise the motion clips. Yan et al. [START_REF] Yan | MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics[END_REF] processed small pose sequences called motion modes that capture short-term motion features. During training, their VAE maps a pair of (past, future) mode features to a random latent code (encoder) then to a prediction of the future mode feature (decoder). It thereby captures the transition between the two modes. At inference time, the VAE and RNN decoders generate a stochastic prediction of the future mode, given a past conditioning mode and a draw of the VAE random latent code. Aliakbarian et al. [START_REF] Sadegh Aliakbarian | A Stochastic Conditioning Scheme for Diverse Human Motion Prediction[END_REF] argued that stacking the past sequence information and the random generating seed in a vector and feeding it to the decoding network leaves the possibility that the stochastic component is assigned low weights during training and is thus effectively ignored by the network. This concern is confirmed by experimental evidence. To avoid this, they proposed a mix-and-match perturbation strategy and formed a vector by replacing randomly selected components of the past sequence feature by corresponding components of the VAE latent code. Feeding this vector to the decoder forces it to account for both the past context and the stochastic input. Generative adversarial networks (GANs) are a popular alternative to VAEs for generating samples from random seeds. In a GAN, new samples are synthesised by a generator network that operates in conjunction with a discriminator network (see Figure 2.4).

Generative Adversarial Networks

The generator transforms a random seed drawn from a known prior distribution to a sample which aims to be similar to the contents of the training dataset, the discriminator assesses this similarity. The two networks are trained jointly with adversarial losses, the generator being driven to produce samples that the discriminator ultimately should not be able to distinguish from the training dataset samples. In motion synthesis, Barsoum et al. [START_REF] Barsoum | HP-GAN: Probabilistic 3D human motion prediction via GAN[END_REF] pointed out that relying on a GAN has intrinsic advantages. GANs do not suffer from the temporal error accumulation issues of RNNs and will produce one instance of all possible outputs instead of averaging these possible outputs. Moreover, GANs also natively embed a stochastic component. Any motion generation network can be wrapped into a GAN by adding a discriminator network to improve the plausibility of the generated sequences [START_REF] Barsoum | HP-GAN: Probabilistic 3D human motion prediction via GAN[END_REF][START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Wang | Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control[END_REF].

Most other works focused their contribution on optimising the architecture of the GAN generator network to facilitate and improve its training. Wang et al. [START_REF] Wang | Adversarial learning for modeling human motion[END_REF] proposed an adversarial autoencoder architecture [START_REF] Makhzani | Adversarial Autoencoders[END_REF] where a GAN enforces a prior on the distribution of the latent code of an autoencoder. Yan et al. [START_REF] Yan | Convolutional Sequence Generation for Skeleton-Based Action Synthesis[END_REF] fed the generator with a sequence of random samples drawn from a Gaussian process with a predetermined covariance function. Through a series of purely convolutional modules made up of a spatio-temporal upsampling layer followed by a graph convolution on the skeleton features, they gradually increased the spatial and temporal resolution of their output to produce a pose sequence. Wang et al. [START_REF] Wang | Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions[END_REF] split their generator into separate spatial and temporal sub-networks: the lower layer maps the input random seed and conditioning action label via an RNN to a sequence of low-dimensional latent codes that model temporal transitions. The upper layer decodes each latent code into a skeletal pose. The generator is regularised by a helper action classifier network through a cycle consistency constraint: the conditioning action label fed to the generator should agree with the result of the classification of the motion sequence it produces. Finally, both Men et al. [START_REF] Men | GAN-based reactive motion synthesis with class-aware discriminators for human-human interaction[END_REF] and Goel et al. [START_REF] Goel | Interaction Mix and Match: Synthesizing Close Interaction using Conditional Hierarchical GAN with Multi-Hot Class Embedding[END_REF] proposed a GAN-based approach for close interaction synthesis, i.e. synthesising the motion of a character (output) reacting to another character's motion (input). Both approaches rely on an LSTM-based generator with seq2seq attention mechanism [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF] and fed with separated human body parts, together with an LSTM-based binary and multi-class discriminator which tries to distinguish between fake and real samples as well as between different classes of interaction.

Different from most other approaches, Li et al. [START_REF] Li | GANimator: Neural Motion Synthesis from a Single Sequence[END_REF] learn a GAN-based motion synthesis model from a single motion sequence instead of a large motion dataset. To avoid overfitting the single training motion sequence, the receptive field of the discriminator is limited, following Patch-GAN [START_REF] Isola | Image-to-Image Translation with Conditional Adversarial Networks[END_REF][START_REF] Li | Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks[END_REF]. Moreover, following the progressive approach of Karras et al. [START_REF] Karras | Progressive Growing of GANs for Improved Quality, Stability, and Variation[END_REF] with images, motion sequences are generated progressively by upsampling them at each level. Finally, their architecture rely on skeleton-aware operators [3] as a backbone (see Section 2.2.3).

Normalising Flows

Generative models based on normalising flows (NF) [START_REF] Dinh | NICE: Non-linear Independent Components Estimation[END_REF] synthesise samples by applying a composition of invertible elementary transforms to a latent variable drawn from a known prior distribution. Unlike in GANs or VAEs, owing to the form of the elementary transforms, the probability density function of the generated samples can be computed in closed form. Thus, the generative network can be trained by maximum likelihood optimisation. Henter et al. [START_REF] Henter | MoGlow: Probabilistic and controllable motion synthesis using normalising flows[END_REF] adapted this framework to the generation of motion sequences. Each transform layer in the generator network is conditioned by a control signal that encodes the past trajectory of the root joint and holds an LSTM unit whose hidden state captures temporal dependencies.

Deterministic Generation

Holden et al. [START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF] conditioned their motion generation approach on purely deterministic constraints that specify the trajectory and velocity of the character. They leveraged an autoencoder operating on temporal chunks of poses to learn a latent manifold of human motion. A separate feedforward network maps the autoencoder embedding to high-level, semantically interpretable controls of the motion. Generating high-dimensional motion embedding samples from a set of low-dimensional control parameters is severely under-constrained. To disambiguate the generation to the largest possible extent, the scope of the approach is restricted to human locomotion, and a comprehensive set of trajectory and foot contact constraints is imposed.

In-betweening

The task of interpolating motion between sparse character poses, i.e. keyframes, is referred to as in-betweening. In early works [START_REF] Rose | Efficient Generation of Motion Transitions Using Spacetime Constraints[END_REF][START_REF] Witkin | Spacetime Constraints[END_REF], spacetime constraints and inverse kinematics were used to interpolate motion between keyframes. Then, probabilistic models including MAP optimisers [START_REF] Chai | Constraint-Based Motion Optimization Using a Statistical Dynamic Model[END_REF][START_REF] Min | Interactive Generation of Human Animation with Deformable Motion Models[END_REF] or Gaussian processes [START_REF] Wang | Gaussian Process Dynamical Models for Human Motion[END_REF] have also been used for in-betweening. However, as pointed out by Harvey et al. [START_REF] Gingras | Robust Motion In-betweening[END_REF], these models are specific to action and actors, making combinations of actions looking scripted and sequential. Recent works relied on the scalability and expressiveness of DNNs to tackle this issue.

Yu et al. [START_REF] Yu | Fast Terrain-Adaptive Motion Generation using Deep Neural Networks[END_REF] proposed a simple scheme for interpolating a motion sequence from starting and ending positions of end-effector joints. Their main objective is computational efficiency, in order to generate terrain-adaptive character motion in real time for video games. They cascaded two FC networks: the first interpolates the trajectories of the end effectors in time, the second infers full poses at each frame.

Then multiple works focused on generating plausible and diverse motion over long transitions bounded by a starting and an ending keyframe. Kaufmann et al. [START_REF] Kaufmann | Convolutional Autoencoders for Human Motion Infilling[END_REF] leveraged a deep convolutional denoising autoencoder, in which pooling layers ensure large receptive fields to capture long-range spatial and temporal joint correlations. A curriculum learning scheme feeds the encoder with sequences containing increasingly large temporal gaps to improve the training. Xu et al. [START_REF] Xu | Hierarchical Style-based Networks for Motion Synthesis[END_REF] proposed a temporally hierarchical scheme in which the transition segment is split into equal length subsegments. Trajectory constraints are provided at each sub-segment endpoint. The transition sequence is initialised by sampling a motion clip from the training dataset for each subsegment. Next, the style of each clip is changed to match the style of a reference sequence throughout the whole transition sequence. This stage leverages a motion autoencoder with separate content and style embeddings, that is trained in an unsupervised way. Style transfer is performed by linearly combining the content latent code of the initial clips and the style latent code of the reference style sequence. Finally, transitions between the endpoints of consecutive sub-segments are generated using forward and backward LSTM networks, and the plausibility of the generated sequence is enhanced by combining the generation network with a discriminator in an adversarial framework. Harvey et al. [START_REF] Gingras | Robust Motion In-betweening[END_REF] pointed out that in-betweening between distant keyframes may result in stalling or teleportation artifacts if the temporal evolution of the motion is not monitored during the generation process. To deal with this issue, Harvey et al. [START_REF] Gingras | Robust Motion In-betweening[END_REF] relied on an encoder-recurrent-decoder (ERD) architecture [START_REF] Fragkiadaki | Recurrent Network Models for Human Dynamics[END_REF] that is fed with the pose representation deltas between the current and the target frame, in addition to the character pose at the current timestep and the end pose. The time-to-arrival is encoded in a sine wave and added (similar to positional encoding), rather than stacked, to the latent code that is input to the RNN, forcing the network to take this piece of information into account during training. Later on, Tang et al. [START_REF] Tang | Real-Time Controllable Motion Transition for Characters[END_REF] proposed a real-time approach to in-betweening, by learning a deep representation of the low-level short-horizon motion dynamics through a conditional VAE modelling the distribution of frames conditioned on the preceding frame, as well as on the next frame hip velocity to disambiguate the next frame. Moreover, a conditional mixture-of-experts scheme is used in the decoder to achieve multi-modal frame transitions. Then, an additional network is learned to sample next frame hip velocity and latent code from previous and target frames and target duration. This enables in-betweening by iteratively conditioning the VAE on the previous frame. According to the authors, 50-frame sequences are too long for the learning to converge, and training sequences were therefore restricted to 25 frames.

Another limitation to this approach is that the target frame is not guaranteed to be reached.

One of the major issues of the aforementioned methods is that they struggle to capture long-term correlations. Indeed, autoregressive models are known to accumulate prediction errors while temporal horizons in convolutional architectures are limited by the size of the kernels and the number of layers. According to Starke et al. [START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF], these models also often suffer from low responsiveness due to the large variation of the memory state in the case of interactive character control, as the internal memory state is high dimensional.

To overcome these limitations, Holden et al. [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF] proposed the use of a specialised architecture called phase-functioned neural network (PFNN), which provides the phase variable to represent the progression of the motion. In their seminal work, the phase is defined based on alternating foot contacts, and used to generate the weights of the regression network at each frame. A trade-off between compactness and runtime speed can then be achieved by precomputing the phase-function for a number of fixed intervals, then interpolating the precomputed elements at runtime. One major limitation of PFNN is that phase functions need to be manually defined, which can be in some cases extremely complex [START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF]. Zhang et al. [START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF] therefore proposed to rely on a mixture-of-experts scheme to dynamically compute the weights of a motion prediction network (see Figure 2.5 for an illustration of the general concept). In

Gating Network Expert Pool Motion Prediction Network

Inputs: motion, controls, actions, etc.

α 1 α 2 α 3 ... α n x t

Blending coefficients

Network Weights

Blending Optional Encoder

Pose at time t +1

x t

x t+1

x t -2

x t -1

Figure 2.5 -Overall idea of the mixture-of-experts scheme [START_REF] Li | A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF]: a gating network first computes blending weights for a number of expert networks, based on a number of features extracted from the current frame x t . Each expert specialises in a particular movement. They are then blended together to dynamically compute the weights of a motion prediction network, which outputs information relevant to the next frame x t+1 including the pose of the animated character. This output is typically fed back into the network (autoregression) at time t + 1.

their architecture, a gating network first computes blending weights for a number of expert networks, each specialising in a particular movement. This approach was first demonstrated for creating complex quadruped character controllers and then extended by Starke et al. [START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF] to compute goal-directed series of motions and transitions, while potentially interacting with the environment. The idea was pushed one step further through the use of local motion phases [START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF], which are defined based on how each body part contacts external objects. Unlike previous approaches where different actions are considered to be synchronised by a single global phase variable, their approach describes each motion by a set of multiple independent and local phases for each bone.

It then enables neural networks to learn asynchronous movements of each bone, as well as its interaction with external elements of the virtual environment. Later on, Starke et al. [START_REF] Starke | DeepPhase: Periodic Autoencoders for Learning Motion Phase Manifolds[END_REF] proposed to learn a periodic autoencoder from unstructured motion data such that the features of its latent space serve as deep motion phases. Based on a temporal convolutional autoencoder [START_REF] Holden | Learning Motion Manifolds with Convolutional Autoencoders[END_REF], the latent space is further structured by parameterising each of its dimension as a sinusoidal function whose amplitude, frequency, offset and phase shift are learnt, enforcing periodicity encoding. A neural motion controller can then be obtained using a framework similar to the gated mixtureof-experts scheme (see above) [START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF] but using the deep motion phases as input to the gating network. Moreover, this approach can also be used in motion matching frameworks, by matching deep motion phases instead of higher-dimensional pose or velocity features of the current character state. Finally, while the previous approaches relied on autoregressive DNNs to generate controlled character motions, Ling et al. [START_REF] Yu | Character Controllers using Motion VAEs[END_REF] demonstrated that a VAE using a similar mixture-of-experts scheme is also viable to

motion editing

In the previous section, we looked at how motion data can be synthesised, either through offline generative approaches or interactive frameworks. Another important area of character animation is motion editing, which greatly diversifies the creative capabilities of artists. In this section, we focus on DL-based approaches divided into three topics: motion cleaning, motion retargeting and style transfer. The methods presented in this section are summarised in Table 2.4. Section Dataset Architecture

Cleaning

As mentioned by several authors, projection to and inverse projection from learnt manifolds of human motion can be used to clean motion data, e.g. to perform operations such as denoising, fixing corrupted information or filling in missing motion sequences. Such problems have for instance been explored using RBMs [START_REF] Wang | Deep Signatures for Indexing and Retrieval in Large Motion Databases[END_REF], convolutional autoencoders [START_REF] Holden | Learning Motion Manifolds with Convolutional Autoencoders[END_REF], temporal autoencoders [START_REF] Bütepage | Deep representation learning for human motion prediction and classification[END_REF][START_REF] Lohit | Recovering Trajectories of Unmarked Joints in 3D Human Actions Using Latent Space Optimization[END_REF], spatio-temporal RNNs [START_REF] Wang | Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling[END_REF],

sequential RNNs [START_REF] Jang | Constructing Human Motion Manifold with Sequential Networks[END_REF], BiLSTMs [START_REF] Li | Bidirectional recurrent autoencoder for 3D skeleton motion data refinement[END_REF], as well as RNN-based GANs [START_REF] Wang | Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control[END_REF]. While most approaches typically clean motion data through direct projection and inverse projection (e.g. [START_REF] Holden | Learning Motion Manifolds with Convolutional Autoencoders[END_REF][START_REF] Wang | Deep Signatures for Indexing and Retrieval in Large Motion Databases[END_REF]), then fix residual errors as a post-process, it is also possible to include additional constraints during training. For instance, it is possible to enforce bone length constraints and smoothness by including specific loss functions [START_REF] Li | A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network[END_REF][START_REF] Li | Bidirectional recurrent autoencoder for 3D skeleton motion data refinement[END_REF], or to include an additional perceptual loss measuring the difference in high-level features extracted by a pre-trained perceptual autoencoder [START_REF] Li | A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network[END_REF], which improves overall visual quality at the cost of a slight increase in reproduction error. Lohit and Anirudh [START_REF] Lohit | Recovering Trajectories of Unmarked Joints in 3D Human Actions Using Latent Space Optimization[END_REF] also proposed to optimise the latent representation to minimise the error between the reconstructed sequence and the correct information of the input sequence, which they applied to filling missing joint trajectories. While most approaches focus on cleaning directly kinematics skeletal data, Holden [START_REF] Holden | Robust Solving of Optical Motion Capture Data by Denoising[END_REF] proposed an approach producing joint transforms directly from raw marker data, in a way which is robust to errors in the input data. This approach is based on learning a deep denoising feedforward neural network using marker locations synthetically reconstructed from motion capture data, where the marker data is corrupted in terms of occlusions and positional shifts.

Footskate

Apart from general-purpose motion cleaning addressed above, researchers have explored footskate cleaning in particular [START_REF] Kovar | Footskate Cleanup for Motion Capture Editing[END_REF]. Artefacts referred to as footskate constitute a family of artifacts well-known in character animation and consist in any motion inconsistency between the ground and the feet, e.g. sliding on, passing through or floating above the ground. They are common and easily introduced when processing existing motion data and are known to be very detrimental to the perceived realism and disturb human perception even at very low intensities [START_REF] Pražák | Perceptual Evaluation of Footskate Cleanup[END_REF]. Moreover the problem of footskate is closely related to foot contacts which are necessary to quantify, prevent or fix such artefacts. In particular, most recent approaches remove footskate artefacts from motion sequences by enforcing foot contacts via IK [3,[START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Min | Motion graphs++: a compact generative model for semantic motion analysis and synthesis[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF]. As a consequence, foot contacts are needed to clean footskate artefacts and foot contact detection is therefore closely related to the problem of footskate.

The most widespread family of approaches in both animation research and industry to extract foot contact labels from motion data relies on simple heuristics with handcrafted thresholds, applied to velocity and proximity relative to the ground as proposed by Bindiganavale and Badler [START_REF] Bindiganavale | Motion Abstraction and Mapping with Spatial Constraints[END_REF] or Lee et al. [START_REF] Lee | Interactive Control of Avatars Animated with Human Motion Data[END_REF]. However, such methods lack of temporal precision and are not reliable [START_REF] Le | Robust Kinematic Constraint Detection for Motion Data[END_REF]. Therefore, they are either limited to algorithms that are insensitive to the accuracy of contact labels [START_REF] Park | Learning predict-and-simulate policies from unorganized human motion data[END_REF], or require tedious manual checking and corrections [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Mason | Few-shot Learning of Homogeneous Human Locomotion Styles[END_REF][START_REF] Wang | Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control[END_REF] to produce faithful contact detection results. Moreover, the accuracy of these approaches often dramatically drops when decreasing motion sequences quality, e.g. increasing noise or artefacts.

Other heuristic approaches have been sparsely investigated, including the work of Le Callennec and Boulic [START_REF] Le | Robust Kinematic Constraint Detection for Motion Data[END_REF]. Stationary or rotating point constraints are computed by solving linear equation systems assuming rigid transformations and a uniform noise pattern, itself estimated from known user-defined constraint(s). While addressing the unreliability of common heuristics-based methods due to the fact that the zero velocity assumption does not hold in the presence of noise, this approach is not fully automatic.

Moreover, the noise pattern is assumed to be uniform along time, which is not always true.

Researchers also investigated learned contact detection models. E.g. Ikemoto et

al. [START_REF] Ikemoto | Knowing When to Put Your Foot Down[END_REF] proposed a k-nearest neighbours (KNN) classifier to determine when the feet should be planted and claimed to be more accurate than heuristics-based approaches.

However, a very low amount of manually annotated motion data was used (about 3 minutes, single subject) preventing the approach to generalise well. The proposed KNN classifier achieves 90.78% accuracy over about 33 seconds of hand-labelled motion data, compared to 57.45% and 57% for speed-based and height-based thresholding baselines, respectively. However, these relatively low accuracies suggest that the corresponding thresholds are not optimal.

More recently, researchers leveraged neural networks to solve this problem. Smith et al. [START_REF] Harrison | Efficient Neural Networks for Real-time Motion Style Transfer[END_REF] detected foot contact labels using a dedicated MLP to remove footskate artefacts through IK in a motion style transfer pipeline. Zou et al. [START_REF] Zou | Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints[END_REF], Shi et al. [START_REF] Shi | MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency[END_REF],

and Rempe et al. [START_REF] Rempe | Contact and Human Dynamics from Monocular Video[END_REF] concurrently proposed to leverage foot contact detection to refine human motion estimation. In these works, foot contact labels are detected from 2D key points, themselves estimated from images using OpenPose [START_REF] Cao | OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields[END_REF], a state-of-the-art 2D pose estimator. Both Zou et al. [START_REF] Zou | Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints[END_REF] and Rempe et al. [START_REF] Rempe | Contact and Human Dynamics from Monocular Video[END_REF] used a dedicated module for contact detection while Shi et al. [START_REF] Shi | MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency[END_REF] detected them together with 3D joint rotations and global root positions. Shimada et al. [START_REF] Shimada | PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time[END_REF] extended the contact detection module from Zou et al. [START_REF] Zou | Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints[END_REF] to additionally detect at each frame whether the subject is stationary or not. However, in all these works the foot contact detection modules were trained with ground truth contacts obtained from simple heuristics, as previously described. Tedious manual screening has occasionally been used to correct or label motion sequences, but this approach limits the amount of labelled data which is crucial with neural networks.

Finally, researchers also embedded foot contact detection into their model with the goal of improving foot positioning consistency, but do not support detection from motion data. Yang et al. [START_REF] Yang | LoBSTr: Real-time Lowerbody Pose Prediction from Sparse Upper-body Tracking Signals[END_REF] estimated lower-body pose and foot contacts from upper-body tracking devices. Harvey et al. [START_REF] Gingras | Robust Motion In-betweening[END_REF] proposed a motion in-betweening method where pose and contact labels are interpolated between their known values at user-specified keyframes. Likewise, Holden et al. [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF] and Starke et al. [START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF] include foot contact detection at the next frame in character control frameworks, which enables foot contact continuation. Min and Chai [START_REF] Min | Motion graphs++: a compact generative model for semantic motion analysis and synthesis[END_REF] modelled foot contacts into a graph-based framework called Motion Graphs++, by annotating individual motion primitives with embedded contact information. Then, this framework is able to randomly synthesise motion along with corresponding contacts at runtime.

Retargeting

Retargeting [START_REF] Gleicher | Retargetting Motion to New Characters[END_REF] refers to the task of transferring the movements of a source character in a skeletal animation sequence to a target character with a different skeleton, i.e. different bone lengths (morphology) and possibly different joints or connections between joints (topology). As pointed out by Aberman et al. [3], there is no formal specification of the task. The purpose at large is to abstract out the dynamics of the source sequence and to reproduce it on a character whose body differ. Effectively, the sought goal is to create retargeted sequences with new morphologies and topologies whose motion mimics the source while remaining visually plausible and natural. Since it is difficult in practice to obtain ground-truth pairs of (source, target) sequences with exactly the same motion, most deep learning approaches to retargeting, and all the schemes surveyed in this section, rely on unpaired training data without motion correspondences across characters.

In the seminal work of Villegas et al. [START_REF] Villegas | Neural Kinematic Networks for Unsupervised Motion Retargetting[END_REF], the retargeting network is built around two RNNs. An encoder RNN captures the motion context of the source sequence in its hidden state and forwards it to a decoder RNN that outputs each frame of the retargeted sequence. A reference pose of the target skeleton is provided to the decoder. Besides regularisation loss terms, network training is driven by an adversarial loss and a cycle consistency loss. The adversarial loss attempts to minimise discrepancies between the joint velocities of ground truth (true) and retargeted (fake) sequences. The cycle consistency loss ensures that a source motion sequence that is retargeted to a target character and then retargeted back to its original character remains as close as possible to the original.

Lim et al. [START_REF] Lim | PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting[END_REF] and Kim et al. [START_REF] Kim | Motion Retargetting based on Dilated Convolutions and Skeleton-specific Loss Functions[END_REF] reported that the above approach tends to generate unrealistic motion. To mitigate this issue, Lim et al. [START_REF] Lim | PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting[END_REF] retargeted the motion of the root joint separately from the poses at each time step, and combined the results to construct the output sequence. The retargeted poses are computed as joint angles, represented as quaternions, to be applied to a reference pose of the target skeleton.

The cycle consistency loss of Villegas et al. [START_REF] Villegas | Neural Kinematic Networks for Unsupervised Motion Retargetting[END_REF] is replaced by a reconstruction loss associated to self-retargeting to the same character. Li et al. [START_REF] Li | An Iterative Solution for Improving the Generalization Ability of Unsupervised Skeleton Motion Retargeting[END_REF] then proposed a similar approach without adversarial loss but using an iterative solution: at runtime, the retargeted motion is iteratively optimised such that source and retargeted motions have the same end-effectors velocities (normalised by kinematic chain lengths). Moreover, a smoothness term between adjacent frames is also used in this iterative optimisation scheme. Kim et al. [START_REF] Kim | Motion Retargetting based on Dilated Convolutions and Skeleton-specific Loss Functions[END_REF] argued that CNNs are better suited than RNNs for retargeting because they can more accurately capture the short-term motion dependencies that mostly condition the performance of the task. Accordingly, their scheme relies on a purely convolutional network with temporally dilated convolutions that retargets whole motion sequences in one batch.

In the previous approaches, retargeting was done only across different morphologies, only supporting source and target characters with the same skeleton joints. Aberman et al. [3] extended the scope of retargeting to skeletons with different topologies, subject to the condition that source and target topologies are homeomorphic. This implies that they can all be reduced to a common primal skeleton by merging pairs of adjacent bones.

Their representation of skeletal motion separates the temporally invariant bone offset vectors that define the character morphology from the time-varying bone rotations at each joint that capture the motion dynamics (see Figure 2 skeleton topology prevents this approach to easily generalise to any homeomorphic skeleton topology.

Style Transfer

Style Transfer refers to algorithms manipulating data such as images, videos or human animations with DNNs to make the stylistic components look like another data sample. Gatys et al. [START_REF] Gatys | Image Style Transfer Using Convolutional Neural Networks[END_REF] first introduced a method to perform neural style transfer on images, using the Gram matrix of the deep features as the artistic style information of an image. In human animation, style transfer aims at transferring the style from one motion sequence to another whose content is retained, called hereafter style and content motion sequences, respectively. For example, we might want to edit a particular motion by affecting the state of mind of the character (e.g. enthusiastic, sad, angry)

while preserving the performed action, e.g. locomotion from point A to point B.

In the following, we split approaches to motion style transfer into two categories, supervised and unsupervised. Both categories are data-driven but the former uses motion sequences annotated with style labels while the latter does not. Human motion databases with style labels obviously facilitate the learning of style transfer frameworks; however they are rare and hence their usage limits the scope and capability to generalise of such frameworks.

Unsupervised

Holden et al. [START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF] pioneered human motion style transfer using their deterministic generation model (see Section 2.3.1). In this framework, different types of constraints can be applied on the generated motion, such as trajectories, bone lengths or joint positions, solving a constrained optimisation problem in a learnt human motion manifold.

Gradients are back-propagated through an autoencoder representing the manifold to optimise latent representations. Style transfer constitutes a particular case, where both joint positions and style are constrained with respect to the content and style motion sequences, respectively. Moreover, Holden et al. [START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF] followed prior work in image style transfer [START_REF] Gatys | Image Style Transfer Using Convolutional Neural Networks[END_REF] by using the Gram matrix in the latent representation as a style similarity measure. They further improved this approach by training a feedforward network to perform style transfer thousands of times faster [START_REF] Holden | Fast Neural Style Transfer for Motion Data[END_REF]. Gradients are back-propagated to train the feedforward network instead of optimising the latent representation. More recently Jang et al. [START_REF] Jang | Motion Puzzle: Arbitrary Motion Style Transfer by Body Part[END_REF] proposed to break down style information into local per-bodypart style components to enable local style editing and increasing the range of stylised motions. Spatial-temporal GCNs are used into a framework that can be learnt from a motion dataset without labels or paired motion sequences. To do so, four objectives are used: a reconstruction loss on unchanged styles, a cycle consistency loss to preserve the style-invariant features, a third loss to preserve the linear and angular velocities of the root joint, and a smoothness term between adjacent frames.

Supervised

Alternatively, style annotations can also be used to guide the learning of a style transfer model. Smith et al. [START_REF] Harrison | Efficient Neural Networks for Real-time Motion Style Transfer[END_REF] divided the task of style transfer into spatial and temporal style variations networks, both taking as inputs joint positions as well as a onehot style vector, where the networks are applied consecutively to predict corresponding style variations. However, this approach requires motion data registered with similar poses in different styles. Wang Beyond general-purpose style transfer, a few particular cases have been explored.

Kuriyama et al. [START_REF] Kuriyama | Context-based Style Transfer of Tokenized Gestures[END_REF] investigated the topic of style transfer of expressive or exaggerated upper-body gestures. To this end, a transformer-based autoencoder is leveraged to first embed both content and style gestures into its latent space, where another transformer network called style transformer swaps content and style gesture tokens. Finally, Dong et al. [START_REF] Dong | Adult2child: Motion Style Transfer using CycleGANs[END_REF] focused on translation from adult to child motions and vice versa. In this particular case, retargeting is not sufficient since it does not capture the natural stylistic differences between adults and children [START_REF] Dong | Adult2child: Motion Style Transfer using CycleGANs[END_REF]. This work leverages the capacity of CycleGAN [START_REF] Zhu | Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks[END_REF] to learn the mapping between adult and child motion distributions without paired training data, which is critical due to the very limited availability of child data.

Whereas most of the aforementioned approaches rely on style labels, Victor et al. [START_REF] Victor | Pose Metrics: A New Paradigm for Character Motion Edition[END_REF] recently pointed out that style is still loosely defined and arbitrary categories are too subjective, arguing that style labels are to high-level. As a compromise, they introduced the idea of objective intermediate-level descriptors (e.g. openness of the shoulders), referred to as pose metrics, computed single poses and intended to be provided by the user. Then metric networks operating into the latent space of a pre-trained pose autoencoder learn to edit pose metrics (one metric per network) given input and target training pairs.

conclusion

As presented throughout this chapter, skeletal character animation is a wide topic consisting of a lot of different but closely related problems. As an example, retargeting and style transfer have clearly distinct purposes, while simultaneously being both interested in extracting and transferring high-level semantic information (decoupling the motion from the skeleton and the style from the motion, respectively).

A lot of promising methods have been proposed to create new motions and edit existing motions. Even though deep learning has brought a diversity of novel tools for character animation, current solutions often suffer from artefacts, such as footskate, which are typically mitigated afterward through inverse kinematics for example. In addition, current deep data-driven models lack capacity to generalise, in part due to the difficulty of capturing high-quality and in-the-wild motion data, requiring an expensive motion capture system that not all research teams can afford. Also, large general-purpose motion databases are missing and there is no proper methodology to learn models from multiple smaller, specialised databases, as one can do in computer vision or natural language processing with image and text data, respectively. Indeed, most human motion databases have different skeleton structures and there is currently no satisfactory way to unify them. For these reasons, deep models struggle to integrate workflows in animation studios which still mostly rely on traditional animation approaches.

The goal of this thesis is then to propose effective approaches to enhance workflows based on deep data-driven models and foster their use in animation studios. In particular, we attempt to ease and enhance acquisition, unification and processing of human motion data, three critical aspects to improve state-of-the-art methods as well as to enlarge interoperability of animation systems, pipelines and databases.

J U M P S : J O I N T S U P S A M P

L I N G M E T H O D F O R P O S E S E Q U E N C E S 3

Chapter abstract

Human pose estimation is a low-level task useful to easily collect pose and motion data with little equipment compared to motion capture systems, promising for animation of synthetic characters. It also offers interesting perspectives for human action recognition, scene understanding at large, and surveillance. For these applications, and especially the former, accurately estimating the position of many joints from images is desirable for larger high-quality in-the-wild human motion data volumes. In this chapter we present a novel method called JUMPS for increasing the number of joints in 2D pose estimates and recovering occluded or missing joints. We believe this is the first attempt to address the issue. We build on a deep generative model that combines a generative adversarial network to learn the distribution of high-resolution human pose sequences with an encoder that maps the input low-resolution sequences to its latent space. Joints completion and upsampling are then obtained by computing the latent representation whose decoding by the generator optimally matches known joints. Post-processing a 2D pose sequence using our method enhances and enriches representations of the character motion which reduces ambiguity and eases inverse mapping from 2D to 3D, commonly done in 3D pose estimation. We show experimentally that the localisation accuracy of the additional joints is on average on par with the original pose estimates. Our approach completes and upsamples joints in a spatially and temporally consistent way to enrich estimated pose sequences, and yields the corresponding green 28-joint skeletons (bottom). In both images, the background has been stitched from a video with a smaller field of view and five images of the jumper at different frames have been overlaid as foreground.

introduction

Human pose estimation refers to the problem of estimating joint positions (either in 2D or 3D) of one or more people in an image or video. In addition, it is appealing in the context of skeletal character animation as an alternative to acquire in-the-wild motion data with little equipment in opposition to traditional motion capture systems which are cumbersome and expensive. It has been a topic of active research for several decades, and all state-of-the-art solutions rely on deep learning, e.g. [START_REF] Artacho | UniPose: Unified Human Pose Estimation in Single Images and Videos[END_REF][START_REF] Cao | OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields[END_REF][START_REF] Sun | Deep High-Resolution Representation Learning for Human Pose Estimation[END_REF][START_REF] Zhang | Distribution-Aware Coordinate Representation for Human Pose Estimation[END_REF].

Even then, the best approaches extract skeletons with a limited number of joints, usually from 12 to 16, which is too rough for the movie industry or video games applications.

This issue concerns both the 2D and 3D cases since 3D joint extraction often relies on 2D pose estimation. Moreover, in the presence of strong foreshortening such left-right ambiguities, (self-)occlusions, or on previously unseen complex poses, both the 2D pose estimation and the mapping from 2D to 3D are severely underconstrained.

In this chapter we present a novel approach to enhance existing state-of-the-art human pose estimation solutions by upsampling human joints and completing occluded ones, thereby paving the way for downstream applications that require complete poses and higher joint resolutions. Starting with a temporal sequence of partially occluded poses, we recover missing joint locations and improve the resolution of the skeleton animation by estimating the position of additional joints. To the best of our knowledge, no work has been previously proposed to recover missing joints and increase joint resolutions of animated skeletons in 2D. We believe that enriching the representation provide a better understanding of the motion which helps in many cases, especially for extremities such as feet and hands. For instance, extracting the toe in addition to the ankle provides an finer sense of feet contacts.

To this purpose, we draw inspiration from past research on human pose estimation, human motion modelling and image inpainting: we leverage a deep generative model that provides an effective spatio-temporal prior on human motion. Our model builds on a GAN, which we complement by a front-end encoder to learn a mapping from the space of human motions to the GAN latent space. The encoder helps selecting better samples in latent space and stabilises the training of the GAN. At runtime, we leverage our deep generative model to complete and upsample joints through an optimisation in the latent space of our model to best match a 2D pose sequence to be enhanced, typically estimated from video beforehand.

related work

In this section, we overview works related to our approach, i.e. recent approaches for image inpainting in computer vision from which we draw inspiration and then methods to interpolate and extrapolate motion sequences in the context of character animation which are related to the joints completion component of the problem we address.

Image Inpainting

Deep generative models have demonstrated impressive performance on image inpainting [START_REF] Bao | CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training[END_REF][START_REF] Iizuka | Globally and Locally Consistent Image Completion[END_REF][START_REF] Yeh | Semantic Image Inpainting with Deep Generative Models[END_REF][START_REF] Yu | Generative Image Inpainting with Contextual Attention[END_REF]. For this task, the need to faithfully reproduce the visible surroundings of missing image regions adds an additional constraint to the generative synthesis process. Yeh et al. [START_REF] Yeh | Semantic Image Inpainting with Deep Generative Models[END_REF] rely on a GAN [START_REF] Goodfellow | Generative Adversarial Nets[END_REF] whose generator is fed with noise samples from a known prior distribution. In the inference stage, the closest noise sample to the corrupted image is obtained by backpropagating the gradients of the generator network, and fed to the generator to obtain the inpainted image. In their seminal paper, Pathak et al. [START_REF] Pathak | Context Encoders: Feature Learning by Inpainting[END_REF] take a different approach that combines a GAN and a context encoder. The latter network is trained to reconstruct a full image from an input with missing parts. It acts as the GAN generator, while the discriminator enforces the plausibility of the inpainting result. Iizuka et al. [START_REF] Iizuka | Globally and Locally Consistent Image Completion[END_REF] enrich this architecture with two discriminators that separately process small-scale and large-scale image textures.

Yu et al. [START_REF] Yu | Generative Image Inpainting with Contextual Attention[END_REF] further add a self-attention module to better take advantage of distant image patches to fill the missing regions. Bao et al. [START_REF] Bao | CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training[END_REF] replace the context encoder with a VAE [START_REF] Diederik | Auto-Encoding Variational Bayes[END_REF] and incorporate an image classifier to specialise the generative process to sub-categories.

Motion Prediction and In-Betweening

As seen in Chapter 2, motion in-betweening refers to the process of temporally interpolating motion between input keyframes that can be manually edited by an artist.

By definition, in-betweening completes pose sequences in the temporal domain and usually operates on 3D motion data [START_REF] Habibie | A Recurrent Variational Autoencoder for Human Motion Synthesis[END_REF][START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Qin | Motion In-Betweening via Two-Stage Transformers[END_REF]. More closely related to our work is the approach of Hernandez Ruiz et al. [START_REF] Hernandez Ruiz | Human Motion Prediction via Spatio-Temporal Inpainting[END_REF], addressing motion prediction through spatiotemporal inpainting. They combine an ERD [START_REF] Fragkiadaki | Recurrent Network Models for Human Dynamics[END_REF] with a GAN whose generator operates in the latent space of the encoder. Their scheme can both temporally interpolate motion frames and recover missing joints within a pose. Unlike these approaches that are fed with 3D motion data, our scheme takes as input 2D joint locations that could typically be first estimated from video using a 2D human pose estimation framework. We believe 2D pose inpainting is more challenging: although depth is missing, it needs to be accounted for by the inpainting process in order to obtain 2D joint location estimates that are consistent with the motion of the character in 3D space.

method

In this section we describe the method we propose to upsample and complete a 2D pose sequence, typically previously estimated from video, to infer the location of missing or unseen joints and provide a higher-resolution representation of the body.

To this purpose we leverage a deep generative model that we train on pose sequences, rather than static poses, in order to lower ambiguities with spatio-temporal cues. data denote motion data distributions at with higher and lower number of joints, respectively. The latter is sampled by discarding predefined joints from motion sequences sampled from the former.

GAN

Network Architecture

As illustrated in Figure 3.2, our model consists of a GAN coupled with an additional encoder intended to map pose sequences with lower number of joints in the latent jumps space of the GAN which represents pose sequences with higher number of joints. Hence, this model benefits from the generative power of GANs and mitigates instability during training by introducing supervision from the encoder.

Our network conforms to the architecture of deep convolutional GANs (DCGANs) [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF], using fractionally-strided transposed convolutions in the generator and strided convolutions in the discriminator (see detailed architecture in Section 3.4.1). Moreover, DCGANs use rectified linear units (ReLUs) in the generator and leaky rectified linear units (LReLUs) in the discriminator, and batch normalisations (BNs) are applied after almost each convolutional layer. Except for the size its first and last layers, our encoder follows the architecture of the discriminator. each entry holds four coordinates, i.e. the 2D coordinates for two joints. The symmetric joints (e.g. feet, knees, etc.) are paired to form a single entry, while the other joints (e.g. pelvis, thorax, etc.) are duplicated 1 to obtain consistent four-channel entries.

Data Representation and Notation

In the following, we note E, G and D the encoder, generator and discriminator networks, respectively. P (h) data is the GAN data distribution, whose samples are pose sequences picked from the training dataset. Then, P (l) data denotes the data distribution used to feed our encoder, which will encode pose sequences with lower number of joints into the latent space of our GAN which represents pose sequences with higher number of joints. It corresponds to P (h) data mapped by discarding predefined joints to obtain pose sequences with lower number of joints. Finally P z denotes the prior distribution and P X stands for the distribution of uniformly sampled points along straight lines between pairs of points sampled from the GAN data distribution P (h) data and the generator distribution, i.e. P z mapped through G, used to improve the training of Wasserstein GANs [START_REF] Gulrajani | Improved Training of Wasserstein GANs[END_REF].

Training Adversarial Loss

Traditionally, a GAN consists of a generator and a discriminator. The former is trained to produce realistic samples while the latter aims at distinguishing those from real samples, both competing against each other. The ability to generate realistic samples can be expressed more formally as the similarity between two probability distributions that are the data distribution and the distribution of samples produced by the generator. The original formulation of GANs [START_REF] Goodfellow | Generative Adversarial Nets[END_REF] measures the similarity with the Jensen-Shannon divergence. However, this divergence fails to provide meaningful values when the overlap between the two distributions is not significant which often makes GANs quickly diverge during training. Arjovsky et al. [START_REF] Arjovsky | Wasserstein Generative Adversarial Networks[END_REF] introduced Wasserstein generative adversarial networks (WGANs), showing that, under the hypothesis that the discriminator is 1-Lipschitz, the Jensen-Shannon divergence can be replaced by the 1. Both discriminator and encoder duplicate those joints while the generator produces duplicated joints in a first step and then outputs the average of the two versions. jumps Wasserstein distance that have better properties for convergence. Then, Gulrajani et al. [START_REF] Gulrajani | Improved Training of Wasserstein GANs[END_REF] proposed a gradient penalty term in the WGAN loss function to enforce the 1-Lipschitz hypothesis on the discriminator. Therefore, we opt for the gradient-penalised WGAN and have the following loss functions for the generator and the discriminator, respectively:

L G = -D G(z) (3.1) L D = D G(z) -D(X) + λ gp ∥∇ X D( X)∥ 2 -1 2 (3.2)
where λ gp is the gradient penalty coefficient, z is a latent code sampled from the pior distribution P z , and both X and X are pose sequences sampled from P X and P (h) data , respectively.

Reconstruction Losses

As previously mentioned, we complement the GAN of our model with an encoder.

Then we encourage our model to reconstruct inputs that are encoded and then decoded through the generator with a (forward) reconstruction loss L f orward rec minimising differences between input and output like in autoencoders. We also incite our model to be consistent when generating samples from the prior distribution and then encoding them back into the latent space with a backward reconstruction loss L backward rec , as in cycle-consistent VAEs [START_REF] Harsh | Disentangling Factors of Variation with Cycle-Consistent Variational Auto-encoders[END_REF]. Such backward reconstruction loss facilitates the convergence but more importantly it enforces encoder outputs to follow the prior distribution P z imposed on our GAN. As illustrated in Figure 3.4b, our reconstruction loss is then

L rec = L f orward rec + L backward rec (3.3)
where L f orward rec is itself made up of two terms, respectively minimising the divergences of position and velocity of the reconstructed pose sequence X = G(E(X)). More formally, we use the MPJPE [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF] to quantify the positional divergence between two pose sequences X and Y:

MPJPE(X, Y) = 1 F 1 J F ∑ f =1 J ∑ j=1 ∥X f ,j -Y f ,j ∥ 2 (3.4)
where F and J denotes the number of frames and joints, respectively. In analogy with the MPJPE, we define the mean per joint velocity error (MPJVE) to quantify the velocity divergence between X and Y as:

MPJVE(X, Y) = 1 F 1 J F ∑ f =1 J ∑ j=1 ∥v(X f ,j ) -v(Y f ,j )∥ 2 (3.5)
where v(•) denotes the velocity of a joint, which is in practice computed via numerical differentiation. This secondary term acts as a powerful regulariser that speeds up the convergence in early iterations and mitigates temporal jitter in generated pose sequences. Finally, L f orward rec is the weighted sum of MPJPE and MPJVE:

L f orward rec = λ p • MPJPE X, G E(X) + λ v • MPJVE X, G E(X) (3.6)
where λ p and λ v are hyperparameters and X a pose sequence sampled from P

L backward rec = MSE z, E G(z) (3.7)

Mixed Loss

We further encourage the generation of realistic sequences with an additional loss term to penalise unrealistic reconstructed pose sequences. Here we make use of the discriminator to tell both the generator and the encoder whether the reconstructed pose sequence X = G(E(X)) is realistic or not. We use the same formulation as for the generator adversarial loss (see Section 3.3.3) but applied to X instead of G(z):

D G X ∼P data (l ) z∼P z G (z ) L D D (G ( z)) D ( X ) (a) Flow during discriminator's training step. D L rec ( forward ) L rec (backward ) X ∼P data (l ) L G L mix G E z∼P z E ( X ) E (G( z)) G(z ) G(E ( X )) (
L mix = -λ m • D G E(X) (3.8)
where λ m is an hyperparameter to balance L mix with respect to L G and L D , and

X ∼ P (l)
data .

Optimisation

In summary, the encoder, the generator and the discriminator are trained by minimising the loss functions L rec + L mix , L G + L rec + L mix and L D , respectively. Similarly to a GAN, we optimise at each iteration the discriminator in a first step and then both the generator and the encoder. Figure 3.4 illustrates the computational flows through the network during both training steps.

Temporal Variance Regularisation

GANs are known to produce sharp samples, but for the considered task this can lead to perceptually disturbing temporal jitter in the output pose sequences. To optimise the trade-off between sharpness and temporal consistency, we feed the discriminator with stacked joint positions and velocities (computed through numerical differentiation).

The velocities favour the rejection of generated samples that are either temporally too smooth or too sharp. This idea is inspired from Karras et al. [START_REF] Karras | Progressive Growing of GANs for Improved Quality, Stability, and Variation[END_REF], where the variation of generated samples is regulated by concatenating mini-batch standard deviations at some point of the discriminator.

Inference

At runtime, we leverage the generative model learnt to recover missing joints in an incomplete pose sequence X and upsample joints to increase the level of details. Given an input pose sequence X, we iteratively optimise the latent code z initialised to E(X) by backpropagating gradients through our generator to minimise the divergence of G(z) from X over available joints. In addition to this contextual loss term, we add a prior term which maximises the discriminator score on the generated pose sequence. This process is closely related to the semantic image inpainting approach proposed by Yeh et al. [START_REF] Yeh | Semantic Image Inpainting with Deep Generative Models[END_REF]. Formally, we solve

z * = arg min z L Inp (3.9)
where L Inp is our inpainting objective function composed of a contextual and a prior term. The contextual term minimises the weighted sum of MPJPE and MPJVE between input X and output X = G(z), while the prior term maximises the plausibility of the output by maximising the discriminator score D( X):

L Inp = γ p • MPJPE X, X + γ v • MPJVE X, X contextual loss -γ d • D X
prior loss (3.10) where γ p , γ v and γ d are additional hyperparameters. Then, we finally generate X * = G(z * ) -which best reconstructs X with respect to L Inp -to get an enhanced version of input X where missing joints have been recovered and additional joints have been upsampled.

Moreover, we translate, scale and rotate X to best match X at each iteration of the gradient descent. Ordinary Procrustes analysis (OPA) is used to analytically find optimal translation, scaling and rotation, and hence has a low overhead while making the gradient descent convergence several times faster and improving results.

Finally, our network only supports fixed-length pose sequences. Here we describe a simple yet effective mechanism to handle longer variable-length pose sequences. Given a pose sequence X longer than F frames, one can independently inpaint fixed-length sub-sequences of X and then concatenate the results into a single inpainted pose sequence having the same length as X. However, this approach does not guarantee that two consecutive sub-sequences will be concatenated smoothly. To prevent such discontinuities we use half overlapping sub-sequences and for each frame we select between the two overlapping inpainted frames the closest to the input, in the sense of the minimal contextual loss term in L Inp .

evaluation

In this section, we assess the proposed method. First, we provide necessary implementation details in Section 3.4.1. Then, we evaluate our method on pure joint upsampling in Section 3.4.2 and on 2D pose estimation enhancement in Section 3.4.3, i.e. joints completion and upsampling from pose sequences estimated from video.

Implementation Details

data.

To train and evaluate our method, we rely on public databases having the highest number of joints, i.e. 28 joints. Since our method focuses on fixed-length 2D pose sequences, we generated a set of around 835'000 2D pose sequences of F = 24 frames by randomly projecting original 3D pose sequences.

We also selected around 166'000 of images annotated with 2D poses directly from MPI-INF-3DHP with no preprocessing for testing.

architecture.

The architecture of our model is detailed in Figure 3.5. It has about 3 millions learnable parameters that are almost equally distributed over the encoder, the generator and the discriminator.

training.

Our implementation is written in Python and deeply relies on PyTorch.

Training and experiments have been executed on an NVidia Tesla P100 GPU. We trained our model for 60 epochs (about 11 hours) with a mini-batch size of 256 using Adam optimisation algorithm [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with learning rate α = 0.0001 and hyperparameters β 1 = 0.5, β 2 = 0.999 and ϵ = 10 -8 . We followed the suggestions for DCGAN from Radford et al. [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF] to reduce α and β 1 , with respect to Kingma and Ba [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF]. As Radford et al. [START_REF] Radford | Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[END_REF],

we observed that β 1 = 0.5 helped to stabilise the training. We set the Wasserstein gradient penalty weight to λ gp = 10 as proposed by Gulrajani et al. [START_REF] Gulrajani | Improved Training of Wasserstein GANs[END_REF], and our loss weights as follows: λ p = 200, λ v = 100, λ z = 2 and λ m = 1. We empirically found these values to work well. jumps inference.

We optimise the latent code again using Adam algorithm with α = 1, β 1 = 0.8, β 2 = 0.999 and ϵ = 10 -8 . The weights of the inpainting loss terms are set to γ p = 10, γ v = 5 and γ d = 15. We run the optimisation for 200 iterations. These hyperparameter values have been chosen to enforce the convergence of the optimisation in a limited number of iterations and avoid matching noise or imperfections in inputs.

These hyperparameter values have been chosen to enforce the convergence of the optimisation in a limited number of iterations and hence avoid to reproduce input noise and imperfections. Finally, to improve inference results we perform several optimisations of the latent code in parallel for a single input, starting from different initialisations. One of these starting points is computed as the output of the encoder fed by the input pose sequence, the others are randomly sampled from the prior distribution. We keep the one closest to the input, in the sense of the inpainting loss

L Inp .
ablative study.

We perform our evaluation on different variants of our method in order to show the relevance of the different components of our method. In the first variant considered, Ours w/o OPA, we disable the OPA alignment mechanism during inference (see Section 3.3.4) to measure its impact. Then, we trained from scratch our model without its encoder (Ours w/o ENC.), falling back to a pure GAN. Finally, we evaluate our method without using our overlapping sub-sequences mechanism described in Section 3.3.4 (Ours w/o overlap).

human pose estimation.

In the second part of our evaluation (see Section 3.4.3), we apply our method on real-world pose sequence estimates. We rely on AlphaPose 2 to perform 2D pose estimation from videos picked from our test set. AlphaPose provides 12-joint estimates that are then enhanced using our method, recovering missing (e.g. occluded) joints and upsampling to 28 joints.

metrics.

Finally, we report our results in the following with the percentage of correct key points normalised with head size (PCKh) [START_REF] Andriluka | 2D Human Pose Estimation: New Benchmark and State of the Art Analysis[END_REF] and the area under the 2. Implementation based on [START_REF] Fang | RMPE: Regional Multi-Person Pose Estimation[END_REF][START_REF] Li | CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark[END_REF][START_REF] Xiu | Pose Flow: Efficient Online Pose Tracking[END_REF] available at https://github.com/MVIG-SJTU/AlphaPose curve (AUC) [START_REF] Insafutdinov | DeeperCut: A Deeper, Stronger, and Faster Multiperson Pose Estimation Model[END_REF] metrics. The PCKh consider a joint as correct if its distance to the ground truth normalised by head size is less than a fixed threshold and the AUC aggregates the PCKh over an entire range of thresholds. We refer to the PCKh with threshold α with PCKh@α and we compute the AUC over the range [0, 1] of thresholds.

Joints Upsampling

Our first experiment focuses on pure joint upsampling. To do so, we purposely downsample ground truth pose sequences from 28 joints to the 12 joints that are common to our test set and AlphaPose outputs (see Figure 3.7 left), upsample them back to 28 joints using our method, and compare the result to the ground truth. Table 3.1

provides PCKh and AUC obtained by our method and its variants.

Assuming a typical human head size, the main variant of our method achieves a positioning error lower than 2.25 cm for half of the upsampled joints (i.e. PCKh threshold of 0.1) and lower than 11.25 cm for 95% of them (i.e. PCKh threshold of 0.5).

Removing the encoder in front of the GAN in our architecture (Ours w/o ENC.) substantially degrades performance. Indeed, the encoder regularises the generative process and improves the initialisation of the latent code at inference time, yielding sequences that better match available input joints. Then, Table 3.1 show that our OPA alignment mechanism is critical to the performance of our approach.

Table 3.1 -Quantitative evaluation of our method and its variants on pure joint upsampling. We purposely remove 16 of the 28 joints from ground truth pose sequences, apply our method to recover them and compare the result to the ground truth. We report the percentage of correct key points normalised with head size (PCKh) at different thresholds and corresponding area under the curve (AUC) over [0, observed that the per-frame joint positioning accuracy drops at the extremities of the processed chunks, probably because of the reduced temporal context information at the extremities. Without our overlapping sub-sequences mechanism, an increased temporal jitter is introduced at the chunk boundaries of the generated pose sequence, which is likely to incur perceptually disturbing artifacts e.g. when applying our method to character animation.

Human Pose Estimation

Our second experiment evaluates the performance of our method and its variants when applied on 2D human pose sequences estimated from videos using AlphaPose.

Table 3.2 summarises the results for this experiment. The first row, entitled AlphaPose, gives the amount of error in AlphaPose estimates. By comparing it to our main variant (Ours) we see that our method is able to upsample input pose sequences from 12 to 28 joints without significantly increasing joint position errors. Moreover, estimated joint positions with low confidence are sometimes corrected using our method, as illustrated in Figure 3.7. The results obtained by the other variants of our method tend to confirm the conclusions drawn in the previous section.

Table 3.2 -Quantitative evaluation of our method and its variants on joint completion and upsampling applied on 2D human pose estimated from videos using AlphaPose [START_REF] Xiu | Pose Flow: Efficient Online Pose Tracking[END_REF].

We report the percentage of correct key points normalised with head size (PCKh) at different thresholds and corresponding area under the curve (AUC) over [0, 1] (see Section 3.4.1). The first row reports errors observed in AlphaPose outputs before applying our method. The ablation studies confirm the conclusions drawn from our pure joint upsampling evaluation (see Table 3.1). Higher PCKh and AUC mean better performance. Model PCKh@0.1 PCKh@0.5 PCKh@1.0 AUC AlphaPose [START_REF] Xiu | Pose Flow: Efficient Online Pose Tracking[END_REF] 0 [START_REF] Xiu | Pose Flow: Efficient Online Pose Tracking[END_REF] but recovered by our method based on human motion priors. Note that these images have been intentionally whitened except for the area around the occlusion for clarity purposes.

conclusion

In this chapter we presented a novel method relying on a hybrid adversarial genera-

introduction

In this chapter, we propose a general approach to learn an abstract representation of motion that is agnostic to the skeleton topology and morphology. Possible applications are diverse and numerous since our representation is convenient to embed motion data from different sources. For example it could be beneficial to apply a number of data-driven methods on both scarce specialised motion datasets (e.g. with style or contact annotations) and larger general motion datasets for improved performance and generalisation ability. Specific applications include motion denoising and motion retargeting, by mapping in two steps given motion sequences into our representation, and then to the desired representation.

Motion retargeting is a long-standing problem in character animation. Specifically addressed by Gleicher [START_REF] Gleicher | Retargetting Motion to New Characters[END_REF] in 1998, it refers to the problem of transferring and adapting the motion of a source character to another target character. However, it has no formal specification since the goal is to abstract out motion dynamics across different characters, which makes motion retargeting difficult to address. Typical applications in animation pipelines include the creation of novel motion sequences by transferring onto new characters off-the-shelf motions. Motion retargeting could also be very interesting to make existing animation systems and motion databases interoperable. Indeed, they often differ in the structure of the skeleton(s) handled, which are difficult to swap.

Finally, finding effective models for motion retargeting might coincide with building expressive and powerful motion representations in which operations such as cleaning or editing are easier.

Motion retargeting has not been much explored compared to other topics in character animation such as generative synthesis of human motion. Nonetheless, a few approaches have been proposed in the last years, exclusively relying on deep learning.

In the general case, source and target characters considered in retargeting might differ in the length of their bones (morphological variations) or in which joints and bones they are composed of (topological variations). However, most existing methods only address morphological variations. As an exception, Aberman et al. [3] recently extended this scope to homeomorphic skeletons only and is limited to known skeleton topologies.

Moreover, there is no significant amount of pairs of source and retargeted motion sequences available. In consequence, researchers rely on unsupervised approaches and often additionally introduce various heuristics to disambiguate the learning phase of their model.

To this end, we present in this chapter a novel approach to learn an abstract deep Motion retargeting can then be performed through encoding conditioned on the source character followed by decoding conditioned on the target character. Moreover, as our decoder has learnt to produce clean motion sequences, motion denoising can be performed through self-retargeting (retargeting with identical source and target characters) of noised motion sequences.

related work

In the following, we give an overview of existing works related to our transformerbased approach for motion retargeting.

Motion Retargeting

Motion retargeting at large might be addressed within different representations of human dynamics. For instance, Aberman et al. [2] proposed a 2D approach based on visual features where inputs and outputs are image sequences. Other variants include the retargeting of body shapes which considers the dynamics of body surfaces, and is notably addressed in parallel to [START_REF] Villegas | Contact-Aware Retargeting of Skinned Motion[END_REF] or on top of [START_REF] Marsot | Correspondence-free online human motion retargeting[END_REF] skeletal retargeting. Our work is specifically focused on the latter, i.e. 3D skeletal motion retargeting, which considers motion dynamics of character skeletons as commonly done in character animation.

Before Gleicher [START_REF] Gleicher | Retargetting Motion to New Characters[END_REF], techniques specifically addressing skeletal motion retargeting have been little explored. In his work, Gleicher [START_REF] Gleicher | Retargetting Motion to New Characters[END_REF] The temporal horizon of CNNs is strictly limited by the size of their receptive field, itself determined by the number of convolutional layers and the size of their kernel.

GCNs suffer from the same limitations as CNNs, but might better model spatio-temporal dependencies. To the best of our knowledge, no transformer-based architecture has yet been explored in the context of motion retargeting, even though it is very promising especially for modelling long-term temporal dependencies (see Section 2.2.4).

Beyond network architectures, a wide diversity of training losses have been explored

without any consensus emerging. They can be sorted into three categories:

-unsupervised losses, e.g. adversarial, reconstruction and cycle consistency losses;

-retargeting heuristics, e.g. to reproduce end-effector velocities or to bound joint twists;

-regularisation losses, e.g. to foster smoothness or quaternion validity.

Losses in the first category shape the model learnt while the ones in the second increase supervision, translating the difficulty to learn such models from unpaired motion sequences. Losses in the third category are typically used to ease model training and guarantee some properties. For a detailed review of methods cited beforehand, please refer to Section 2.4.2.

In contrast with existing methods, ours gets rid of retargeting heuristics, as we only rely on a reconstruction loss to learn a deep motion representation and a temporal bone lengths consistency loss to enforce constant bone lengths over time, since we use a positional pose representation. Moreover, we leverage transformers which are a promising alternative to better model distant spatio-temporal correlations. Last but not least, our learnt deep motion representation is unified across skeleton morphologies and topologies, pushing the scope of retargeting beyond homeomorphic topologies, and generalises well to novel topologies unseen during training.

Transformers

A transformer is a scheme of neural network architecture adopting the mechanism of self-attention, in which the network itself finds out which part of input data it should pay more attention to (see Section 2.2.4 for an introduction to transformers).

First introduced in natural language processing, transformers are getting more and more attention in all topics leveraging deep learning. For instance in computer vision, transformers have been leveraged in specialised architectures for images, such as vision transformers (ViTs) [START_REF] Dosovitskiy | An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale[END_REF], and tend to replace CNNs as self-attention layers with sufficient number of heads are at least as expressive as any convolutional layer [START_REF] Cordonnier | On the Relationship between Self-Attention and Convolutional Layers[END_REF]. In animation, they have been explored for 3D facial animation [START_REF] Chandran | Facial Animation with Disentangled Identity and Motion using Transformers[END_REF] for instance. In skeletal character animation in particular, solutions to style transfer [START_REF] Kuriyama | Context-based Style Transfer of Tokenized Gestures[END_REF] and motion in-betweening [START_REF] Duan | Single-Shot Motion Completion with Transformer[END_REF][START_REF] Oreshkin | Motion In-betweening via Deep Delta-Interpolator[END_REF][START_REF] Qin | Motion In-Betweening via Two-Stage Transformers[END_REF] have also been built around transformer architectures.

In our work, we draw inspiration from the architecture proposed by Chandran et al. [START_REF] Chandran | Shape Transformers: Topology-Independent 3D Shape Models Using Transformers[END_REF] for 3D shape modelling, who relied on a transformer-based autoencoder to model topology-independent static 3D shapes such as human faces, hand shapes or full human body shapes. Later on, Chandran et al. [START_REF] Chandran | Facial Animation with Disentangled Identity and Motion using Transformers[END_REF] extended their method to include the time dimension for facial animation, where static facial identity and dynamics are separately encoded such that the model learns a disentangled representation.

On our side, we instead adapt our transformer-based architecture to process spatiotemporal sequences of joint positions. To this end, we introduce point-wise 1D temporal convolutions at different places to encode, process and extract human joint trajectories (see next section). During training, our model is guided by the reconstruction loss L rec as well as the temporal bone lengths consistency loss L rbl (see Section 4.3.4). At inference, applications of our model include motion retargeting that can be performed by setting the decoding template to the target character skeleton, as well as motion cleaning by encoding and decoding with the same template as our decoder has learnt to produce clean motion sequences.

Deep Motion Representation

method

In this section, we describe our methodology to build a deep motion representation unified across skeleton topologies and morphologies. Our goal is twofold: first, we need an architecture flexible enough to encode variable size inputs to latent variables lying in our deep representation and then to decode them to variable size outputs. In particular, inputs and outputs are motion sequences with variable number of frames and joints. Second, we not only want our deep representation to be able to encode motion sequences with variable numbers of joints, but also to be shared across skeleton topologies and morphologies. In other words, we want similar motions to be represented by close latent codes in our deep representation, even when skeleton topologies and morphologies differ. To this end, we propose a model relying on an autoencoder (see Figure 4.1) in which both encoder and decoder have transformer-based architectures and are conditioned on skeleton templates (see below), representing morphological and topological variations. Our deep motion representation then corresponds to the latent space of our autoencoder and is supposed to encode motion free from skeleton topology and morphology. Hence, we make the hypothesis that "pure" motion features can be reasonably separated from morphological and topological features.

After a brief overview of data representations and notations, we present in detail the proposed model in the following: we begin with the description of our conditioning mechanism used to control and abstract skeleton topology and morphology, followed by the proposed associated architecture. Finally, we present the procedure used to train our model and how to exploit it at inference.

Notation and Representation

Our deep representation operates on variable-length motion sequences. We represent them by the position of the joints at each frame, expressed in the global Euclidean space.

We note X (a) ∈ R J (a)×3×F a motion sequence of F frames "performed" by the skeleton a

Template Conditioning

Our neural network is an autoencoder made up of an encoder E and a decoder D, with our deep motion representation corresponding to the latent space lying in between.

A first consequence of our goal to abstract out human motion in the latent space independently of skeleton topology and morphology is the following: no information about topology and morphology should be encoded into our latent space. Then, the decoding of such skeleton-agnostic latent codes must be conditioned on the desired topology and morphology of the output motion sequence. To this end, we introduce skeleton templates as conditioning representatives of topological and morphological features. In the following, the short form "template" is equivalent to skeleton template if not stated otherwise. In our approach, a skeleton template is a static neutral pose, i.e. standing with arms along the body (sometimes referred to as N-pose in animation, in opposition to T-pose and A-pose), which we represent by the position of its joints expressed in the global Euclidean space. Furthermore, we normalise them such that the position of the pelvis is at the origin, the transverse 1 plane is parallel to the XY-plane and the coronal 2 plane is parallel to the XZ-plane. See Figure 4.2 for a typical skeleton template aligned with transverse and coronal planes. These skeleton templates explicitly encode topological features through the order and position of their joints while morphological features are implicitly encoded through the relations between joint positions which correspond to the bone lengths. These poses are required to be neutral for our network to be able to compare templates and to register motion sequences with respect to their respective skeleton templates.

Such static neutral poses are straightforward to obtain. Indeed, topological variations generally appear across animation systems or motion databases, while morphological variations are more common and appear between any different characters or persons.

Therefore, after manually predefining a single neutral pose per database or animation system considered, the skeleton template corresponding to a given motion sequence is obtained by scaling the bone lengths of the predefined neutral pose according to the bone lengths observed in the motion sequence. Moreover, the simplicity of neutral poses makes their manual editing straightforward.

Finally, in addition to our decoder, we condition our encoder on the very same skeleton templates which are expected to be consistent with input motion sequences to be encoded. Still, both encoder and decoder conditionings are independent and different templates might be used to encode and then decode a given motion sequence, e.g. to transfer a motion from one character to another, i.e. to perform retargeting. As we will see in the following, these templates additionally serve as spatial references when encoding or decoding motion sequences, in a similar way to positional encoding which is critical to transformers performance (see Section 2.2.4).

Network Architecture

Our model relies on an autoencoder network based on the recent transformer models and whose latent space constitutes the proposed unified deep motion representation.

In this section we describe its architecture in detail. For a general overview of what a transformer is and how it works, please refer to Section 2.2.4. Moreover, to reproduce our approach, please also see implementation details provided in Section 4.4.1.

Joint-wise MLP

Joint-wise CNN Transformer blocks

Replicate

Template

Motion

Latent code Hence, for each joint, motion features are registered to template features of the same joint (e.g. left wrist highlight in yellow), telling the network which piece of information is associated to which joint. Thereby, the skeleton template acts as a reference frame for motion features.

Positional Encoding

Inputs and outputs of our model are sequences of joint positions given in an arbitrary order in the spatial domain but in a consistent order across frames. As commonly done in transformer networks, we resort to positional encoding with sine and cosine functions of varying frequencies to give both encoder and decoder knowledge about each frame position in the sequence and hence exploit the temporal structure of motion sequences.

To further give information about the spatial structure of input sequences, we rely on another type of positional encoding which consists in concatenating reference features to input features. In our case, this corresponds to concatenating skeleton template features and joint position features, joint by joint, as illustrated in Figure 4.3. In other words, each joint position of an input motion sequence is registered with the same joint in input skeleton template, and hence this mechanism is closely related to our conditioning scheme and to the representation of skeleton templates. Previous work in 3D shape modelling has found this kind of positional encoding to be beneficial [START_REF] Chandran | Shape Transformers: Topology-Independent 3D Shape Models Using Transformers[END_REF][START_REF] Lin | End-to-End Human Pose and Mesh Reconstruction with Transformers[END_REF]. As previously stated, our model is expected to encode a given motion sequence X (a) conditioned on skeleton template T (a) to z = E(X (a) |T (a)) and might be asked to decode it conditioned on another template T (b) toward X(b) = D(z|T (b)). In such a scenario, both X (a) and X(b) will have the same number of frames, but might have a different number of joints. As a consequence, the whole process conforms to the sequence-to-sequence paradigm, as in original transformers, in the temporal dimension.

However, it is not true in the spatial domain as the number of joints is variable across inputs and outputs. This observation coupled with our goal to have latent codes free from skeletal features imposes spatially fixed-size latent codes, and that our encoder and decoder act as sequence-to-item and item-to-sequence functions, respectively, in the spatial domain. To address this issue, we follow Chandran et al. [START_REF] Chandran | Shape Transformers: Topology-Independent 3D Shape Models Using Transformers[END_REF] who use an additional learned token, called query token, which is appended to input sequences of tokens. First introduced in BERT [START_REF] Devlin | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding[END_REF], such additional sequence-level token, sometimes referred to as [CLS], is a common design pattern in transformer architectures. In our particular case, it is used as a fixed-size container for output features while other output tokens of the final transformer block are discarded. During training, the network is thus forced to progressively migrate necessary information toward the query token along transformer blocks.

We use a hybrid approach following Chandran et al. [START_REF] Chandran | Shape Transformers: Topology-Independent 3D Shape Models Using Transformers[END_REF] with the query token. We further temporally replicate it to match the number of frames of the input motion sequence to obtain latent codes with fixed spatial size but variable temporal size.

More formally, the query token is a tensor of shape 1 × C, where C is a number of intermediate features. Given an input motion sequence X (a) ∈ R J (a)×3×F , the sequence of tokens obtained after positional encoding will be a tensor of shape J (a) × C × F.

Then, the query token is replicated F times, producing a tensor of shape 1 × C × F and appended to the sequence of tokens, resulting in a tensor of shape (J (a) + 1) × C × F, which will be the actual input sequence to the transformer blocks. Finally, at the end of the final transformer block, the first J (a) slices are discarded to get a tensor of shape 1 × C × F which is further processed by a linear layer toward the latent code z ∈ R Z×F where Z is the spatial size of the latent space. See Figure 4.4 for a better understanding of how these different tensors are manipulated into our encoder. The result is used as input tokens to a stack of transformer blocks, as in our encoder.

However, these blocks differ from the ones used in our encoder to inject information from the latent code. In particular, we inject information derived from the latent code into each block rather than only as input to the first one. This scheme consisting in disseminating latent information at different points of the network, sometimes called style modulation, has proven to be effective when it comes to synthesising new samples, e.g. in the StyleGAN series [START_REF] Karras | A Style-Based Generator Architecture for Generative Adversarial Networks[END_REF][START_REF] Karras | Alias-Free Generative Adversarial Networks[END_REF][START_REF] Karras | Analyzing and Improving the Image Quality of Style-GAN[END_REF]. In our decoder, each style-modulated transformer block corresponds to the same transformer block as in our encoder with an additional style module which maps the latent code into a style code. The style code is then used to modulate input tokens of the transformer block through multiplication. Style modules are 1D temporal CNNs, well suited to process latent codes that are temporally structured (see previous section). Most of the first layers of style modules are shared across blocks (see Section 4.4.1).

Finally, the output of the last transformer block is further processed by a joint-wise 1D temporal CNN to get the final motion sequence.

Cross-Covariance Attention

Transformers typically transform a sequence of input tokens into a desired output sequence by modelling token-to-token interactions using self-attention [START_REF] Vaswani | Attention is All You Need[END_REF]. In this work, we consider motion sequences and model interactions between joint positions at any frames. However, one of the limitations of transformers is the time and memory complexity of self-attention layers which increase quadratically with the number of input tokens [START_REF] El-Nouby | XCiT: Cross-Covariance Image Transformers[END_REF]. In our model, we instead leverage cross-covariance attention (XCA), recently proposed by El-Nouby et al. [START_REF] El-Nouby | XCiT: Cross-Covariance Image Transformers[END_REF], which is a transposed version of self-attention with linear complexity with respect to the number of tokens. It operates across feature channels rather than tokens where interactions are based on the cross-covariance matrix between keys and queries (see Section 2.2.4). El-Nouby et al. [START_REF] El-Nouby | XCiT: Cross-Covariance Image Transformers[END_REF] further proposed a cross-covariance image transformer (XCiT) block adapted from original transformer block with three main changes: first, self-attention layer is replaced with XCA layer; second, a local interaction layer is added in-between attention and feed-forward layers; third, layer normalisation is applied before rather than after each layer of the block. The additional local interaction layer enables explicit communication across tokens (only implicit in XCA layer) and consists in two 2D convolutional layers. Even though our model operates on motion sequences rather than images, we build it on top of the same cross-covariance transformer blocks. Indeed, the only part that might be sensitive to the type of data processed is the local interaction layer made of 2D convolutional layers.

Since its goal is to make tokens interact locally, 2D convolutions are as much relevant to image patches as to spatio-temporal motion sequence patches.

Training

After describing the architecture of our network in the previous section, we present here the training procedure used to successfully learn the sought unified deep motion representation.

Data Preparation

Basically, the goal of our approach is to learn the structure of the human body from data instead of relying on a single predefined skeleton topology. To this end, we collected a large amount of motion data gathered from U nder P ressure (see Chapter 5), existing public databases (Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF], MPI-INF-3DHP [START_REF] Mehta | Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision[END_REF], AMASS [START_REF] Mahmood | AMASS: Archive of Motion Capture As Surface Shapes[END_REF] and PSU-TMM100 [START_REF] Scott | From Image to Stability: Learning Dynamics from Human Pose[END_REF], see Table 4.1) and internal motion data. In total we have about 65 hours of motion data, spanning across 5 different skeleton topologies (see Figure 4.6)

and 494 different morphologies. Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF], MPI-INF-3DHP [START_REF] Mehta | Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision[END_REF], AMASS [START_REF] Mahmood | AMASS: Archive of Motion Capture As Surface Shapes[END_REF], PSU-TMM100 [START_REF] Scott | From Image to Stability: Learning Dynamics from Human Pose[END_REF] and U nder P ressure [125]). The right-most column gives an insight about topological variations by overlaying the different topologies. Then, our data preprocessing consists in three steps:

1. We resample all motion sequences to a common framerate (i.e. 30 Hz) using spherical linear interpolation on motion sequences represented with angular pose representations and linear interpolation on motion sequences represented with positional pose representations (see Section 2.2.1).

2.

We convert all motion sequences to joint positions expressed in the same XYZ Euclidean space, by switching their axes, converting their coordinates to meters and/or applying forward kinematics (FK) on motion sequences represented with angular pose representations to get joint positions when needed.

3.

We extract 1-second overlapping chunks (i.e. 30-frame chunks overlapping over 24 frames, or 0.8 s) and normalise them separately by removing the mean position computed over a set of major joints (i.e. ankles, hips, knees, shoulders, elbows, wrists).

The resulting unified, augmented and normalised motion sequences constitute our dataset, for a total of about 300 hours of motion data.

Training Objective

In an typical autoencoder, inputs are fed to the encoder to produce latent codes which are then decoded, and the resulting differences between inputs and outputs (known as reconstruction loss) is minimised to train the network. We slightly change this scheme to further prevent our model from being tied to one or more particular skeleton topology, and rely on a stochastic joint subsampling of input and target motion sequences.

Moreover, we use temporal consistency loss over bone lengths. Both objectives are described in the following paragraphs.

reconstruction loss.

The idea of stochastic joint subsampling comes by thinking of motion sequences tied to their skeleton topologies as proxy representations of their underlying motion features. Within this context, we can discard a few joints from motion sequences and still consider their underlying motion features to be the same. Then, given an input motion sequence X (a) sampled from the training set, we first derive two sub-versions X (a 1 ) = S(X (a) ) and X (a 2 ) = S(X (a) ), where S(•) stands for stochastic joint subsampling and discards a random set of joints. Skeleton template T (a) is also subsampled consistently with X (a 1 ) and X (a 2 ) subsamplings to get templates T (a 1 ) = S(T (a)) and T (a 2 ) = S(T(a)), respectively. Hence, we consider these two sub-versions as different proxy representations of the same underlying motion features X. We encode X (a 1 ) and then decode the resulting latent code z = E(X (a 1 ) |T (a 1 ))

conditioned on a 2 to get X(a 2 ) = D(z|T (a 2 )). Finally, our reconstruction loss is imple- mented to minimise the deviation of X(a 2 ) with respect to X (a 2 ) . This mechanism is critical to the training of our model as it forces both encoder and decoder to strictly follow the conditioning template skeletons they are given to encode and decode motion sequences. Indeed, the variations in skeleton topologies seen by our network is much larger. Formally, we compute the deviation of X(a 2 ) with respect to X (a 2 ) as the mean squared position error:

L rec = 1 F 1 J (a 2 ) F ∑ f =1 J (a 2 ) ∑ j=1 ∥X (a 2 ) f ,j - X(a 2 ) f ,j ∥ 2 2 (4.1)
As described in Section 4.3.1 and consistently with the formulation of our reconstruction loss above, input and output motion sequences are represented by their joint positions. However, as explained in Section 2.2.1, positional pose representations do not constrain bone lengths to remain constant over time which is be problematic when synthesising motion sequences. A common approach to solve this issue is to add constraints to ensure that distance between pairs of adjacent joints, referred to as bones, are consistent [START_REF] Li | A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network[END_REF][START_REF] Li | Bidirectional recurrent autoencoder for 3D skeleton motion data refinement[END_REF].

temporal bone lengths consistency. Therefore, we rely on an additional temporal bone lengths consistency loss to train our model such that the motion sequences produced by the decoder have consistent bone lengths over time. It consists in minimising the temporal variance of bone lengths in the decoder output. However, bone lengths have large variations across the different bones and thus minimising absolute temporal variance of bone lengths might mostly fail to constrain short bones.

To mitigate this issue, we instead use the relative temporal variance of bone lengths obtained by normalising, per bone, output lengths with respect to the ground truth lengths before computing the temporal variance. In other words, we compute the temporal variance of output bone length ratios with respect to the true bone lengths.

Our temporal bone lengths consistency loss is then:

L rbl = 1 B(a 2 ) B(a 2 ) ∑ b=1 Var X(a 2 ) b X (a 2 ) b (4.2)
where function B(•) gives the number of bones of a given skeleton and X

b ∈ R F denotes the vector of lengths of the bth bone at each of the F frames of motion sequence X (a) .

In summary, we train our model with two objectives, a reconstruction loss and a temporal bone lengths consistency loss. Moreover, we rely on a stochastic joint subsampling mechanism within our reconstruction loss in order to augment topological and morphological variations seen by our model during training, and hence improve generalisation capability. This reduces the dependency of the network on specific joints of the input topology, in the same way as dropout reduces the dependency on specific nodes of a network. The total loss used to train our model is

L = L rec + λ rbl • L rbl ,
where λ rbl is a hyperparameter to balance both losses.

evaluation

In this section, we evaluate our approach. First, implementation details necessary for reproducibility are provided in the next sub-section. Second, we quantify the representation accuracy of our deep motion representation. Third, we further show that our deep motion representation is well structured through the evaluation of a side task, i.e. motion denoising. Finally, we evaluate our model on motion retargeting and demonstrate that it achieves state-of-the-art performance.

Implementation Details

data.

To train and evaluate our model, we split our dataset (see Section 4.3.4) into a training set and validation set. The latter is constituted of all chunks from MPI-INF-3DHP [START_REF] Mehta | Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision[END_REF], to allow validation on unseen skeleton topology, as well as about 10% of chunks from other sub-datasets for validation on known skeleton topologies.

This results in about 114'275 seen and 26'249 unseen 30-frames chunks reserved for evaluation.

skeleton templates.

As explained in Section 4.3.2, we obtain the skeleton template for a given motion sequence by scaling the bone lengths of the corresponding generic neutral pose. We manually predefined a generic neutral pose for each data source as illustrated in Figure 4.6. Then, given a motion sequence, we scale bone lengths of the corresponding predefined neutral pose to match the bone lengths observed in the given motion sequence. Since some of our sources of motion data use a positional pose representation, the bone lengths are not always constrained and sometimes vary a little over time. In consequence, we use the temporal median of bone lengths of a given motion sequence as observed bone lengths.

architecture.

As illustrated in Figure 4.4, our encoder begins with two joint-wise embedding modules: a 3-layer MLP is used to embed skeleton templates (16, 32 and convolutional layers (3, 16, 32 and 64 output features) followed by 3 linear layers (64 output features each). Each layer of both modules is followed by exponential linear unit (ELU). Then, the core of our encoder is made of As seen in Section 4.3.4, we apply our reconstruction loss on two sub-versions of a given sample motion sequence, in which we discard random sets of joints. In practice, we discard each joint independently with probability p = 0.1 for major joints (i.e. ankles, hips, knees, shoulders, elbows, wrists) that are common to all skeleton topologies of our dataset, and with probability p = 0.5 for other joints. Moreover, we compute our temporal bone lengths consistency loss on reconstructed subsampled motion sequences. Since some joints have been discarded, we compute this loss over bones whose both joints have not been discarded for increased computational efficiency. Otherwise, it is required to encode and decode the original sample motion sequence in addition to its subsampled version, which roughly doubles the computation time.

training.

Our implementation is written in Python and relies on PyTorch. Training was performed on an NVidia Ampere A100 GPU while other results were obtained either on an NVidia GeForce RTX 2060 GPU or on CPU. We trained our model for 10 7 iterations (about 385 epochs and 26.5 days) using Adam optimisation algorithm [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with mini-batch size of 32 and hyperparameters β 1 = 0.9, β 2 = 0.999. Moreover, we used a triangular cyclic learning rate [START_REF] Smith | Cyclical Learning Rates for Training Neural Networks[END_REF] with a period of 10 epochs and lower and upper bounds set at 10 -4 and 10 -5 , respectively. Finally we set the weight of our temporal bone lengths consistency loss to λ rbl = 0.5 (see Section 4.3.4). motion retargeting.

In the following, we notably perform a quantitative evaluation of the performance of our model on motion retargeting. This evaluation is performed on motion sequences drawn from Mixamo dataset (see Section 2.2.2), as commonly done for motion retargeting [3,[START_REF] Kim | Motion Retargetting based on Dilated Convolutions and Skeleton-specific Loss Functions[END_REF][START_REF] Li | An Iterative Solution for Improving the Generalization Ability of Unsupervised Skeleton Motion Retargeting[END_REF][START_REF] Lim | PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting[END_REF][START_REF] Villegas | Neural Kinematic Networks for Unsupervised Motion Retargetting[END_REF]. However, models from other approaches are usually trained on a subset of Mixamo while ours is not. To make the comparison more fair, we fine-tune our model on a few motion sequences from Mixamo before performing our motion retargeting evaluation. The procedure is the same as the training of the model, except that the amount of motion data is far smaller (about 1 hour vs 65 hours for training) and its duration is much shorter (less than 2 hours vs 26.5 days). We also used a constant learning rate set at 5 • 10 -5

instead of the cyclic learning rate. Finally, since the pelvises of skeleton templates are aligned (see Section 4.3.2), our model aligns the global trajectory of output motion sequences with respect to input motion sequences around the same region of the body. However, when performing motion retargeting on source and target characters of different scales, the resulting global trajectory might not be consistent with target character scale. To fix this issue, we further scale the global trajectory of output motion sequences when performing motion retargeting based on the leg length ratio of source and target characters, as typically performed in the literature [START_REF] Hecker | Real-Time Motion Retargeting to Highly Varied User-Created Morphologies[END_REF][START_REF] Kulpa | Morphology-independent representation of motions for interactive human-like animation[END_REF] ablative study.

Throughout our evaluation, we notably show that our joint subsampling mechanism (see Section 4.3.4) is critical to the performance of our method.

To this end, we perform a simple ablative study which considers a variant of our approach, called Ours -no SJS, in which the same neural network was trained without this mechanism.

Representation Accuracy

A key aspect of deep representations is their ability to accurately encode information, sometimes called the representation accuracy. In this section, we evaluate this aspect by quantifying the amount of distortion introduced by our deep motion representation when encoding and then decoding motion sequences drawn from our validation set (see previous section). We measure this amount of distortion using the mean per joint position error (MPJPE) of reconstructed motion sequences with respect to ground truth motion sequences given in input.

In Table 4.2, we provide the corresponding results for both our deep motion representation and its variant without stochastic joint subsampling and for each skeleton topology. Moreover, we separate the evaluation between skeleton topologies that have been seen during training from unseen skeleton topology, i.e. the one associated to MPI-INF-3DHP.

Table 4.2 -Quantitative evaluation of the representation accuracy of our deep motion representation. We measure the distortion introduced when encoding and then decoding ground truth motion sequences drawn from our validation set with the mean per joint position error (MPJPE) in centimetres. We distinguish skeleton topologies that have been seen during training from unseen topologies. Columns are associated to the different skeleton topologies in our validation set, named after the data sources used to constitute our dataset (see Section 4.3.4). Lower MPJPE means higher representation accuracy.

Model Seen Unseen

Human3.6M AMASS PSU-TMM100 U nder P ments such as leg swings when the character is running or jumping over the fence. Still, the MPJPE is of 1.33 cm on this example, which is slightly above global and topology averages (1.13 cm and 1 cm, respectively; see Table 4.2). Results are averaged over seen and unseen skeleton topologies. Flatter curves correspond to higher denoising ability.

Motion Denoising

In this section, we demonstrate that our model has also the ability to substantially clean noised motion sequences, which suggests that the underlying deep motion representation is well structured and prevents from encoding artefacts like noise. To this end, we pick clean motion sequences from our validation set and purposely add Gaussian noise on joint positions to obtain corresponding noised motion sequences.

Then we simply encode and then decode these noised motion sequences and compare them to original clean motion sequences. We follow this procedure for an entire range of noise levels, controlled by the standard deviation of offset coordinates (sampled from a centred Gaussian) that are added to coordinates of joint positions.

In Figure 4.8, we vertically plot the MPJPE of denoised motion sequences with respect to clean motion sequences against the level of noise introduced horizontally. The MPJPE is computed for all motion sequences from our validation set and averaged over seen and unseen skeleton topologies. Without noise (at x = 0 in Figure 4.8), we fall back on the representation errors found when evaluating the representation accuracy of our model (see Table 4.2). Then, positional errors observed in denoised motion sequences increase with noise but slowly compared to noise increase, indicating that our model reduces noise. The right-hand side of Figure 4.8 demonstrates the ability of our model to reduce noise as the output positional error is lower than input positional error observed in noised motion sequences.

Motion Retargeting

As pointed by Aberman et al. [3], the task of motion retargeting has no formal specification as its goal is to abstract out motion dynamics. Nevertheless, research in motion retargeting relied on the so-called Mixamo dataset (see Section 2.2.2) as a source of motion sequences performed by different characters, considered as ground truth for evaluation purposes [3,[START_REF] Kim | Motion Retargetting based on Dilated Convolutions and Skeleton-specific Loss Functions[END_REF][START_REF] Li | An Iterative Solution for Improving the Generalization Ability of Unsupervised Skeleton Motion Retargeting[END_REF][START_REF] Lim | PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting[END_REF][START_REF] Villegas | Neural Kinematic Networks for Unsupervised Motion Retargetting[END_REF].

In this section, we provide quantitative retargeting results following the evaluation procedure from Aberman et al. [3], as well as qualitative visual results. Relying on Mixamo, 106 test motion sequences are considered. The evaluation is divided in two modes, called intra-structural and cross-structural. The former corresponds to motion retargeting with the same skeleton topology but different body proportions, while the latter additionally considers source characters with topology different from the target.

Five characters are considered: BigVegas, Goblin, Mousey, Mremireh and Vampire. The first one has a different topology from others and is used as the source character to evaluate cross-structural retargeting. For intra-structural retargeting, Aberman et al. [3] considered all combinations of the four last characters as source and target characters.

However, according to their publicly released implementation 3 , these combinations are drawn with replacement, meaning that their evaluation of intra-structural retargeting includes self-retargeting, i.e. retargeting with same source and target characters. In the following, we further evaluate self-retargeting apart from intra-structural retargeting.

3. https://github.com/DeepMotionEditing/deep-motion-editing Finally, we measure the retargeting error with the MPJPE normalised by the height of the skeleton.

In the results provided by Aberman et al. [3], the retargeting error is averaged over target characters. However, we observed variations in the performance of our model depending on whether the characters involved have body proportions close to humans.

The reason here is that some of these humanoid characters have body proportions very different from human, such as a head about the size of the body, and that we exclusively trained our model on motion data captured from human subjects. As a result, our model is a bit less accurate on non-human morphologies. In consequence, we detail our evaluation per character to emphasise the performance and limitations of our model when body proportions are close to human or not. In particular, BigVegas, Goblin and Mremireh have body proportions close to humans, while Vampire and Mousey body proportions differ, especially for the latter.

self-retargeting. Table 4.3 shows that our model performs well on all test characters for self-retargeting, even on non-human morphologies. Still, the retargeting error is significantly higher for Mousey compared to other characters. We notably outperform the skeleton-aware networks proposed by Aberman et al. [3], with an average retargeting error normalised by character height at 0.009, which is equivalent to an error of 1.62 cm for a 1.8 m tall character. As shown in Table 4.4, the performance of our model on intra-structural retargeting is a bit lower because non-human morphologies are encountered more often. In particular, the retargeting error when Mousey is either the source or target character is increased with respect to other characters, which points out that the ability of our model to generalise to morphologies very different from the ones in the training set is a bit limited. Still, its performance is competitive with respect to the state-of-the-art skeleton-aware networks proposed by Aberman et al. [3]. cross-structural retargeting. Table 4.5 confirms our conclusions on the performance of our model on human morphologies and its relative limitation on very different morphologies. Nevertheless, our approach outperforms skeleton-aware networks [3], while it does not require a specific encoder/decoder pair to be trained for each new skeleton topology. Surprisingly, our model is more accurate on crossstructural retargeting than on intra-structural retargeting, which is presumably easier.

G o ↔ M o G o ↔ M r G o ↔ V a M o ↔ M r M o ↔ V a M r ↔ V
The performance on cross-structural retargeting suggests that our model effectively The lower performance on intra-structural retargeting indicates that our model might wrongly transfer some morphological features. We further discuss the reasons to that in Section 4.5, and the possible directions to solve it. In Figures 4.9 and 4.10, we provide visual results of motion retargeting performed using our approach and skeleton-aware networks [3].

Input

Ground Truth SAN Ours Figure 4.9 -Visual result of cross-structural retargeting. In this example, a motion sequence (1st row) drawn from Mixamo, consisting in fighting moves, is retargeted from BigVegas to Vampire using either Skeleton-Aware Networks (SAN) [3] (3rd row) or our model (4th row). The corresponding sequence for character Vampire (2nd row) is considered as ground truth when evaluating motion retargeting. stochastic joint subsampling.

A A → P A → P → A A → M A → M → A Figure 4.
Finally, Tables 4.3 to 4.5 show that our stochastic joint subsampling mechanism is effective, with performance of the variant Oursno SJS consistently outperformed by our main variant. It is worth noting that the difference is larger on self-retargeting rather than intra-structural and cross-structural retargeting, and on characters with morphology close to human.

conclusion

We have proposed a novel framework in which an autoencoder network based on transformers learns to encode diverse motion sequences with variable skeleton topologies and morphologies in a shared latent space, i.e. our unified deep motion representation. The reasons why our approach is successful are threefold: first, we rely on a skeleton template conditioning mechanism to tell both encoder and decoder which morphology and topology is given and targeted, respectively. Moreover, it acts as a spatial variant of positional encoding since we require skeleton templates to be We have demonstrated the ability of our model to accurately encode motion sequences, perform motion retargeting, and denoise motion sequences. Indeed, both motion retargeting and denoising can be performed reasonably well through encoding into our deep motion representation followed by decoding. This suggests that our deep motion representation is rather well structured to encode clean motion sequences with abstract features that are common to multiple topologies and morphologies. In the case of retargeting in particular, the results obtained with our model are very close to the state-of-the-art skeleton-aware networks proposed by Aberman et al. [3].

However, both models reach an average normalised retargeting error ranging from 0.03 to 0.06 on cross-structural and intra-structural, corresponding to an abolute mean per joint position error of 5.4 cm to 10.8 cm for a 1.8 m tall character which is relatively consequent. When looking at visual results such as those illustrated in Figure 4.9,

we might think that both skeleton-aware networks and our method have essentially reached the limitation of using Mixamo's source and target motion sequence pairs as ground truth for retargeting evaluation. If our thoughts are correct, further reducing the retargeting error evaluated over Mixamo would mostly be overfitting. Nevertheless, our deep motion representation unified across morphology and topology has the advantage over skeleton-aware networks [3] to not require additional encoder/decoder pairs to be trained for each new skeleton topology. This is particularly beneficial when few motion data are available for a given skeleton topology, or when training an additional encoder/decoder pair is not possible.

Beyond motion retargeting, our model might be useful for numerous applications, in a similar way that recent powerful natural language models are effective on multiple different natural language tasks. For instance, joint positions might be completed in a similar way to JUMPS, i.e. our approach presented in Chapter 3. In the temporal domain, this might include frames completion, motion prediction and in-betweening.

Moreover, a well structured deep motion representation might also facilitate numerous editing applications, e.g. to affect the trajectory followed, the style expressed, the interactions with environment or other characters, etc. Last but not least, our deep motion representation, which is able to embed together motion sequences from different sources (e.g. databases or animation systems outputs), might enhance the performance and extend the scope of numerous approaches in skeletal character animation. On one hand, data-driven approaches might be applied on large general databases coupled with smaller specialised ones (e.g. annotated with style or contact labels) to increase supervision, which is currently difficult when skeleton topologies differ. On the other hand, existing methods and animation systems might be more easily used as building blocks for novel methods of higher level and complexity since topological and morphological variations constitute a source of incompatibility.

It is tempting to think that our model might generalise to any morphology. However, our evaluation demonstrated that our model learnt on human characters is more effective on unseen human-like morphology rather than on exaggerated morphologies.

Moreover, we showed that our model is less accurate on intra-structural retargeting, i.e.

with source and target characters having the same topology. The reason here might be that the skeleton templates lack complexity, pushing our deep motion representation to not entirely be freed from some morphological features. When performing intra-structural retargeting, such wrongly encoded morphological features would be transferred, explaining the lower performance. Moreover, skeleton templates that we represented with joint positions have no mechanism to specify correspondences. For instance, a long half arm and a short full arm result in similar structure in skeleton templates even though the nature of their dynamics is different. In other words, skeleton templates do not tell anything about dynamics which leaves space for ambiguity when dealing with different morphologies and topologies. Increasing expressiveness of skeleton templates is a possible direction to improve our deep motion representation.

Finally, our approach, as other existing methods for motion retargeting, do not take care of foot contacts. This is an issue when transferring motion dynamics from one character toward another because annoying footskate artefacts (i.e. inconsistencies between feet motion and ground) are likely to be introduced. Even though approaches based on angular pose representations are more prone to produce such artefacts, it is still an issue with positional representations, such as in our case, since neither guarantees are given nor explicit constraints applied to ensure that legs movements are consistent with the global trajectory. In the next chapter, we present an approach intended to automatically solve such artefacts typically applied as a post-process.

Chapter abstract

Human motion synthesis and editing are essential to many applications like video games, virtual reality, and film post-production. However, they often introduce artefacts in motion data, which can be detrimental to the perceived realism. In particular, footskate is a frequent and disturbing artefact, which requires knowledge of foot contacts to be cleaned up. Current approaches to obtain foot contact labels rely either on unreliable threshold-based heuristics or on tedious manual annotation. In this chapter, we explore foot contact detection from motion data with deep learning. To this end, we first publicly release U nder P ressure, a novel motion capture database labelled with pressure insoles data serving as reliable knowledge of foot contact with the ground. Then, we design and We further clean up footskate artefacts through an optimisation-based inverse kinematics algorithm while enforcing vGRF invariance.

introduction

Perceived realism is central in animation; however, when synthesising or processing motion sequences, annoying artefacts can be introduced, such as footskate well-known in skeletal character animation. As seen in Section 2.4.1, It is a family of artefacts which can be defined as motion inconsistencies of the feet with respect to the ground such as when the feet are sliding on, passing through or floating above the ground while they should be static and on the ground. The problem of footskate is twofold: first, it easily appears when processing or synthesising motion sequences. Second, it is known to disturb human perception even at very low intensities [START_REF] Pražák | Perceptual Evaluation of Footskate Cleanup[END_REF] and to be highly detrimental to the realism of synthetic characters. In this chapter we explore foot contact detection and footskate artefacts removal, the former being a necessary step of the latter.

As of today, foot contacts are derived from motion sequences using simple heuristics, most commonly hand-crafted thresholds applied over the velocity and height of the feet to obtain binary contact labels. These approaches suffer from three main limitations.

First, the terrain height must be known everywhere, meaning that these approaches are generally not applicable to uneven or inclined terrain. Second, those thresholds have no universal optimal values and therefore must be manually tuned, ideally for every motion type, morphology and contact location (e.g. heel or toe). Finally, even with optimal values thresholding approaches are far from being 100% accurate and particularly lack of robustness, which implies that tedious manual checking is necessary when using these approaches. Especially in the context of footskate cleaning, foot contact detection must be robust to the presence of, at least, footskate artifacts.

In this chapter, we present a novel data-driven method for foot contact detection from motion sequences which significantly outperforms traditional heuristic approaches accuracy and robustness. First, we captured U nder P ressure, a novel database publicly released and composed of motion sequences synchronised with pressure insoles data, from which vertical ground reaction forces (vGRFs) and accurate foot contact labels can be derived. Then, our key idea is to model vGRFs with a deep neural network, providing a finer representation of interactions of the feet with the ground than binary contact labels. Furthermore, vGRFs are directly related to the dynamics of motion unlike binary labels. To this end, we train our model to estimate the distribution of vGRFs exerted under the feet, from which foot contact labels can be calculated, encouraging robustness and generalisability since the network needs a relatively deep understanding of motion dynamics to accurately estimate vGRFs.

We evaluate our model against an OT baseline that relies on thresholding the velocity and height above ground of foot joints. In addition, we make an ablative study which considers learned linear and non-linear generalisations of the OT baseline as well as variants of the proposed model. We further experiment how these models behave in different challenging conditions representative of real-world applications.

Finally, we demonstrate the generalisability of our approach on motion sequences from other databases, as well as its integration in a fully automatic footskate cleanup workflow. The main novelty of this workflow is to leverage our deep neural network by enforcing invariance between vGRFs estimated from input and optimised motion sequences to better guide the IK optimisation used to clean footskate and maintain the consistency of interactions between feet and ground.

related work

Early works in human animation already considered kinematic constraints such as foot contacts [START_REF] Bindiganavale | Motion Abstraction and Mapping with Spatial Constraints[END_REF][START_REF] Ikemoto | Knowing When to Put Your Foot Down[END_REF][START_REF] Kovar | Footskate Cleanup for Motion Capture Editing[END_REF][START_REF] Le | Robust Kinematic Constraint Detection for Motion Data[END_REF][START_REF] Lee | Interactive Control of Avatars Animated with Human Motion Data[END_REF]. Foot contact labels are helpful in numerous applications: they are often necessary to clean up foot artefacts such as footskate [START_REF] Kovar | Footskate Cleanup for Motion Capture Editing[END_REF], e.g. by enforcing foot constraints via IK [3,[START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Min | Motion graphs++: a compact generative model for semantic motion analysis and synthesis[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF]; likewise, they are required to quantify such foot artefacts, e.g. for evaluation purposes [START_REF] Yu | Character Controllers using Motion VAEs[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF]; they are also helpful to mitigate the well-known problem of mean collapse / drift away in human motion prediction, particularly with deterministic and recurrent models [START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Henter | MoGlow: Probabilistic and controllable motion synthesis using normalising flows[END_REF][START_REF] Martinez | On human motion prediction using recurrent neural networks[END_REF][START_REF] Wang | Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control[END_REF], to disambiguate human motion modelling [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF][START_REF] Pavllo | QuaterNet: A Quaternionbased Recurrent Model for Human Motion[END_REF][START_REF] Pavllo | Modeling Human Motion with Quaternion-Based Neural Networks[END_REF][START_REF] Xia | Realtime Style Transfer for Unlabeled Heterogeneous Human Motion[END_REF] (see Section 2.3.1), and more generally to leverage contact-based loss functions for increased quality and robustness [START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Lee | Interactive Character Animation by Learning Multi-Objective Control[END_REF][START_REF] Park | Learning predict-and-simulate policies from unorganized human motion data[END_REF][START_REF] Shi | MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency[END_REF][START_REF] Wang | Spatio-temporal Manifold Learning for Human Motions via Long-horizon Modeling[END_REF]; foot contact information is also deeply involved in character control based on physical simulation [START_REF] Abdolhosseini | On Learning Symmetric Locomotion[END_REF][START_REF] Kwon | Fast and Flexible Multilegged Locomotion Using Learned Centroidal Dynamics[END_REF][START_REF] Won | Learning Body Shape Variation in Physics-based Characters[END_REF][START_REF] Xie | ALLSTEPS: Curriculum-driven Learning of Stepping Stone Skills[END_REF][START_REF] Yu | Learning Symmetric and Low-Energy Locomotion[END_REF] where ground reaction forces are explicitly modelled in the physics engine, and also relates to foot-placement strategies that are a real challenge for locomotion policies [START_REF] Xue | Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?[END_REF]. In the following, we overview existing approaches for foot contacts labels detection (Section 5.2.1) and ground reaction forces estimation (Section 5.2.2), as well as existing databases of motion data labelled with information on foot contacts (Section 5.2.3).

Foot Contact Labels Detection

In Chapter 2, we have extensively reviewed existing approaches for foot contacts labels detection (see Section 2.4.1). In summary, both animation research and industry mostly rely directly or indirectly on simple heuristics with hand-crafted thresholds, applied to velocity and height of the feet as proposed by Bindiganavale and Badler [START_REF] Bindiganavale | Motion Abstraction and Mapping with Spatial Constraints[END_REF] or Lee et al. [START_REF] Lee | Interactive Control of Avatars Animated with Human Motion Data[END_REF]. However, they lack of temporal precision and are not reliable [START_REF] Le | Robust Kinematic Constraint Detection for Motion Data[END_REF], in particular in the presence of noise or artefacts. As a result, they are not suited in the context of footskate cleanup without tedious manual checking and corrections. Beyond thresholds, researchers have explored learned contact detection models, e.g. KNN classifier [START_REF] Ikemoto | Knowing When to Put Your Foot Down[END_REF] and neural networks [START_REF] Rempe | Contact and Human Dynamics from Monocular Video[END_REF][START_REF] Shi | MotioNet: 3D Human Motion Reconstruction from Monocular Video with Skeleton Consistency[END_REF][START_REF] Shimada | PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time[END_REF][START_REF] Harrison | Efficient Neural Networks for Real-time Motion Style Transfer[END_REF][START_REF] Zou | Reducing Footskate in Human Motion Reconstruction with Ground Contact Constraints[END_REF]. However, these approaches are learned from ground truth foot contacts that are either obtained with heuristics or manually annotated which limits the amount of data and hence the generalisability of the derived model.

Ground Reaction Forces Estimation

In physics, the force exerted by the ground on a body in contact, such as the human body, is called a ground reaction force (GRF). It is generally difficult to measure but nonetheless important in many fields of study including biomechanics, biomedical engineering and physics-based animation. Researchers in biomechanics and biomedical investigated GRF estimation from plantar pressure sensors [START_REF] Jung | Ground reaction force estimation using an insole-type pressure mat and joint kinematics during walking[END_REF][START_REF] Morin | Foot contact detection through pressure insoles for the estimation of external forces and moments: application to running and walking[END_REF][START_REF] Rouhani | Ambulatory assessment of 3D ground reaction force using plantar pressure distribution[END_REF], inertial and optical motion capture systems [START_REF] Faber | Estimating 3D L5/S1 moments and ground reaction forces during trunk bending using a full-body ambulatory inertial motion capture system[END_REF][START_REF] Fry | Validation of a motion capture system for deriving accurate ground reaction forces without a force plate[END_REF][START_REF] Karatsidis | Estimation of Ground Reaction Forces and Moments During Gait Using Only Inertial Motion Capture[END_REF][START_REF] Mundt | Prediction of ground reaction force and joint moments based on optical motion capture data during gait[END_REF], 3D accelerometers [START_REF] Leporace | Residual Analysis of Ground Reaction Forces Simulation during Gait Using Neural Networks with Different Configurations[END_REF], and Kinect [START_REF] Eltoukhy | Prediction of ground reaction forces for Parkinson's disease patients using a kinect-driven musculoskeletal gait analysis model[END_REF]. However, GRF estimation for biomechanics or biomedical applications is beyond the scope of our approach that uses vGRF distribution as a proxy representation and is intended for human animation applications. For more details on GRF estimation in biomechanics and biomedical engineering, we refer the reader to the systematic review by Ancillao et al. [START_REF] Ancillao | Indirect Measurement of Ground Reaction Forces and Moments by Means of Wearable Inertial Sensors: A Systematic Review[END_REF].

Early works in motion reconstruction leveraged pressure sensors to measure GRFs, because of their importance in dynamics. Ha et al. [START_REF] Ha | Human Motion Reconstruction from Force Sensors[END_REF] formulated the problem as a per-frame optimisation of end-effector positions obtained from a hand tracking device, and linear and angular momentums measured with pressure platforms. Later, Zhang et al. [START_REF] Zhang | Leveraging Depth Cameras and Wearable Pressure Sensors for Full-Body Kinematics and Dynamics Capture[END_REF] leveraged a pair of pressure sensing shoes as well as three depth cameras to develop a full-body motion reconstruction framework consisting in kinematic pose reconstruction followed by physics-based motion optimisation.

More recently, several approaches instead estimated GRFs from monocular images, starting with 2D and 3D pose estimation and then solving physical optimisation problems. Zell et al. [START_REF] Zell | Joint 3D Human Motion Capture and Physical Analysis from Monocular Videos[END_REF] proposed to estimate inner and exterior forces by optimising camera parameters and 2D pose reconstruction with a linear combination of base poses in a first step, and then GRFs and inner joint torques to satisfy the equations of motion and resolve camera projection ambiguities. Li et al. [START_REF] Li | Estimating 3D Motion and Forces of Person-Object Interactions from Monocular Video[END_REF] estimated 3D motion and forces between a subject and its environment by minimizing the discrepancy between the observed and reprojected 2D poses, with priors on estimated 3D poses, trajectory smoothness and physical plausibility for regularisation. Rempe et al. [START_REF] Rempe | Contact and Human Dynamics from Monocular Video[END_REF] and Shimada et al. [START_REF] Shimada | PhysCap: Physically Plausible Monocular 3D Motion Capture in Real Time[END_REF] proposed a similar pipeline but focusing on more dynamic and diverse human motions, without object interactions. Shimada et al. further corrected imbalanced stationary poses. Later on, they extended their approach with additional DNN [START_REF] Shimada | Neural Monocular 3D Human Motion Capture with Physical Awareness[END_REF]: TPNet first regresses target 3D poses and contact states from 2D key points.

Then, GRFNet and DyNet iteratively estimate GRFs and proportional-derivative (PD) controller gain parameters in a dynamic cycle where the character pose is updated at each step after FK.

Different from motion reconstruction from images, Zell et al. [START_REF] Zell | Weakly-supervised Learning of Human Dynamics[END_REF] proposed a weakly-supervised approach to inverse dynamics. An MLP is trained to estimate GRFs, moments and joint torques from motion such that the input motion is reconstructed using forward dynamics in an optimisation loop. Motion capture data synchronised with force plates enable supervision during training: reconstructed ground reaction force and moment and joint torque divergences are penalised while GRF are minimised whenever feet are not in contact with the ground.

To the best of our knowledge, the closest work to the proposed method is the deep learning approach to improve computing stability in human pose estimation proposed by Scott et al. [START_REF] Scott | From Image to Stability: Learning Dynamics from Human Pose[END_REF]. In this work, body dynamics analysis from joint positions is investigated while most papers on human pose estimation only focus on skeleton

database

In this work, we release U nder P ressure, a motion capture database annotated with pressure insoles data, designed primarily for character animation purposes. In the following, we provide information about the capture, the motion characteristics, and the preprocessing steps.

We recorded 10 healthy adult volunteers (2F, 8M) with diverse morphologies aged between 21 and 55 years (32 ± 11 yr), weighing between 65 and 91 kilograms (79 ± 9 kg), and measuring between 167 and 187 centimetres (177 ± 5 cm). Each subject performed the same set of activities, including forward and backward locomotion at different paces, sitting, standing, passing obstacles, climbing stairs, as well as motions on uneven motion trackers with respect to the corresponding segments. After MVN Animate processing, motion data consist of pose sequences with 23 segments sampled at 240 Hz.

Foot Pressure

In addition to motion capture, we recorded the spatial distribution of plantar foot pressures. To this end, subjects were also equipped with Moticon's OpenGo Sensor Insoles [START_REF] Rego | OpenGo Sensor Insole Specification[END_REF] placed into their shoes. Each insole has 16 plantar pressure sensors with a resolution of 0.25 N/cm 2 and a 6-axis IMU, both running at 100 Hz (see Figure 5.2a). Moreover, we weighed each subject with full equipment to enable vGRF normalisation and equipped subjects with the same shoes whose soles are thin and flexible insole for accurate and faithful pressure measures as well as for controlling the grip.

Ψ Γ Contacts Validation X ∈ℝ T ×J ×3 F ∈ℝ T ×2×16 F =Ψ ( X ) L vGRFs = 1 N ∑ i=1 N ln 2 ( F i + 1 F i + 1 )
Figure 5.3 -Overview of our approach. Our database provides synchronised input motion sequences and target vertical ground reaction forces (vGRFs) to train our network Ψ depicted in red. The blue trapezoid represents the contact function Γ (see Section 5.3.3). At runtime, our network estimates vGRFs from which foot contact labels can be derived using the contact function Γ, both useful in many applications, e.g., reconstructing motion from images, cleaning footskate, finding suitable transition frames for motion blending, adapting animations to uneven terrain, and many more. As illustrated, both estimated vGRFs and detected contacts are evaluated in Sections 5.5.2 and 5.5.3, respectively. match the framerate of pressure insole data, and trimmed the beginning and end of each sequence to remove the synchronisation patterns. Original motion sequences at 240 Hz are also provided in our database.

deep neural network

In this section, we describe the proposed method to learn a deep model for vGRFs estimation from motion capture data. Learning vGRFs instead of binary contact labels encourages our deep neural network to more accurately model interactions between feet and ground, and enforces motion dynamics understanding. See Figure 5.3 for an overview of our approach.

Data Representation input

At each frame t, the human pose X t ∈ R J×3 is represented by the position of its J = 23 joints in a global Euclidean space. We design our deep network Ψ to output vGRFs and contact labels at each frame from a few surrounding input frames with padding when needed. The full input pose sequence is then X ∈ R T×J×3 , where T is a variable number of frames.

output As previously described, our deep network Ψ estimates the vGRF distribution from motion data. At each frame, it outputs F = Ψ (X) with F ∈ R T×2×16 , i.e. 16

positive real-valued vGRF components, corresponding to the 16 insole pressure cells for each foot, expressed proportionally to subject weight.

Network Architecture

We designed our network Ψ to process variable-length sequences. To this end, the network is composed of four 1D temporal convolutional layers with 7-frame wide kernels, followed by three fully-connected layers applied at each frame independently to preserve the support variable-length sequence. Each convolutional or fully-connected layer is followed by ELUs as activation, except for the last one which is a softplus activation to output nonnegative vGRF components.

Training and Inference

During training, our network is iteratively exposed to sequences of human poses and tries to estimate corresponding vGRF components as depicted in Figure 5.3. To encourage robustness and smooth convergence, we make use of stochastic data augmentation. First, similar to random crops and rotations used on images in computer vision, we apply random vGRF-invariant transformations on input pose sequences including translations, horizontal rotations, scaling, and left-right mirroring.

For each sequence, we also randomly draw its skeleton which is then animated by joint angles to further robustify our network. To do so, we precomputed (offline) an singular value decomposition (SVD) basis of the skeletons captured in our database.

At training time, we draw new skeletons by linearly combining precomputed singular vectors with randomly sampled weights. We then further edit these skeletons by randomly moving joint relative positions and rescaling bone lengths to obtain morphological variations. The resulting input motion sequences purposely suffer from artefacts since kinematic chains (i.e. from root joint to feet) have been randomly edited, which encourages the network to be resilient with respect to perturbed inputs. Joint positions are finally computed through forward kinematics and fed to the network.

To train our deep network Ψ, we minimise a reconstruction loss of vGRF components.

Instead of the standard MSE, we use the mean squared logarithmic error (MSLE). It has the property to only focus on the relative difference between target and estimated values (see right-hand side of Equation (5.1)), which is convenient when the values considered can be several orders of magnitude apart. In our case, actual vGRFs can be strictly positive and arbitrarily low (e.g. during transition from the double leg stance to the single leg stance) as well as very high (e.g. during jump landing). The loss function used to train our network is then

L = 1 N N ∑ i=1 ln(F i + 1) -ln( Fi + 1) 2 = 1 N N ∑ i=1 ln 2 F i + 1 Fi + 1 (5.1)
where F i are the ground truth vGRF components, and Fi = Ψ(X) i their estimated counterpart. Adding 1 to both F and F ensures that the loss is defined when vGRF component value goes to zero.

At inference, our deep network estimates vGRF components from joint positions as inputs. Then, foot contact labels can be calculated from vGRF estimates using the contact function Γ (see Section 5.3.3).

evaluation

In this section we present results of our method to assess estimated vGRFs and detected foot contacts. After providing implementation details necessary for reproducibility, we assess foot contact labels detection and vGRF estimation. Finally, we also evaluate foot contact detection performance on different perturbed motion sequences, simulating challenging conditions encountered in concrete applications. with a mini-batch size of 64, learning rate α = 3 • 10 -5 and hyperparameters β 1 = 0.9 and β 2 = 0.999.

Foot Contacts Detection

metric.

To evaluate foot contact detection, we use the F 1 score which is the harmonic mean of precision (fraction of correctly detected labels among detected labels)

and recall (fraction of correctly detected labels among expected contact labels).

baseline.

To evaluate our model against commonly used heuristics-based approaches using thresholds, we define an optimal thresholds (OT) baseline. It has two thresholds that are applied on foot height and velocity norm. To demonstrate the effectiveness of our approach, we set these two thresholds to the values which maximise the F 1 score over the training set, computed by recursive grid search. Note that in practical applications, such optimal values are not available. Therefore, this baseline constitutes an upper bound of threshold-based approaches.

ablative study.

We also investigate the relevance of our architecture with an ablative study. We first consider two linear models taking as inputs joint positions and velocities, as learned generalisations of the OT baseline: the former (Linear-Feet), takes foot and ankle joints as inputs while the latter (Linear) takes all joints. Then, we consider a 3-layer MLP, i.e. the architecture of the foot contact detection module in the style transfer framework proposed by Smith et al. [START_REF] Harrison | Efficient Neural Networks for Real-time Motion Style Transfer[END_REF], introducing non-linearities with respect to Linear-Feet and Linear models. . Intuitively, the farther misdetected contacts are from contact phases, the more severe they are. False positive rates of learned models like ours quickly decrease outside contact phases while the optimal thresholds (OT) baseline keeps it significantly higher in-between contacts. In this section we assess performances of our method on vGRFs estimation from motion capture data. We use the root mean squared error (RMSE) of per-foot total vGRF which is mainly sensitive to global biases (at foot scale). In complement, we also evaluate the centre of pressure (CoP) computed from vGRF estimates, which is more sensitive to local errors (at pressure cell scale) since it is calculated as the weighted mean of vGRF component application points.

results. motions perturbed with gaussian noise. First, we simply evaluate how contact detection performances are affected by additive isotropic Gaussian noise. To this end, we first take with ground truth sequences from our test set and purposely add Gaussian noise to joint positions. Then we try to detect foot contacts from the noised motion sequences using various models evaluated in Section 5.5.2 and report their performances with the F 1 score. We repeat this procedure for an entire range of noise levels whose standard deviation varies from 0 to 5 centimetres which corresponds to a MPJPE ranging from 0 to 8 centimetres. Finally, we plot the resulting F 1 score curves in

footskate cleanup

In this section, we leverage our robust foot contact detection for a downstream task and propose a novel fully automatic workflow for footskate cleanup. After detecting foot contact labels, our goal here is remove footskate artefacts by setting the feet static on the ground during detected contact phases, which can be seen as a particular case of IK. Traditional IK algorithms often rely on numerical optimisation to reach the target(s).

Few works based on deep learning have been proposed, such as the framework of Victor et al. [START_REF] Victor | Learning-based pose edition for efficient and interactive design[END_REF] for interactive design which allows to edit end-effectors locations in a static pose while keeping a globally consistent pose. In our case, we rely on an hybrid optimisation-based approach in which our deep network is used to enforce preservation of vGRFs throughout the optimisation.

Let X = ( Q, S, P) be the input motion sequence suffering from footskate, where Q denote joint angles, S the skeleton and P the global trajectory positions of the root joint. Our goal is to find Q and P such that X = (Q, S, P) is the footskate-cleaned version. First, we compute F = Ψ(FK( Q, S, P)) and C = Γ( F), being respectively the vGRF components estimated by our deep neural network Ψ and the foot contact labels calculated using the contact function Γ defined in Section 5.3.3. The function FK(•) refers to forward kinematics, i.e. the computation of joint positions from joint angles, skeleton and global trajectory. After initialising Q and P with Q and P, we iteratively optimise both for a small fixed number N of iterations to best satisfy foot contact constraints into a gradient-based optimisation loop. In the following paragraphs we describe each term L i of the our loss function

L = ω q L quat + ω f L f oot + ω t L traj + ω v L vGRFs (5.2)
where weights ω i are hyperparameters to balance our objectives.

quaternions loss.

We parameterise joint angles with unit quaternions, which are well-suited for such an optimisation since they are free of singularities, computationally efficient, and numerically stable [START_REF] Mourot | A Survey on Deep Learning for Skeleton-Based Human Animation[END_REF]. To keep valid rotations with quaternions throughout the optimisation, L quat penalise quaternion norm deviations from 1:

L quat = ∑ t ∑ j ∥Q j (t)∥ -1 2 (5.3)
where Q j (t) is the quaternion representing the orientation of joint j at time t.

foot contacts loss.

Satisfying foot contact constraints is not an easy task since we do not have directly access to actual contact locations with respect to the skeleton.

Moreover, since foot joints are located above actual contact points, they are allowed to rotate around the latter during part of contact phases (e.g. when the heel is in contact with the ground, the ankle is constrained to rotate around). To alleviate these issues, we artificially insert contact joints in S corresponding to contact points, i.e. under foot joints at the ground level (see Figure 5.11). We assume that each contact joint is located vertically under its foot joint in the T-pose at the ground height, has constant orientation with respect to its foot joint, and is assumed to be static during contact phases. Then, imposing zero velocity on contact joints instead of foot joints during the contact phases better reflects the kinematics of the interaction with the ground, allowing slight rotation of foot joints located above the ground level. To this end, we minimise the mean squared distance from contact joints to contact positions during contact phases to faithfully constrain foot joint positions through forward kinematics:

Foot joints Contact joints

L f oot = C ∑ c=1 t 1 (c) ∑ t=t 0 (c)
Π(t) ∥FK(Q, S, P) j c (t) -p c ∥ 2 (5.4)

where j c and p c are respectively the joint in contact with the ground during contact phase c and its contact position during the contact phase spanning from t 0 (c) to t 1 (c). Moreover, function Π(•) is a rectangular wave function with smoothed edges (synchronised with contact phases) to avoid sharp foot position changes.

trajectory loss.

Since foot contact constraints are not guaranteed to be reachable by leg extensions, the global trajectory might be affected. However, it is closely related to body dynamics and its optimisation must be carefully controlled to avoid artefacts.

In particular, constraining trajectory positions might introduce implausible velocity changes, hurting motion dynamics realism, e.g. slowing down to reach some contact position might lead to speeding up afterwards to catch up the target trajectory. To this end, we minimise the trajectory velocity deviations instead of position deviations:

L traj = ∑ t (∥v(t)∥ -∥ ṽ(t)∥) 2 , v(t) = P(t + ∆T) -P(t) ∆T (5.5)

where v(t) is the global velocity of root joint at time t, computed from global trajectory position P by numerical differentiation.

vgrf invariance loss.

The main novelty in our gradient-based optimisation approach is the use of our deep neural network through which gradients can flow.

Since it is able to estimate consistent vGRF components from either clean or perturbed inputs, we further guide the optimisation by minimising the deviation between initially estimated vGRFs components F and dynamically estimated vGRF components F = Ψ(FK(Q, S, P)):

L vGRFs = MSLE(F, F) (

where the MSLE measures vGRFs deviation as previously (see Section 5.4.3).

We used the same motion sequences blended for evaluation purposes in Section 5.5.4

to test the proposed footskate cleanup workflow. With parameters N = 100, ω q = 10 -3 , ω f = 10 -5 , ω t = 10 2 and ω v = 5 • 10 -5 (see Equation (5.2)), we achieve to significantly reduce the amount of footskate (the velocity of the feet during contact phases is approximately halved) while noticeably improving the realism of the sequences, assessed by visual inspection.

conclusion

In this chapter, we presented a novel approach that improves the state-of-art of foot contact detection and footskate cleanup in human character animation. Building on U nder P ressure, a novel motion capture database synchronised with pressure insoles which we publicly released, we proposed to learn the relationship between human motion and interactions between feet and ground with a deep neural network estimating vertical ground reaction forces, from which foot contact constraints can be derived and exploited for an effective removal of foot sliding artefacts.

As of today, the proposed database is one of a kind in animation as it provides synchronised motion capture and pressure insoles data for a wide variety of human motion. Our approach significantly outperforms thresholds-based heuristic approaches in detecting foot contact labels, which are limited in their capability to generalise.

We show its robustness to perturbed input motion sequences, which is crucial in concrete use cases. We have shown that estimating forces greatly helps the accurate and robust detection of contacts. Finally, we demonstrate the usefulness of our foot contact detection approach for footskate cleanup, leveraging vGRF estimates to improve the quality of the results.

As described in Section 5.2, foot contact labels are relevant to a lot of tasks in human animation and are typically obtained with simple heuristics, or manually annotated on an exceptional basis. Thus, many of these downstream tasks could probably benefit from the improved accuracy of our foot contact detection approach. Moreover, motion reconstruction relies more and more on modelling physical interaction of the feet with the ground (see Section 5.2.2). Resolving inverse dynamics problems like ground reaction forces estimation from motion capture data could provide a valuable regularisation for such underconstrained problems as well as other tasks involving physics-based models. We believe our approach is a step in that direction.

C O N C L U S I O N

In the last decade, artificial intelligence has revolutionised the topic of skeletal character animation as many others, especially through deep learning. Novel frameworks, e.g. leveraging variational autoencoder (VAE) [START_REF] Sadegh Aliakbarian | A Stochastic Conditioning Scheme for Diverse Human Motion Prediction[END_REF][START_REF] Du | Stylistic Locomotion Modeling and Synthesis using Variational Generative Models[END_REF][START_REF] Habibie | A Recurrent Variational Autoencoder for Human Motion Synthesis[END_REF][START_REF] Yan | MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics[END_REF] or generative adversarial network (GAN) [START_REF] Barsoum | HP-GAN: Probabilistic 3D human motion prediction via GAN[END_REF][START_REF] Gingras | Robust Motion In-betweening[END_REF][START_REF] Makhzani | Adversarial Autoencoders[END_REF][START_REF] Yan | Convolutional Sequence Generation for Skeleton-Based Action Synthesis[END_REF], pushed generative motion synthesis up to unprecedented realism. Sophisticated schemes such as mixture-of-expert networks [START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF] or local motion phases [START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF] yielded breakthroughs in the control of virtual characters.

The examples are numerous. Though, these advances still lack quality, flexibility and ability to generalise to fully outperform skilled animators' manual work quality and to replace them when dealing with tedious low-creativity tasks. Therefore, the purpose of this thesis is to tackle current obstacles preventing them from doing so.

In particular, we strived to alleviate the lack of high-quality motion data in the context of deep learning and the inclination of neural networks to introduce artefacts when synthesising or processing motion data. To this end, we first investigated the enhancement of 2D human pose sequences estimated from video using prior knowledge captured in a deep generative model. Then, we proposed a novel versatile deep representation of human motion, in which motion features are abstracted out from skeleton structure and morphology, allowing for gathering, processing or retargeting motion sequences with variable skeleton topologies. Finally, we tackled foot contact detection to automatically clean up footskate artefacts, well-known in character animation. unified human motion representation.

Another way to tackle the bottleneck constituted by the limited availability of motion data is to increase the interoperability of existing human motion databases, which are difficult to gather and exploit together. For example, combining a limited amount of motion data annotated with style labels with a larger general motion database might be more thoughtful when developing a style transfer method than acquiring a large amount of motion data annotated with style labels. However, combining existing databases is not straightforward since virtual characters often have different sets of joints and interconnections and might even be as tedious as acquiring a large amount of novel motion data. To overcome this issue, we proposed in Chapter 4 a novel deep representation of human motion unified across skeleton topologies and morphologies. We learnt this representation with a transformer-based neural network in an autoencoder scheme from a large amount of motion data collected from different existing motion databases. We condition both encoder and decoder on skeleton topological and morphological features allowing to abstract "pure" motion features in our deep representation. The effectiveness of transformers to discover complex patterns from data is known to be unprecedented. In our case, it overcomes the difficulty to learn such a unified human motion representation, which itself makes the training of our transformer-based architecture possible. Indeed, transformers require large data volumes but our unified model can be learnt from the gathering of multiple existing motion databases. As a result, our deep motion representation constitutes an interesting space to embed and work with motion sequences from different sources. Moreover, the effectiveness of transformers coupled with the generality of the approach makes our model well-suited for a variety of downstream applications including motion retargeting, motion denoising and joint upsampling.

foot contact detection and footskate cleanup.

Finally, we also investigated in this thesis the cleaning of artefacts. We believe that deep learning is also promising to effectively clean artefacts as post-processing rather than indefinitely increasing data quality, data volumes and deep model sizes to perfect the results obtained.

In particular, we focused on footskate which is known to be detrimental to the perceived realism even at low intensities [START_REF] Pražák | Perceptual Evaluation of Footskate Cleanup[END_REF] and also easily introduced when processing motion data. Knowledge of foot contacts is required for footskate cleanup, however it is often obtained through manual annotation or using simple heuristics (e.g. hand-crafted foot velocity threshold). Therefore, we proposed a reliable foot contact detection solution on top of which we built an algorithm to automatically clean footskate artefacts. To this end, we first captured and publicly released a novel database of motion data synchronised with pressure insoles data, from which accurate foot contacts can be derived. Hence, this database provides a significant amount of motion data annotated with foot contacts compared to scarce existing motion data manually annotated. Then, we built a robust foot contact detection approach by training a deep neural network to estimate vertical ground reaction forces (vGRFs) from motion on our database. vGRFs serve as a proxy representation which captures physical interactions between feet and ground better than binary foot contact labels. At inference, binary foot contact labels can be computed from vertical ground reaction forces (vGRFs). We demonstrate that our approach outperforms traditional heuristics-based solutions in terms of accuracy, generalisability and robustness. On top of our foot contact detection approach, we also presented a fully automatic workflow to remove footskate artefacts from motion sequences. Following recent approaches, footskate is removed by enforcing foot constraints (detected with our model) using an inverse kinematics (IK) algorithm, iteratively optimising joint angles to satisfy the constraints. We additionally leverage our deep model to preserve ground reaction forces during this optimisation process and keep motion dynamics globally consistent.

perspectives and future work

In this thesis, we have explored approaches in the general goal of improving deep learning based systems for skeleton character animation. In the following, we present possible research perspectives in the continuation of our work. First, we propose shortterm perspectives, i.e. possible improvements and experiments which naturally stem from the work presented in this thesis. Then, we suggest a set of possible directions to extend our work inspired from existing literature in character animation and beyond.

Finally, we open up the discussion to long-term perspectives requiring to solve several challenges and rethink part of the current paradigms in character animation.

Short-Term Perspectives

In Chapters 3 and 4 we have presented two approaches relying on deep motion representations. In the latter, we proposed an abstract motion representation independent of skeleton topology and morphology. However, as pointed out in Section 4.5, our model has some limitations with regard to morphological variations, including decreased performance on non-human morphologies.

A first perspective would be to investigate whether or not the proposed model is able to accurately encode both human and humanoid morphologies in a single unified motion representation. In other words, is it possible to unify the motion of humanoid characters (i.e. two legs, two arms and a head but possibly exaggerated body proportions) in a deep representation learnt following our approach? Moreover, non-humanoid characters such as quadrupeds might also be explored in future work.

A simple approach to do so that might be effective would be to diversify the training dataset beyond human motion sequences. However, other changes more involved might be needed to handle humanoid or other kind of characters, and are overviewed in the medium-term perspectives hereafter.

Another limitation of our work is that footskate artefacts are likely to be introduced in some cases. As of today, this issue concerns most, if not all, deep motion models used to produce or process motion data. A possible approach to improve in this direction would be to introduce knowledge about interactions between feet and ground.

For instance, this could be possible by introducing some supervision with respect to foot contact or vertical ground reaction forces (vGRFs) during training. This can be directly implemented using the motion database proposed in Chapter 5 in which motion data is synchronised with pressure insole data. Several ways to add supervision are possible and can be combined. Finally, the amount of footskate artefacts introduced is rarely evaluated. The reasons here are the scarcity of motion data annotated with accurate foot contacts, and the difficulty to obtain accurate ground truth foot contacts. The database and foot contact detection approach proposed in Chapter 5 could be leveraged to define some standard evaluation procedure of footskate.

Medium-Term Perspectives

A rising challenge in character animation is to propose methods that are versatile enough to be usable on characters with large topological and morphological variations, and possibly generalise beyond humanoid characters in some cases. Extending some of the deep motion representations proposed in this thesis to handle such large variations might therefore need to rethink the conditioning mechanism we proposed. In particular, we currently proposed to use neutral poses as skeleton templates, represented by their joint positions which might be limited to fully capture morphological and topological features. While this issue could be solved by a more thoughtful design, we believe that an elegant solution would consist in additionally learning a deep representation of skeleton topologies and morphologies together with a deep motion representation.

Such an approach would have the benefit to avoid manually predefining skeleton templates while alleviating the limitations of our current approach. To further extend the scope of our approach, one might also need to consider personalising skeleton templates (either predefined or learnt) for each subject, i.e. more finely tuning our conditioning mechanism to better capture fine morphological variations, e.g. hip sexual dimorphism.

Beyond skeleton templates, novel deep learning models might be leveraged for improved learning. For instance, diffusion models [START_REF] Sohl-Dickstein | Deep Unsupervised Learning Using Nonequilibrium Thermodynamics[END_REF] are very attractive to build deep representations useful for multiple tasks. Indeed, these models outperform GANs on image synthesis, and have also been promising in a variety of domains [START_REF] Yang | Diffusion Models: A Comprehensive Survey of Methods and Applications[END_REF]. Moreover, diffusion models also improve on scalability and parallelisability. Finally, their progressive nature might be key to numerous applications that can be formulated as progressive transitions. These include most completion tasks, such as motion completion, in-betweening or joint upsampling, but also cleaning tasks such as denoising or footskate cleanup. The progressive nature of diffusion models might even extend the scope of current problems. For instance, motion-to-motion diffusion models (in analogy with image-to-image diffusion models [START_REF] Saharia | Palette: Image-to-Image Diffusion Models[END_REF]) could learn to progressively retarget a motion sequence from a character to another, notably allowing character morphing in motion sequences, i.e. progressively switching from one character to another over time.

There are also multiple perspectives in the direction of cleaning and preventing artefacts in output motion sequences. In the context of footskate cleanup in particular, one of the limitations comes from the binary representation of foot contacts which are then generally used to enforce foot constraints using inverse kinematics (IK) algorithms.

Indeed, interactions between feet and ground are not trivial and modelling them as in or out of contact is overly simplified and conducts to manually tune IK algorithms for specific situations to avoid replacing footskate by other kinds of artefacts. A possible direction to solve this issue is to use a finer representation of feet and ground interactions, such as vGRFs, and design an IK algorithm specifically targeting footskate cleanup, e.g. by enforcing target profiles of vGRFs instead of binary in-contact phases.

This is something that we have only slightly explored in this thesis, but which we believe shows promising potential for novel approaches. Another ambitious direction to prevent footskate artefacts would be to explicitly model feet and ground interactions together with motion, e.g. by building a common embedding of vGRFs and motion sequences similarly to visual semantic embedding in computer vision that are shared between text and images.

Long-Term Perspectives

In the long term, we envision that character animation might need to change paradigm to make animation synthesis and editing more accessible. This might include replacing current technical human-computer interfaces in animation systems by semantic controls more intuitive for designers and artists. In this direction, semantic embeddings are promising. Indeed, they have recently demonstrated impressive capabilities at the intersection of natural language processing and computer vision, e.g.

OpenAI's novel image generation model DALL•E 2 [START_REF] Ramesh | Hierarchical Text-Conditional Image Generation with CLIP Latents[END_REF] which relies on a version of GPT-3 [START_REF] Tom | Language Models are Few-Shot Learners[END_REF] modified for image synthesis. In particular, semantic embeddings based on natural languages seem to be very efficient to yield powerful representations and capture necessary information for applications involving human-computer interactions.

In character animation, a lot of applications include human-computer interactions at some point, such as motion synthesis and editing systems driven by artists and animators or virtual characters controlled by gamers. For example, in analogy with capabilities of current image synthesis models, we could imagine to quickly produce draft motion sequences from textual descriptions, such as "a tall character running for 4 seconds then accelerating jump above an obstacle". However, building a semantic embedding for character animation based on natural language would require to solve several tough challenges. These challenges include the greediness in term of data volume of recent successful models such as GPT-3. In the case of images associated with text descriptors, billions of examples are available on the internet. On the contrary, motion data are currently much more scarce that images. Also, motion sequences are not often annotated with text descriptors in existing databases, needed to build such a semantic embedding of natural language and character motion. Another important challenge is that dedicated natural language model(s) might be necessary, as for DALL•E 2. Even though it might require less complexity than general natural language models like GPT-3, little work has been done in this direction. A recent attempt is the language-directed physical character controller proposed by Juravsky et al. [START_REF] Juravsky | PADL: Language-Directed Physics-Based Character Control[END_REF].

We believe that the contributions presented in this thesis are a first step in such directions, and will open the door to novel approaches for animating virtual characters based on deep learning approaches. Though, these advances still lack quality, flexibility and ability to generalise to replace animators when dealing with tedious low-creativity tasks. Therefore, the purpose of this thesis is to tackle current obstacles preventing from doing so. In particular, we strived to alleviate the lack of high-quality motion data in the context of deep learning and the inclination of neural networks to introduce artefacts when processing motion data. To this end, we first investigated the enhancement of 2D human pose sequences estimated from video using prior knowledge captured in a deep generative model. Then, we abstracted out motion features from skeleton topology and morphology in a deep motion representation, with the goal of gathering, processing or retargeting motion sequences with variable skeleton topologies and morphologies. Finally, we tackled foot contact detection to automatically clean up footskate artefacts, well-known in character animation.

R

  É S U M É Les êtres humains et leurs représentations sont omniprésents dans toutes les cultures. Depuis la nuit des temps, nos ancêtres ont peint ou sculpté des myriades d'oeuvres d'art où figurent souvent des représentants humains, comme les peintures pariétales de la grotte de Lascaux, il y a environ 20'000 ans, la Vénus de Milo, il y a 2'000 ans, ou le David de Michel-Ange, il y a quelques siècles. Au cours des dernières décennies, les technologies numériques ont véritablement révolutionné la manière dont les êtres humains peuvent créer et accéder à la culture. De plus en plus d'outils sont apparus pour aider tous les types d'artistes, y compris les concepteurs et les animateurs, en particulier pour accroître leurs capacités créatives et le réalisme de leurs oeuvres d'art tout en réduisant les coûts de production. Par exemple, des personnages virtuels ont donné vie de manière très convaincante à des créatures fictives non humaines comme les Na'vi dans Avatar ou même à l'identité visuelle d'acteurs décédés depuis longtemps, comme le personnage Grand Moff Tarkin d'abord interprété par l'acteur britannique Peter Cushing en 1977 dans Star Wars : Episode IV -A New Hope, puis synthétisé Rogue One en 2016, bien que l'acteur soit décédé en 1994 (voir figure 1.1).

  poses complètes et suréchantillonnées qui y correspondent, ainsi qu'avec une cohérence temporelle améliorée. Dans le chapitre 4, nous proposons une nouvelle représentation profonde polyvalente pour unifier les séquences de mouvement avec différentes topologies et morphologies de squelette. Étant donné les variations d'emplacement, de nombre et d'interconnexion des articulations entre les systèmes, les méthodes et les bases de données en animation, notre objectif est d'accroître leur interopérabilité avec une représentation abstraite du mouvement indépendante des caractéristiques structurelles et morphologiques du squelette. Pour atteindre cet objectif, nous concevons une architecture d'autoencodeur dédiée basée sur les transformeurs qui permet de traiter des séquences de mouvement d'entrée et de sortie avec un nombre variable d'articulations et en conditionnant à la fois l'encodage et le décodage sur le squelette. Notre modèle a de multiples applications, notamment le rééchantillonnage des articulations ou le transfert d'un mouvement d'un squelette source à un squelette cible, c'est-à-dire le retargeting.

contributionsArticle 1 :Article 2 :Article 3 :

 123 Cette thèse est basée sur les contributions suivantes : Lucas Mourot, Ludovic Hoyet, François Le Clerc, François Schnitzler and Pierre Hellier, "A Survey on Deep Learning for Skeleton-Based Human Animation", Computer Graphics Forum 41.1, 2021, pp. 122-157, DOI : 10.1111/cgf.14426. Lucas Mourot, François Le Clerc, Cedric Thébault and Pierre Hellier, "JUMPS : Joints Upsampling Method for Pose Sequences", Proceedings of 2020 25th International Conference on Pattern Recognition (ICPR), 2021, pp. 1096-1103, DOI : 10.1109/ICPR48806.2021.9412160. -Presenté à la 25th International Conference on Pattern Recognition (ICPR), Virtual / Milano, Italy, 2021. Lucas Mourot, Ludovic Hoyet, François Le Clerc and Pierre Hellier, "Transformers-based Unified Deep Motion Representation", article en préparation.

6. 2 . 3 Figure 1 . 1

 2311 Figure 1.1 Grand Moff Tarkin, a fictional character in the Star Wars franchise, first played by Peter Cushing in Star Wars: Episode IV -A New Hope in 1977 (left) and synthesised in Rogue One: A Star Wars Story in 2016 (right), 22 years after Peter Cushing passed away. . . . .

Figure 2 . 1

 21 Figure 2.1 Illustration of the error accumulation problem with angular representations: even small angular errors along the kinematic chain can lead to large accumulated joint positioning errors (left-hand stick figure, see error colour gradient). This is problematic in optimisation frameworks such as deep learning when penalising joint orientation deviations. This is not the case with positional representations (right-hand stick figure) where joint positions are directly optimised. . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 2 . 2

 22 Figure 2.2 Representation of skeletal motion data as a graph proposed by Aberman et al. [3]. The nodes of the graph correspond to joints and the edges to armatures. Each of the J armatures holds a timevarying tensor Q modelling the temporal sequence of rotations at its corresponding joint, and a time-independent vector S modelling the bone offset to the parent joint. The global motion of the root joint R is processed separately. Illustration in courtesy of Aberman et al. [3]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 2 . 3

 23 Figure 2.3 In a variational autoencoder (VAE), a motion sequence is mapped by an encoder to a random latent code that is constrained to follow a prior Gaussian distribution. This code is mapped back by a decoder to the input motion data. Once the full network is trained, feeding the decoder with samples drawn from the prior generates stochastic motion sequences. . . . . . . . . . . . . . . .

Figure 2 . 4 A

 24 Figure 2.4 A generative adversarial network (GAN) consists of two networks: the generator produces synthetic motion samples from random seeds, the discriminator tries to differentiate them from real ones drawn from the training dataset. The two modules are trained jointly, aiming at an equilibrium where the generator outputs cannot be distinguished from the training data. . . . . . . . . . .

Figure 2 . 5

 25 Figure 2.5Overall idea of the mixture-of-experts scheme[START_REF] Li | A Perceptual-Based Noise-Agnostic 3D Skeleton Motion Data Refinement Network[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF]:agating network first computes blending weights for a number of expert networks, based on a number of features extracted from the current frame x t . Each expert specialises in a particular movement. They are then blended together to dynamically compute the weights of a motion prediction network, which outputs information relevant to the next frame x t+1 including the pose of the animated character. This output is typically fed back into the network (autoregression) at time t + 1. . . . . . . . . . . . . . . .

Figure 3 . 1

 31 Figure 3.1 Illustration of our method applied to human pose estimation: the yellow 12-joint skeletons (top) depicts poses estimated by AlphaPose [192] at different timestamps. Our approach completes and upsamples joints in a spatially and temporally consistent way to enrich estimated pose sequences, and yields the corresponding green 28-joint skeletons (bottom). In both images, the background has been stitched from a video with a smaller field of view and five images of the jumper at different frames have been overlaid as foreground. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 2

 32 Figure 3.2 Coarse representation of our deep generative model. The upper right part is the basis of our model: a generative adversarial network (GAN) with generator G and discriminator D operating on motion sequences with higher number of joints. Moreover, we complement our model with encoder E to map motion sequences with lower number of joints to the latent space of the GAN. P z denotes the prior distribution while P (h) data and P (l)data denote motion data distributions at with higher and lower number of joints, respectively. The latter is sampled by discarding predefined joints from motion sequences sampled from the former. . . . . . . . . .

Figure 3 . 3

 33 Figure 3.3 Illustration of how our model internally represents a pose sequence whose topology is depicted on the left part. The right part shows how joint coordinates are arranged, with body parts ordered following the human skeleton from hands to feet. x 1 , y 1 , x 2 and y 2 denote the 2D coordinates of left and right joints, respectively, for symmetrical joints (e.g. hips) and duplicated 2D coordinates of asymmetrical joints (e.g. pelvis). . . . . . . . . . .

Figure 3 . 4

 34 Figure 3.4 Computational flows through our deep generative model during training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 5

 35 Figure 3.5 Detailed description of our network architecture. Notations: Convolution (Conv), Transposed Convolution (Tr.Conv), Batch Normalisation (BN), Rectified Linear Unit (ReLU) and Leaky Rectified Linear Unit (LReLU). . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 6

 36 Figure 3.6 Qualitative example of the influence of overlapping temporal chunks. This subset of four consecutive frames in a longer sequence is inpainted with no overlap (top) and half overlap (bottom). The frames are located at the end (first two) and the beginning (last two) of two consecutive chunks. Note the temporal discontinuity of joint locations (e.g. head top, elbows, ankles) in the sequence inpainted with no overlap at the chunk boundary (dashed red line, top). The temporal consistency is much better with half overlap (bottom). . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 7

 37 Figure 3.7 Example of a limb (right forearm) occluded by the subject's body inaccurately estimated by AlphaPose [192] but recovered by our method based on human motion priors. Note that these images have been intentionally whitened except for the area around the occlusion for clarity purposes. . . . . . . . . . . . . . . . . . . . .

Figure 4 . 1

 41 Figure 4.1 Overview of our model which is a transformer-based autoencoder whose encoder E and decoder D are conditioned on skeleton templates to control morphological and topological features. Skeleton templates consist in neutral poses (see Section 4.3.2). During training, our model is guided by the reconstruction loss L rec as well as the temporal bone lengths consistency loss L rbl (see Section 4.3.4). At inference, applications of our model include motion retargeting that can be performed by setting the decoding template to the target character skeleton, as well as motion cleaning by encoding and decoding with the same template as our decoder has learnt to produce clean motion sequences. . . . . . . . . . . . . . . . . .

Figure 4 . 2

 42 Figure 4.2 Illustration of a typical skeleton template used to condition our model, consisting in a neutral standing pose with arms along the body (i.e. N-pose) with the pelvis at the origin and transverse (green) and coronal (blue) planes aligned with XY and XZ planes, respectively. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 3

 43 Figure 4.3 Illustration of the spatial positional encoding used in our encoder, performed via joint-wise concatenation of intermediate features of skeleton template and motion. Hence, for each joint, motion features are registered to template features of the same joint (e.g. left wrist highlight in yellow), telling the network which piece of information is associated to which joint. Thereby, the skeleton template acts as a reference frame for motion features. . . . . . .

Figure 4 . 4

 44 Figure 4.4 Illustration of our encoder architecture: output latent code (right) is computed from input template (top-left) and motion (bottomleft) through the different network layers (dark cyan rounded rectangles) and other functions (see top-right caption), following black arrows. Coloured rectangular slices depict tensor shapes of template (red), motion (blue), query token (cyan) and latent (grey) features at different points in the network. Colour gradients across slices illustrate varying features over time dimension. . .

Figure 4 . 5

 45 Figure 4.5 Illustration of our decoder architecture: output motion sequence (right) is computed from input template (bottom-left) and latent code (top-left) through the different network layers (dark red rounded rectangles) and other functions (see top-right caption), following black arrows. Coloured rectangular slices depict tensor shapes of template (red), motion (blue) and latent (grey) features at different points in the network. Colour gradients across slices illustrate varying features over time dimension. . . . . . . . . . .

Figure 4 . 6

 46 Figure 4.6 Illustration of the different skeleton topologies present in the dataset we gathered, and their structural variations, represented by the corresponding predefined neutral poses. The left-most five columns show front and side views of the different topologies. Each one is associated to a subset of the data (from left

  The right-most column gives an insight about topological variations by overlaying the different topologies. . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 7

 47 Figure 4.7 Illustration of the representation accuracy of our deep motion representation on a challenging motion sequence, consisting in a speed vault to cross some fence or wall, depicted in white. The ground truth test motion sequence (top row, green) is given as input to our model which encodes and then decodes it to get the corresponding reconstructed motion sequence (middle row, red). The bottom row shows both ground truth and reconstructed motion sequences overlaid to emphasise reconstruction errors. .

Figure 4 . 8

 48 Figure 4.8 Quantitative evaluation of our model on motion denoising. Gaussian noise is purposely added to clean motion sequences from our validation set. The resulting noised motion sequences are then encoded and decoded using our model to perform denoising. For an entire range of levels of noise, positional error on denoised sequences is plotted vertically against the positional error observed in noised sequences, horizontally. The top axis indicates the standard deviation of the different levels of noise, while the bottom axis gives the mean per joint position error (MPJPE) observed in corresponding noised sequences, which is proportional to the standard deviation. Results are averaged over seen and unseen skeleton topologies. Flatter curves correspond to higher denoising ability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 9

 49 Figure 4.9 Visual result of cross-structural retargeting. In this example, a motion sequence (1st row) drawn from Mixamo, consisting in fighting moves, is retargeted from BigVegas to Vampire using either Skeleton-Aware Networks (SAN) [3] (3rd row) or our model (4th row). The corresponding sequence for character Vampire (2nd row)is considered as ground truth when evaluating motion retargeting.

Figure

  Figure 4.10 Visual examples of motion retargeting. A motion sequence (see

Figure 5 . 5

 55 Figure 5.5 False positive distribution during normalised off-contact phases, i.e. mapped to [0, 1]. Intuitively, the farther misdetected contacts are from contact phases, the more severe they are. False positive rates of learned models like ours quickly decrease outside contact phases while the optimal thresholds (OT) baseline keeps it significantly higher in-between contacts. . . . . . . . . . . . . . .

Figure 5 . 6

 56 Figure 5.6 Scatter plot of 2D offsets between centres of pressure computed from ground truth and estimated vGRFs. The concentric solid and dashed circles respectively represent the mean and median norm of 2D offsets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 5 . 7

 57 Figure 5.7 Illustration of vGRF components at different timestamps during a walking cycle. The top, middle and bottom rows respectively depict the ground truth, vGRFs estimated from motion by our deep neural network and the corresponding absolute error. . . . . . .

Figure 5 . 8 F

 58 Figure 5.8 F 1 score on foot contacts detection from motion sequences purposely noised with additive isotropic Gaussian noise. Each curve represents the F 1 score against the amount of noise introduced, measured with the MPJPE in centimetres indicated by the bottom horizontal axis, while the top horizontal axis gives the corresponding standard deviations of the Gaussian noise. The results indicate that our method is more robust to noise. . . . . . . . . . . . . . .

Figure 5 . 9 F

 59 Figure 5.9 F 1 score on foot contacts detection from motion sequences distorted by a motion autoencoder early-stopped at different epochs to emulate different amounts of distortion. Each curve displays the F 1 score against the amount of distortion introduced, measured with the MPJPE in centimetres. The results indicate that our method is more robust to distorted sequences. . . . . . . . . . . .

Figure 5 . 10 F

 510 Figure 5.10 F 1 score against amount of footskate. Test motions with matching foot contact patterns have been blended to purposely introduce footskate while preserving contact labels. Foot contact detection is then evaluated on such motions suffering from footskate, and compared to the ground truth contact labels. The amount of footskate introduced through blending is measured using the mean velocity of the feet during contact phases. Non-linear learned detection models keep reasonably high accuracy while accuracy of linear models significantly decrease and accuracy of optimal thresholds (OT) baseline quickly collapse. . . . . . . . . . . . . . .

Figure 5 . 11

 511 Figure 5.11 Illustration of contact joints (green dots) inserted under corresponding foot joints at the ground level in the skeleton to properly enforce foot contact constraints (See details below). . . . . . . . .

1 I

 1 N T R O D U C T I O N Humans and their representations are ubiquitous in all cultures. Since the beginning of times, our ancestors have painted or sculpted myriads of works of art often with human representatives portrayed like the parietal wall paintings in Lascaux Cave about 20 thousands years ago, the Venus de Milo 2 thousands years ago or Michelangelo's David a few centuries ago. In the past decades, digital technologies have truly revolutionised the way humans can create and access culture. More and more tools have appeared to assist all kind of artists including designers and animators, especially to increase their creative capabilities and the realism of their artworks while reducing production costs. For instance, synthetic characters have very convincingly brought to life non-human fictional creatures like the Na'vi in Avatar or even visual identity of long time deceased actors like Grand Moff Tarkin in Rogue One in 2016 portrayed by the British actor Peter Cushing even though he passed away in 1994, as illustrated in Figure 1.1.

Figure 1 . 1 -

 11 Figure 1.1 -Grand Moff Tarkin, a fictional character in the Star Wars franchise, first played by Peter Cushing in Star Wars: Episode IV -A New Hope in 1977 (left) and synthesised in Rogue One: A Star Wars Story in 2016 (right), 22 years after Peter Cushing passed away.

Figure 2 . 1 -

 21 Figure 2.1 -Illustration of the error accumulation problem with angular representations: even small angular errors along the kinematic chain can lead to large accumulated joint positioning errors (left-hand stick figure, see error colour gradient). This is problematic in optimisation frameworks such as deep learning when penalising joint orientation deviations. This is not the case with positional representations (right-hand stick figure) where joint positions are directly optimised.

Figure 2 . 1 ,

 21 Figure 2.1, making it difficult to accurately handle end effectors. This is especially true in motions sequences involving fast or ample movements, e.g. running.

Figure 2 . 2 -

 22 Figure 2.2 -Representation of skeletal motion data as a graph proposed by Aberman et al. [3]. The nodes of the graph correspond to joints and the edges to armatures. Each of the J armatures holds a time-varying tensor Q modelling the temporal sequence of rotations at its corresponding joint, and a time-independent vector S modelling the bone offset to the parent joint. The global motion of the root joint R is processed separately. Illustration in courtesy of Aberman et al. [3].

  proposed HDM05, a public well-documented database of systematically recorded motion capture data. Complementary to CMU which contains a large number of diverse motion sequences, HDM05 is composed of a limited number of specific motion sequences (one hundred) which were executed from 10 to 50 times by five actors. As an example, a cartwheel starting with the left hand has been performed 21 times. More recently, Shahroudy et al.[START_REF] Shahroudy | NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis[END_REF] proposed NTU RGB+D, one of the largest datasets for 3D skeleton-based action recognition. It contains 60 action classes as well as RGB and infrared (IR) videos and depth map sequences (D) synchronised with motion sequences. Later on, Liu et al.[START_REF] Liu | NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding[END_REF] added another 60 classes to constitute NTU RGB+D 120, roughly doubling the size of the dataset. However, the motion sequences in these two datasets are represented only by joint positions which limits their use in skeleton-based human animation. Since motion capture systems need dedicated environments e.g. for a multicamera setup, human motion datasets are mostly captured in the lab, resulting in a lack of in the wild data. To this end, Marcard et al.[START_REF] Timo Von | Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera[END_REF] proposed a pose estimation method, leveraging inertial measurement units in addition to a hand-held camera, accurate enough to capture a new dataset called 3DPW consisting of human pose sequences in the wild synchronised with RGB videos.

[ 3 ]

 3 resorted to a simple implementation of spatial GCNs. The supports of convolution kernels around each joint are defined as d-ring neighbourhoods on the skeleton graph in the spatial dimension and extended to the temporal axis to obtain 2D skeleto-temporal convolutions. The operation of such GCNs is limited to skeletons sharing the same topology. However, the motion retargeting network they proposed includes skeletal pooling/unpooling layers, based on the fusion/duplication of the signals of adjacent edges. The pooling layers bring the input topology to a common primal skeleton on which the core processing is performed. The result is transformed back to the original topology by the unpooling layers. Such a network can cope with any topology that is homeomorphic to the primal skeleton.

Figure 2 . 3 -

 23 Figure 2.3 -In a variational autoencoder (VAE), a motion sequence is mapped by an encoder to arandom latent code that is constrained to follow a prior Gaussian distribution. This code is mapped back by a decoder to the input motion data. Once the full network is trained, feeding the decoder with samples drawn from the prior generates stochastic motion sequences.

Figure 2

 2 Figure 2.4 -A generative adversarial network (GAN) consists of two networks: the generator produces synthetic motion samples from random seeds, the discriminator tries to differentiate them from real ones drawn from the training dataset. The two modules are trained jointly, aiming at an equilibrium where the generator outputs cannot be distinguished from the training data.

  .2). These two components are processed in distinct parallel branches of the retargeting network. Importantly, a motion sequence is modelled as a graph whose edges correspond to armatures. This provides a principled formalism for processing motion data sampled on the skeletal graph. The retargeting network includes three types of modules: space-time graphconvolutional operators acting on spatio-temporal joint neighbourhoods, graph pooling and graph unpooling operators. Graph pooling merges features of two adjacent edges (armatures) into a single feature. Each graph unpooling operator is designed as the inverse of a graph pooling operator, splitting one edge into two adjacent edges whose features are copied from the original feature. Even though homeomorphic skeletons are supported, a separate autoencoder has to be learnt for each skeletal structure to encode motion sequences into a common latent space expressed in the common primal skeleton topology, and to decode back latent code to motion sequences. Retargeting is achieved by composing the encoder for the source character with the decoder for the target character. The need of a separate autoencoder for each different homeomorphic

  et al.[START_REF] Wang | Adversarial learning for modeling human motion[END_REF][START_REF] Wang | Transferring Style in Motion Capture Sequences with Adversarial Learning[END_REF] leveraged an LSTM-based sequential adversarial autoencoder whose encoder learns to map motions to separate content and style encodings: two additional discriminators are trained to recover style labels from content and style encodings, respectively. The encoder tries to fool the former and helps the latter in order to free the content encoding from the style information while preserving it in the style encoding. At runtime, both the content and style motion sequences are encoded into separate content and style embeddings. The decoder combines the content embedding of the content motion sequence and the style embedding of the style motion sequence to produce the style transfer output. Following these works, Aberman et al.[4] also extracted content and style encodings but with two separate temporal convolutional encoders. Furthermore, the style encoder learns a common embedding from both 2D and 3D joint positions with a triplet loss, which enables style extraction from videos. The output motion is synthesised from the content encoding while the style is controlled by the style encoding through temporally invariant adaptive instance normalisation (AdaIN). Moreover, a multi-style discriminator assesses the output motion style. Park et al.[START_REF] Park | Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model[END_REF] pointed out two limitations of the latter framework: firstly, spatial relations between joints are not considered with 1D convolutions, which might be an issue when style and content motions are significantly different. Secondly, input style in previous works can only be given from an example motion, which is not always available in real-world applications. To tackle the former issue, Park et al.[START_REF] Park | Diverse Motion Stylization for Multiple Style Domains via Spatial-Temporal Graph-Based Generative Model[END_REF] presented a GAN-based framework relying on spatial-temporal GCNs to extract style features in both spatial and temporal domains. In addition, a mapping network is learnt to generate style codes sampled from a Gaussian distribution and to match style codes encoded from style motions. At runtime, the space of style codes can be explored through the mapping network and used to generate new style codes without requiring example motions from which to extract style codes, addressing the latter issue.

Figure 3 . 1 -

 31 Figure 3.1 -Illustration of our method applied to human pose estimation: the yellow 12-joint skeletons (top) depicts poses estimated by AlphaPose [192] at different timestamps.Our approach completes and upsamples joints in a spatially and temporally consistent way to enrich estimated pose sequences, and yields the corresponding green 28-joint skeletons (bottom). In both images, the background has been stitched from a video with a smaller field of view and five images of the jumper at different frames have been overlaid as foreground.

Figure 3 . 2 -

 32 Figure 3.2 -Coarse representation of our deep generative model. The upper right part is the basis of our model: a GAN with generator G and discriminator D operating on motion sequences with higher number of joints. Moreover, we complement our model with encoder E to map motion sequences with lower number of joints to the latent space of the GAN. P z denotes the prior distribution while P (h) data and P (l)

1 Figure 3 . 3 -

 133 Figure 3.3 -Illustration of how our model internally represents a pose sequence whose topology is depicted on the left part. The right part shows how joint coordinates are arranged, with body parts ordered following the human skeleton from hands to feet. x 1 , y 1 , x 2 and y 2 denote the 2D coordinates of left and right joints, respectively, for symmetrical joints (e.g. hips) and duplicated 2D coordinates of asymmetrical joints (e.g. pelvis).

  second component L backward rec focuses on the reconstruction of latent codes sampled from the prior distribution P z . It minimises the mean squared error (MSE) between z ∼ P z and its reconstructed version E(G(z)):

  b) Flow during generator/encoder's training step.

Figure 3 . 4 -

 34 Figure 3.4 -Computational flows through our deep generative model during training.

Figure 3 . 5 -

 35 Figure 3.5 -Detailed description of our network architecture. Notations: Convolution (Conv), Transposed Convolution (Tr.Conv), Batch Normalisation (BN), Rectified Linear Unit (ReLU) and Leaky Rectified Linear Unit (LReLU).

Finally, in absenceFigure 3 . 6 -

 36 Figure 3.6 -Qualitative example of the influence of overlapping temporal chunks. This subset of four consecutive frames in a longer sequence is inpainted with no overlap (top) and half overlap (bottom). The frames are located at the end (first two) and the beginning (last two) of two consecutive chunks. Note the temporal discontinuity of joint locations (e.g. head top, elbows, ankles) in the sequence inpainted with no overlap at the chunk boundary (dashed red line, top). The temporal consistency is much better with half overlap (bottom).

Figure 3 . 7 -

 37 Figure 3.7 -Example of a limb (right forearm) occluded by the subject's body inaccuratelyestimated by AlphaPose[START_REF] Xiu | Pose Flow: Efficient Online Pose Tracking[END_REF] but recovered by our method based on human motion priors. Note that these images have been intentionally whitened except for the area around the occlusion for clarity purposes.

  tive model which is able to upsample joints and complete missing joints in 2D human pose sequences. Intended to enhance 2D pose estimates, we achieve to increase joint resolution without of loss accuracy. To the best of our knowledge, this is the first attempt to solve this problem in 2D with deep learning. Our framework considers a 12-joint 2D pose sequence as input and produces a valuable 28-joint 2D pose sequence by inpainting the input. The proposed model consists of a DCGAN coupled to a frontend encoder. Ablation studies have shown the strong benefit of the encoder, since it provides some supervision that greatly helps the convergence and accuracy of the combined model. Given an input sequence, inpainting is performed by optimising the latent representation that best reconstructs the low-resolution input. The encoder provides the initialisation and a prior loss based on the discriminator is used to improve the plausibility of the generated output.The obtained results are encouraging and open up future research opportunities toward high-quality human pose estimation and increased availability of high-quality inthe-wild human motion data. Improvements could come from an improved consistency with respect to actual human motion. Explicitly modelling or enforcing biomechanical constraints could help. Adapting our method to 3D pose sequences might also benefit from the richer information of 3D joints. Additionally, a potentially fruitful line of research could be to tackle as a whole 3D human pose estimation and joint upsampling.Finally, temporal consistency can be improved. Even though the overlapping subsequences mechanism introduced helps to provide smooth output pose sequences, the underlying architecture is still limited to fixed-length sequences. Moreover, CNNs are known to struggle to model long-term correlations. Transformers could be a promising candidate to better model long-term dependencies and handle variablelength sequences. Indeed, as explained in Section 2.2.4, transformers have recently been successful in natural language processing, in particular to efficiently learn distant correlations, and lately became very popular. In the following chapter, we explore such a transformer model to learn an abstract deep motion representation of variable-lengthChapter abstractMotion retargeting is a long-standing problem in character animation, which consists in transferring and adapting the motion of a source character to another target character.A typical application is the creation of motion sequences from off-the-shelf motions by transferring them onto new characters. Motion retargeting is also promising to increase interoperability of existing animation systems and motion databases, as they often differ in the structure of the skeleton(s) considered. Moreover, since the goal of motion retargeting is to abstract and transfer motion dynamics, effective solutions might coincide with expressive and powerful human motion models in which operations such as cleaning or editing are easier. In this chapter, we present a novel abstract representation of human motion agnostic to skeleton topology and morphology. Based on transformers, our model is able to encode and decode motion sequences with variable morphology and topology -extending the scope of retargeting -while supporting skeleton topologies never seen during the training phase.More specifically, our model is structured as an autoencoder and encoding and decoding are separately conditioned on skeleton templates to extract and control morphology and topology. Beyond motion retargeting, our model has many applications since our representation is convenient to embed motion data from different sources. It might be benefical to a number of data-driven methods, allowing them to combine scarce specialised motion datasets (e.g. with style or contact annotations) and larger general motion datasets for improved performance and generalisation ability. Other applications include motion denoising and joint upsampling.

  motion representation unified across human morphologies and topologies. Based on transformers, our model is able to encode and decode motion sequences with variable skeleton topology and morphology, extending the scope of retargeting beyond homeomorphic skeletons while supporting skeleton topologies never seen during the training phase. More specifically, our model consists in an autoencoder in which both encoder and decoder follow a transformer-based architecture, support motion sequences with variable number of joints and are separately conditioned on skeleton templates to extract and control morphology and topology. Trained on a large amount of data gathered from multiple existing databases with different skeleton topologies, our model successfully abstracts out motion features from morphological and topological features.

Figure 4 . 1 -

 41 Figure 4.1 -Overview of our model which is a transformer-based autoencoder whose encoder E and decoder D are conditioned on skeleton templates to control morphological and topological features. Skeleton templates consist in neutral poses (see Section 4.3.2).During training, our model is guided by the reconstruction loss L rec as well as the temporal bone lengths consistency loss L rbl (see Section 4.3.4). At inference, applications of our model include motion retargeting that can be performed by setting the decoding template to the target character skeleton, as well as motion cleaning by encoding and decoding with the same template as our decoder has learnt to produce clean motion sequences.

  having J (a) ∈ N + joints, where the function J (•) gives the number of joint of a given skeleton. Then X (b) ∈ R J (b)×3×F denotes a motion sequence consisting of the same motion features as X (a) but performed by a another skeleton b ̸ = a. Finally, we note E and D our encoder and decoder modules, E(X (a) |T (a)) the encoding of a motion sequence X (a) conditioned on the skeleton template T (a) associated to skeleton a, and D(z|T (b)) the decoding of latent code z conditioned on the template T (b) associated to skeleton b, where the function T (•) gives the template of given skeleton. In the next section we will see what skeleton templates are.

Figure 4 . 2 -

 42 Figure 4.2 -Illustration of a typical skeleton template used to condition our model, consisting in a neutral standing pose with arms along the body (i.e. N-pose) with the pelvis at the origin and transverse (green) and coronal (blue) planes aligned with XY and XZ planes, respectively.

Figure 4 . 3 -

 43 Figure 4.3 -Illustration of the spatial positional encoding used in our encoder, performed via joint-wise concatenation of intermediate features of skeleton template and motion.Hence, for each joint, motion features are registered to template features of the same joint (e.g. left wrist highlight in yellow), telling the network which piece of information is associated to which joint. Thereby, the skeleton template acts as a reference frame for motion features.

Figure 4 . 4 -

 44 Figure 4.4 -Illustration of our encoder architecture: output latent code (right) is computed from input template (top-left) and motion (bottom-left) through the different network layers (dark cyan rounded rectangles) and other functions (see top-right caption), following black arrows. Coloured rectangular slices depict tensor shapes of template (red), motion (blue), query token (cyan) and latent (grey) features at different points in the network. Colour gradients across slices illustrate varying features over time dimension.

Figure 4 . 5 -

 45 Figure 4.5 -Illustration of our decoder architecture: output motion sequence (right) is computed from input template (bottom-left) and latent code (top-left) through the different network layers (dark red rounded rectangles) and other functions (see top-right caption), following black arrows. Coloured rectangular slices depict tensor shapes of template (red), motion (blue) and latent (grey) features at different points in the network. Colour gradients across slices illustrate varying features over time dimension.

Figure 4 . 6 -

 46 Figure 4.6 -Illustration of the different skeleton topologies present in the dataset we gathered, and their structural variations, represented by the corresponding predefined neutral poses. The left-most five columns show front and side views of the different topologies. Each one is associated to a subset of the data (from left to right:Human3.6M[START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF], MPI-INF-3DHP[START_REF] Mehta | Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision[END_REF], AMASS[START_REF] Mahmood | AMASS: Archive of Motion Capture As Surface Shapes[END_REF], PSU-TMM100[START_REF] Scott | From Image to Stability: Learning Dynamics from Human Pose[END_REF] and

Figure 4 . 7 -

 47 Figure 4.7 -Illustration of the representation accuracy of our deep motion representation on a challenging motion sequence, consisting in a speed vault to cross some fence or wall, depicted in white. The ground truth test motion sequence (top row, green) is given as input to our model which encodes and then decodes it to get the corresponding reconstructed motion sequence (middle row, red). The bottom row shows both ground truth and reconstructed motion sequences overlaid to emphasise reconstruction errors.

Figure 4 . 7

 47 Figure 4.7 illustrates the representation accuracy of our model on a challenging example motion sequence. In this example, we can see that the dynamics is relatively well encoded with fine details, such as hand orientations which are kept consistent throughout the sequence. Positional errors mostly occur during fast and ample move-

Figure 4 . 8 -

 48 Figure 4.8 -Quantitative evaluation of our model on motion denoising. Gaussian noise is purposely added to clean motion sequences from our validation set. The resulting noised motion sequences are then encoded and decoded using our model to perform denoising. For an entire range of levels of noise, positional error on denoised sequences is plotted vertically against the positional error observed in noised sequences, horizontally. The top axis indicates the standard deviation of the different levels of noise, while the bottom axis gives the MPJPE observed in corresponding noised sequences, which is proportional to the standard deviation.Results are averaged over seen and unseen skeleton topologies. Flatter curves correspond to higher denoising ability.

  normalised and consistently aligned neutral poses, and we concatenate intermediate features of motions and templates per-joint. Second, we apply during training a wellthought variant of dropout, consisting in randomly subsampling joints of input and output motion sequences and skeleton templates consistently. As our model is forced not to rely too much on specific joints, this mechanism encourages our deep motion representation to be shared across skeleton topologies, instead of representing each topology in a distinct region. Third, our autoencoder network follows a thoroughly designed transformer-based architecture, coupled with a large amount of motion data gathered from multiple existing databases with different skeleton topologies and morphologies.

Figure 5 . 1 -

 51 Figure 5.1 -We leverage U nder P ressure, a novel publicly available dataset of motion capture synchronised with pressure insoles data, to learn a deep model for vertical ground reaction forces (vGRFs) estimation from motion data and foot contact detection. We further clean up footskate artefacts through an optimisation-based inverse kinematics algorithm while enforcing vGRF invariance.

Figure 5 . 5 -

 55 Figure 5.5 -False positive distribution during normalised off-contact phases, i.e. mapped to [0, 1]. Intuitively, the farther misdetected contacts are from contact phases, the more severe they are. False positive rates of learned models like ours quickly decrease outside contact phases while the optimal thresholds (OT) baseline keeps it significantly higher in-between contacts.

Figure 5 . 6 -Figure 5 . 7 -

 5657 Figure 5.6 -Scatter plot of 2D offsets between centres of pressure computed from ground truth and estimated vGRFs. The concentric solid and dashed circles respectively represent the mean and median norm of 2D offsets.

Figure 5

 5 Figure 5.8.

F

Figure 5 . 11 -

 511 Figure 5.11 -Illustration of contact joints (green dots) inserted under corresponding foot joints at the ground level in the skeleton to properly enforce foot contact constraints (See details below).

6. 1

 1 summary of contributions joint upsampling and completion.Human pose estimation attempts to capture human body pose or motion from image or video, respectively. Requiring little equipment, it is a promising alternative to traditional motion capture systems which are expensive and cumbersome. Hence, capturing high-quality motion data from video might vastly tackle the lack of human motion data, especially experienced with recent data-greedy models such as transformers. Since many 3D human pose estimation solutions rely on 2D pose estimation as a first step and solve inverse mapping from 2D to 3D afterwards, we presented in Chapter 3 a method to upsample and complete (e.g. missing joints due to occlusions) human joints in 2D as an intermediate step.Indeed, completed 2D pose sequences with higher spatial resolution disambiguate inverse mapping. Moreover, in such a use case, the higher resolution of joints would be reflected in estimated 3D pose sequences. To this end, we leveraged a deep generative model to learn the distribution 2D human pose sequences at high resolution of joints.At inference, completion and upsampling are performed by finding the point in the latent space of our model which best matches observed 2D low-resolution estimates, and then generating the corresponding completed high-resolution pose sequences. As our model is based on a GAN, this process is performed through optimisation in the latent space of our model, and high-resolution pose sequences are produced by the generator module of our model. We showed that our method is able to upsample joints in 2D pose estimates while keeping localisation accuracy of the underlying pose estimator. Moreover, prior knowledge on human motion learnt by our model allows to plausibly fill occluded joints (e.g. when a limb passes behind the body).

  For example: ground truth foot contacts might be used to measure and minimise footskate in output motion sequences; vGRFs or foot contacts could be estimated from the latent representation of motion sequences; vGRFs or foot contacts consistency loss might be applied on vGRFs or foot contacts estimated from reconstructed or retargeted motion sequences. Moreover, the unified motion representation proposed in Chapter 4 and the foot contact detection model proposed in Chapter 5 might be directly combined to automatically clean footskate artefacts. The former would retarget motion sequences to the skeleton topology of the latter to obtain foot contacts needed to clean footskate.

T

  I T R E : A P P R E N T I S S A G E P R O F O N D P O U R L' A N I M AT I O N D U S Q U E L E T T E D E P E R S O N -N A G E S V I R T U E L S : É D I T I O N D E L A TO P O L O G I E , R E TA R G E T I N G E T N E T T OYA G EMot clés : animation, personnages virtuels, apprentissage profond, mouvement humain Résumé : L'apprentissage profond a révolutionné l'animation de personnages durant la dernière décennie. Des modèles novateurs et sophistiqués ont permis d'obtenir un réalisme sans précédent. Cependant, ces avancées ne permettent pas encore de remplacer les animateurs sur toutes les tâches fastidieuses et peu créatives. L'objectif de cette thèse est de s'attaquer aux obstacles qui les en empêchent. En particulier, nous avons abordé le manque de données de mouvement de qualité et la tendance des réseaux de neurones à introduire des artéfacts lors du traitement de données de mouvement. Nous avons d'abord exploré l'amélioration de séquences de poses humaines 2D estimées à partir de vidéos en utilisant des connaissances a priori apprises par un modèle génératif profond. Ensuite, nous avons abstrait le mouvement de la topologie et de la morphologie dans une représentation profonde, afin de rassembler, de traiter ou de squeletto-morphoser des séquences de mouvement avec des topologies et des morphologies variables. Enfin, nous nous sommes attaqués à la détection des contacts des pieds avec le sol dans le but de nettoyer automatiquement les artéfacts de glissement des pieds. T I T L E : D E E P L E A R N I N G F O R S K E L E TA L C H A R A C T E R A N I M AT I O N : T O P O L O G Y E D I T-I N G , R E TA R G E T I N G A N D C L E A N I N G Keywords: character animation, deep learning, human motion Abstract: Deep learning has revolutionised skeletal character animation in the last decade. Novel frameworks and sophisticated schemes pushed toward to an unprecedented realism.

  -Demande de brevet : Lucas Mourot, Ludovic Hoyet, François Le Clerc and Pierre Hellier, "Topology-Agnostic Skeletal Motion Retargeting", Lucas Mourot, Ludovic Hoyet, François Le Clerc and Pierre Hellier, "UnderPressure : Deep Learning for Foot Contact Detection, Ground Reaction Force Estimation and Footskate Cleanup", Computer Graphics Forum 41.8, 2022, pp. 195-206, DOI : 10.1111/cgf.14635. Deep Neural Network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Data Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . Training and Inference . . . . . . . . . . . . . . . . . . . . . . . . . . Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Foot Contacts Detection . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.3 vGRFs Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.4 Foot Contacts Detection in Challenging Conditions . . . . . . . . . 5.6 Footskate Cleanup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	2023.
	Article 4: -Base 5.4.3	de	donnée	et	implémentation	disponible	ici	:
	https://github.com/InterDigitalInc/UnderPressure.

-Demande de brevet : Lucas Mourot, Ludovic Hoyet, François Le Clerc and Pierre Hellier, "UnderPressure : Deep Learning of Foot Contacts and Forces", 2022. -Presenté aux 24èmes journées du Groupe de Travail Animation et Simulation (GTAS), Vannes, France, 2022. -Presenté au 21th ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA), Durham, UK, 2022. 5.4 5.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusion 6.1 Summary of contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Perspectives and future work . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Short-Term Perspectives . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Medium-Term Perspectives . . . . . . . . . . . . . . . . . . . . . . .

Table 3

 3

	.2	Quantitative evaluation of our method and its variants on joint
		completion and upsampling applied on 2D human pose estimated
		from videos using AlphaPose [192]. We report the percentage of
		correct key points normalised with head size (PCKh) at different
		thresholds and corresponding area under the curve (AUC) over
		[0, 1] (see Section 3.4.1). The first row reports errors observed in
		AlphaPose outputs before applying our method. The ablation
		studies confirm the conclusions drawn from our pure joint up-
		sampling evaluation (see

Table 3

 3 .1). Higher PCKh and AUC mean better performance. . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 4 . 1

 41 Overview of the different datasets gathered to train our model. .

Table 4 .

 4

2 Quantitative evaluation of the representation accuracy of our deep motion representation. We measure the distortion introduced when encoding and then decoding ground truth motion sequences drawn from our validation set with the mean per joint position error (MPJPE) in centimetres. We distinguish skeleton topologies that have been seen during training from unseen topologies. Columns are associated to the different skeleton topologies in our validation set, named after the data sources used to constitute our dataset (see Section 4.3.4). Lower MPJPE means higher representation accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table 4.3 Quantitative evaluation of self-retargeting, i.e. from a source to a target character with the same skeleton topology and body proportions. We measure the retargeting error with the MPJPE normalised by the height of the skeleton, and provide results per character (middle) as well as overall average (right-most). Lower values means higher retargeting accuracy. . . . . . . . . . . . . .

Table 4 . 4

 44 Quantitative evaluation of intra-structural retargeting, i.e. from a source to a target character with the same skeleton topology. We measure the retargeting error with the MPJPE normalised by the height of the skeleton, and provide results per pair of source and target characters (middle) as well as overall average (rightmost). G o , M o , M r and V a stand for Goblin, Mousey, Mremireh and Vampire characters, respectively. Lower values means higher retargeting accuracy. . . . . . . . . . . . . . . . . . . . . . . . . . .

	Table 4.5	Quantitative evaluation of cross-structural retargeting, i.e. from a
		source (BigVegas here) to a target character with different skeleton
		topologies. We measure the retargeting error with the MPJPE

normalised by the height of the skeleton, and provide results per target character (middle) as well as overall average (right-most).

Lower values means higher retargeting accuracy. . . . . . . . . .

Table 5 . 1

 51

Table 5 .

 5

	AdaIN	adaptive instance normalisation
	AE	autoencoder
	AUC	area under the curve
	BCE	binary cross-entropy

3 Root mean squared error (RMSE) of the estimated vGRF normalised by body weight. Estimating foot contacts in addition to vGRFs (Ours-C&F) seems slightly detrimental compared to estimating only vGRFs (Ours). . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 5.4 Median absolute deviation (MAD) in milimetres of centre of pressure (CoP) computed from estimated vGRF components. Similarly to estimated vGRF, modelling foot contact labels in addition to vGRFs (Ours) slightly affects CoP accuracy. . . . . . . . . . . . . . A C R O N Y M S BoS base of support BN batch normalisation BiLSTM bidirectional LSTM CNN convolutional neural network CoP centre of pressure DCGAN deep convolutional GAN DL deep learning DNN deep neural network DOF degree of freedom ELU exponential linear unit ERD encoder-recurrent-decoder FC fully-connected FCRBM factored CRBM FK forward kinematics GAN generative adversarial network GCN graph convolutional network GRF ground reaction force GRU gated recurrent unit GSL Gram-Schmidt-like IK inverse kinematics IMU inertial measurement unit KNN k-nearest neighbours LSTM long short-term memory LReLU leaky rectified linear unit MAD median absolute deviation MLP multilayer perceptron MSE mean squared error MSLE mean squared logarithmic error MPJPE mean per joint position error MPJVE mean per joint velocity error NF normalising flows OPA ordinary Procrustes analysis OT optimal thresholds PD proportional-derivative PFNN phase-functioned neural network PCKh percentage of correct key points normalised with head size RBM restricted Boltzmann machine RMSE root mean squared error RNN recurrent neural network ReLU rectified linear unit SVD singular value decomposition VAE variational autoencoder vGRF vertical ground reaction force ViT vision transformer WGAN Wasserstein generative adversarial network XCA cross-covariance attention

Table 2 .

 2 1 -Summary of the main datasets presented in Section 2.2.2.

	Dataset	URL

Table 2 .

 2 2 -Summary of the methods presented in Section 2.3.1. Miscellaneous data includes hand-crafted, synthetic, proprietary, unspecified or other public datasets. of interest in those fields. For this reason, motion prediction is out of the scope of the thesis. Therefore, we do not cover it in the following (see Article 1 for a comprehensive review of DL-based motion prediction in the context of skeletal character animation).

	Reference	URL

  is further constrained to follow a predefined prior distribution, typically a multivariate normal distribution, endowing the autoencoder with a generative capability. Thus, VAEs provide a convenient way to embed a stochastic component into the generation of human motion.Various approaches have been proposed to extend the VAE framework to the modelling of temporal sequences. Toyer et al.[START_REF] Toyer | Human Pose Forecasting via Deep Markov Models[END_REF] relied on a Deep Markov Model, essentially a VAE in which the latent code is conditioned on its value at the previous time step. Habibie et al.[START_REF] Habibie | A Recurrent Variational Autoencoder for Human Motion Synthesis[END_REF] combined a VAE and an RNN. Motion generation is driven by the random latent code samples, as well as by control variables that constrain the trajectory and velocity of the character. The RNN is conditioned on encodings on these control variables. At inference time, its cell state is initialised with the latent code value. The concatenation of cell state outputs at each time step is fed to the decoder to produce the synthesised motion sequence.

Du et al.

[START_REF] Du | Stylistic Locomotion Modeling and Synthesis using Variational Generative Models[END_REF]

built on the motion graph framework proposed by Min and Chai

[START_REF] Min | Motion graphs++: a compact generative model for semantic motion analysis and synthesis[END_REF]

, in which motion sequences are represented as a graph of motion primitives. The segmentation of motion sequences into primitives is dependent on the type of motion and typically hand-crafted. Autoencoders learn embeddings for each primitive, and the latent codes for these embeddings are further encoded by conditional VAEs trained on dataset samples for the considered primitives. The conditioning of the primitive-specific VAEs ensures that they reproduce the style of the input motion, which is encoded as a

Table 2 .

 2 and other autoregressive models are often considered to be more appropriate than CNNs, as the future pose is predicted from the previous motion as well as a 3 -Summary of the methods presented in Section 2.3.2. Miscellaneous data includes hand-crafted, synthetic, proprietary, unspecified or other public datasets.

	wrapped by 1D temporal convolutional layers. Missing frames between keyframes are
	first interpolated using spherical linear interpolation before being fed to the model,
	which is expected to refine interpolated frames and bring global consistency. Oreshkin
	et al. [131] then also relied on a transformer-based architecture and interpolation of

To tackle this issue, researchers recently investigated transformers which are known to efficiently model long-term dependencies (see Section 2.2.4). Duan et al.

[START_REF] Duan | Single-Shot Motion Completion with Transformer[END_REF]

first used a transformer encoder

2.3.2 Character Control

In this section, we explore the task of controlling character motions that react naturally to user inputs, while accounting for environment constraints, which is another challenge involved in creating believable virtual characters. Character control can itself be sorted into kinematic, physical and biomechanical control. Kinematic approaches directly produce motions as joint angles. In contrast, both physical and biomechanical approaches strive to obey the laws of physics, while differing in the actuation model: physical models are actuated by forces and torques, while biomechanical models (a.k.a. musculoskeletal models) are driven by muscle activations. Moreover, both rely most of the time on deep reinforcement learning which is well-suited when dealing with agents (characters) in a simulated environment. This section only treats kinematic character control, the other two being slightly out of the scope of this thesis (see Article 1 for a comprehensive review of physical and biomechanical character control). We summarise DL-based methods presented in this section in Table

2

.3.

Kinematic character control approaches typically produce motions as joint angles,

based on a set of motion examples and high-level controls (e.g. user inputs, interactions with the environment). Because of the requirement of generating motions in an online fashion when controlling characters in video games or other interactive applications, RNNs control signal. For instance, Lee et al.

[START_REF] Lee | Interactive Character Animation by Learning Multi-Objective Control[END_REF]

used a four-layer LSTM model for controlling characters playing basketball and tennis. However such approaches often tend to fail in the long run, as errors in the prediction are fed back into the input and accumulate, eventually either converging to an average pose or introducing high frequency artifacts.

Table 2 .

 2 4 -Summary of the methods presented in Section 2.4. Miscellaneous data includes hand-crafted, synthetic, proprietary, unspecified or other public datasets.

	Reference	URL

  .1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 Image Inpainting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Motion Prediction and In-Betweening . . . . . . . . . . . . . . . . . 3.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3.2 Data Representation and Notation . . . . . . . . . . . . . . . . . . . 3.3.3 Training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.3.4 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 Joints Upsampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Human Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  which correlations are learnt. Moreover, the sequential processing of RNNs is inefficient since parallelism is little exploited compared to CNNs, which are more efficient, allowing for larger data volumes and longer training times.

	temporal horizon across
	agree on what is the most effective architecture for this task: recurrent, 1D temporal
	convolutional and spatio-temporal graph-convolutional architectures have all been
	leveraged.
	Indeed, both convolutional and recurrent architecture have advantages and disad-
	vantages, as seen in Section 2.2.4. Theoretically, RNNs can model correlations across
	any temporal horizons. However, as they process each timestep of time series one
	after another, training gradients exponentially vanish, limiting the effective maximal

uses different constraints, e.g. on joint locations or body segment orientations, to define specific motion dynamics features that must be preserved or transferred. Retargeting is then performed by numerically solving the corresponding constrained optimisation problem. About two decades later, researchers in character animation tried again to tackle this long-standing problem, relying on deep learning.

As detailed in Section 2.4.2, recent approaches address only morphological retargeting

[START_REF] Kim | Motion Retargetting based on Dilated Convolutions and Skeleton-specific Loss Functions[END_REF][START_REF] Li | An Iterative Solution for Improving the Generalization Ability of Unsupervised Skeleton Motion Retargeting[END_REF][START_REF] Lim | PMnet: Learning of Disentangled Pose and Movement for Unsupervised Motion Retargeting[END_REF][START_REF] Villegas | Neural Kinematic Networks for Unsupervised Motion Retargetting[END_REF]

, except for Aberman et al.

[3]

who address homeomorphic skeleton topologies. Besides the scope addressed, these recent methods mostly dis-

1 .

 1 Anatomical plane that divides the body into superior and inferior sections; see Figure 4.2. 2. Anatomical plane that divides the body into dorsal and ventral sections; see Figure 4.2.

Table 4 .

 4 1 -Overview of the different datasets gathered to train our model.

	Dataset Human3.6M [66] MPI-INF-3DHP [119] 1.3×10 5 Frames Framerate 3.6×10 6 50 Hz 25 Hz AMASS [111] 1.8×10 7 60-250 Hz PSU-TMM100 [148] 1.3×10 6 50 Hz U nder P ressure [125] 4.9×10 6 240 Hz	Size 20.0 h 1.5 h 41.5 h 7.4 h 5.6 h	Joints 25 27 52 17 23	Repr. angular positional angular positional On request Availability On request Public Public angular Public

  4 transformer blocks with 8 heads and 128 features each. Finally, intermediate and final linear layers (see Figure 4.4) have both 128 output features. Similarly to our encoder, templates and latent codes are also embedded at the beginning of our decoder (see Figure 4.5). Skeleton templates are embedded by a 4-layer MLP (16, 32, 64 and 128 output features) with ReLU after each layer except the last. Latent codes are embedded by a CNN made of 3 1D convolutional layers followed by a linear layer (128 output features and ELU activation each). Then, the core of our decoder is also made of 4 transformer blocks with 8 heads and 128 features each. Moreover, a 2-layer MLP with 128 output features and ReLU activation is associated to each block to further independently process latent features before modulating input tokens (see Section 4.3.3). After transformer blocks, a final joint-wise

CNN is used to reconstruct motion sequences. It is made of 3 1D convolutional layers (128 output features) followed by 3 linear layers (128, 128 and 3 output features), with ELU activation after each layer except the last. As explained in Section 4.3.3, transformer blocks in both our encoder and decoder leverage cross-covariance attention. In total, our model has 2'475'511 learnable parameters. stochastic joint subsampling.

Table 4 .

 4 3 -Quantitative evaluation of self-retargeting, i.e. from a source to a target character with the same skeleton topology and body proportions. We measure the retargeting error with the MPJPE normalised by the height of the skeleton, and provide results per character (middle) as well as overall average (right-most).

	Lower values means

Table 4 .

 4 4 -Quantitative evaluation of intra-structural retargeting, i.e. from a source to a target character with the same skeleton topology. We measure the retargeting error with the MPJPE normalised by the height of the skeleton, and provide results per pair of source and target characters (middle) as well as overall average (right-most). G o , M o , M r and V a stand for Goblin, Mousey, Mremireh and Vampire characters, respectively. Lower values means higher retargeting accuracy. Model

Table 4 .

 4 5 -Quantitative evaluation of cross-structural retargeting, i.e. from a source (BigVegas here) to a target character with different skeleton topologies. We measure the retargeting error with the MPJPE normalised by the height of the skeleton, and provide results per target character (middle) as well as overall average (right-most).

	Lower values means higher retargeting accuracy.
	Model	Goblin	Mousey	Mremireh	Vampire	Average
	Aberman et al. [3] Ours -no SJS Ours	0.060 0.072 0.045	0.047 0.128 0.060	0.062 0.076 0.035	0.071 0.076 0.036	0.060 0.088 0.044

  [START_REF] Sadegh Aliakbarian | A Stochastic Conditioning Scheme for Diverse Human Motion Prediction[END_REF] -Visual examples of motion retargeting. A motion sequence (see 1st row, in green) with AMASS skeleton topology A is retargeted to PSU-TMM100 and MPI-INF-3DHP topologies, noted P and M, respectively (see 2nd and 4th rows, respectively, in blue). Then, both are retargeted back to AMASS topology (see 3rd and 5th rows, respectively, in red). Note that our model performs well on MPI-INF-3DHP topology, even though it has never seen it during training.

  kinematics. A CNN called PressNet is proposed to estimate 2D foot pressure maps from joint positions and validated on a novel dataset of tai chi sequences (see Section 5.2.3).To the best of our knowledge, the closest database to the one we propose is PSU-TMM100. Recently released to the computer vision community by Scott et al.[START_REF] Scott | From Image to Stability: Learning Dynamics from Human Pose[END_REF], this dataset provides videos from two views, motion capture markers, body joints and foot pressure recorded with insole sensors. It contains about 7.6 hours of data during which 10 subjects are performing 24-form simplified tai chi. Although similar in terms of scale and nature of the captured data, the database we propose has quite different types of motion. While PSU-TMM100 contains specific tai chi sequences mostly composed of slow body movements with long and stable foot supports, U nder P ressure provides diversified sequences focused on but not restricted to locomotion at different paces including on non-flat environments (see Table5.1), i.e. more challenging conditions for foot contacts detection.

	5.2.3 Foot Contact & Ground Reaction Force Databases
	As of today, motion capture data annotated with accurate foot contact information
	are scarce. Researchers in biomechanics and biomedical engineering have released a
	few databases of motion capture with GRFs e.g. Kulbacki et al. [87], however most
	of these databases are not suitable for animation purposes as they typically focus on
	specific aspects of movement, or target stage and symptoms recognition of diseases in
	pathological subjects.
	CoP and base of support (BoS) are computed from pressure values and used for val-
	idation. Unlike Scott et al., our work specifically targets foot contact detection and
	footskate cleanup, tasks that are relevant to human character animation. We explore
	more diverse motion sequences including different types of locomotion at different
	paces performed in different ways (forward, backward, sideways) as well as sequences
	in non-flat environments such as stair climbing and stepping on solid obstacles.

Table 5 .

 5 1 -Motion sequence categories in U nder P ressure. terrain like going up and down stairs. The detailed composition of our dataset is provided in Table 5.1. Motion capture data for each subject last approximately 34 minutes, for a total of 5.6 hours of motion capture. -Left) Pressure cell layout of Moticon's OpenGo Sensor Insoles [6]. Blue (1 to 4) and red (9 to 16) cells are the groups of cells used to compute heel and toe contacts, respectively. Axes at insole centres represent inertial measurement units. Right) Xsens MVN's [147] motion capture skeleton with 23 joints.

	Category	Motion Type	Duration [mn]
	Locomotion, forward	slow walking normal walking fast walking running	43.9 42.0 42.9 30.1
	Locomotion, backward	slow walking normal walking fast walking running	21.1 21.8 20.8 15.5
	Locomotion, miscellaneous	running sideways hopping stairs 1 at a time stairs 2 at a time	12.3 13.9 18.0 13.9
	Locomotion with obstacles	stepping on obstacles stepping over obstacles jumping over obstacles	5.7 11.4 10.5
	Idle	leg stand-up sit-down crouched down	4.8 4.8 4.8
	Total		338.2
	5.3.1 Motion Capture
	Subjects were equipped with the Xsens MVN Link motion capture system [147]. The
	hardware consists of 17 inertial measurement units (IMUs) running at 240 Hz embedded
	in the MVN Link suit. Each IMU contains a 3D accelerometer, a gyroscope, and a
	magnetometer. Capturing was performed using the Xsens MVN Animate software, an
	engine customised for 3D character animation that combines tracking data of the 17
	individual IMUs with a 23-segment biomechanical model (see Figure 5.2b) to obtain
	segment positions and orientations. We calibrated MVN Animate for each subject with
	height, arm span and shoe length measurements as inputs while other body dimensions
	and proportions were estimated through the calibration, as well as the orientation of

vGRFs Validation Validation Training Joint Positions Ground Truth vGRFs vGRF Estimates

  Since pressure is defined as the perpen-dicular force per unit area, we additionally compute vGRF components by multiplying pressure values by the corresponding cell areas. The motivation here is that groups of vGRFs are easier to aggregate (by summation). We also normalise these values to express them as subject weight proportions. In this work, we consider two contact locations per foot, i.e. heels and toes as commonly done in human animation[START_REF] Holden | Phase-Functioned Neural Networks for Character Control[END_REF][START_REF] Ikemoto | Knowing When to Put Your Foot Down[END_REF][START_REF] Shimada | Neural Monocular 3D Human Motion Capture with Physical Awareness[END_REF]. To compute contact labels, vGRF components are first smoothed with a Gaussian filter to avoid rapidly alternating labels due to threshold effects. Then, smoothed vGRF components are summed per contact location (see red and blue cells in Figure5.2a), rescaled such that the sum of blue and red cell vGRFs is equal to total vGRF (to properly ignore grey cells which particularly suffer from noised measures and can be activated during either toes or heels contact) and then a threshold at 5% of the body weight is applied to obtain raw labels. Finally,

	5.3.3 Post-Capture Processing ground reaction forces from vGRFs. IMUs with peaks computed from motion captured foot positions. Although numerical Captured data include motion sequences, plantar We derive ground truth foot contact labels deterministically differentiation is known to amplify high frequency noise, we found that the framerate of our motion capture data was sufficiently high and measurement noise was small enough for synchronisation. downsampling and trimming After synchronising our data, we downsampled pressure distribution and foot acceleration. foot contact labels motion capture data from 240 Hz to 100 Hz using spherical linear interpolation to

raw labels are discarded whenever per-foot total vGRF (including grey cells) is below 10% of the body weight to avoid false positives triggered by noise. Contact phases shorter than 0.1 s are also discarded for the same reason. In the following, we refer to this binary contact labels calculation as the contact function Γ.

synchronisation

Since we jointly capture motion and foot pressure data with separate devices, our records must be accurately synchronised in absence of a genlock signal. To this end, subjects were asked to perform a simple control movement at the beginning and end of each capture sequence, consisting in an in-place double-leg jump. This allows us to match vertical acceleration peaks measured on pressure insole

  These three models have real-valued outputs, for which positive values are considered as foot contacts with the ground, and are trained with binary cross-entropy (BCE). Finally, we explore two variants of our deep network. First, in Ours-C variant, we adapt our architecture to directly estimate foot contact labels instead of vGRFs by change the last layer and its activation (i.e. number of neurons and sigmoid instead of softplus) and train this adapted model with BCE. Then, in Ours-C&F variant, we combine Ours-C variant with the main proposed model Ours. Our architecture is adapted to output both contacts and vGRF components (convolutional layers are shared while FC layers are duplicated and separately learned for each output) and trained with both MSLE and BCE over vGRF and contacts, respectively. foot contact detection results.Table5.2 reports the F 1 score for each model of our ablative study and each motion category as well as overall results. First, Linear and Linear-Feet show improved performances with respect to the OT baseline, which tends to confirm that thresholds based approaches lack complexity. Increased performances obtained by Linear model with respect to Linear-Feet model suggests that foot contact detection also benefits from other body joints, pointing out one of the limitation of thresholds applied on foot joints. The higher F 1 score of the 3-layer

	MLP with respect to linear models confirms their limitations. Finally the proposed
	architecture further increases the detection accuracy with relatively small differences
	among variants.

Table 5 .

 5 2 -F 1 score on foot contact labels detection of our method, its variants for ablative study purposes, and the OT baseline. Bold and underline respectively indicate per-column best and second best. Our method outperforms the OT baseline and its linear generalisations, and the proposed architecture seems relevant with respect to the 3-layer MLP. Model phase changes are less severe in the sense that they would result in less severe biases or artefacts in most applications, e.g. footskate cleanup. To this end, we provide a finer analysis in the following. In Figure5.4, we plot the F 1 score against an increasing temporal tolerance to detection errors. In Figure5.5, the distribution of false positive rates is given according to a normalised temporal frame where ground truth off-contact phase intervals are mapped to [0, 1]. Both figures confirm the limitations of heuristics approaches represented by the OT baseline whose false positive rate is significant in the middle of off-contact phases. On the contrary, learned models show convergence of accuracy close to 100% with increasing tolerance as well as low false positive rates in the middle of off-contact phases. -F 1 score curves against temporal tolerance. Here we compute the F 1 score with tolerance t ∈ [0, 0.25] by considering contact labels correct whenever they are located at most t seconds away from the closest contact phase. At t = 0, we fall back to the Overall column in Table5.2. The optimal thresholds (OT) baseline and its linear generalisations have large errors far from contact phase changes, resulting in relatively low F 1 scores compared to our model even with high temporal tolerances.

		Walking	Running	Obstacles Hopping	Stairs	Idle	Overall
	OT baseline 0.927	0.869	0.926	0.859	0.882	0.826	0.909
	Linear-Feet 0.936 Linear 0.937 3-layer MLP 0.940	0.863 0.868 0.883	0.921 0.926 0.926	0.824 0.855 0.882	0.906 0.925 0.947	0.812 0.912 0.921	0.913 0.923 0.930
	Ours-C	0.946	0.917	0.941	0.930	0.956	0.941	0.942
	Ours-C&F	0.948	0.918	0.942	0.923	0.954	0.946	0.943
	Ours	0.949	0.930	0.944	0.931	0.959	0.948	0.947
	temporal analysis of foot contact detection.	Up to now, we provided
	temporally global foot contact detection results. However, misdetected labels located
	closer to contact

Table 5 .

 5 3 -Root mean squared error (RMSE) of the estimated vGRF normalised by body weight. Estimating foot contacts in addition to vGRFs (Ours-C&F) seems slightly detrimental compared to estimating only vGRFs (Ours).

	Model	Walking	Running	Obstacles	Hopping	Stairs	Idle	Overall
	Ours-C&F Ours	9.5% 9.1%	14.8% 14.3%	12.4% 11.6%	14.3% 14.1%	11.8% 10.9%	13.1% 11.9%	11.4% 10.9%

Table 5 .

 5 4 -Median absolute deviation (MAD) in milimetres of CoP computed from estimated vGRF components. Similarly to estimated vGRF, modelling foot contact labels in addition to vGRFs (Ours) slightly affects CoP accuracy. Model

		Walking	Running	Obstacles	Hopping	Stairs	Idle	Overall
	Ours-C&F Ours	16.7 13.3	12.3 11.2	17.8 15.1	15.5 12.3	17.9 13.9	28.9 25.9	16.4 13.4
	5.5.3 vGRFs Estimation
	metrics.

Table 5 .

 5 3 gives the RMSE of the estimated vGRFs proportionally to the subjects' body weight while Table5.4 provides the MAD (or median distance to ground truth) of the CoP. Moreover, Figure5.6 depicts the distribution of offsets from predicted to ground truth CoP. As for contact detection, these results suggest that learning to model foot contact labels in addition to vGRF (Ours-C&F) reduces accuracy, although the performances of both variants are close. Note that other variants evaluated on contact detection in the previous section only detect contact labels and hence cannot be evaluated on vGRF estimation. Finally, visual results of vGRFs estimated using our main variant are depicted in Figure5.7.

	Y Offset [mm]
	X Offset [mm]

  discussion on pressure insoles accuracy.In biomechanics, force plates are generally considered gold standard to measure vGRF and CoP; however, the environment in which we captured the proposed database, i.e. spanning over a significant area including obstacles and stairs, prevented us from resorting on force plates to capture force or pressure data. As a result, our evaluation relies on pressure insole measures as ground truth. Existing works in biomechanics[START_REF] Jönsson | Foot centre of pressure and ground reaction force during quadriceps resistance exercises; a comparison between force plates and a pressure insole system[END_REF][START_REF] Nakazato | A Comparison of Ground Reaction Forces Determined by Portable Force-Plate and Pressure-Insole Systems in Alpine Skiing[END_REF] evaluated pressure insoles accuracy with respect to force plates, and tell us that vGRFs measured with pressure insoles suffer from a RMSE up to approximately 10% of the subject weight, being subject to variations depending on experimental conditions. Thus, Table 5.3 indicates that vGRF estimation errors of our deep neural network are approximately of the same order of magnitude as the measurement error this is expected with pressure insoles. To this end, we purposely introduce three types of artefacts in motion sequences from our database such that foot contact labels can still be used as ground truth to measure contact detection performance. First we add Gaussian noise to joint positions. Second we introduce distortions caused by going through a partially trained motion autoencoder. Finally, we generate footskate artefacts by blending different motion sequences from our database. Figure 5.8 -F 1 score on foot contacts detection from motion sequences purposely noised with additive isotropic Gaussian noise. Each curve represents the F 1 score against the amount of noise introduced, measured with the MPJPE in centimetres indicated by the bottom horizontal axis, while the top horizontal axis gives the corresponding standard deviations of the Gaussian noise. The results indicate that our method is more robust to noise.

						Standard deviation [cm]
		1	0	1		2	3		4		5
		0.9
		0.8
		0.7
	1 score	0.5 0.6
	F	0.4
		0.3
		0.2					OT baseline		Linear-Feet	Linear
		0.1					3-layer MLP Ours		Ours-C	Ours-C&F
		0	0	1	2	3	4	5	6	7	8
							MPJPE [cm]

5.5.4 Foot Contacts Detection in Challenging Conditions

As explained in Section 5.2, applications requiring foot contact labels detection include quantifying or correcting foot artefacts. By definition, motion sequences considered in such applications are expected to be perturbed at least by footskate artefacts, and sometimes by other artefacts like noise or distortions. As a consequence, the performance of a method for foot contact detection is not only relevant on clean motion capture data (as evaluated in Section 5.5.2), but also on motion sequences suffering from perturbations of all kinds. In this section, we evaluate the performance of our model compared to our baseline and ablative variants of our model on perturbed motion sequences.

  -F 1 score on foot contacts detection from motion sequences distorted by a motion autoencoder early-stopped at different epochs to emulate different amounts of distortion. Each curve displays the F 1 score against the amount of distortion introduced, measured with the MPJPE in centimetres. The results indicate that our method is more robust to distorted sequences. F 1 score curves obtained against the amount of perturbations measured at each of those epochs using the MPJPE. F 1 score against amount of footskate. Test motions with matching foot contact patterns have been blended to purposely introduce footskate while preserving contact labels. Foot contact detection is then evaluated on such motions suffering from footskate, and compared to the ground truth contact labels. The amount of footskate introduced through blending is measured using the mean velocity of the feet during contact phases. Non-linear learned detection models keep reasonably high accuracy while accuracy of linear models significantly decrease and accuracy of optimal thresholds (OT) baseline quickly collapse. To do so, we randomly picked pairs of chunks from motion sequences in our test set having identical foot contact patterns on either left or right foot. Making the reasonable hypothesis that foot contact patterns are preserved when motion sequences with identical foot contact patterns are blended together, we can then blend our random pairs of chunks and thus introduce footskate artefacts while still having ground truth foot contacts needed to evaluate foot contact detection. We constituted a set of approximately 40 thousands blended chunks of 80 frames long (i.e. 0.8 s) and quantified the amount of footskate introduced using the mean horizontal velocity of the feet during contact phases[START_REF] Yu | Character Controllers using Motion VAEs[END_REF][START_REF] Starke | Local Motion Phases for Learning Multi-Contact Character Movements[END_REF][START_REF] Starke | Neural State Machine for Character-Scene Interactions[END_REF][START_REF] Zhang | Mode-Adaptive Neural Networks for Quadruped Motion Control[END_REF]. Then, we attempt to detect foot contacts using the same models as before, and measure the performances with the F 1 score. Finally, we obtain and plot corresponding F 1 score curves in Figure5.10 using simple moving average.As depicted in Figures 5.8 to 5.10, the proposed approach for foot contact labels detection is much more robust than threshold-based heuristics approaches represented by the OT baseline, regardless of the type of perturbation applied to motion sequences.Moreover, the improvement of modelling vGRFs instead of foot contact labels (Ours vs Ours-C) is much larger when facing perturbed motion sequences. Since vGRFs are much more related to motion dynamics than binary contact label, vGRFs estimation requires a deeper understanding of motion than contact labels detection. Indeed,

		1
		0.9
		0.8
	F 1 score	0.7	1 score	OT baseline OT baseline
		0.6		Linear-Feet Linear Linear-Feet Linear
				3-layer MLP 3-layer MLP
		0.5		Ours-C Ours-C&F	Ours-C Ours-C&F
				Ours	Ours
	0.25 Figure 5.9 0 Figure 5.10 -footskate generated with motion blending. 0.5 0.75 1 1.25 0.4 Amount of footskate, i.e. mean velocity of the feet during contact phases [m/s] 1.5 To push further our eval-1.75 2
	uation toward concrete applications, we additionally evaluate contact detection on
	motion sequences that have been blended. Motion blending (or motion interpolation)
	is a well-known technique widely used in animation that consists in mixing existing
	motions with dynamic blending weights to create new motions. In our case, motion
	blending is interesting because it is known to easily introduce footskate artefacts in
	resulting motions, which can then help us to evaluate contact detection on such per-
	turbed motions.

motions distorted by an autoencoder.

Second, to simulate perturbations more faithful to what is encountered in real-world applications, we evaluate detection performances on motion sequences that have been encoded and decoded by an autoencoder. To this purpose, we trained a convolutional autoencoder similar to the one proposed by Holden et al.

[START_REF] Holden | A Deep Learning Framework for Character Motion Synthesis and Editing[END_REF]

to reconstruct motion sequences from our test set for 100 epochs (about 10 minutes). Then we evaluate foot contact detection on motion sequences which have been encoded and decoded with our autoencoder. Moreover, to emulate different level of perturbations, we leveraged the fact that the amount of perturbations introduced when passing through the autoencoder continuously decrease during training. Hence, we ran this evaluation scheme at different epochs and plot in Figure

5

.9

approximately 90% of contact labels can be correctly detected with foot position and velocity thresholding (see Table

5

.2), i.e. with almost no understanding. Then, in challenging conditions like perturbed input motion sequences, the performances of the variant modelling vGRFs are logically more stable since a deeper understanding of motion is intuitively more robust.

https://en.wikipedia.org/wiki/Mixamo

L I S T O F TA B L E S