H Wallach

H Larochelle

A Beygelzimer

F D'alché-Buc

E Fox

Cette thèse, non contente de ne s'être écrite toute seule ou un jour, est le fruit de riches et nombreuses interactions, tant professionnelles que simplement humaines, du court au long terme et par tous les moyens, avec une floppée de personnes qu'il est bon de citer.

Je tiens tout d'abord à remercier mes directeurs: Olivier et Matthieu à Jussieu, pour les longues discussions scientifiques et stylistiques; Frédéric et Jean-Yves à Limours, pour leur expertises respectives; à tous, pour leurs encouragements et bienveillance vis-à-vis d'un thésard parfois tête-en-l'air.

Je remercie également mes collègues en tous genres, colorant le quotidien d'un arc-en-ciel collectif.

Je n'oublie non plus Christelle et Gilles à l'ONERA, sans qui cette thèse n'a pas commencé.

J'adresse ma reconnaissance au groupe de travail OTAN SET245, qui a bien voulu transmettre une base de données radar riche en contenu, largement exploitée lors de ces travaux.

Enfin, je remercie mes proches, sans qui rien de tout cela n'aurait de sens. Mes parents, dont le soutien sans faille m'a toujours porté de l'avant, mes amis, collègues et Yann, certains voisins qui se reconnaîtront, Miel, la cuisine basque, les danseurs et danseuses. Tout particulièrement, je remercie de plein coeur Céline pour m'avoir supporté et accompagné avec patience et tendresse. v C h a p t e r 1 I N T R O D U C T I O N 1.1

Context

) to mastering the game of Go (AlphaGo Zero 2019), progress has been phenomenal and its usage sprawling to an ever more diverse field of applications. Historically, many of the leading innovations in the field were and are primarily designed for Computer Vision, i.e. the field of image analysis. Since the first applications by

A B S T R A C T

Machine Learning, and in particular Deep Learning, is a powerful tool to model and study the intrinsic statistical foundations of data, allowing the extraction of meaningful, human-interpretable information from otherwise unpalatable arrays of floating points. While it provides a generic solution to many problems, some particular data types exhibit strong underlying physical structure: images have spatial locality, audio has temporal sequentiality, radar has time-frequency structure. Both intuitively and formally, there can be much to gain in leveraging this structure by adapting the subsequent learning models. As convolutional architectures for images, signal properties can be encoded and harnessed within the network. Conceptually, this would allow for a more intrinsic handling of the data, potentially leading to more efficient learning models. Thus, we will aim to use known structures in the signals as model priors. Specifically, we build dedicated deep temporal architectures for time series classification, and explore the use of complex values in neural networks to further refine the analysis of structured data.

Going even further, one may wish to directly study the signal's underlying statistical process. As such, Gaussian families constitute a popular candidate. Formally, the covariance of the data fully characterizes such a distribution; developing Machine Learning algorithms on covariance matrices will thus be a central theme throughout this thesis. Statistical distributions inherently diverge from the Euclidean framework; as such, it is necessary to study them on the appropriate, curved Riemannian manifold, as opposed to a flat, Euclidean space. Specifically, we contribute to existing deep architectures by adding normalizations in the form of data-aware mappings, and a Riemannian Batch Normalization algorithm. We showcase empirical validation through a variety of different tasks, including emotion and action recognition from video and Motion Capture data, with a sharpened focus on micro-Doppler radar data for Non-Cooperative Target Recognition drone recognition. Finally, we develop a library for the Deep Learning framework PyTorch, to spur reproducibility and ease of use. i

R É S U M É

L'apprentissage automatique, et en particulier l'apprentissage profond, unit un arsenal d'outillages puissants pour modeler et étudier les distributions statistiques sous-jacentes aux données, permettant ainsi l'extraction d'informations sémantiquement valides et interprétables depuis des séquences tabulaires de nombres par ailleurs indigestes à l'oeil humain. Bien que l'apprentissage fournisse une solution générique à la plupart des problèmes, certains types de données présentent une riche structure issue de phénomènes physiques: les images ont la localité spatiale, les sons la séquentialité temporelle, le radar la structure temps-fréquence. Il est à la fois intuitif et démontrable qu'il serait bénéfique d'exploiter avec astucieuse ces formations fondatrices au sein même des modèles d'apprentissage. A l'instar des architectures convolutives pour les images, les propriétés du signal peuvent être encodées et utilisées dans un réseau de neurones adapté, avec pour but l'apprentissage de modèles plus efficaces, plus performants. Spécifiquement, nous oeuvrerons à intégrer dans la conception nos modèles profonds pour la classification de séries temporelles des sur leurs structures sous-jacentes, à savoir le temps, la fréquence, et leur nature proprement complexe.

C O N T E N T S

L I S T O F F I G U R E S

Chapter 1: introduction The proposed architecture for radar signal classification . . Evolution of the scattering points and normals for the Vario, Phantom2 and S1000+ drones . . . . . . . . . . . . . . . . . Figure 3.12 Spectrograms of noisy and uncluttered signals . . . . . . . Figure 3. [START_REF]NVIDIA/pix2pixHD[END_REF] Classification accuracies for three learning models . . . . . Figure 3. [START_REF] Tao | Unsupervised Spectral-Spatial Feature Learning With Stacked Sparse Autoencoder for Hyperspectral Imagery Classification[END_REF] Performance of the learning models for increasingly good conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3. [START_REF] Liang | Hyperspectral Imagery Classification Using Sparse Representations of Convolutional Neural Network Features[END_REF] Growth of classification confidence over time in a very challenging environment (SN R = 10dB and P RF = 2kHZ) . . Figure 3. [START_REF] Tupin | Detection of linear features in SAR images: application to road network extraction[END_REF] Growth of classification confidence of a Fully Temporal Convolutional Network (FTCN) in audio recognition . . . Chapter 4: advances in spd neural networks 3. 1 The parameters are found or estimated from drone specifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table 3. 2 Simulation parameters and their default values . . . . . . . Table 3. 3 Performance comparison of real and complex deep structures on radar data on various amount of noisy data . . . . Table 3. 4 Performance comparison of real and complex deep structures on radar data on various amount of less noisy data . Table 3. 5 Performance comparison of first-and second-order models on radar data . . . . . . . . . . . . . . . . . . . . . . . . . . . More specifically, most of the frenzy in this paradigm shift revolves around Deep Learning, which is a subclass of Machine Learning mostly involving Deep Neural Networks. In short, Deep Neural Networks build a latent representation of data through the learning of a hierarchy of layers; as such, Deep Learning can be seen as a generalization of certain classes of standard Machine Learning algorithms, by jointly learning feature representation space and separation within a single architecture. A major subgroup of Machine Learning is the field of Supervised Learning, where a dataset of pairs of inputs and outputs is given to allow the learning of a decision function through the optimizing of a loss function. While Deep Learning has successfully been used in the context of supervised classification, there exists a plethora of different possible tasks. These may include Visual Question Answering, where the inputs are couples of images and text, image captioning, where the output given an image is text, object or instance segmentation, where the output is bounding boxes or a list of pixels given an input image. While each one involves task-specific engineered model and loss functions, the general principle remains the same: to learn a decision function on a dataset of inputs and outputs, based on the outputs of a loss function.

Despite the popularity of Computer Vision, the instrinsic genericity of Deep Learning makes it a suitable match for scores of other applicative fields such as health, neuroscience, automated systems and robotics, social network studies, Brain-Computer Interfaces, audio, radar, lidar... Of these, some closely follow pure Computer Vision, such as hyperspectral [START_REF] Tao | Unsupervised Spectral-Spatial Feature Learning With Stacked Sparse Autoencoder for Hyperspectral Imagery Classification[END_REF]; Liang and Q. [START_REF] Liang | Hyperspectral Imagery Classification Using Sparse Representations of Convolutional Neural Network Features[END_REF] or Synthetic Aperture Radar imaging [START_REF] Tupin | Detection of linear features in SAR images: application to road network extraction[END_REF][START_REF] Tupin | A contrario comparison of local descriptors for change detection in Very High spatial Resolution (VHR) satellite images of urban areas[END_REF][START_REF] Atto | Multidate Divergence Matrices for the Analysis of SAR Image Time Series[END_REF]). However, some types of data exhibit a rich structure within their inherent construction, as for example the case of micro-Doppler radar data. Here, the stream of physical processes involved in the signal formation shape it in a visually recognizable fashion (Figure 1.1). Leveraging the resulting structures within the associated learning models thus seems a promising idea.

The usage of micro-Doppler radar data is gaining momentum in diverse branches of the industry, such as in autonomous vehicles (Inman, M. 2020) or fine-grained gesture recognition in smartphones [START_REF] Lien | Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar[END_REF]; S. [START_REF] Wang | Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum[END_REF]. Radar roots its historical foundations in the theoretical works of [START_REF] Maxwell | VIII. A dynamical theory of the electromagnetic field[END_REF], H. [START_REF] Hertz | Die Kräfte electrischer Schwingungen, behandelt nach der Maxwell'schen Theorie[END_REF] and N. Tesla [START_REF] Page | The Early History of Radar[END_REF], then grounds its preliminary applications with the notable patents of G. Marconi and C. Hülsmeyer (Fahie 1899; Blanchard 2019). Although radar has been key in its proven ability to detect potentially hostile airborne crafts, the recent emergence of less conventional, smaller and erratic-behaving Unmanned Aircraft Vehicles confronts the classification task of radars to an unprecedented challenge. Unrecognized drone sightings over airports ("Gatwick 'drone sighting' diverts flights" 2019) or nuclear plants [START_REF] Bouchaud | Drones Have Been Spotted Flying Over French Nuclear Power Plants[END_REF] bring a new kind of threat to civilian security, while military operations both harness ("Top Iranian general killed by US in Iraq" 2020) and heed the imminent danger of stealthy strikes [START_REF] Press | Russia implies US involvement in drone strikes on Syria military bases[END_REF]; "Syrian army foils drone attack on military base in northwest" 2019), to the point of redefining strategical paradigms and international law altogether (Boyle 2013; Shah 2014). In this modern perspective, new approaches with a focus on versatility, efficiency and scalability should provide key elements to tackle these issues. This thesis focuses on Deep Learning in the Supervised Learning setting, also termed deep supervised learning. It brings particular attention to developing models adapted to the underlying structure of the data. Furthermore, experimental approaches thus far, while spanning a variety of different applicative fields, mainly focus on micro-Doppler radar data. We motivate the work thereupon below.

Motivations

The core research in industry is shifting towards Artificial Intelligence, for better or worse. Indeed, alongside insightful and game-changing applications of Deep Learning, some vulnerabilities, or even failures leading to real-world fiascos, have tainted the otherwise gleaming aura of the field. One can think of, for instance, Microsoft's chatbot Tay which became a genocidal racist after some hours on Twitter [START_REF] Vincent | Twitter taught Microsoft's friendly AI chatbot to be a racist asshole in less than a day[END_REF]), Amazon's recruiting tool's strong bias to recommend Caucasian males ("Amazon scraps secret AI recruiting tool that showed bias against women" 2018), or Florida's crime recidivism software unreasonably over-biased against minority communities (Julia Angwin 2016). Even in simple Computer Vi-sion tasks, models with outstanding performance are not necessarily as robust as we may believe: as shown in [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF])'s seminal work, slightly perturbing the image with no visual impact on humans can quickly cause a seasoned model to completely fail at recognizing the initial object. At the root of this major issue, one can argue that data representation plays a key role: indeed, while humans naturally infer concepts, causality and relationships from given data, all the models "see" are matrices of floating numbers; the mathematical space spawned by these data points is thus a true corner stone in building robust and trustable models (Tackling bias in artificial intelligence (and in humans) | McKinsey 2019). While the fact that Deep Learning's unchallenged performance is matched but by our lack of understanding of that performance has been a long-standing meme in the community (see Figure 1.2), these seemingly instrinsic problems further accent the need for a better understanding of how models interact with data. The shift from Deep Neural Networks to Convolutional Neural Networks in Computer Vision is a paramount example of architectures adapting to their input data: while standard Deep Neural Networks feature fully-connected, or dense layers (i.e. full Matrix Multiplication), Convolutional Neural Networks replace the latter with convolution banks, which can be seen as sparse, locally connected Matrix Multiplications. Contrary to a full Matrix Multiplication, a convolution harnesses the spatially local structure of images, producing much better results with many less parameters, with an additional level of robustness to perturbation in the data: indeed, convolutions are formally equivariant to translation.

Aiming for more instrinsic representations of data and designing adapted models with a notion of robustness will be a common leitmotiv throughout this thesis. Simply put, this issue can be explored both through data pre-processing and model design. Again, micro-Doppler data will constitute a illustrative example of mult-faceted structured data. We give a brief overview of these two complementary approaches.

Pre-processing data for better representativity

The idea of processing raw data to a more intrinsic representation is by no means recent, as indicated in the early literature of Computer Vision. Indeed, its first and foremost champion probably is the Scale-Invariant Feature Transform [START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF]), an image representation algorithm integrating into a single vector information about color, gradients, angles, such that it be robust to scale, rotation, translation and illumination. Even earlier, the idea of deformable kernels, or steerable filters was developed in the same objective of robustness of the representation to known transformations [START_REF] Perona | Deformable Kernels for Early Vision[END_REF]. These ideas secured a rich progeny all the way to modern Computer Vision, as we will see below.

Again, this idea is by no means limited to Computer Vision. In the analysis of structured time series, such as audio, or radar data, the Fourier transform (Shannon C. E. 2013) -from here on out, we will rather refer to the fast implementation ubiquitously used, the Fast Fourier Transform -is the most used representation, rather than the raw time data. In the Digital Signal Processing literature, there exists an abundance of alternative representations, most of them with a strong theoretical link to the Fourier transform. The Capon spectrum introduced by Capon (1969) mimicks the behaviour of the Fourier transform while adding constraints to improve on the time-frequency resolution, inherently limited by Heisenberg's incertainty principle [START_REF] Heisenberg | Les Principes Physiques de la Théorie des Quanta[END_REF]. In the same principle, the Wigner-Ville spectrum, instance of the so-called Cohen class of time-frequency operators (L. Cohen 1989), allows for a more fine-grained resolution as illustrated in Figure 1.3. Other approaches build hierarchical frequencies at different time scales, circumventing the resolution problem altogether; the founding innovation in that regards is the notion of wavelets [START_REF] Mallat | A Wavelet Tour of Signal Processing[END_REF], which are now ubiquitous in signal compression.

A tightly-linked notion [START_REF] Flandrin | Time-Frequency/Time-Scale Analysis[END_REF], that of covariance, is also a powerful representation of the data, albeit less used. We now dwell a bit longer on the covariance representation of structured time series data, as it will be a central theme throughout this thesis. Conceptually, the Sample Covariance Matrix captures temporal fluctuations in a compact and meaningful way. In other words, the Sample Covariance Matrix is a powerful discriminative feature representation for structured time series, and deserves the exploration of adapted Machine Learning models. This holds even more for signals with strong physical structure; again, one can think of radar data, Brain-Computer Interface, Magnetic Resonance Imagery or audio.

Designing architectures for powerful representation learning

It is broadly accepted than one main factor for what is now known as the "Artificial Intelligence winters" is the lack of data and computing power to successfully train competitive models.

Enforcing constraints for intrinsic data modelling

The quest to models intrinsically adapted to their input data has thus become an ongoing pursuit in the Deep Learning community. Indeed, since the a Deep Neural Network learns its own feature representation through the hierarchy of layers, clever engineering of these layers hopefully leads to more intrinsic representations of the data. This engineering often draws inspiration from the older data processing techniques mentioned above, transferring them in Deep Neural Networks in a differentiable fashion. This is for instance embodied in part-based image segmentation [START_REF] Mordan | Deformable Part-based Fully Convolutional Network for Object Detection[END_REF] or steerable Convolutional Neural Networks (T. S. Cohen and Welling 2016). This idea of models being invariant or robust to a given class of transformations, is key to learning better representations, as illustrated in Figure 1.5: a learning model should be expected to recognize that both images feature the exact same content, although deformed through given transformations.

In a similar vein, other works design layers to stabilize models through normalization, which can be seen as explicitly enforcing invariance to scale and translation within the network layers. The seminal effort in this direction is found in the introduction of Batch Normalization in Deep Neural Networks (Ioffe and Szegedy 2015). Further works have improved upon the idea, such as in Layer Normalization (J. [START_REF] Xu | Understanding and Improving Layer Normalization[END_REF] . Others have studied the generalizability of such normalizations across different domains (X. [START_REF] Wang | Transferable Normalization: Towards Improving Transferability of Deep Neural Networks[END_REF]) . The recurring motivation behind inner normalizations is to reduce the network's dependence the representation space covariate shift at each layer. Digging even deeper within the elder literature, modern works have made use of Information Geometry. Whether it be in graph analysis (Bronstein et [START_REF] Rao | Information and the Accuracy Attainable in the Estimation of Statistical Parameters[END_REF], [START_REF] Fréchet | Sur l'extension de certaines evaluations statistiques au cas de petits echantillons[END_REF] and [START_REF] Amari | Information Geometry and Its Applications[END_REF], "is an interdisciplinary field that applies the techniques of differential geometry to study probability theory and statistics. It studies statistical manifolds, which are Riemannian manifolds whose points correspond to probability distributions." (Information geometry 2019). The core idea of using Information Geometry in Machine Learning is to model the individual data points as the underlying statistical distribution they are sampled from, instead than simply considering a Euclidean structure to the data. Fundamentally, these distributions form a differential manifold, which then constitutes the subject study of the subsequent Machine Learning algorithm. A key example lies within Exponential Familys, more specifically the family of Gaussian distributions. Indeed, many natural phenomena can be reasonably modelled as the realization of a Gaussian law, characterized by the first-and second-order moments, i.e. the mean and covariance: whenever some form of structure is involved, whether temporal, spatial or more abstract relational principles, moments can naturally found within the sample data.

To this day, learning models based on Information Geometry do not yet enjoy the popularity of Deep Learning methods. However, some particular instances seem to be known by a relatively larger audience in the general field of Machine Learning. The Fisher kernel method for instance [START_REF] Sánchez | Image Classification with the Fisher Vector: Theory and Practice[END_REF]) has know success in general Machine Learning, including Bag of Words methods for image retrieval [START_REF] Jegou | Aggregating Local Image Descriptors into Compact Codes[END_REF]. Another method named natural gradient, introduced by Amari (1998), aims at modifying the gradient of an optimization problem by fitting it to the manifold spanned by the problem's parameters. It is nowadays not uncommon to see it used to regularize the learning in Deep Neural Networks (the term "neuromanifold" is then used), and has since then enjoyed an upsurge in interest (Pascanu and Bengio 2014; Marceau-Caron and Ollivier 2016; Marceau-Caron and Ollivier 2017). While both methods rely on Information Geometry, they considerably differ in terms of purpose and realization. Given this information, it becomes important to distinguish two broad learning settings making use of Information Geometry. In the first one, its theoretical background is summoned to refine "traditional" methods; it is the case for the Fisher kernel and the natural gradient. In the second scenario, objects with a particular geometry, i.e. belonging to a Riemannian manifold, formally require the tools of Information Geometry for Machine Learning methods to be built upon them. Henceforth, because this thesis will attempt to study structured time series data with an underlying physical geometry, we will focus on the second class of settings.

Machine Learning models within the Information Geometry framework understandably lag behind the complexity and diversity found within the standard Euclidean framework. As argued above, filling this gap is a pressing matter, especially for fields where much is to gain when exploiting the strong underlying structures. Most of this thesis will thus deal with developing such models; specifically, the focus is set on the manifold of Gaussian distributions, with applications to a variety of fields. Next section details the particular proposed contributions.

1.3

Contributions and outline

For the reasons above, the works presented throughout this thesis bring particular care to developing Deep Learning models adapted to the underlying form of the data, incorporating these structural priors within. We both explore standard deep architectures, and less explored Deep Neural Networks on covariance matrices, rooted in Information Geometry. Specifically, we take inspiration from Computer Vision's intuitions and translate them to a different field of application -mainly, micro-Doppler radar. Furthermore, we expand upon existing architectures on covariance matrices to seek general improvement in a variety of tasks -in addition to radar, emotion classification from video, action recognition from Motion Capture data.

¢ Chapter 2: theoretical background

The presented works require cross-overs from a variety of fields. Thus, we first go over the core aspects of radar data formation, Machine Learning, and Information Geometry, with a focus on the concepts which will then be further utilized.

¢ Chapter 3: second-order pipeline for temporal classification This chapter introduces various learning models on structured time series data. We show how knowledge of its physical morphology can help design adapted deep models; specifically, we present:

1. A Fully Temporal Convolutional Network (FTCN), a Deep Learning model operating on real-valued spectrograms of temporal data;

2. An Information Geometry-based Deep Neural Network operating on covariance matrix representations of the time series;

3.

A complex-valued version of either model above, allowing to generalize the learning to complex-valued time series.

Finally, we show how these models can be combined into a single pipeline for structured time series classification, called Second-Order Fully Temporal Network (SOFTNet). We report empirical properties of all introduced models through experimentations on micro-Doppler radar data.

¢ Chapter 4: advances in spd neural networks Here, we set our focus entirely on deep models for covariance data, or SPD neural networks: at a high level, we improve upon the existing art by introducing geometry-and data-aware normalizations within the network.

We first propose the Data-Aware Mapping Network (DAMNet) architecture, which generalizes the mathematical framework leveraged by the original SPD neural network. In both theory and practice, a DAMNet allows to refine both the inference and training by using the input data's underlying geometric information, by refining the Euclidean mapping stage at network's final layer.

Then, we develop the Riemannian equivalent, for SPD neural networks, to the well-known Batch Normalization (BatchNorm) algorithm in Deep Neural Networks, leading to an architecture we dub Batch-Normalized SPDNet (SPDNetBN). Using the intuitions from the DAMNet architecture which it further generalizes, and additonal concepts of Information Geometry, we again show improvement over the previous methods. Finally, we propose an incipient 1D convolutional layer for SPD matrices.

Throughout this chapter, various applicative fields are set forth; though radar data remains the main focus, we set up experiments on video and Motion Capture data, respectively for emotion and action recognition. Finally, we present an implementation of all proposed algorithms as library within the renouned PyTorch Deep Learning framework.

1.4

Related publications

This thesis is based on the material published in the following papers: C h a p t e r

@ D.

T H E O R E T I C A L B A C K G R O U N D

Chapter abstract

This chapter details the various fields of interest for the work at hand. The primary applicative domain, radar signal formation and processing, will be covered at first. Conceptually speaking, radar signals posses a rich underlying physical structure; as such, many mathematical representations, whether equivalent or not, aptly capture the signal's many properties. The key point is, while these different representation spaces feature a common ground, their structure may differ to a point where a thorough statistical analysis would require an altogther different learning framework. Specifically, we will focus on three main ways of representing a raw radar signal: the raw received radar wave, the Fourier transform of the wave (or more generally, any timefrequency representation), and the covariance of the wave. As such, concepts of Euclidean Machine Learning (ML), along with the fundamental algorithms used on the data, will follow the radar description. Then, we propose an introduction to Information Geometry (IG), which constitutes the rigorous, Riemannian framework for the analysis of the covariance representation, and finally give an overview of existing Riemannian ML methods.

Introduction

The classification of structured time series using methods of both Euclidean and Riemannian Machine Learning (ML) involves a rather involved comprehension of several different fields.

We begin this chapter with a description of micro-Doppler (µ-D) radar data, which both acts as a glowing instanciation of structured time series, and as the main applicative focus in our works. In a nutshell, a radar signal is formed by the reflection of an emitted wave on a target. The emitted signal is itself designed in a specific manner to assuage inherent inner physical constraints, and to optimize the reflected signal's resolution and ambiguity. A key feature in the received signal lies within the Doppler effect (Doppler 1842), from which speed and frequency information is deductible. By briefly explaining the core physical formation process of the signal, we hope to convince the reader that much is to gain by leveraging its underlying physical structure in learning models; by revealing the rich variety of representations arising from such structure, we wish to inspire the reader into believing this variety can also be incorporated within.

We then turn to the fundamentals of ML and its overall contextualization; in particular we distinguish between Supervised Learning (SL) and Unsupervised Learning (UL) (is the task given labelled examples to learn upon or not), and parametric and non-parametric learning (whether a parameterized distribution models the task or not). We rapidly tunnel our focus to Deep Learning (DL) as it will be the main algorithmic class we ambition to build upon, and specifically target our attention to the convolutional architectures initially developed in the context of Computer Vision (CV), and in particular Fully Convolutional Networks (FCNs), which heavily inspire the works presneted in Chapter 3.

We then switch to an overview of the field of Information Geometry (IG). The overall goal of this section is the introduction of mathematical framework allowing the manipulation of statistical distributions. In the process of doing so, we also wish to convey the fundamental differences between Euclidean, vectorial objects, and the more delicate members of Riemannian manifolds. In essence, we showcase useful tools for the following section, dedicated to... Riemannian ML. In this final section, we aim at providing insight into algorithmic methods adapted from common-knowledge ML to the Riemannian realm of IG, through the demonstration of several simple learning algorithms. The objective is to pave the way to both following chapters (Chapter 3 and Chapter 4) by set-ting a reasonably comprehensive contextualization of the proposed contributions within.

2.2

Radar Signal and Simulation

Radar Core Concepts

A standard radar consists in both an emitter and a receiver. The former emits a signal at a given wavelength, or more generally a given waveform (typically a linearly evolving frequency as depicted in Figure 2.1), while the latter is left to interpret all incoming signals. Let us consider a simple model where the emission frequency f e is kept constant. The base waveform is itself emitted repeatedly, at every time intervall called Pulse Repetition Intervall (PRI), the inverse of which is called Pulse Repetition Frequency (PRF). The latter can be seen as the sampling frequency. This dual system induces many problems to be dealt with such as ambiguities in distance and velocity or compromises in time versus distance resolution. Different compromises to solve different problems lead to a wide variety of radars which exist for different purposes: active radars combine emitter and receiver while passive radars only receive estranged signals, surveillance radars span a wide area of space by being poorly resolved while tracking radars sweep only a small portion of space to gain resolution on targets, antennaes are either fixed for a longer integration time or rotating for a better coverage, and the list goes on. The type of target also contributes to the choice of radar parameters; for instance, targets with smaller Radar Cross-Section (RCS) (see Figure 2 While our work does not focus on a specific set of radars, it does give special attention to Unmanned Aircraft Vehicles (UAVs). These particular objects exhibit shared characteristics, such as being blade-propelled or fix-winged (contrary to birds, for instance), small (contrary to wind turbines) and relatively symmetric RCS (contrary to airborne missiles), and potentially rapidly varying radial velocity V R (contrary to commercial airplanes). • Radial velocity V r , the amplitude of the UAV's body's velocity vector projected on the axis linking it to the radar. The Vario helicopter, DJI's Phantom2, a quadrocopter, and S1000+, an octocopter. Scale is not respected.

As we can see, radar detection and classification is a task highly dependent on the parameters involved; to make a first analogy with CV, the finesse required to obtain "good" signals can compare to finetuning an ancient camera to particularly harsh photographic conditions. The catch is that no camera will ever be capture a drone hidden in a forest several kilometers away, while a properly calibrated radar will. Here we give a non-exhaustive list of intrinsic problems which are bound to limit our ability to aptly classify objects, which will hence become matter of attention in future study of radar models:

• Signal-to-Noise Ratio (SNR): classification can only be possible when the signal is actually detected; a scalar quantity which determines a detection threshold is the Signal-to-Noise Ratio, expressed in decibels, which represents how high in amplitude the signal is w.r.t. the background noise; Here, a PRF of 8kHz on a signal of 250ms yields a time series of 2000 complex points, jointly drawn as modulus and real part.

• PRF: the sampling frequency should by all means be high enough to avoid frequency amibiguities; indeed, according to the well-known Shannon condition (or Nyquist condition), the spectrum of a signal sampled at a lower rate than the maximum frequency of the signal is bound to fold on itself, thereby somewhat corrupting the spectrum; for instance, in classifying blade-propelled UAV, the PRF should ideally be proportionally higher than the fastest point on the UAV (typically the blade tip);

• f e : the emission frequency range determines how finely we resolve variations in frequency between signals along with the spectral bandwidth; also considering a blade-propelled UAV, f e should also be both proportionally higher than the fastest blade tip, but not so high as to induce spectral foldings.

Representation of radar Signals

As discussed in the previous section, a radar signal is the result of an emitted wave reflected off a target, sampled at a given frequency, which yields a numerical time series of complex points (amplitude and phase), as illustrated in Figure 2.4. This will serve as our base representation.

A good representation is the first step to any statistical inference, be it classification or feature analysis. In our case, the signal has an intrinsic time-frequency nature, an intermediary step being a pure frequency representation, i.e. a Fourier transform. A mere spectrum already allows for human interpretation and feature • To the right: Zoom-in of the original signal;

• Top left: Complete Fourier transform (P RF = 8kHz, SN R = 50dB), where we define B as the signal's bandwidth (the plateau in the spectrum);

• Bottom left: Zoom-in of the spectrum, where we define ν the observed period of its assumed periodicity.

Then we have ν = N b Ω and B = 4L b Ωfe c , which gives access to the three unkowns up to a hypothesis on the number of blades, clean analysis pending. Note that for the sake of clear figures, the signal representation parameters such as as the PRF and the SNR were set to wishfully accomodating values. It is also important to point out that a gain in SNR of 10 log(n f f t) ≈ 15dB is achieved in performing the Fourier transform. The given SNR values take that gain into account. However, the extraction will likely not be robust to real-world variations, thus nor will any subsequent classification algorithm.

Euclidean Machine Learning

This section gives an overview of the baseline concepts of ML developed in our classification algorithms. From a general point of view, the core of most learning algorithms is finding a representation space where the data are linearly separable, and then finding an optimal separation hyperplane in that space. In a more formal setting, ML aims at designing and building a function f on the inputs x, such that f outputs the decisions y involved in the task at hand. Although we have until now described f as a function, it can take the more general form of any algorithm yielding decisions from inputs. For instance, the form taken by f is usually split in non-parametric functions, and parametric functions. In our works we focus on the latter, where f is parameterized by a set of parameters Θ (we then note f |Θ , or f (•, Θ), which are to be learnt through optimization of the loss function l. As briefly described in the introduction, ML can cover a variety of tasks, which are by no means limited to the usual framework of SL, which we notwithstanding focus on throughout this work. For excellent references on the matter of Machine Learning, we mention the following books, considered near biblical by many: [START_REF] Bishop | Neural Networks for Pattern Recognition[END_REF], [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF], [START_REF] Haykin | Neural networks and learning machines[END_REF], and LeCun et al. (2015), amongst of course many others.

Supervised Learning

The SL framework allows to learn f based on a dataset D of pairs {x i , y i } i∈ [1,N ] of inputs and corresponding decisions by minimzing a loss function l, designed to be high when the prediction f (x) differs from the true decision y and vice-versa.

The most common example would be supervised image classification, where the inputs x are images and the decisions y are the image labels. For instance, f could be a linear discriminator f : x -→ w t x, and l could be the Mean-Squared Error (MSE):

l(f (x), y) = ||f (x) -y|| 2 , ( 2.1)
or the Logistic Loss (LL):

l(f (x), y) = -y log(f (x)) -(1 -y) log(1 -f (x)), (2.2)
which both encourage in their own way f (x) to be close to y, as illustrated in Figure 2.6. The two loss functions shown share the same goal, but differ in behaviour and properties; as such, MSE in rather used in regression tasks while LL is dominant in classification. We further describe the major classes of "traditional" ML algorithms. We first give the example of a class of non-parametric functions.

Nearest neighbours algorithms

Although parametric methods are usually favoured in SL, non-parametric functions have the conceptual edge of being straightforwardly adaptable in Unsupervised Learning (US), where clustering is a preferred task to classification. The nearest neighbours algorithms is an commonplace instance of the class of non-parametric learning algorithms.

k-nearest neighbours (k-NN) algorithm

The main instanciation of nearest neighbours is the k-NN algorithm. Its mechanism is trivial to state: given a dataset of labelled inputs D = {x i , y i } i∈ [1,N ] , we seek to classify a new example x. To do so, we collect the k data points {x i } i≤k (with a different ordering of points) which are the closest to x (given a metric d, usually the Euclidean metric). The label assigned to x is then the voted through the majority label in {y i } i≤k .

Minimum Distance to Mean (MDM) algorithm A popular alternative to k-NN is the MDM algorithm. This time, given the dataset D, we first compute the barycenters {µ c } c≤C (again, given the metric d) of each class in the dataset, with C the total number of classes. Then, the new data point x is assigned to the closest barycenter. It is interesting to note that this algorithm is highly moldable depending on the choice of the metric d -more on that later.

k-means algorithm The k-means algorithm, first formulated in [START_REF] Steinhaus | Sur la division des corp materiels en parties[END_REF] and popularized in MacQueen (1967), is a popular choice for unsupervised clustering, where only the number of classes C is known. Simply put, it is an iterative version of the MDM in an unsupervised setting: given a random barycenter initialization for each class c ≤ C, points are assigned to their closest barycenter, as in the MDM, which yields C clusters. Then, barycenters are updated from the new clusters; this process in then iterated until convergence.

We now shift to parametric learning in its the simplest form, i.e. supervised linear classification, which performs the separation of data in the original input space.

Logistic Regression as a Baseline Classification Algorithm

Logistic regression is considered one of the most efficient (generalized) linear classification algorithms along with the Support Vector Machine (SVM) , and has known a tremendous success since its creation by [START_REF] Cox | The regression analysis of binary sequences[END_REF] in many a field. It builds on logarithmically-scaled probability ratios between events: in a binary classification scenario, this model formalizes as ln( Pω(y=1|x) Pω(y=0|x) ) = ω T x, with ω being the separating hyperplane, x ∈ R n the random variable to be classified and y ∈ (0, 1) the binary class. Put simply, if x belongs to class 0, ω T x is negative, and positive otherwise, which naturally corresponds to modelling the distrinution y|x as a Bernouilli random variable of parameter η. Furthermore, considering P ω (y = 1|x) = 1 -P ω (y = 0|x), we have P ω (y = 1|x) = σ(ω T x) = η with:

σ(z) = 1 1 + e -z (2.3)
being the logistic, or sigmoid function, illustrated in Figure 2.7. Two useful properties of σ are σ(-z) = 1σ(z) and σ (z) = σ(z)σ(-z). We will now detail how the hyperplane ω is computed. Though this computation is standard is the ML field, it will help us understand more advanced concepts in future sections. Given a dataset D of N pairs D = {x i , y i } i∈ [1,N ] , we wish to maximize the distribution log-likelihood of our Bernouilli distribution over D:

max ω∈R n l D (ω) = 1 N ln( N i=1 η y i i (1 -η i ) 1-y i ) = 1 N N i=1 y i ln(η i ) + (1 -y i )ln(1 -η i ) = 1 N N i=1 y i ω T x i + ln(σ(-ω T x i )) (2.4)
To maximize l D , we minimize its opposite using Newton's method, which extends and refines gradient descent by multiplying to the left of the gradient the inverse of the Hessian:

∇ -l D (ω) = 1 N N i=1 -y i x i + x i σ (-ω T x i ) σ(-ω T x i ) = 1 N N i=1 x i (η i -y i ) = 1 N X T (η -Y ) (2.5) ∆ -l D (ω) = 1 N N i=1 η i (1 -η i )x i x T i = 1 N X T D η X, (2.6)
where X ∈ R N,n is data design matrix such that X(i, j) = x i (j), similarly for η and Y , and

D η = diag(η i (1 -η i )).
The update rule for computing the optimal hyperplane ω is thus as follows:

ω (t+1) = ω (t) -α (t) ∆l D (ω (t) ) -1 ∇l D (ω (t) ). (2.7)
In practice, we do not explicitly compute the inverse Hessian but rather solve for Z the linear system ∆l D Z = ∇l D . It is also important incorporate bias in the otherwise purely linear model. This can be done with no additional variable by using the bias trick: ωx + b = ω + x + , with ω + = (ω • • • b) and x + = (x • • • 1). This trick is implicitly used throughout all algorithms.

The Perceptron as a link between Logistic Regression and Neural Networks

Also in 1958, F. Rosenblatt independently published a novel classification algorithm named the Multi Layer Perceptron (MLP) [START_REF] Rosenblatt | The perceptron: A probabilistic model for information storage and organization in the brain[END_REF], which aspires to mimick the way neurons and synapses process information within the brain. From another perspective, it aims to build a seperating hyperplane in a learnt feature space, as allustrated in Figure 2.8. We show in this section how a single layer perceptron with entropic loss is mathematically equivalent to logistic regression, thus providing a link between "black box"-ness of Deep Neural Networks (DNNs) and well-known and studied classical learning algorithms. Given the same scenario as above, nodes (which represent neurons) are connected together with weights (synapses) which in turn fire in response to an activation function σ (axon gated channel). The goal is to adjust the weights such that a given input x, when passed through the network, outputs an estimation ỹ which matches the ground-truth y. Again, in the special case of binary classification, y ∈ (0, 1), but we can now generalize to multi-class classification. There are different approaches to generalizing binary classification, such as one-versus-all or one-versus-one strategies, but the most used in DNNs and thus is in our work is the one-hot encoding, in which each label y is a C-dimensional vector with y(c) = 1 and 0 everywhere else, where C is the number of classes and c the particular class of y. The weights w ij can be summarized in matrix W , hence the building block of MLPs: reason to be linearly separable in its raw form. A good representation space aims to that goal through the mapping K, from where we can then separate and classify the data.

K(x, y) = (x, y, x 2 + y 2 )
X (k+1) = f (k) (X (k) ) := σ(X (k) W (k) ) (2.8)
In the equation above, we again use the bias trick, and X is the design matrix as defined in Equation 2.5. In a multi-class problem, the final sigmoid function is replaced by its generalized softmax formulation:

σ(z) j = e z j k≤K e z k
(2.9)

A vanilla MLP is merely a stacking of L such layers such that ỹ = f

(L) • • • • • f (0) (x)
, upon which a loss is computed w.r.t. the ground truth y. The L2 loss is a widespread choice, but we will see how the cross-entropy loss, also popular, is more naturally suited to the classification problem:

l D (Y, Ỹ ) = - 1 N N i=1 y i ln(ỹ i ) + (1 -y i )ln(1 -ỹi ) = - 1 N N i=1 y i ln(f {x i }) + (1 -y i )ln(1 -f {x i }) (2.10)
At this point, we observe that the loss for a single layer perceptron matches exactly that of the logistic regression defined in Equation 2.4, with W acting as the separating hyperplane ω and η as f (x), which yields the same weight update. This formal equivalence of these two algorithms not only allows for a interpretation of DNNs as stacked generalized linear models, it also inspires to attempt second order derivation of the loss as is done in logistic regression. However, the first step is standard gradient computation on stacked MLPs, which we describe in the next section.

Deep Learning

Using the notations dfined above: in DL, f is designed in a general fashion as a hierarchy of L sub-functions {f (k) } l∈[0,L) , commonly known as layers. Then,

f = f (L-1) • • • • • f (0)
. Individual layers are made of elementary modules, such as linear layers, non-linear activations and so on, the idea being to allow the DNN to learn the representation space rather than fixing it through expert human decision. Indeed, while traditional ML models seek to learn a suitable decision function, the concept behind the stacking of layers is to learn both the feature representation space and the decision function, in a seamless end-to-end fashion (see Figure 2.9).

First-order Backpropagation

Gradients in DNNs are derived using the chain rule, leading to the gradient backpropagation algorithm, popularized by [START_REF] Cun | A Theoretical Framework for Back-Propagation[END_REF]. Let us first introduce the useful intermediate variable P (k) = X (k) W (k) at each layer (k), such that X (k+1) = σ(P ) and Ỹ = σ(P (L) ). We first compute ∂l D ∂P (L) :

∂l D ∂P (L) = - 1 N ∂l D ∂ Ỹ ∂ Ỹ ∂P (L) = - 1 N N i=1 y i 1 ỹi + (1 -y i ) -1 1 -ỹi σ (P (L) ) = - 1 N N i=1 y i 1 σ(p (L) i ) + (1 -y i ) - 1
1 -σ(p (L) i ) σ(p (L) i )σ(-p (L) i ) = - 1 N N i=1 y i σ(p (L) i )σ(-p (L) i ) σ(p (L) i ) + (y i -1) σ(p (L) i )σ(-p (L) i ) σ(-p (L) i ) = - 1 N N i=1 y i (1 -ỹi ) + (y i -1)ỹ i = 1 N (Y -Ỹ ) (2.11)
Knowing that P (k) = σ(P (k-1) )W (k) for any layer (k), we can then recursively derive ∂P (k) ∂P (k-1) using the chain rule:

∂P (k) ∂P (k-1) = ∂P (k+1) ∂P (k) W (k)T σ (P (k) )
, where is the element-wise Hadamard product (2.12)

We now have immediate recursive access to the derivatives of l D w.r.t. the weights or data at every level to the network:

∂P (k) ∂W (k-1) = X (k-1) ∂P (k+1) ∂P (k) W (k)T σ (P (k) ) ∂P (k) ∂X (k-1) = ∂P (k+1) ∂P (k) W (k)T σ (P (k) ) W (k-1)
(2.13)

The weights at every layer are then adjsuted accordingly. It is interesting to note there are "good" and "bad" ways to use the chain rule: this one initiates the recursion at the last layer; doing so from the first layer is possible but leads to computationally expensive high-rank tensor products. Backpropagation is the core of deep learning algorithms: more sophiticated architectures than MLPs require but per-layer gradients to implement. Such sophistication is shown in the next section.

Convolutional Neural Networks for Computer Vision

Further algorithmic refinements, formalizations and design branchings led the concept of perceptron to extend to that of fully-connected DNNs. It The core component of a 2D CNN is the convolutional block, which inputs and outputs 3-D image volumes as shown in Figure 2.13. As in the figure, x ∈ R n×n×m is passed through the m filter banks f i of filters f ij to output y ∈ R n ×n ×m , such that: A set of m filters are applied to the input block. Each n×n×m block can be seen as an n×n image with m artificial channels. For the sake of notational simplicity, we consider a square image, which may not be the case.

Anecdotally, Equation 2.14 can be rewritten as a fully-connected DNNs' core element defined in Equation 2.8 using a vectorized form of the image blocks

Fully-Convolutional Networks

We now study a modification of CNNs which, although seemingly trivial, allows for semantic segmentation, i.e. pixel-level, or, in the case of radar classification, timestep-level classification: the FCN. A standard CNN such as VGG (Simonyan and Zisserman 2015) (see Figure 2.14) typically ends with one to three fullyconnected layers, a legacy from MLPs: indeed, it seems natural to keep final representations as vectors rather than 3-D image blocks. One disadvantage is that all spatial relationships, which to this point were maintained in the previous convolutional layers, are now lost. It is in 2016 that J. Long and E. Shelhamer introduced the concept of FCNs [START_REF] Long | Fully convolutional networks for semantic segmentation[END_REF], where dense layers of size n are replaced with convolutional layers of size 1 and depth n. This artificial transformation is referred to in the paper as the "convolutionalization trick", illustrated in Figure 2.15. As explained in the caption, final image blocks are feature maps, whose pixels contain semantic information (labels). Upsampling the feature maps (typically 7 × 7 in the original paper), yields pixel-level segmentation. FCNs became an elegant state-of-the-art after the region-based family of CNNs: in 2013, Girshick et al. (2014) introduced the Region Convolutional Neural Network (RCNN) algorithm, which consists of region proposals sub-images within the image which are then passed through a pre-trained classification network; strong final activations are matched as semantically relevant. RCNN, followed by fast-RCNN (Girshick 2015) and faster-RCNN [START_REF] Ren | Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks[END_REF] in 2015, although a huge step forward in detection, lacked the unified framework and precision naturally allowed by FCNs, which we will in turn use for fine-grained signal classification. The convolutionalization trick consists in replacing the fully-connected layers by 1 × 1 convolution filters, which actually sums up to a fully-connected layer, shared across the features. Indeed, continuing Equation 2.14, we have: Building a FCN upon a backbone CNN is a conceptually trivial step to accomplish: a mere replacement of dense final layers, followed by a Global Average Pooling (GAP) or Global Max Pooling (GMP) layer, which sums up information from the feature maps to 1D features, necessary for classification (note that the segmentation information is taken right before the GAP).

(f * x) n,m = k,l f k,l x n-k,m-l = f x n,m ⇒ ∀i ≤ m , y i = m j=1 f ij * x j = m j=1 f ij x j (2.15)

Batch Normalization

CNNs and other convolutional architectures owe their success to careful design of, amongst many others parameters, the number of layers, the choice of activation functions, the filter sizes... However, novel layers aiming to regularize, lessen overfitting, or normalize outputs also proved to be major contributors to the phenomenal performances brought by these architectures. Batch Normalization (BatchNorm) is probably a most famed example of the latter, as first demonstrated by [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF] and its (to this date) 16 078 citations.

The problem tackled by BatchNorm is the intensity imbalance between different firing neurons. While the learning of a network might favour a certain group of neurons, some with as much underlying relevance may be left aside by the vicissitudes of stochastic training. The formulation undertaken by the original authors is that of "covariate shift": the difficulty for marginally out-of-distribution, yet relevant examples to influence the weight updates within the network. BatchNorm aims to a fairer learning procedure, by imposing a normalization step after each block in the network. Concretely, given a batch of training examples, it subtracts the batch mean, then divides by the batch standard deviation. It then multiplies the result by a learnable scaling parameter, and adds a learnable parameter bias.

The consequence is, that whichever sub-distribution the current batch samples, the neurons fire to a standardized output.

The BatchNorm algorithm is now a default component in many state-of-the-art architectures. Its conception also resonates with our guiding principle of adapting learning models to underlying data distributions; as such, the concept will be summoned in further chapters, in a different context.

Recurrent Neural Networks

Recurrent Neural Networks (RNNs) admit their foundations in the works of Bengio et al. (1994) and Hochreiter and Schmidhuber (1997) originally stem from standard perceptrons, but loop the inner states to learn on sequences of data rather than on individual, unordered points as illustrated in Figure 2.16. Let us recall the buiding block equation for MLPs Equation 2.8. The analog equation for RNNs is quite similar for it only adds the hidden state time dependency; using the notations in Figure 2.16, we have Equation 2.16:

s l+1 = x l+1 U l + s l W l o l+1 = σ(s l+1 V l ) (2.16)
Figure 2.16.: RNN with one hidden layer. The series of data points x i is passed in sequence to the network to become hidden states s i which are function not only of x i but also of all the s j , j < i, hence of the previous data points themselves.

RNNs do not naturally handle out time-frequency images as they are vectorbased. Nonetheless, by considering the spectrogam no longer as an image but rather as a series of 1D spectrums, which in essence it is, we face no more concern. Figure 2.17 shows the two different yet equivalent representations for a same signal. In the recurrent framework, the finesse coefficient k is equal to its maximal value τ as we are dealing with individual spectrums. We can nonetheless relax k by adding a stride in the sequential spectrums, which amounts to subsampling the original time-frequency representation. We have described here a standard, or vanilla, RNN, the one major drawback of which being what is called the problem of vanishing gradient, which we shortly describe here. RNNs accept in theory arbitrarily long sequences, and, as any neural network, are optimized using gradient backpropagation (specifically, the temporal unfolding of the network leads to an algorithm more complicated, yet identical in essence called Backpropagation Through Time (BPTT)). In a standard network, the gradient loses an order of magnitude in its computation at each layer, and by the time it reaches the first one it has decreased exponentially so, or vanished. RNNs suffer from the same drawback, not only in depth but also in time. To put it in a nutshell, a vanilla RNN has trouble learning long-term temporal relationships, as [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF] Until now, evoqued models handle vector-like data in a Euclidean fashion. As hinted in the introduction, much of our work focuses on models built on covariance representation of the data, i.e. on data lying within a Riemannian manifold. Although presented methods of DL are built to achieve high levels of representative abstraction, the acknowledgement of working on a possibly irregular grid opens up the possibility of combining the powerhouse mechanisms of DNNs and the inherent mathematical precision of the underlying data manifold. The curious reader may for example consult the works of [START_REF] Bronstein | Geometric Deep Learning: Going beyond Euclidean data[END_REF] on deep graph representation. The following section is dedicated to introducing the relevant fied of Information Geometry (IG), with the goal set in mind to fuse the concepts of Euclidean ML together with mathematical tools on Riemannian manifolds.

Information Geometry

Many signals reasonably lie within Euclidean spaces, where inner products and metrics are naturally defined and simple operations such as finding a barycenter of points are straightforwardly tractable and defined globally across the space. However, some entities intrinsically lie within a curved "sub-space", called manifold, defined by geometric, statistical or physical constraints. A simple example is illustrated in Figure 2.19. This section deals with the notion of locally defined metrics on curved Riemannian manifolds, which generalize the concept of a flat Euclidean space. For instance, µ-D radar signals are successive series of pulses which can be represented as many realisations of a centered Gaussian process, the instantiation of which verifies certain constraints. The differential geometric study of statistical manifolds is the field of IG, for which we will introduce the important concepts in the following paragraphs. The data however intrinsically lie on the colored sphere, which is a manifold, i.e. can be locally approximated to its tangent plane.

Riemannian manifold

Formally, a n-manifold M is locally homeomorphic to a n-dimensional Euclidean space, i.e. ∀{x,

R} ∈ (M, R * + ), ∃r ∈ R * + and φ ∈ I 0 (M, R n ) | φ(B x,R M ) = B x,r R n , where I 0 (M, R n ) contains all homeomorphisms, or continuous bijections, from M to R n and B x,r
R n is the topological ball of R n centred in x and of radius r. The manifold is also said to be Riemannian when it is equipped with a positive definite matrix F (ξ) defined ∀ξ ∈ M, which is to be at least twice differentiable in ξ, such that the local squared distance ds 2 between two infinitesimallly close points ξ 0 and ξ = ξ 0 + dξ can be written as:

ds 2 = dξ T F (ξ 0 )dξ, (2.17)
where we note an element on the manifold ξ = {ξ 1 , . . . , ξ n } in a coordinate system for one homeomorphic local Euclidean space at point x in M. We recall as a sidenote that the previous definitions also generalize in infinite dimension (the homeomorphic spaces are then pre-Hilbertian) and in addition for complex values (then they are Hilbert spaces). We will detail the results for real-valued n-manifolds.

Fisher information

The manifold we will put to use, is the space of multivariate centered Gaussian laws, which we will from now on note S + * , or S + * (n) in a specific finite dimension n. The elements of the manifold are then entirely characterized by the n × n covariance matrix Σ n . The parameter ξ := Σ n of one such distribution constitutes a coordinate system for the local Euclidean space. The fundamental definition of the covariance Σ of a random vector X of dimension n is:

Σ := Cov(X, X) = E((X -E(X)) T (X -E(X))). (2.18)
Above, E(X) denotes the expectation of X. In practice, given a sequence of Independent and Identically Distributed samples {x i } i∈[1,N ] of X, the Sample Covariance Matrix (SCM) writes:

Σ = 1 N -1 N i=1 ((x i -x) T (x i -x)). (2.19)
Above, x ∈ R n denotes the sample expectation of the time series, i.e. the mean x = N i=1 x i . As stated before, the SCM Σ ∈ R n×n synthesizes the data's characteristic features in a compact and meaningful way.

Though we will later on focus on Gaussian distributions, the following computations hold for any member of an Exponential Family (EF). First of all, it is important to introduce the Kullback-Leibler (KL) divergence, which is a widespread tool to express how one distribution p ξ 1 "diverges" from another p ξ 2 :

D KL (p ξ 1 || p ξ 2 ) = p ξ 1 ln( p ξ 1 p ξ 2 ).
(2.20)

The term "distance" would not be appropriate because D is not symmetric, nor does it verify the triangle inequality. It is however positive and equals zero i.f.f. p ξ 1 = p ξ 2 , i.e. ξ 1 = ξ 2 for the same distribution family. From now on we will use the following simplified notations: p ξ ≡ p and p ξ 0 ≡ p 0 , again with ξ = ξ 0 + dξ, and we will use interchangeably a distribution and its parameter.

The KL divergence is derived from the even more fundamental concept of Shannon entropy (Shannon C. E. 2013):

H(p ξ ) = -p ξ ln(p ξ ) (2.21) Put simply, D KL (p 1 || p 2 ) = H(p 1 ) -H(p 1 , p 2 )
, which respectively represent the entropy of p 1 and the cross-entropy of p 1 and p 2 . We will now see how equipping M with the KL divergence effectively provides it with a Riemannian structure. We define the function f 0 :

f 0 (ξ) = D KL (p || p 0 ). (2.22)
As we are dealing with exponential families, f 0 is indefinitely differentiable and by definition:

f 0 (ξ 0 ) = 0. (2.23)
We also have: We fix a first distribution at (µ 0 , σ 0 ) and plot the KL divergence from the second to the first. We see it annuls i.f.f. (µ 0 , σ 0 ) = (µ 1 , σ 1 ).

∇f 0 = ∂ ξ p 0 (x) ln( p 0 (x) p(x) )dx = 0 -p 0 (x)∂ ξ ln(p(x))dx = p(x) p 0 (x) ∂ ξ p(x)dx ⇒ ∇f 0 (ξ 0 ) = 1 * ∂ ξ p(x) (p 0 )dx = ∂ ξ p(x)dx = ∂ ξ (1) = 0 (2.24) µ -6 -4 -2 0
By noting l x (ξ) = ln(p(x)) the log-likelihood of p, we have ∇f 0 = -E ξ 0 (∂ ξ l x (ξ)), which in p 0 yields the intuitive equality E ξ 0 (∂ ξ l x (ξ 0 )) = 0, which means the loglikelihood expectedly reaches a maximum at the optimal parameter ξ 0 . Figure 2.20 plots the KL divergence between two one-dimensional Gaussian distributions, one being fixed, and exposes the to above cancellations. We now proceed to a secondorder Taylor expansion in p 0 , omitting the term in o(||dξ|| 2 ). The last two equalities will annull orders 0 and 1, leaving only the second-order term:

f 0 (p) = f 0 (p 0 ) + dξ T ∇f 0 (p 0 ) + 1 2 dξ T ∇ 2 f 0 (p 0 )dξ = 0 + 0 + 1 2 dξ T ∇ 2 f 0 (p 0 )dξ = 1 2 dξ T -∇E ξ 0 (∂ ξ l x (ξ 0 )) dξ = 1 2 dξ T -E ξ 0 (∂ ξ ∂ ξ l x (ξ 0 )) dξ ⇔ D KL (p || p 0 ) = 1 2 dξ T F (ξ 0 )dξ.
(2.25)

F is the Hessian matrix of f 0 , and specifically

F (ξ 0 ) = -E ξ 0 (∂ ξ ∂ ξ l x (ξ 0 )) is the Fisher Information Matrix (FIM) F (p 0 ) of p 0 .
In other words, the Hessian of the entropy is the Fisher matrix of the distribution. To be exact, the formal and equivalent definition at index level of the FIM for a distribution p is:

F ij (p) = E ξ (∂ i l x (ξ)∂ j l x (ξ)) = p∂ i ln(p)∂ j ln(p) = ∂ i p ∂ j ln(p) , using ∂ξ i ln(p) = ∂ i p p = p∂ j ln(p) -p∂ i ∂ j ln(p)
, using integration by parts

⇔ E ξ (∂ i l x (ξ)∂ j l x (ξ)) = -E ξ (∂ i ∂ j l x (ξ)) , using E ξ (∂ i l x (ξ)) = 0 (Equation 2.24), (2.26)
where ∂ i abbreviates the operator ∂ ξ i . The above equations show the compliance of the KL divergence with a Riemannian structure as defined in Equation 2.17. The equations presented above show that, given a definition of entropy and divergence between distributions, we can build a notion of local metric on the distribution manifold. These remain valid for any EF; the following subsection studies the induced metric on the family of Gaussian distributions.

Explicit computation of the Riemannian metric on Gaussian distributions

To summarize, we have showed that the manifold of centred Gaussian laws admits a Riemannian metric which is the FIM. Typically, in

E = R n , we define < • | • > : E 2 -→ R as < x | y > = n i=1 x i y i . The induced metric is then || • || : E -→ R + as ||x|| = < x | x > . Considering manifold M, the metric is local to each point ξ and its tangent hyperplane T ξ , where < ξ 1 | ξ 2 > ξ = (ξ 2 -ξ 1 ) T F (ξ)(ξ 2 -ξ 1 ), where F (ξ) is the FIM of ξ. In particular, for ξ 1 = ξ, < ξ 2 | ξ > ξ = ∇ 2 |ξ D KL (• | ξ).
We now explicitly compute the FIM for Gaussian distributions by finding the Hessian of its entropy, which is known and easily derivable from the distribution's expression.

∇ 2 ln (2πe) n det(Σ) = ∇Σ -1 .
(2.27)

This in turn amounts to the natural local inner product of two symmetric matrices S 1 and S 2 on the tangent plane to any on covariance matrix P ∈ S + * :

< S 1 | S 2 > P = T r(S 1 P -1 S 2 P -1 ), (2.28)
From there, integration along the shortest pathleads to the geodesic distance between two Symmetric Positive Definite (SPD) matrices P 1 and P 2 , located anywhere on the manifold, also called Affine Invariant Riemannian Metric (AIRM), defined using the Frobenius norm || • || F :

δ R (P 1 , P 2 ) = 1 2 || log(P -1 2 1 P 2 P -1 2 1 )|| F . (2.29)
The alert reader may note that while the above metric is the correct one from the information geometric viewpoint, it is notoriously computation-heavy, mostly because of the involvement of eignevalue decomposition of symmetric matrices. Other metrics or divergences, either closely approximate it or provide an alternate theoretical apporach, while contributing the highly desirable property of lightweight computational complexity, especially in the modern context of machine learning. Notable examples may include the usage of the Fisher-Bures metric (Sun et Nonetheless, the AIRM remains a founding cornerstone for the exact study of covariance matrices on their correct manifold. Contrary to the local metric, related to the asymmetric KL divergence, it is a valid, globally-defined metric, usable as is within potential contenders for learning algorithms. This connection between entropy and differential metrics was first made in 1945 by [START_REF] Rao | Information and the Accuracy Attainable in the Estimation of Statistical Parameters[END_REF] and in 1943 by [START_REF] Fréchet | Sur l'extension de certaines evaluations statistiques au cas de petits echantillons[END_REF], and further axiomatized in 1965 by [START_REF] Cencov | Statistical Decision Rules and Optimal Inference[END_REF] and the 1976 confidential reportby S.T. Jensen cited in the works of [START_REF] Atkinson | Rao's Distance Measure[END_REF]. The keen reader may refer to [START_REF] Atkinson | Rao's Distance Measure[END_REF] for the detailed derivations relative to the AIRM, and [START_REF] Burbea | Informative Geometry of Probability Spaces: en[END_REF] and [START_REF] Skovgaard | A Riemannian Geometry of the Multivariate Normal Model[END_REF] for further details and proofs. We now move on to studying the link between the Riemannian metric and the manifold's tangent bundle.

Tangent space

The concept of tangent space allows for local Euclidean bearings, but to utilize it to its full extent we must define how to project points from the manifold to the tangent space and vice-versa, as is visualized in Figure 2.21. A Riemannian manifold defines a tangent space at each point; the set of all tangent spaces is referred to as the tangent bundle. Moreover, at each point G, there exists a mapping operator to project from the manifold to the tangent space at this point, and its inverse. These are the exponential mapping and the logarithmic mapping, which are known in closed form on the manifold of SPD matrices S + * :

∀S ∈ T G , Exp G (S) = G 1 2 exp(G -1 2 SG -1 2 )G 1 2 ∈ M ∀P ∈ M, Log G (P ) = G 1 2 log(G -1 2 P G -1 2 )G 1 2 ∈ T G .
(2.30)

In Equation 2.30, exp and log are the matrix exponential and logarithm functions, that is:

∀A ∈ S + * such that A = U diag(λ i )U T , U being an orthonormal diagonalising basis of A, exp(A) = U diag(exp(λ i ))U T log(A) = U diag(log(λ i ))U T (2.31)
Notice non-linear operations intervene in the mappings' definitions: inverse matrix, square root; in fact, for symmetric matrices, these all reduce in complexity to eigen-decomposition, followed by the operation being applied on the vector of eigenvalues. Furthermore for SPD matrices, the Singular Value Decomposition (SVD), more stable and faster can replace the eigen-decomposition, as its eigenvalues are by definition strictly positive.

These two mappings are fundamental to the geometric understanding of the manifold, as they provide a bridge from a curved Riemannian setting to a stright Euclidean one. For instance, taking the reference point as the identity matrix, the mappings equate to the standard matrix log and exp functions, which tells us the set of symmetric matrices is spanned by the log of SPD matrices. As a sidenote, which will eventually be developed upon in further chapters, this correspondance at the identity matrix defines a particular metric, the Log-Euclidean Metric (LEM).

The algorithmic consequence of these mappings, is they open the possibility of implementing learning methods in a Euclidean setting on the tangent space of some given reference point, which is a key feature of tremendous importance, illustrated throughout the various methods exposed in the following section on Riemannian ML.

Riemannian Machine Learning

In this section we describe the generalization of standard ML algorithms from Euclidean spaces to Riemannian manifolds. In fact, we limit ourselves to S + * , the manifold of SPD matrices, in order to be able to fully derive the actual algorithms used later on. Moreover, we make an important distinction between the presented methods: on the one hand, some act directly upon manifold objects; on the other hand, some summon IG as a refinement to an otherwise Euclidean method. In this thesis, we rather use the first category, as the objects we consider are covariance matrices. We therefore describe three such methods: two non-parametric neighbour-based, and Tangent Space Linear Regression (TSLR). Finally, to illustrate the second category, we describe the better-known natural gradient.

Nearest neighbours in Riemannian space

Being now equipped with a Riemannian metric and manifold mappings, we will now build the very simple machine learning algorithm seen previously, the k-NN algorithm, only this time adapted to manifold-valued data, specifically here SPD matrices. Given the dataset D of pairs {P i , y i } i∈[1,N ] , we now classify new points according to the AIRM defined in Equation 2.29. We showcase the process in its basic form in Algorithm 2.1.

L i ← [δ R (P i , P ) , y i ] 4: end for 5: sort ↓ (L[:, •]) // sort ↓ sorts a list in descending order 6: y ← mode(L[•, 1 : K])
// mode returns the most frequent element of a list 7: return y // Label of P

Karcher algorithm for nearest Riemannian barycenter

Similar to above, we can also define the nearest barycenter algorithm in the Riemannian setting. Given a dataset D of N covariance matrices P i spread in C classes, the latter takes place in two steps:

• Compute the barycenter for each class;

• Classify new points to the closest barycenter.

The first subtlety here is, while it may be possible to use the arithmetic mean

G = Bar({P i } i∈[1,N ] ) := arg min G∈S + * N i=1 δ 2 R (G, P i ).
(2.32)

When the metric δ is Euclidean (i.e. based on a standard, globally-defined inner product), the minimization is tractable and leads to the arithmetic mean. However when it is Riemannian (i.e. based on a locally-defined inner product), it isn't (except in certain configurations, for instance in the centred Gaussian case, when the dimension is 1 or 2). The definition above is trivially extensible to a weighted Riemannian barycenter, noted Bar w ({P i } i∈[1,N ] ) or Bar w (D), where the weights w := {w i } i∈[1,N ] respect the convexity constraint:

G = Bar w ({P i } i∈[1,N ] ) := arg min G∈S + * N i=1 w i δ 2 R (G, P i ) , with w i ≥ 0 i∈[1,N ] w i = 1 (2.33)
When N = 2, i.e. when w = {w, 1 -w}, a closed-form solution exists, which exactly corresponds to the geodesic between two points P 1 and P 2 , parameterized by w ∈ [0, 1] (Bonnabel and Sepulchre 2010):

Bar (w,1-w) (P 1 , P 2 ) = P 1 2 2 P -1 2 2 P 1 P -1 2 2 w P 1 2
2 , with w ≥ 0.

(2.34)

Unfortunately, when N > 2, the solution to the minimization problem is not known in closed-form, whether weighted or not: thus G is usually computed using the so-called Karcher flow algorithm [START_REF] Karcher | Riemannian center of mass and mollifier smoothing[END_REF]Yang et al. 2010b), which we chiefly describe in Algorithm 2.2 and illustrate in Figure 2.22. In short, the Karcher flow is an iterative process in which data points projected using the logarithmic mapping (Equation 2.30) are averaged in tangent space and mapped back to the manifold using the exponential mappings (Equation 2.30), with a guaranteed convergence on a manifold with constant negative curvature, which is the case for S + * [START_REF] Karcher | Riemannian center of mass and mollifier smoothing[END_REF]. The initialization of G is arbitrary, but a reasonable choice is the arithmetic mean. Another point of interest is that selecting K = 1 (that is, only one iteration of the flow) and α = 1 (unit step size) in the Karcher algorithm (Algorithm 2.2), corresponds exactly to the barycenter from the LEM viewpoint (Pennec et

i } i∈[1,N ] , iterations K, step α 1: G ← i∈[1,N ] P i 2: for k ≤ K do 3: G ← 1 N i∈[1,N ] Log G (P i ) 4:
G ← Exp G (αG) 5: end for 6: return G Once the geometric mean is known for all classes, we simply pick the one with the smallest geometric distance for a new point to classify. It is interesting to note (and easy to derive) that the geometric mean in S 1 = R + * does correspond to the definition learnt in highschool:

G c = √ x 1 • • • x Nc .

Tangent space linear regression

The generalization of nearest neighbours to manifolds was straightforward once the appropriate tools were identified and known, i.e. the metric, and, perhaps less trivial, the Riemannian barycenter. Nonetheless, their non-parametric nature helped bridge the gap quite effortlessly. On the other hand, parametric learning methods are by design often specifically built for Euclidean spaces: in the founding case of linear regressions, the algorithms' parameters involve projecting to seperating hyperplanes, a tricky notion in manifolds. The key idea to convey here is that one can circumvent this mathematical obstacle by instead projecting the manifold data to the tangent bundle, through the logarithmic mapping defined in Equation 2.30, and setup the linear regression in this now Euclidean space: this procedure is called TSLR, and is successfully used in the analysis of highly-structured data, such as in BCI [START_REF] Barachant | Multiclass Brain-Computer Interface Classification by Riemannian Geometry[END_REF][START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF]).

The choice of projection anchor is thus of prime relevance. One could default to the identity matrix by lack of better choice. However, chosing a point better suited to the data could perhaps enjoy better final performances: for instance, the Riemannian barycenter of the studied data points seems a promising candidate.

In the case of covariance matrices {P i } i∈[1,N ] , we now know how to build the Riemannian barycenter G using the Karcher flow (Algorithm 2.2). The manifold points P i are then mapped to vectors s i as such:

s i ← vec G 1 2 log(G -1 2 P i G -1 2 )G 1 2
.

(2.35)

Then, any linear method, such as SVM or logistic regression can be applied to the transformed problem. The projection can even be reformulated as a kernel transform, as shown in [START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF], which paves the way for further generalizations.

Natural gradient

We have seen how the standard, Euclidean gradient works: to rephrase it synthetically, we simply choose, in the parameter space, an optimal direction vector ν * on an -ball around current parameter θ such that it minimizes the loss function l:

ν * = arg min d(θ,ν)= l(θ + ν).
(2.36)

The optimization yields the negative gradient (-∇ θ l) of the loss w.r.t. the parameter, thus the descent reads:

θ ← θ -α ∇ θ l.
(2.37)

Above, α is the learning rate.

In gradient descent, the result is dependent on the topological shape of theball around θ. In this Euclidean version, which constitutes the industry standard, this ball is isotropic as it is defined by the Euclidean metric. However, we have also learned from the section on IG that the Euclidean metric is an ill-advised candidate for studying distributions: rather, the natural, Riemannian metric δ R inherent to the distribution's underlying manifold should be used. In this context of optimization on an -ball, the infinitesimal version of the metric suffices, i.e. by construction the KL divergence, or equivalently the second-order term of the Taylor expansion, which involves the FIM G(θ) of l(θ) (recall Equation 2.25). The feasible surface of the problem now follows the curvature of the distribution's manifold:

ν * = arg min D KL (θ|ν)= l(θ + ν).
(2.38)

We now solve Equation 2.38, by first writing its Lagrangian form, using the equivalence between the KL divergence and the Taylor expansion and also developing the loss function to the first order:

ν * = arg min D KL (θ|ν)-=0 l(θ + ν) ⇔ ν * ∼ arg min ν l(θ) + (∇ θ l) T ν + λ ν T G(θ)ν - (Lagrangian) ⇔ 0 = ∇ ν l + ∇ θ l + λG(θ)ν (setting the gradient w.r.t. ν to zero) ⇔ -λG(θ)ν = 0 + ∇ θ l
(l only depends on θ)

⇔ ν = - 1 λ G -1 (θ) ∇ θ l (we assume invertibility) (2.39)
We can now finally write the natural gradient update:

θ ← θ -αG -1 (θ)∇l(θ). (2.40)
The resolution above reveals the gradient update is distorted by the inverse Fisher matrix, i.e. the Cramer-Rao bound of the distribution family parameterized by θ. The FIM being the Hessian of a convex function, the operation can be seen as an orthonormal basis change: to complete the link with IG concepts, this amounts to rigidly transforming the canonical Euclidean space to the tangent plane of the manifold at θ. Again, this manifold represents the feasible distributions according to l(θ) -in the case of a neural network distribution, it is referred to as the neuromanifold, first introduced by Amari (1998), and perhaps made popular in CV by [START_REF] Pascanu | Revisiting natural gradient for deep networks[END_REF].

Notice the resemblance of the natural gradient with second-order optimization, notably Newton's method, where the FIM is replaced by the Hessian of the loss [START_REF] Boyd | Convex Optimization. en. Google-Books[END_REF]. It is interesting to note the two methods are mathematically equivalent for the logistic regression, one single-layered perceptron, presented previously, although in general they are not. They do however share the prohibitively computational cost of high-order methods: while the standard Stochastic Gradient Descent (SGD) requires O(n) operations (n being here the dimension of θ, in other words the number of scalar parameters), the former typically require O(n 3 ). While computational improvements along with attempts to democratize the Riemannian gradient, it remains to this day a niche method, seldom used in DL applications.

Yet again, the full potential of such Riemannian methods based on IG seems yet rather unexploited by the general community. These methods show great theoretical and practical advantages, and strongly hint to the possibility of successfully applying them to structured data, a hint we will make sure to follow in the next chapter.

Conclusion

In this chapter, we gave an overview of radar data formation and structure, Euclidean Machine Learning, Information Geometry, and finally Riemannian Machine Learning. With the presented tools at hand, we are now able to imagine an inceptive framework to perform, say, micro-Doppler radar classification.

Given a signal, which we may sensibly represent as a real-valued spectrogam, we can build a vector form out of it, and train, for instance, a Multi Layer Perceptron on the dataset. Recall that a single-layered perceptron is equivalent to logistic regression, which can serve as a baseline algorithm. Of course, doing so destroys the structure of the data from the start, which defies the guiding principle behind our works. It is however a valid approach used in the literature, which we will compare against.

Following the same example, a finer method would consist in interpreting the spectrogam as a time series of spectra. In this scenario, an individual spectrum takes a vector form, and a sequence of such can be modelled by any form of Recurrent Neural Network. Alternatively, one may consider joint temporal and frequential locality, i.e. interpret the spectrogam as an image, to be modelled through a Convolutional Neural Network. In both cases the structure of the signal is taken into account within the learning models.

While using convolutional architectures on micro-Doppler data constitutes a conceptual improvement over the naive purely vectorial approach, treating the time and frequency dimensions in a symetric fashion proves an infringement to their inherent physical difference. To this end, designing rectangular (as in, non square) filter banks allows to better control each dimension's behaviour within the network; using separable convolutions would push the concept further by handling time and frequency independently. Moreover, as it sometimes happens in Computer Vision, inputs may have different shapes; this is even more common in radar signals, where the acquisition pipeline, as well as the tracking of noncooperative targets, potentially leads to a high disparity in signal durations. Here, a standard Convolutional Neural Network might strugle to handle this variability, to the point of malevolently discarding it in the final fully-connected layers. This issue justifies the use of Fully Convolutional Networks, which we also described in the chapter. To repeat the main interest of the Fully Convolutional Network, the temporal structure of the input data is preserved throughout the whole network, allowing a fine-grained classification -in fact, we will see later on it is possible to exactly control the temporal resolution of the network.

The above developments give a first example of adapting popular notions in Machine Learning to the structure of data. In the case of radar data, this kind of approach had not yet been thoroughly explored: while Convolutional Neural Networks and Recurrent Neural Networks had indeed been used, filter bank design and temporal resolution analysis was non existent; we will argue these notions are key to implementing more efficient models, which perform better while remaining relatively lightweight. Specifically, we will demonstrate the development of a Fully Temporal Convolutional Network (FTCN) adapted to the structure of the data, based on the key elements described throughouth this chapter. Moreover, this architecture will be expanded in to the complex domain, pushing further the potential power and flexibility of these new deep architectures, named CR Neural Network (CRNet) and Fourier Neural Network (FourierNet).

Going further on, we will also harness the strengths of Information Geometry to directly study the underlying distribution of the data; specifically, centered Gaussian distributions, characterized by their Sample Covariance Matrix. To this end, we will make use of Riemannian Machine Learning models presented above, along with a covariance-based Riemannian neural model, and go even further by proposing to unite in both Euclidean and Riemannian models in a single classification pipeline, thus leveraging the data's structure from multiple angles. This pipe-like model, which we call Second-Order Fully Temporal Network (SOFTNet), is expected to gather the best from both Euclidean and Riemannian worlds, i.e. yield competitive performance on large datasets while remaining robust to the lack of data, a key issue throughout our studies, as previously stated.

The final chapter of the thesis will focus on the deep covariance-based models, making use of the IG knowledge to propose novel layers and improve upon existing architectures. Concretely, we will first generalize to the AIRM mathematical framework presented in this chapter of a key layer to the main existing deep SPD architecture, yielding a Data-Aware Mapping Network (DAMNet) architecture. We then further generalize the DAMNet to a Riemannian BatchNorm algorithm respecting the manifold's geometry, amounting to the Batch-Normalized SPDNet (SPDNetBN) architecture, which thus posses another level of regularization compared to the original models. All improvements are based on knowledge of the distribution's underlying manifold, such as notions of manifold mappings or Parallel Transport (PT), allowing for models intrinsically better fit to the data.

A proposal of convolutional SPD architecture will also be described, in resonance with the convolutional nets described in this chapter.

The detailed presentation of the proposed Fully Temporal Convolutional Network and complex-valued counterparts, the base Riemannian model and the full SOFTNet pipeline is the subject of Chapter 3. Then, Chapter 4 proposes the theoretical and practical improvements on the covariance-based neural model.

C h a p t e r

S E C O N D -O R D E R P I P E L I N E F O R T E M P O R A L C L A S S I F I C AT I O N

Chapter abstract

This chapter is purposed to disentangle the possible dilemma arising from the multiplicity of valid representations and associated learning models introduced in the previous chapter in the context of temporal signals. The chapter's overall contribution is to show how we can make use of all representations in a single pipeline incorporating both Euclidean and Riemannian learning frameworks. To achieve this end, we first describe a deep Riemannian learning model, specifically a neural network operating directly on Symmetric Positive Definite (SPD) matrices. We then show how to combine the Euclidean models operating on first-order components (raw signal, time-frequency representation), and the Riemannian model operating on second-order components (covariance matrix), which we refer to as Second-Order Fully Temporal Network (SOFTNet). We also demonstrate experimentally the compound benefits of this pipeline model, compared to its elementary components.

Introduction

The first part of this chapter makes use of the structured time series representations, along with the possible learning models introduced in the previous chapter. Succintly, a temporal signal such as micro-Doppler (µ-D) data can take a variety of forms, each one highlighting a different set of properties within the signal. Established references in the signal analysis community, such as Flandrin (1998) and Hlawatsch and Auger (2013). The second part shows the collaboration of these representations and associated models in a single pipeline for temporal classification, along with experimental comparisons on µ-D radar signals.

The raw signal, as a time series of phase and amplitude, doesn't reveal much information to the naked eye. It does however house a rather rich family of semantic features, potentially revealing information of key relevance about the signal. Building a hierarchy of layers to construct an even more compact and meaningful feature representation certainly would benefit to the subsequent classification task. Learning models adapted to this kind of representation include Recurrent Neural Networks (RNNs) and 1D or 2D Convolutional Neural Networks (CNNs).

The spectral representation obtained through the Fourier transform, on the other hand, yields a synthetic view of frequency information throughout the signal. Global trends in the signal's overall behaviour are easily spotted, which makes the spectral representation a more favourable candidate for representing the signal. Because it posseses the same structure as the raw signal, similar models can be used on the spectral representation.

However, a fully spectral feature space, by construction, drops all temporal information. More than the overall frequency information, it is perhaps in the fluctuation of such information that resides the most discriminative features, even more so in a context of challenging data with a wide and interlaced spectrum of behavioural activity, such as in Non-Cooperative Target Recognition (NCTR) on drone µ-D radar data. As such, a windowed Fourier transform seems to constitute the resolving compromise , by providing an effective time-frequency representation of the signal: there, local trends in frequency are seen evolving through time. See for instance [START_REF] Stoica | Spectral analysis of signals[END_REF] and Moruzzis and Colin (1998) for a background review of learning models on µ-D data.

The two dimensions embodied by time and frequency add a layer of structure in the data: while Recurrent Neural Network (RNN) are still an option, Computer Vision (CV) literature strongly hints towards the usage of 2D CNNs. However, the locality found in the time-frequency "image" is not symmetric as the spatial locality in actual images: time and frequency don't spawn the same space. While omitting this fact is entirely an option, taking it into account during architecture design could perhaps lead to more efficient models. Previously cited literature generally does not take this step; while CNNs and to a lesser RNNs exist for µ-D classification, the architectures are more often than not taken directly off the shelf.

As already stated in previous sections, a companion notion to the Fourier transform is that of covariance; like the Fast Fourier Transform (FFT), it allows for a compact representation of trends within the signal. By examining the underlying Gaussian process, the Sample Covariance Matrix (SCM) resulting from the signal thus constitutes a highly-structured, promising representation to make use of in learning models. As was also mentioned before, its inherent Riemannian structure calls for carefully constrained design on the manifold of centered Gaussian distributions, i.e. the manifold of Symmetric Positive Definite (SPD) matrices S + * . While some models have been presented in the previous chapter, we will rather rely on a Deep Neural Network (DNN)-like formulation of the problem, and base our works on a so-called SPD neural network (SPDNet), introduced by Huang and Van Gool (2017). To the best of our knowledge, such an SPD architecture has not yet been adapted to time-frequency data, as we will here do.

Before concluding the chapter's introduction, we would like to remind, that while the proposed models theoretically fit any structured time series data, the experimental validations are centered around µ-D radar data classification. As asserted in the introduction and Chapter 2, radar data is plagued by the cost of its acquisition, its inherently sensitive nature, along with the variety of different physical configurations. It thus and ends up a rather rare commodity. This particular characteristic, shared with many real-life applications (medical data being a prominent example), beg for solutions, not all found within the powerful machinery of DNNs. As previously intuited, Riemannian models fitting to the underlying data manifold hopefully allows for more efficient learning. Moreover, the simulation of data is a parallel track to attempt at filling the missing gaps needed to leverage the full expressiveness of deep models. In this context, we also propose a drone µ-D radar data simulator.

To summarize our proposed contributions:

• A Fully Temporal Convolutional Network (FTCN) designed to respect the temporal structure of the data;

• Two closely resembling complex declinations of the FTCN, notably:

¨A CR Neural Network (CRNet), with a design inspired from signal analysis principles;

¨A Fourier convolution layer, allowing the training from raw complex temporal data, yielding a Fourier Neural Network (FourierNet) architecture;

• A full pipeline for structured time series classification, combining multiple representations and models;

• A µ-D drone radar simulator to allow for extensive experimental studies;

• Validation, comparison and interpretation of the various models, on both simulated and real data.

The chapter begins in Section 3.2 by a description of the proposed models on the different representations, from the SPDNet on covariance matrices to the CNN architectures adapted to the time series data. Then, Section 3.3 introduces the proposed classification pipeline, the Second-Order Fully Temporal Network (SOFTNet) model. We end the chapter in Section 3.4 with the experimental validations, along with the description of our drone simulator.

Learning on structured time series representations

As we have seen, time series data can be seen from various viewpoints, all rooted in sound theoretical background, and different learning models may operate independently on the different representations. Since the experimental model validations rely on µ-D radar data, we will also generalize the framework to complexvalued data. We begin with the SPDNet on covariance matrices. We then introduce a FTCN deep architecture on real-valued time-frequency representations, i.e. spectrograms. Finally, we generalize the latter to the complex domain, and by doing so exhibit a formal link between CNN architectures operating on the time series on the one hand, and on a time-frequency representation on the other hand.

SPD neural networks

The SPDNet architecture mimics that of classical neural networks with a first stage devoted to compute a pertinent representation of the input data points and a second stage which allows to perform the final classification. The particular structure of S + * , the manifold of SPD matrices, is taken into account by layers crafted Bilinear Mapping layer In analogy to the classical linear layer, the Bilinear Mapping (BiMap) layer transforms a SPD matrix P (of size n) at layer (k -1) to a SPD matrix X (of size n ) at layer (k), which we wish to be more compact and discriminative in terms of the final loss function, via a basis matrix W of Gl(n, n ) (the group of left-invertible matrices of size n × n ) as illustrated in 3.1. Note that n ≤ n is required to guarantee W 's full-rank, thus the positive definiteness of the output matrix. However, Gl(n, n ) being a group of matrices unbounded in norm, optimizing on it constitutes a threat of gradient divergence. Thus, as in Huang and Van Gool (2017), the transformation matrices are constrained to the compact manifold of semi-orthogonal matrices, also known as Stiefel manifold, noted O(n, n ). Finally, the forward pass of the BiMap layer is expressed as follows:

X (k) = W (k) T P (k-1) W (k) with W (k) semi-orthogonal (3.1)
Rectified Eigenvalues layer Inspired from the Rectified Linear Unit (ReLU)perhaps the most popular activation in DNNs (B. Xu et al. 2015) -Huang and Van Gool (2017) introduce the Rectified Eigenvalues (ReEig) layer, which lifts small (as in close to zero) eigenvalues to a given threshold . The function being applied solely to the matrix' eigenvalues, we first need to eigen-decompose the matrix P as P = U ΣU T , where the ortogonal matrix U groups P 's eigenvectors in columns and Σ is the diagonal matrix of n eigenvalues. Then, the forward ReEig layer is expressed as follows:

X (k) = U (k) max(Σ (k) , I n )U (k) T , with P (k) = U (k) Σ (k) U (k) T (3.2)
While the ReEig acts a layer activation, another viewpoint would deem it a regularization of the covariance's positive definiteness. This different perspective opens the door to considering other options for the activation, possibly inspired from the covariance estimation, or more generally the Random Matrix Theory (RMT) community. While not the subject here, interesting ideas improving the notion of matrix regularization can be found in [START_REF] Ledoit | A well-conditioned estimator for large-dimensional covariance matrices[END_REF] (2004), reduces to the Euclidean distance between the matrix logarithms. This particularity reveals a natural mapping from the manifold of SPD matrices to the space of symmetric matrices via the matrix logarithm log, which is chosen as the mapping; additionally, the resulting matrix is vectorized to an n * n-dimensional vector, an operation we note vec:

X (k) = vec( U (k) log(Σ (k) )U (k) T ) , with P (k) = U (k) Σ (k) U (k) T (3.3)
We illustrate such an SPDNet architecture in Figure 3.2.

Fully Temporal Convolutional Network

Here we present the DNN for structured temporal signals. We first justify the choice of developing a fully convolutional model, then detail the proposed architecture and comment on its properties, notably the preservation of temporal structure and temporal resolution.

All you need is convolutions

Interpreting a time-frequency representation as an image to be analyzed via a CNN constitutes a first attempt at deep modelling. One first subtlety to handle is that choosing the same representation for input signals of potentially different lengths will lead to frequency dilution, exposed in Figure 3.3. The more natural solution is to use the same projection instead of the same representation, e.g. a spectrogam of fixed window size and overlap. This of course allows input data of different dimensions, not easily handled by standard CNNs. A natural bypass is to consider the problem as Multiple Instance Learning (MIL). In this framework, one input data point is seen as a collection of multiple atomic instances of the same class. Concretely, we divide spectrogams in segments of equal length, which we denote as τ . The choice of τ is rather arbitrary, and is interpreted as the minimal duration one must observe the signal to accurately determine its class; its influence on results will thus be an interesting analysis. As an analogy to speech or music recognition's phonemes (Lee et we may call these atomic portions of signal "noisemes". We thereby consider that observing a noiseme-length signal portion is sufficient to classify the whole signal, but that observing multiple portions is expected to boost the classification confidence. From left to right in Figure 3.3:

• the spectrogram of the original signal;

• that of the same signal downsampled four times;

• that of the signal downsampled sixteen times.

We observe how projecting to the same frequency domain signals of different intrinsic time length yields dilutions in frequency, which in turn unavoidably leads to bad classification.

Although convolutions do preserve temporal structure, the final layers of a typical CNN being fully-connected, that structure is lost, precisely when we reach the most representative feature spaces in the network. The conservation of the temporal structure requires the usage of a FTCN, described in the previous chapter.

Deep architecture for temporal classification

The desirable consequence of being fully convolutional is that, having transformed the time series to a deep feature representation through the network's layers, these learnt features can also be modelled and analysed as a final temporal representation of the signal; for instance, it now becomes possible to study their covariance matrix, which will constitute a cornerstone for our future classification pipeline.

The resulting architecture, which as mentioned above focuses on µ-D radar data, is illustrated in Figure 3.4. Such a network, which we may name FTCN, naturally allows input signals of varying length, and outputs temporal feature maps when fed with longer signals. The choice of hyperparameters for the network directly influences the inherent temporal precision achieved by the FTCN, in particular the filter and pooling sizes and associated strides. Figure 3.5 gives a reminder of how strided filters work, along with a quick intuition to the concept of receptive field. Since the FTCN is built upon the CNN classifier, the temporal feature map length l o equals 1 when given an input of length l i = τ . We note S(l i ) := l o the global sizing function of the network, composed of L layer-wise such functions S l as defined in Figure 3.5. The values taken by S for multiples of τ define the temporal finesse of the network, obtained by successive receptive field overlaps. For instance, no receptive overlap, achieved through non-overlapping strided filters, gives ∀k ∈ N * , S(kτ ) = k. In the case of the proposed architecture, (Figure 3.4), we have: We introduced in the above equation the finesse coefficient f which quantifies the super-resolution reached within a noiseme. By construction, f ∈ [1, τ ], and its explicit formula is simply f = ( L l=1 s l ) -1 (s l being the stride at layer (k)). For instance with τ = 20 and f = 5 2 , we can classify the signal every 8 timesteps instead of 20, i.e. perform a segmentation of controled precision. The takeaway message here is that reducing the amount of pooling and strided filters leads to better temporal resolution. A trade-off thus arises, as strided filters are usually seen as a way for the network to better generalize.

∀k ∈ N * , S(kτ ) = S L • • • • • S 1 (kτ ) = 5 2 kτ - 5 2 + 1 := (k -1)fτ + 1 (3.4)
In Figure 3.5:

• To the left:

-Above: Standard convolution with k = 3 and n = 7 with no stride. The output is of size n = nk + 1.

-Below: Strided convolution with k = 3, s = 2 and n = 7. The output is of size n = n-k s + 1.

• To the right: the same concept applies for pooling: in this example we have k = 2 = s.

In addition to stride, padding is often used to constrain the output to the same size as the input. In general, the sizing function from layer (k) to layer (k + 1)

then reads S l (n) := n l+1 = n l -k l +2p l s l + 1.
The notion of receptive field arises from how many pixels in the previous image the next one "sees". In general, CNNs are designed so that the final receptive field more or less covers the input area.

We conclude this first model description by insisting that, while the usage of CNNs is not unheard of in the time-frequency classification literature, a fully-convolutional architecture, designed to fit the structure of data as detailed above, is novel. The next paragraphs generalize the described model family to the complex domain.

Complex Fully Temporal Convolutional Network

We begin this generalization with a reminder of complex calculus. In the context of DNNs, the backpropagation involves defining gradients of complex-valued functions, which are non-holomorphic, i.e. which requires itself a generalization from the usual framework.

Non-holomorphic complex calculus

It is at first tempting to handle complex numbers in CNNs by simply considering a 2-channeled input containing the real and imaginary parts. One should however take care of respecting the inherent structure of complex numbers; both channels are neither independent nor interchangeable. In this section we describe how complex numbers can indeed be handled in a 2-channel fashion given certain constraints and interactions.

First, we establish the formal equivalence between complex numbers and 2D real vectors as developed by [START_REF] Wirtinger | Zur formalen Theorie der Funktionen von mehr komplexen Veränderlichen[END_REF] and rediscovered by [START_REF] Brandwood | A complex gradient operator and its application in adaptive array theory[END_REF], and later on by [START_REF] Bos | Complex gradient and Hessian[END_REF]. This rather old framework is named Wirtinger calculus, or CR calculus (Kreutz-Delgado 2009). The context of these original developments was the generalization of the complex gradient to non-holomorphic complex functions, to which the proper conceptualisation of complex differentiability is usually limited to. As such, the following equations also establish the complex gradient operators for non-holomorphic functions, which in turn may be used in the backpropagation phase of subsequent neural networks.

The key idea equates the Taylor expansions of a function f : z ∈ C → y ∈ R with its counterpart, which by abuse of notation is also noted f : (u, v) T ∈ R 2 → y ∈ R; both functions map to the same scalar y. The Taylor expansion for both forms is written:

f (z) ≈ f (z 0 ) + ∇ z f z 0 (z -z0) (3.5) f (u, v) ≈ f (u 0 , v 0 ) + ∇ u f ∇ v f (u 0 ,v 0 ) u 0 v 0 (3.6)
We now introduce the 2 × 2 real-to-complex matrix T :

T := 1 j 1 -j such that:        T H T = 2I = T T H T u v = u + jv u -jv = x x * := x (3.7)
In the equation above, • * and • H denote the complex conjugate and transconjugate. We can now rewrite the first-order moments of the vector function f as a function on complex values:

∇ u f ∇ v f (u 0 ,v 0 ) u 0 v 0 = 1 2 ∇ u f ∇ v f (u 0 ,v 0 ) T H T u 0 v 0 := ∇ x f x 0 x -x 0 (3.8)
In the equation above, we defined the complex gradient:

∇ x f x 0 = ∇ x f x 0 ∇ x * f x 0 = 1 2 (∇ u f -j∇ v f ) (u 0 ,v 0 ) 1 2 (∇ u f + j∇ v f ) (u 0 ,v 0 ) (3.9)
The complex gradient is composed of the complex differential operator ∇ x and the complex conjugate differential operator ∇ x * , which can then be inserted in the backpropagation framework such as HIPS/autograd (2020) for any neural network using the formal representation of complex numbers as 2D vectors. In the simple case of a convex optimization problem (Boyd and Vandenberghe 2004), we see that equating the complex vector gradient to zero is equivalent to setting either component to zero. By chosing for instance the second component, and using the equivalent 2D vector form, we arrive to a simpler optimization problem:

1 2 (∇ u f + j∇ v f ) (u 0 ,v 0 ) = 0 (complex domain) ⇔ (∇ u f ) u 0 (∇ v f ) v 0 = 0 (real domain) (3.10)
Similarly for gradient descent, we chose (∇ u f + j∇ v f ) as the gradient to propagate backwards, as done in [START_REF] Brandwood | A complex gradient operator and its application in adaptive array theory[END_REF] and [START_REF] Böddeker | On the Computation of Complex-valued Gradients with Application to Statistically Optimum Beamforming[END_REF]. Thus, in practice, by using the CR calculus framework, it suffices to represent complex numbers as 2-channeled inputs, design the inference layers with correct interactions between the real and complex channels, and simply compute the gradient as a 2-channeled vector to use as is during backpropagation. We refer to [START_REF] Trabelsi | Deep Complex Networks[END_REF] for a description of the usual CNN layers translated to the complex domain, which we in turn implement in our proposed model. The latter builds upon the real-valued architecture introduced in Figure 3.4; while the CR framework allows for a purely complex network, we rather implement a semi-complex network, i.e. the final layers stay in the real domain. The reasons for this are mainly empirical, and detailed in the dedicated Section 3.4. The transition from real to complex values done through the decibel (dB) function, as suggested from signal analysis standards:

∀z ∈ C, dB(z) = 10 log 10 (|z| 2 ) ∈ R (3.11)
The choice of the dB function, though guided by common practice, also empirically proves useful in experiments as shown later on. See Figure 3.6 for a description of the semi-complex model. 2018) for founding elements of deep complex networks, the architectural exploration which is detailed further on, along with the improvement of the complex Batch Normalization (BatchNorm) layer, in the context of truly appropriate data types (i.e., inherently complex-valued), remains a novel orientation of the matter.

HPD neural network

In a similar trend to the complex-valued FTCN, and following generalizations of the Minimum Riemannian Distance to Riemannian Mean (MRDRM) scheme to the complex domain (Barbaresco 2013), we develop a HPD neural network (HPDNet), designed through the same principles of Wirtinger calculus. In this section, we present how we adapt the SPDNet layers to complex values, making use of the previously used formal two-channel representation. We note H + * the manifold of Hermitian Positive Definite (HPD) matrices.

Formal HPD representation in a real-valued computational framework

Because calculus of non-holomorphic functions is to this day not widespread in computational frameworks (for instance as of end 2018, complex tensors are not integrated in the deep learning framework PyTorch Paszke et al. 2017), it remains interesting to provide a custom integration, especially given the simplicity of the resulting implementation. From the results derived above, we can thus represent an HPD matrix H = H +jH as a two-channel SPD matrix (H , H ). In any neural architecture, the gradient of the loss function l involving (H , H ) is computed as ∇ H l + j∇ H l, which corresponds exactly to ∇ H l. For this reason, it is natural to choose ∇ H l over ∇ H * l, as foreseen above. The backpropagation through the network can therefore be done with no additional pain, using any out-of-the-box backpropagation algorithm in a real-valued computational framework. Similar to before, the inference still needs to respect the internal structure of complex numbers. Below we show how to adapt the BiMap layer to H + * .

Complex bilinear mapping

Here we show how to generalise the BiMap layer described previously to complex values, using the Wirtinger formalism. First we write the complex expresison of the bilinear mapping, then structure it to fit the CR framework:

P = W XW H = (W + jW )(X + jX )(W H + jW H ) = W X W H -W X W H -W X W H -W X W H + j W X W H -W X W H + W X W H + W X W H := W X W H -W X W H -W X W H -W X W H W X W H -W X W H + W X W H + W X W H (3.12)
Note that the parameter matrix is now unitary (that is, complex orthogonal), and the transpose becomes a transconjugate.

Complex structured non-linearities

We study here the case of the non-linear structured complex functions involved in the HPDNet, the complex ReEig and complex Log Eigenvalues (LogEig). As stated previously, both take the form a non-linear function f acting on the an HPD matrix P 's eigenvalues. We assume an eigen-decomposition of P = U ΣU H . Note that although U is unitary, Σ is real because P is Hermitian. This is a very strong result, which greatly simplifies the generalisation of aforementioned functions to the complex setting: in fact, there is nothing in f to actually generalise, since the input eigenvalues are already real. However, one must take care to correctly handle the eigen-decomposition itself, separating the real and imaginary parts in order to respect the CR formalism:

X = f (P ) = U f (Σ)U H = (U + jU )f (Σ)(U -jU ) T = U f (Σ)U T -U f (Σ)U T + j U f (Σ)U T + U f (Σ)U T := U f (Σ)U T -U f (Σ)U T U f (Σ)U T + U f (Σ)U T (3.13)
Using the equations above, we are now able to perform inference through the complex BiMap, ReEig and LogEig layers. Posterior to the LogEig, a fully-connected layer (or several) handles the hyperplanar separation for classification. At any point in the network, it is possible to go back to R using the C2R transfer function below:

∀H ∈ H + * , 1 2 (H + H ) = S ∈ S + * . (3.14)
This is necessary because, again, we cannot yet perform the classification in the complex manifold. Furthermore, it is not obvious that the best performing model would maximize the use of complex numbers; it remains of interest to study the influence of the C2R's positioning in the network's hierarchy to optimize its performance andd robustness.

Full pipeline for temporal classification

To sum up, we have presented various models adapted to different representations of the time series data, specifically the real-valued spectrogam, the complex spectrogram and the covariance matrix. The cunning reader may have noticed we have not presented a model for the raw time series data, notwithstanding the fact we had hinted in previous sections it was possible either with 1D CNNs or RNNs. Before moving on to the full pipeline, we first present a simple trick which allows to partly elude this matter, and naturally integrate the 1D raw time series within the models shown above.

The Fourier convolution layer

First we recall the Discrete Fourier Transform (DFT) of a discrete complex signal z of total duration N :

∀k ≤ N, DF T z (k) = l≤N z(l)e -2iπk l N (3.15)
Thus, a windowed Fourier transform of length n can be replaced by a convolution of the signal by the n Fourier atoms, which are the n base vectors of n-th-roots of the unit, (e -2iπk • n ) k≤n . Attempting to learn filter banks on 1D inputs has been used in the context of audio recognition [START_REF] Lee | Unsupervised feature learning for audio classification using convolutional deep belief networks[END_REF]; Dieleman and Schrauwen 2014); however, while they observed the learned filters did seem to converge to frequency-selective filters, the Fourier atoms were not used. Thus, instead of fixing the DFT, we propose to prepend to the FTCN presented in Figure 3.6 a 1D complex convolutional layer initialized with the Fourier atoms, and allow the weights to be learnable, i.e. we allow Fourier atoms to vary to best accomodate the optimization crtiterion. By doing so, we allow additional flexibility in the learning process, all the while utilizing the expert knowledge of a proper time-frequency representation. We illustrate this novel convolutional layer in Figure 3.7.

Fourier Convolution

Pipeline bifurcations

We now introduce the proposed pipeline, displayed in Figure 3.8, and show how branching through the its blocks leads to different models on the signal representations. Four global models, noted from (1) to (4) in the figure, can be extracted from the pipeline, which we detail throughout the section.

As a first note, never bifurcating to covariance analysis, i.e. remaining on the first row in Figure 3.8 exactly amounts to the FTCN taking as input the raw signal, sequentially building a spectral and hierarchical feature representations. If the DFT is allowed to be fine-tuned, we call such an architecture a filter-bank learning network, or FourierNet; if not, we simply refer to it as a CRNet. While the FourierNet thus consitutes a generalization of the CRNet, both correspond to model (4) in Figure 3.8; the distinction is made to allow to see potential benefits from learning upon the Fourier atoms. Using a SPDNet to classify the covariance of the signal amounts to branching out downwards in the pipeline. This can be done either at the raw signal representation, or at the Fourier representation, whether it be a proper DFT or the Fourier convolution. In other words, it is possible to learn on the covariance of the raw temporal data, or on that of the spectral, Fourier trnasform of the latter. The two resulting SPDNet models correspond to (1) and (2) respectively.

The classification in the FTCN uses Global Average Pooling (GAP) to pool the feature maps' temporal evolution to a single dimension. However, as mentioned before it is possible to branch out in the pipeline at any stage, precisely thanks to the fully-temporal property of the network. It is precisely the property of conservation of temporal structure verified by the FTCN which allows to extract temporal feature maps, and thus, gives the possibility to study the covariance of these maps. Doing so hints towards a powerful representation learning model, making use both of a potentially complex-valued Euclidean FTCN operating on first-order moments of the data, and a Riemannian SPDNet operating on secondorder moments of the data. We thus call this particular model the Second-Order Fully Temporal Network (SOFTNet), and illustrate it in Figure 3.9.

Covariance pooling

A key module linking the first-and second-order models is the covariance pooling (CovPool): given an n-dimensional signal sampled during N timesteps s = (s i ) i∈ [1,N ] , where s i ∈ R n , its covariance matrix X ∈ S + * (n) is estimated as:

X = i∈[1,N ] si sT i (3.16)
In the equation above, often referred to as Maximum Likelihood (MLE), si is the centered version of s i , i.e. si = s i -1 N i∈[1,N ] s i . In our context, the raw signal is univariate as it is, by construction, a scalar complex time series. We overcome this problem by considering the raw signal as a series of possibly overlapping windowed elementary sub-signals, introducing effective multi-variability through the length τ of the window. Thus, one s i in Equation 3.16 becomes [x(l) • • • x(l + τ )] T . In essence, the windowing is similar to performing a windowed filtering, and can also be represented as a convolution, for which the weights are no longer the Fourier atoms but adequately placed zeros and ones. Figure 3.10 further on illustrates the signal splitting.

Another problem may arise from the MLE estimation: theoretically, for X to be regular, i.e. positive definite, it is required to have n ≥ N , i.e. to observe more samples than their dimension. This can be problematic in estimating the covariance of the learnt features, i.e. when the s i in Equation 3.16 are the FTCN's final fea-ture maps f i . In this scenario, the temporal sampling has been reduced through the FTCN's convolutions and possible poolings, while feature size could have been set to arbitrarily large dimensions. The intent reader may refer to DeepKSPD (2018), [START_REF] Acharya | Covariance Pooling for Facial Expression Recognition[END_REF], and Yu and Salzmann (2017) for details on how to best extrapolate covariance from CNN features: in summary, it is possible to use robust estimators such as described in [START_REF] Ledoit | A well-conditioned estimator for large-dimensional covariance matrices[END_REF] and [START_REF] Ledoit | Nonlinear shrinkage estimation of large-dimensional covariance matrices[END_REF], to split the feature maps in smaller groups, or even to introduce an transitional convolutional layer to learn a dimension reduction suited for the covariance pooling. In the experimental validation section below, several combinations of the above were tried; without going into much detail, we give examples of theoretically better conditioned covariance estimators used, but not cited, in the experiments:

Σ regul = (1 -λ) Σ + λ T r( Σ) n I d Σ adjust = U 1 α Λ + ( 1 -2α 2α ) 2 U T + 1 -α 2α 1 (3.17)
In the two equations above Σ is the SCM of dimension n, and Λ its eigenvalues; Σ regul tempers the covariance eigenvalues with a constant value λ = 1e -6 (typical value); Σ adjust fits eigenvalues to a parametric oracle constraint imposing regularity, with a typical value α = 0.75 and 1 a n × n matrix of ones. See the established Scikit-learn package [START_REF] Pedregosa | Scikit-learn: Machine Learning in Python[END_REF], and more precisely the documentation for covariance estimation (2.6. Covariance estimation -scikit-learn 0.22.1 documentation 2020).

Experimental validation

As stated above, we perform experiments on µ-D radar data, specifically on the task of drone recognition. We will compare the different models performing on the different kind of input representations (raw signal, spectrum, covariance), and the full pipeline making use of all the latter. Before diving head-on in tables and numbers, one must take special care to ensure fair comparisons between the models; indeed, how does comparing the performances, for example, of a FTCN on spectrograms against a SPDNet on covariance matrices? This concern is solved by considering the input data processing -signal splitting, Fourier transform or convolution, covariance pooling -as particular instances of filtering; conceptually, it suffices to chose the same elementary duration τ for all processings to make all comparisons fair. This is illustrated in Figure 3.10: as can be seen, the formal analysis of the full signal through different representations is made possible by chosing a temporal integration framework coherent throughout all representations. This concept refines Figure 1.4 shown in the introduction, where the Fourier transform and covariance pooling were performed on the signal's global time scale.

• • • • • • Split signal Fourier transform Raw signal s 0 s N f 0 s N s 1 s N -1 f 1 f N -1 s 2 s N -2 f 2 f N -2 s 3 s N -3 f 3 f N -3
Signal covariance Spectral covariance Spectrogram This founding issue being now addressed, we now describe in detail the bulk of the data used in the experiments. Specifically, we propose a data generator, which simulates the behaviour and physical properties of various drones, and then emulates a radar wave forming upon the latter targets. We now describe our proposed drone data simulator.

Drone micro-Doppler radar data simulator

Radars are a sensitive field for companies and states as they are usually involved in defense and security. For our problem, this amounts to a lack of real data and reproducibility. Moreover, the proliferation of Unmanned Aircraft Vehicle (UAV) models begs for generic modelling. Building an accurate and powerful simulator is thus key the task at hand, especially for deep learning models, which traditionnally require massive amounts of data, or rather close-to-exhaustive variety in the dataset. Here, we introduce a simple yet expressive drone simulator.

A physical drone model

Firstly, the UAV is modelled by a discrete set of N p scattering points disseminated along its physical structure. Scaterring points model the reflectance of the target, as a discrete set instead of a continuous ensemble. Figure 3.11 shows the distribution of the scattering points, their reflection direction and Radar Cross-Section (RCS) along with their temporal evolution. Note the model is 2D as we consider the UAV to remain roughly in the same plane w.r.t. the radar. This assumption is fairly reasonable in many situations, greatly simplifies the simulation, and slight changes in inclination can be rather accurately modelled by variations in the RCS. The N p scattering points moving in time yield a set of N p series of 2D coordinates, which are then fed to wave equations described below. Equation 3.20 shows the final result of raw µ-D radar signal of the body and the helices, respectively S 0 and S.

The azimuth Az ∈ R 2 of the target w.r.t. the radar varies in function of time and the cuurent Rounds per Minute (RPM):

Az k (t) = -2π RP M k (t) 60 t + Az k (0) (3.18)
The total angular vector α ∈ R 2 modulating the final signal depends on the azimuth of the target and the orientation n ∈ R 2 (we drop the time dependency for clarity):

α k = Az k -n k π 180 (3.19)
Finally, the received signal S 0 from the drone's body is modelled as a trigonometric function of a spatial dot product of the azimuth and the positional vector X k of the target's body scattering points and its RCS, and similarly for the contribution S of the drone's blades' scattering points:

S 0 = N P 0 k=1 RCS k exp 4iπ f e c ( X k • Az k ) S = N P k=1 RCS k max(cos( X k • α k ), 0)sinc(2 f e c L k sin( X k • α k ))exp 4iπ f e c ( X k • α k ) (3.20)
Finally, the signal is immersed in a noisy and cluttered environment. The noise is unavoidable white thermic noise from the sensing mechanism, the clutter sums up the influence the outside environment on the signal of interest; Billingsley's model was used to model ground clutter [START_REF] Billingsley | Low-angle Radar Land Clutter: Measurements and Empirical Models[END_REF]. The following section extends the simulator to account for unpredictible behavior, thereby introducing intra-class variety.

Dataset generation

The simulator described above gives a deterministic output given one configuration. Real data are however subject to multiple variations even within one single measurement, whether it be responses to an unkown controller, alterations of the environment or the data acquisition itself, all in all to NCTR. These variations will define the intra-class disparity and thus the inter-class separability. Table 3 π . Note we have listed velocity and not radial velocity. We consider the case where a drone alternates between straight line to circular trajectories, we thus have V r = ±V sin(κ(t)), where κ(t) is the curvature evolving through time. Figure 3.12 shows intra and inter-class variations for the three drones Vario, Phantom2 and S1000+ for a given set of representation parameters. The reader may refer to Table 3.2 for an exhaustive list of default values. Intra-class variety is achieved by sampling pairs of drone parameter values in the acceptable ranges set in Table 3.1 and linearly interpolating the resulting values in time. Therefore the parameters are defined in time as in Equation 3.21: The reader may observe that the S1000+, under strong variations, bypasses the allowed bandwidth set by the PRF, which leads to the well known frequency folding phenomenon (Shannon C. E. 2013), or Doppler ambiguity, which in turn can hurt the representation's credibility as a robust one. Unfortunately, though we can set the parameter arbitrarily high in simulations (again, which we do here for the sake of visual clarity), real-world radar costs constrain the PRF to possibly sub-optimal thresholds.

V (t) = V 1 + (V 2 -V 1 ) t T with V 1 , V 2 ∼ U(V min , V max ) Ω(t) = Ω 1 + (Ω 2 -Ω 1 ) t T with Ω 1 , Ω 2 ∼ U(Ω min , Ω max ) κ(t) = κ 0 ∼ U(0, κ max ) V r (t) = sV (t)
We are now ready to present results upon a simulated database of µ-D radar signals.

Experiments and results

Experimental setup All datasets contain N = 1000 examples for each of the C = 3 drone classes presented in Figure 2.3, of which we reserve 25% for validation and another 25% for testing. All models are run in a 5-fold cross-validation. Each example simulates a recording of length 250ms, i.e. 1000 sample points for a standard PRF of 4kHz. This amounts to a total of ≈ 4 minutes of data per class, i.e. 2 minutes of training data. We choose the noiseme length τ ≈ 10ms, which corre- sponds to 20 timesteps at 4Hz. The choice is guided by the largest approximate period in all signals, i.e. in the case of drones, approximately one blade rotation. Conceptually, this means the elementary decision resolution spans about one blade rotation.

Validation and comparison of the deep first-order models

We begin with the simplest proposed FTCN, i.e. operating on real-valued spectrograms obtained through a valid Fourier transform, and ratify its usage against two benchmarks: a logistic regression (i.e. a single-layered Multi Layer Perceptron (MLP)), and a Long Short-Term Memory (LSTM) architecture, which we will simply refer to as the RNN.

Throughout the experiments we train the three classifiers (MLP, RNN and FTCN) with the same strategy to remain consistent: Stochastic Gradient Descent (SGD) with initial learning rate α = 0.5 for the MLP, RMSProp [START_REF] Ruder | An overview of gradient descent optimization algorithms[END_REF]) with α = 2e -3 and decay β = 1e -6 for the RNN, and accelerated SGD with α = 4e -3 and momentum µ = 0.9 for the FTCN. All optimizations are initialized by Glorot uniform sampling and run for K = 100 epochs without early stopping nor crossvalidation (except manual hyper-parameter finetuning). All learning rates are divided by 2 every 25 epochs. These initial models were implemented on both the Keras ( General performance and robustness First we train the classifiers multiple times on a standard configuration to evaluate overall performance and robustness to initialization to evaluate the simulator's expressivity, seen in figure 3.13. Running the training 20 times for each classifier (we actually test three FTCN architectures with different finesse coefficients defined in Equation 3.4, i.e. different temporal resolutions), we observe consistent test accuracy results, the most consistent being the FTCN, which exhibits the smallest spread. In terms of accuracy, the MLP falls way behind the RNN and FTCNs, which are close though the FTCNs seem to generally perform better.

The FTCNs show better performance than the RNN and MLP, along with less variance over cross-validations.

Impact of radar configuration parameters

Here we train all models in different configurations; specifically we vary SNR and PRF from extremely challenging to wishfully accomodating. Results are found in figure 3.14. The most challenging configuration being at SN R = 0dB and P RF = 4kHz, we nevertheless achieve ≈ 58% with the FTCN, versus total confusion (≈ 33%)) with the MLP. Note this is an unrealistically challenging configuration, as an SN R of 0dB means an original signal to noise ratio before Fourier transform of ≈ -15dB. Performances seem to plateau when reaching high PRF, which is sufficient from a practicioner's point of view. This could potentially be explained by the high dilution of the signal when the sampling frequency PRF is much higher than the signal's bandwidth.

Again, the FTCN seems to be more robust to bad quality signals. Super-resolved classification We now come back on the notion of temporal reosolution, or finesse, of the proposed FTCN architecture. We showed that less pooling in the newtork increased its temporal resolution, with a potential risk in loosing ability to generalize. In the following experiment, we plot, in Figure 3.15, the accuracy of the three FTCNs used above, in function of the duration of the input signal. In Figure 3.15, the input spectrograms span 40 timesteps. Using τ = 20 and three different networks with finesse f ∈ { 1 4 , 1 2 , 1}, the input is thus seen as a series of 4 noisemes and the corresponding output feature maps' lengths are respectively 6, 11 and 21 according to equation 3.4. Confidence is plotted on the 21 timesteps of the most resolved FTCN, the other two being subsampled accordingly. To ensure a fair comparison, the three networks are identical in all except for the convolution strides: each thus has its own resolution, and will require different accumulation of data before updating its confidence. The figure shows that the gradual income of data best profits the most resolved architecture, i.e. here the one with no strides at all, i.e. with fully overlapping receptive fields.

From this experiment, we prove that controling the network's temporal resolution constitutes a thought-through architecture design choice adapted to the type of input data, which resonates with the overall philosophy of the thesis. We also present in Figure 3.16 a similar result in a different field of application, here Environmental Audio Recognition (EAR), specifically on the UrbanSound8K dataset [START_REF] Salamon | A Dataset and Taxonomy for Urban Sound Research[END_REF]), a commonly used dataset in this field. Again, we use a pre-trained FTCN architecture, similar in essence to the one presented above in the case of radar classification, and input signals of longer duration to observe a general increase in performance.

The key idea to retain here is that chosing a finer temporal resolution, perhaps to the detriment of spectral resolution, seems to favor better performances whether it be for the µ-D radar or EAR applications.

Exploring the use of complex-value data and models

Having now validated the performance, and described the advantages of the FTCN architecture, we now study how the usage of complex values within the network impacts on performance. The first step is to introduce the new design choices arising from the complex framework, with the end goal being the comparison with a real-valued conterpart.

Number of parameters Intrinsicly, a complex network will have twice as many parameters as its real counterpart; in practice, it is not obvious how this increase would affect performance. For instance, neural networks tend to better generalize in the case of a large dataset when allowed more parameters, but may also suffer from overfitting when a sufficient amount of diversified data is not met. A reasonable way to experiment on this interrogation is to allow half as many channels in the CRNet's convolutional blocks and focusing experiments on small amounts of data. Results show that keeping the same number of channels as in the real network still performs better, which is a conclusive statement as, while the practical number of parameters has doubled, the network did not suffer even when presented with few data. As a sanity check, doubling the number of channels in the complex network performs the worst of all cases.

The CRNet architecture may keep the same number of parameters as its realvalued FTCN counterpart, even though it doubles in parameters and quadruples in operations.

Signal scaling and complex representation Raw radar data, along with their Fourier transforms, often exhibit major variations in scale, due to different intervening physical phenomena operating in a variety of scales. This translates to the practical habit of converting spectrograms to a logarithmic scale, most often dBs, whether it be for visualisation or further analysis. A real-valued network benefits from this rescaling from the start as the inputs are the decibel-spectrograms. In the CRNet however, the log-scale is ambiguously defined for complex values, which allows potentially harmful variations in scale to propagate within. Proper weight initialization and BatchNorm explicitly combat this issue, but formally fail to recover a log-scale as they remain linear transformations. To this end, we propose a partially complex network for which the output complex representation is log-scaled after the passage to absolute value, and heuristically study the impact on performance of the complex-to-real (C2R(x) = 10 log(|x| 2 |)) function's position in the layer hierarchy. The empirical conclusion is conceptually satisfying as it places the C2R right after the final temporal representation layer, ie right before the convolutionalized fully-connected layers, as represented in Figure 3.6. This result leads to a rather natural interpretation:

While the complex spectral representation of the signal in a real-valued FTCN ends with the FFT, complex feature mapping in a CRNet explores a hierarchy of further filter banks in addition to the Fourier-like filtering.

Fourier convolution parameters

The first layer of the real-valued FTCN on spectrograms is conceptually preceded by a windowed Fourier transform, which remains a fixed pre-processing. While the previsou paragraph dealt with the scaling of the complex spectrogram within the CRNet, the FourierNet however directly handles the raw complex data, and as such, its first layer is a 1D convolution. While conventional initialization schemes such as Xavier initialization (Glorot and Bengio 2010) can be applied, we may benefit from exploiting the spectral properties of the radar signal: indeed, experiments show a consistent improvement when using the proposed Fourier-like convolutional layer. Similarly, the window overlap percentage or hop length of the Fourier transform corresponds to the convolution stride. In the context of learning the 1D filter banks, a low stride (set to 1 in the experiments, ie maximum overlap) proved paramount to the network's performance, regardless of initialization. On the other hand, real-valued counter-parts seemed much more robust to this hyperparameter. One interpretation of this phenomenon is that the passage from raw complex data to real-valued spectrograms averages through coherent integration any potential added information from a higher overlap, while keeping both amplitude and phase sensitizes further processing to this added information.

Using the Fourier convolution in a FourierNet may require careful hyper-parameter finetuning for it to outperform the fixed FFT in its CRNet counterpart. For instance, a strong overlap (i.e. low stride) seems necessary to a successful training.

Quality and amount of data Throughout conducted experiments, a general trend seemed to emerge: complex networks overpowered real networks when presented with a large yet complicated dataset. Specifically, we observed improvement for SNRs on the raw data close to zero or in the negatives, positive SNRs leading to insignificant improvements. Furthermore, when the amount of training data was kept relatively small (in our scenario, less than 5 minutes), CRNets performed poorly to worse than their real-valued FTCN.

In short, a CRNet seems to yield significant improvements mostly when the data are generally challenging and their amount is vast.

Numerical results

We now give numerical comparisons between FTCN, CRNet and FourierNet architectures, the latter two designed based on the previous empirical evaluations. The simulation configurations are first set to an extremely noisy case, where the raw data is 5dB below noise (SN R = -5dB). We also initially quintuple the size of the simulated dataset, yielding 20 minutes of total continuous recording per class instead of 4, our usual standard (i.e. again, 10 minutes of training data). For reference, a coherent integration of 20 timesteps brings the spectrum 8dB above noise. A PRF of 4kHz is used; at this frequency and with the considered drones, Doppler ambiguity, i.e. spectral folding, is omnipresent. As stated above, we voluntarily chose a large amount of data in a very challenging configuration. We also give performance results for the models when trained on a fraction of the data to quantify the robustness of the models to lack of data: 20% of this larger dataset corresponds to the original size, 5% to ≈ 1 minute per class (30 seconds of training data). The rest of the experimental setup remains the same. Results on the three models are presented in table 3.3. The first observation is the improvement of the two complex networks over the real counterpart when given all 10 minutes of training data (the 50% training split of the total 20 minutes), the FourierNet being superior to the CRNet. Given 20% of available training data (2 minutes), the FourierNet still outperforms all models, but the CRNet starts decreasing towards the FTCN's performance. Given only 5% of training data (30 seconds), all complex models perform worse than the FTCN. Finally, we repeat the experiments on a cleaner dataset, by changing the SN R from -5dB to 5dB (we limit ourselves to the FTCN and FourierNet). Results shown in table 3.4 naturally exhibit better performances overall, but the FourierNet struggles to outperform the FTCN, which supports the argument of complex networks working noticeably better in challenging configurations only.

In summary, the complex models seem to improve upon a real-valued couterpart, in situations where data is plentiful and inherently challenging. They thus seem a poor choice in scarce data scenarios, or when a real-valued model already comes close to solving the task. The cost is twice as many parameters and four times as many computations, leading to slower training and inference.

Comparison with second-order models and validation of the full pipeline

We now add to the experiment pool the SPDNet operating on covariance matrices, which will then allow us to proceed to evaluating the proposed pipeline. The 3.10. We bifurcate the introduced pipeline at various stages in various configurations, which amounts to different learning models which we relate to in Figure 3.8, specifically the following:

1. SPDNet: a SPDNet on the raw complex data's covariance immediately made real, estimated over 99 windows of 20 samples;

2. HPDNet: a HPDNet on the raw complex data's covariance, estimated over 99 windows of 20 samples;

3. Spectral SPD Neural Network (SpectroSPD): a SPDNet on spectrum time series;

4. FTCN: a FTCN on spectrograms (mathematically equivalent to spectrum time series);

5. FourierNet: a FTCN on raw complex data as described above, where Fourier filter banks are fine-tuned;

6. SOFTNet: a Second-Order Fully Temporal Network, where a SPDNet is appended to the the final feature representation of the FTCN.

Furthermore, we repeat the experiments with decreasing amount of training data in the hope that injecting geometric information in the learning through second-order modelling would compensate for lack in data volume. Results are displayed in Table 3.5.

The first remark is that the SPDNet yields the worst accuracy when all training data is available. Intuitively, it makes sense that it score lower than SpectroSPD and SOFTNet as the covariance is then sampled from a more adapted or discriminative model, i.e. respectively a spectrogram and FTCN learnt features. The HPDNet performs better mostly when much data is available, losing its potential when data is scarce while being more greedy on computation time. As for the FTCN and FourierNet, these are deep learning models with a total of about 14000 parameters or more, whereas the SPDNet is the Riemannian equivalent a rather shallow network, with only about 700 parameters. However, as training data decreases, we see the SPD methods become better than the traditional deep models: they seem to exhibit much higher robustness to lack of data, which validates the usefulness of exploiting the geometric information of the data.

The final observation concerns the SOFTNet (Figure 3.9), which performs best across all configurations. In a sense, this model benefits both from deep temporal feature learning, and from covariance geometry modeling. In practice, affordable overhead is observed when using this larger model: one epoch lasts ∼ 7s, compared to ∼ 1.5s for the FTCN trained on a Nvidia GTX 1070M GPU and ∼ 3.5s for the SPDNet trained on a i7-6700HQ CPU.

To conclude on the classification pipeline:

Riemannian SPD-based models don't compete against Euclidean deep models when data is plentiful. However, the exhibit strong robustness to the lack of data, outperforming the latter in scarcity scenarios. The full SOFTNet pipeline exhibits complimentary behaviours, yielding competitive performance throughout all situations. Details execution times, code snippets illustrating the models mentioned throughout this chapter, along with indications for implementation can be found in Appendix A.

Conclusion

In the order of things, we first introduced an expressive simulator for bladepropelled engines such as drones, or Unmanned Aircraft Vehicles (UAVs). The simulator sports sufficiently variational characteristics, and allows for the generation of realistic µ-D radar signals. The synthetic datasets thereby obtained were thus suited for the Machine Learning (ML) task of classification on several distinct models: Multi Layer Perceptrons (MLPs), Recurrent Neural Networks (RNNs), and Convolutional Neural Networks (CNNs). In a first set of experimentations, we assessed these models performed well even in harsh simulation conditions, from a radar operation viewpoint. We furthermore justified the assumption that the recurrent and convolutional models, because they inherently learn temporal fluctuations, are particularly appropriate to the task of radar classification.

In particular, we proposed a relatively lightweight Fully Temporal Convolutional Network (FTCN), which, being free of fully-connected layers, is able to handle signals of varying length over time. The FTCN showcases two interesting properties: its output feature maps preserve the temporal structure of the signal, and are themselves a time series, and, it is possible to control exactly the temporal resolution of this output feature time series. The properties of the FTCN, along with its experimental success on both synthetic and real datasets, make it a satisfying candidate for the main streamline of this thesis, the search for efficient models, adapted to the underlying structure of the data.

We also developed a partially complex-valued counterpart to the FTCN, in order to take into account the inherent complex structure of µ-D data, which again follows the line of research throughout this thesis. We furthermore introduced a Fourier-like convolutional layer, which harvests the advantages of both the Fourier transform and of learning filter banks on the raw data, an intuition proved to be consistently true in practice. We performed extensive experimentation on synthetic data to isolate the cases where performance benefitted from complex values. The main conclusions obtained were, that above a certain amount of observed data (a couple of minutes for our datasets), in challenging configurations (under 5dB of SN R in our scenarios), complex-valued networks significantly outperformed their real counterparts. These results initiate a hopeful stance on introducing complex values in deep learning-based classification methods on µ-D radar data.

Finally, we introduced a neural pipeline consisting of a first-order FTCN, onto which a second-order SPD neural network (SPDNet) can be appended at any stage of the first-order model. We find that the deeper down the first-order model, the better the performance of the global second-order model. The best-performing, and also deepest model issued was baptized Second-Order Fully Temporal Network (SOFTNet), and consists in a succession of meaningful representations of the data: raw complex form, Fourier transform, deep features and covariance. Furthermore, models exploiting covariance structure seem to be more robust to lack of data than the first-order models, even outperforming them in critically sparse scenarios (with only a few seconds of training data per class). Finally, the end-toend second-order model outperforms all other models in any of the experimented configurations, which points to the definite possibility of getting the best from the two worlds: a set of discriminative features learnt by convolutional layers, and a Riemannian processing on the resulting temporal covariance matrices.

Opening perspectives envisioned throughout this chapter are numerous. The drone radar simulator on the one hand, could surely benefit from further developments such as its refinement to real-life physical and maneuvering subtleties, and its extension to other kinds of UAVs, as it is for now limited to rotary-blade drones (the most common type, however). As for the ML perspective, the presented temporal architectures pave the way to more difficult and varied tasks than classification, for instance detection, labelling and segmentation, other key tasks in the field of radar operations. On the other hand, concerning the SOFTNet pipeline, many undiscussed technicalities on how to connect the first-and second-order networks give rise to interesting challenges; the briefly mentioned covariance estimation operator, in itself, constitutes a field of research by itself. Other ideas one might come up with include the smoothing of the connection: penalizing the gradients passing by the connection (either with L2 loss or an additional Stiefel constraint), using a smooth transition kernel after the first-order representation space, or after the covariance estimation, adding layers in between... On the architectural side, one may look back at Figure 3.8 and wonder whether a fusion of each of the four feature representations could be imagined; again, the semantic feature fusion promises major research material.

C h a p t e r

A D VA N C E S I N S P D N E U R A L N E T W O R K S

Chapter abstract

This chapter addresses the budding area of geometric deep architectures for covariance processing. Specifically, it is devoted to the invention of novel layers for the SPD neural network (SPDNet) presented earlier on. The driving challenge throughout the chapter will be the conformation of these layers to the particular manifold geometry of Symmetric Positive Definite (SPD) matrices. In this spirit, we first introduce a natural improvement of these architectures, based on the precepts of Information Geometry (IG). This improvement consists in a data-aware normalization of the final layer, the Euclidean mapping, which is of paramount importance to the SPDNet. We propose two distinct normalization schemes, one based on the data's barycenter, called Barycentric Normalization (BarNorm), the other on a learned parameter, called Parametric Normalization (ParNorm). We name Data-Aware Mapping Network (DAMNet) an architecture equipped with either of the normalization layers. Then, we introduce a Riemannian Batch Normalization (BatchNorm) algorithm, set within a novel Batch-Normalized SPDNet (SPDNetBN) architecture. Finally, we develop a convolutional layer for the SPD matrices. These novel layers make use of geometric operations on the manifold, notably the Riemannian barycenter, parallel transport and non-linear structured matrix transformations. We derive a new manifold-constrained gradient descent algorithm working in the space of SPD matrices, allowing to learn the BatchNorm, or BatchNorm layer. We validate our proposed approach with extensive experiments in three different contexts on diverse data types: drone recognition data from radar observations, and on emotion and action recognition datasets from video and Motion Capture (MoCap) data. Experiments show that the SPDNetBN systematically gives better classification performance compared with leading methods and a remarkable robustness to lack of data.

Introduction

Covariance matrices are ubiquitous in any statistical related field but their direct usage as a representation of the data for machine learning is less common. However, it has proved its usefulness in a variety of applications: object detection in images [START_REF] Tuzel | Region Covariance: A Fast Descriptor for Detection and Classification[END_REF]), analysis of Magnetic Resonance Imagery (MRI) data [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF], classification of electroencephalography (EEG) time series for Brain-Computer Interfaces (BCIs) [START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF]. It is particularly interesting in the case of structured temporal data since a global covariance matrix is a straightforward way to capture and represent the temporal fluctuations of data points of different lengths. As previously stated, the main difficulty resides in their inherently curved nature, which calls for the usage of tools from non-Euclidean geometry. A full overview of the Riemannian nature of Symmetric Positive Definite (SPD) matrices, along with a plethora of theoretical justifications and properties on the matter, is given in the book of [START_REF] Bhatia | Positive Definite Matrices[END_REF]. For this reason most of classification methods (which implicitly make the hypothesis of a Euclidean input space) cannot be used successfully on SPD matrices. Furthermore, this mathematical difficulty comes with a much higher cost: computation often involves eigenvalues decompositions or expensive iterative algorithms. Lack of straightforward parallelizability and super-quadratic algorithmic complexity often hinders the direct usage of covariate data in learning models.

Interestingly, relatively lightweight machine learning techniques can nonetheless produce state-of-the-art results as soon as the particular Riemannian geometry is taken into account. This is the case for BCI: [START_REF] Barachant | Classification of covariance matrices using a Riemannian-based kernel for BCI applications[END_REF] and [START_REF] Barachant | Multiclass Brain-Computer Interface Classification by Riemannian Geometry[END_REF] respectively use a nearest Riemannian barycenter scheme (described in Section 2.5.2) and a tangent-space Support Vector Machine (SVM) (described in Section 2. As mentioned before, a SPD neural network (SPDNet) architecture specifically adapted for these matrices has been proposed in the context of neural networks (Huang and Van Gool 2017). While the overall aspect is similar to a classical (Euclidean) network (transformations, activations and a final stage of classification), each layer processes a point on the SPD manifold; the final Riemanian layer transforms the feature manifold to a Euclidean space for further classification. Following this seminal work, more architectures have followed, proposing alternatives to the basic building blocks: in Dong All in all, most of the developments focus on improving or modifying existing blocks in an effort to converge to their most relevant form, both theoretically and practically; throughout this chapter, we will propose new building blocks, with a shared goal of normalizing the inner mechanics of the SPDNet architecture. As an additional, independent SPD building block, this novel layer is agnostic to the particular way the other layers are computed, and as such can fit into any of the above architectures; we do however focus on the original one (Huang and Van Gool 2017).

In our first proposition, we introduce a new architecture, called a Data-Aware Mapping Network (DAMNet), which focuses on the final layer of the SPDNet, the Euclidean mapping: we argue that a better projection can be done by making the layer dependent on the data. The original projection layer relies on the Log-Euclidean Metric (LEM) framework [START_REF] Arsigny | Log-Euclidean metrics for fast and simple calculus on diffusion tensors[END_REF], which endows the manifold of SPD matrices with a Lie group structure. This framework is much simpler than the full Riemanian setting and allows efficient computations while keeping good theoretical properties [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF]. While useful, this framework is only a particular case: we thus introduce an improved projection layer working in the broader Riemaniann setting. This new projection maps the points to the tangent space of some reference matrix and comes in two variants: a barycentric projection, called Barycentric Normalization (BarNorm), which uses the Riemannian barycenter as the reference matrix; and a parametric projection, called Parametric Normalization (ParNorm), which uses a parameter SPD matrix, learnt during training.

Our second main contribution is inspired by the well-known and well-used Batch Normalization (BatchNorm) layer, introduced in the context of (Euclidean) CNNs for Computer Vision (CV) tasks in [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF]. This layer makes use of batch centering and biasing, operations which in our case need to be defined on the SPD manifold. Although the overall structure of the original BatchNorm is preserved, its generalization to SPD matrices requires geometric tools on the manifold, both for the forward and backward pass. The overall architecture, which we call Batch-Normalized SPDNet (SPDNetBN), is expected to perform better than either the SPDNet or DAMNet.

In this study, we further assess the particular interest of batch-normalized SPDNets in the context of learning on scarce data with lightweight models: indeed, many fields are faced with costly, private or evasive data, which strongly motivates the exploration of architectures naturally resilient to such challenging situations. Medical imagery data is well-known to face these issues [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF], as is the field of drone radar classification [START_REF] Brigant | Geometric barycenters of time/Doppler spectra for the recognition of non-stationary targets[END_REF]. Indeed, radar signal acquisition is prohibitively expensive, the acquired data is usually of confidential nature, and drone classification in particular is plagued with an ever-changing pool of targets, which we can never reasonably hope to encapsulate in comprehensive datasets. Furthermore, hardware integration limitations, such as for real-time embedded systems-on-chip in the context of radar systems, further motivate the development of lightweight models based on a powerful representation of the data.

To summarize our proposed contributions:

• A neural architecture called Data-Aware Mapping Network (DAMNet) with two new layers (which are mutually exclusive):

¨A barycentric normalization layer using a Riemannian barycenter, called BarNorm;

¨A parametric normalization layer using a SPD-constrained parameter, called ParNorm;

• A neural architecture called Batch-Normalized SPDNet (SPDNetBN) with one new layer:

¨A Riemannian BatchNorm layer for SPD neural networks, respecting the manifold's geometry;

• A generalized gradient descent allowing to learn the DAMNet and SPDNetBN models;

• A convolutional layer for SPD matrices;

• Extensive experimentations on three datasets from three different fields.

The chapter is organized as follows: we begin Section 4.2 by describing the DAMNet architecture with the BarNorm or ParNorm layers, along with a brief mathematical reminder of its driving motivation. We then move on in Section 4.3 to describe our proposed Riemannian BatchNorm algorithm; again, essential concepts to the layer are debriefed. Section 4.4 ends the technical contributions with by devising the projected gradient descent algorithm for learning the aforementioned layers. Finally, we validate experimentally and assess the properties of our proposed architectures in Section 4.6.

Data-Aware Mapping Network

The DAMNet architecture improves upon the original SPDNet. The core incentive behind this improvement lies in the formal generalization of the background mathematical framework the Euclidean mapping layer us based upon. Specifically, the Log Eigenvalues (LogEig) layer described in Section 3.2.1 is explicitly defined within the realms of the LEM framework; we propose to elevate the notion to the more general Riemannian framework. We begin by briefly describing this LEM framework.

Log-Euclidean Metric

We recall the notations S + * the space of SPD matrices and S + the space of symmetric matrices. Let us define the logarithmic product of two SPD matrices:

∀P 1 , P 2 ∈ S + * , P 1 P 2 = exp(log(P 1 ) + log(P 2 )) (4.1)
We now endow S + * with the group structure defined by the logarithmic product. We claim that (S + * , I d ) is an abelian Lie group, the neutral element I d being the identity matrix and the inversion the standard matrix inversion (•) -1 . This is easy to see, as the logarithmic product commutes (yielding a group structure) and involves smooth differentiable operations in matrix space (yielding a Lie group structure). Furthermore, it reduces to the standard matrix product when the matrices commute: The main idea to retain is the passage from the manifold to a certain Euclidean space.

∀P 1 , P 2 ∈ S + * | P 1 P 2 = P 2 P 1 , P 1 P 2 = P 1 P 2 (4.2)
The Lie group exponential associated to the Lie group is by definition the exponential mapping Exp of the associated manifold S + * at the neutral element; recall from its definition in Equation 2.30, that Exp I d = exp, the standard exponential mapping. Its inverse is thus the standard logarithm, and thus the associated Lie algebra s corresponds to S + , as log : S + * -→ S + constitutes a diffeomorphism. The LEM noted δ Log is then defined as the Euclidean metric on the Lie algebra:

δ Log (P 1 , P 2 ) = || log(P 1 ) -log(P 2 )|| (4.3)
From the elements described above, we now see that the LogEig layer (Equation 3.3) corresponds to a special case of the logarithmic mapping on S + * (Equation 2.30), specifically with the reference G point being the identity matrix. We illustrate this important notion in the context of a SPDNet in Figure 4.1. In fact, the LEM itself is a special case of the Affine Invariant Riemannian Metric (AIRM) in the sub-group of commuting SPD matrices. Our initial main idea thus consists in generalizing the concept by considering data-and geometry-aware mappings based on the more general AIRM framework, i.e. in practice mapping the points at the LogEig layer other than at the identity matrix. The question then becomes, how to chose this reference point.

Barycentric Normalization

Our first idea is to map data points at their barycenter's tangent space. It is important here to distinguish the Riemannian (or geometric) barycenter G defined on the manifold from the standard arithmetic mean 1 N i≤N P i . Note that the arithmetic mean might in the general case not even belong to the manifold, although it actually does in S + * . Recall Section 2.5.2 for the computation of G using the Karcher flow algorithm.

As the batch barycenter changes with each batch, the tangent space will be different at each iteration. Posterior to the mapping, classification is performed by learning a separating hyperplane, i.e. a standard dense layer. However, this process may deal poorly with an ever-evolving projection space to learn on: indeed, each step in the optimization would yield a different vector space. Therefore, we instead center, or parallel transport, each point P

G := G {P (k) i } i≤N
(barycenter of the mini-batch)

X (k) = log(G -1 2 P (k) G -1 2 ) (4.4)

Parametric Normalization

In the same vein as previously, we aim to generalize the Euclidean projection from the LEM framework to the AIRM framework, for which a reference point G is required. G can be the barycenter, which takes data into account by optimizing a minimal dispersion criterion within the batch. Since this criterion has no reason to be related to the training objective, we introduce a second data-driven mapping layer, this time explicitly optimizing the SPD reference point G for the global training loss.

Contrary to the BarNorm layer described previously, G is now an additional parameter of the network to be learnt alongside the others. The learning mechanics linked to such an operation are covered further on in this chapter. The difference with the standard LogEig then reduces to a congruence by G -1 2 before applying the log. However, G needs to belong to S + * , which we will need to enforce in the gradient update, as done in [START_REF] Yger | Supervised LogEuclidean Metric Learning for Symmetric Positive Definite Matrices[END_REF]. Note that a similar constraint applies to the Bilinear Mapping (BiMap) weights (Equation 3.1) of the SPDNet, only the latter is orthogonal rather than symmetric. Finally, the parametric projection for the learnable logarithm of the transformed data points P (k) at the last SPD layer reads:

X (k) = log(G -1 2 P (k) G -1 2 ), G ∈ S + * (4.5)

DAMNet architecture

To sum up, the difference between a SPDNet and a DAMNet is the Euclidean mapping, which in the case of the former is fixed to the matrix logarithm, contrary to the latter which depends on a data-aware reference point M , defined either as an additional parameter or a barycenter; as such; SPDNet is a special case of DAMNet, as illustrated in Equation 4.6 and in Figure 4.2:

X (k) = log(M -1 2 P (k) M -1 2 ) , with M =        Id → LogEig with no regularization G batch → BarNorm (barycenter) G ∈ S + * → ParNorm (gradient descent) (4.6)
In the DAMNet architecture, the two normalization schemes BarNorm and ParNorm are mutually exclusive. However, they are not incompatibe, and can in principle be chained. Moreover, while our theoretical justification for their existence focused on the LogEig layer, it still holds at any echelon of the network. The centering of a batch within the network, followed by a parametric displacement of this batch, may remind the acute reader of the famed Batch Normalization algorithm. We thus extend in the next section the ideas underlying the DAMNet, to develop a Batch-Normalized SPDNet (SPDNetBN) architecture.

Batch-Normalized SPDNet

This section is dedicated to the Riemanian BatchNorm and its associated architecture, the SPDNetBN, a core contribution in our works. As stated above, it generalizes our DAMNet architecture, which itself generalizes the original SPDNet. The Riemannian BatchNorm shares the goal of its Euclidean counterpart, that is essentially the normalization of data conditioning through the network by the reduction of internal covariate shift [START_REF] Ioffe | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[END_REF]. Recall Section 2.3.2 for a brief overview of the standard, Euclidean BatchNorm algorithm. We begin the section by clarifying a matter of importance, sheepishly skimmed over in the previous section. While we initially introduced the normalization schemes as different logarithmic mappings, in reality they are defined from Parallel Transport (PT). Similarly, the centering and addition of bias within the proposed Riemannian BatchNorm will involve PT. We thus clarify this ambiguity in the context of the Riemannian BatchNorm.

Centering SPD matrices using Parallel Transport

The Euclidean BatchNorm involves centering and biasing the batch B, which is done via subtraction and addition. However on a curved manifold, there is no such group structure in general, so these seemingly basic operations are ill-defined. To shift SPD matrices around their mean G or towards a bias parameter G ∈ S + * , we propose to rather use parallel transport on the manifold (Amari 2016).

Parallel Transport and SPD transport

In short, the operator Γ P 1 →P 2 (S) of a vector S ∈ T P 1 in the tangent plane at P 1 , between P 1 , P 2 ∈ S + * defines the path from P 1 to P 2 such that S remains parallel to itself in the tangent planes along the path. The geodesic γ P 1 →P 2 is itself a special case of the PT, when S is chosen to be the direction vector γ P 1 →P 2 (0) from P 1 to P 2 . The expression for PT is known on S + * :

∀S ∈ T P 1 , Γ P 1 →P 2 (S) = (P 2 P -1 1 ) 1 2 S (P -1 1 P 2 ) 1 2 ∈ T P 2 (4.7)
The equation above defines PT for tangent vectors, while we wish to transport points on the manifold. To do so, we simply project the data points to the tangent space at the initial point P 1 using the logarithmic mapping at P 1 , parallel transport the resulting vector from Equation 4.7 towards P 2 which we then map back to the manifold using exponential mapping at P 2 . We show in Theorem 4.1 that the resulting operation, which we call SPD transport, turns out to be exactly the same as the formula above, which is not an obvious result in itself. Note that by abuse of notation, we also use Γ P 1 →P 2 to denote the SPD transport. We first show two useful lemmas for the proof: These values are by construction all strictly positive, therefore they are the unique square roots of the respective squares, and it thus suffices to prove A 2 = B 2 . We have:

B 2 = P 1 2 1 (P -1 2 1 P 2 P -1 2
1 )

1 2 P -1 2 1 P 1 2 1 (P -1 2 1 P 2 P -1 2
1 )

1 2 P -1 2 1 = P 1 2 1 (P -1 2 1 P 2 P -1 2
1 )P -1 2 1 = P 2 P -1 1 = A 2 ⇒ B = A Lemma 4.2. ∀P 1 , P 2 ∈ S + * , X := P -1 2 2 BP 1 2
1 is unitary, i.e. XX T = X T X (using the same notations as in Lemma 4.1).

Proof.

XX T = P -1 2 2 B P 1 2 1 (P -1 2 1 P 2 P -1 2 1 ) 1 2 I d P -1 2 1 P 1 2 1 I d P 1 2 1 P -1 2 1 (P 1 2 1 P 2 P -1 2 1 ) 1 2 T P 1 2 1 B T P -1 2 2 = (P -1 2 2 I d P 1 2 1 ) ∈S + * (P -1 2 1 P 2 I d P -1 2 1 ) 2 2 (P 1 2 1 P -1 2 2 ) = P -1 2 2 P 2 P -1 2 2 = I d
As a consequence of X being unitary, we have that exp(XM X T ) = X exp(M )X T for any matrix M (by definition of the matrix exponential exp(M ) = N M k k! ). We now show the main result, establishing the SPD transport Γ P 1 →P 2 (S) of S ∈ S + * from P 1 to P 2 . Theorem 4.1. SPD transport of SPD matrix S from P 1 to P 2 on S + * .

∀P 1 , P 1 , S ∈ S + * , Γ P 1 →P 2 (S) = (P 2 P -1 1 )

1 2 S (P -1 1 P 2 )

1 2 ∈ S + * (4.8)
Proof.

Γ P 1 →P 2 (S) = Exp P 2 Γ P 1 →P 2 Log P 1 (S) (by definition) = P 1 2 2 exp P -1 2 2 =A=B (P 2 P -1 1 ) 1 2 P 1 2 1 X log(P -1 2 1 SP -1 2
1 ) P 1 2 1 =A T =B T (P -1 1 P 2 ) 1 2 P -1 2 2 X T P 1 2 2 (Lemma 4.1) = P 1 2
2 X exp log(P Note that the transport is transitive [START_REF] Yair | Parallel Transport on the Cone Manifold of SPD Matrices for Domain Adaptation[END_REF], so both operations can be concatenated in a single transport. We can now fully define the batch centering and biasing:

-1 2 1 SP -1 2
Centering from G := Bar(B): ∀i ≤ N, Pi = Γ G→I d (P i ) = G -1 2 P i G -1 2 (4.9a) Biasing towards parameter G: ∀i ≤ N, Pi = Γ I d →G ( Pi ) = G 1 2 Pi G 1 2 (4.9b)
At this point, the next step in the original BatchNorm algorithm would be the standardization. We instead choose, amongst the possible definitions of normal distributions on S + * , one which naturally involves no notion of standard deviation.

Statistical distribution on SPD matrices

In traditional Neural Networks (NNs), BatchNorm is defined as the centering and standardization of the data within one batch, followed by the multiplication and addition by parameterized variance and bias, to emulate the data sampling from a learnt Gaussian distribution. In order to generalize to batches of SPD matrices, we must first define the notion of Gausian density on S + * . Although this definition has not yet been settled for good, several approaches have been proposed. In Jaquier and Calinon (2017), the authors proceed by introducing mean and variance as second-and fourth-order tensors. On the other hand, Said et al. (2017) derive a scalar variance. In another line of work synthesized in [START_REF] Barbaresco | Jean-Louis Koszul and the Elementary Structures of Information Geometry[END_REF], which we adopt in this work, the Gaussian density is derived from the definition of maximum entropy on exponential families using Information Geometry (IG) on the cone of SPD matrices. In this setting, the natural parameter of the resulting exponential family is simply the Riemannian mean; in other words, this means the notion of variance, which appears in the Eucidean setting, takes no part in this definition of a Gaussian density on S + * . Specifically, such a density p on SPD matrices P of dimension n writes: p(P ) ∝ det(α G -1 )e -tr(α G -1 P ) , with α = n + 1 2 (4.10)

In the equation above, G is the Riemannian mean of the distribution. Again, there is no notion of variance: the main consequence is that a Riemannian BatchNorm on SPD matrices will only involve centering and biasing of the batch.

Riemannian BatchNorm algorithm

While the normalization is done on the current batch during training time, the statistics used in inference are computed as running estimations. For instance, the running mean over the training set, noted G S , is iteratively updated at each batch. In a Euclidean setting, this would amount to a weighted average between the batch mean and the current running mean, the weight being a momentum typically set to 0.9. The same concept holds for SPD matrices, but the running mean should be a Riemannian mean weighted by η, i.e. Bar (η,1-η) (G S , G B ), which amounts to transporting the running mean towards the current batch mean by an amount (1η) along the geodesic. We can now write our full Riemannian BatchNorm in Algorithm 4.1.

In practice, Riemannian BatchNorm is appended after each BiMap layer in the network. Though we have now introduced the full inference phase of the algorithm, its backpropagation remains to be solved. The next section dives in the gory details of Riemannian manifold-constrained optimization, addressing the gradient derivations both for the DAMNet architecture and the Riemanian BatchNorm, as they share common ground.

Riemannian manifold-constrained optimization

The specificities of a the proposed DAMNet architecture and BatchNorm algorithm are the non-linear manipulation of manifold values in both inputs and parameters and the use of a Riemannian barycenter. Here we present the two results Pi ← Γ G S →I d (P i ) // center batch using set statistics

Algorithm

3:

Pi ← Γ I d →G ( Pi ) // bias batch using learnt parameter 4: end for 5: return normalized batch { Pi } i≤N necessary to correctly fit the learning of the Riemannian BatchNorm in a standard backpropagation framework.

Learning with SPD constraint

The bias parameter matrix G of the Riemannian BatchNorm is by construction constrained to the SPD manifold. However, noting L the network's loss function, the usual Euclidean gradient ∂L ∂G , which we note ∂G eucl , has no particular reason to respect this constraint. To enforce it, ∂G eucl is projected to the tangent space of the manifold at G using the manifold's tangential projection operator ΠT G , resulting in the tangential gradient ∂G riem . The update is then obtained by computing the geodesic on the SPD manifold emanating from G in the direction ∂G riem , using the exponential mapping Exp G defined in Equation 2 The Euclidean gradient is projected to the tangent space, then mapped to the manifold through retraction, which can be the exponential mapping.

We illustrate this two-step process in Figure 4.3, explained in detail in Edelman et al. (1998), which allows to learn the parameter in a manifold-constrained fashion. However, this is still not enough for the optimization of the layer, as the BatchNorm involves not simply G and G, but G 1 2 and G -1 2 , which are structured matrix functions of G, i.e. which act non-linearly on the matrices' eigenvalues without affecting its associated eigenspace. The next subsection deals with the backpropagation through such functions.

Structured matrix backpropagation

Classically, the functions involved in the chain rule are vector functions in R n , whereas we deal here with structured (symmetric) matrix functions in the S + * , specifically the square root (•) 1 2 for the parametric bias and the inverse square root (•) -1 2 for the barycenter (in Equation 4.9 and Equation 4.6).

General principles

We recall, a neural network of depth L can be seen as a hierarchical function 1) of input data X (0) := X and associated label Y composed of L elementary blocks and a loss function l penalizing the distance of the output X (L) to Y . We also note

f = l • f (L) • . . . • f (
L (k) := l • f (L) • . . . • f (k) the

∂X

given the forward pass P -→ X, is thus required for the backpropagation through the Riemannian BatchNorm and the ParNorm layers to be correct. Note that a similar requirement applies to the ReEig and LogEig layers, respectively with a threshold and log function. We generically note f a monotonous non-linear function; both (•) 1 2 and (•) -1 2 check out this hypothesis. To be clear, the formula for the gradients through the ReEig and LogEig layers stems for a shared, more general formula, which we then put to use in our application, i.e. the square root and inverse square root functions. This general formula for the gradient of f , applied on a SPD matrix' eigenvalues (σ i ) i≤n grouped in Σ's diagonal, was independently developed by [START_REF] Ionescu | Matrix Backpropagation for Deep Networks with Structured Layers[END_REF] and [START_REF] Brodskiȋ | Thirteen Papers on Functional Analysis and Partial Differential Equations[END_REF]. Theorem 4.2. Given the forward pass f : P -→ X, the backward pass

∂L (k+1) ∂X -→ ∂L (k)
∂P is constructed as follows:

∂L (k) ∂P (P, Y ) = F * (k) ∂L (k+1) ∂X (X, Y ) (4.12)
In the equation, F * (k) is the adjoint operator to the directional derivative (Df (k) ) P (dP ) of f (k) at point P in direction dP . Conceptually, F (k) is the operator mapping dP to dX.

Proof. The proof, as overviewed in [START_REF] Ionescu | Matrix Backpropagation for Deep Networks with Structured Layers[END_REF] and [START_REF] Yger | Supervised LogEuclidean Metric Learning for Symmetric Positive Definite Matrices[END_REF], stems for equating variation to the loss function given a perturbation dP in P at layer (k), and the resulting perturbation dX of X at the following layer (k + 1).We simplify the notations by dropping the layer indices (k) and (k + 1) such that L := L (k+1) and using L (k) = L • f . We retrospectively introduce F : dP -→ dX. The proof thus initiates with equating the loss variation at the two successive layers:

L • f (P + dP ) -L • f (P ) = L(X + dX) -L(X)
A first-order Taylor expansion then procedes; we will further omit the remainders as the method is first-order only:

⇔ ∂L • f ∂P , dP + O(||dP || 2 ) = ∂L ∂X , dX + O(||dX|| 2 ) (Taylor expansion) ⇔ ∂L • f ∂P , dP = ∂L ∂X , F(dP ) (by definition) ⇔ ∂L • f ∂P , dP = F * ( ∂L ∂X ), dP (adjoint operation) ⇔ ∂L • f ∂P = F * ( ∂L ∂X ) (inner product valid ∀ dP )
In the Euclidean case of Equation 4.12, F * (k) would typically reduce to the standard linear form of the chain rule. We now show a practical usage of this generalized chain rule, through the demonstration of a central example: the nonlinear eigenvalue modification, generically noted: Proof.

U T U = I d ⇒ U T Z + Z T U = 0 (noting Z := dU ) ⇒ Ω ∈ S †
dU = U (K T (U T dP U )) (with K ij = 1 σi-σj (i = j) 0 (i = j) ) (4.14)
Proof.

P = U ΣU T ⇒ dP = dU ΣU T + U dΣU T + U ΣdU T ⇒ U T dP U = (U T dU )Σ + dΣ + Σ(U T dU ) T
In the above equation, we know from the proof of Lemma 4.3 that U T dU is skew-symmetric; thus, its diagonal is zero, and therefore its multiplication with diagonal matrix Σ yields a zero-diagonal matrix. Furthermore, dΣ is also diagonal by construction, so in the end we have dΣ = 0 + (U T dP U ) diag + 0. Continuing the equalities, we have:

⇒ X = (U T dU )Σ + A diag -Σ(U T dU ) (with A := U T dP U ) ⇒ à = (U T dU )Σ -Σ(U T dU ) (with à := A -A diag ) ⇒ (U T dU ) ij σ j -(U T dU ) ij σ i = Ãij (i = j) (U T dU ) ij = 0 (i = j) ⇒ (U T dU ) ij = 1 σ j -σ i Ãij (i = j) (U T dU ) ij = 0 (i = j) ⇒ U T dU = K T à ⇒ dU = U (K T (U T dP U )) (because K T A diag = 0)
Finally, we show the backward pass of the Eig function in Theorem 4.3; all we need now given Lemma 4.4 is to compute the adjoint operator:

∂L • Eig ∂P = U (K T (U T ∂L ∂U )) + ( ∂L ∂Σ ) diag U T (with K ij =    1 σ j -σ i (i = j) 0 (i = j) ) (4.15)
Proof. To compute the adjoint aoperator, only a few linear algebra tricks are involved, which we bunch in one single line:

∂L ∂U , dU + ∂L ∂Σ , dΣ = ∂L ∂U , U (K T (U T dP U )) + ∂L ∂Σ , (U T dP U ) diag = U (K T (U T ∂L ∂U ))U T , dP + U ( ∂L ∂Σ ) diag U T , dP = U (K T (U T ∂L ∂U ))U T + U ( ∂L ∂Σ ) diag U T , dP ⇒ ∂L • Eig ∂P = U (K T (U T ∂L ∂U )) + ( ∂L ∂Σ ) diag U T
We now derive in Theorem 4.4 the backward pass for the eigenvalue modification g : (U, Σ) -→ X: Proof.

X = U g(Σ)U T ⇒dX = dU g(Σ)U T + U g (Σ)dΣU T + U g(Σ)dU T ⇒dX = Msym:= M +M T 2 (dU g(Σ)U T ) sym +U g (Σ)dΣU T
Given this newfound differential form, we now compute its adjoint; similarly to before, we use the inner product adjoint property:

∂L ∂X , dX = ∂L ∂X , (dU g(Σ)U T ) sym + U g (Σ)dΣU T = g (Σ)U T ∂L ∂X U, dΣ + 2( ∂L ∂X ) sym U g(Σ) ⇒ ∂L•g ∂Σ = 2( ∂L ∂X )) sym U g(Σ) ∂L•g ∂U = g (Σ)U T ( ∂L ∂X ))U
The investigative reader may refer to other established references of matrix computation and calculus for a deeper understanding of the matter, such as [START_REF]Matrix and tensor calculus with applications to mechanics, elasticity, and aeronautics[END_REF], [START_REF] Bers | Review: A. D. Michal, Matrix and tensor calculus with applications to mechanics, elasticity and aeronautics[END_REF], [START_REF] Dwyer | Symbolic Matrix Derivatives[END_REF], [START_REF] Papadopoulo | Estimating the Jacobian of the Singular Value Decomposition: Theory and Applications[END_REF], Petersen et al. (2006), [START_REF] Giles | Collected Matrix Derivative Results for Forward and Reverse Mode Algorithmic Differentiation[END_REF], and Magnus and Neudecker (2019)

Takeaway backpropagation formulas

We now summarize the findings above in one simple formula, obtained by chaining the backward passes of Eig and g (we omit the calculations). In short: given the function P -→ X := g(P ) and the succeeding gradient ∂L (k+1) ∂X , the output gradient ∂L (k) ∂P is:

∂L (k) ∂P = U L (U T ( ∂L (k+1) ∂X )U ) U T (4.16)
The equation above, also decribed in Nielsen and Bhatia (2013), is called the Daleck ȋi-Kre ȋn formula and dates back to 1956, (but was translated from Russian 9 years later), predating the other formulation by 60 years. It involves the eigenspace U of the input matrix P , and the Loewner matrix L, or finite difference matrix defined by:

L ij = g(σ i )-g(σ j ) σ i -σ j if σ i = σ j g (σ i ) otherwise (4.17)
In the case at hand, (•) In summary, the Riemannian barycenter (approximation via the Karcher flow for a batch of matrices, or exact formulation for two matrices), the parallel transport and its extension on the SPD manifold, the SPD-constrained gradient descent and the derivation of a non-linear SPD-valued structured function's gradient allow for training and inference of the proposed Riemannian BatchNorm algorithm.

-1 2 = -1 2 (•) -3 2 and (•) 1 2 = 1 2 (•) -

Convolution for covariance time series

Here we introduce a novel temporal convolution layer for time series of SPD matrices, inspired from the usual Euclidean 1D convolution layer, and using the BiMap layer defined above. This layer produced no satisfactory results, so does not appear in the experimental validations.

Single-channel convolution

Input P is a time series of SPD matrices of dimension n i and of duration T i , i.e. an input of shape (T i , n i , n i ). The convolution parameter K is a kernel of length k of BiMap operators of shape (n i , n o ), i.e. a block of shape (k, n i , n o ). The output X of the layer, of duration T o and overal shape (T o , n o , n o ), is computed as a valid temporal convolution using the bilinear mapping as base operation:

X := P * K ∀t ≤ T o , X t = 1 k l≤k K T l P t+k-1-l K l (4.18)
Stride and padding can also be used to influence the duration T o of the output signal.

Weighted average convolution

A key difference between the Euclidean 1D convolution and the one defined above is the BiMap operator preserves the matrix norm (up to normalization by the dimension), which gives the same importance to all the SPD "pixels" in the time series; essentially, the convolution is valid, but weightless. Therefore, we introduce weights w of shape (k, ) to the convolution, constrained to represent a valid distribution, i.e. ∀l ≤ k, w l ≥ 0 and l≤k w l = 1. The convolution then becomes:

X := P * w K ∀t ≤ T o , X t = l≤k w l K T l P t+k-1-l K l (4.19)

Riemannian convolution using the weighted Fréchet mean

The convolution as defined above takes the form of an arithmetic mean; one could instead imagine using a Riemannian mean, better-suited to the SPD data:

X := P * w K ∀t ≤ T o , X t = Bar w l≤k (K T l P t+k-1-l K l ) (4.20)
In the equation above, one must take care not to confuse the weighted Fréchet mean of the operands with the unweighted Fréchet mean of the weighted operands.

Raw signal

S 0 • • • P 0 S 1 • • • P 1 S 2 • • • P 2 S 3 • • • P 3 S 4 • • • P 4 w 0 .K 0 w 1 .K 1 w 2 .K 2 BiMap filter bank (k = 3) X 0 X 1 X 2 X := P * K w Output covariance time series X (T o = ((5-3)+0) 1 + 1)=3

Multi-channel convolution

More realistically, we wish to instigate the network's potential expressiveness by further building multi-channeled SPD representations. We do so in the usual fashion; input P is now of shape

(T i , C i , n i , n i ), kernel K of shape (C o , C i , k, n i , n o ),
weights w are still of shape (k, ), and output X of shape (T o , C o , n o , n o ), and is computed as follows:

∀c o ≤ C o , X co = c i ≤C i P c i * w K co,c i (4.21)
In all definitions above, the convolution uses the BiMap layer (and therefore induces dimension reduction), which is novel.

Experimental validation

Here we evaluate the gain in performance of the SPDNetBN against the baseline SPDNet on different tasks: micro-Doppler (µ-D) radar data classification, emotion recognition from video, and action recognition from MoCap data. We call the depth L of an SPDNet the number of BiMap layers in the network, and denote the dimensions as {n 0 , • • • , n L }. The vectorized input to the final classification layer is thus of length n 2 L . To be precise, the symmetry of the matrix allows to gather a vector of length n(n+1)

2

, but our experiments seemed to slightly favor using the whole matrix. All networks are trained for 200 epochs using Stochastic Gradient Descent (SGD) with momentum set to 0.9 with a batch size of 30 and learning rate 5e -3 , 1e -2 or 5e -2 . We provide the data in a pre-processed form alongside the PyTorch (Paszke et al. 2017) code for reproducibility purposes, and also show snippets to demonstrate its ease of use. We recall that the SPDNetBN appends a Riemannian BatchNorm after each BiMap layer of the associated SPDNet. Finally, we also report performances of shallow learning method on SPD data, namely the Minimum Riemannian Distance to Riemannian Mean (MRDRM) scheme presented in Section 2.5.2, in order to bring elements of comparison between shallow and deep learning on SPD data.

Drones recognition

Our first experimental target focuses on drone µ-D radar classification. First we validate the usage of our proposed method over a baseline SPDNet, and also compare to the previously exposed deep learning methods. Then, we study the models' robustness to lack of data, a challenge which, as stated previously, plagues the task of radar classification and also a lot of different tasks. Experiments are conducted on a confidential dataset of real recordings issued from the North Atlantic Treaty Organization (NATO). To spur reproducibility, we also experiment on synthetic, publicly available data.

Radar data description As previously described, the radar signal is the result of an emitted wave reflected on a target; as such, one data point is a time series of N values, which can be considered as multiple realizations of a locally stationary centered Gaussian process, as done in [START_REF] Charon | A new approach for target detection in radar images based on geometric properties of covariance matrices[END_REF]. The signal is split in windows of length n = 20, the series of which a single covariance matrix of size 20 × 20 is sampled from, which represents one radar data point. We refer to Figure 3.10 for a visual clarification of the splitting operation. The NATO data features 10 classes of drones, whereas the synthetic data is generated by a realistic simulator of 3 different classes of drones following the protocol previously described. We chose here to mimick the real dataset's configuration, i.e. we consider a couple of minutes of continuous recordings per class, which correspond to 500 data points per class, so 1500 in total.

Comparison of Riemannian and Euclidean models

We test the SPD-based models in a {20, 16, 8}, 2-layer configuration for the synthetic data, and in a {20, 16, 14, 12, 10, 8}, 5-layer configuration for the NATO data, over a 5-fold crossvalidation, split in a train-test of 75% -25%. We also wish to compare the Riemannian models to the common Euclidean ones, which consitute our state-ofthe-art in µ-D classification. We compare two Fully Temporal Convolutional Networks (FTCNs): the first one is the same as in Section 3.2.2; for the second one, the number of parameters is set to approximately the same number as for the SPD neural nets, which amounts to an unusually small Deep Neural Network (DNN). This second "deep" architecture is chosen to ensure fairness from the viewpoint of parameter intesity. All in all, the SPD neural models and the small FTCN on the one hand, and the full-size FTCN on the other hand respectively have approximately 500 and 10000 parameters.

Table 4.1 reports the average accuracies and variances on the NATO data of the SPDNet against the Euclidean models, and the MRDRM scheme. A first obvious result, is that the MRDRM performs relatively poorly, albeit showcasing great stability. Secondly, the Euclidean method outperforms both Riemannian methods by a large margin. However, the Euclidean method crumbles when given ten times less data to learn upon, while the Riemanian method remains quite stable, to the point of reversing the tendency. This robustness to lack of data, already observed in the previous chapter, will recur in the following experiments, providing a strong argument in favour of the Riemannian methods. We observe from these results a strong gain in performance on the SPDNetBN and DAMNets over the SPDNet and over the small FTCN, which validates the usage of the Riemannian BatchNorm along with the exploitation of the geometric structure underlying the data. All in all, we reach better performance with much fewer parameters, which again is a key feature for radar classification, and much others. The raw SPDNet, though not competing with the FTCN when all data is available, remains more robust to the lack data. The SPDNetBN on the other hand, exhibits a strong gain in performance to the point of competing with the deep model. Note that we iterate only once for the barycenter estimation in the BarNorm or BatchNorm (K = 1 in the Karcher flow). This is most likely due to the fact, that within the normalization of a batch, a noisy estimate is probably preferable as the estimation itself is restricted to a small sample size. This hypothesis was validated in experiments we do not include in the text: in the end, using the single iteration yielded better results than more iterations, and no iteration (i.e., the arithmetic mean).

Finally, in the interest of convergence analysis, we also report learning curves for the model's accuracy with and without Riemannian BatchNorm in Figure 4.5. In Figure 4.5, the SPDNetBN exhibits a steeper learning curve. For the same number of epochs, it does take more time overall, but reaches better accuracy much faster, allowing to reduce the number of epochs. A similar results is found for the BarNorm regularization, while the ParNorm yields an overall less stable curve, which is a reasonable behaviour condsidering the additional SPD parameter to be learnt, without the benefit of normalizing around the barycenter first.

For the sake of expression, we also show in Figure 4.6 a confusion matrix of the SPDNetBN model for the NATO dataset. Notice the strong imbalance throughout the classes precisions, due to the same imbalance found within the class samples. A thorough description of the NATO dataset is geiven in Appendix B. Robustness to lack of data As stated previously, it is of great interest to consider the robustness of learning algorithms when faced with a critically low amount of data. The last line in both Table 4.1 and Table 4. We can conclude from these that the SPDNetBN both exhibits higher robustness to lack of data and performs much better than the Euclidean deep methods with much fewer parameters. When the available training data allowed skyrockets, we do observe that the FTCN comes back to par with the SPDNetBN to the point of outperforming it by a small margin in the extremal scenario; in the meantime, the SPDNet lags behind by a large margin to the SPDNetBN, which thus seems to benefit strongly from the normalization. In any case, the manifold framework seems well suited in a scarce data learning context, especially considering the introduced normalization layers, which again pinpoints the interest of taking into account the geometric structure of the data, all the while without introducing specific prior knowledge during training.

SPD Data separability

To study how the reference projection point influences the separability of datapoints, we choose two arbitrary classes c 1 and c 2 in the radar dataset; we map all points from the test set belonging to c 1 and c 2 to the final SPD representation layer of a pre-trained SPDNet; we compute the barycenter G of all mapped points, and the discretized geodesic [START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF]), parameterized by t ∈ [0, 1] linking the identity matrix I d to G; we also further extrapolate the geodesic for better visualisation; we compute the Euclidean projections of the mapped SPD points at all reference points along the geodesic; t = 0 corresponds to LogEig, and t = 1 to the BarNorm; for one given projection reference point, we evaluate class separability as the Euclidean distance between the Euclidean class means of the projected points; separability is evaluated at each reference point along the geodesic; our hope is to observe high separability at the barycenter (i.e. at t = 1).

1. We choose two arbitrary classes c 1 and c 2 ;

2. We map all points from the test set belonging to c 1 and c 2 to the final SPD representation layer of a pre-trained SPDNet;

3. We compute the barycenter G of all mapped points, and the discretized geodesic, parameterized by t ∈ [0, 1] linking the identity matrix I d to G; we also further extrapolate the geodesic for better visualisation;

4. We compute the Euclidean projections of the mapped SPD points at all reference points along the geodesic; t = 0 corresponds to LogEig, and t = 1 to the BarNorm;

5. For one given projection reference point, we evaluate class separability as the Euclidean distance between the Euclidean class means of the projected points;

6.

Separability is evaluated at each reference point along the geodesic; our hope is to observe high separability at G (i.e. at t = 1).

The red curve in Figure 4.8 measures the distance between the two classes: we see it increasing along the geodesic. Note it is not obvious (though it is intuitive) that the barycenter criterion should provide this result, as class separability on the manifold does not necessarily translate to the tangent space; in the case of S * + however, negative curvature actually ensures this. This increase experimentally validates the interest of the batch centering BarNorm scheme.

For a more extensive visual study, we also shoot geodesics in random directions around the identity (t = 0, green curves), and around the barycenter G (t = 1, blue curves), and also plot the corresponding class separability at each reference point along each random geodesic. We observe that the neighborhood of G seems to constitute local maximum in class separability. This observation validates the practical relevance of information geometric theoretical elements involved in our normalization schemes in the development of manifold-valued neural networks. Interest of the Riemannian barycenter Within the BarNorm scheme and the Riemannian BatchNorm, the Fréchet mean is computed using the Karcher flow (Algorithm 2.2). Actually, we run one single step of the Karcher flow, which when the learning rate is set to 1 amounts to the barycenter from the LEM viewpoint (Section 4.2.1). Conceptually, step 0 of the flow, i.e. the initialization, amounts to the Euclidean barycenter, or arithmetic average. One could question the efficiency of using the Karcher flow, which involves eigenvalue decompositions, in terms of the network's final performance.

The first rebuttal, is that within the SPD networks, the decomposition is already performed, and the results are stored for the backpropagation step. The second counter-argument lies within the following experiment. We execute the Minimum Distance to Mean (MDM) classification scheme, successively using the three possible barycenters (arithmetic mean, LEM, Fréchet with several steps) on the synthetic radar data. Since the three candidates instantiate as iterations through the Karcher flow, we plot in Figure 4.9 a single curve of validation accuracy. Figure 4.9 shows performance increasing with the fidelity to the manifold's geometry, justifying the interest of the Fréchet mean. However, the computational overhead induced from multiple iterations may not be worth the gain in accuracy depending on the application. Furthermore, we found that adding iterations of the Karcher flow in the batch centering scheme actually deteriorates performance. This could be explained by the fact that one single batch may not adequately represent the underlying data distribution, thus a theoretically less accurate batch barycenter may reduce the variance in the reference point computation and ease training.

We now move on to validating the DAMNet and SPDNetBN on a broader set of tasks.

Emotion recognition

In this section we experiment on the Acted Faces Expressions in the Wild (AFEW) dataset [START_REF] Dhall | Static facial expression analysis in tough conditions: Data, evaluation protocol and benchmark[END_REF], which consists of videos depicting 7 classes of emotions; we follow the setup and protocol in Huang and Van Gool (2017). The dataset consists of 2118 SPD matrices of size 400 × 400 split in 1747+371 training and validation sets, each matrix in the dataset summarizing the pixel cross-correlations of a video clip scaled down to frames of size 20 × 20. Since Huang and Van Gool (2017) perform no cross-validation and use a fixed seed for initialization, we also initialize with the same exact weights, as we observed a rather high variance in accuracy for different random initializations. We train 1-, 2-, 3-and 4-layer networks; results are summarized in Table 4.3. In comparison, the MRDRM yields a 20.5% accuracy. We first clarify we do not necessarily seek state-of-the-art in the general sense for the following tasks, but rather in the specific case of the family of SPD-based methods. Our own implementation's performances closely match that those cited in Huang and Van Gool (2017), ensuring a fair comparison. We observe a small yet consistent improvement using barycentric centering of the data. It also seems the improvement is higher the fewer the layers. This could be attributed to the fact the network learnt a more discriminative SPD representation in deeper layers, which then benefits less from centering. We also observe a consistent improvement using the Riemanian BatchNorm, with systematically better scores, although the BarNorm rivals it closely in certain scenarios in the higher end of deepness spectrum. Again, we blame the additional parameters to be learnt, in the ParNorm or BatchNorm This dataset being our largest-scale experiment, we also report the increase in computation time using the Riemannian BatchNorm, specifically for the deepest net: one full training lasted on average 81s for SPDNet, and 88s (+8.6%) for SPDNetBN. Computational complexity inherent to SPD-based methods is counterbalanced by having much fewer parameters to learn than an usual deep net.

Action recognition

In this section we experiment on the Hochschule der Medien 05 (HDM05) [START_REF] Müller | Documentation Mocap Database HDM05[END_REF]) dataset, which consists in MoCap data depicting various actions performed by 5 human actors. The actions are divided in 14 main action classes, which are subdivided in 130 classes. We perform the experiments in the 130-class setting. One data point is a sequence of m body states evolving through time, one body state being a frame of 3-D coordinates of the n joints = 31 joints constituting the body. For each body state, we consider the n = 3n joints = 93-dimensional feature vector concatenating all coordinates; the whole sequence is then described by its centered n × n joint covariance, ie the summed centered covariances of each frame feature vector. The resulting matrix can in theory be SPD i.f.f. n ≤ m. The database contains 2337 sequences. Some of the 130 classes being vastly under- As in Huang and Van Gool (2017), we train a {93, 30} SPDNet on half the data and test on the other half with 10-fold cross-validation; results are shown in Table 4.4. Note that all tested models exhibit noticeable variance depending on the weights initialization and the initial random split of the dataset; the results displayed were obtained by setting a fixed seed of 0 for both. In comparison, the MRDRM yields a 27.3% ± 1.06 accuracy. Again, we validate a better performance using the proposed normalization schemes, the Riemanian BatchNorm outperforming the other models.

Conclusion

This chapter focused on the introduction of novel layers in the context of deep architectures for SPD matrices. Making use of the geometry of the underlying manifold, we presented two main levels of contributions.

We first proposed a Data-Aware Mapping Network (DAMNet), a new architecture with two mutually exclusive layers which improve on the performances of an equivalent SPDNet. Both rely on the generalization of the mathematical background upon which the final Euclidean projection layer is based on, from the Log-Euclidean Metric (LEM) framework to the Affine Invariant Riemannian Metric (AIRM) framework. The first one, the Barycentric Normalization (BarNorm), uses the Riemaniann barycenter to improve the logarithmic mapping, making it computationally tractable by only computing the barycenter on a batch of reduced dimension: incoming SPD matrices are centered to their barycenter, ensuring a better conditioned distribution in the arrival Eucidean space. The second one, the Parametric Normalization (ParNorm), makes the projection dependent on the classification loss. More precisely, a parametric SPD matrix acts as the reference projection point, and thus guides the mapping according to the loss.

The second proposition is a Batch Normalization (BatchNorm) algorithm for SPD NNs, mimicking the original BatchNorm in Euclidean nets. The resulting architecture is dubbed SPDNetBN. The algorithm makes use of the SPD Riemannian manifold's geometric structure, namely the Riemannian barycenter, parallel transport, and manifold-constrained backpropagation through non-linear structured functions on SPD matrices. It is interesting to note the proposed Riemanian BatchNorm is an architectural generalization of the DAMNet; indeed, it conceptually acts as the succession of a BarNorm, then a ParNorm, appended after each layer instead of simply the final one. However, experiments show the parametric normalization scheme sometimes hurts the learning, possibly due to the additional degree of freedom and the inherent complexity of learning on the SPD manifold. A final proposition, for which no satisfactory experimental results were obtained, consists in a 1D convolutional layer based on the BiMap, the idea being to allow a more fine-grained temporal splitting of the signal, and a more expressive SPD neural network.

Results were led on µ-D radar data, and also on video data for emotion recognition, and MoCap data for action recognition. They show that both new architectures improve upon the performances of the baseline SPDNet, on multiple datasets. Although the SPDNetBN usually performs best, the BarNorm presents a more reliable, smooth training curve. Also, neither the BarNorm or the ParNorm seems to systematically prevail over the other, justifying the interest of both for the DAMNet in different application cases. A comparison to a well-used, better known Riemannian learning method, the MRDRM scheme, systematically showcases much better performances for the neural-based methods.

An additional result, obtained exclusively on µ-D radar data, is the better robustness to lack of data obtained with our two architectures, compared to the baseline SPDNet and to the Fully Temporal Convolutional Network (FTCN) proposed in the previous chapter. The overall performances and robustness of our proposed SPDNetBN makes it a suitable candidate in learning scenarios where data is structured, scarce, and where model size is a relevant issue, which is of frequent occurrence in the radar scenery, by and large in other fields as well.

Also on radar data, the practical interest of the Riemannian barycenter was shown, with two different visualisations. A first one simply illustrates the gain in performance in a simple MDM scheme: we see this gain increasing from the Euclidean mean, through the LEM mean, to the AIRM mean, or Fréchet mean. The second visualisation, richer in content, shows the performance of a frozen DAMNet architecture, given different Euclidean mapping reference points. The best plotted result is the Riemanian barycenter, with the identity matrix performing better than random.

All in all, the potential inspirations from this chapter are quite mind-boggling. Indeed, each and every of the developments of neural networks led from 1953 to today in 2020 constitute a source of inspiration for the development of SPD neural models, and in general neural nets on any Riemannian manifold. A main chunk of work in this chapter was inspired by the BatchNorm algorithm, while the same could be done with past novelties as diverse as recurrent networks, entropy regularization, more sophisticated learning schemes... Innovation in Riemannian neural models will also most logically stem from the field of Information Geometry itself, for instance to allow the classification to be done directly on the manifold through "separating hyper-geodesics" instead of requiring the projection to a Euclidean plane. Other tools in statistics, linear algebra and general mathematics can also be involved in the refinement or speedup of such models: for instance, the logarithmic mapping could be approximated or replaced (under certain conditions) by a matrix square root, which is approximable through an iterative algorithm requiring inly linear operations, making it easily parallelizable. In the end, we believe the research in Riemannian neural models, not only shows great potential in yet untested fields, but also shows evident fields of exploration for future works.

C h a p t e r

C O N C L U S I O N A N D P E R S P E C T I V E S

We first provide a final conclusion to this thesis, before discussing opening perspectives for future ideas.

Final conclusion

In this thesis, we worked on developing neural models inherently adapted to the data, in the context of structured time series classification, with a more thorough spotlight on micro-Doppler (µ-D) radar data.

The primary goal of this thesis was finding learning models, with the internal structure naturally suited to the input data. As a part of this goal, the understanding of the input data itself is of prime importance, and in particular the various representations used to extract meaningful sense out of it. Two main classes of representations were considered in addition to the raw time series: a time-frequency spectrogram, i.e. some form of Fourier transform, and a covariance matrix.

The first form of representation, along with raw time series, was mainly the concern of Chapter 3. Since the spectrogram can be seen as both a time series of spectral vectors, or an image of time and frequency dimensions, a first benchmark was established between the use of Recurrent Neural Networks (RNNs) on the one hand, and Convolutional Neural Networks (CNNs) on the other hand, or more precisely a Fully Temporal Convolutional Network (FTCN). While the convolutional architecture prevailed, it was interetsing to notice a strong upper hand of both models w.r.t. a standard logistic regression, justifying the incorporation of signal knowledge within the learning models. Other features of interest of the FTCN are, it preserves the temporal structure of the input data, and the resolution of the output feature time series is also fully controlable. Furthermore, a complex-valued version of the FTCN was devised; we also introduced a novel Fourier convolution layer, allowing for a flexible refining of the Fourier filter-bank; the two complex architectures proposed were thus the CR Neural Network (CRNet) and the Fourier Neural Network (FourierNet), one operating on the complex spectrogram, the other on the raw complex time series. Subsequent experiments on µ-D radar data further showed the practical interest of using the complex architectures -mainly in the context of plentiful, and challenging data.

The second form of representation, the covariance, was mainly the concern of Chapter 4. The covariance matrix posseses a particular geometry, that of being Symmetric Positive Definite (SPD). As such, it does not belong to a flat, Euclidean space but rather to a curved, Riemannian manifold; learning models must thus be suited to this underlying manifold. We mainly focused on the Riemannian version of Deep Neural Networks (DNNs), a founding instance of which we base ourselves on, the SPD neural network (SPDNet). In this context, we developed two series of contributions. A first one, the Data-Aware Mapping Network (DAMNet) architecture, dwelled upon the final Euclidean projection layer, generalizing it with two distinct layers, the Barycentric Normalization (BarNorm) making use of the Riemannian barycenter, and the Parametric Normalization (ParNorm) making use of a parametric, learnable SPD matrix. Then, these normalization layers were pushed to their natural evolution in the Batch-Normalized SPDNet (SPDNetBN), in which we introduced a Riemannian equivalent to the well-established Batch Normalization (BatchNorm) layer. Experimentations were conducted on radar data, but also on video data for emotion recognition, and Motion Capture (MoCap) data for action recognition. They showed rather systematic improvements from the SPDNet, through the DAMNet, to the SPDNetBN, although it seems the DAMNet with BarNorm regularization eased the learning in some cases.

Given these different learning models associated to different input representations, it may be tough to chose between one or the other in a given scenario. The second purpose of Chapter 3 was also to provide a congregating solution, by making use of all representations and models in a single pipeline, called Second-Order Fully Temporal Network (SOFTNet). This final new model consists in plugging a SPD neural net to a FTCN, which is made possible by the FTCN feature maps being of temporal nature. This amounts to building a first-order representation of the data through a DNN, and then appending a second-order neural model to study the covariance of the feature time series, which makes for a conceptually strong model, making use of a variety of representations and adapted learning models of the data. This pipeline makes for a fine transition to perspectives on future works.

Opening perspectives

As parting words, let us now jot down a few thoughts brought to light in the midst of this thesis.

As hinted above, the SOFTNet pipeline provides a good starting point: from what one may gather for its figurative aspect, its modularity potentially allows a combinatorial expanse of re-configured models suited to the task at hand: to our mind, the interaction of first, second and even higher statistical orders of the data within adapted Euclidean or Riemannian models shows the strongest promises for future research potential. This is already the case for second-order models, which simply plug a SPD neural net after a CNN to improve the performance of the CNN in its classification task. The independent blocks of Euclidean and Riemannian models may advance seperately in their journey towards more efficient and powerful learning abilities, but their union may prove the ultimate provider of cutting-edge performance.

These independent evolutions are also a cornerstone subject of interest, specifically in our case developments in SPD architectures, not only on the technical advances, but also in domain applications. The use of Parallel Transport (PT) for domain adaptation is starting to be explored, with potential applications as diverse as there are domains to adapt; for instance, the self-driving car problem of having to perform equally well in very different landscapes or weather conditions. The usage of local covariance descriptors for image segmentation is also a incipient possibility, paving the way to possible innovations such as pixelcovariance-RNN, or more simply an application of a convolutional SPDNet.

We end this thesis by reenacting our original guiding principle, and perhaps the most obvious perspective of our works: whenever any kind of local structure defines the data to be studied, a covariance structure may be extracted. Then, SPDbased methods may be applied, in possible collaboration with a deep Euclidean model. For it is together, not isolated one from the other, that their individual qualities may be expressed in emergence towards a dominant performance. import t o r c h . nn as nn import t o r c h s p d n e t . nn as nn_spd model=nn . S e q u e n t i a l ( nn_spd . BiMap ( 1 , 1 , 2 0 , 1 5 ) , nn_spd . ReEig ( ) , nn_spd . BiMap ( 1 , 1 , 1 5 , 1 0 ) , nn_spd . LogEig ( ) , nn_spd . V e c t o r i z e ( ) , nn . L i n e a r ( 1 0 * * 2 , 3 ) )

Note that our implementation of the BiMap module supports an arbitrary number of channels, represented by the additional parameters all set to 1 in this example.

A.3.0.2 First-order and second-order combined In a more complex example, an SPDNet acts upon the features maps of a convolutional network. For an image recognition task, these features may come from a pre-trained deep network but nothing keeps from training the whole network in an end-to-end fashion or to fine-tune the parameters. Here we describe the combination of a pre-trained ResNet-18 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]) on the CIFAR10 (Krizhevsky n.d.) challenge and of SPDNet layers. We recall we call such a model a SOFTNet.

Organization of the provided PyTorch library

The library is organized following the PyTorch module and functional logic:

1. spd/: building blocks for the SPDNetBN architecture and training:

functional.py: core machinery for inference and backpropagation in a SPDNetBN;

nn.py: modules used to build a SPDNetBN architecture;

2. cplx/: building blocks for the complex data handling (only useful for radar data); functional.py: core machinery for inference and backpropagation in a complex neural net; nn.py: modules used to build a complex net architecture;

A.5 Implementation details

Finally, we list some details of interest to reproduce the results:

1. Training did not benefit from GPU acceleration, seemingly bottlenecked at eigenvalue operations: computation time desribed in the paper is not decreased using an Nvidia GTX 1070M;

2. Symmetric matrix vectorization: following the Euclidean mapping, it is possible to vectorize only the upper triangular part of the matrix (and normalize the outer-diagonal coefficients by √ 2 to conserve the norm). We conducted all experiments in both settings and observed no significant impact, except for the parametric centering, which seemed to suffer from a smaller representation dimension. The reported results use full matrix vectorization;

1 2 3 4 5 1 2 3 4

 123451234 and properties of the proposed learni n g models a. Model sizes and speeds . . . . . . . . . . . . . . . . . . . . . . . . . a. Sample code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a. Use cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . a. Organization of the provided PyTorch library . . . . . . . . . . . . . a. Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . b d e tailed description of the nato radar database b. Global overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . b. Description of the data . . . . . . . . . . . . . . . . . . . . . . . . . . b. Contents of the disk . . . . . . . . . . . . . . . . . . . . . . . . . . . . b. Possible setting for learning . . . . . . . . . . . . . . . . . . . . . . .

Figure 2 . 21 Figure 3 . 1

 22131 Figure 2.21 Manifold mappings . . . . . . . . . . . . . . . . . . . . . . . Figure 2.22 Illustration of one iteration of the Karcher flow (Karcher 1977) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . C h a p t e r 3: s e c o n do r d e r p i p e l i n e f o r t e m p o r a l classification Figure 3.1 Illustration of the Bilinear Mapping (BiMap) layer . . . . . . Figure 3.2 Illustration of a generic Symmetric Positive Definite (SPD) neural network . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.3 Downsampling of a spectro-temporal signal . . . . . . . . . Figure 3.4The proposed architecture for radar signal classification . . Figure 3.5 Illustration of how strided filters of size k and stride s work on an input of size n . . . . . . . . . . . . . . . . . . . . . . Figure 3.6 The proposed complex-valued architecture for radar signal classification . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.7 Illustration of the proposed Fourier-like convolutional layer Figure 3.8 Illustration of the global pipeline proposed for the classification of time-frequency signals . . . . . . . . . . . . . . . . Figure 3.9 Illustration of the Second-Order Fully Temporal Network (SOFTNet) . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.10 Structured time series representations . . . . . . . . . . . . Figure 3.11 Evolution of the scattering points and normals for the Vario, Phantom2 and S1000+ drones . . . . . . . . . . . . . . . . . Figure 3.12 Spectrograms of noisy and uncluttered signals . . . . . . . Figure 3.13 Classification accuracies for three learning models . . . . . Figure 3.14 Performance of the learning models for increasingly good conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.15 Growth of classification confidence over time in a very challenging environment (SN R = 10dB and P RF = 2kHZ) . . Figure 3.16 Growth of classification confidence of a Fully Temporal Convolutional Network (FTCN) in audio recognition . . .

Figure 3 . 5

 35 Figure 3.1 Illustration of the Bilinear Mapping (BiMap) layer . . . . . . Figure 3.2 Illustration of a generic Symmetric Positive Definite (SPD) neural network . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.3 Downsampling of a spectro-temporal signal . . . . . . . . . Figure 3.4 The proposed architecture for radar signal classification . . Figure 3.5 Illustration of how strided filters of size k and stride s work on an input of size n . . . . . . . . . . . . . . . . . . . . . . Figure 3.6 The proposed complex-valued architecture for radar signal classification . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.7 Illustration of the proposed Fourier-like convolutional layer Figure 3.8 Illustration of the global pipeline proposed for the classification of time-frequency signals . . . . . . . . . . . . . . . . Figure 3.9 Illustration of the Second-Order Fully Temporal Network (SOFTNet) . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.10 Structured time series representations . . . . . . . . . . . . Figure 3.11 Evolution of the scattering points and normals for the Vario, Phantom2 and S1000+ drones . . . . . . . . . . . . . . . . . Figure 3.12 Spectrograms of noisy and uncluttered signals . . . . . . . Figure 3.13 Classification accuracies for three learning models . . . . . Figure 3.14 Performance of the learning models for increasingly good conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 3.15 Growth of classification confidence over time in a very challenging environment (SN R = 10dB and P RF = 2kHZ) . . Figure 3.16 Growth of classification confidence of a Fully Temporal Convolutional Network (FTCN) in audio recognition . . .

Figure 4 . 1

 41 Figure 4.1 Log Eigenvalues (LogEig) as a special case of logarithmic mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.2 Illustration of the Data-Aware Mapping Network (DAMNet) architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.3 Illustration of manifold-constrained gradient update . . . . Figure 4.4 The convolutional BiMap . . . . . . . . . . . . . . . . . . . . Figure 4.5 Validation accuracy and loss curves of SPD neural network (SPDNet) and Batch-Normalized SPDNet (SPDNetBN) on the North Atlantic Treaty Organization (NATO) dataset . .

Figure 4 . 2

 42 Figure 4.1 Log Eigenvalues (LogEig) as a special case of logarithmic mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.2 Illustration of the Data-Aware Mapping Network (DAMNet) architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.3 Illustration of manifold-constrained gradient update . . . . Figure 4.4 The convolutional BiMap . . . . . . . . . . . . . . . . . . . . Figure 4.5 Validation accuracy and loss curves of SPD neural network (SPDNet) and Batch-Normalized SPDNet (SPDNetBN) on the North Atlantic Treaty Organization (NATO) dataset . .

Figure 4 . 3

 43 Figure 4.1 Log Eigenvalues (LogEig) as a special case of logarithmic mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.2 Illustration of the Data-Aware Mapping Network (DAMNet) architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.3 Illustration of manifold-constrained gradient update . . . . Figure 4.4 The convolutional BiMap . . . . . . . . . . . . . . . . . . . . Figure 4.5 Validation accuracy and loss curves of SPD neural network (SPDNet) and Batch-Normalized SPDNet (SPDNetBN) on the North Atlantic Treaty Organization (NATO) dataset . .

Figure 4 . 4

 44 Figure 4.1 Log Eigenvalues (LogEig) as a special case of logarithmic mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.2 Illustration of the Data-Aware Mapping Network (DAMNet) architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 4.3 Illustration of manifold-constrained gradient update . . . . Figure 4.4 The convolutional BiMap . . . . . . . . . . . . . . . . . . . . Figure 4.5 Validation accuracy and loss curves of SPD neural network (SPDNet) and Batch-Normalized SPDNet (SPDNetBN) on the North Atlantic Treaty Organization (NATO) dataset . .

Figure 4 . 6 123 Figure 4 . 7 124 Figure 4 . 8 126 Figure 4 . 9

 46123471244812649 Figure 4.6 Confusion matrix on the NATO dataset . . . . . . . . . . . . 123 Figure 4.7 Performance of all models in function of the amount of synthetic radar data . . . . . . . . . . . . . . . . . . . . . . . 124 Figure 4.8 Separability of two classes of the radar dataset. Separation between classes is higher with a reference matrix near the barycenter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Figure 4.9 Validation accuracy for the closest barycenter algorithm. Iteration 0 corresponds to the arithmetic mean, iteration 1 to the LEM barycenter. . . . . . . . . . . . . . . . . . . . . . 127 Figure 4.10 Distribution of class instance number and sequence length for the Hochschule der Medien 05 (HDM05) dataset . . . . . 129Appendix A: alorithmic details and properties of the proposed learning models 157

Figure 1 . 1 .:

 11 Figure 1.1.: Example of a drone's micro-Doppler signal. This time-frequency representation exhibits rich structure; much information about the target is inferable from Digital Signal Processing tools.

Figure 1 . 2 .:

 12 Figure 1.2.: The race to ever bigger Deep Neural Networks. Image source: Deep Learning in Practice (2019).

Figure 1 . 4

 14 Figure 1.3.: Spectrogram versus Capon sepctrogram. The Capon spectrum is a variant of the Wigner-Ville spectrum, making use of parametric constraints on the spectrum to enhance its time-frequency resolution.

Figure 1 . 4

 14 gives the intuition of building different learning models upon different representations of the data.

Figure 1 . 4 .:

 14 Figure 1.4.: Raw signal and its Fourier and covariance representations The raw signal is split through a sliding window, from which either covariance or Fourier transform can be computed.

Figure 1 . 5 .:

 15 Figure 1.5.: Example of image deformation through scale, rotation, translation and illumination. Ideally, a learning model should be insensitive to the deformation to infer in both cases the same image class: cat.

Figure 2 . 1 .:

 21 Figure 2.1.: The basic form of a standard signal emitted by a radar. The frequency typically grows linearly in time. Imposing a known frequency evolution helps solve ambiguities in the received signal.

  .2) will require a more powerful radar, leading to further compromises. The curious reader may dig in detailed explanations in excellent references such as V. C. Chen et al. (2006).

Figure 2 . 2 .:

 22 Figure 2.2.: Illutration of the RCS. Expressed in m 2 , it represents the spatial distribution of reflectance across an object: as such, it is intrinsic to the object, and corresponds to a certain extent to its global shape size from the radar's point of view. This particular figure shows the RCS along a single rotation axis. Much research has been devoted to minimizing it to manufacture stealth jets.

Figure 2 . 3

 23 shows three different drones studied in initial works: The Vario helicopter and DJI's Phantom2 and S1000+. Many UAVs such as those featured in the figure share specific characteristics: • Blade rotation speed Ω = RP M 60 expressed in rad.s -1 (Rounds per Minutes (RPMs) being rotations per minute); • RCS; • Number N b and length L b of blades;

Figure 2 . 3 .:

 23 Figure 2.3.: Depiction of the three drones used in simulations. From left to right:The Vario helicopter, DJI's Phantom2, a quadrocopter, and S1000+, an octocopter. Scale is not respected.

Figure 2 .

 2 Figure 2.4.: Illustration of the simulated signal. Here, a PRF of 8kHz on a signal of 250ms yields a time series of 2000 complex points, jointly drawn as modulus and real part.

Figure 2 .

 2 Figure 2.5.: Illustration of the Fourier spectrum of a radar signal. From a simple analysis of the featured spectrum, we can deduce three important characteristics of the UAV: rotation speed Ω, number N b and length L b of blades.

Figure 2 . 6 .

 26 Figure 2.6.: MSE versus LL. Inputs to the functions are supposed normalized to (0, 1); the plot shows the cost of labelling input x with true class y = 1.

Figure 2 .

 2 Figure 2.7.: Activation functions Along with the sigmoid used in logistic regression, two other popular choices of so-called "activations" are depicted: the hyperbolic tangent and the ReLU.

Figure 2 . 8 .:

 28 Figure 2.8.: Illustration of the kernel trick. Input data usually has no particularreason to be linearly separable in its raw form. A good representation space aims to that goal through the mapping K, from where we can then separate and classify the data.

Figure 2 . 9 .:

 29 Figure 2.9.: Representation power of DNNs through hierarchical learning. Image source: Lawrence et al. (1997).

Figure 2 .

 2 Figure 2.10.: Illustration of the convolution as a linear transformation. The weights in a CNN are shared and locally connected throughout the layers, contrary to the fully-connected DNN.

Figure 2 .Figure 2 .

 22 Figure 2.11.: LeNet, the historical first CNN.

Figure 2 .

 2 Figure 2.12.: AlexNet, the 2012 ILSVRC winner.

Figure 2 .

 2 Figure 2.14.: V GG neural network, second place in the 2014 ILSVRC challenge.

Figure 2 .

 2 Figure 2.15.: The convolutionalization trick. The trick consists in replacing dense layers by convolutions of size 1, which maintains end-to-end spatial structure. The final convolutional layer can be seen as feature maps, or downsampled versions of the input image containing semantic information, i.e. spatial labels.

Figure 2 .

 2 Figure 2.17.: Illustration of the fundamental representation equivalence between a 2D time-frequency image and a series of 1D spectrums..

  shed light on, later supported by, notably, Pascanu et al. (2013). More sophisticated forms of recurrent networks have since been introduced, the most established at the time being Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber 1997) (see Figure 2.18 for a comparison of LSTMs with vanilla RNNs) and Gated Recurrent Unit (GRU) networks (Cho et al. 2014; B. Xu et al. 2015), along with blends with CNNs (Y. Xu et al. 2017). These architectures are widely spread in Natural Language Processing (NLP) and time series analysis and prediction (Sutskever et al. 2011). Further developments led to the now praised attention mechanisms in neural nets (Luong et al. 2015; Vaswani et al. 2017). One interesting theoretical justification of LSTM's architecture is given in Tallec and Ollivier (n.d.), where the authors build upon vanilla RNNs by adding invariance to slight time warpings, and end up with an architecture similar to the LSTM.

Figure 2 .

 2 Figure 2.18.: Illustration of a Long Short-Term Memory network. LSTMs include a memory stream, which allows for unobtrused temporal data flow, along with a forget gate which leaves aside unnecessary information. Image credits to Colah's excellent blog article on RNNs: Understanding LSTM Networks -colah's blog (2017).

Figure 2 .

 2 Figure 2.19.: The sphere: a simple manifold. In this figure, the cube is the Euclidean space R 3 .The data however intrinsically lie on the colored sphere, which is a manifold, i.e. can be locally approximated to its tangent plane.

80 D 1 Figure 2 .

 8012 Figure 2.20.: Kullback-Leibler divergence between two univariate Gaussians.We fix a first distribution at (µ 0 , σ 0 ) and plot the KL divergence from the second to the first. We see it annuls i.f.f. (µ 0 , σ 0 ) = (µ 1 , σ 1 ).

  al. 2019), the Bregman divergence (Boissonnat et al. 2010; Siahkamari et al. 2019; Banerjee et al. 2005), and optimal transport (Arjovsky et al. 2017).

Figure 2 .

 2 Figure 2.21.: Manifold mappings. This figure illustrates the mappings between the manifold M and the tangent space T G in G. The bijection from M to T G is called logarithmic mapping, and its reverse exponential mapping.

Algorithm 2 . 1

 21 Riemannian k-NN algorithm on SPD matrices. Require: Dataset D := {P i , y i } i∈[1,N ] , new data point to classify P 1: L ← [0, 0] N // List to store distances to and labels of neighbours 2: for i ∈ [1, N ] do 3:

Figure 3 . 1 .:

 31 Figure 3.1.: Illustration of the BiMap layer The input manifold is transformed to a lower-dimensional, more discriminative one.

  , Ledoit and Wolf (2012), Y. Chen et al. (2011), Tiomoko et al. (2019a), Tiomoko et al. (2019b), Cao and Bouman (2009), Cao et al. (2011), and Mezzadri (2006). Log Eigenvalues layer The first stage of a SPD neural network consists in a stack of manifold-to-manifold BiMap+ReEig layers. Once we reach a satisfying feature manifold, we aim to perform classification on the corresponding matrix P . To do so, Huang and Van Gool (2017) propose to map P to a Euclidean space, where it is possible to resort to classical Machine Learning (ML). They do so by harnessing the work of Arsigny et al. (2006), in which the authors endow the SPD manifold with a Lie group structure, for which the associated Riemannian distance computed from infinitesimal integration across the associated Lie algebra, the Log-Euclidean Metric (LEM), first introduced in Pennec et al. (2006) and Harris

Figure 3 . 2 .:

 32 Figure 3.2.: Illustration of a generic SPD neural network Successive bilinear layers followed by activations build a feature SPD manifold, which is then transformed to a Euclidean space to allow for classification.

Figure 3 . 3 .:

 33 Figure 3.3.: Downsampling of a spectro-temporal signal. Here we choose the same input representation, e.g. a 512 × 1024 time-frequency spectrogram.

Figure 3

 3 Figure 3.4.: The proposed architecture for radar signal classification. The boxes represent the successive filter banks. Horizontal is time, vertical is frequency. Contrary to CV, where the feature maps are spatial, our feature maps extend solely in time; the frequency domain is flattened through the convolutions.

Figure 3 . 5 .:

 35 Figure 3.5.: Illustration of how strided filters of size k and stride s work on an input of size n. Image source: Deshpande (2017).

Figure 3 . 6 .:

 36 Figure 3.6.: The proposed complex-valued architecture for radar signal classification. All layers are complex, except for the final ones. The decibel function transitions between the two domains.

Figure 3 . 7 .:

 37 Figure 3.7.: Illustration of the proposed Fourier-like convolutional layer. The Fourier atoms are represented as sine waves of increasing frequency.

Figure 3 . 8 .:

 38 Figure 3.8.: Illustration of the global pipeline proposed for the classification of time-frequency signals. Possible bifurcations to covariance analysis stem either from the raw data, the spectral representation or the learnt temporal features. The first convolutional layer can be either learnt fom the data and the Fourier atoms or used as a Fourier transform.

Figure 3

 3 Figure 3.9.: Illustration of the SOFTNet. This model makes use of all cited representations for structured time series data: time, frequency and covariance.

Figure 3 .

 3 Figure 3.10.: Structured time series representations. The input raw signal is split in sub-signals of elementary duration τ . From there, either Fourier transform or covariance pooling then operate at the same time scale, resulting in output spectrograms and covariance matrices representing the same object, and fit to be compared against in further processings: FTCN, SPDNet, and eventually SOFTNet.

Figure 3 .

 3 Figure 3.11.: Evolution of the scattering points and normals for the Vario, Phan-tom2 and S1000+ drones. The total simulation time is 250ms, of which we only illustrate the first 3ms (corresponding to 7 frames) for the sake of visual simplicity. The body is modelled as an isotropic reflector whereas blades are scattered with directional punctual reflectors in the direction of the normals, the length of which are proportional to the points' reflectance. Notice the multicopters' helices alternatively turn clockwise and counter-clockwise. The 2D scale is in meters; dimensions and distances are accurately scaled w.r.t. reality. Time goes from red to green.

Figure 3 .

 3 Figure 3.12.: Spectrograms of noisy and uncluttered signals. These are sampled from the three drones mentioned in Figure 2.3, from left to right: the Vario, the Phantom2 and the S1000+. Each group is organized as follows: four varying spectrogams of the same class are displayed; the top-left one always corresponds to a version constant in time. Below we plot an arbitrary time cut of one of the varying spectrograms.

  Chollet 2015) library with Tensorflow (Abadi et al. 2016) backend and PyTorch (Paszke et al. 2017) deep learning framework, and trained on a single Nvidia GTX 1070M Graphical Processing Unit (GPU). Training time for any of the above lasts less than an hour.

Figure 3 .

 3 Figure 3.13.: Classification accuracies for three learning models. Here we chose a setup of cluttered, noisy signals with SN R = 30dB on the left and SN R = 10dB to the right (P RF = 4kHz), which is more than reasonable from a practicioner's point of view.

Figure 3 .

 3 Figure 3.14.: Performance of the learning models for increasingly good conditions. The graphs show intuitive behavior w.r.t. the configurations, i.e. a general increase.

Figure 3 .

 3 Figure 3.15.: Growth of classification confidence over time in a very challenging environment (SN R = 10dB and P RF = 2kHZ). The result of interest here is how a lower finesse coefficient, i.e. higher temporal resolution, provides finer and faster confidence during measurements with no additional cost in global accuracy.

Figure 3 .

 3 Figure 3.16.: Growth of classification confidence of a FTCN in audio recognition. The graph plots the confidence in function of the duration of input data. Each line groups signals of same total length; the thickness of the line represents the proportion of such signals within the dataset.

5 . 3 )

 53 to successfully classify covariances matrices computed on multivariate EEG signals; in the same field, Yger and Sugiyama (2015) propose kernel methods for metric learning on the SPD manifold . Another example is in MRI, where Pennec et al. (2006) and Arsigny et al. (2006) develop a Riemannian knearest neighbours (k-NN) as described in Section 2.5.1. Motion recognition from Motion Capture (MoCap) skeletal data also benefits from Riemannian geometry, as exposed in Cavazza et al. (2017), Huang et al. (2018), and Huang et al. (2016).

  et al. (2017) and Gao et al. (2017), a more lightweight transformation layer is proposed; in T. Zhang et al. (2018) and Chakraborty et al. (2018), the authors propose alternate convolutional layers, respectively based on multi-channel SPD representations and Riemannian means; a recurrent model is further proposed in Chakraborty et al. (2018); in Mao et al. (2019) and P. Li et al. (2018), an approximate matrix square-root layer replaces the final Euclidean projection to lighten computational complexity. Transversally, works on accelerating the optimization of SPD layers are emerging, in answer to the frustratingly slow pace the optimization usually takes place at (H. Zhang et al. 2016; Liu et al. 2017; Alimisis et al. 2020). In Acharya et al. (2018) and Yu and Salzmann (2017), a SPDNet is appended to a Euclidean Convolutional Neural Network (CNN) to improve on performance. In this case, a so-called covariance pooling (CovPool) layer is devoted to compute a Sample Covariance Matrix (SCM) of the final feature maps.

Figure 4 .

 4 Figure 4.1.: LogEig as a special case of logarithmic mapping. While the LogEig is defined within the LEM framework, its generalization to the AIRM framework allows for a more flexible, powerful data-aware mapping.The main idea to retain is the passage from the manifold to a certain Euclidean space.

  (k) i at the final SPD layer (k) in the batch around their geometric barycenter G {P (k) i } i≤N , as done in Yger (2013) and Barachant et al. (2013), which amounts to omitting the final congruence in the logarithmic map in Equation 2.30. More details on centering with parallel transport are given in the next section. At each iteration, we compute the barycenter G {P (k) i } i≤N of the current batch of transformed data points {P i } i≤N . All in all, this BarNorm layer becomes:

Figure 4 .

 4 Figure 4.2.: Illustration of the DAMNet architecture. When the reference point of the Euclidean mapping is the identity matrix, it reduces to the SPDNet architecture.

Lemma 4 . 1 .

 41 ∀P 1 , P 2 ∈ S + * , A := (P 2 P Proof. A and B trivially are similar matrices, as in they share the same eigenvalues.

1 ) X T P 1 2 2( 2

 22 SPD centering and biasing The definition of the SPD transport Γ clarifies the BarNorm and ParNorm operators, and allows the formal definition of • The batch centering of matrices {P i } i≤N with Riemannian barycenter G as the PT from G to the identity I d ; • The biasing of the batch towards a parametric SPD matrix G as the PT from I d to G.

. 30 ,Figure 4 . 3 .:

 3043 Figure 4.3.: Illustration of manifold-constrained gradient update. The Euclideangradient is projected to the tangent space, then mapped to the manifold through retraction, which can be the exponential mapping.

gLemma 4 . 3 .

 43 • Eig : P -→ (U, Σ, U T ) -→ U g(Σ)U T =: X (4.13) Equation 4.13 maps P to its eigenspace U and eigenvalues Σ, then to the nonlinear (monotonous) modification of the eigenvalues. By construction, U ∈ O (where we reuse the notation O for orthogonal matrices), and Σ is diagonal. We decompose the derivation of ∂L•g•Eig ∂P through ∂L••Eig ∂P and ∂L•g ∂(U,Σ) We begin with Lemma 4.3. The tangent plane T U O of U ∈ O is U × S † (where S † is the space of skew-symmetric matrices). In other words, dU = U Ω with Ω T = -Ω.

( 4 : 4 . 4 .

 444 noting Ω := U T Z) ⇒ dU = U Ω In practice, making use of Theorem 4.2 amounts for the Eig function to finding dΣ and dU in function of dP , given the differential F form of dP , which we do in Lemma 4.Lemma The function Eig : P -→ (U, Σ, U T ) | P = U ΣU T admits the following differential form:      dΣ = (U T dP U ) diag (using the operation (•) diag which zeroes all off-diagonal elements)

Theorem 4 . 3 .

 43 The function Eig : P -→ (U, Σ, U T ) | P = U ΣU T admits the following backward pass:

Theorem 4 . 4 .

 44 The function Eig : P -→ (U, Σ, U T ) | P = U ΣU T admits the following backward pass:

1 2 .

 2 We credit Engin et al. (2018) for first showing the equivalence between the two cited formulations, of which we expose the most concise.

Figure 4 . 4

 44 Figure 4.4 illustrates the somewhat convoluted process of performing the SPD BiMap convolution.

Figure 4 . 4 .:

 44 Figure 4.4.: The convolutional BiMap. The input raw signal is split in T i subsignals corresponding to a first level of temporal resolution. Each sub-signal is then split in sub-sub-signals of elementary duration τ (from Section 3.2.2). From there, a covariance matrix is sampled for each timestep t ≤ T i , yielding a time series of covariances, which is convolved with the temporal BiMap filter bank. The output duration T o follows the standard rule of convolutions, here with no stride and padding.

Figure 4 . 5 .:

 45 Figure 4.5.: Validation accuracy and loss curves of SPDNet and SPDNetBN on the NATO dataset. The x-axis measures hours instead of epochs for the accuracy, epochs for the loss.

Figure 4 .

 4 Figure 4.6.: Confusion matrix on the NATO dataset. Class imbalance within the dataset yields rather strongly imbalanced results.

2 Figure 4

 24 Figure 4.7.: Performance of all models in function of the amount of synthetic radar data. The SPDNetBN model outperforms the other ones and continues to work even with a little fraction of the train data.

Figure 4 . 8 .:

 48 Figure 4.8.: Separability of two classes of the radar dataset. Separation between classes is higher with a reference matrix near the barycenter.

Figure 4 . 9 .:

 49 Figure 4.9.: Validation accuracy for the closest barycenter algorithm. Iteration 0 corresponds to the arithmetic mean, iteration 1 to the LEM barycenter.

  represented, we select those with at least 5 sequences yielding SPD descriptors (as done in Harandi et al. (2014)), which trims the dataset down to 2086 points scattered throughout 117 classes.

Figure 4 .

 4 10 plots the distribution of sequence lengths across the retained classes (minimum, mean and maximum) along with the instances per class. Note that by construction the minimal sequence length possible is 93 and the smallest number of instances is 5.

Figure 4 .

 4 Figure 4.10.: Distribution of class instance number and sequence length for the HDM05 dataset. Visibly, there are less instances per class than there are classes, and some sequences are too short to yield positive definite covariance.

  import t o r c h . nn as nn import t o r c h s p d n e t . nn as nn_spd from r e s n e t import ResNet18 c l a s s SOCNN( nn . Module ) : def _ _ i n i t _ _ ( s e l f ) : super ( _ _ c l a s s _ _ , s e l f ) . _ _ i n i t _ _ ( ) # f i r s t -o r d e r m o d e l s e l f . model_fo=ResNet18 ( ) s t a t e _ d i c t =th . load ( ' p r e t r a i n e d /ResNet18 . pth ' ) [ ' s t a t e _ d i c t ' ] s e l f . model_fo . l o a d _ s t a t e _ d i c t ( s t a t e _ d i c t ) # c o n v o l u t i o n a l c o n n e c t i o n s e l f . c o n n e c t i o n =nn . Conv2d ( 5 1 2 , 2 5 6 , k e r n e l _ s i z e = ( 1 , 1 ) ) # s e c o n d -o r d e r m o d e l s e l f . model_so=nn . S e q u e n t i a l ( nn_spd . BiMap ( 1 , 1 , 2 5 6 , 1 2 8 ) , nn_spd . ReEig ( ) , nn_spd . BiMap ( 1 , 1 , 1 2 8 , 6 4 ) , ) . t o ( s e l f . d e v i c e _ s o ) s e l f . dense=nn . S e q u e n t i a l ( nn . L i n e a r ( 6 4 * * 2 , 1 0 2 4 ) , nn . L i n e a r ( 1 0 2 4 , 1 0 ) ) def forward ( s e l f , x ) : x _ f o= s e l f . model_fo ( x ) x_co= s e l f . c o n n e c t i o n ( x _ f o ) x_sym=nn_spd . CovPool ( ) ( x_co . view ( x_co . shape [ 0 ] , x_co . shape [ 1 ] , -1 ) ) x_so= s e l f . model_so ( x_sym ) x_vec=nn_spd . LogEig ( ) ( x_so ) . view ( x_so . shape [ 0 ] , x_so . shape [ -1 ] * * 2 ) y= s e l f . dense ( x_vec ) r e t u r n y A.4

3 .

 3 experiments/: radar.py: code for lauching experiments on the synthetic radar dataset (results on the NATO dataset are not reproducible beacause of data confidentiality issues); hdm05.py: code for lauching experiments on the Hochschule der Medien 05 (HDM05) dataset; afew.py: code for lauching experiments on the Acted Faces Expressions in the Wild (AFEW) dataset; radar_mrdrm.py: code for lauching minimum Riemannian distance to Riemannian mean (Minimum Riemannian Distance to Riemannian Mean (MRDRM)) experiment on the synthetic radar dataset (results on the real dataset are not reproducible beacause of data confidentiality issues); hdm05_mrdrm.py: code for lauching MRDRM experiment on the HDM05 dataset; afew_mrdrm.py: code for lauching MRDRM experiment on the AFEW dataset; data/: the folder to copy the data to; the others are made available for download 1 . Radar, HDM05 and AFEW datasets respectively weigh 47Mb, 139Mb and 1.3Gb. To launch the code in the experiments/ directory, execute for instance python radar.py. The code was developed in Python3 and tested under PyTorch (Paszke et al. 2017) v0.4.1 on CUDA version 8.0.61 and run on a laptop i7-6700HQ CPU.

  Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Motivations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Contributions and outline . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Data-Aware Mapping Network . . . . . . . . . . . . . . . . . . . . . 100 4.3 Batch-Normalized SPDNet . . . . . . . . . . . . . . . . . . . . . . . . 104 4.4 Riemannian manifold-constrained optimization . . . . . . . . . . . 108 4.5 Convolution for covariance time series . . . . . . . . . . . . . . . . . 116 4.6 Experimental validation . . . . . . . . . . . . . . . . . . . . . . . . . 119 4.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	a b s t r act	i
	r é s u m é	iii
	r e m e rciements	v
	c o n t e nts	vii
	l i s t o f figures	ix
	l i s t o f tables	xiii
	ac ro n yms	xv
	1 i n t roduction	1
	1.1 4 a d vances in spd neural networks
	4.1

2 t h e oretical background 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Radar Signal and Simulation . . . . . . . . . . . . . . . . . . . . . . 2.3 Euclidean Machine Learning . . . . . . . . . . . . . . . . . . . . . . 2.4 Information Geometry . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Riemannian Machine Learning . . . . . . . . . . . . . . . . . . . . . 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 s e c ond-order pipeline for temporal classification 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Learning on structured time series representations . . . . . . . . . . 3.3 Full pipeline for temporal classification . . . . . . . . . . . . . . . . 3.4 Experimental validation . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Figure 1.1 Example of a drone's micro-Doppler signal. . . . . . . . . . Figure 1.2 The race to ever bigger Deep Neural Networks. . . . . . . . Figure 1.3 Spectrogram versus Capon sepctrogram . . . . . . . . . . . Figure 1.4 Raw signal and its Fourier and covariance representations Figure 1.5 Example of image deformation through scale, rotation, translation and illumination. . . . . . . . . . . . . . . . . . . . . . Figure 2.14 V GG neural network, second place in the 2014 ILSVRC challenge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.15 The convolutionalization trick . . . . . . . . . . . . . . . . . Figure 2.16 Recurrent Neural Network (RNN) with one hidden layer . Figure 2.17 Illustration of the fundamental representation equivalence between a 2D time-frequency image and a series of 1D spectrums. . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.18 Illustration of a Long Short-Term Memory network . . . . Figure 2.19The sphere: a simple manifold . . . . . . . . . . . . . . . . .

Chapter 2: theoretical background Figure 2.1

The basic form of a standard signal emitted by a radar . . Figure

2

.2 Illutration of the Radar Cross-Section (RCS) . . . . . . . . . Figure 2.3 Depiction of the three drones used in simulations . . . . . Figure 2.4 Illustration of the simulated signal . . . . . . . . . . . . . . Figure 2.5 Illustration of the Fourier spectrum of a radar signal . . . . Figure 2.6 Mean-Squared Error (MSE) versus Logistic Loss (LL). . . . . Figure 2.7 Activation functions . . . . . . . . . . . . . . . . . . . . . . . Figure 2.8 Illustration of the kernel trick . . . . . . . . . . . . . . . . . Figure 2.9 Representation power of Deep Neural Networks (DNNs) through hierarchical learning. . . . . . . . . . . . . . . . . . Figure 2.10 Illustration of the convolution as a linear transformation . Figure 2.11 LeNet, the historical first Convolutional Neural Network (CNN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 2.13 Illustration of a convolutional block . . . . . . . . . . . . . . Figure 2.12 AlexNet, the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) winner . . . . . . . . . . . . . . . . . . . Figure 2.20 Kullback-Leibler divergence between two univariate Gaussians . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix x List of Figures

  This concern led to a further paradigm shift: from Deep Learning pioneers blaming models to simply learn dataset biases (Colloquium d'Informatique de Sorbonne Université 2018) and encouraging the push towards reasoning, rather than purely learning models (Yoshua Bengio | From System 1 Deep Learning to System 2 Deep Learning | NeurIPS 2019), to the setup of the "xAI" (for "explainable Artificial Intelligence") Defense Advanced Research Projects Agency (Gunning 2017), focus shifted towards smaller, more efficient models. Following this trend, another 2015 famed architecture, the ResN et101 (He et al. 2016), exhibited a mere 1.7M parameters, while outperforming all previously cited models. Many followed, such as the AllConvolutionalN et (Springenberg et al. 2015), or the HybridN et architecture (Robert et al. 2018), which lead us to the following paragraph.

The year 2012 is now seen as a defining landmark to the renewal and explosion of interest in Artificial Intelligence with the seminal work implementing a Convolutional Neural Network dubbed AlexN et

[START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]

for image classification on the ImageNet dataset within the ImageNet Large Scale Visual Recognition Challenge, outperforming for the first time, and by a large margin, traditional methods. The following years led to a trend, simplistically illustrated in Figure

1

.2, which constituted the mainstream approach for several years, with the emergence of deeper and more complex architectures (notably the V GG model Simonyan and Zisserman (2015)), to the point in 2015 when models beat human performance on the ImageNet Large Scale Visual Recognition Challenge challenge (Tsang 2019). However, the models now boasted orders of magnitude more parameters than the number of examples: 60M for AlexN et, 133M for V GG, when the large-scale ImageNet dataset contained "only" 1.2M images. Alongside the practical challenges of training such large models, this disproportion arose the concern that perhaps models were mostly overfitting on the datasets.

  al. 2017), domain adapation (Yair et al. 2019), Neural Network optimization (Marceau-Caron and Ollivier 2016), Brain-Computer Interface (Barachant et al. 2012), Magnetic Resonance Imagery (Pennec et al. 2006), and even Computer Vision (Acharya et al. 2018; Mollahosseini et al. 2016; Tuzel et al. 2006). The field of Information Geometry, initiated with the works of

  Brooks, O. Schwander, F. Barbaresco, J. Schneider, and M. Cord. "Temporal Deep Learning for Drone Micro-Doppler Classification". In: 2018 19th International Radar Symposium (IRS). June 2018, pp. 1-10;

handwritten digits database (Lecun et al. 1998). However, it was only in 2012 that Krizhevsky et al. (2012) popularized on a world- wide scale the usage of CNNs by winning by a large margin the tough ImageNet Large Scale Visual Recognition Challenge (ILSVRC) competition (Russakovsky et

  al. 2015) with their ALexN et CNN depicted in Figure 2.12. Since then, CNNs have known an explosive amount of development, mostly in the field of CV, and until more recently, in other fields dealing with temporal series, such as sleep analysis (Aboalayon et al. 2016), human Brain-Computer Interface (BCI) (Lawhern et al.

	was in 1989
	that LeCun et al. (1989) first proposed to replace dense layers with shared and
	locally connected layers (see Figure 2.10) to exploit locality in images as it was in
	Markov random fields (Cross and Jain 1983; S. Z. Li 1994), hereby creating the
	first convolutional network, or Convolutional Neural Network (CNN), popular-
	ized five to ten years later as LeN et (Lecun et al. 1998), described in Figure 2.11,
	which beat all results over the six previous years (Bottou et al. 1994) of dense (or
	fully-connected) networks on the historical Modified National Institute of Stan-
	dards and Technology (MNIST) 2018), Automatic Speech Recognition (ASR) and Environmental Audio Recogni-
	tion (EAR) (Takahashi et al. 2018; Piczak 2015) and, to a more limited extent, µ-D
	radar classification (Trommel et al. 2016).

  ] P i , we will rather use the more geometrically appropriate Riemannian barycenter G, also known as the Fréchet mean (Yang et al. 2010a) or geometric average, which we also note Bar({P i } i∈[1,N ] ) or Bar(D). The Riemannian barycenter has shown strong theoretical and practical interest in Riemannian data analysis[START_REF] Pennec | A Riemannian Framework for Tensor Computing[END_REF], which justifies its usage in this context. By definition, G is the point on the manifold that minimizes inertia in terms of the Riemannian metric defined in Equation 2.29:

	1
	N	i∈[1,N

2 Karcher flow Karcher 1977 to compute the Riemannian mean of N SPD matrices. Require: data points {P

  al. 2006).

	Algorithm 2.

of one iteration of the Karcher flow (Karcher 1977).

  Data points P i are logarithmically mapped to the S i on the tangent space at G(t) . The S i are then arithmetically averaged, the result of which is exponentially mapped back to the manifold, yielding G (t+1) .

The nearest Riemannian barycenter, also named Minimum Riemannian Distance to Riemannian Mean (MRDRM) in works such as in

[START_REF] Barachant | Multiclass Brain-Computer Interface Classification by Riemannian Geometry[END_REF]

, achieves state-ofthe-art results in certain fields of highly structured time series data, notably in Figure 2.22.: Illustration BCI applications. Also, pioneering works such as Yger and Sugiyama (2015) for electroencephalography (EEG) modelisation, or Yang et al. (

2010b

) for radar classification, make use of a MRDRM scheme to reach high performance with relatively lightweight machinery. Nearest baryencter schemes still remains a simple algorithm from the ML viewpoint, which strongly encourages the usage of IG in ML applications, allowing models better fit to the inout data and its inherent physical conditioning.

  Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Learning on structured time series representations . . . . . . . . . . . . . . . . 3.2 .1 SPD neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 .2 Fully Temporal Convolutional Network . . . . . . . . . . . . . . . . . . 3.2 .3 Complex Fully Temporal Convolutional Network . . . . . . . . . . . . 3.2 .4 HPD neural network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. 3 Full pipeline for temporal classification . . . . . . . . . . . . . . . . . . . . . . . 3.3 .1 The Fourier convolution layer . . . . . . . . . . . . . . . . . . . . . . . . 3.3 .2 Pipeline bifurcations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 .3 Covariance pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. 4 Experimental validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3.4 .1 Drone micro-Doppler radar data simulator . . . . . . . . . . . . . . . . 3.4 .2 Experiments and results . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. 5

  al. 2009; Bartz et al. 2017; Dieleman and Schrauwen 2014; Parascandolo et al. 2016; Cakir et al. 2017; Hershey et al. 2017),

Table 3 .

 3 .1 introduces the varying parameters as well as ranges of variations for each drone. = 5e -8 . A directional scattering point's RCS is computed as being proportional to the dimensions d of the surface it represents, i.e. RCS = d 2

	Drone parameters	Vario	Phantom	S1000+
	Maximal flight curvature κ max (10 -4 m -1 )	[0; 2.55]	[0; 18.5]	[0; 48.5]
	Velocity V (m.s -1 )	[-10; 10]	[-10; 10]	[-10; 10]
	Blade RPM (tr.min -1 )	[1445; 1655] [3000; 7000] [1500; 6500]
	Body's scattering point RCS (m 2 )	0.1	0.1	0.1
		258	3.18	11.4
	Blade's scattering point RCS (10 -3 m 2 )	230	1.79	8.66
		0.796	0.199	0.796

1.: The parameters are found or estimated from drone specifications. Their exact values and variations still need to be heuristically estimated in order to allow for more expressive sampling. Maximal flight curvature is computed as being proportional to mass over carrying surface, i.e. κ max = a κ m π * L 2 b with a κ

Table 3 .

 3 2.: Simulation parameters and their default values. w s is the Fourier window size and p ov the overlap percentage. The table is split in two, to the left are radar parameters, to the right representation parameters.

	sin(κ(t)) , being the element-wise Hadamard product and s a random spin.
	(3.21)

• Simulation time T : 250 ms

• N f f t : 64 points • Fourier window size w s = 20 points • Window overlap percentage p ov = 50%

Again, we take into account the gain in he Fourier transform. Visible temporal variations are as accurate as possible, although exaggerated, w.r.t. reality. They exhibit discriminative behaviours which we hope to capture in learning algorithms.

Table 3 .

 3 3.:

Performance comparison of real and complex deep structures on radar data on various amount of noisy data.

  Models are compared on a decreasing amount of training data, from the full 10 minutes to 30 seconds.

	Train size	100%	20%	5%
	FTCN CRNet FourierNet 70.8 ± 0.22 67.8 ± 0.40 62.1 ± 0.90 67.2 ± 0.27 65.1 ± 0.39 63.5 ± 0.46 68.8 ± 0.17 65.1 ± 0.50 59.3 ± 1.51

Table 3 .4.: Performance comparison of real and complex deep structures on radar data on various amount of less noisy data.

 3 Models are compared on a decreasing amount of training data, from the full 10 minutes to 30 seconds.

	Train size	100%	20%	5%
	FTCN FourierNet 99.0 ± 0.07 94.4 ± 0.14 88.7 ± 1.12 98.6 ± 0.34 94.3 ± 0.57 91.6 ± 0.98

Table 3 .

 3 5.:

Performance comparison of first-and second-order models on radar data.

  The SOFTNet model combines both orders in a single pipeline; its performance expectedly matches or tops that of the individual, unit models. ± 0.[START_REF] Heisenberg | Les Principes Physiques de la Théorie des Quanta[END_REF] 93.4 ± 1.21 84.3 ± 2.51 FourierNet 99.4 ± 0.17 96.2 ± 1.12 87.4 ± 1.94 SpectroSPD 95.1 ± 0.49 91.9 ± 0.82 84.6 ± 3.49 SOFTNet 99.5 ± 0.16 97.2 ± 0.90 93.9 ± 0.74 same windowing is used for covariance and spectral representations to keep comparisons fair, as per illustrated in Figure

	Train size	100%	20%	5%
	SPDNet HPDNet FTCN	92.6 ± 0.54 91.5 ± 0.74 88.4 ± 3.06 94.4 ± 0.76 91.8 ± 0.80 87.1 ± 1.11 98.9

  4.1 Riemannian batch normalization on S + * , training and testing phase SPD matrices {P i } i≤N , final running mean G S , learnt bias G 1: for i ≤ N do

	TRAINING PHASE
	Require: batch of N SPD matrices {P i } i≤N , running mean G S , bias G, momen-tum η
	1: G B ← Bar({P i } i≤N ) 2: G S ← Bar η (G S , G B ) 3: for i ≤ N do 4: Pi ← Γ G B →I d (P i ) 5: Pi ← Γ I d →G ( Pi ) 6: end for 7: return normalized batch { Pi } i≤N	// compute batch mean // update running mean // center batch // bias batch
	INFERENCE PHASE
	Require: batch of N 2:

  intermediate loss function of layer (k -1). A generalization of the chain rule to S * + , linking ∂L (k) ∂P to ∂L (k+1)

Table 4 .

 4 1.:

Performance of SPDNet, FTCN and MRDRM on the NATO dataset. Ac

  curacy is compared on a 5-fold cross-validation; experiments are repeated with only 10% of training data. Model

	SPDNet	MRDRM	FTCN
	# Parameters Acc. (all data) Acc. (10% data) 69.1 69.1 69.1% ± 0.97 67.1% ± 2.17 61.1% ± 3.50 65.6% ± 2.74 ∼ 500 -∼ 500 ∼ 10000 72.6% ± 0.61 69.7% ± 1.12 73.4% ± 3.66 88.7 88.7 88.7% ± 0.83

Table 4 .

 4 2 reports the average accuracies and variances of the proposed DAMNet and SPDNetBN architectures, compared with the original SPDNet.

Table 4 .

 4 2.:

Performance of SPDNet, DAMNet and SPDNetBN on the NATO dataset

	Model	SPDNet	DAMNet	SPDNetBN
	Normalization	-	BarNorm	ParNorm	BatchNorm
	Accuracy Acc. (10% data) 69.1% ± 0.97 73.8% ± 0.25 70.2% ± 1.74 77.7 77.7 77.7% ± 0.95 72.6% ± 0.61 79.9% ± 1.19 80.3% ± 0.55 82.3 82.3 82.3% ± 0.80

. Accuracy is compared on a 5-fold cross-validation; experiments are repeated with only 10% of training data.

Table 4 .

 4 3.:

Performance of SPDNet, DAMNet and SPDNetBN on the AFEW dataset.

  As in previous works, accuracy is measured on a held-out test set.

	Model architecture	SPDNet	DAMNet	SPDNetBN
	Normalization	-	BarNorm ParNorm BatchNorm
	29.9% 31.2% 34.5% {400, 300, 200, 100, 50} 33.7% {400, 50} {400, 100, 50} {400, 200, 100, 50}	32.1% 33.2% 36.1% 34.3%	32.6% 31.8% 35.0% 36.3%	34.9 34.9 34.9% 35.2 35.2 35.2% 36.2 36.2 36.2% 37.1 37.1 37.1%

Table 4 .

 4 4.: Accuracy comparison of SPDNets and DAMNets on the HDM05 dataset. Model 6% ± 1.35 64.2% ± 1.10 63.5% ± 0.98 65.2 65.2 65.2% ± 1.15

		SPDNet	DAMNet	SPDNetBN
	Norm.	-	BarNorm	ParNorm	BatchNorm
	Acc.	61.

Table B .

 B 1.: Duration of recorded data in the database. The duration is split through the classes, and gives a reference amount of number of data points as individual spectrograms. number of 256 × 32 spectrograms. Note that number is obtained with the formula N = T * P RF pov * N f f t * n .

		Phantom3 X8	Iris	Firefly Anaconda Opterra Skywalker
	Total duration (s)	1186	68 1776	740	636	530	12
	Number of spectrograms	7239	415 10840 4516	3882	3235	73

En allant plus loin dans une veine similaire, l'on peut s'atteler à la tâche d'étudier non pas le signal en tant que tel, mais bel et bien la distribution statistique dont il est issu. Dans ce scenario, les familles Gaussiennes constituent un candidat de choix. Formellement, la covariance des vecteurs de données caractérisent entièrement une telle distribution, pour peu qu'on la considère, à peu de frais, centrée; le développement d'algorithmes d'apprentissage, notamment profonds, sur des matrices de covariance, sera ainsi un thème central de cette thèse. L'espace des distributions diverge de manière fondamentale des espaces Euclidiens plats; il s'agit en fait de variétés Riemanniennes courbes, desquelles il conviendra de respecter la géométrie mathématique intrinsèque. Spécifiquement, nous contribuons à des architectures existantes par la création de nouvelles couches inspirées de la géométrie de l'information, notamment une couche de projection sensible aux données, et une couche inspirée de l'algorithme classique de la Batch Normalization. La validation empirique de nos nouveaux modèles se fera dans trois domaines différents: la reconnaissance d'émotions par vidéo, d'action par squelettes, avec une attention toute particulière à la classification de drones par signal radar micro-Doppler. Enfin, nous proposerons une librairie PyTorch aidant à la reproduction des résultats et la facilité de ré-implémentation des algorithmes proposés.iii

The synthetic radar, HDM05 and AFEW datasets may be found at https://www.dropbox.com/ s/dfnlx2bnyh3kjwy/data.zip?dl=0

A p p e n d i x

Here we benchmark both model's properties and illustrate with some sample code. Benchmark are done on two different datasets: a synthetic one of the three drone classes mentioned above (see Figure 2.3), and one of real recordings, the North Atlantic Treaty Organization (NATO) dataset introduced in Section 4.6, the description of which is given in Appendix B (in the experiments, we use only about 3% of the dataset). To recall in short, the synthetic data gathers 1000 examples per class of 250 ms signals sampled at 4kHz, so of initial size (1000, ), rendered to a (99, 20) time-frequency spectrogam after Fourier convolution.

A.1 Model sizes and speeds

A.1.1 Model size

The number of parameters per model are as follows:

A.2

Sample code

The library we propose seamlessly integrates manifold-constrained optimization of structured functions on S + * : the code for setting up the learning of a model in PyTorch is only modified in the usage of the MixOptimizer class, which mixes a conventional optimizer with the Riemannian ones: import t o r c h . nn as nn from mixoptimizer import MixOptimizer . . . model = . . . # d e f i n e t h e m o d e l . . . l =nn . CrossEntropyLoss ( ) # d e f i n e t h e l o s s f u n c t i o n and mixed o p t i m i z e r opt=MixOptimizer ( model . parameters ( ) , l r = l r , momentum= 0 . 9 , weight_decay =5e -4) . . . l . backward ( ) # i n t h e t r a i n i n g l o o p , com pute g r a d i e n t s and u p d a t e w e i g h t s a s u s u a l l y d o n e opt . s t e p ( )

A.3 Use cases

Here we show how to use the library in practice. Following the PyTorch logic, elementary functions are defined in torchspdnet.functional and high-level modules in torchspdnet.nn.

A.3.0.1 Basic SPDNet model

Here we give the most basic use case scenario: given input covariance data of size 20 × 20, we build an SPDNet which reduces its size to 15 then 10 through two Bilinear Mappings (BiMaps) and a Rectified Eigenvalues (ReEig) activation, followed by the Log Eigenvalues (LogEig) and vectorization. Finally, a standard fully-connected layer allows for classification over the 3 classes A p p e n d i x

B D E TA I L E D D E S C R I P T I O N O F T H E N AT O R A D A R D ATA B A S E

This appendix details the North Atlantic Treaty Organization (NATO) database used in experiments, as we have found it to be of major interest both experimentally and of practical interest.

B.1

Global overview

The NATO database consists of recordings of 8 different aiborne subjects, including one class encompassing birds and 7 different drones: The Phantom and the Iris come in two versions: carbon fiber or nylon blades, which can consitute child classes. The drones are categorized in three types: quadcopter, fixed wing and hybrid, which can consitute parent classes. The radar signals are accessible in their raw form of time series of complex points of amplitude and phase. The subjects were furthermore recorded in 6 frequency bands (L,S,C,X,Ku and Ka) and with both vertical and horizontal polarization (except for the L band).

B.2 Description of the data

We call one data point the continuous recording of one subject for an arbitrary amount of time. Signals vary in lenth from 7s to 672s. The Pulse Repetition Frequency (PRF) is set to 25kHz, meaning a signal of 1 min totals 1.5e6 complex numbers. One data point is furthermore segmented into N non-overlapping chunks of size M = 1024 upon which Fast Fourier Transform (FFT) is performed, thus outputting a spectrogram of size M × N . Again, for a signal lasting 1 min, the corresponding spectrogram would be of size 1024 × 1465.

B.3

Contents of the disk

The disk is divided in 5 CDs, in which folders separate the recordings of different subjects. We compactly describe one such folder with the length in seconds of the different signals within it; we omit the multiplicity of bi-polarization and multiple frequency bands, such the description 'Phantom3 (nylon): 465 + 84' means that the folder contains two different data points of respective time length 465s and 84s of the Pantom 3 with nylon blades, repeated across all polarizations and frequency bands. Matlab code is also provided to read and plot the data.

B.4 Possible setting for learning

The database is complete, varied, and as such can inspire many possible settings for learning (usage of both polarizations, usage of hierarchy in classes...). We propose here one simple setting, using single polarization in the S band. We also do not consider hierarchy and retain only the 7 drones, which we do not split in subclasses of blade fabric. Furthermore, from a machine learning point-of-view, it is reasonable to trim potentially overkill information: for instance, M = 1024 FFT points may be more resolution than we need; we propose instead to use a p ov = 50%-overlapping N f f t = 256-point FFT. Also, we wish to consider a minimal elementary signal duration for which learning performance is not affected much. In previous works, it was found a sane choice to operate on a duration of the order of one complete blade rotation. We do not know these values for the drones at hand; however, given that a DJI Phantom 2 rotates at about 4800tr/min, we propose to set the minimal duration around 10ms, which discretizes to 250 sample points, which we round up to 256. We thus propose to apply the same setting here, which converts a T = 1 min signal to a spectrogram of length 11720, or equivalently a set of N = 360 spectrograms of length n = 32 and frequency resolution 256. In the following Table B.1, we sum up the proposed setting by grouping the total recording time for each class along with the corresponding