Chiara Bartolozzi

Timothée Masquelier

Michel Paindavoine

Ryad Be- Nosman

Elisabetta Chicca

Keywords: Réseaux de neurones impulsionnels, Caméras événementielles, Neuromorphique, Vision par ordinateur, Neurosciences computationnelles, Reconnaissance de motifs. viii Spiking neural networks Spiking neural networks, Event-based cameras, Neuromorphic, Computer vision, Computational neuroscience, Pattern recognition. viii Simulation type GPU Backpropagation Auryn

Réseaux de neurones impulsionnels pour la vision embarquée basée sur les événements Résumé

La vision par ordinateur embarquée est récemment devenue omniprésente. Elle englobe des tâches telles que la détection, la reconnaissance et le suivi d'éléments visuels, avec des applications en robotique (conduite autonome), dans l'industrie (évaluation de la qualité des produits, automatisation de tâches répétitives), dans la sécurité, pour l'expérience client, dans les réseaux sociaux, etc. Cet engouement généralisé ne fait que renforcer la nécessité de surmonter les défis posés par ce domaine de recherche, à savoir une consommation d'énergie gargantuesque, une grande mémoire et la prise en charge d'un large éventail d'algorithmes. Nous pensons qu'une réponse prometteuse à ces défis peut être amenée par l'utilisation combinée de réseaux de neurones à impulsions (SNNs) et de caméras événementielles. Les SNNs sont des réseaux de neurones artificiels bio-inspirés qui visent à imiter la dynamique des neurones biologiques en traitant l'information sous forme de séries d'impulsions. Les caméras événementielles sont un nouveau type de capteur visuel bio-inspiré qui génère des données asynchrones en fonction des changements d'intensité des pixels. Elles sont idéales pour les applications en temps réel, mais la grande quantité d'informations temporisées qu'elles génèrent est difficile à traiter à l'aide de modèles traditionnels de vision par ordinateur. Cependant, les données événementielles se combinent naturellement aux SNNs en termes d'inspiration biologique, d'économie d'énergie, de latence et d'utilisation de la mémoire, pour le traitement dynamique des données visuelles.

Cependant, la nouveauté des SNNs et caméras événementielles laisse place à de nombreuses améliorations en termes de prétraitement optimal des données ainsi que leur traitement subséquent, en tirant le meilleur parti des particularités de ces concepts scientifiques. Dans cette thèse, nous avons identifié plusieurs problématiques liées à ce vaste champ de recherche, que nous avons condensées en deux thèmes principaux. La première problématique concerne l'optimisation du prétraitement embarqué des données événementielles acquises par une caméra embarquée pour en faciliter l'analyse ultérieure. Nous proposons trois solutions : les événements pourraient soit 1) être réduits dans l'espace ou dans le temps, online ou offline ; 2) ne conserver que les éléments saillants et rejeter le reste ; 3) être soumis à un mécanisme de fovéation, selon un compromis bio-plausible entre les deux solutions précédentes. Nous avons comparé qualitativement et quantitativement les données obtenues après chaque méthode de prétraitement afin d'évaluer laquelle de ces méthodes amène à un meilleur compromis entre entre la quantité de données (c'est-à-dire le nombre d'événements) conservées et la pertinence de l'information préservée. Le second défi est l'exploitation de l'adéquation des SNNs pour traiter la temporalité spécifique des événements dans un contexte embarqué. Dans l'ensemble, nous espérons avoir apporté une contribution utile à l'exploitation des avantages uniques de la combinaison des SNNs avec les caméras événementielles pour la vision par ordinateur embarquée, en particulier en ce qui concerne le prétraitement des données événementielles.

Introduction

Human beings have always drawn inspiration from the biological world surrounding us to create new technologies. Over 500 years ago, Leonardo da Vinci was one of the first people to theorise the concept of bio-mimicry -"Go take your lessons from nature, that's where our future lies" -and his inventions paved the way for bio-inspired technologies such as aeroplanes, Velcro, solar panels, turbines, pressure-resistant helmets, efficient heating systems, hammers, vaccination, ... and more recently, artificial intelligence.

Talos, Golems, Frankenstein and other manufactured intelligent humanoids have been part of the popular consciousness since ancient times. Still, it was only in 1950 that Alan Turing published his landmark paper "Computing Machinery and Intelligence", where he discussed machines' potential consciousness and thinking ability. Six years later, the historic workshop Dartmouth Summer Research Project on Artificial Intelligence was the birthplace of the field of artificial intelligence (AI) research, especially the Logic Theorist program, which aimed to mimic human problem-solving skills. This conference was followed by a long period of successive success and setbacks, which finally ended in the 2000s with the successful application of machine learning (ML) to academia and industry tasks combined with the increase in computer hardware performance and the size of available data.

How can visual neuroscience improve computer vision ?

1.1.1 Why should computer vision draw inspiration from visual neuroscience ?

In recent years, the performance of computer vision algorithms increased tremendously. In 2010, the first winner of the ImageNet Large Scale Visual Recognition Challenge (ILS-VRC) [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]) reached a 52.9% accuracy [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], which AlexNet, i.e. the first deep convolutional neural network (CNN) applied to ImageNet, quickly outperformed in 2012 (63.3% accuracy) [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]. According to [START_REF] Alom | The history began from AlexNet: A comprehensive survey on deep learning approaches[END_REF], this "significant breakthrough in the field of ML and computer vision for visual recognition and classification tasks [...] is the point in history where interest in deep learning increased rapidly". From this point on, the successive ILSVRC winners produced increasingly good results, reaching in 2023 an impressive 91.1% accuracy on a global dataset of more than 14, 2M samples distributed in almost 22K classes (X. [START_REF] Chen | Symbolic discovery of optimization algorithms[END_REF]. It is worth noting that these DL models' performance overcame the human performance in 2015 (see Fig. 1.1), only three years after the breakthrough bought by AlexNet. Indeed, it was experimentally assessed that trained human annotators reach a maximum accuracy of 94.9% (top-5 classification error) while classifying samples from ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], which was surpassed by ResNet-152's validation error of 95.4% [START_REF] He | Deep residual learning for image recognition[END_REF]. An "optimistic" human 1 2 CHAPITRE 1 -Introduction accuracy was approximated at 97.6% -equally surpassed by a DL model in 2018 [START_REF] Mahajan | Exploring the limits of weakly supervised pretraining[END_REF]. Overall, these recent achievements of DL models could suggest that despite its early bioinspiration, ML no longer needs visual neuroscience to improve its performance. As recently declared by Yann LeCun, one of the scientific leaders in AI research, the latter should no longer "be blinded by biology" [START_REF] Cox | Neural networks and neuroscience-inspired computer vision[END_REF].

However, this progress must be nuanced: the superhuman performances are to be counterbalanced by the proportionate size of these networks and by the gargantuan quantity of energy consumed to produce these results. Let us take a closer look at some of the models mentioned previously: ResNet-152 used 40M parameters ; AlexNet consists of 13 layers, including 5 convolutional ones, implemented with 60M parameters ; and the current ILSVRC 2023 laureate reached a substantial 2440M parameters. While the computer performance in terms of floating-point operations per second (FLOPS) is known * for each of these architectures, the energy consumption has not yet been studied -however, it has been estimated that a recognition task among 1,000 different objects performed on a standard computer consumes about 250W [START_REF] Weerasinghe | Neuromorphic spiking neural network algorithms for machine learning and pattern recognition[END_REF].

If we direct our gaze to more modern systems, Chat-GPT3 uses 175 billion parameters, and its energy consumption during training was approximated at 1, 287M W h [START_REF] Patterson | Carbon emissions and large neural network training[END_REF], i.e. around 550 tonnes of CO 2 eq. This corresponds to the equivalent of 550 round trips from Paris to New York by plane and 250 times what a European citizen is supposed to consume per year to achieve carbon neutrality ! We do not know the actual energy consumption of the most recent version of Chat-GPT (based on GPT4), but we can only assume from its staggering number of parameters (170 trillion) that it must approximate a 1, 000 times the consumption of GPT3.

1.1 -1.1.2 Which neuro-inspired tools should one use to improve computer vision ? 3

In contrast to the neural networks described above, the human brain from which their designers draw inspiration is significantly more energy efficient: with a power budget of only 20W , it allows us to "make sense of a complex and ever-changing world, to plan complex actions, to navigate our social environment and intuit the minds of others, and to learn and remember across our entire lifespans" [START_REF] Cox | Neural networks and neuroscience-inspired computer vision[END_REF]. Its architecture enables these impressive feats: over 100 billion neurons operating simultaneously, interconnected by 100 trillion synapses. The latter propagate action potentials at a relatively low speed ranging from 1 to 100m/s, with an additional slowdown caused by dendritic and axonal delays ; the visual signal perceived by the retina reaches the primary visual cortex after 50ms, and the later stages of visual processing take place after almost 200ms. However, even though these propagation times are way slower than manufactured silicon computers performing up to billions of computational cycles per second, the cerebral structure's high parallelism and dense connectivity allow information to span between distant brain regions rapidly [START_REF] Liao | Small-world human brain networks: Perspectives and challenges[END_REF].

All the figures described above underline the brutal difference in energy consumption between computer vision and a human being, which even the superhuman performances mentioned previously do not make up for. To close this gap and reduce the impact of the digital world on the environment (responsible for 3.8% of greenhouse gas emissions and consuming 4.2% of primary energy in 2019 [START_REF] Bordage | Empreinte environnementale du numérique mondial[END_REF]), it is crucial to draw inspiration from visual neuroscience to improve computer vision.

Which neuro-inspired tools should one use to improve computer vision ?

Introduced as the "third generation of neural network models", SNNs represent an asynchronous type of artificial neural network (ANN) closer to biology than traditional artificial networks, mainly because they seek to mimic the dynamics of neural membrane and action potentials over time [START_REF] Maass | Networks of spiking neurons: The third generation of neural network models[END_REF]. Their event-driven nature (the SNN's neurons with no spikes remain idle while the ANN's units are always active), as well as their summation operations (binary inputs compared to the ANN's floating point values) that are inherently less computationally expensive, contribute to their energy efficiency over a given period [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF]. The SNNs' bio-inspiration allows for them to be implemented on neuromorphic hardware, i.e. silicon architectures whose elemental building blocks work as biological neurons and transmit the information as electric impulses (the "spikes").

Event cameras are a novel bio-inspired sensor, akin to a "silicon retina" [START_REF] Mahowald | The Silicon Retina[END_REF], which encodes the visual information as spikes ; they are bringing an emerging vision paradigm by mimicking the biological retina. In 2008, Lichtsteiner, Posch and Delbruck presented the first complete design of a Dynamic Vision Sensor (DVS) (Lichtsteiner, Posch, & Delbruck, 2008a) responding only to temporal brightness change in a scene with no consideration for colours, similar to the organic retina. This asynchronous sensor records visual data with a microsecond temporal resolution and a high dynamic range (120dB compared to the human eye's 35dB and traditional red-green-blue (RGB) camera's 60dB), with significantly low power (10mW compared to the human eye's 10mW ) [START_REF] Gallego | Event-based vision: A survey[END_REF]. Event data require low memory to be stored, 1000 times less than RGB videos (Kilobytes versus Megabytes) (Rebecq, Gallego, Mueggler, & Scaramuzza, 2018).

SNNs receive and process information as spike trains, meaning as a non-monotonous sequence of activations. Therefore, they make for a suitable candidate for the efficient processing of incoming event patterns measured by event-based cameras: each event is assimilated to an activation CHAPITRE 1 -Introduction spike between two spiking neurons. As stated by [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF], "a suitable spiking neuron model with proper synaptic plasticity while exploiting event-based, data-driven updates is a major goal among neuromorphic engineers to enable computationally efficient intelligent applications".

Both those scientific concepts fall within the spectrum of "neuromorphic computing", a concept invented in the 1980s by Prof. Carver Mead at Caltech "which combines biology, electrical engineering, computer science and mathematics to create ANNs able to handle information as a human brain and nervous system would" (L, 2021). Nowadays, this concept includes both different processing strategies, such as spintronics, in-memory computing, SNNs, photonics, etc [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF], as well as various sensors. We can mention some for each of the five senses: [START_REF] Mahowald | The Silicon Retina[END_REF] and (Lichtsteiner et al., 2008a) for the sight, [START_REF] Lyon | An analog electronic cochlea[END_REF] and [START_REF] Chan | Aer ear: A matched silicon cochlea pair with address event representation interface[END_REF] for the hearing, [START_REF] Koickal | Analog vlsi circuit implementation of an adaptive neuromorphic olfaction chip[END_REF] and [START_REF] Koickal | Analog vlsi circuit implementation of an adaptive neuromorphic olfaction chip[END_REF] for the smell, [START_REF] Yang | Neuromorphic gustatory system with salt-taste perception, information processing, and excessive-intake warning capabilities[END_REF] and [START_REF] Lebow | Real-time edge neuromorphic tasting from chemical microsensor arrays[END_REF] for the taste and [START_REF] Birkoben | A spiking and adapting tactile sensor for neuromorphic applications[END_REF] and [START_REF] Macdonald | Neuromorphic tactile edge orientation classification in an unsupervised spiking neural network[END_REF] for the touch.

In this thesis, we propose tackling this challenging task by capitalising on two compatible scientific concepts: SNNs and event cameras, both allowing for neuromorphic computing. We present below the scientific context and motivations of this work, as well as the contributions achieved.

Scientific context

This PhD work has been conducted under the supervision of Prof. Jean Martinet, within the SPARKS Team (i3S, 2023b) at the I3S laboratory in Sophia Antipolis, France (i3S, 2023a) ; and has been co-supervised by Dr Laurent Perrinet, a researcher at the Institute of Neurosciences of Timone in Marseille, France (INT, 2023).

This thesis is undertaken in the context of the European APROVIS3D project (April 2020-September 2023) (APROVIS3D, 2023), supported by the European Union's ERA-NET CHIST-ERA 2018 research and innovation programme under grant agreement ANR-19-CHR3-0008. The objective of the APROVIS3D project is to design and implement ML methods based on SNNs to extract visual features and infer useful information from the visual scene using stereo event cameras. In particular, the project focuses on two use cases: coastline tracking and depth estimation.

The consortium from four EU countries shares a common interest in analog-based computing and computer vision and offers a unique combination of expertise this project requires. It was created with the following objective: SNNs specialists from various fields, such as visual sensors (IMSE, Spain), neural network architecture and computer vision (UCA, France and University of Lille, France) and computational neuroscience (INT, France), will team up with robotics and automatic control specialists (NTUA, Greece) and low power integrated systems designers (ETHZ, Switzerland) to help geoinformatics researchers (UNIWA, Greece) build a demonstrator UAV for coastal surveillance, evaluated at software level for comparison with standard image-based approaches in terms of accuracy and energy consumption, before hardware integration.

The key challenges of this project will be to design an end-to-end analog system (from sensing to AI-based control of the UAV and 3D coastal volumetric reconstruction) which is energy efficient and adapted to real conditions. The project relies on SpiNNaker applied to a stereopsis system dedicated to coastal surveillance using an aerial robot. This project is additionally the oc-1.3 -Our motivations and objectives casion to embed and experimentally assess the neuromorphic foveation mechanism as designed by IMSE [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF]. APROVIS3D aims to show that such a bioinspired analog design will bring considerable benefits regarding energy efficiency and adaptability needed to make coastal surveillance with UAVs practical and more efficient than digital approaches.

Our motivations and objectives

As described above, the main objectives of the APROVIS3D project are coastline detection for autonomous navigation of a UAV and 3D reconstruction from a stereo stream of events, using an SNN implemented on the neuromorphic hardware SpiNNaker and interfaced with two DVSs, both embedded on a UAV. Our work falls within this broad spectrum and, more specifically, addresses the issue of optimising † the preprocessing of event data acquired by an event camera to facilitate subsequent analysis. We specifically target their combined use for embedded systems, i.e. hardware systems with software designed to perform real-time operations for a dedicated function either independently or as a part of a larger system (HEAVY. AI, 2022). The UAV application task of APROVIS3D dictates this choice, which leads to the development of frugal neural networks.

Neuromorphic data preprocessing for optimised online analysis

Indeed, embedded systems and even early realised neuromorphic embedded processors are quite limited in terms of memory bandwidth. A UAV has available low processing power and low memory: its limited battery lifetime leads to limited flight time and requires planning UAV missions to minimise energy consumption [START_REF] Abeywickrama | Empirical power consumption model for uavs[END_REF].

While the data produced by event cameras are inherently sparse and seem ideally suited to minimise the issue of embedded energy consumption and data storage, the event flow can be very high. Event cameras have a very high temporal resolution (in the order of microseconds), meaning that one pixel can theoretically generate 1 million events per second at most. A DVS128 of resolution 128 × 128 could produce thus 16.4 billion events per second. While this case never happens in real use due to the events' intrinsic sparsity and non-redundancy, several million events per second have been recorded in highly dynamic and textured scenes [START_REF] Tayarani-Najaran | Event-based sensing and signal processing in the visual, auditory, and olfactory domain: A review[END_REF]. These heavy event streams require significant resources to handle and may be too dense to be correctly processed by the state-of-the-art low-power embedded system. Indeed, as stated in [START_REF] Sorbaro | Optimizing the energy consumption of spiking neural networks for neuromorphic applications[END_REF]: since most neuromorphic architectures require the neuron state to be fetched from memory at each synaptic event then be rewritten and each of those operations consume a certain amount of energy (usually around 10 -11 J [START_REF] Indiveri | The importance of space and time for signal processing in neuromorphic agents: The challenge of developing low-power, autonomous agents that interact with the environment[END_REF]) -8x10 -9 J for SpiNNaker [START_REF] Stromatias | Power analysis of large-scale, real-time neural networks on spinnaker[END_REF] and 23.6x10 -12 J for Loihi [START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF]), "reducing the number [of synaptic operations] is therefore the most natural way to keep energy usage low". Furthermore, recent study shows that in certain lighting conditions, high-resolution event cameras produce data susceptible to temporal noise and with an increasingly high per-pixel event rate, thus leading to the decreased performance of some traditional computer vision tasks (D. [START_REF] Gehrig | Are high-resolution cameras really needed ?[END_REF]. While 6 CHAPITRE 1 -Introduction desktop event vision solutions can afford considerable computing resources, embedded processing can suffer from hardware limitations such as energy resources, memory, computational power and communication bandwidth. In such situations where a dense event stream can neither be stored nor processed online, it is likely that some (if not most) events will be randomly dropped, which will obviously impede correct processing.

Additionally, this heavy flow of information may not always be relevant to the task at hand. Some work simply chose to reduce this information by pooling the events into frames, then processing this data as standard visual images (Fang, Yu, Chen, Masquelier, et al., 2021 ;[START_REF] Huang | Event-based timestamp image encoding network for human action recognition and anticipation[END_REF]. A concrete example can be found in real-time event-based odometry applied to robotics, which requires only movement information and whose needs can typically be answered by frame analysis [START_REF] Rebecq | Real-time visual-inertial odometry for event cameras using keyframe-based nonlinear optimization[END_REF]. However, its main drawback is that the asynchronicity of the event data, which is a real advantage of event data, is disregarded ; all events are gathered, and their timestamps are discretised at a higher temporal grain.

Other works aim to reduce the induced high storage requirements by implementing event data compression methods, such as entropy coding [START_REF] Gallego | Event-based vision: A survey[END_REF], dictionary-based compression [START_REF] Ziv | A universal algorithm for sequential data compression[END_REF][START_REF] Alakuijala | Brotli: A general-purpose data compressor[END_REF], Internet of Things specific compression [START_REF] Blalock | Sprintz: Time series compression for the internet of things[END_REF][START_REF] Gallego | Event-based vision: A survey[END_REF], fast integer compression [START_REF] Khan | Lossless compression of data from static and mobile dynamic vision sensors-performance and trade-offs[END_REF], 2021), etc. These compression techniques can be lossy -see Poisson Disk Sampling-based Lossy Event Compression [START_REF] Banerjee | Lossy event compression based on image-derived quad trees and poisson disk sampling[END_REF]) -or lossless -see [START_REF] Khan | Lossless compression of data from static and mobile dynamic vision sensors-performance and trade-offs[END_REF][START_REF] Khan | Time-aggregation-based lossless video encoding for neuromorphic vision sensor data[END_REF]. Their major disadvantage is their inability to compress data in real-time during acquisition.

Furthermore, deep SNN models can have difficulties handling a great amount of data simultaneously while minimising their energy consumption, especially at a high temporal resolution. Some chose to compress these models using attention, as proposed in [START_REF] Kundu | Spike-thrift: Towards energy-efficient deep spiking neural networks by limiting spiking activity via attention-guided compression[END_REF].

All in all, drawing from the multiple examples presented above, we are convinced that the issue of event data optimisation for onboard analysis must be addressed. Therefore the first core problem of this thesis is: How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for any embedded data processing ? In other words, what is the best method to reduce the number of events to be processed while retaining a maximum amount of information relevant to this process to optimise the embedded system performance ?

Event-data downscaling Our first approach to solving this wide-ranging issue is to reduce the event stream by simply downscaling the spatial and temporal resolution. Downscaling vision data is an essential feature in a system, especially if embedded, so as to be able to adjust the complexity of data to the available resources, such as processing capability and energy consumption. Reducing the size of visual data is straightforward for grayscale and RGB images with interpolation. When it comes to event data, size reduction is much less trivial.

It is important to note here that our main goal above all is to reduce the number of events in the input stream above. To solve this problem, we opted to reduce either the spatial dimension of the sensor or the grain size of the data recording time. These solutions should reduce the number of events by decreasing the number of possible spatial or temporal coordinates. However one should note that the impact of event reduction on the energy consumption depends on the downstream system: if it is event-based, the goal of lower energy consumption is valid. However, in a clockbased system, the downscaling of the temporal resolution is more impacting than a lower number of events: a lower temporal resolution leads to a smaller number of timesteps to simulate, thus a lower energy consumption and processing time ; which is not necessarily the case for a lower number of events. Regarding spatial downscaling, the reduced map size is beneficial for both event-and clock-based systems. Additionally, one could argue that the reduced data provided as input would only have an impact on the resources consumed by the first layer of the downstream network, especially in DL models. This argument can easily be dismissed here, as we are looking to facilitate embedded event data processing and such DL models are rarely embedded due to their high energy consumption. Furthermore, it is easily observable in many deep SNN models that the output activity is proportional to the output activity.

Most existing event vision approaches handle event data preprocessing by simply downscaling the pixel coordinates, which is actually a max-pooling operation in the event domain. Such as strategy is adopted by the Tonic's "Downsample" method [START_REF] Lenz | Tonic: event-based datasets and transformations[END_REF] (see Section 3.2.3), which is, to the best of our knowledge, the only existing tool for event data downscaling postrecording. One obvious issue of this simple event "funnelling", especially in the spatial dimension, is that the resulting event density drastically increases in the reduced spatial dimension, causing a critical information loss. Indeed, the precision of (x, y) coordinates decreases and the fine details of the recorded objects are lost.

Attempts to overcome this issue have been made by using hardware-integrated filters to reduce the event rate. For example, the event camera Prophesee Gen4 implements a row-wise arbitration, introducing systematic errors in the event timestamps, thus randomly skipping events [START_REF] Finateu | 5.10 a 1280×720 back-illuminated stacked temporal contrast event-based vision sensor with 4.86µm pixels, 1.066geps readout, programmable event-rate controller and compressive data-formatting pipeline[END_REF]. The authors in [START_REF] Delbrück | Feedback control of event cameras[END_REF] introduce fixed-step feedback controllers, which automatically adjust the camera parameters according to the input event rate and noise: they adjust the threshold and refractory period to regulate the event rate and adapt the bandwidth to regulate the noise. However, these filters significantly alter the visual data and therefore are not a viable solution.

In neuromorphic ML, SNNs consider each pixel as an input neuron, and each event is produced as an input spike to the neural model. However, they usually cannot handle the flow of information the event data represents as this first layer is often too important for efficient computation ; thus, most authors applying this type of neuromorphic computing on event data resolve to convolve the input. For example, [START_REF] Cordone | Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks[END_REF] apply multiple convolution layers to the dataset DVS128 Gesture before processing it. We can also mention the residual graph convolutional techniques, which handle each input event as a node in the graph [START_REF] Bi | Graph-based object classification for neuromorphic vision sensing[END_REF]: this leads to low computational efficiency and a substantial simulation run and only produces satisfying results on a sparse dataset with well-defined empty areas like the one they use.

The aforementioned examples are just a few reasons why event downscaling is a real challenge in neuromorphic computer vision and needs to be formalised as a step further towards sparse data processing.

Visual attention for salient selection of OoIs Our second approach to solving the wide-ranging issue of event data preprocessing is to process only the events belonging to the salient elements of the visual scene, i.e. the elements noticeable and/or relevant for the task at hand. We identify regions of interest (RoIs -or salient regions) as a set of pixels recording a high spatiotemporal density of events. The objects of interest (OoIs) thus correspond to events that cluster together spatiotemporally to form the structure of a visual object.

CHAPITRE 1 -Introduction

We believe that detecting RoIs in the original visual scene to select the corresponding OoIs (i.e. the events taking place in the detected RoIs) is the missing element to optimising the tradeoff between the quantity and the quality of the reduced event data. Additionally, it is a promising approach to increase accuracy, reducing both latency and memory requirements. Indeed, the study of many biological organisms highlighted the importance of limiting the processing of sensory RoIs, thus minimising the energy spent on this task as well as the reaction delay to what is perceived [START_REF] Tsotsos | Analyzing vision at the complexity level[END_REF]. For prey animals, for example, this means a quicker detection of a potential predator and a more efficient escape from danger. Attention was first described as the coalition of "focalisation, concentration and consciousness" by William James in 1890 [START_REF] James | The principles of psychology[END_REF]. According to [START_REF] Borji | State-of-the-art in visual attention modeling[END_REF], visual attention can be defined as the behavioural and cognitive process of selectively focusing on a discrete aspect of sensory cues while disregarding other perceivable information. In psychology, this mechanism is more recently defined as the allocation of limited cognitive processing resources on one or a few relevant environmental elements while ignoring others [START_REF] Anderson | Cognitive Psychology and Its Implications[END_REF]. It can be either subjective or objective, as well as voluntary or instinctive.

Taking inspiration from the primates' visual attention mechanisms [START_REF] Moosmann | Learning saliency maps for object categorization[END_REF][START_REF] Itti | Computational modelling of visual attention[END_REF][START_REF] Martinet | Gaze based quality assessment of visual media understanding[END_REF][START_REF] Aydemir | Tempsal -uncovering temporal information for deep saliency prediction[END_REF], many computer vision tasks (classification, object tracking, autonomous navigation, etc.) rely on small salient items in the global scene to limit the amount of data to be processed. As technologies for sensory information processing expanded, an increasing number of them assimilated this idea for their own purposes. In the deep learning community, for instance, this concept was converted into a neural network's component weighting features by the task's level of importance. Attention can thus be applied to regions in images, words in text, phonemes in speech, etc.

Works that implement neuromorphic visual attention applied to event data are rare in the literature. Only a few models genuinely take advantage of the intrinsic dynamics of SNNs and the uniqueness of event data: in particular, [START_REF] Renner | Event-based attention and tracking on neuromorphic hardware[END_REF]) make use of the mathematical model of DNF [START_REF] Sandamirskaya | Dynamic neural fields as a step toward cognitive neuromorphic architectures[END_REF] as a soft Winner-Takes-All (WTA) to implement tracking of salient objects, which position in the sensor was known due to an initial man-made pre-activation. Therefore, we believe that the neuromorphic salient selection of OoIs is a promising axis of research to optimise the event data to be processed.

Neuromorphic foveation

We propose a third and last approach to optimise the on-and off-line processing of event data as a combination of the two previously targeted solutions: foveation, a visual neuroscience mechanism allowing the complex eye to acquire relevant information selectively. We define here neuromorphic foveation as a feedback loop between an event camera and a neuromorphic saliency detector, merging events at multiple resolutions according to identified RoIs. Teresa Serrano-Gotarredona and Bernabé Linares-Barranco, two APROVIS3D collaborators, recently patented an electronically foveated DVS (e-Fov DVS) [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF], whose development was considerably delayed by the sanitary crisis (see Section 5.2.3). As a result of this impediment, this sensor has not been associated with a feedback loop saliency detection mechanism and validated on a computer vision task within the timeline of this work -our objective is to implement software neuromorphic foveation as a first step towards this direction.

1. 4 -1.3.2 Neuromorphic embedded processing of spatiotemporal patterns

Neuromorphic embedded processing of spatiotemporal patterns

While this PhD thesis' core objective is the optimisation of event data preprocessing for embedded neuromorphic analysis (as described in the previous section), it only answers part of the APROVIS3D project goals. Embedded coastline detection and 3D reconstruction will benefit significantly from our results but cannot be achieved solely thanks to them. We thus turned our attention to solving these tasks.

Our analysis of these problems raised the importance and relevance of applying SNNs to event cameras to solve computer vision tasks dedicated to spatiotemporal pattern processing. The second problem we addressed is thus the following: How can we exploit the relevance of the SNN to exploit the specific chronology of event data in an embedded context ? In other words, how do we handle precise movements recorded by an event camera using the crucial suitability of SNNs ?

The specific tasks we identified as linking this problem to APROVIS3D are coastline detection using delay learning or simple SNN architectures ; 3D reconstruction ; and lip-reading.

Our contributions

We established in the previous section our motivations to tackle the following two wideranging issues: How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for any embedded data processing ? And how can we exploit the relevance of the SNN to process the specific chronology of event data ? In the paragraphs below, we describe the work carried out and the results obtained during this thesis.

1.4.1 How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for any embedded data processing ?

We sought to address this issue by experimenting with several neuromorphic and nonneuromorphic mechanisms.

Event data downscaling As a first approach to solve the problem posed by the excessive flow of events, we implemented ten event data downscaling methods:

-six "spatial" methods, i.e. influencing the spatial resolution according to a spatial dividing factor with no effect on the temporal resolution: > spatial funnelling ; > event count ; > linear log-luminance reconstruction ; > cubic log-luminance reconstruction ; > SNN pooling with separate handling of the polarity ; > SNN pooling with mutual handling of the polarity ; -four "temporal" methods, i.e. influencing the event flow temporally according to a temporal dividing factor with no effect on the spatial resolution: > synchronous temporal funnelling ; > asynchronous temporal funnelling ; > stochastic structural downscaling ; > deterministic structural downscaling.

To compare these various methods, we study quantitative and qualitative descriptors of the two event-based datasets DVS 128 Gesture (Amir et al., 2017) and N-MNIST [START_REF] Orchard | Converting static image datasets to spiking neuromorphic datasets using saccades[END_REF], downscaled using these methods. The quantitative descriptors correspond to the number of events, temporal density, information entropy (as defined in Shannon's information theory) and the computation time for on-and off-line downscaling (respectively corresponding to the reduction of event data as it is being recorded or of the whole dataset at once). The qualitative descriptors correspond to the optical coherence (as estimated by our collaborators at IMSE, specialists in neuromorphic sensors) and the classification performance of the two neuromorphic classifiers using respectively the Parametric Leaky Integrate-and-Fire (PLIF) neuron model (Fang, Yu, Chen, Masquelier, et al., 2021) and the Spike Layer Error Reassignment (SLAYER) framework [START_REF] Shrestha | SLAYER: Spike layer error reassignment in time[END_REF]. We also describe the results of a gesture classification task performed by a set of human users to the latter.

Considering that our application is embedded processing of the reduced data, we give more weight to the criteria "number of events", "online computation time" and "classification performance" (either machine or human). The SNN pooling methods are both implemented on the neuromorphic hardware SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF] ; the results tend to show that the best choice to perform spatial event reduction on an embedded system to adjust the size of data to the available resources, such as processing capability and energy consumption, can be found among the SNN pooling methods. Suppose that we do not limit our criterion to spatial downscaling. In that case, the stochastic structural method that consists in randomly dropping events shows exciting performances, with a classification accuracy that remains very high even when dropping a high ratio of events.

Our human gesture classification assessment showed that the spatial dividing factor needs to be maintained at a certain threshold (8 for a 128 × 128 sensor) to ensure that human performance does not fall below the chance level. Overall, the quality of the data obtained from our downscaling methods is not uniform, and we identified some surprising discrepancies between human and ML approaches to gesture classification using event data: human accuracy is higher than machine accuracy in specific dividing factors for specific methods, such as the synchronous temporal funnelling and mutual and separate SNN techniques. This study sheds light on the potential limitations of event data downscaling and provides insights into the human perception of gesture classification using event data.

Visual attention for salient selection of OoIs Following the results presented above, we hypothesised that the trade-off between the quantity of events versus the quality of the retained information might be improved by only processing events corresponding to OoIs, i.e. salient elements of the visual scene. In collaboration with the groups of Michele Magno (partner of APROVIS3D) and Luca Benini, both at ETH Zürich, we implemented to this end a low-latency neuromorphic model of visual attention and salient spatiotemporal selection, which simultaneously outputs one or multiple OoIs detected in an event-based scene. Our architectures are designed to be lightweight enough to enable running in real-time. The RoIs are detected thanks to intrinsic SNN dynamics that accumulate spikes with a leaky behaviour The generalisability of the model is ensured by dynamic adaptation rules applied to synaptic weights and neural thresholds.

We implemented various versions of our visual attention model: -a saliency detector, which detects RoIs ; -a model for the salient selection of one OoI at a time ; -a model for the simultaneous salient selection of multiple OoIs at a time.

1. 4 -1.4.1 How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for any embedded data processing ? 11

Those models have been developed in close collaboration with fellow members of APROVIS3D at ETH Zürich and IMSE. They have been experimentally validated on a central processing unit (CPU) and on the neuromorphic hardware Loihi [START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF] and the academic platform Kraken (Di [START_REF] Mauro | Kraken: A direct event/frame-based multi-sensor fusion soc for ultra-efficient visual processing in nano-uavs[END_REF].

We demonstrated the quality of the visual attention mechanism implemented thanks to quantitative and qualitative descriptors. After salient selection of one OoI in the DVS 128 Gesture dataset (where we hypothesise the OoI to be the moving arms), we obtained a repartition of events in space reduced by 80% and a classification accuracy maintained at an acceptable 70% of the one achieved on the original dataset. On a custom-made dataset with multiple OoIs, we confirmed that our model performs a saliency detection akin to biological visual attention. When confronted with a visual scene with multiple OoIs and asked to select only one, we expect a human viewer to always select either the one with the highest density of information (i.e. highest number of events and spatiotemporal density) or the one that comes first when the shift between the multiple OoIs is big enough. Our model exhibited similar behaviour in our results, which could be improved by experimental validation of our hypothesis concerning the human-like visual attention our model mimics.

Additionally, we demonstrated the usefulness of dynamic rules for online weight and threshold adaptations, which, to our knowledge, cannot be implemented on Loihi: the results obtained with the adaptation rules (on CPU and Kraken) are significantly better than those obtained without those rules (on Loihi).

Our innovative model for simultaneous salient selection of multiple OoIs at a time equally achieves good results. While it only accurately filters out n OoIs out of n initially present in the input event data, it reaches a selection latency of 15ms (5ms in best-case scenarios). It selects OoIs with a quality leading to a classification performance reaching 73% of the performance obtained on original data while reducing the activated surface of the sensor (i.e. the average number of pixels activated over time) by 1000, all with no training phase and a reduced number of neurons and connections compared to state-of-the-art salient detection methods.

Implementing the models described above on SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF] is currently being undertaken to align with APROVIS3D constraints.

Neuromorphic foveation

We finally worked on drawing benefits from both approaches described below (event data reduction and neuromorphic visual attention mechanisms) to offer a third and last solution to the problem posed by the excessive flow of events. Our neuromorphic foveation model merges events at multiple resolutions according to the RoIs detected using the first model described in the previous section. We experimentally validated this mechanism on software as a preliminary work to its hardware implementation. We implemented a "binary" version of this mechanism, akin to the combination of sample events in high resolution and low resolution using a mask. We applied it to the two datasets DDD17 [START_REF] Binas | DDD17: End-To-End DAVIS Driving Dataset[END_REF] and DVS 128 Gesture (Amir et al., 2017) to study the performance evolution for two computer vision tasks, respectively semantic segmentation and classification, according to the remaining number of events. We deepened the study using additional quantitative descriptors, i.e. the spatial and temporal densities of the RoIs.

The semantic segmentation results demonstrated that the foveated data managed to keep on average the same performance as with the original dataset while decreasing by two-thirds the mean number of events to reach the low-resolution value. On the classification side, the average number of events per sample drops to 1% of the original metric after reduction and 50% after foveation, while the performance decreases respectively by 63% and only by 25%. This difference is explained by the intrinsic distinction between the datasets and tasks: classifying a DVS 128 Gesture sample can be done by identifying the movement of the hands, i.e. by following the trajectory of the RoIs while segmenting a driving scene requires more information and cannot rely solely on the data provided in the RoIs, although these greatly facilitate the task. Combined observations confirm our core thesis that neuromorphic foveation leads to the best trade-off between information quantity and quality, at least on a software level.

Furthermore, we studied the performance evolution of the dataset pre-processed with various methods (reduction, visual attention and neuromorphic foveation) when reduced using an increasing factor of stochastic structural downscaling. Interestingly, while all three data types show the same behaviour, the averaged foveated data outperforms the high-resolution data when the 60% threshold of decrease is crossed. This is explained by the fact that most events kept in the foveated dataset provide relevant information to the semantic segmentation model. In contrast, a significant part of the events in the original dataset is not as valuable.

In summary, we demonstrated the stakes of foveation applied to event data for semantic segmentation and classification. Our work shows that such a strategy does concurrently preserve the accuracy of event data processing and greatly reduces the amount of data needed for the task, at least with the current software implementation. Further work will implement the hardware neuromorphic foveation feedback process by concurrently using the foveated DVS presented in [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF] and the neuromorphic chip SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF] to record foveated event data in an embedded fashion.

1.4.2 How can we exploit the relevance of the spiking neuron to process the specific chronology of event data ?

We identified multiple computer vision tasks answering this problem while falling within the spectrum of APROVIS3D, the results of which are detailed below.

Coastline detection

We participated in implementing this application, i.e. the main use case to be addressed within the APROVIS3D project, thanks to three approaches.

The first one corresponds to ANN-to-SNN learning and is carried out by a dedicated APRO-VIS3D workgroup composed of NTUA, INT and UCA collaborators. NTUA records event data and RGB videos of the sea and the coastline ground, separated and mixed, using cameras embedded on a UAV. Once the data is acquired, UCA verifies the provided videos' quality and translates the data into a universal format. This modified dataset is then forwarded to INT: having implemented the Hierarchy Of event-based Time-Surfaces (HOTS) classifier [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] in previous work, they update this architecture to differentiate between sea and ground patches efficiently. Once correctly adapted to this task, UCA will implement the HOTS network with the hyperparameters learned during training on the neuromorphic hardware SpiNN-3. The final topology will finally be embedded in the UAV set up by NTUA. At UCA, we handle the translation of data between different file formats (from AEDAT2 to NPY) as well as the verification of the quality of the data, as the embedded DVS required multiple adjustment steps to provide a usable dataset.

The second approach corresponds to delay learning, i.e. an SNN learning rule which, instead of changing the synaptic weight, changes the delay of the electrical spike journey in the presynaptic neuron's axon [START_REF] Hüning | Synaptic delay learning in pulse-coupled neurons[END_REF]. We aim to implement a delay learning 1.4 -1.4.2 How can we exploit the relevance of the spiking neuron to process the specific chronology of event data ? 13 mechanism to classify between spatiotemporal patterns in an event scene, mainly applied to our application task of coastline detection. Indeed, depending on whether the events recorded correspond to the ground or the sea, their movements are quite different: the events corresponding to the ground are generated by the sensor's ego-motion [START_REF] Chapel | Moving objects detection with a moving camera: A comprehensive review[END_REF] while the movement of the events corresponding to the sea is the sum between the movement generated by the ego-motion (vertical motion) and by the waves (mostly a motion parallel to the coastline). To this end, we extended the bio-plausible unsupervised delay learning model introduced in [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF] to a greater number of convolution layers and noisy and noiseless synthetic data. Additionally, we applied various metrics (accuracy, recall, F1-score, GINI index) to assess the training phase and ease the future adaptation to real-world data.

3D reconstruction Another use case of the APROVIS3D project targets the reconstruction of the 3D architecture of a visual scene from two stereo event streams recorded by two DVS interfaced to a neuromorphic platform. Building on [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] model for neuromorphic 3D reconstruction, we embed the model on a SpiNN-3 by 1) downscaling the input data, 2) modifying the network's architecture, and 3) cropping the data. Once we experimentally validated our results on the toy datasets provided in [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF], we produced some first results on the real-world dataset DSEC (M. [START_REF] Gehrig | Dsec: A stereo event camera dataset for driving scenarios[END_REF].

Neuromorphic lip reading Finally, we distance ourselves from the APROVIS3D spectrum and propose an innovative application task that would strongly benefit from the SNN's intrinsic temporal processing and event data's crucial chronology: lip reading. In order to tackle this task, we recorded an event-based dataset of lip movement corresponding to various words. This dataset, only the second in state of the art, subsequently allowed us to develop a neuromorphic lip reading model classifying short sequences of events into word categories. This SNN architecture leverages the advantages of neuromorphic computing, making it suitable for real-time embedded scenarios. We experimentally validated this approach ; we demonstrated that this model, the first to our knowledge to apply an SNN to event-based data of lip movement, produces promising results while keeping a relatively small size. This task is specifically relevant to our scientific problem because a contrario to the others, it could not have been solved using a small set of frames (consisting of accumulated events) but only thanks to the fine temporal resolution of events.

Overview of publications

All in all, the different articles and extended abstracts we produced during this PhD answer the different problems described above according to the following repartition: CHAPITRE 1 -Introduction Subject:

How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for any embedded data processing ?

Manuscript outline

This manuscript is organised as follows: Chapter 1 introduces the thesis by discussing its motivations, scientific context and various objectives. The state-of-the-art is distributed between the Chapters 2, 3, 4 and 5. SNNs and event cameras are introduced, and their history, mechanism and applications are described in detail respectively in Chapter 2 and 3. The benefits of the combined use of embedded SNNs and event cameras are amply demonstrated and documented in Chapter 4. Finally, Chapter 5 develops the various concepts from neuroscience, which potential or achieved application to computer science brings a significant performance boost. Chapters 6, 7, 8 and 9 constitute the contributions part. Our first issue concerning the trade-off between quantity and quality is addressed in 6, 7 and 8, which respectively present the design, implementation and achieved results of event data reduction, salient detection of OoIs by event density and neuromorphic foveation. Chapter 9 tackles the issue of drawing benefits from SNNs temporality to handle the chronology of event data by describing the various embedded applications and achieved results. Finally, Chapter 10 concludes this work and summarises the results achieved during this PhD thesis.

Notations

Spiking neural networks

This chapter introduces the concept of spiking neural networks, a type of artificial neural network closer to biology than traditional artificial networks. Indeed, spiking neural networks receive and process information in the form of spike trains, meaning as a nonmonotonous sequence of activations. Therefore, they make for a suitable candidate for efficiently processing incoming event patterns measured by silicon retinas.

-Neuronal dynamics 21

SNNs are a new type of ANN closer to biology than traditional artificial networks, also known as the third generation of neural network model [START_REF] Maass | Networks of spiking neurons: The third generation of neural network models[END_REF]. In accordance with this high bio-inspiration, they seek to mimic the dynamics of neural membrane and action potentials over time [START_REF] Paugam-Moisy | Computing with Spiking Neuron Networks[END_REF]. Additionally, whereas information was precedently encoded and processed as multi-bit numbers, SNN encodes the signal using sequences of spikes (or activations) with varying weights, delays and frequencies. SNNs receive and process information in the form of spike trains, meaning as a non-monotonous sequence of activations, as represented in Fig. 2.1a. Therefore, they make for a suitable candidate for efficient processing and classifying of incoming event patterns measured by silicon retinas. In biology, the term "neuron" designates one type of the cells forming the nervous system. An adult human brain comprises roughly 85 billion neurons, each 0.01 -0.05mm in diameter and dedicated to "sensing the environment, communicating the perceived changes and commanding the body's response to these sensations" [START_REF] Bear | The Human Eye[END_REF]. As seen in Fig. 2.2, each neuron is composed of the soma (i.e. the body of the neural cell, around 20µm in diameter and also called perikarion) and two types of neurites, the axon and the dendrites.

The axon transfers information over long distances in the nervous system, ranging from 1mm to over a meter long and with a diameter varying between 1 and 25µm in the human brain. We'll develop on the impact of both those factors on the speed of information transmission in Section 5.3.1. At most 2mm long and forming a "dendritic tree", dendrites extend from the soma and act as the "antennae" by connecting with the axons of other neurons to receive their incoming signals. The connection point between a first neuron's axon and a second neuron's dendrite (or, in certain occasions, soma) is called a synapse, derived from the Greek "to fasten together". Each dendrite of each neuron is covered with thousands of synapses.

In order to transit from the presynaptic to the postsynaptic neuron, the information is transmitted from the presynaptic neuron's soma to the synapse as a train of electrical impulses, i.e. "action potentials", that triggers the delivery of neurotransmitters from the pre-to the postsynaptic neuron which finally translates the received information back into an electro-chemical format. These are mediated by neurotransmitters, which are chemical substances endogenous to the nervous systems and responsible for the transmission of various information ; the human body contains more than 40 different types of neurotransmitters, including acetylcholine, norepinephrine, dopamine, gamma-aminobutyric acid and serotonin [START_REF] Vasković | Neurotransmitters. Kenhub[END_REF].

Scientists have developed various classification mechanisms to distinguish between different types of neurons according to their number of neurites, the shape of their dendritic tree, the length of their axon, the organs with which they connect (either sensory surfaces of the body, muscles or other neurons) or the differences in gene expression.

Neurons are not the only cells present in the nervous system ; another broad category is the glial cells, whose representatives are just as numerous as neurons. Also known as the "sleeping giants of neuroscience", glia "contribute to brain function mainly by insulating, supporting and nourishing neighbouring neurons" [START_REF] Bear | The Human Eye[END_REF]. Without it, the brain would be unable to perform the vast array of tasks it is responsible for.

Many different types of glial cells exist, of which the most represented are the astrocytes. Its role is not yet entirely clear, but it is believed to regulate the neurites' growth by controlling the chemical content of the extracellular space. Oligodendrocytes and Schwann cells compose the myelinating glia, i.e. responsible for the concentric wrapping of myelin, a thin protein sheet interspersed with lipid layers, around the axon. Scientific experiments have proven its essential role in information transmission and facilitation of both the neural circuit function and the behavioural performance (see Section 5.3.1 for more details). Other, less represented glial cells also exist in the nervous system: we can cite the ependymal cells (cell migration the organism development) and microglia (phagocytosis).

Chemoelectric mechanism of the biological neuron

As stated above, the neuron transmits information along its axon as a train of "action potentials", electric impulses encoding data in their frequency, pattern and timing.

This mechanism is made possible by the chemical balance between the intracellular and extracellular environment surrounding the neurons. There is a high concentration of potassium within the neuron, alimented by a high concentration of sodium and calcium in the extracellular environment enabling the activity of the membrane sodium-potassium pumps. This disequilibrium leads chemically to the creation of an outward potassium movement thanks to membrane potassium channels, leaving the inside of the neuron negatively charged.

Due to this difference in electrical charge, the neuronal membrane has a resting potential of -65mV . When the neuron is activated, its extracellular environment becomes positively charged relative to the extracellular one during 1ms at most: this corresponds to the depolarisation or integration. The action potential, also called "nerve impulse", "discharge", or "spike", takes place only if the membrane potential reaches a specific value called "threshold" during the depolarisation phase. In this case, the neuron enters a phase of hyperpolarisation where it undershoots below the resting value and cannot be activated again before increasing back to the resting value. Otherwise, the membrane potential will slowly decrease back to its resting value. This behaviour is presented in Fig. 2.1b.

Behaviour of a spiking neuron

As seen in Fig. 2.1b, spikes are emitted by each neuron depending on its membrane potential, which increases according to the input activation (much like the biological membrane potential increases when the biological neuron is electrically activated) and decreases over time in the absence of stimulation (thus simulating the leak observed in the biological neurons) (Paugam-Moisy & Bohte, 2012). Once the membrane potential u crosses the threshold θ with a positive slope, a spike is produced, and the membrane potential is reset at u reset . This is coherent with a biological neuron's behaviour when an action potential: those two steps correspond respectively to the neuron's depolarisation (or overshoot) and hyperpolarisation (or undershoot) as described in Section 2.1.1.2.

By definition, a spiking neuron follows a model based on parameters describing its internal state and its reaction to the input current with more or less faithful biological realism. A complete review of those models is presented in (E. M. [START_REF] Izhikevich | Which model to use for cortical spiking neurons ?[END_REF]) (see Fig. 2.3), but we describe the most common ones below. hikevich, 2004). The x-axis label "# of FLOPS" is an approximate number of floating point operations (addition, multiplication, etc.) needed to simulate the model during a 1 ms period.

Hodgkin-Huxley model

Historically the first spiking neuron model, the Hodgkin-Huxley model was developed in 1952 [START_REF] Hodgkin | Currents carried by sodium and potassium ions through the membrane of the giant axon of loligo[END_REF] and describes the neuronal dynamic by its variations of ionic concentrations in the intra and extracellular environments. It is part of the so-called "conductancebased" models and has been "successfully compared to numerous data from biological experiments on the giant axon of the squid" [START_REF] Paugam-Moisy | Computing with Spiking Neuron Networks[END_REF]. Its high degree of realism makes it far too complex to be of any real use in SNN simulation.

Integrate-and-Fire models

Drawing inspiration from Hodgkin-Huxley, the Integrate-and-Fire (IF) model places itself at a less precise level of detail to be more computationally efficient [START_REF] Lapicque | Recherches quantitatives sur l'excitation electrique des nerfs traitee comme une polarization[END_REF]. It integrates the spikes and increases the membrane potential u only at the time of the input, according to Eq. 2.1:

C d dt U = - u(t) -u rest R + I(t) (2.1)
The spike firing time t f is defined in the following threshold crossing equation:

u(t f ) = θ u ′ (t f ) > 0 ⇒ u(t f ) = u reset (2.
2)

The main difference with Hodgkin-Huxley is the simplification of the action potential processing. While the latter considers the shape of the action potentials, the IF (and LIF described below) consider each spike as a uniform event defined only by its timing.

Leaky Integrate-and-Fire model

The Leaky Integrate-and-Fire (LIF) model varies from its inspiration IF by adding a leaky behaviour: if incoming spikes activate the neuron, the membrane potential increases. If it does not reach the threshold, it decreases slowly back to its resting value when the input stops (leak). If the membrane potential overcomes a threshold, an output spike is produced, and the membrane potential is reset.

The dynamics of the LIF neuron's membrane potential u are described by the equations 2.3:

τ m du dt = u rest -u(t) + RI(t) (2.3)
where τ m is the membrane's time constant and I the input current modulated by a resistance R. Without any input current, the membrane potential is at rest and is of value u rest . When activated, it increases according to the input current. Moreover, at each timestep a slow decrease towards u rest is driven by the time constant τ m , thus modeling the voltage leakage. The firing time t f is defined by the threshold crossing Equation 2.2. While implementing SNN simulations, the IF and LIF models are often preferred because of their simplicity, conferring an advantage on the issue of efficiency for computational resources. The Izhikevich model makes a good compromise between biophysical plausibility and computational cost: according to the authors, "using this model, one can simulate tens of thousands of spiking cortical neurons in real time (1ms resolution) using a desktop PC"(E. M. [START_REF] Izhikevich | Which model to use for cortical spiking neurons ?[END_REF].

Spike Response Model

Defined by Gerstner in 1993, the Spike Response Model is a phenomenological model of the neuron based on the occurrence of spike emissions, which makes it more intuitive and more straightforward to implement [START_REF] Gerstner | Why spikes ? hebbian learning and retrieval of time-resolved excitation patterns[END_REF][START_REF] Paugam-Moisy | Computing with Spiking Neuron Networks[END_REF]. It generalises the IF model and its implementation abandons coupled differential equations (as used by IF, LIF and Izhikevich) in favour of the use of kernels.

Parameters

Overall, the computational implementation of a spiking neuron requires setting different parameters influencing the network performance. The list of parameters might vary between different models ; below are described most of the LIF parameters and their impact on neuron behaviour.

Neuronal parameters

This section brings together the parameters influencing the neuron directly.

Membrane potential Three parameters, expressed in mV , define the membrane potential:

-the resting membrane potential u rest , corresponding to the value at rest (around 65mV in the biological neuron) ; -the reset potential u reset , corresponding to the value reached by the membrane potential during the hyperpolarisation phase (see Fig. 2.1b) ; -the threshold θ, setting the potential limit before a spike is emitted.

Membrane time constant

The membrane time constant τ m corresponds to the "amount of time (in ms) it takes for the change in potential to reach 63% (i.e. 1 -exp -1 %) of its final value" [START_REF] Byrne | Chapter 3: Propagation of the action potentia[END_REF]. In other words, the higher the value, the longer it will take to finish its depolarisation phase, and the later the spike will be produced from the incoming excitation, and vice versa.

Refractory period Expressed in ms, the refractory period τ refrac is the time needed by the membrane after the hyperpolarisation phase to increase from the reset potential back to the resting potential, i.e. the amount of time during the neuron cannot be excited after it produced a spike.

Synaptic parameters

This section covers the parameters influencing the synapse.

Weight The weight ω defines the importance of the influence of a synapse activity on the postsynaptic neuron. In traditional ANN, the input data is multiplied by it within the hidden layer. In SNN, it increases the activation or inhibition of the post-synaptic neuron following the activation of the presynaptic neuron. It can be assimilated to the post-synaptic transmitters released in the biological synapse: the higher the value, the more important the effect of the information transmitted.

Delay The delay δ corresponds to the amount of time it takes for the information to leave the presynaptic neuron's soma and reach the post-synaptic soma. Biologically speaking, this parameter corresponds to the combination of two elements: the dendritic delay and the axonal delay. In addition, it can be learned using a variety of biological mechanisms and is at the origin of many neuronal behaviours, all of which are described in more detail in Section 5.3.1. While many may consider that this parameter concerns the neuron, we decided to include it in the "synaptic parameters" as it influences the synaptic connection between two neurons, and most simulation libraries implement it at the synapse level.

Excitatory and inhibitory decay time

The decay time simulates synaptic fatigue, a biological phenomenon taking place when the synapse is too highly activated as a way to regulate the nervous system activity physiologically. It results in the temporary inhibition of neuron activity and influences either excitatory or inhibitory synapses, depending on its type.

Network architecture

An SNN is constructed using populations of neurons linked together with connections with respect to certain rules and specific architecture. A population or layer reunites neurons in a unit that often shares the function and the initial parameters. The neurons can be arranged either linearly or in 2D or 3D space (depending on the simulator used), allowing for the implementation of distance-dependant rules.

The connections can be either excitatory, meaning that they activate the post-synaptic neuron ; or inhibitory, indicating that the post-synaptic neuron is inhibited. The two most common types of connective patterns are One-To-One, meaning that each neuron of the presynaptic population is connected to only one neuron of the postsynaptic population at the same index in the layer ; and All-To-All, meaning that each presynaptic neuron is connected to all the postsynaptic neurons.

Information encoding

By definition, SNNs use time as a basis for information coding. Indeed it has been experimentally demonstrated that biological neurons transmit information with impressive efficiency: for example, a human being requires only 100-150ms to process complex visual stimuli (S. J. [START_REF] Thorpe | Speed of processing in the human visual system[END_REF]. Scientists used to theorise that the information was encoded using a Poisson-like spike rate ; however, the visual system is composed of multiple layers firing at an average rate of 10ms, too slowly to allow for rate coding. Instead, it seems more likely that the information is encoded using the precise timing of the spikes.

Thorpe defined in 2001 various bio-plausible encoding mechanisms for SNN relying on time (S. [START_REF] Thorpe | Spike-based strategies for rapid processing[END_REF]. Among them can be found the latency coding, where the information is transmitted in the time it took for the spike to reach the postsynaptic neuron after a stimulus ; and the rank order coding, where the information is encoded in the order in which the spikes reached the postsynaptic layer. A visual example highlighting the importance of the choice of information encoding is presented in Fig. 2.4.

Furber later introduced an additional temporal encoding mechanism: N of M coding counts the input spikes using a feedback inhibition circuit and blocks their spiking once a certain number has been reached, thus implementing a k-WTA and allowing for the detection of specific combinations of spikes (S. B. [START_REF] Furber | Sparse distributed memory using n-of-m codes[END_REF]. The implications of these different strategies on neuromorphic circuits are discussed by [START_REF] Panzeri | Constraints on the design of neuromorphic circuits set by the properties of neural population codes[END_REF].

Dynamic adaptation rules

An SNN can be applied to various tasks (some examples can be found below in Section 2.3) that can be achieved with and without learning. In the second case, it relies on intrinsic dynamic behaviours and specific network architectures.

A first adaption mechanism worth mentioning is the WTA, observed in the cortical architecture, is a first adaption mechanism worth mentioning. This mechanism allows for the selective activation of a neuron, enabled by a pattern of lateral inhibition retroactively regulating the layer. It is implemented using an All-To-All inhibitory connection, without autapses (self-connections) and where the input and output layers are the same. Each neuron activation leads to the inhibition of the others to prevent too many efferent neurons from receiving too many spikes from a particular afferent region, thus ensuring that different neurons code different inputs [START_REF] Rolls | A model of invariant object recognition in the visual system: Learning rules, activation functions, lateral inhibition, and information-based performance measures[END_REF]. In the brain, lateral inhibition leads to WTA situations. Yet, other forms exist, such as intergroup WTA competition (instead of intra-group), leading to a cohort of winner-neurons rather than a single one [START_REF] Delorme | Network of integrate-andfire neurons using rank order coding b: spike timing dependant plasticity and emergence of orientation selectivity[END_REF]. Despite WTA's core purpose being enhanced selectivity, it also increases feature sparsity [START_REF] Falez | Unsupervised visual feature learning with spike-timing-dependent plasticity: How far are we from traditional feature learning approaches ?[END_REF] and maintains synaptic weights within bounds [START_REF] Masquelier | Unsupervised learning of visual features through spike timing dependent plasticity[END_REF]. An example of such a mechanism is provided in the model described in Section 7.2.2 and Fig. 7.2.

CHAPITRE 2 -Spiking neural networks

Homeostasis makes a second interesting adaptation mechanism. Globally speaking, we call "homeostasis" any mechanism that allows a system to regulate a key factor to maintain its beneficial value, guaranteeing flexibility. Applied to SNN, this mechanism is "crucial in providing a more efficient solution to the emergence of independent components" [START_REF] Perrinet | Role of homeostasis in learning sparse representations[END_REF]. It corresponds to the dynamic adaptation of neuronal thresholds often applied to reducing the high spike frequency induced by complex patterns and networks, as observed in the nervous system (W. [START_REF] Zhang | The other side of the engram: experience-driven changes in neuronal intrinsic excitability[END_REF][START_REF] Perrinet | Sparse spike coding in an asynchronous feed-forward multi-layer neural network using matching pursuit[END_REF]).

Learning rules

In biology, learning is "achieved via local plasticity mechanisms that operate at various spatial and temporal scales" [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF]. SNN can learn patterns and specialise for specific tasks by updating their parameters at each timestep until a specific aim is achieved. The modified parameter is more often than not the weight ; however, other parameters can also be the subject of learning, such as the synaptic delay (see Section 5.3), the membrane time constant (Fang, Yu, Chen, Masquelier, et al., 2021 ;[START_REF] Rathi | Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization[END_REF], the neuronal threshold [START_REF] Rathi | Diet-snn: A low-latency spiking neural network with direct input encoding and leakage and threshold optimization[END_REF][START_REF] Zhang | Spiking transformers for event-based single object tracking[END_REF], or even the membrane potential which distribution is "rectified" into a Gaussian (Y.-Z. Guo et al., 2022 ;[START_REF] Wang | Membrane potential distribution adjustment and parametric surrogate gradient in spiking neural networks[END_REF].

The strategies adopted to perform learning with SNN can be differentiated into various groups: conversion-based versus spike-based [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF], online versus offline (J. [START_REF] Wang | Online versus offline learning for spiking neural networks: A review and new strategies[END_REF], on-chip versus off-chip [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF], supervised versus unsupervised [START_REF] Paugam-Moisy | Computing with Spiking Neuron Networks[END_REF].

Conversion-based versus spike-based. [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF] proposes to segregate the SNN learning mechanisms between conversion-based and spike-based approaches. The first one corresponds to the conversion of a set of parameters trained by an ANN into parameters understandable by an SNN with a similar architecture ; this approach produces the best results on large-scale SNN (for example, in the context of ImageNet classification [START_REF] Sengupta | Going deeper in spiking neural networks: VGG and residual architectures[END_REF]). However, it is to be noted that this method leads to a significantly large inference time, so an increased latency and decreased energy efficiency.

The second spike-based approach is more biologically plausible because of its exploitation of the intrinsic temporal aspect of SNNs. Two research directions have been adopted: unsupervised (trained without labelled data) and supervised (trained with labelled data) learning. The first one uses local STDP-based learning rules, where the STDP is a bio-inspired unsupervised rule reinforcing the synaptic connections between neurons activating closely in time. This mechanism is heavily inspired by Hebb's rule: "cells that fire together wire together" [START_REF] Hebb | The organization of behavior: A neuropsychological theory[END_REF]. It allows for the selective selection of repeating patterns of input spikes [START_REF] Delorme | Network of integrate-andfire neurons using rank order coding b: spike timing dependant plasticity and emergence of orientation selectivity[END_REF]Perrinet & Samuelides, 2002 ;[START_REF] Masquelier | Unsupervised learning of visual features through spike timing dependent plasticity[END_REF] and is suitable for energy-efficient on-chip implementation (see below). However, it is difficult to adapt to large-scale SNN due to the "vanishing forward-spike propagation", i.e. the decreased firing rate in deeper layers, and current research are trying to solve this issue using feedback-based learning methods [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF].

Initiated with the early ReSuMe [START_REF] Ponulak | Supervised learning in spiking neural networks with resume: Sequence learning, classification, and spike shifting[END_REF] and tempotron [START_REF] Gütig | The tempotron: a neuron that learns spike-timing based decisions[END_REF], the research on the supervised spike-based approach has recently directed its attention to the implementation of a "global backpropagation-like spike-based error gradient descent to enable supervised learning in multi-layer SNNs" [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF]. Recent works in this vein include SpikeProp [START_REF] Bohté | Error-backpropagation in temporally encoded networks of spiking neurons[END_REF] and variants, which implement a differentiable approximate function for SNN using a specific output spike train as the target ; others perform stochastic gradient-descent on membrane potentials [START_REF] Lee | Training deep spiking neural networks using backpropagation[END_REF][START_REF] Mostafa | Supervised learning based on temporal coding in spiking neural networks[END_REF]. Even though promising, these supervised methods have not been able yet to overcome conversion-based approaches.

Offline versus online learning In (J. [START_REF] Wang | Online versus offline learning for spiking neural networks: A review and new strategies[END_REF], the authors underline the segregation of SNN learning approaches between offline and online learning and the lack of representation of the latter despite its suitability for large datasets and non-stationary tasks. The offline learning method updates the trained parameters only once the whole dataset has been presented ; new data cannot be added without retraining the whole network. It is thus quite fast but subject to local minima issues. Among those methods can be found both supervised -SpikeProp [START_REF] Bohté | Error-backpropagation in temporally encoded networks of spiking neurons[END_REF], Synfire Chains (E. [START_REF] Izhikevich | Polychronization: Computation with spikes[END_REF] -or unsupervised -STDP [START_REF] Masquelier | Unsupervised learning of visual features through spike timing dependent plasticity[END_REF] and Hebbian learning mechanisms.

However, online learning updates the network parameters at each new training sample to minimise the error per sample. It is adapted to continuous environmental changes due to its "always-on" approach and is thus significantly more bio-coherent. Few examples of such a mechanism exist: we can cite the four-layer hierarchical model for face recognition introduced in [START_REF] Wysoski | On-line learning with structural adaptation in a network of spiking neurons for visual pattern recognition[END_REF]) and the gustatory model for taste recognition based on an evolving learning algorithm in [START_REF] Soltic | Evolving spiking neural networks for taste recognition[END_REF], which both lack adaptive fine-tuning of the learning parameters as well as suffer from an excessive training speed. It has been more recently applied to robotics with the self-adaptation model for optimal navigation in dynamic environments implemented in [START_REF] Alnajjar | A spiking neural network with dynamic memory for a real autonomous mobile robot in dynamic environment[END_REF], and to event-based object recognition in [START_REF] Grimaldi | A robust eventdriven approach to always-on object recognition[END_REF]. Online learning is specifically relevant in the current research field since it overcomes the issue of generalisability, i.e. the mismatch between the training dataset and the real world, by learning continuously and correcting errors as they unfold [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF].

Off-chip versus on-chip learning Finally, the training phase can either occur off or on a synaptic circuit [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF]. In the first case, the SNN is trained on a separate computer with significant memory and computational power, then deployed on neuromorphic hardware (see Section 2.2. This approach generally achieves the best inference accuracy on practical tasks since it performs well on large available datasets and often makes use of convolutional architectures [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF].

On-chip learning consists primarily of learning dynamics taking inspiration from computational neuroscience, such as the STDP variants or, more recently, the 3F rules [START_REF] Ercsey-Ravasz | A predictive network model of cerebral cortical connectivity based on a distance rule[END_REF], and allow for online learning. However, this approach requires a large amount of memory and routing resources, thus limiting the size of the architectures ; promising hardware alternatives are currently being developed to allow for the storage and computation to take place at the same place [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF]. Overall, this approach helps overcome the challenge posed by the high power requirement of state-of-the-art AI technologies [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF].

A third option can be found in hybrid learning, also known as "chip-in-the-loop", combining both approaches while allowing for online learning.

A detailed state-of-the-art of recent SNN models with their corresponding learning mechanism can be found in Tables 2 and3 from [START_REF] Yamazaki | Spiking neural networks and their applications: A review[END_REF]. It is interesting to CHAPITRE 2 -Spiking neural networks note that the increasing size of SNN induces many challenges, amplified compared to those faced with smaller-scale systems: controlling the neuronal activity with homeostasis, compensating local failures with adaptation mechanisms, the application of unsupervised learning mechanisms and online learning, and synaptogenesis and neurogenesis [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF].

Neuromorphic hardware

As of today, SNN simulations on traditional von Neuman computers require a substantial simulation time ; indeed, "their simulation requires the sequential computation of several time steps, and simulation time thus scales with the deepness of the network as well as the time resolution of the simulation" [START_REF] Bouvier | Spiking neural networks hardware implementations and challenges: a survey[END_REF].

However, researchers have recently designed hardware specifically adapted to the fast and low power simulations of SNNs, using electronic circuits that faithfully reproduce the dynamics of neurons in real time [START_REF] Sorbaro | Optimizing the energy consumption of spiking neural networks for neuromorphic applications[END_REF]. This novel technology is called "neuromorphic hardware" and can be divided into three different designs [START_REF] Javanshir | Advancements in Algorithms and Neuromorphic Hardware for Spiking Neural Networks[END_REF]:

-analog designs model neurons using physical processes leading to a very efficient implementation of operations with low power consumption thanks to the natural dynamics of the system ; -digital designs model neurons with bits, allowing for a "controllable and guaranteed precision of variables" [START_REF] Javanshir | Advancements in Algorithms and Neuromorphic Hardware for Spiking Neural Networks[END_REF] but with a significant impact on memory requirements and energy consumption ; -mixed designs take the best of both analog and digital, for example implementing the spike communication digitally while modeling neurons with analog elements. Those designs are often implemented on Application-Specific Integrated Circuits (ASICs) or Field Programmable Gate Array (FPGA) devices: while most known neuromorphic boards are developed on ASICs (such as the digital SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF], TrueNorth [START_REF] Akopyan | Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip[END_REF] and Loihi [START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF], or mixed BrainScaleS [START_REF] Schemmel | A waferscale neuromorphic hardware system for large-scale neural modeling[END_REF] and Neurogrid [START_REF] Benjamin | Neurogrid: A mixed-analog-digital multichip system for large-scale neural simulations[END_REF]), FPGAs are gaining momentum as "a promising candidate for acceleration of SNN, achieving better speed-up than CPUs and less energy consumption than graphics processing units (GPUs)" [START_REF] Javanshir | Advancements in Algorithms and Neuromorphic Hardware for Spiking Neural Networks[END_REF]. We can mention the recent applications of SNNs on FPGAs to edge-detection [START_REF] Qi | Fpga design of a multicore neuromorphic processing system[END_REF], bio-mimetic pattern generation [START_REF] Ambroise | Real-time biomimetic central pattern generators in an fpga for hybrid experiments[END_REF] and event-driven vision processing [START_REF] Yousefzadeh | Fast pipeline 128×128 pixel spiking convolution core for event-driven vision processing in fpgas[END_REF].

We present in this section first a short review of CPU and GPU simulators, then a selection of neuromorphic hardware dedicated to SNN simulation. A more comprehensive review of existing neuromorphic hardware and their particularities can be found in [START_REF] Bouvier | Spiking neural networks hardware implementations and challenges: a survey[END_REF], [START_REF] Roy | Towards spike-based machine intelligence with neuromorphic computing[END_REF] and (J. Wang, 2022).

Simulation on CPU/GPU

SNN simulators can be divided into two approaches, depending on their synchronicity [START_REF] Javanshir | Advancements in Algorithms and Neuromorphic Hardware for Spiking Neural Networks[END_REF]. The "event-driven" simulators are asynchronous, i.e. it only updates the neuron when there is an incoming spike. They allow for a high operation speed and good bio-plausibility at the cost of an increased implementation complexity, thus are more suitable for sparsely activated networks. The "clock-driven" simulators however update the neuron synchronously "at every tick of a clock" [START_REF] Javanshir | Advancements in Algorithms and Neuromorphic Hardware for Spiking Neural Networks[END_REF] thus allowing for parallel computing and an easier implementation on CPUs (small networks) and GPUS (greater networks with higher mathematical complexity).

We adapt and complete in Table 2.1 below the list of available software frameworks for SNNs simulation from [START_REF] Yamazaki | Spiking neural networks and their applications: A review[END_REF], with the additional information provided in (Pehle & Pedersen, 2021). (Fang, Yu, Chen, Masquelier, et al., 2021) Python SNN simulator extending PyTorch, using stateful neurons Clock-driven Yes Yes

Human Brain Project's SpiNNaker

SpiNNaker is a neuromorphic hardware platform developed at the University of Manchester under the Flagship Human Brain Project. Each SpiNNaker chip communicates and process spike events: it contains a router that delivers each incoming spike to the corresponding neural processor where the spike should arrive. The SpiNNaker computation is based on the availability of 18 parallel ARM processor cores per chip. Each ARM core is tightly coupled to local memory (TCM), where the neuron parameters and states are stored. Additionally, each chip is packaged with an external memory communicated through a DMA (direct memory access) controller allowing direct data transference between the internal and external memory. This memory stores the synaptic connectivity between the different neural populations allocated in the chip.

Each SpiNNaker core can simulate up to 256 spiking neurons -the initial aim was to simulate 1,000 neurons, but it proved to be a goal too optimistic. A SpiNNaker chip can thus allocate about 4,500 neurons (S. [START_REF] Furber | Spinnaker -a spiking neural network architecture[END_REF]. Additionally, each neuron can receive 1,000 incoming synapses at most, hence a SpiNNaker chip can simulate up to 4,500,000 synapses.

Two versions of SpiNNaker boards are available. The first SpiNN-3 board contains 4 SpiN-Naker chips, i.e. 72 cores. However, only 63 cores are available out of this total of 72, as the remaining are used for OS, monitoring, data I/O, etc. A SpiNN-3 board can thus implement SNN with a maximum of 18K neurons. Currently available SpiNN-5 boards contain 48 chips, having the capacity for 195K neurons.

SpiNNaker allows the description of the SNN system to be allocated in the hardware using sPyNNaker, a library developed for SpiNNaker in the PyNN language (University of Manchester, 2021). The network and corresponding connectivity described using the PyNN SpiNNaker library are mapped to the corresponding hardware. A software tool partitions and maps the defined network and connectivity into the physical cores and generates the corresponding routing and synaptic tables.

Intel's Loihi

Intel Loihi is a recent neuromorphic research chip [START_REF] Davies | Advancing neuromorphic computing with loihi: A survey of results and outlook[END_REF] that can be used as a processing platform. In particular, the Kapoho Bay platform comes in a universal serial bus (USB) form factor with 2 Loihi chips with a total of 256 neuro-cores able to simulate 262.144 neurons and up to 260Mn synapses. It can be easily interfaced for live communication with a host system by programming the three embedded x86 processors used for monitoring and I/O spike management embedded on the Loihi chip.

Center for Project-Based Learning's Kraken

The Kraken chip (Di [START_REF] Mauro | Kraken: A direct event/frame-based multi-sensor fusion soc for ultra-efficient visual processing in nano-uavs[END_REF] comprises three main subsystems composing a heterogeneous architecture. The first subsystem, called the Fabric Controller, is built around a 32bit RISC-V core that acts as the central control unit for the entire System-on-Chip. The Fabric Controller contains the main interconnection busses to the main L2 memory and the Advanced Peripheral Bus, which controls all the System-on-Chip peripherals. Additionally, the Fabric Controller hosts a complete set of on-chip peripherals and a power management unit.

The second subsystem is a RISC-V-based general-purpose accelerator known as the "Cluster", which hosts eight RISC-V cores enhanced with specialised Instruction Set Architecture exten-2.3 -Applications sions. The cluster also features a 128 KiB L1 Tightly Coupled Data Memory and a dedicated hardware block for fast event management, parallel thread dispatching, and synchronisation.

The third subsystem, the External Hardware Processing Engine, hosts two accelerators, including the Sparse Neural Engine neuromorphic accelerator and CUTIE, a ternary weight neural network accelerator. The accelerators operate on two independent clock domains and are programmed via memory-mapped register interfaces.

The chip features a vast set of IO peripherals that can generate interrupts depending on data transmission events. An autonomous IO subsystem called the µDMA hosts all the IO peripherals. It can be programmed to orchestrate data transfers from the peripherals to the L2 memory and vice versa. Kraken also has a power management scheme that allows the on/off switching of unused system parts to control the overall energy consumption.

The Kraken chip is mounted on an evaluation board incorporating external components essential to execute complete applications. A USB-C connector is used to power the board and for JTAG and UART data transfer. The three power domains of Kraken are supplied by buck converters that are individually runtime-configurable, permitting application-controlled DVFS. External memory includes a combined HyperFlash/HyperRAM chip and a quad-SPI flash memory chip that can store application code for standalone booting. Additional connectivity is provided by Arduino headers, and a Camera Parallel Interface ribbon connector and level shifters are present between each off-chip connector and Kraken's I/O pins.

Applications

SNN can be applied to multiple application tasks, ranging from computer vision to gustative and tactile classification models. In 2012, (Paugam-Moisy & Bohte, 2012) outlined a first review of existing applications of SNN to pattern recognition ; we can mention the tasks of speech processing, computer vision, robotics, motor control, and rehabilitation in medicine. In 2022, [START_REF] Yamazaki | Spiking neural networks and their applications: A review[END_REF] published an SNN review where they summarise recent applications of SNN in computer vision (Table 2) and robotics (Table 3). Additionally, the same year, [START_REF] Christensen | roadmap on neuromorphic computing and engineering[END_REF] presented a "roadmap on neuromorphic computing and engineering" with a complete section dedicated to SNN applications, including robotics, self-driving cars, audition, olfaction and chemosensation.

It is interesting to mention here the recent advances in deep SNNs: these networks aim to "bridge the gap between the big success of deep learning in AI applications and the promise of SNNs" [START_REF] Pfeiffer | Deep learning with spiking neurons: Opportunities and challenges[END_REF]. Indeed, since its invention, the application of deep learning methods allowed for significant improvements in "the state-of-the-art in speech recognition, visual object recognition, object detection and many other domains such as drug discovery and genomics" [START_REF] Lecun | Deep learning[END_REF]. Deep neural networks (DNNs) are composed of multiple layers (divided into one input, one output and at least two hidden ones) allowing for multiple levels of data representation and abstraction. Thanks to the feature discovery hierarchy, the different features are not designed by humans but acquired from data using non-linear activation functions with increasing complexity, abstraction and invariance [START_REF] Tavanaei | Deep learning in spiking neural networks[END_REF]. Even though DNNs strongly drew inspiration from neuroscientific models of cortical hierarchies, such as pattern recognition in the primate's brain, they lack fundamental bio-plausible elements compared to the brain, most notably the communication of information as spikes.

The introduction of deep SNNs therefore aims to overcome this issue. They seem particularly promising for processing event-based visual and auditory inputs while deployed on neuromorphic hardware [START_REF] Tavanaei | Deep learning in spiking neural networks[END_REF]. The SNN intrinsic behaviours should lead to fast propagation of salient information through multiple layers with low-power consumption [START_REF] Pfeiffer | Deep learning with spiking neurons: Opportunities and challenges[END_REF] -however, these promises are not yet fulfilled. Some notable limiting factors are the lack of spikeoriented training algorithms, benchmarks not suited to the bio-inspired aspect of SNNs (more suited to tasks based on continuous data perceived in the real world than frame-based datasets) and evaluation metrics measuring efficient real-world performance. To bypass this pitfall, recent works took a heightened interest in spike-based training methods and local learning rules in order to develop a neural network "as efficient and biologically plausible as SNNs but as powerful as DNNs in performing different tasks" [START_REF] Tavanaei | Deep learning in spiking neural networks[END_REF].

Of note, the recent application of surrogate gradient to SNNs, although distancing itself from biology, seems to offer a "particularly flexible and efficient method to overcome the aforementioned challenges" [START_REF] Neftci | Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[END_REF]. This strategy consists in applying to SNNs the supervised learning algorithm "back-propagation", popular in DL and which updates synaptic weights at each timestep in order to minimise the error, i.e. the distance between the current output and the desired one [START_REF] Masquelier | Back-propagation now works in spiking neural networks ![END_REF]. Applying back-propagation to SNNs is not trivial as it requires differentiable activation functions, which are inherently absent in SNNs. [START_REF] Neftci | Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[END_REF] propose the Surrogate Gradient-Learning, a global approach commonly acclaimed which approximates the LIF basis equations into a recurrent relation for the potential, thus allowing the application of back-propagation with the help of a "surrogate gradient". Thanks to this method's approximations, SNN models can maintain low firing rates while obtaining nearly state-of-theart accuracy. We can also mention the S4NN, a latency-based backpropagation for static stimuli which estimates the gradients of the loss with respect to all the weights but is constrained to bellow state-of-the-art performance due to its "at-most-one-spike-per-neuron" limitation [START_REF] Kheradpisheh | S4NN: temporal backpropagation for spiking neural networks with one spike per neuron[END_REF].

CHAPTER 3

Event-based cameras

In this chapter, we introduce the notion of event-based cameras. Also called silicon retinas, they represent a new kind of visual sensor that measures pixel-wise changes in brightness and accordingly outputs asynchronous events. This novel technology allows for a sparse, non-redundant and energy-efficient recording and storage of visual information, which has recently led to a substantial increase in its use for computer vision tasks, notably in robotics. With the recent advent of event cameras, we seek to use this unusual type of data as input to neural networks applied to selected vision tasks (scene segmentation, feature tracking, depth estimation, etc.). Additionally, event data streams are highly compatible with spiking neural networks due to the similarity in the information encoding ; therefore, these features make their combined use unavoidable for the future of real-time embedded vision systems.

-39

Conventional video cameras are based on the periodic acquisition of frames. Each frame is a static representation of the visual scene acquired by measuring the average light intensity during a certain sub-period, commonly known as exposure time. The intensity of each pixel is periodically measured (with a typical frame period of 20-30ms) and communicated regardless of whether it contains relevant information. Furthermore, objects moving at high speed relative to the frame time appear blurred in the image frame.

However, biological vision systems are not frame-based, but the transmitted information is coded as spikes. The idea of a novel bio-inspired event-based sensor, akin to a "silicon retina" [START_REF] Mahowald | The Silicon Retina[END_REF], has been developed since the 1990s. Pixels of such sensors emit spikes whenever relevant information is captured in the visual field. In the last two decades, different technological developments of spiking silicon retinas implementing spatial and temporal filtering have been published: retinas converting luminance to spike frequency, spatial contrast retinas, etc. They did not reach the maturity of commercial applications due to several limitations such as high fixed pattern noise, low fill factor, and complex circuitry resulting in low sensor resolution.

In 2008, Lichtsteiner, Posch and Delbruck presented the first complete design of a DVS (Lichtsteiner et al., 2008a): it implements a simplified retinal model where each pixel computes autonomously the relative time difference of the illumination received on its photosensor and responds only to temporal brightness changes in a scene with no consideration for colours, similar to the biological retina. When pixel illumination increases and its relative change exceeds a certain threshold, the pixel generates a positive ON output spike. Similarly, the pixel generates a negative OFF output spike if the illumination decreases and its relative change exceeds a certain negative threshold. DVS circuitry can be implemented with compact circuitry, which results in low fixed pattern noise and higher resolution sensors. Furthermore, DVS exhibit high temporal resolution (below 1 microsecond) and intrascene dynamic range as high as 120dB (Lichtsteiner et al., 2008a). Due to these features, these sensors have emerged as the first commercially available neuromorphic sensors and have overcome the integration density and mismatch limitation exhibited by previous neuromorphic retinas. Advanced megapixel DVS sensors have been developed (C. Li, Longinotti, Corradi, & Delbruck, 2019 ;[START_REF] Suh | A 1280×960 dynamic vision sensor with a 4.95-micro m pixel pitch and motion artifact minimization[END_REF][START_REF] Kubendran | A 256x256 6.3pj/pixel-event query-driven dynamic vision sensor with energy-conserving row-parallel event scanning[END_REF][START_REF] Guo | Live demonstration: A 768 × 640 pixels 200meps dynamic vision sensor[END_REF]) and new high-speed vision applications and systems based on DVS sensors are emerging.

The main advantages of such an event-based camera are [START_REF] Gallego | Event-based vision: A survey[END_REF]: -the high temporal resolution (microsecond time grain), thanks to which an event can be emitted on the timescale of microseconds and avoid motion blur ; -the high dynamic range (up to 120dB), which makes it possible to use them for extremely dim as well as under bright sun illumination ; -the high contrast range, which allows for highly contrasted images avoiding dazzling effects caused by sudden illumination changes, thus scenes with illumination expanding over 120dB can be sensed without suffering saturation ; -the low latency and asynchronicity enabled by the independence between each pixel ; -the absence of redundancy in the information transmitted, as compared to frame-based sensors (1,000 times fewer data produced by an event camera than in RGB videos [START_REF] Prophesee | What is event-based vision[END_REF]) ; -the low power consumption (10mW at the die level and less than 100mW for the whole embedded system [START_REF] Gallego | Event-based vision: A survey[END_REF]), following the model of biological retinas (3mW for the retina and less than 10mW for the whole eye [START_REF] Skorka | Toward a digital camera to rival the human eye[END_REF]) and substituting the biological photoreceptors with photodiodes in the electrical circuits.

By definition, a DVS generates sparse data: since each DVS pixel generates a string of spikes which respond to the temporal variation of the illumination, the pixel remains silent under a static background, saving communication and computation energy of the subsequent recognition network. This spatiotemporal filtering eliminates data redundancy over time and space ; this technology therefore allows for an energy-efficient recording and storage of data evolving over time and space. However, the increasing resolution of DVS prototypes (S. [START_REF] Chen | Live demonstration: Celex-v: A 1m pixel multi-mode event-based sensor[END_REF] has motivated the research on active reduction methods for DVS data to save communication, computation bandwidth, and energy.

Operation of an event camera

Biological retina

In biological retinas, photoreceptors sense the visual scene and lead to continuous and asynchronous spike generation whenever relevant information is detected, i.e. when high spatial or temporal contrast is caught in the scene [START_REF] Meister | The neural code of the retina[END_REF].

The human retina performs vision thanks to a wide array of photosensitive cells, of which around 100 million (the rods) are responsible for black-and-white vision and 4 million (the cones) for vision in colour [START_REF] Steffen | Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms[END_REF]. These photoreceptors make one of the three retinal layers presented in Fig. 3.1 ; the other two are the outer and inner plexiform layers, respectively containing bipolar cells and ganglion cells.

The bipolar cells can be divided into ON and OFF-types, depending on their response to photoreceptor activation linked to their glutamate receptors: an OFF cell is hyperpolarised, thus "turned off", when subjected to light whereas an ON cell is depolarised, i.e. "turned on' [START_REF] Bear | The Human Eye[END_REF]'. The receptive field of a bipolar cell is divided into its center, from which comes direct photoreceptor input, and its surround, receiving input from the horizontal cells (see Fig. 3.1) ; the activation of the center and the surround lead to different responses from the membrane potential. The bipolar cells are thus said to have "antagonistic center-surround receptive fields". A similar ON-OFF and center-surround organisation can be found in the ganglion cells [START_REF] Bear | The Human Eye[END_REF].

The bipolar and ganglion sensory cells lower the activity rate produced by the photoreceptors and received via the horizontal cells before transmitting the information through the optic nerve to be processed by the neural layers of the visual cortex. This spike-based coding of the visual scene results in high temporal resolution and low communication.

Silicon retina

Fig. 3.2 shows the conceptual schematic of an event camera pixel. A photosensor produces a current I ph proportional to the illumination. A trans-impedance logarithmic circuitry converts the detected current into a voltage V log = A T log(I ph ), which depends logarithmically on the photocurrent. This voltage is the input to a differencing capacitive amplifier so that the voltage difference

∆V diff = -C 1 /C 2 ∆V log = -C 1 /C 2 A T ∆I ph I ph (3.1)
Figure 3.1 -Human retina, as presented in [START_REF] Steffen | Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms[END_REF], reduced to essential layers for neuromorphic visual sensors: the photoreceptor layer, the outer plexiform layer including bipolar cells and the inner plexiform layer made up of ganglion cells. Additionally, horizontal cells and amacrine cells connect these layers. That way, voltage V diff is proportional to the relative temporal variation of the photocurrent. Voltage V dif f is compared with upper and lower voltage thresholds. Each time it goes over the upper (or below the lower) threshold, the pixel will generate an output ON (or OFF) event and voltage V dif f is reset to a resting value V R . Fig. 3.3 illustrates the resulting behaviour of the event camera pixel. The upper subfigure plots the variation of a pixel photocurrent over time, while the lower subfigure illustrates the generated ON and OFF output events. As can be observed, each time the photocurrent increases (or decreases) by a certain relative variation given by,

∆I ph I ph = C 2 (V t -V R ) C 1 A T (3.2)
the pixel generates an ON (OFF) output event.

Multiple event camera designs have been developed since the first bio-inspired sensor was built by Misha Mahowaldin in 1991. We can differentiate between three main architectures, whose characteristics are outlined in Table 2 of [START_REF] Steffen | Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms[END_REF]: the Dynamic Vision Sensor (DVS), the Asynchronous Time-based Image Sensor (ATIS), and the Dynamic and Active Pixel Vision Sensor (DAVIS). [START_REF] Gallego | Event-based vision: A survey[END_REF] presents a comprehensive comparison of existing commercial and prototype event cameras in their Table 1.

Event data representations

Formally, an event is a tuple (x, y, t, p) that indicates the pixel x, y where the event occurred, a timestamp t (with 10 -6 s time resolution), and a polarity p that indicates the direction of change: positive for brighter and negative for darker. However, event data is often preprocessed and transformed into alternative representations more adapted to given tasks [START_REF] Gallego | Event-based vision: A survey[END_REF]. These representations can be divided into two categories described below according to whether they retain temporal accuracy or not.

Representations retaining temporal accuracy

The four representations maintaining temporal accuracy are individual events (before any preprocessing), which is a format particularly suited for SNN handling [START_REF] Paredes-Valles | Unsupervised learning of a hierarchical spiking neural network for optical flow estimation: From events to global motion perception[END_REF][START_REF] Weikersdorfer | Event-based particle filtering for robot selflocalization[END_REF] ; the voxel grid, a 3D histogram of events where each voxel represents a particular pixel and time interval (L. [START_REF] Wang | Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks[END_REF][START_REF] Zhu | Unsupervised event-based learning of optical flow, depth, and egomotion[END_REF] ; and the 3D point set, akin to a graph where each event is treated as a point in 3D space [START_REF] Benosman | Event-based visual flow[END_REF][START_REF] Sekikawa | Eventnet: Asynchronous recursive event processing[END_REF].

Representations losing temporal accuracy

Many methods represent events by accumulating them over time, thus losing their temporal accuracy. Among them can be found the frame or 2D histogram, where the events are summed and/or accumulated into 2D grids [START_REF] Kogler | Bio-inspired stereo vision system with silicon retina imagers[END_REF][START_REF] Liu | Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors[END_REF][START_REF] Maqueda | Event-based vision meets deep learning on steering prediction for self-driving cars[END_REF] ; the time surface, a 2D map where each pixel stores the timestamp value of the last event at that pixel (Delbruck, 2008a ;[START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF][START_REF] Grimaldi | A robust event-driven approach to always-on object recognition[END_REF] ; the event packet, which gathers events in a spatiotemporal neighbourhood to process them together [START_REF] Reinbacher | Real-time panoramic tracking for event cameras[END_REF]Mueggler, Gallego, Rebecq, & Scaramuzza, 2018) ; reconstructed brightness images that are more motion-invariant than time surfaces [START_REF] Rebecq | High speed and high dynamic range video with an event camera[END_REF] ; and motion-compensated event image where the events are wrapped to a reference time, and their alignment is maximised [START_REF] Gallego | Accurate angular velocity estimation with an event camera[END_REF]Gallego, Rebecq, & Scaramuzza, 2018).

Tools for event data handling

As seen above, event cameras produce a novel kind of data whose novel format complicates the processing and application of methods traditionally used in computer vision. A particular com-3.3 -Applications and corresponding datasets 43 plication can be found in the wide array of file formats (AEDAT, CSV, NPY or NPZ, MAT, DAT, HDF5... and others presented in Tab. 3.1) and the possible incompatibility of specific formats and computer languages (for example, between AEDAT2 and Python).

Multiple tools were thus made available over time to handle event data, with various specifications described in (Robotics & Perception Group, 2020).

Among these, we were particularly interested in Tonic. This Python library for neuromorphic data handling has been created on an original idea dating back from the 2019 Telluride neuromorphic workshop [START_REF] Lenz | Tonic: event-based datasets and transformations[END_REF]. It facilitates dataset downloads and their conversion to different event representations (frames, time surfaces, voxel grids, etc.). It also proposes event transformation, such as denoising, event dropping, polarities merging, spatial and temporal jittering and downsampling.

Applications and corresponding datasets

The event cameras' particularities grant them advantages in many application cases compared to more traditional visual sensors [START_REF] Gallego | Event-based vision: A survey[END_REF]. Because of their operating mode, event-based cameras are helpful in tasks featuring movements, such as object tracking [START_REF] Glover | Event-driven ball detection and gaze fixation in clutter[END_REF], gesture recognition (Amir et al., 2017), optical flow estimation [START_REF] Orchard | A spiking neural network architecture for visual motion estimation[END_REF][START_REF] Paredes-Valles | Unsupervised learning of a hierarchical spiking neural network for optical flow estimation: From events to global motion perception[END_REF], Simultaneous Localisation and Mapping [START_REF] Kim | Real-time 3d reconstruction and 6-dof tracking with an event camera[END_REF][START_REF] Vidal | Ultimate slam ? combining events, images, and imu for robust visual slam in hdr and high-speed scenarios[END_REF], etc. Their high resolution and contrast range is also valuable for astronomical studies [START_REF] Cohen | Event-based sensing for space situational awareness[END_REF][START_REF] Chin | Star tracking using an event camera[END_REF]. Research on their combined use with traditional RGB cameras becomes more common, with applications on high-resolution image reconstruction (Z. [START_REF] Zhang | Image reconstruction from events. why learn it ?[END_REF][START_REF] Rebecq | High speed and high dynamic range video with an event camera[END_REF] and video deblurring (L. Zhang et al., 2021). A significant field of applications is robotics: autonomous vehicles, drones and other embedded systems facing limitations in terms of energy resources, memory, computational power, and communication bandwidth.

In 2020, [START_REF] Gallego | Event-based vision: A survey[END_REF]) presented a complete survey of event-based vision according to the tasks addressed. The authors keep an up-to-date list of existing event-based datasets and corresponding applications publicly available at (Robotics & Perception Group, 2020). We introduce in the paragraphs below the datasets most commonly used in current neuromorphic research and their corresponding applications. The last paragraph brings together 38 datasets (including the ones detailed previously) summarised in table form.

DVS 128 Gesture

Hand gesture recognition is a skill used daily in human society and is tightly integrated with verbal communication, hence the need for computational learning of such data. Since it relies heavily on temporal information (i.e. a static representation is not enough to solve this task), this task is well-suited for event-based computation. Building on this notion, Amir et al. presented at CVPR 2017 a complete hand gesture neuromorphic dataset called DVS128 Gesture (Amir et al., 2017), which has now become a standard benchmark in event data classification. To this end, 29 subjects were recorded performing 11 different hand gestures under three kinds of illumination conditions by a DVS128 camera. A total of about 133 samples are available for each gesture, each composed roughly of 400,000 events and of dimension 128 × 128 pixels, for a duration of approximately 6 seconds. The dataset is split into two sub-datasets to facilitate ML training: the train set consists of 80% of the recorded samples and the test set 20%, with a balanced distribution of the 11 gestures between them.

N-MNIST

The Neuromorphic-MNIST dataset is an event translation of the original MNIST dataset (for Mixed National Institute of Standards and Technology), which was presented in 1998 by Y. Le-Cun and which gathers size-normalised handwritten digits for pattern recognition [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]. To record this neuromorphic adaptation, Orchard et al. presented an average of 7,000 MNIST samples on a screen for each digit from 0 to 9 during 500 ms to an ATIS sensor of resolution 28 × 28 pixels mounted on a motorised unit, moving in saccades [START_REF] Orchard | Converting static image datasets to spiking neuromorphic datasets using saccades[END_REF]. This produces 80,000 neuromorphic samples, each roughly composed of 4,200 events. The set was split into train (88%) and test (12%) sub-sets for ML purposes. The authors showcase the advantages its small resolution brings, which is a "reduce[d] processing time, allowing for rapid testing and iteration of algorithms when prototyping new ideas".

N-MNIST is particularly heavy in events: according to the parameters presented in the above paragraphs, N-MNIST displays, on average, 10.71 events per pixel per second, whereas DVS128 Gesture only displays about 4.06 events per pixel per second.

DDD17

The DAVIS Driving Dataset 2017 (DDD17) [START_REF] Binas | DDD17: End-To-End DAVIS Driving Dataset[END_REF] contains 40 different driving sequences of event data captured by an event camera (around 447GB). However, since the original dataset provides only both grayscale images and event data without semantic segmentation labels, [START_REF] Alonso | EV-SegNet: Semantic Segmentation for Event-based Cameras[END_REF] provides segmentation labels they produced from 20 different sequence intervals taken from 6 of the original DDD17 sequences as well as a multi-channel representation of the corresponding events (normalised sum, mean and standard deviation for each polarity). One can extract the corresponding original events from DDD17 with the traditional < x, y, p, t > structure using DDD20 tools * . The resulting dataset is split into a training dataset consisting of 15,950 frames and a testing one consisting of 3,890 frames.

Ev-SegNet is a semantic segmentation model applied to DDD17 and built by [START_REF] Alonso | EV-SegNet: Semantic Segmentation for Event-based Cameras[END_REF], which outperforms all existing studies using event cameras in this kind of task. Semantic segmentation is a crucial task in visual scene understanding and corresponds to a visual classification problem which assigns a label, i.e. a given class, to each pixel in the image. This model is inspired by current state-of-the-art semantic segmentation CNNs, slightly adapted to use the event data encoding. As shown in Fig. 3.4, it consists of an encoder-decoder architecture: an encoder represented by the Xception model in which all the training is concentrated, and a light decoder connected to the encoder via skip connections to help deep neural architecture to avoid the vanishing gradient problem and also to make the fine-grained details learned in the encoder part used in the decoder to construct the initial image. Moreover, [START_REF] Alonso | EV-SegNet: Semantic Segmentation for Event-based Cameras[END_REF] uses an auxiliary loss which increases convergence speed.

The model takes as input six channels representing the count, mean and standard deviation of the normalised timestamps of events happening at each pixel within an interval of 50ms for Figure 3.4 -CNN architecture of Ev-SegNet, from [START_REF] Alonso | EV-SegNet: Semantic Segmentation for Event-based Cameras[END_REF]. the positive and negative polarities. It is applied to the DDD17 dataset described previously. Finally, the training is performed via backpropagation to minimise the soft-max cross-entropy loss measured by summing the error between the estimated pixels' classes and the true ones.

The semantic segmentation performance is measured thanks to standard metrics of semantic segmentation: the Accuracy (Eq. 3.3) and the Mean Intersection Over Union (MIoU -Eq. 3.4).

Accuracy(y, y) = 1 N N i=1 δ(y i , y i ) = T P + T N T P + T N + F P + F N (3.3) MIoU(y, y) = 1 C C j=1 N i=1 δ(y i,c , 1)δ(y i,c , y i,c ) N i=1 max(1, δ(y i,c , 1)δ( y i,c , 1)) = T P T P + T N + F P + F N (3.4)
where y and y are the desired output and the system output respectively. C is the number of classes.

N is the number of pixels and δ denotes the Kronecker delta function. T P , T N , F P , and F N respectively stand for true positive, true negative, false positive, and false negative.

DSEC

The Driving Stereo Even t Camera dataset DSEC is a hybrid stereo event camera and video camera dataset in driving scenarios, in both favourable and very challenging illumination conditions (M. [START_REF] Gehrig | Dsec: A stereo event camera dataset for driving scenarios[END_REF]. The scenes are recorded using two event cameras of resolution 640 × 480, combined with an RGB camera and a lidar. The dataset is composed of 53 sequences in urban, suburban and rural areas in Switzerland, amounting to a total of 3193 seconds. The authors wished to overcome issues present in existing similar datasets (such as the Multi Vehicle Stereo Event Camera Dataset MVSEC, DDD17 or DDD20) and evaluate the datasets' quality using several established metrics in stereo matching.

Lip reading

Tan et al. [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF] proposed a novel dataset of event-based recordings of lip motions and an associated architecture to perform event-based lip reading: the MSTP network. As discussed in [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF], the existing event-based action recognition methods, such as pointcloud-based, graph-based, and fixed-frame-based approaches, are not suitable for the lip reading task since it requires the perception of fine-grained spatiotemporal features from the event data.

The performance of the MSTP on this task was experimentally proven to be superior to the state-of-the-art event-based action recognition models and video-based lip reading models. Within the MSTP network, the input events are consequently converted to low-rate and high-rate event frames with different temporal bins to better preserve the spatiotemporal information of the event stream. These two types of event frames are then fed into a multi-branch network with message flow modules between different branches designed to perceive both complete spatial features and fine temporal features from the event data. Following that, a sequence model decodes the visual features into words. In their study, SNNs are mentioned but discarded because they lack an efficient backpropagation algorithm to train them. Hence, SNNs are yet to be applied to this problem. [START_REF] Bryner | Event-based, Direct Camera Tracking from a Photometric 3D Map using Nonlinear Optimization[END_REF][START_REF] Gallego | Event-based, 6-DOF camera tracking from photometric depth maps[END_REF]

Overview of existing datasets

Camera tracking

Indoor and outdoor camera trajectories 244M B (ROSBAG)

Event-based Lip-reading [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF] Lip reading Close-ups of lips pronouncing words (100 classes) 87.4M B (NPY) EVIMO [START_REF] Mitrokhin | Ev-imo: Motion segmentation dataset and learning pipeline for event cameras[END_REF][START_REF] Mitrokhin | Learning visual motion segmentation using event surfaces[END_REF][START_REF] Parameshwara | SpikeMS: Deep spiking neural network for motion segmentation[END_REF] Motion PRED18 [START_REF] Moeys | Steering a predator robot using a mixed frame/event-driven convolutional neural network[END_REF][START_REF] Moeys | PRED18: Dataset and further experiments with DAVIS event camera in predator-prey robot chasing[END_REF] Predator/prey robot chasing Left, right, center and non-visible 298GB (AEDAT) ROSHAMBO17 [START_REF] Lungu | Live Demonstration: Convolutional Neural Network Driven by Dynamic Vision Sensor Playing RoShamBo[END_REF] Symbol recognition Rock, paper, scissor and no hand 79GB (AEDAT) SL-ANIMALS-DVS [START_REF] Vasudevan | Introduction and Analysis of an Event-Based Sign Language Dataset[END_REF][START_REF] Vasudevan | Sl-animals-dvs: event-driven sign language animals dataset[END_REF] Action recognition Sign language words for 20 animals 749M B (AEDAT)

3.4 -Limits 49

Limits

While the event camera provides the significant advantages described above, one could ponder whether its microsecond resolution is relevant in any situation. In recent works, some of its application cases could have been solved using a less precise temporal grain. for example, human pose estimation and gesture recognition are studied in the widely used benchmark dataset DVS 128 Gesture described above -but does one really need a microsecond resolution when one gesture is completed in roughly 1 second ? It could be of greater interest to apply this technology to domains where the phenomenons of interest are significantly brief, such as in high-energy physics or detonics.

Additionally, due to their differential imaging mechanism event cameras are more subject to noise than standard RGB cameras: both are subject to 'inherent shot noise in photons and transistor circuit noise, but the additional quantisation of temporal contrast in event cameras is more complex to control [START_REF] Gallego | Event-based vision: A survey[END_REF]. The noise in event data is mainly caused by to movements of background objects [START_REF] Padala | A noise filtering algorithm for event-based asynchronous change detection image sensors on truenorth and its implementation on truenorth[END_REF]. When the overall brightness of the visual scene decreases, the noise in the recorded event data increases due to the camera continual sampling and the logarithmic conversion [START_REF] Ding | E-mlb: Multilevel benchmark for event-based camera denoising[END_REF]. To overcome this issue, different strategies to implement denoising either at sensor level or in post-processing were developed in recent works (see Section 1.3.1). [START_REF] Dilmaghani | Control and evaluation of event cameras output sharpness via bias[END_REF] introduces in 2022 a complete evaluation of the different biases (i.e. sensor settings) proposed by Prophesee in their Gen4H camera and compares the quality of output events thanks to the "average image gradient magnitude" metric assessing their sharpness. It has also been demonstrated that the contrast sensitivity of an event camera influences the amount of noise and pixel-to-pixel mismatch: the contrast should be set above a lower bound to avoid "a storm of events" [START_REF] Gallego | Event-based vision: A survey[END_REF].

CHAPTER 4

Spiking neural networks applied to an embedded event camera

The joint use of spiking neural networks and event cameras, respectively introduced in previous Chapters 2 and 3, in an embedded setting is a promising combination for dynamic visual data processing. Both technologies have recently emerged separately about a decade ago from electronics and neuroscience communities, sharing many features: biological inspiration, temporal dimension, model sparsity, aim for higher energy efficiency, etc. This chapter further develops this initial statement and presents an overview of state-of-the-art embedded models interfacing event cameras and SNNs.

Because of their asynchronous operation principle, event cameras are a natural match for SNNs: each event can be assimilated to an activated spike between two neurons. Furthermore, the behaviour of the log-luminance recorded by the sensor displays a substantial similarity with one of a LIF neuron, as shown in Fig. 6.10. Their combined use is of such high interest from the point of view of biological inspiration, energy savings, decision latency, and memory use that it is gaining momentum in the field of embedded computer vision [START_REF] Gallego | Event-based vision: A survey[END_REF].

Embedded systems

According to De Micco et al., an embedded system is an "electronic equipment with a computing core which, unlike a personal computer, is designed to meet a specific function and is usually optimised to satisfy strict requirements of processing time, reliability, power consumption, size and cost" [START_REF] De Micco | A literature review on embedded systems[END_REF]. As underlined in [START_REF] Biglari | A review of embedded machine learning based on hardware, application, and sensing scheme[END_REF], those requirements still hinder the implementation of most ML algorithms. One must consider the specific application to identify the appropriate embedded system and its equipment, such as the sensors used. A review of the state-of-the-art research in the embedded ML area is presented in [START_REF] Biglari | A review of embedded machine learning based on hardware, application, and sensing scheme[END_REF], according to the application tasks (autonomous driving, security, healthcare, agriculture, drones, etc.). Among the sensors available, they mention LIDAR, electrocardiograms, microphones, radars, motion sensors and RGB, depth, thermal and 360-degree cameras. They make the remarkable omission of event-driven cameras, which are nevertheless particularly well suited to onboard computer vision due to their high dynamic range and temporal grain as well as their low memory and low energy consumption (see Chap. 3).

Why is it such a good idea ?

The emerging event-based vision technology promises to significantly improve energy efficiency and latency in the next generation of computer vision applications [START_REF] Gallego | Event-based vision: A survey[END_REF]. However, classic computer vision algorithms which rely on frame-based data for applications such as object classification [START_REF] Song | Improved convolutional neutral network based model for small visual object detection in autonomous driving[END_REF] and scene segmentation [START_REF] Peng | Multi-task adas system on fpga[END_REF] are unable to handle sparse, asynchronous event data coming from DVS. Approaches combining classic methods with event-based data as input require a first step of data pre-processing at the cost of losing temporal information (A. Z. [START_REF] Zhu | Ev-flownet: Self-supervised optical flow estimation for event-based cameras[END_REF].

However, recent work has shown that SNN can operate on raw event data and achieve similar performance as conventional methods (Paredes-Vallés, Hagenaars, & de Croon, 2021). Thanks to neuromorphic chips, and ultra-fast low-power processors, SNNs can be implemented on hardware and interfaced directly to an embedded event camera. They thus make for a promising alternative for the embedded handling of event-based data.

Indeed, even though the maturity of SNN-based vision algorithms is still far from state-of-theart conventional methods, which provide excellent results for vision tasks with traditional cameras, asynchronous event sequences require special handling for which SNNs are better suited. Indeed, many previous works have demonstrated the benefits of SNN and the perfect match with eventbased cameras to reach both exceptionally low latency and energy efficiency [START_REF] Gallego | Event-based vision: A survey[END_REF]) -all of this, however, without achieving equivalent performance to DNNs.

Pushed by the objective of closing the gap with DNN accuracy, many previous works do not consider the limits of memory and complexity that state-of-the-art embedded neuromorphic pro-cessors pose. Another drawback that limits the success of today's event-based cameras and neuromorphic computing, especially for embedded low-power systems, is the fact that the promised low power, low latency and energy efficiency are overpowered by the interface's high power consumption, often due to non-standard communication protocol. Although nowadays data is mainly acquired using a high-power FPGA and a USB interface [START_REF] Mauro | Flydvs: An event-driven wireless ultra-low power visual sensor node[END_REF], novel embedded processors are being developed, providing the research community with reference platforms and resources to address this issue. Large-scale, neuromorphic processors like SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF] or Intel Loihi [START_REF] Davies | Advancing neuromorphic computing with loihi: A survey of results and outlook[END_REF] can simulate hundreds of thousands or even millions of spiking neurons in real time. To achieve genuinely power-efficient real-time operation, such platforms need to be even smaller, reach even lower power consumption and integrate the camera interface directly on board. Such an example is given by Kraken (Di [START_REF] Mauro | Kraken: A direct event/frame-based multi-sensor fusion soc for ultra-efficient visual processing in nano-uavs[END_REF], which can interface directly with event-based cameras without power-hungry FPGAs or external interface adapters.

Applications

Traditional methods for standard vision tasks (e.g. recognition, tracking, segmentation, motion analysis, etc.) cannot be applied straightforwardly to event-based cameras due to their unconventional output type. Silicon retinas bring considerable potential in indexing and retrieval, yet only a limited number of methods have been explored in this direction. In 2020, Gallego et al. [START_REF] Gallego | Event-based vision: A survey[END_REF] establish inter alia a state of the art of existing SNN algorithms for feature detection, object tracking, 3D reconstruction and motion segmentation applied to event data. This state of the art is maintained up to date on the team's Github (Robotics & Perception Group, 2020).

We present below some state-of-the-art models applied to different application tasks and extend those relevant to the experiments performed during this PhD.

Classification

A classification task aims to assign a label to each dataset sample.

SLAYER benchmark classifier

Shrestha and Orchard developed in 2018 a Python framework called SLAYER (for Spike Layer Error Reassignment in Time) [START_REF] Shrestha | SLAYER: Spike layer error reassignment in time[END_REF], based on the ML library Py-Torch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF] and designed to simulate "backpropagation based SNN learning" on GPU. A specific "SLAYER Loihi" module has been implemented to run SNN models initially developed on SLAYER, on Intel's Loihi neuromorphic chips [START_REF] Davies | Advancing neuromorphic computing with loihi: A survey of results and outlook[END_REF] without further adaptation to the code. This module and, more globally this framework, were benchmarked on a classification task applied to different spiking and non-spiking visual datasets, such as MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], N-MNIST [START_REF] Orchard | Converting static image datasets to spiking neuromorphic datasets using saccades[END_REF] and DVS128 Gesture (Amir et al., 2017), and the audio dataset TIDIGITS converted into spikes [START_REF] Leonard | Tidigits. Philadelphia: Linguistic Data Consortium[END_REF]. According to the authors, when the input is a spiking dataset, "the spike data from the DVS is directly fed into the classifier" which removes any step of pre-process and allows the SNN to be trained using "significantly less number of neurons and layers" than any state-of-the-art CNN classifier. (Fang, Yu, Chen, Masquelier, et al., 2021). This model aims to better represent the heterogeneity of biological neurons by learning the membrane time constant, whereas membrane parameters usually correspond to hyperparameters set before training. To reproduce the biological dynamic where all neighbouring neurons share similar properties, the time constant is the same for all neurons belonging to the same layer but differs between layers, thus implementing diverse phase-frequency responsiveness. The authors also opted for a max-pooling method instead of the average-pooling method traditionally used in SNN, thus preserving the asynchronous characteristic of neuron firing.

This model allows the authors to apply a backpropagation learning algorithm to a classification task. They present the results obtained when classifying traditional RGB datasets, as well as neuromorphic datasets such as DVS128 Gesture (Amir et al., 2017), N-MNIST [START_REF] Orchard | Converting static image datasets to spiking neuromorphic datasets using saccades[END_REF] and CIFAR10-DVS (H. Li et al., s. d.).

It is important to note that the authors chose to process the neuromorphic datasets as frames precisely because of the high number of events. The events E are thus translated from their traditional representation E(x i , y i , p i , t i ) (with x i and y i the pixel's coordinate of the i th event, p i the polarity and t i the timestamp) into T slices of same time length. The frames F are represented as follows:

F (j, p, x, y) = tmax t=t min δ(x t , x).δ(y t , y).δ(p t , p) (4.1)
with t min and t max the minimal and maximal timestamp in the j th slice and δ the Kronecker delta function. When run on neuromorphic datasets, this classifier considers each frame as the input for one timestep ; T is the number of timesteps on which the classifier is run for one epoch.

HOTS

The online classification algorithm introduced in (Grimaldi et al., 2023) is an extension of a previous study entitled HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF]. In this work, they make object recognition on a stream of events through a feedforward hierarchical architecture using time surfaces, an event-driven analog representation of the local dynamics of a scene. Using a form of Hebbian learning, the network can learn in an unsupervised way progressively more complex spatiotemporal features which appear in the event stream. Once trained, one layer of this network transforms any incoming event from the stream of events into a novel event as it is selected in a layer of spiking neurons. Using it as a building block, such layers can be stacked together, each layer's output address space defining a novel input address space for the next layer.

Inspired by this dynamical processing of the visual information, (Grimaldi et al., 2023) added a Multinomial Logistic Regression as an online classifier and transformed the previous post hoc classification process, which counted the activity of the neurons of the last layer, into an always-on decision process. The Multinomial Logistic Regression layer also takes time surfaces as input, and a formal demonstration was made to assimilate this algorithm to an SNN with Hebbian learning and homeostasis.

The classifier introduced in (Grimaldi et al., 2023) will be referred to as "HOTS" in the remainder to facilitate its designation.

3D reconstruction

Stereo vision is the computation of depth based on the binocular disparity between the images of an object in the left and right eyes. This requires matching up features in the two eyes, that is, identifying features in the left and right retinas that are both images of the same point in the visual scene. This allows to obtain the 3D information from a pair of images by estimating the relative depth of points in the scene. These estimates are represented in a stereo disparity map, which is constructed by matching corresponding points in the stereo pair. The disparity is computed as:

disparity = x -x i = B × f Z (4.2)
where x and x i are the distance between points in the image plane corresponding to the 3D scene point and their camera centre, B is the baseline that represents the distance between two cameras, Z is the depth, and f is the focal length of the camera.

The models we detail below will tackle this computer vision task using a stereo-matching approach. It is interesting to note that it is far from the only existing neuromorphic approach one could apply: for example, the StereoSpike model introduced in [START_REF] Rançon | Stereospike: Depth learning with a spiking neural network[END_REF] reconstructs depth from a stereo-event stream using a modified U-Net-like encoder-decoder architecture, which performs supervised learning using a surrogate gradient and achieves good results with a remarkably low number of parameters compared to the state-of-art.

As stated above, the models we studied in this thesis tackle 3D reconstruction using stereo matching. This "correspondence problem" is the problem of finding a pair of corresponding pixels or patches in a set of stereo images. Having the matching objects or pixels, we can calculate the depth, which is inversely proportional to the disparity-the displacement along the epipolar lines-between the matches.

The geometrical characteristics of the stereo cameras can be formulated to map a single pixel of one image into a set of possible corresponding pixels in the other image. In classic computer vision, there are two general approaches for matching pixel pairs: area-based and feature-based matching. The use of DVSs introduces a separate clue that can be used for matching: time. Just as with its biological counterpart, the asynchronous operation of the silicon retina opens the possibility of matching events based on their time of appearance. However, due to inherent sensor variations and spatial aliasing, event coincidence alone is not reliable enough for event matching and has to be combined with additional constraints to achieve a reasonable matching quality. 4.3.2.1 Osswald et al., 2017[START_REF] Osswald | A spiking neural network model of 3d perception for event-based neuromorphic stereo vision systems[END_REF] introduced in 2017 a "spiking neural network model of 3d perception for event-based neuromorphic stereo vision systems", composed of three neuronal populations: the retinal population, the coincidence detectors and the disparity detectors. Retina cells serve as input to the network and project excitatory connections towards the coincidence detectors. The disparity detectors pool the responses from the coincidence detectors via excitatory and inhibitory connections. The network outputs a disparity event only when the spikes produced by a disparity neuron and a corresponding coincidence neuron (i.e. located in a nearby representation in disparity space) take place within a few milliseconds.

This architecture is implemented on an FPGA interfaced with the neuromorphic processor ROLS [START_REF] Qiao | A re-configurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128k synapses[END_REF], comprising an array of 256 analog IF neuron circuits and 512 × 256 dynamic synapses. All in all, this model successfully solves the stereo correspondence problem, as evidenced by the small local average disparity error ϵd, inferior to 1 pixel.

Dikov et al., 2017

The "spiking cooperative stereo-matching" model introduced in [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] is composed of a three-dimensional array of cells, where each cell encodes a belief in matching a heterolateral pair of pixels from the left and the right visual scenes from the stereo event stream. Due to intrinsic structural constraints, this model aims to respect the biologically plausible principles devised by [START_REF] Marr | Cooperative computation of stereo disparity[END_REF]:

-within-disparity continuity, implemented using a weak potentiation between neighbouring cells representing similar disparity and leading to a smooth disparity map ; -cross-disparity uniqueness, implemented via a strong inhibition between cells to allow for the pairing of two corresponding pixels ; -prevention of the homolateral matching, i.e. the activation of disparity neurons solely due to the activity of one retina. The corresponding connectivity patterns are implemented via excitatory and inhibitory synaptic pictured in Fig. 4.1. [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF].

This network measures the disparity from the event stream recorded by two parallel stereo event cameras. It is implemented using PyNN [START_REF] Davison | Pynn: a common interface for neuronal network simulators[END_REF] on the neuromorphic board SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF].

The disparity is contained in a range [d min , d max ]. In this work, the disparity is strictly positive ; since the cameras are parallel, d min = 0 corresponds to very distant objects. As the neural micro-ensembles are arranged in each layer according to a trapezoid, and each layer corresponds to one coordinate y, we can calculate the total number of neural micro-ensemble N nme as:

N nme = n × y max with n = w baseline + w topline 2 × h (4.3)
where n is the number of neural micro-ensembles in each layer and w baseline and w topline are respectively the wide and small bases of the trapezoid, computed according to Eq. 4.4. All in all, knowing that each neural micro-ensemble contains three neurons, the number of neurons N n in the network is:

       h = d max -d min + 1 w baseline = x max -d min w topline = x max -d max
N n = 3 × (x max - d min + d max 2 ) × (d max -d min + 1) × y max (4.5)
With a sensor size of x max = y max = 128 with a [0, 127] disparity range, the network implementation requires 3.15 million neurons, corresponding to at least 12,288 SpiNNaker cores, 723 SpiNNaker chips or 15 SpiNN-5 boards.

This spiking cooperative stereo-matching network is experimentally validated on three datasets: a synthetic dataset presenting letters in space at various depths, with no jitter or noise ; and two experimental datasets, "two fans" and "pendulum", recorded in a controlled environment. In the "two fans" dataset, two identical 3-blade fans with a diameter of 35cm are recorded during 10s by two event cameras set up with a horizontal offset of 10cm. The fans are respectively 1.8m and 2.8m away from the cameras and generate approximately 140, 000 events per second per camera. In the "pendulum" dataset, a pendulum (i.e. a tennis ball suspended on a string) swinging back and forth at around 0.6Hz is recorded during 10s by two event cameras set up with a horizontal offset of 10cm. The pendulum swings along the line-of-sight of the left camera and generates around 20K events per second per camera. The results are thus obtained on three datasets, one synthetic and two real-world tests, of which one is quasi-stationary and one dynamic.

The network achieves an accuracy of 86% on the synthetic dataset. For the "two fans" dataset, 97% of the output events around the distant fan indicate a disparity d = 7 ± 1 and 06.7% of the events around the proximate fans indicate a disparity d = 10 ± 1. Finally, 62.2% of all events produced when presented to the "pendulum" dataset are within ±2 pixels of that metric. A systematic error causes this relatively bad matching performance: the network tends to fuse the ball images mirror-symmetrically in this dynamic scene. According to the authors, this behaviour can be counteracted by object-level matching or a simple ordering constraint. 4.3.2.3 Risi et al., 2020[START_REF] Risi | A spike-based neuromorphic architecture of stereo vision[END_REF] proposes a spiking stereo neural network adapted from the architecture presented above [START_REF] Osswald | A spiking neural network model of 3d perception for event-based neuromorphic stereo vision systems[END_REF]. Firstly, each coincidence detector is connected to the retina cells via a slow NMDA-like synapse on one side and a fast AMPA-like synapse circuit block on the other. This mechanism demands that both synapses be stimulated rapidly so that the coincidence detector neuron fires. Secondly, the authors implement each cell as a micro-ensemble to prevent homolateral matching, similarly to [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF].

This architecture is implemented on an FPGA platform interfacing two DAVIS cameras with three neuromorphic processors DYNAPs [START_REF] Moradi | A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs)[END_REF]. It is experimentally validated on multiple synthetic datasets and in real-time scenarios.

Visual attention

An extensive review of SNN models of visual attention applied to event data can be found in Section 5.1, along with the biological basis of visual attention.

CHAPTER 5

Concepts from neuroscience

This chapter extends Chapter 4 by reviewing applications combining spiking neural networks and event cameras inspired by biology and the associated biological mechanism.

This chapter introduces SNN models applied to event data to solve application tasks inspired by biological behaviours. The corresponding biological behaviour is described in detail ; each application is the object of a comprehensive review. The three biological behaviours adapted to neuromorphic computing studied in this PhD are: visual attention, foveation and delay learning.

Visual attention

Visual attention can be defined as the behavioural and cognitive process of selectively focusing on a discrete aspect of sensory cues while disregarding other perceivable information. This biological mechanism, more specifically saliency detection, has long been used in computer vision to drive the analysis only on relevant parts of images or videos for further processing.

It is important here to differentiate the attention we are interested in with the concept of attention as most often used in AI. Indeed the AI community comprehends attention as a piece of additional information calculated and updated during runtime for each input vector in order to focus learning on the most relevant input elements. This attention mechanism was originally applied to recurrent neural networks such as encoder-decoder models, which tended to give more importance to the last input elements. It was then used as building blocks for the transformer model, where they are now computed in parallel instead of sequentially [START_REF] Vaswani | Attention is all you need[END_REF]. Transformers were recently adapted to SNNs with the recent example of Spikformer [START_REF] Zhou | Spikformer: When spiking neural network meets transformer[END_REF].

What interests us in this work is the attention behaviour observed in biological organisms, as described in further detail below.

To limit the amount of data to be processed, several approaches inspired by primates' visual attention mechanisms [START_REF] Itti | Computational modelling of visual attention[END_REF][START_REF] Martinet | Gaze based quality assessment of visual media understanding[END_REF] have proposed to focus the processing only on the most salient parts of the scene. Indeed, the study of many biological organisms highlighted the importance of limiting the processing of sensory RoIs, thus minimising the energy spent on this task as well as the reaction delay to what is perceived [START_REF] Tsotsos | Analyzing vision at the complexity level[END_REF]. For prey animals, this means a quicker detection of a potential predator and a more efficient escape from danger. Attention was first described as the coalition of "focalisation, concentration and consciousness" by William James in 1890 [START_REF] James | The principles of psychology[END_REF]. In psychology, this mechanism is now defined as the allocation of limited cognitive processing resources on one or a few relevant environmental elements while ignoring others [START_REF] Anderson | Cognitive Psychology and Its Implications[END_REF]. It can be either subjective or objective, as well as voluntary or spontaneous.

As technologies for sensory information processing expanded, an increasing number of them assimilated this idea for their own purposes. In the DL community, for instance, this concept was converted into a neural network's component weighting features by the task's level of importance. Attention can thus be applied to regions in images, words in text, phonemes in speech, etc. For example, some studies aim to exploit this mechanism in order to optimise CNNs [START_REF] Jetley | Learn To Pay Attention[END_REF]. We believe that studying the biological aspect of visual attention will allow for the development of new models applicable to event vision.

Biological visual attention

Early work by Jonides [START_REF] Jonides | Further toward a model of the Mind's eye's movement[END_REF] distinguish two stages of visual attention, defined by the type of processing: it can be processed either sequentially when it is concentrated on a specific area of the visual scene ; or it can be processed in parallel in case of a more uniform distribution over the scene. Multiple models of visual attention have been defined over time, all coexisting with one another. The earlier one is the spotlight model, stating that "attention has a focus, a margin and a fringe" [START_REF] James | The principles of psychology[END_REF], meaning it does not entirely prevent the perception of elements outside the attended visual area. The zoom-lens model [START_REF] Eriksen | Visual attention within and around the field of focal attention: A zoom lens model[END_REF], the attentional engagement theory [START_REF] Raftopoulos | Cognition and Perception: How Do Psychology and Neural Science Inform Philosophy ?[END_REF] and the gradient model [START_REF] Laberge | Theory of attentional operations in shape identification[END_REF] were all three elaborated between 1986 and 1989 and corresponded to non-exclusive psychological and biological observations in humans. Treisman went one step further and produced the Feature Integration Theory [START_REF] Treisman | A Feature-Integration Theory of Attention[END_REF], according to which visual searches are done through combined parallel and serial stages ; the blended use of those two processes enables the integration of the global features of a complex visual scene.

Selective visual attention can relate to other well-known bio-inspired concepts. Curiosity can be considered as a driver for visual attention since a novel element catches more of the eye than a known element [START_REF] Shi | A Curiosity-Based Learning Method for Spiking Neural Networks[END_REF]. In addition, selective visual attention suffers from the exploration versus exploitation trade-off: should one spend more time studying a specific element with the risk of missing crucial information or losing energy processing the entire visual scene at all times ?

Eye and visual cortex in the context of the visual attention

This section outlines what is known about the organs and pathways enabling human vision to review human visual attention.

Human eye and retina Humans have complex eyes, allowing for colour detection and binocular vision, thus depth perception. It is composed of three parts: the outermost layer comprises the cornea and the sclera, whereas the middle layer contains the iris and muscles. The innermost layer, situated at the back of the eye, is mainly composed of the retina, which permits the detection of light and colours. The retina contains two types of light-sensing cells: the photoreceptor cells (rods and cones) and the photosensitive ganglion cells, emulated by the event cameras (see Chapter 3).

The inside of the human eye is filled with humour which the light rays cross until they reach the retina. They are primarily captured by the macula, a retina spot responsible for central colour vision. A subdivision of this macula is the fovea, where the cones are more closely packed than anywhere else in the retina. Since cones are responsible for colour discrimination and delicate detail perception, the fovea corresponds to the central spot where the vision is optimal in bright light. Hence any attention mechanism will aim to direct the eyes to point the "foveal gaze" towards the interesting visual feature. It should be noted that the small size of the fovea limits the overall perception of fine details. The organism overcomes this issue by redirecting the eye in saccades, up to 3 times per second. A similar mechanism could be implemented, allowing for a higher resolution on the spot targeted by the "foveal gaze", thus letting a larger amount of specifically relevant information be processed.

Visual cortex Once the visual information has travelled from the eyes and through the thalamus, it is processed by the visual cortex in the brain's occipital lobe. Each hemisphere has a visual cortex cross-handling the information output by the opposed eye. The visual data is received firstly by Figure 5.1 -Schematic depiction of the visual cortex, adapted from [START_REF] Bear | The Human Eye[END_REF].

Located in the frontal cortex, the Frontal Eye Field (FEF) has a role in visually guided saccades' target selection and selective spatial attention [START_REF] Moore | Neural Mechanisms of Selective Visual Attention[END_REF]. It encodes spatial information into retinocentric coordinates, that is, coordinates established using the retina as a reference. It receives inputs from and projects onto most of the visual cortex.

According to Knudsen et al. [START_REF] Knudsen | Neural Circuits That Mediate Selective Attention: A Comparative Perspective[END_REF], high activity in the FEF brings an increased neuronal response in V4 in a "space-specific, attention-dependant manner". Indeed, the prefrontal cortex integrates the task according to a non-retinocentric frame of reference and translates it into retinocentric signals by sending it to the lateral intra-parietal area. The FEF convert those signals into topographical maps and finally outputs those maps to the retinocentric visual regions of the posterior cortex and superior colliculus. This may hint at a pathway enabling the influence of saccade-related signals on visual representation. Once understood, this pathway could be implemented as a neural network to specifically direct attention towards the retinocentric space privileged by the FEF.

Forebrain and midbrain networks. It is interesting to note that the forebrain and midbrain networks are strongly involved in attention [START_REF] Knudsen | Neural Circuits That Mediate Selective Attention: A Comparative Perspective[END_REF]. The forebrain selects information based on its relevance to the task at hand or its saliency. It encodes visual information via enhanced distributed representations regulated by heterogeneous dynamics. A similar type of information encoding, with each element interpreted according to its saliency, would speed up its processing significantly.

The midbrain enables spatial attention by monitoring the environment for behaviourally relevant stimuli and then directing the gaze to the region of highest interest thanks to a powerful inhibitory competition. It represents the visual input from the retina and the visual forebrain as a topographic map of space following a retinocentric frame of reference. A midbrain-like neural network would allow for the identification of the data relevant to the task at hand in order to optimise the bandwidth of information to process.

Neural mechanisms

Visual attention can be categorized using three dichotomies, differentiated by distinct and divergent neural mechanisms [START_REF] Moore | Neural Mechanisms of Selective Visual Attention[END_REF].

The first dichotomy can be defined between top-down and bottom-up attention. The top-down one corresponds to a selective type of attention depending on a previously set motivation or rule. This "endogenously generated signal" influences visually-driven signals in primates. The bottomup depends on physical saliency, including brightness, movements and colours. Those two interdependant types of attention interact strongly during visual search, which helps to focus on salient features relevant to the task at hand and disregard the less fitting ones.

A second dichotomy exists between spatial, temporal and feature-based attention. Spatial attention is directed towards a specific location in space thanks to the occulomotor system, which leads to the prioritisation of an area in the visual field. Furthermore, temporal attention focuses on a specific instant in time ; this type is often tapped into for video processing and emphasises critical video frames. It could be particularly interesting to exploit this kind of attention in human action recognition. Finally, feature-based attention selects elements based on their resemblance to a behaviourally relevant object by representing the corresponding features in the prefrontal cortex anterior and the ventral pre-arcuate [START_REF] Bichot | A source for feature based attention in the prefrontal cortex[END_REF].

The last dichotomy opposes overt and covert attention. It respectively corresponds to the presence (overt) or absence (covert) of motor commands leading to saccadic eye movements. Those two types of attention are simultaneous and complementary: overt attention comes from orientating eye movements, actively guided by salient features determined thanks to concurrent covert attention. When attention comes to orientating movements (overt), the posterior parietal cortex activity increases. In addition, the ventral stream visual area is actively involved in guiding this motion according to salient features determined thanks to concurrent covert attention. This is confirmed by the observation of a significant temporal correlation between the visual processing of targets and eye movement [START_REF] Moore | Neural Mechanisms of Selective Visual Attention[END_REF]. Interestingly, covert attention is the one most often studied by visual neuroscientists. It improves the detection and discrimination of features [START_REF] Chevallier | Covert Attention with a Spiking Neural Network[END_REF] both at the fovea and in the visual periphery, thanks to a visual enhancement in V4 and the inferior temporal cortex.

Neuromodulators

Neuromodulators are a class of extraneuronal molecules released within cortical and subcortical structures which allow and influence the signal transmission between neurons. According to Moore and Zirnsak [START_REF] Moore | Neural Mechanisms of Selective Visual Attention[END_REF], three neuromodulators have a particularly relevant role in attention: acetylcholine, dopamine and norepinephrine.

Acetylcholine is synthesised by the nucleus basalis of Meynert, the substantial innominata and the basal forebrain. When collected by a certain type of receptors present on the neuron surface (metabotropic muscarinic or ionotropic nicotinic receptors), it can enhance selective visual attention. Moreover, the release of acetylcholine seems to play a role in the enhanced processing of sensory information: its increase triggers glutamate release from retinal ganglion cells. This release of extracellular molecules by those photosensitive cells boosts their effect and amplifies the visual data captured from the corresponding retinal space.

Another neuromodulator of interest is dopamine: synthesised by midbrain nuclei, it appears to alter the strength and reliability of converging glutaminergic synapses in the prefrontal cortex, Also implicated in this disorder, norepinephrine is involved in the selective response to salient sensory stimuli, depending on the relevance of the stimuli for the task. It is synthesised by the locus coeruleus.

Much like dopamine pathways were studied and imitated when developing various models of reinforcement learning, it would be highly appealing and relevant to tailor those neuromodulators' mechanisms in order to enhance computational models applied to computer vision tasks.

Neuromorphic visual attention

Applications of visual attention to computer vision have been studied for the past two decades. However, the possible impact of SNNs and event-based cameras on this domain has been little studied so far. In the following section, we establish a review of state-of-the-art attention mechanism models applied to event camera data and/or using SNNs.

Itti and Koch presented a complete overview of bottom-up visual attention models in 2001 [START_REF] Itti | Computational modelling of visual attention[END_REF]. They define saliency as a bottom-up type of attention, operating very quickly and independently to the nature of the task. Most of the computer vision domain work on saliency relies upon this definition.

In 2014, Le Callet and Niebur reviewed the various existing applications of visual attention to multimedia technologies (Le [START_REF] Callet | Visual Attention and Applications in Multimedia Technologies[END_REF]. Their convincing synthesis of the domain emphasises the benefit of attention particularly regarding "multimedia delivery, retargeting and quality assessment of image and video, medical imaging, and the field of stereoscopic 3D images applications". It strengthens our conviction that applying attention mechanisms to computer vision, using SNNs and event data, is a highly enticing line of scientific inquiry.

Attention mechanism with SNNs

The use of SNNs instead of traditional artificial neurons has been prioritised in the modelling of attention for some years, maybe due to their common bio-inspiration. Chevallier et al. [START_REF] Chevallier | Efficient neural models for visual attention[END_REF] compared the relevance of their use for this task and concluded in favour of spiking neurons over traditional ones, following their previous work on attention in the exploration of a prey-predator environment [START_REF] Chevallier | Distributed processing for modelling real-time multimodal perception in a virtual robot[END_REF].

SNNs have thus been used to model different types of attention mechanisms applied to various tasks. For instance, Katayama et al. [START_REF] Katayama | Neural network model of selective visual attention using Hodgkin-Huxley equation[END_REF]) used SNNs to implement overt attention through selective visual attention with gaze shift. This model used two layers: one representing the virtual cortex and the other the hippocampal formation. The correlation of firing times of spikes output by these two layers determines three states: the attention state, the non-attention state and the shift of attention.

Another application of SNNs is the modelling of covert attention (visual attention without eye movement): Chevallier and Tarroux [START_REF] Chevallier | Covert Attention with a Spiking Neural Network[END_REF] use SNNs to extract saliency and focus on the attention of a moving stimulus. This is implemented using a neural filter, a saliency map and a focus map. The filter thresholds enforce a convolution on low and high spatial frequencies. The first map gathers information from visual features on different spatial frequencies thanks to synchrony detectors. The second map finally allows covert attention using a self-connection mask. In this model, saliencies are temporally coded and arise in hierarchical order, close to the forebrain representation of the visual scene. [START_REF] Thakur | Neuromorphic visual saliency implementation using stochastic computation[END_REF] implement a neuromorphic model of visual saliency in RGB data on digital hardware, using stochastic computation and achieving very low power consumption. They believe their model to be useful for "information triaging, compression and analysis in computer vision tasks". The neuromorphic dynamic visual saliency model introduced in [START_REF] Molin | A neuromorphic proto-object based dynamic visual saliency model with a hybrid fpga implementation[END_REF]) is bottom-up, feed-forward and built on the concept of proto-objects, i.e. low-level elementary features formed rapidly and in parallel in the visual cortex. The authors compare their results to the ground truth obtained from human eye prediction as well as implement their model on a hybrid FPGA to enable real-time applications.

Chick et al. [START_REF] Chik | Selective attention model with spiking elements[END_REF]) defined a model for selective visual attention applied to the sequential selection of objects. This top-down, temporal, feature-based attention uses a 2-layer architecture of inhibitory, excitatory and peripheral neurons. This SNN model used two types of inhibition to implement respectively attention focus and shift thanks to short-term plasticity.

SNNs can also allow for object detection with saliency indexing, mimicking top-down attention. Such a model has been implemented by [START_REF] Wu | A visual attention model based on hierarchical spiking neural networks[END_REF], in the form of three successive stages: a network inspired by the biological retina extracts the low-level image features, which are then decomposed into multiple visual pathways. Finally, "attention area maps" similar to the retinocentric output of the FEF are created, approximated by spike rate maps and enabling the detection of RoIs.

A model of interest has been defined by [START_REF] Bernert | An Attention-Based Spiking Neural Network for Unsupervised Spike-Sorting[END_REF] with an application to spike sorting, a common pattern recognition problem in neurosciences. Spike sorting consists of detecting action potentials emitted by biological neurons, thanks to the classification of the patterns present in the membrane potential measured in said neurons. This unsupervised network reproduces overt temporal attention after a short learning period with little data: an attention neuron modulates intermediate and output layers according to a combined short-term plasticity, hysteresis and threshold adaptation mechanism.

Finally, since curiosity is one of the main drivers of attention, we can mention the work recently introduced in [START_REF] Shi | A Curiosity-Based Learning Method for Spiking Neural Networks[END_REF]. To perform object recognition on MNIST data, the authors regard attention as the novelty of a stimulus. Thus their SNN model learns by estimating the novelty of the visual samples and updating those samples when the novelty exceeds a certain threshold. According to the authors, the attention mechanism takes place in one of the cerebral pathways responding to curiosity: more specifically, it follows the pathway starting from the ventral tegmental area and reaching the hippocampus via the prefrontal cortex and the precuneus.

Attention mechanisms applied to silicon retinas

The processing of event-based data often takes place via the adaptation of computer vision models initially dedicated to classic RGB visual information. One example is given by Cannici et al. [START_REF] Cannici | Attention Mechanisms for Object Recognition with Event-Based Cameras[END_REF], who designed two visually attentive models for event-based data. One aims to locate RoIs using event activity within the field of view, while the other is based on the Deep Recurrent Attentive Writer neural model. This model was designed in 2015 by [START_REF] Gregor | DRAW: A recurrent neural network for image generation[END_REF] to generate complex images and emulate the foveation of the human eye, thus implementing a spatial attention mechanism. Another example of adaptation from RGB to event data is given by the evProto model introduced in [START_REF] Iacono | Protoobject based saliency for event-driven cameras[END_REF]: the authors extend the algorithm for proto-object detection and spatialscale estimation in RGB data introduced in [START_REF] Russell | A model of proto-object based saliency[END_REF], based on border ownership -which reacts to edges potentially belonging to a border -and grouping -strongly activating a region enclosed by several object borders. [START_REF] Ghosh | Event-driven proto-object based saliency in 3d space to attract a robot's attention[END_REF] further develops the previous work into a depth-based attention model by combining it with a biologically inspired stereo disparity estimation algorithm.

Feng et al. [START_REF] Feng | Event density based denoising method for dynamic vision sensor[END_REF] implemented an algorithm applied to denoising of event data: the background of a scene recorded by a DVS camera is denoised thanks to the detection of event density in a spatiotemporal neighbourhood, which can be assimilated to covert visual attention.

Attention mechanisms with SNNs applied to silicon retinas

Works addressing visual attention by combining SNNs and event data are rare in the literature. Recently, some authors have studied the combination of SNNs used on event-based data in order to convey an attention mechanism. An example of such a combination is given by Bogdan et al. [START_REF] Bogdan | Eventbased computation: Unsupervised elementary motion decomposition[END_REF], where they develop an unsupervised decomposition of elementary motion detectors. According to them, "fast localised motion detection is crucial" for an efficient visual attention mechanism.

Additionally, neuromorphic saliency detection models applied to event data are most often than not derived from existing models implemented with traditional neural networks and applied to RGB or grayscale images: (D'Angelo, Perrett, Iacono, Furber, & Bartolozzi, 2022) is for example adapted from [START_REF] Iacono | Protoobject based saliency for event-driven cameras[END_REF], itself originally adapted from [START_REF] Russell | A model of proto-object based saliency[END_REF]. Another noteworthy example is given by the model recently introduced in [START_REF] Lele | Bio-mimetic high-speed target localization with fused frame and event vision for edge application[END_REF], where the authors combine CNN and SNN applied respectively on frames and event data to achieve embedded target localisation in a prey/predator scenario. Both networks are implemented on a drone and their fusion outperforms CNN-only processing in realistic 3D virtual environments. The CNN is trained for prey detection on the intensity image reconstructed from the embedded DVS, while the SNN allows for the cancellation of events generated from the drone ego-motion. This combination is inspired by insects and primates' biological behaviours: their vestibular sensors interact with the vision "to cancel the activity corresponding to the predator's self-motion".

Only a few models genuinely take advantage of the intrinsic dynamics of SNNs and the uniqueness of event data. One popular bioinspired mechanism applicable to the task of saliency detection is the Dynamic Neural Field (DNF), a spatiotemporal mathematical description of activity patterns in neuronal populations in the visual and motorcortex [START_REF] Sandamirskaya | Dynamic neural fields as a step toward cognitive neuromorphic architectures[END_REF]. Renner et al. [START_REF] Renner | Event-based attention and tracking on neuromorphic hardware[END_REF]) make use of this mechanism as a soft WTA to implement saliency tracking of pre-activated objects, by manually increasing the activity in a pre-defined region. While neuromorphic, this approach is not applicable in real scenarios where the RoI is not known beforehand. DNF was also used by [START_REF] Rasamuel | Specialized visual sensor coupled to a dynamic neural field for embedded attentional process[END_REF] to extract a saliency map from combined DVS and standard camera, with an application for real-world exploration and exploitation. Renner et al. also proposed a combined use of SNNs and silicon retina, applied to object tracking [START_REF] Renner | Event-based attention and tracking on neuromorphic hardware[END_REF]: they implemented a recurrent SNN emulating the activity of large homogeneous populations of biological neurons. Via a soft WTA, this model thus adopts the attractor dynamics of DNFs.

Foveation

Biological foveation

The majority of biological visual sensors have very irregular sensor grids. For example, some insects have a peripheral vision grid used for navigation, while another is specialised to a very specific location in the visual space and dedicated to mating behaviour. Most predatory mammals have a central visual area on the retina with a high density of photoreceptors and the ability to move their eyes, and thus the localisation of this area of high acuity around the centre of the gaze. This action is driven by visual attention mechanisms (see previous Sec. 5.1.1) and can be learned by means of a saccadic mechanism [START_REF] Daucé | A dual foveal-peripheral visual processing model implements efficient saccade selection[END_REF]) that has been shown to optimise the efficiency of information gain at each saccade (Daucé & Perrinet, 2020). The wide variety of anatomical configurations illustrates that this arrangement is closely related to the behavioural repertoire of each animal [START_REF] Land | Eyes to See: The Astonishing Variety of Vision in Nature[END_REF], and the introduction of such an irregularity, which we refer to here as foveation for simplicity, can provide significant improvements in information processing in event cameras.

As described in Paragraph Human eye and retina, foveation is a visual process in which the eye directs its focus to a particular point of interest, typically by moving or adjusting the position of the eye. In the human eye, the fovea is the central area of the retina responsible for sharp and detailed vision. By directing the fovea towards the point of interest, the visual system can acquire more information and process it with higher accuracy and resolution.

Foveation in computer vision

Foveation is important in various applications, such as image and video compression, where selective compression is applied to regions of low visual acuity, and in robotics, where foveated sensors can reduce the amount of data and processing required for visual tasks. [START_REF] Perrinet | Implementing a retinotopic transform using 'grid_sample' from pytorch[END_REF]. "A picture (extract from the painting "The Ambassadors" by Hans Holbein the Younger) can be represented on a regular grid represented by vertical (red) and horizontal (blue) lines. Retinotopy transforms this grid, and in particular, the area representing the fovea (shaded gray) is over-represented. Applied to the original image of the portrait, the image is strongly distorted and represents more finally the parts under the axis of sight (here the mouth)."

The concept of foveation has been explored in [START_REF] Geisler | Real-time foveated multiresolution system for lowbandwidth video communication[END_REF] in the late 1990s with a foveated multiresolution pyramid with the same objective to reduce the amount of data transmitted in remote vision tasks such as vehicle teleoperation, surveillance, or telemedicine. While a pointing device was used in their experiments to select the foveated area, authors recommend a gaze tracker like in [START_REF] Martinet | Gaze based quality assessment of visual media understanding[END_REF] as a preferable solution to select areas of interest. It has been demonstrated that the retinotopic representation of images (as described in Fig. 5.3) is beneficial in Spatial Transformer Networks among others "to learn CNN to be invariant to affine transforms" [START_REF] Dabane | What you see is what you transform: Foveated spatial transformers as a bio-inspired attention mechanism[END_REF].

Most artificial sensors, such as the CMOS chip that powers the cameras in the average smartphone, have evenly spaced sensors. This is also the case for all event cameras. This is an optimal choice when considering the trade-off between the increasing miniaturisation of each pixel and the growing demand for higher image resolutions. However, the introduction of an irregularity such as the one described above, which we refer to here as foveation for simplicity, can provide significant improvements in information processing in event cameras.

A foveated sensor that mimics the retinotopic layout of a biological retina will generally use fewer pixels than a conventional sensor and will therefore be more energy efficient. Indeed, this would allow us to maintain a high amount of meaningful information while significantly reducing the amount of raw data to be processed. In the particular case of neuromorphic architectures such as SpiNNaker or Intel Loihi, the power consumption is directly proportional to the number of spikes/events processed, and reducing this number is an efficient way to reduce power consumption, a heavy constraint for embedded applications. However, we believe that the development of a mechanism mimicking foveation would greatly improve the processing of event-driven data beyond the energy efficiency aspect. Indeed, recent studies have shown that the particular geometrical layout of the log-polar mapping observed in the human retina has several advantages [START_REF] Hao | Retina-like Imaging and Its Applications: A Brief Review[END_REF]. In particular, a rotation or a zoom is transformed into translations into a log-polar mapping [START_REF] Traver | Designing the Lattice for Log-Polar Images[END_REF]. Further rotation and zoom-invariant processing can for example be easily implemented in a CNN, thus improving the classification performance [START_REF] Jérémie | Retinotopy improves the categorisation and localisation of visual objects in cnns[END_REF]. Moreover, foveation can lead to several improvements in image compression [START_REF] Araujo | An introduction to the log-polar mapping[END_REF], in the efficiency of image registration [START_REF] Sarvaiya | Image registration using log-polar transform and phase correlation[END_REF], or for object recognition [START_REF] Pramod | Human peripheral blur is optimal for object recognition[END_REF]). Yet, it is still not known whether such foveation could be beneficial for event-driven imaging.

A mechanism akin to neuromorphic foveation was introduced by D'Angelo and colleagues in 2020 [START_REF] D'angelo | Event-based eccentric motion detection exploiting time difference encoding[END_REF] and drew inspiration from the Spiking Elementary Motion Detector, which detects optical flow in event data, as well as from varying sizes of receptive fields in the mammalian retina. D'Angelo and colleagues extend this work in order to "decrease the computational resources required, by filtering the number of incoming events into the visual field while maintaining a fine resolution in the centre of the visual field", which comes close to the foveation definition. However, they extrapolate the visual attention from the centre of the optical flowwhich may not contain all the salient elements of the visual scene.

Foveated event-based camera

The introduction of foveation at the sensor level for event-driven sensors should reduce the energy and bandwidth consumption from the sensor level up to the computing system by dynamically allocating more bandwidth and pixel resolution to the RoIs and reducing the information of peripheral (non-interesting) regions. Recently, an electronically foveated DVS has been proposed [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF] where DVS pixels can be dynamically configured in highresolution foveal regions or grouped into low-resolution regions with arbitrary sizes. The sensor can attend in parallel to an arbitrary number of foveal high-resolution regions. Furthermore, the electronic reconfiguration of foveal regions makes it faster and has lower energy compared with the configuration of the centre of the RoI using mechanical control of the sensor position.

Delay learning and detection of spiking motifs

As described in Chapter 2, an SNN is constructed using populations of neurons linked together with connections, with respect to a certain architecture and certain rules allowing it to learn a behaviour. These rules often concern the evolution of the synaptic weight between two preand post-synaptic neurons, i.e. the amount of neurotransmitters released in the synapse following the pre-synaptic neuron's excitation. Hebb's rule can be mentioned: "cells that fire together wire together" [START_REF] Hebb | The organization of behavior: A neuropsychological theory[END_REF].

Delay learning is another of those learning rules which, instead of changing the synaptic weight, changes the delay of the electrical spike propagation in the pre-synaptic neuron's axon [START_REF] Hüning | Synaptic delay learning in pulse-coupled neurons[END_REF]. Depending on the definition, the delay refers to either the time a spike takes to travel through the presynaptic axon, the postsynaptic dendrites or both.

We present below a short review of biological delay learning and neuromorphic models implementing this bio-inspired learning. A more complete review of this topic can be found in our review article (Grimaldi et al., 2022).

Biological delay learning

The first evidence of any neuronal delay in the information propagation within the animal brain came from the interaural time difference, allowing for the azimuthal localization of sound by barn owls. According to Gerstner in 1996, there is a true paradox in auditory neural systems since "neural networks encode behaviourally relevant signals in the range of a few µs with neurons that are at least one order of magnitude slower" [START_REF] Gerstner | A neuronal learning rule for sub-millisecond temporal coding[END_REF]. Various biological experiments have thus revealed the existence of a biological axonal delay precisely adjusted according to variations of parameters in the brainstem (with our without application of plasticity) ; one of them, the "Jeffress model" is presented in Fig. 5.4. We can also mention the Reichardt detector, a model inspired by the drosophila's elementary motion detector for basic motion detection [START_REF] Reichardt | Autocorrelation, a principle for evaluation of sensory information by the central nervous system[END_REF][START_REF] Franceschini | Optic Flow Based Autopilots: Speed Control and Obstacle Avoidance[END_REF] whose mechanism is presented in Fig. 5.5. [START_REF] Kubke | Morphological variation in the nucleus laminaris of birds[END_REF]. Neurons located in the dorsal edge of the nucleus laminaris respond maximally to sounds originating from far contralateral space, and neurons located in more ventral positions respond maximally to sounds originating from the front. Figure 5.5 -Reichardt detector model as introduced in [START_REF] Reichardt | Autocorrelation, a principle for evaluation of sensory information by the central nervous system[END_REF]. Two input neurons A and B connect to a common output. The input from B and A are delayed according respectively to a value δ B and comprised in [δ A1 , δ A2 , δ A3 ]. If the stimulus to detect, moving from A to B with a time t stimulus , is timed with those two connections so that δ Ax = δ B + t stimulus , then the spikes will reach the output at the same time and the model will fire.

Over and over, myelin has been identified as one of the parameters mentioned above. Indeed, this multilaminar coating formed by the glial cells in the Vertebrates' nervous system facilitates both the neural circuit function and the behavioural performance [START_REF] Fields | A new mechanism of nervous system plasticity: activity-dependent myelination[END_REF]. Experiments on mammals show that myelination is directly related to learning and memory consolidation, especially motor training and in enriched environments, both at an early age and in older animals, due to its involvement in coupling the activity of distant neuron populations. It is worth highlighting that myelination becomes increasingly important in larger brains where conductance delays are substantial, and brainwave rhythms are critical ; synchrony errors can lead to neuropsychiatric and neurological dysfunctions. Some further studies show that oligodendrocytes may detect neuronal activity thanks to growth factors or neurotransmitters released through ion channels or via exocytosis. Fig. 5.6 outlines the biological mechanism behind myelination and its impact on learning.

Neuromorphic models for delay learning

Delay learning is a striking example of a computer science concept effectively reinforced by ongoing neuroscience work. Indeed, more and more SNN models are being developed with an impetus to learn by updating delays, not just synaptic weights. One convincing instance among many is given by (Paugam-Moisy, Martinez, & Bengio, 2008), which combines delay adaptation and polychronisation for reservoir computing using an STDP and a supervised delay learning algorithm. [START_REF] Wright | Learning transmission delays in spiking neural networks: A novel approach to sequence learning based on spike delay variance[END_REF] presents a Spike-Delay-Variance Learning which adapts the shape of the postsynaptic profiles according to temporal error. [START_REF] Kerr | Delay selection by spike-timing-dependent plasticity in recurrent networks of spiking neurons receiving oscillatory inputs[END_REF] selectively potentiates recurrent connections with different axonal and dendritic delays during oscillation activity. [START_REF] Matsubara | Spike timing-dependent conduction delay learning model classifying spatio-temporal spike patterns[END_REF] implements an unsupervised delay learning algorithm adjusting conduction delays and synaptic weights. [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF] proposes an STDP extended to the delay learning repeating spatiotemporal patterns described in further detail in the paragraph below. More recently, [START_REF] Grimaldi | Learning heterogeneous delays of spiking neurons for motion detection[END_REF] introduces a supervised SNN model akin to a Reichardt detector learning polychronous groups, i.e. repeated activations of prototypical spiking motifs ; this work was then extended in (Grimaldi & Perrinet, 2023).

Bio-plausible unsupervised delay learning [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF] introduce a nonsupervised learning mechanism of synaptic delays in an SNN model composed of four convolution layers with a kernel size of 5 × 5 and with shared weights and delays (i.e. all the weights and delays of one convolution layer is updated homogeneously). The network is given as input a set of stimuli formed by a dot moving diagonally during 5 timesteps randomly in either one of four directions (northeast, north-west, south-east and south-west -see Fig. 5.7), with additional noise. With such an architecture and input, each convolution layer specialises in the recognition of one specific pattern among the four given as input (see Fig. 5.8, thanks to four unsupervised learning rules described below.

The authors propose an STDP updating both weights and delay as follows:

∆t i,j =t j -t i -d i,j ∆δ i,j = G(t i,j ) =    -B -× exp -∆t i,j σ - if ∆t i,j ≥ 0 B + × exp ∆t i,j σ + if ∆t i,j < 0 (5.1)
where i and j are respectively the pre and post-synaptic neurons, t x is the spiking time of the neuron x, d i,j is the synaptic delay between i and j, ∆t i,j is the delay variation after STDP, B + and B -are the parameters controlling the value of synaptic delay reduction and increment and σ + and σ -are the STDP time constant.

-5.3.2 Neuromorphic models for delay learning 75

Secondly, they implement a homeostasis mechanism based on the neural activity: the activity of each neuron is compared to a targeted frequency ; in case of low activity, their weight will be decreased and delay increased to approach this objective (and vice versa on the contrary) according to the following equation:

K(t + 1) = R target + R observed R target δ(t + 1) =δ(t) -λ δ × K(t + 1) ω(t + 1) =ω(t) -λ ω × K(t + 1) (5.2)
where R target and R observed are respectively the targeted and observed activation frequencies, ω(t) is the current weight, δ(t) is the current delay and λ ω and λ δ are the learning rate of homeostasis mechanism, respectively applied to weights and delays. Additionally, the thresholds are linearly adapted according to the neural activity: the threshold of a neuron is increased when it spikes and decreased otherwise. Finally, they include in their model a stop condition, developed to avoid the convergence of all delays to zero (corresponding to the biological hyper-myelination) and the loss of any temporal feature in the learnt pattern. If one of the synaptic connections of a neuron has a delay δ inferior to the constant c (with c > B -), the learning is stopped and the STDP is no longer applied to any synaptic connections of this neuron.

Contributions

CHAPTER 6

Event data downscaling

This chapter presents our work on embedded event data downscaling. Indeed, embedded systems face limitations in terms of energy resources, memory, computational power, and communication bandwidth. Hence, the increasing use of event cameras in embedded computer vision tasks calls for a way to reduce the amount of events to be processed while keeping the relevant information for the task at hand. We thus believe that a formal definition of event data reduction methods will provide a step further towards sparse data processing. Ten methods will be introduced in this chapter, achieving either spatial or temporal downscaling in order to reduce the number of events in the dataset. For each, we will study the trade-off between their capacity to handle events in real-time, the amount of information kept after reduction and the performance of a computer vision task on the processed data.

Introduction

Neuromorphic computer vision and event sensor-based applications are receiving increasing interest from the computer vision community (classification, detection, tracking, segmentation, etc.), especially for robotics or autonomous driving scenarios.

Downscaling event data is an important feature in a system, especially if embedded, so as to be able to adjust the complexity of data to the available resources such as processing capability and energy consumption. Reducing the size of visual data is straightforward for grayscale and RGB images with interpolation. When it comes to event data, size reduction is much less trivial: event downscaling is a real challenge in neuromorphic computer vision and needs to be formalised as a step further towards sparse data processing.

To this end, we designed different methods to reduce the number of events in a dataset or in an event stream. We decided to tackle this problem by reducing the spatial dimension of the sensor, or by reducing the grain size of the data recording time. In total, we implemented ten event data downscaling methods: six reducing the spatial resolution (designated as "spatial" methods) and four temporally reducing the event flow ("temporal" methods). We aim to design methods which can handle events in real time so that they can be deployed on an embedded system and reduce the input flow of events as they come to ease their processing. Additionally, the spatial methods are developed so that the reduced events correspond to the events that would be produced by an event camera of smaller dimensions than the original, recording the same scene.

In order to study the impact of spatial and temporal downscaling, we compare several features of the resulting data, such as the total number of events, event density, information entropy, optical consistency, computation time and performance of a computer vision task as assessment criteria.

We also carried out what is to the best of our knowledge the first-ever human perception study with event data, where we assess and compare human performance on a gesture classification task. The implementation of this study was the subject of a work placement by three Master students, Ms Lucía Trillo Carreras, Ms Marina Bueno Garcia and Ms Ewa Kupczyk, carried out under our supervision. The contributions of this study are 3-fold: (1) we establish a size threshold under which the human performance falls behind the chance level, (2) we compare several spatial and temporal event downscaling methods and show that all methods give unequal data quality, and (3) we highlight some unexpected discrepancies in a comparison between human vs machine performance.

To the best of our knowledge, the proposed methods make the first attempt to formalise event data downscaling. The methods were implemented in Python * .

The results presented in this chapter were published on three occasions, in the proceedings of the VISAPP 2022 conference (spatial funnelling and log-luminance reconstruction methods - (Gruel, Martinet, Serrano-Gotarredona, & Linares-Barranco, 2022)), the WACV 2023 conference (neuromorphic spatial methods, i.e. SNN pooling - [START_REF] Gruel | Performance comparison of dvs data spatial downscaling methods using spiking neural networks[END_REF]) and the CVPR 2023 Workshop on Event-based Vision (temporal methods and human assessment - [START_REF] Gruel | Frugal event data: how small is too small ? a human performance assessment with shrinking data[END_REF]).

Spatial downscaling

The following subsections present the six methods that we designed and implemented to downscale spatially the data produced by an event camera. It should be noted that a seventh method is introduced but quickly discarded as it does not meet our criterion for embedded data downscaling: indeed it cannot be performed online and requires previous training of an ANN model (see Section 6.2.3). Each method takes as input a sample of events at maximum resolution as well as the reduction factor for this sample and returns the downscaled events as illustrated in Fig. 6.1. The temporal dimension is not affected by the reduction. Depending on the downscaling method used, the number and frequency of the downscaled events will not necessarily equal those of the events occurring in the corresponding fullscale regions. These specific methods are particularly relevant for ML models limited in memory and bandwidth (i.e. limited in terms of the spatial resolution of the data they handle) or tasks requiring multi-scale resolution -see Chapter 8 and the work conducted in (D. [START_REF] Gehrig | Are high-resolution cameras really needed ?[END_REF]). Figure 6.1 -A set of fullscale events captured by a sensor of size height × width is r according to a downscaling factor div, set to 2 in this example. Here, all the events of the original sample occurring in the blue area's pixels will correspond to the events occurring in the green pixel after reduction.

Spatial funnelling

A first intuitive solution to reduce event data is to simply divide each event's coordinates in x and y by the downscaling factor, then remove all the reduced events duplicates (i.e., keep only one occurrence of each event for a specific set of coordinates (x, y, t, p) -see Fig. 6.2). This method follows the same logic as Tonic's downscaling approach, with the slight difference that Tonic keeps duplicates while our implementation discards them.

From a computational point of view, this downscaling method consists simply of updating the memory address of the event's x, y coordinates. Since this relies on one elementary operation repeated n times, with n the number of events processed, it has a complexity O(n).

Thanks to its quickness and simplicity, this solution could answer the need for downscaling events in real time on embedded systems. However, the main drawback is that this process does not drop any event (except the duplicates). The reduced area undergoes a high density of events, yielding an information loss due to the accumulation of pixel activation: the precision of (x, y) coordinates decreases and the fine details of the recorded objects are lost (see Fig. 6.2). Spatial funnelling drastically increases pixel activity (see Fig. 6.2), especially when the downscale factor is significant. In the extreme yet unlikely case where the data is downscaled to a 1 × 1 target size, the resulting pixel is likely to be constantly active. Finally, the mechanism does not correlate with the physical behaviour of an actual event camera: the optical consistency is very low for this method.

Log luminance reconstruction

From an optical point of view, downscaling the spatial size of an event recording boils down to averaging the luminance captured by a subset of pixels in the original sensor. In a physical implementation of an event camera formed by a 2D array of pixels as the one shown in Fig. 3.2, this method could be implemented by averaging the photocurrents I ph generated by neighbour pixels and performing the trans-impedance logarithmic conversion and posterior voltage amplification and differentiation (as explained in Chapter 3). In order to implement this behaviour, the log-luminance is reconstructed for each pixel in the full-scale dataset: each event occurring on the same pixel increases (if positive) or decreases (if negative) the log-luminance level by one unit. The mean log-luminance is then computed for each region of size f actor 2 in the sensor. Lastly, the downscaled events are generated: an event occurs at every timestep where the average logluminance crosses a threshold line ; the event is positive if the gradient is positive, and vice-versa.

To achieve a pixel's log-luminance reconstruction, all events from this pixel are translated into points of coordinates (x shows the timestamps, y is the level reached) where the initial level is arbitrarily set to 0 and the contrast threshold set to 1 (see Fig. 3.3). Figure 6.2 -Schematic illustration for spatial funnelling and event count methods. A sensor of size 2 × 2 outputs negative and positive at each pixel across time, which are downscaled using spatial funnelling (upper red frame) and log-luminance event count (lower green frame). In this example, the downscaling factor is set to 2.

Event count

A first approach to log-luminance reconstruction consists in estimating the mean logluminance value reached by a downscaled pixel every time an event occurs in the corresponding fullscale region. In other words, we realise a normalised event count in the fullscale region as presented in Fig. 6.2. If its value crosses a contrast threshold, an event is produced ; an additional value keeps track of the slope in order to set this event's polarity depending on the slope. It is also necessary to handle the constraint posed by the fact that an event is only produced if the entire threshold has been crossed since the last event. The slope value allows us to ensure this by identifying the extrema, thus preventing events from occurring when the log-luminance crosses a threshold directly following an extremum. Therefore, no event is produced at timestamps 8,13 et 17 in Fig. 6.2, consistently with the native behaviour of an event camera.

Since the complexity of this method relies on the number n of events, it is therefore also O(n). This approach's main disadvantage is the loss of neuromorphic asynchrony. A reduced event can only occur at the timesteps where a full-scale event took place, which would not be the case with actual log-luminance reconstruction, as depicted in the following paragraphs. This drawback is also an asset: this reduction method can easily downscale events online as they are being recorded, and its algorithm simplicity is a plus for an embedded system.

Linear interpolation

The second approach consists in actually reconstructing the log-luminance curve as a polyline, where each event in the log-luminance graph is linked to its neighbours in time by a line as presented in Fig. 6.3. This allows for a quick computation and interpolation of the intersections between the thresholds and the curve.

However, it is not as faithful to the physical reality as we would like it to be: the log-luminance curves are reconstructed as polylines and not cubic curves similar to the actual behaviour of logluminance recorded by an event camera. Furthermore, it has one disadvantage that it shares with the cubic estimation approach described below: both those approaches cannot be computed strictly in real time since the interpolation requires knowing the future behaviour of the curve, i.e. the next events that will occur on the monitored pixel, which are obviously unknown at the current time. In real-time scenarios, these methods require an adaptation: we propose to work around this problem by inserting a slight delay between the data recording and processing.

The proposed adaptation strategies describe the behaviour of the reconstructed log-luminance curves when no input event is given at a time t:

constant strategy: the reconstructed curve remains constant until new events are triggered, follows the equation y = c with c a constant equals to the log-luminance value reached last (see Fig. 6.4) ; -interpolation strategy: the reconstructed curve is interpolated (linear/cubic) given the slope at the location of the last triggered event when it crossed the previous log-luminance threshold (see Fig. 6.5). Note that the interpolation strategy presented in Fig. 6.5 leads to negative values, which is physically incoherent. Indeed, the log-luminance curve recorded by an event camera would not go into the negative (as it is not physically sound) by would have a minimum bound of 0. In our implementation, we made the choice to allow for negative values for the two following reasons:

1. the curves are interpolated for an "unseen future", i.e. for an unknown time up until the arrival of an event at the corresponding pixel. We thus expect the next event to arrive quickly enough for the curve not to go too low into the negatives.

2. the curves are interpolated using a null log-luminance value at the origin. Since we do not know when the event camera began to record, this value is arbitrary and we could decide to set it at a much higher value, thus preventing the issue of interpolated negative log-luminance value. The only information we need is the time at which the interpolated curves cross the thresholds ; here we set the cross with the origin at 0 to facilitate the reader's comprehension.

This note also concerns the interpolation strategy applied to the cubic lo-luminance reconstruction described below (Fig. 6.8).

Cubic interpolation

In this last approach, the log-luminance of each neuron is interpolated as a cubic curve strictly running through the events on the log-luminance graph as seen in Fig. 6.6. The cubic interpolation was implemented using Scipy's method PChipInterpolator, an implementation of the Piecewise Cubic Hermite Interpolating Polynomial. This algorithm is a third-degree piecewise polynomial function which seeks to match the derivatives of each point with its neighbours. Thus its extrema coincide with the data's extrema, which is not the case for all cubic interpolation algorithms that tend to greatly overshoot [START_REF] Rabbath | A comparison of piecewise cubic hermite interpolating polynomials, cubic splines and piecewise linear functions for the approximation of projectile aerodynamics[END_REF]. Similarly to other cubic interpolation algorithms, this cubic interpolator I takes as input a set of values (x, y, x ′ ) and returns the y ′ data interpolated for all x ′ values with respect to the relation between y and x: In our case, the coordinates are those of the events in the corresponding log-luminance graph: the x-axis corresponds to the timestamps and the y -absciss to the events' log-luminance. From a set of known log-luminance levels x ′ , the cubic interpolation outputs the timestamps y ′ where the events' log-luminance crosses those levels. This approach reconstructs a curve close to an actual log-luminance curve recorded by an event camera, granting a high optical consistency to this approach. Indeed, this method emulates the real-time signal averaging among neighbouring pixels that is implemented in hardware in the electronically foveated DVS proposed by our collaborators at IMSE (Serrano-Gotarredona et al., 2022). That way, the log-luminance reconstruction generates the output event flow that most closely resembles the output event flow obtained by the electronically foveated DVS real hardware observing the same scene driven by the same attention mechanism.

I(x, y, x ′ ) = y ′ with x ′ = ⌊ xmax threshold ⌋ x=⌊ x min threshold ⌋+threshold (
However, the existing tools allowing the cubic interpolation of y values according to a set of x parameters require the x set to be strictly monotonous, without any duplicates. The first hinder can be removed by splitting the (x, y) dataset presented in Eq. 6.1 at the different log-luminance's extrema:

S i (x, y) = next i=0 ty ext i+1 t=ty ext i (x t , y t ) (6.2)
with S i the i th subset of the events comprised between two extrema, n ext the total number of extrema in the dataset i.e. the number of segments S i , and t yext i and t yext i+1 respectively the timestamps of the i th and (i + 1) th extrema.

The second hindrance mentioned above is more complex to avoid: although it is a rare occurrence, multiple events still happen at the same coordinates and in the same timestamps. This The handling of event data in real-time by the cubic log-luminance interpolation method follows the strategies described for linear log-luminance reconstruction: the behaviour of the constant strategy is described in Fig. 6.7 and the behaviour of the interpolation strategy in Fig. 6.8.

Events to video to events

Rebceq and colleagues introduced in [START_REF] Rebecq | High speed and high dynamic range video with an event camera[END_REF] a method to reconstruct grayscale videos from an event stream using a recurrent neural network trained on a large amount of simulated event data. Using this model, a simple technique would be to use standard image downscaling methods on grayscale frames reconstructed from the event data and then translate this reduced image back into events. This is equivalent to reconstructing a downscaled luminance and computing downscaled events from it, but it has some important drawbacks. The main one lies in the complete loss of event data's asynchronicity since the grayscale video produced from the event data is built from the accumulation of information into frames. This method needs two sets of grayscale data to be created during the process (fullscale from events, then downscale) leading to high memory consumption, while embedded systems are limited in data storage. Moreover, its implementation in practice is computationally heavy, which is not suited for embedded applications. Lastly, the chance of errors in event data reduction is quite high due to the two approximations and one interpolation required by this method.

A pipeline implementing such a downscaling process is the following:

1. translate the event data into grayscale frames thanks to Events-to-Video library introduced in (Rebecq, Ranftl, Koltun, & Scaramuzza, 2019a). This module actually consists of a ML model trained on a set of datasets comprising neuromorphic and grayscale recordings of the same scenes.

2. rescale the grayscale frames thus obtained with a standard interpolation using one of the many existing image processing tools.

3. finally translate the downscaled grayscale frames into events by using the library Videoto-Events from (D. [START_REF] Gehrig | Video to events: Recycling video datasets for event cameras[END_REF]. This library reconstructs the log-luminance variations between each grayscale frame, and then outputs the corresponding events according to user-defined negative and positive contrast thresholds.

Some works implement multi-scale resolution of event data using part or all of the pipeline described above. For example, the authors in (D. [START_REF] Gehrig | Are high-resolution cameras really needed ?[END_REF] implement steps 2 and 3 to assess the evolution of the performance of various computer vision tasks.

This pipeline requires the event dataset one wishes to downscale either to be one of the datasets the Events-to-Video model has been trained on or to contain a set of grayscale frames recorded simultaneously as the event data. Since this method is very arguable considering the motivations for event downscaling, we did not pursue it further during this work. Note that the dataset used in our experiments does not meet either of these two criteria (see Section 3.3).

SNN pooling

This subsection presents two novel SNN architectures for spatial event data downscaling by SNN pooling, introduced in [START_REF] Gruel | Performance comparison of dvs data spatial downscaling methods using spiking neural networks[END_REF]. Here, our core objective lies in designing a fully integrated energy-efficient SNN model able to run dynamically on a neuromorphic platform. We claim the importance of providing downscaling methods for event data in order to filter out events and reduce the data flow with care, as this preprocessing step should be achieved while preserving the information conveyed. The design of such a model is not straightforward, and it requires some hyper-parameter tuning to achieve the results presented below. As far as we know, it is the first time an SNN has been studied as a method to downscale event-based data. Knowing that the combination of SNNs with event-based cameras seems to prove its worth in terms of compatibility of data types [START_REF] Cordone | Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks[END_REF], information retention [START_REF] Hagenaars | Self-supervised learning of eventbased optical flow with spiking neural networks[END_REF], energy efficiency [START_REF] Cordone | Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks[END_REF][START_REF] Debat | Event-based trajectory prediction using spiking neural networks[END_REF] and processing latency [START_REF] Debat | Event-based trajectory prediction using spiking neural networks[END_REF] reinforces the relevance of studying event data reduction using SNNs.

In some ways, this event processing has often been used by SNN models handling neuromorphic data: indeed the use of a stridded convolution or max pooling layer e.g. in [START_REF] Cordone | Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks[END_REF] is close to event downscaling by SNN pooling, as we present here. The events are directly translated as input spikes in a 2D layer of size height × width connected to a smaller 2D layer of size height ratio × width ratio , which implements a convolutional layer with a kernel size ratio × ratio, a stride ratio, without padding. Each spike output from this smaller layer is then interpreted as an event, with each neuron construed as a pixel of a smaller sensor. The connection between each output neuron and the corresponding input region is fully connected (i.e., all the spikes produced by the region are sent to the same neuron). As we will describe in the following paragraphs, we implemented two SNN architectures depending on whether the positive and negative events are considered separately or not.

Both methods were implemented in this work using the Python SNN simulator PyNN [START_REF] Davison | Pynn: a common interface for neuronal network simulators[END_REF] to produce downscaled datasets for further comparisons. However, since this library runs simulations on CPU, it leads to a significant run time. We thus adapted the networks to be executed on the SpiNNaker neuromorphic chip (S. Furber & Bogdan, 2020): we chose SpiNNaker as a simulator instead of NEST (which runs on CPU) and update the implementation of the fully connected connections to be comprehended by the sPyNNaker library.

Separate handling of polarity

A first method of SNN pooling is proposed, where the positive and negative events are handled differently, although during the same simulation. The architecture used is depicted in Fig. 6.9: the 2D input layer is of size height × width × 2 with two 2D grids of LIF neurons stacked together, each one receiving either positive or negative events. Likewise, the output layer is of size height ratio × width ratio × 2 ; each of its LIF neurons is wired to the corresponding input region by an excitatory connection. No learning is necessary here: the weight ω of the connection between the input and the output is set after fine-tuning phase (as is the sensitivity of an event camera recording a scene -see below) and should not be adapted along the simulation. Spatially downscaling using SNN pooling relies entirely on the LIF neuron's intrinsic dynamics, which come close to the behaviour of a DVS pixel (see Chap. 4).

Most SNNs taking events directly as input, with no intermediate preprocessing of the data, translate each of them into a spike at the corresponding coordinates in the input 2D layer of the network with no distinction between positive and negative events ; any time a pixel is triggered, a spike is emitted by the corresponding input neuron (see [START_REF] Cordone | Learning from Event Cameras with Sparse Spiking Convolutional Neural Networks[END_REF]). This may not have an impact on some computer vision tasks, but in our case we aim to reduce event data while keeping the information as close to the original one as possible, thus preserving polarity. From this stems our choice for the particular architecture described above.

However, separately handling the event polarities disregards the physics behind event cameras and embodies the main flaw of this first architecture. In the case of a burst of negative and positive events grouped together and recorded by the same pixel, the luminance is actually quite stable and varies around one value with a variation range slightly higher than the sensor's sensitivity threshold. As can be seen in Fig. 6.10, downscaling this data should lead to a smaller luminance variation range, thus to fewer events produced since they compensate each other.

Mutual influence of positive and negative events

To resolve the aforementioned problem posed by separate processing of polarities, we implemented a second architecture illustrated in Fig. 6.9 (including the grey dotted box). It builds on the structure of the first one and adds to it a mechanism of mutual inhibition between the polarities. Each receptive field of the input layer activates the corresponding output neuron of the same polarity, but it also inhibits the corresponding output neuron of the opposite polarity with the same weight. This way each activation of the output neuron's membrane potential due to an event of a certain polarity is offset by any potential event of the opposed polarity taking place in a close time span to obtain the expected behaviour presented in Fig. 6.10.

This represents more faithfully the behaviour of an event camera of a smaller resolution, recording the same luminance as the sample being downscaled. It should also be noted that the sensitivity of this simulated recording of a downscaled luminance can be adapted simply by adjusting the weights of the connections, as easily as adjusting the sensitivity of a DVS: the stronger the weights, the higher the activation of the output neurons and the more events produced, and reciprocally. Both methods described above will respectively be referenced to as "separate SNN" and "mutual SNN" methods in the following pages.

Influence of hyperparameter tuning on SNN pooling

As with most SNNs, hyperparameter tuning greatly influences the activation of the different layers and the overall spiking dynamic. Hyperparameters include the synaptic weights, the neurons' thresholds, the membrane time constant, etc (see Section 2.1.3). This hyperparameter tuning is not inconsistent with the physical operation set up of the DVS: indeed in a DVS camera, different parameters can be set to record a scene, such as the contrast threshold or the refractory period.

It was thus necessary to study the activity measures of both separate and mutual architectures described in the previous section with varying hyperparameter tuning in order to select the most optimal one. As seen earlier, many hyperparameters can be tuned in order to obtain the ideal results. Here, we limit our study to the synaptic weight ω (see Fig. 6.9) since we wish to simulate the behaviour of a smaller event sensor and the weight can be assimilated to a DVS contrast threshold. The rest of the hyperparameters are set to values commonly used to initialise a LIF Figure 6.9 -Architectures of separate and mutual SNN downscaling methods. The two architectures proposed by the authors for SNN Pooling either handle the positive and negative events separately (excluding the grey dotted box in the figure) or take into account their mutual influence (including the grey dotted box in the figure). The green arrows represent One-To-One excitatory connections between an input receptive field and the corresponding output neuron of the same polarity, and the red arrows All-To-All inhibitory connections between different polarities. The synaptic weight ω is homogeneous. The green dots correspond to positive events and the blue dots to negative events. The first step of this model is to translate these input events into input spikes while separating the polarity ; the output spikes are translated back into events in the final step. In this example, both architectures are applied to data recorded with a sensor size of 128 × 128 pixels, which is downscaled by a factor of 2. The output layer of each polarity is thus of size Figure 6.10 -Schematic illustration for SNN pooling. A sensor of size 2 × 2 outputs events both negative (blue ticks) and positive (green ticks) at each pixel across time, which are downscaled with SNN pooling handling separately the events (green frame on top) or allowing a mutual influence (pink frame at the bottom). In this example, the downscaling factor is set to 2. Best seen in colours.

6.2 -6.2.4 SNN pooling 93 neuron in an SNN (presented in Tab. 6.1). As pictured in Fig. 6.11, a strong ω will allow for a stronger activation of the output layer, thus letting more events through the connections and leading to a higher output event density. If we consider the standard deviation of the density per pixel as a measure of information provided by the event data, we could set an upper and lower bound for the hyperparameter in order to ensure the production of suitable downscaled data (i.e. which output number of events is smaller than the input, but still high enough to provide information, and which output spatial density is proportional to the input one). Conversely, a lower bound should also be provided in order to keep enough events in the output to have a minimum of information. This is coherent with the contrast threshold used to set up a DVS camera: the higher the threshold, the more events are filtered and the less information the event data provides. Hyperparameter tuning is therefore an important aspect to keep in mind when using SNNs to downscale events. Fig. 6.11 also interestingly shows that the mutual SNN tends to output fewer events in a lower temporal density than separate SNN, which will be confirmed in Fig. 6.14. Indeed the mutual inhibition tends to normalise the membrane potential of each neuron (see Fig. 6.10), which can be assimilated to the log luminance recorded by each sensor's pixel. In order to investigate the relevance of the SNN downscaling method for real-time event handling, we implemented both SNN downscaling methods on the SpiNN-3 board (see Section 2.2.2). The run time of this method thus suffers from the limitations of the SpiNN-3 board. The number of input events per simulation time-step does not influence the simulation run time, contrary to the number of neurons and connections. As described in Eq. 6.3, the proposed architecture is convolutional, meaning that the number of connections c depends on the number of neurons n input and n output in both layers as well as on the convolutional kernel size kernel. The latter is actually the same as ratio in the proposed topology: kernel = ratio. Interestingly, as described in Eq. 6.3 below, for a given width and height, the number of connections c does not depend on ratio, whereas ratio strongly influences the number of output neurons to be simulated on the board.

n input = 2 × height × width n output = n input ratio 2 c = kernel 2 × n output = n input (6.3)
Technically, PyNN uses a SpikeSourceArray (resp. a SpikeInjector) data structure for off-line (resp. online) neurons that do not need to be simulated by the SpiNN-3 board. This important detail means that although the architecture requires n input + n output neurons, the board will only need to simulate n output neurons and their c connections. Furthermore, the SpiNN-3 board does not handle the definition of connections over a smaller group of neurons than the whole population, which hinders the implementation of the connections in our models. This issue was bypassed using the FromListConnector which allows to set the weights between each neuron of the input and output layer in an All-To-All connection, thus enabling the convolution by setting some synapses weight to zero (University of Manchester, 2021). However, this solution requires the implementation of synapses that will not be used during the simulation ; this reduces the overall amount of available synapses that can be implemented on the SpiNN-3 (which upper bound is set to 1,000 synapses according to Section 2.2.2) and might prevent scaling up to larger sensors. Depending on whether the separate or the mutual SNN downscaling version is used, the number of connections varies ; indeed the mutual method adds an inhibitory connection between neurons corresponding to positive events in input to those corresponding to negative events in output and reciprocally. Thus the number of connections c doubles between the separate and mutual SNN methods.

Considering all this information, the simulation run time should decrease when the reduction factor ratio increases for both SNN downscaling methods, while the mutual SNN computation time is likely to remain higher than the separate SNN. This is confirmed below in Section 6.4.2 and Fig. 6.17.

Temporal downscaling

In this section, we introduce four temporal downscaling methods, reducing the data flow by sub-sampling events and with no impact on the spatial resolution. These methods are adapted to scenarios that highly differ from the ones where a spatial downscaling method would be used (i.e. in cases where a model would not have enough memory to handle an input layer of the sensor size): we believe their use is especially relevant for embedded models, whose memory and computing power may not be sufficient to record and process all events at the time of their arrival, but which do not necessarily have a limit on spatial resolution.

Temporal funnelling

The temporal funnelling method comes close to Tonic's temporal "Downsample" method [START_REF] Lenz | Tonic: event-based datasets and transformations[END_REF]. Similarly to the spatial funnelling method we implemented (see Section 6.2.1), it involves the pooling of events according to their timestamp rather than spatial coor-6.3 -6.3.2 Structural downscaling 95 dinates and removes any duplicates. Event data is expressed with a µs temporal precision. Pooling them according to their timestamp amounts to keeping only one event of the current polarity at the current pixel in the time window defined by a temporal factor t (see Fig. 6.12), which leads to changing the temporal precision.

We establish two types of temporal funnelling (see Fig. 6.12):

-asynchronous temporal funnelling: each event kept per pixel, per polarity and per timewindow maintains its original timestamp, with a thinner temporal grain than the one defined by t. -synchronous temporal funnelling: each event is kept but with a timestamp rounded to the end of the current time window set by t. This aims to illustrate the behaviour of an embedded system which would accumulate the kept events into frames once all the events of the corresponding time window are processed.

Figure 6.12 -Schematic illustration of the temporal funnelling downscaling method. A sensor of size 2 × 2 outputs negative and positive events at each pixel across time, which are downscaled using the asynchronous (upper red frame) or synchronous temporal funnelling (bottom green frame) methods with a factor t = 5.

The main disadvantages of this method are the induced loss of temporal precision and the increased event density. As for the spatial funnelling, this amounts to simply updating the content of the memory address of the event temporal coordinates, thus having a complexity O(n) (with n the number of events).

Structural downscaling

We subdivide the structural downscaling methods into a deterministic one and a stochastic one. The deterministic method (see Fig. 6.13 top) keeps every k th event out of N . The stochastic method (see Fig. 6.13 bottom) filters events with a probability p. This strategy has the benefit of maintaining the original time and space scales. It handles events efficiently, simply deciding whether to keep or discard each new event.

It should be highlighted that this strategy simulates the real-world situation where an embedded system is overflowed with a dense event stream and is not able to process them all: some events will just be dropped [START_REF] Gallego | Event-based vision: A survey[END_REF].

One can easily compare both structural downscaling methods by converting k in p as described in Eq. 6.4.

p = 100 × 1 k (6.4)
Figure 6.13 -Schematic illustration of structural downscaling methods. A sensor of size 2 × 2 outputs negative and positive events at each pixel across time, which are downscaled using the deterministic structural (upper red frame) with a factor k = 6 ; and the deterministic structural (lower green frame) with a factor p = 60%.

Experimental comparison

To assess the performance of the different methods of event data downscaling described above, we applied them to the neuromorphic datasets DVS128 Gesture (Amir et al., 2017) and N-MNIST [START_REF] Orchard | A spiking neural network architecture for visual motion estimation[END_REF], widely used as a benchmark in event data processing, and we assessed the results according to the criteria described in the following sections. These datasets were chosen according to their relevance to our motivations: DVS128 Gesture represents dynamic information to be processed temporally, which would be the case in most embedded systems. As for N-MNIST, it is particularly heavy in events: according to the parameters presented in its description, N-MNIST displays on average 10.71 events per pixel per second, whereas DVS128 6.4 -6.4.1 Comparison according to activity measures 97 Gesture only displays about 4.06 events per pixel per second (see Sections 3.3.1 and 3.3.2). All in all, they are different but complementary in many aspects (sensor size, amount of events, visual information, etc). The spatial downscaling of event data aims to significantly reduce the number of events per sample while keeping a sufficient amount of relevant information. Table 6.2 compares the six spatial downscaling methods presented above and the Tonic's "Downsample" tool according to 5 criteria described in the following section. The mean number of events per sample was calculated over the whole DVS28 Gesture dataset at full scale and downscaled by a reduction factor 4.

Comparison according to activity measures

Events' temporal density D t corresponds to the activation probability of pixels averaged over the whole sensor:

D t = x∈{1,w} y∈{1,h} P x,y w • h (6.5)
where w and h are, respectively, the width and height of the sensor. The activation probability P x,y of one pixel of coordinates (x, y) is calculated as the number of events (positive or negative) occurring at a given pixel divided by the time length of the sample:

P x,y = t∈{t min ,tmax} δ(x t , x) • δ(y t , y) t max -t min (6.6)
with t min and t max respectively the minimum and maximum timestamp of the sample, and δ the Kronecker delta function, which returns 1 if the variables are equal, and 0 otherwise. If we consider the information brought by positive (or negative) events according to Shannon's information theory, we can estimate the entropy H (in bits) of the dataset as the average over all pixels and polarities p of the pixel entropy H x,y,p :

H = p∈{-,+} w x=1 h y=1 H x,y,p 2 • w • h with H x,y,p =
Pe∈{Px,y,p,Px,y,p} -P e • log 2 (P e ) (6.7)

where P x,y,p is the probability of event (probability for the pixel of being active), and P x,y,p = 1 -P x,y,p is the dual probability of no event, for the polarity p at location (x, y).

Overall, we believe that an event camera provides interesting information when the sensor is neither too little nor too much activated, i.e. when the pixels have an activation chance close to 50%. H x,y,p becomes lower when the pixel is either never active or always active. Entropy values in Table 6.2 are averaged from both polarities.

Finally, the computation time of each method presented in this paper was calculated on a panel of over 500M events. The values presented in Table 6.2 correspond to the average calculation period for the downscaling of 100K events, expressed in seconds. Optical consistency is a qualitative estimation of the resemblance between the method implemented and the actual behaviour of an event camera of smaller resolution recording the same scene ; this estimation was performed by our collaborators at IMSE, specialising in micro-electronics and event cameras.

It should be noted that the values produced for the SNN Pooling method are not obtained from the same panel of event data used by the other methods. As a matter of fact, since the computation time for one sample using either technique exceeds 10 min, we chose to assess our criteria on a smaller set of downsampled events using SNN Pooling.

The values of these criteria confirm the hypotheses we outlined above. Tonic's "Downscale" method yields the highest number of events, thus the highest event density ; this spatial reduction does not lead to a significant drop of events and thus does not answer our need for reduced data to handle. The same goes for spatial funnelling, which ignores all event duplicates. Even though the computation time of those two methods is low, the output data is almost as large as the input. We should also point out that the information entropy of the spatial funnelling is quite low due to the constant activation of the sensor's pixels. The log-luminance techniques are the most consistent with the actual behaviour of event cameras. It drops a large amount of data, which might significantly improve any embedded computation, while maintaining a relatively high level of information entropy as intended. SNN Pooling is also promising by its behaviour, closely related to one of an event camera, but it is extremely sensitive to the weight of its connection: here the weight is probably too strong hence its poor results. Fig. 6.14 shows that the different event spatial downscaling methods implemented vary strongly by their activity measures. The plot on the left shows quite clearly that the number of events per sample decreases to a varying extent depending on the downscaling method used. Since we aim for a drop in the number of events post-reduction, the least convincing method by this criterion Figure 6.14 -Comparison of activity measures between different methods and reduction factors for DVS128 Gesture (above) and N-MNIST (below) datasets. The y-axes are in log scale. Since N-MNIST has a lower resolution than DVS128 Gesture, we assess its activity measures up until a reduction factor of 10 (instead of 50). We see here that the funnelling method keeps a high number of events and has an increasing temporal density, whereas all the other methods (except for SNN pooling) strongly decrease the number of events kept. The log luminance reconstructions lead to the decrease of temporal density, while the event count maintains one similar to the original data.

is obviously the funnelling one: the number of events stays stable no matter the reduction factor, whereas the most pronounced drop is obtained with the log luminance reconstruction methods.

The plot on the right represents the temporal event density, which we additionally study as the number of events is not a sufficient criterion for deciding which method provides the best preservation of information with an increasing reduction factor. This criterion shows the distribution of the reduced events over the sensor: a high temporal density corresponds to event data produced by a highly triggered sensor. Fig. 6.14 shows that the funnelling and event count methods have a high temporal density, i.e. all the sensor's pixels tend to trigger a lot (significantly more than 50% of the time) across time thus reducing the information transmitted. A contrario, the log luminance reconstruction seems to provide informative data, and both SNN methods seem a good compromise between these two extrema.

Since the spatial downscaling methods are more likely to be applied on an embedded system, it is interesting to consider the computational cost of each method, which give an estimate of the energy consumed. The funnelling method reduces data with nearly zero energy, as the funnelling can be applied in the event transmission from the sensor to the processor by just ignoring some bits in the address of the events (and removing any duplicates). On an embedded computer, the log luminance reconstruction methods are implemented using an algorithm of higher complexity thus requiring a greater energy expenditure. However, if we consider this reduction at the sensor level, it can be produced at zero energy cost. Finally, the SNN methods are to be directly performed by an embedded neuromorphic chip, whose energy consumption is directly proportional to the number of events given as input and the number of synapses implemented between the input and output layer, i.e. proportional to the reduction factor.

Comparison of real-time downscaling

In this section, we propose to compare the different spatial downscaling methods we implemented regarding offline and online data handling. Indeed, while event sensors show a number of advantages over frame-based cameras such as high temporal resolution, low energy consumption, high dynamic range sensing, etc, that make them unavoidable for the future of real-time embedded vision systems, the limited processing resources make it often hard to deal with a high event density in real-time embedded vision applications when handling events as they come † . This dilemma calls for data reduction techniques able to handle input events in real time. We investigate the time each of the methods introduced precedently takes to process 1s of input data from the DVS 128 Gesture dataset while preserving the quality of the information provided by the output data. The run-time is calculated both for the "real-time" processing of events, i.e. the event data is reduced as it is being recorded, or for the "deferred" processing, where the event data is first recorded as a whole and then downscaled. Indeed, some methods need a different implementation depending on this choice of temporality (see the description of log-luminance reconstruction methods above). Note that the online downscaling time of temporal methods, which were implemented more recently, could not be measured at the time of redaction of this manuscript. The online and offline event downscaling studied in this section was run on an Intel 8-core i9-10885H CPU at 2.4GHz. Figure 6.15 -Run time evolution for offline spatial and temporal downscaling of 1s of real-time event data. The dashed line shows the real-time bound. The SNN methods are run on SpiNN-3. Run offline, the spatial funnelling method is the quickest one as expected. Considering the temporal methods, all allow for quicker-than-real-time downscaling. Fig. 6.15 confirms the results presented in Tab. 6.2: spatial funnelling is definitely the quickest method among the spatial downscaling ones, followed closely by the event count. Among the temporal methods, the funnelling methods run with the same time and are the slowest out of the four -however, it still outperforms the log-luminance reconstruction methods and the SNN pooling. The deterministic structural method outruns all methods (spatial included). Furthermore, Fig. 6.15 shows that the temporal and structural methods do process events online relatively fast and stably. When the temporal reduction factor increases, the deterministic structural factor k increases or the structural stochastic filtering probability decreases. Figure 6.16 -Run time evolution for online spatial downscaling of 1s of real-time event data. The dashed line shows the real-time bound. The SNN methods are run on SpiNN-3, while the logluminance reconstruction methods are evaluated for the two strategies constant and interpolation. The SNN methods are undoubtedly the quickest, but still two orders of magnitude slower than real-time. Fig. 6.16 illustrates the computational run-time evolution. It should be noted that this CPU can only perform a maximum of 2.4 billion cycles per second, whereas an event camera can output up to 1M events per second per pixel. For the DVS 128 Gesture dataset, the event camera is of size 128 × 128 pixels, thus has the capacity to output up to 16B events per second ‡ . The DVS 128 Gesture comprises an average of 30k events per second, as described in Section 3.3.1. The speed of our CPU is thus too low to simply cycle through the events in real time, so the real-time reduction is unattainable using this hardware. ‡. The number of 16 billion events per second is hypothetical and corresponds to the worst case scenario where the DVS pixels are constantly triggered during the whole second. It has been calculated as follows: 1M events per second per pixel × 128 2 pixels = 16, 384M events per second ≈ 16B events per second.

It is interesting to note that while the event count is slower than the funnelling when run offline on the whole dataset at once (see Fig. 6.15), it is quicker in an online scenario. This is due to the fact that the event count needs by definition to be run timestamp per timestamp -which is how we believe any online downscaling method should be run -thus giving it an advantage over the spatial funnelling. As for linear and cubic log-luminance reconstructions, they suffer from an important drop in performance accuracy and an increase in run time when the reduction factor increases ; but it is to be noted that the constant strategy is significantly quicker than the interpolation one. Figure 6.17 -Time required by both SNN downscaling methods to process one second of input events (left) or 1000 incoming events (right) while run on the SpiNN-3 board, with a previous asynchronous temporal funnelling of the input data.

The simulation run time for the SNN downscaling methods on SpiNN-3 is displayed with more details in Fig. 6.17. We combine the SNN pooling methods with the asynchronous temporal funnelling to reduce event data both spatially and temporally in order to assess the limitations of the SpiNN-3 board in terms of simulation time for the simple networks that are the SNN pooling methods. Note that this is one of many method combinations for spatiotemporal reduction that one could implement using the methods we introduced. We downscale the event data temporally by a factor t ∈ [1, 1e 3 , 1e 6 ] (in µs). The temporal factor t = 1 corresponds to the absence of preliminary temporal reduction.

The time required to spatially downscale 1s of input data in real-time without preliminary temporal reduction is significantly higher than 1s, even when the reduction is run on the SpiNN-3 board. This seems to point towards a limitation of the SpiNN-3 board: to simulate a simple network composed of two layers (one of which is the input, thus not simulated as SNN by SpiNNaker) and one to two sets of connections, the board requires minimum time significantly greater than 10 -6 seconds (grain of an event timestamp) to simulate a time step. SpiNN-3 therefore seems to require a prior temporal reduction in order to be able to simulate event data in a reasonable time. Indeed, Fig. 6.17 shows that a temporal funnelling of at least 10 3 is needed.

As discussed previously, the simulation run time decreases when the reduction factor increases. Surprisingly, we do not observe a significant difference in run time between mutual SNN and separate SNN methods. Similar behaviours are observed when studying the evolution of the runtime needed by the SNN methods to process 1000 input events, i.e. around 30ms of input data.

Comparison according to performance on a classification task

In the best-case scenario, an optimal spatial downscaling method for event data drastically reduces the number of events while conserving relevant data. This section aims to discuss the relevance of our proposed approaches compared to existing ones regarding this matter. Activity measures are studied above as a first attempt to answer this interrogation. We further investigate the relevance of the retained information by looking at the performance of each method against the specific computer vision task of classification.

We have pre-trained the classifiers implemented using SLAYER and PLIF (that will be designated as "SLAYER benchmark classifier" an "PLIF-based classifier" in the remainder of this chapter -see Section 4.3.1) on the full-scale DVS 128 Gesture and N-MNIST datasets. In order to optimise the simulation run time, we loaded and fine-tuned this classifier's weights in a transfer learning strategy on the downscaled datasets.

SLAYER benchmark classifier

We apply the SLAYER classifier (see Section 4.3.1.1) on the DVS128 Gesture and N-MNIST datasets reduced using the six spatial methods introduced previously § . The resulting performances are presented in Fig. 6.18. The optimal downscaling method should have the best accuracy performance since it retains the most relevant information necessary for effective classification.

Unlike other classification models which accumulate input events into frames (such as the PLIF-based classifier (Fang, Yu, Chen, Masquelier, et al., 2021) -see subsection below), the SLAYER benchmark classifier takes as input sparse events and shows more interesting and coherent results than the PLIF-based one. Fig. 6.18 shows that the downscaling methods introduced in this paper perform well, especially for the DVS 128 Gesture dataset. Regarding N-MNIST, as seen earlier, the large number of events retained for a small reduction factor depends entirely on the SNN pooling hyperparameters, which should be tuned for each new dataset. Moreover, this shortcoming is compensated by the significant advantage of fast computation time for SNN pooling compared to other methods (see Fig. 6.16). In contrast, the "Spatial funnelling" remains on the lower right even when the reduction factor increases ; moreover, past a small reduction factor (2), its performance is lower than most methods and the number of retained events is higher. It should be noted that all methods lead to a decrease in performance when the reduction factor increases, which validates the activity measures study performed in the above section.

PLIF-based classifier

Fig. 6.19 presents the performances obtained using the PLIF-based classifier (see Section 4.3.1.2), which leads to the conclusion that the funnelling method is by far the most appropriate downscaling method, at least in the context of a classification task. This is quite surprising compared to the results obtained using SLAYER (see Fig. 6.18. An explanation of this difference can be found in the fact that the PLIF-based classifier, which takes in input frames, will have a better recognition performance when the number of events is higher, as the corresponding reconstructed frames will provide more information than in a sparser case. Indeed classifiers based on convolution such as the PLIF-based one tend to underperform on sparse data: they can handle §. Due to unforeseen computational problems, we were unable to test SLAYER on the more recently introduced temporal methods. However, these are evaluated on the PLIF-based classifier in the next section. Figure 6.18 -Evolution of the accuracy performance of the SLAYER benchmark classifier applied to DVS128 Gesture (first row) and N-MNIST (second row) when the spatial diving factor increases (columns). The dashed line connects the original data to the origin (0, 0) of the reference system. Note that the points located in the upper left side of the dashed line correspond to results showing interesting trade-offs for performance over data size, i.e. a high accuracy with a small number of events. Globally, most spatial downscaling methods lead to a good trade-off, with the notable exception of the "Spatial funnelling" which remains in the lower right side even when the reduction factor increases. between what we aim for -a decreased number of events for a significantly good accuracy (upper left) -and what we do not aim for -a decreased accuracy for a great number of events (lower right). Globally most results are in the part that we aim for, with the notable exception of the "SNN pooling" results. fullscale event data but quickly lose performance when the number of events drops, i.e. when the data is reduced using any methods except funnelling. The accuracy performance of the PLIF-based classifier obtained on DVS 128 Gesture downscaled temporally using the structural and funnelling methods are presented respectively in Tab. 6.3 and 6.4. According to these results, these methods maintain a quite high amount of information even with a high reduction factor: keeping only 0.05% of the original events, the two structural methods still reach an accuracy exceeding 60% ; when accumulating events over 5s (i.e. more than 80% of the average sample length), the temporal funnelling methods both remains at 95% accuracy.

An interesting aside worth highlighting is that a large part of the motion information is encoded in the event data polarity, as pointed out in [START_REF] Oudjail | Bio-inspired event-based motion analysis with spiking neural networks[END_REF]. In some examples, the motion direction can easily be estimated using simply the spatial information (i.e. the repartition of events of different polarities in space) without the help of temporal information. As explained in Chapter 9 and more specifically in Fig. 9.16, the polarities displayed on a single frame reconstructed from 50ms of a DVS 128 Gesture sample allow us to identify the direction of the hand motion. We confirm this intuition by comparing the results obtained on a classification task on event data with only available polarity using the PLIF-based classifier, a model known to translate events into 2-channel frames (one channel per polarity) and then train on those frames.

We observe in Fig. 6.20 that the PLIF-based classifier did not obtain the same results whether the classification for DVS128 Gesture is performed on both polarities or solely one (here, negative). Instead of learning the dynamics of the events, the PLIF-based classifier easily recognises classes by identifying the direction of the arm's movement according to the distribution of the positive and negative events on the frame. This questions the hardness of the dataset as well as underlines the need to process event data using a temporal model, able to rely on the temporal evolution of data and/or features. Figure 6.20 -Evolution of the accuracy performance of the PLIF-based classifier applied to DVS128 Gesture when the spatial reduction factor increases, with both polarities and only on negative events (clear lines).

Human assessment study

In this chapter, we introduced a wide range of downscaling methods for event data and assessed their usefulness by comparing quantitatively and qualitatively between them. We wished to identify the best method, either overall or depending on the use case, relying on the evaluators described above -however, a last question remained: what would one really "see" in the downscaled data ? The results brought by the qualitative assessment using classifiers were a first step towards answering this broad question from a machine point of view. We completed these results to assess the human point of view by carrying out a human perception study, which protocol and results are described in this section.

With the help of our interns Lucía, Marina and Ewa, we performed a study to assess and compare human performance in a gesture classification task using event data. Original event data from IBM's DVS128 Gesture dataset (see Section 3.3.1) is downscaled with the several spatial and temporal methods described above, and the classification performance is measured with human participants.

Human machine interface and protocol

Event data

We aim to evaluate the evolution of human performance according to different event downscaling methods at different resolutions. To achieve this objective, a participant should not correctly classify each of the 11 gestures comprised in DVS128 Gesture but should demonstrate that they were able to correctly separate similar gestures in order to prove that the reduction method studied retained sufficient relevant information. For this purpose, we decided to display to the participants only four gestures instead of eleven:

-gesture "right hand clockwise" (A), -"right hand counter-clockwise" (B), -"drums" (C), -"hand rolls" (D). The selected gestures were two by two similar together but different between the two pairs ; A and B are both quite similar, only differentiable by the direction of the hand movement ; C and D are 6.5 -6.5.1 Human machine interface and protocol 107 similar through the spatial distribution of events. For each figure, short samples from 5 out of the original 29 recorded subjects were selected and randomly displayed to the participants so that they classify the gesture itself and not the performing subject. Fig. 6.21 presents a screenshot of those four gestures ; the participants could use this support to help them classify the videos shown to them during the experiment. Figure 6.21 -Poster presented to the participants, illustrating the four annotated gestures selected from the DVS128 Gesture dataset to conduct the experiment.

The six spatial event downscaling methods described above were experimentally evaluated in this work at four different reduction factors:

-2 (resolution of 64 × 64), -4 (resolution of 32 × 32), -8 (resolution of 16 × 16), -10 (resolution of 13 × 13).

We also experimentally evaluated the four temporal downscaling methods. The two temporal structural downscaling methods were experimentally evaluated in this work at three different structural reduction factors p: 10%, 1% and 0.1%. In other words, the deterministic factor k was respectively set dividing 10, 100 and 1000 and k was converted into p according to Eq. 6.4. The two temporal funnelling downscaling methods were evaluated at five funnelling reduction factors: with time-window lengths of 0.05s, 0.1s and 0.5s for the synchronous funnelling, and 1s and 5s for the asynchronous funnelling. These were agreed upon as it was immediately obvious to us after studying the downscaled datasets that humans would understand the data obtained with the asynchronous method much better and for a longer time window, while they would have a harder time understanding events accumulated into frames displayed at the end of similar time windows. We decided to distinguish between the factors used between the synchronous and asynchronous methods in order to limit the experimental time for participants and not to collect data that we expected or could easily extrapolate.

Interface

Our interns Lucía, Marina and Ewa implemented an interface allowing participants to classify different samples of the 4 gestures, with different downscaling techniques and resolutions recorded from 5 different users.

To capture the user input, the interface was implemented using OpenSesame [START_REF] Mathôt | Opensesame: An open-source, graphical experiment builder for the social sciences[END_REF], an open-source tool for experiment creation in the fields of psychology, neuroscience, and experimental economics. This interface allows for easy storage of the gesture shown in each sample, its downscaling technique and resolution, enabling also to capture the subject's response and its velocity.

The experiment encompassed a training phase to familiarise participants with the four gestures and a testing phase to record the data for analysis. During both phases, every sample was displayed during 10s in mp4 format. Afterwards, subjects could select one of the 4 options (A, B, C, D) corresponding to the gestures or otherwise select "I don't know" (see Fig. 6.22).

In the training phase, subjects were presented with the 4 different gestures without any downscaling and received short feedback on the correctness of their answer. This training was repeated twice per participant to ensure that they could recognise the original gesture recorded with an event camera, even if they were not previously familiar with event data. Afterwards, in the testing phase, the samples were shown without any feedback.

The data collection for participants starts with a brief experiment explanation, as well as demographic questions, such as gender and age range. To reduce selection bias, participants of different genders across all ages took part in the experiment (see Tab. 6.5). To reduce expectation bias, participants were not informed of the techniques they were classifying at any moment nor were they obliged to complete the experiment within a given timeframe. To minimise order bias, the order of the shown samples was randomised.

Moreover, during the whole experiment, subjects had access to a poster (see Fig. 6.5 -Demographic repartition of the human assessment experiment participants.

Protocol

The study involved two experiments, one to evaluate the spatial techniques and another to evaluate the temporal techniques. Both were performed by 30 subjects each. The participants gave informed consent to participate and were informed of the purpose and nature of the study. The study was conducted in designated quiet rooms intended only for participants, on laptops with screens of similar size and quality. The participants were invited into the room in batches of 3 at different time intervals where they could choose one of the laptops whose positioning prevented access to the answers of other participants (see Fig. 6.23).

-6.5.2 Assessing human performance 109

In the experiment to assess spatial methods, participants had to identify 122 samples, with an average of 8m30s in the total task. In the experiment to assess temporal methods, participants had to identify 44 samples, and we recorded an average of 3m06s in the total task. When setting up both experiments, we expected the participants to take around 15s to select the label corresponding to each sample they were shown. However, in most cases, the subjects were able to respond faster, in 4.45s on average.

Assessing human performance

In this subsection, we assess the quality of human classification by analysing the accuracy, the number of unknown answers, the time to reply and the average number of events per sample for the different downscaling methods. The last criterion "number of events" is a quantitative evaluator relevant to our motivations: indeed, we want to implement event downscaling methods which reduce the quantity (i.e. the number of events) while preserving the quality (i.e. the amount of information carried) in the data. The "unknown" answers correspond to the selection of the "I don't know" choice in the interface, i.e. when participants did not recognise the gesture displayed. Note that the "unknown" answer does not correspond to an additional class per se, as the interface does not display at any point a gesture different from A, B, C or D.

As expected, the human accuracy decreases and the percentage of unknown answers increases as the size of the data decreases across all spatial downscaling methods (see Fig. 6.24). We can observe that both SNN-based and spatial funnelling are the techniques with the best human accuracy while log-luminance reconstruction techniques give the worst results.

Human accuracy drops below the chance level (25%) for a spatial diving factor of 10 in the log-luminance and mutual SNN techniques. The biggest drop in human accuracy occurs from factor 4 to factor 8. Surprisingly, the log-luminance techniques lead to accuracy varying little, even increasing slightly but always staying close to the chance level threshold. The notable difference in human accuracy and percentage of unknown answers between techniques does not translate to the time per answer results, which are relatively homogeneous among techniques (see Fig. 6.24 right).

The temporal techniques achieving similar resolutions are assessed in Fig. 6.25. Amongst the structural techniques, the stochastic outperforms the deterministic. As for the temporal funnelling techniques, the asynchronous technique obtains better results, with a small percentage of unknown Figure 6.24 -Human accuracy, rate of "unknown" responses and human response time to classify event data downsized spatially. In the figure on the left, the dotted line at 25% of the accuracy corresponds to the chance level. answers (5%) and a very notable accuracy of 70% despite an accumulation of events over 5smeaning that events appear only once per pixel for 80% of the original samples (that are in average 6s long). The temporal downscaling techniques result overall in better human accuracy and a lesser amount of unknown answers than the spatial techniques, which might be explained by the higher number of events. The response time per answer is also shorter than for spatial techniques. Only the synchronous temporal funnelling technique falls below the chance level when accumulated over a time window of 0.5s. Fig. 6.26 plots the human accuracy by spatial techniques against the resulting number of events. It clearly shows that the three techniques with the best overall human performance, Spatial Funnelling and both SNN pooling, also keep the highest number of events from reduction factor 2 onwards. Moreover, the techniques with the worst human accuracy, the log-luminance reconstruction techniques, give the smallest number of events kept.

Regarding the temporal downscaling techniques (see Fig. 6.27), we can observe that the number of events kept is the same across all reduction factors. As mentioned, the stochastic technique's accuracy exceeds the deterministic's performance.

Thanks to this study on the downscaling of event data for gesture classification using human participants, we can finally conclude that a certain size threshold needs to be maintained to ensure that human performance does not fall below the chance level. For input data of resolution 128 × 128, this threshold is close to the factor 8 for spatial methods, 0.1 for temporal structural methods and 0.5s for temporal funnelling methods.

Comparing human and machine performance

In this subsection, we assess the quality of human classification compared to the performance achieved by an SNN. We compare the results collected experimentally with the results output by the PLIF-based classifier (see Sec. 4.3.1.2).

As detailed in Chapter 1, many NNs outperform human beings in specialised tasks such as classification. Here we expect the PLIF-based classification to outperform the recorded human performance, especially considering many of the study participants were not familiar with event data. Fig. 6.28 confirms that for the structural and temporal funnelling techniques, the classifier outperforms human results with a minor exception in the synchronous temporal funnelling method with a time window of 0.05s. The difference between human and machine performance is especially stark for the synchronous temporal funnelling since the ratio displayed in Fig. 6.28 is quite close to 0 for this method. Machines significantly outperform humans, which cannot comprehend a motion from successive frames that are too far apart. This confirms that the PLIF-based classifier used here relies on the spatial repartition of events and does not take advantage of its temporal dimension. This underlines the disadvantages of using a frame-based classifier with event data. For the spatial methods, we encounter more obvious exceptions in the mutual and separate SNN at the reduction factors between 2 and 4. Figure 6.28 -Ratio of the human accuracy to the classifier accuracy, for each downscale factor, for spatial and temporal downscaling methods. When the values exceed 100% (dotted grey line), the human performance is higher than the classifier's ; when they fall below 100%, the classifier outperforms the human performance. Fig. 6.29 sums up the four criteria evaluated until now and adds a fourth evaluation metrics factor to the three evaluated so far (human accuracy, machine performance and number of events 6.5 -6.5.3 Comparing human and machine performance 113 compared to the original data): the downscaling time. The best method would optimise human and machine accuracy while minimising the number of events and downscaling time. Amongst the spatial downscaling methods, we observe once again that the techniques with the best compromise seem to be Spatial Funnelling and SNN. Mutual SNN is slightly better because it achieves an equivalent human accuracy with a significantly lower number of events, even though the downscaling time increments significantly. Once again, log-luminance methods do not provide satisfactory results. Even if they give a very small number of events and are optically coherent with the behaviour of an event camera (see (Gruel et al., 2022)), the accuracy is low and the downscaling time is high.

Amongst the temporal structural downscaling methods, the deterministic technique yields a very similar accuracy to the stochastic method, with a much shorter time taken to downscale. Less obvious, for temporal funnelling methods, the synchronous method offers better global results than the asynchronous technique, with a lower downscaling time and number of events ; however these advantages were traded off for a decrease in human accuracy.

The large standard deviations observed in Fig. 6.29 on the human accuracy measured on five methods (spatial funnelling, event count, SNN separate and mutual and especially synchronous temporal funnelling) can be explained by the wide variation in human performance depending on the intensity of reduction: according to Fig. 6.26 and 6.27, the human accuracy drops when the reduction factor increases for all methods. This reinforces the overall value of the asynchronous temporal funnelling method, which produces event data whose standard deviation is smaller although the performances are measured on significantly higher reduction factors, meaning it is quite consistent in its downscaling. Figure 6.29 -Overall comparison between spatial and temporal event downscaling methods, according to the overall human and machine performance, number of events and downscaling time of 1s of input event data (in s). All those criteria are weighted by the downscaling factor: the bigger the reduction, the higher the weight of the corresponding value. The number of events corresponds to the ratio of reduced events to the original number (in %). Each barplot presents the weighted mean of the corresponding value, and the black error bars the weighted standard deviation.

Our results highlight some discrepancies between human and machine-learning approaches to gesture classification using event data. Human accuracy is higher than machine accuracy in specific reduction factors for the synchronous temporal funnelling and mutual and separate SNN techniques. Indeed, this can be explained by the fact that humans are less susceptible to spurious correlation (i.e. associating two samples that are in no way causally connected) in the data than most neural networks [START_REF] Srivastava | Robustness to spurious correlations via human annotations[END_REF] and do not suffer from shortcut learning [START_REF] Geirhos | Shortcut learning in deep neural networks[END_REF], especially on the relatively small dataset that is DVS128 gesture (only 133 samples in its entirety).

Conclusion

While standard frame-based data (grayscale and RGB) are easily downscaled -which is a very common pre-processing step -, there existed no straightforward equivalent for event data up until recently.

In this chapter, we introduced and compared ten methods for event data downscaling for computer vision applications: six "spatial" and four "temporal". Our results show that the choice of the reduction methods greatly influences the event features of the reduced dataset, namely the number of events, the density, and information entropy.

While the proposed methods offer a panel of solutions to choose from, this choice is not trivial but depends strongly on the target vision task (classification, detection, tracking, etc), the application context, other event data preprocess it requires and the embedded computer throughput.

Our goal is to identify the best choice to perform spatial event reduction on an embedded system in order to adjust the complexity of data to the available resources such as processing capability and energy consumption (which is often directly linked to the amount of input datafor example, the energy consumed by certain neuromorphic hardware such as Kraken depend on the number of input events (Di [START_REF] Mauro | Kraken: A direct event/frame-based multi-sensor fusion soc for ultra-efficient visual processing in nano-uavs[END_REF]). It is important to note that our contributions are designed to be used in a global system including downscale and processing ; the implementation of our methods thus yields interest only if the energy consumed by the downscaling mechanism is compensated by a major reduction of the cost of subsequent processes. The results tend to show that this best choice can be found among the SNN pooling methods. Indeed they achieve an efficient trade-off between the optimisation of desired activity measures, performance and energy consumption as well as being significantly closer to real-time data handling than existing methods. Furthermore, when neuromorphic hardware is involved, it can be deemed easier and less costly to use either SNN method described in this chapter as a first preprocessing layer to ML models. As no significant differences can be observed between the run time and the classification performance of mutual and separate SNN pooling, we look at the activity measures to compare both methods. The mutual SNN pooling is definitely the optimal one based on the results of this work regarding these criteria, as it leads to a smaller number of events and a lower temporal density and is more optically coherent with the behaviour of an event camera.

If we do not limit our criterion to spatial downscaling, the structural methods that consist in randomly dropping events show very interesting performances in our experimental settings with the PLIF-based model, with a classification accuracy that remains very high even when dropping a high ratio of events. Additionally, the deterministic method has the lowest downscaling run time in an offline scenario out of all the spatial and temporal methods.

It is interesting to note here that, even though they are standard benchmark event datasets, N-MNIST and DVS 128 Gesture may not be the ideal datasets to assess the impact of temporal downscaling since none of them requires a fine temporal resolution: N-MNIST is essentially a static image translated in space to trigger events, and (Fang, Yu, Chen, Masquelier, et al., 2021) demonstrated that DVS 128 Gesture can be solve with only 20 timesteps of 5ms each, i.e. only around 1s. Thus DVS 128 Gesture and N-MNIST have been useful as a first support to validate 6.6 -Conclusion 115 these temporal methods, but a validation on temporally finer data would be welcome, for example on a lip-reading or even faster motion classification test.

Finally, we completed the previous results with a study quantifying the effect of downscaling of event data for gesture classification using human participants. Our study showed that a certain size threshold needs to be maintained to ensure that human performance does not fall below the chance level and that the quality of the data obtained from these methods is not uniform. It also highlights some discrepancies between human and machine-learning approaches to gesture classification using event data. Human accuracy is higher than that of machine in specific reduction factors for the Synchronous Temporal Funnelling and Mutual and Separate SNN techniques. This study sheds light on the potential limitations of event data downscaling and provides insights into the human perception of gesture classification using event data. The findings of this study have implications for the design and implementation of embedded computer vision systems that rely on event data and may also inform the development of more accurate ML algorithms for gesture recognition.

The spatial downscaling methods subsequently allowed us to implement a software version of the neuromorphic foveation architecture described in Chapter 8.

CHAPTER 7

Saliency detection by event density

The combined use of SNN and event cameras is gaining momentum in the field of embedded computer vision as they promise to reduce latency and computational resource requests. However, state-of-the-art embedded neuromorphic models show little interest in modifying input data to optimise model performance, memory usage, latency and power consumption. We propose here to leverage visual attention to optimise this trade-off.

Visual attention can be defined as the behavioural and cognitive process of selectively focusing on a discrete aspect of sensory cues while disregarding other perceivable information. This biological mechanism, more specifically saliency detection, has long been used in computer vision to drive the analysis only on relevant parts of images or videos for further processing. The recent advent of event cameras raises the question of adapting attention and saliency to the unconventional type of such sensors' output. We introduce in this chapter a neuromorphic model of spatiotemporal salient selection, which simultaneously outputs one or multiple segregated objects of interest detected in an event-based scene. We propose to identify regions of interest as those corresponding to a high spatiotemporal density of events, which allows us to rely on intrinsic spiking neural network dynamics and online adaptation rules. We experimentally validate our model on the open DVS128 Gesture Dataset with implementations on CPU, Loihi and Kraken.

Introduction

Embedded systems and even early realised neuromorphic embedded processors are limited in terms of memory bandwidth. During this PhD, we first attempted to address the issue by proposing neuromorphic and non-neuromorphic spatial reduction techniques (see Chapter 6). However, the trade-off between the quantity and the quality of the reduced event data is not optimal, which could be explained by a lack of emphasis on the salient elements of the visual scene during data reduction. Indeed, we believe that detecting RoIs in the original visual scene to select the corresponding OoIs (i.e. the events taking place in the detected RoIs) is a more promising approach to increase accuracy, reducing both latency and memory requirements.

Building on top of the limited number of existing neuromorphic visual attention models (see Section 5.1.2), we implemented a low-latency neuromorphic model of salient spatiotemporal selection, which simultaneously outputs one or multiple OoIs detected in an event-based scene.

Here we based our work on the hypothesis that in an event-driven scene, the human eye is attracted by a cluster of events that are close in time and space that one could easily assimilate to an object and try to comprehend as such. In order to simulate a mechanism of bottom-up visual attention similar to the behaviour described precedently, we have created a model that focuses on the regions corresponding to a high spatiotemporal density of events. In this work, we differentiate between RoIs and OoIs: we consider that the RoI (or "salient region") corresponds to the sensor's spatiotemporal region where interesting events are emitted, while the OoI (or "salient object") is the object one can reconstruct from those events. Let's visualise this difference with the example of the DVS 128 Dataset: in Fig. 7.1, the red square indicates the RoI, i.e. the region where many events take place closely in time and space ; and the purple outline indicates the OoI, i.e. the events identified as salient and that can be assimilated to an "object" (here, the hand). One RoI can englobe different OoIs and does not influence their shape. The various mechanisms described in this chapter rely solely on intrinsic SNN dynamics and dynamic adaptation rules applied to synaptic weights and neural thresholds. Additionally, they directly take the event data as input by considering each event (with no regard for the polarity) as a spike emitted by a visual organ (for example, the retina). This is a crucial feature as it leads to minimising the latency since it does not require the conversion of spiking events into a frame. The saliency detector is not specialised for any specific context or any specific shape (in other words, there is no training phase), which allows for a good generalisation ability of the network. Our proposed architectures, implemented with LIF neurons due to their simplicity, are designed to be lightweight enough to run in real time.

In order to validate our models, we experimentally validate their implementation on CPU, Loihi, SpiNN-3 and Kraken (see Section 2.2) in collaboration with Michele Magno's group at ETH Zürich and researchers at IMRA Europe (Sophia Antipolis, France): the research assistant Antonio Vitale deployed the model on Loihi ; Dr Alfio di Mauro and his intern Robin Hunziker took care of the implementation on Kraken ; and the intern Hugo Bulzomi * deployed the model on SpiNN-3 and applied to a pedestrian detection task with the help of his co-supervisor Mr Yuta Nakano. We qualitatively and quantitatively assess the RoI detection and OoI selection performed on the DVS 128 Gesture dataset (Amir et al., 2017). In summary, our contribution:

-is able to simultaneously select multiple OoIs present in the visual scene in the visual scene up to 50% of the time, with no training phase ; -selects at least one OoI with a delay of less than 15ms, and reaching 5ms in the best cases ; -selects OoIs with quality leading to a classification performance reaching 73% of the original data with a reduction of the sensor's activated surface by a factor of 1000 ; -should be implemented on neuromorphic hardware such as SpiNNaker, Loihi or Kraken to optimise the simulation time and energy consumption, as we demonstrate that it is amenable to resource-constrained embedded neuromorphic platforms ; -achieves all the above with a number of neurons reduced by 10% to 46% compared to the state-of-the-art.

To the best of our knowledge, it is the first neuromorphic model able to simultaneously select multiple OoIs. The implementation in PyNN (Python library) of the mode † . The results presented in this chapter were published in four articles: the model selecting one OoI and its implementation on CPU and Loihi appear in the proceedings of AICAS 2022 [START_REF] Gruel | Neuromorphic event-based spatiotemporal attention using adaptive mechanisms[END_REF], its implementation on Kraken is discussed in the proceedings of AICAS 2023 [START_REF] Gruel | Embedded neuromorphic attention model leveraging a novel low-power heterogeneous platform[END_REF] and first steps towards a complete implementation on SpiNNaker in the context of pedestrian recognition are described in the proceedings of MVA 2023 [START_REF] Bulzomi | Object detection for embedded systems using tiny spiking neural networks: Filtering noise through visual attention[END_REF]. Finally, the model of simultaneous multi-OoIs selection is introduced in the proceedings of IJCNN 2023 (Gruel, Martinet, & Magno, 2023a). The latter was also presented during ORASIS 2023 [START_REF] Gruel | Simultaneous neuromorphic selection of multiple salient objects for event vision[END_REF]. It is to be noted that a first paper presenting a global bibliographic review of visual attention in biology and associated neuromorphic models was accepted at CBMI 2021 (Gruel & Martinet, 2021) ; however, the first implementation of visual attention with SNN on event-data introduced in this paper did not lead to conclusive results and we shifted our research direction to the model described below.

-Saliency detection 121

Saliency detection

We introduced a saliency detection by event density which integrates the events produced by each pixel at a low resolution and outputs a set of coordinates for one or multiple RoIs. We define the RoIs as the spatial regions of the sensor where the amount of events received over a certain amount of time is more important than elsewhere over the whole scene, i.e. a region where the number of events is significantly big in a small spatiotemporal window. This visual attention mechanism is thus bottom up (i.e. independent from any previously set motivation or rule) and covert (i.e. without simulated saccadic eye movements). Fig. 7.2A presents the architecture of our saliency detector ; it is formed by the "Input events" and "Saliency detector" layers.

Input layer

The input layer translates the events generated by the sensor into spikes, with no regard to the polarity. One neuron is assimilated to one pixel of the sensor. The spikes produced by the input layer are sent to the saliency detector via an excitatory downscaling connection. This corresponds to a convolutional layer with a kernel size S ×S and a stride S, without padding. The input neurons are separated into non-overlapping square regions of size S × S. Each neuron in the input layer's subregions is connected to one corresponding neuron in the saliency detector layer, as illustrated in Fig. 7.2A.

Saliency detector -soft exponential Winner-Takes-All

The model aggregates the neurons activated in the "Saliency detector" layer, corresponding to the salient regions, into distinct segments using a soft exponential WTA. The choice of using WTA instead of another clustering algorithm was made to respect bio-plausibility and implement clustering using a native, SNN-based strategy. Since a strong WTA leads to the activation of only one neuron in the layer and multiple RoI are to be detected by the network, the soft WTA weight has been set experimentally to 0.02. Indeed, the higher the weight, the higher the lateral inhibition and the stronger the selection between the neuronal activations ; a strong weight leads to the detection of only one RoI, while a softer one (such as the one we settled on) allows for the simultaneous detection of multiple RoIs.

In the case of the saliency detector, a specific exponential WTA is implemented according to the radial basis function Eq. 7.1 in order to allow RoI of arbitrary sizes:

ω W T A = max( e d w × h , ω max ) (7.1)
where d corresponds to the Euclidean distance in number of neurons between the active and target neuron subject to inhibition, and w and h are the width and height of the layer. The weight ω W T A has an upper bound of wω max = 50. The Euclidean distance d can be calculated thanks to the arrangement of the neurons of the "Saliency detector" layer according to an orthogonal reference frame (2D grid): each neuron has a set of (x, y) coordinates. As illustrated in Fig. 7.2B, the higher the distance between the inhibiting neuron and its neighbours, the higher is the inhibition. Another visualisation of this mechanism is provided by Fig. A) The blue and green dots and shapes in the input layer corresponds to input events: the dots represent noisy or non-saillant events and the "cloud" shapes illustrate events aggregated into saillant objects (OoIs). B) Precise depiction of the activating convolution between the Input and the Saliency detector layers as well as the soft exponential WTA on the Saliency detector. In this example, the sensor is of size 9 × 9 pixels and the convolution has a kernel size of S × S = 3 × 3. Each neuron is depicted as a circle, and the increased size in the Saliency detector layer simply allows better visualisation of the topology. The neurons are spatially positioned as a 2D rectangle. The red connections depict the lateral inhibition implemented by the soft exponential WTA, and the continuity of the lines corresponds to the intensity of the inhibition: neurons far from the activated one are highly inhibited (continuous lines), while those close are softly inhibited (dashed lines).

to ω W T A = ω max . The maximum inhibition weight corresponds to the weight of the WTA as described above, i.e. the weight by which a traditional WTA inhibits all the neurons in the layer.

Saliency detector -weight adaptation rule

Finally, the adaptive detection of saliency in this layer is enabled by a dynamic weight adaptation rule between the input layer and the saliency detector, inspired by Hebb's rule: "cells that fire together wire together" [START_REF] Hebb | The organization of behavior: A neuropsychological theory[END_REF]. This rule is implemented by increasing or decreasing the weights of synapses which pre-synaptic neurons have recently fired, as we consider them to be more likely to fire again soon (see Eq. 7.2).

ω(t + 1) =      min(ω(t) + ∆ω, ω max ) if f t input ≥ t -1 ω reset if f t input < t -∆t ω(t) otherwise (7.2)
where ω(t) is the weight at the simulation step t of the synapse to which is applied the dynamic weight adaptation rule, ∆ω the positive weight variation at each simulation step, ω reset and ω max the lower and upper bounds of the synaptic weight, f t input the firing time of the last spike transmitted by the synapse and ∆t the delay before the synaptic weight is set back to ω reset . Fig. 7.4 illustrates the behaviour of this rule. The architecture described is simulated with PyNN, a simulator-independent Python interface for SNN simulators [START_REF] Davison | Pynn: a common interface for neuronal network simulators[END_REF], combined with NEST (NEural Simulation Tool) [START_REF] Gewaltig | Nest (neural simulation tool)[END_REF], a Python simulator for SNN on CPUs.

Spatiotemporal filtering -saliency detection of one object of interest at a time

We extended the saliency detection architecture described in the section above into a neuromorphic model of spatiotemporal attention on event data, implemented on both the CPU simulator PyNN [START_REF] Davison | Pynn: a common interface for neuronal network simulators[END_REF]), Intel's neuromorphic hardware Loihi [START_REF] Davies | Advancing neuromorphic computing with loihi: A survey of results and outlook[END_REF] and Kraken, an academic platform that includes a Spiking Neural Accelerators and RISC-V cores [START_REF] Mauro | Kraken: A direct event/frame-based multi-sensor fusion soc for ultra-efficient visual processing in nano-uavs[END_REF]. This model detects the salient regions of the input sensor ; it outputs accordingly either the first corresponding OoI (i.e., the spikes corresponding to the detected region) or the OoI with the greater event density (see Fig. 7.17 for a complete analysis of this behaviour and its bio-plausibility).

Network architecture

The architecture of this model is described in Fig. 7.5It achieves this task using a light architecture: three SNN layers linked by two sets of excitatory synapses (downscale connection between the input and the RoI detector, and One-to-One connection between the input and the output layers) and two sets of inhibitory synapses implementing WTA mechanisms. Additionally, the model can adapt to any input data thanks to two adaptive mechanisms: the weight adaptation rule described above -which favours the activation of the RoI detector depending on its previous activations ; and a threshold adaptation rule -which promotes the passage of spikes in the output regions corresponding to the salient regions identified by the RoI detector and hinders this passage in other places.

The last layer is the output layer, of the same size as the input layer. It receives the activity of the input layer through one-to-one connectors (i.e. each input neuron is solely connected to the neuron located at the same coordinates in the output layer). Only the spikes corresponding to the RoIs identified by the saliency detector, thus forming the OoIs, are emitted by this layer thanks to an exponential WTA mechanism similar to the one described in Eq. 7.1, implemented with a stronger weight value to prevent other neurons in the output layer to spike when one OoI has already been selected and let through.

Threshold adaptation rule

In order to maintain the same information as the OoIs identified in the original data, we set up a dynamic threshold adaptation rule aiming to facilitate the neuronal spiking at the salient coordinates and to hinder it in other neurons of the output layer. The facilitation is implemented via the decrease of the neuronal threshold closer to the resting value of each neuron whenever the corresponding neurons spike in the saliency detector layer. Similarly, the hindering is implemented via the increase of the neuronal threshold, further from its resting value, when the saliency detector identifies no activity over a long period of time. This mechanism is described in Eq. 7.3.

θ(t + 1) =      max(θ(t) -∆θ, θ reset ) if f t saliency ≥ t -1 min(θ(t) + ∆θ, θ max ) if f t saliency < t -t δ θ(t) otherwise (7.3)
where θ(t) is the threshold at simulation step t of the neuron to which is applied the dynamic weight adaptation rule, ∆θ is the positive threshold variation at each simulation step, f t saliency corresponds to the firing time of the last spike transmitted by the saliency detector in the corresponding neuronal regions and t δ is the delay before the neuronal spiking is hindered i.e. before the neuronal threshold is increased. Finally, θ max and θ reset are respectively the upper and lower bound (i.e. the reset value) of the neuronal threshold. Note that the f t saliency used here corresponds to the f t input used in Eq. 7.2. However, while the weight adaptation rule in Eq. 7.2 was applied to the synapse linking the input and the saliency detector relies on the activity of its post-synaptic neurons, here the thresholds modifications carried out by the adaptation rule do not rely on the post-synaptic activity (i.e. the output layer's activity) but on the saliency detector acting as an external supervisor. Fig. 7.6 illustrates the behaviour of this rule.

This filtering by dynamic threshold adaptation requires first identifying the RoIs in the sensor before lowering the corresponding threshold, in order to prevent excessive unwanted output at the beginning of the simulation. Thus the initial thresholds are set to a high value in order to let the saliency detector activate first. We demonstrate in the following experimental validation that such a plasticity rule is highly preferable to a simple system of synaptic activations and inhibitions. Indeed, applying an excitating activation to the neurons corresponding to the salient regions identified by the saliency detector would compete with the input activation and saturate these neurons. This would "blur" the spikes from the input data, i.e. create artefact due to the additional activation of the saliency detector, and cause the filter to lose spatiotemporal accuracy. The first experimental validation of the model described in this section was originally performed using PyNN combined with NEST (NEural Simulation Tool) (see Section 2.2), which allows for dynamic adaptation rules. However, this software implementation is significantly limited by the high simulation time.

Loihi

Our collaborator Antonio Vitale, from Michele Magno's group at ETH Zürich, then adapted this model with our help to Intel's neuromorphic hardware Loihi. As depicted in Fig. 7.5, the key elements in the presented SNN are both the weight and threshold adaptation mechanisms. Currently, the Loihi platform does not support these features (i.e. the modification of neuronal and synaptic parameters during the simulation) thus the proposed method had to be modified to implement it on the neuromorphic chip and perform an experimental validation. The input and the RoI layer were connected to the output in a way such that at any given timestep the neurons in the output layer would spike only if the corresponding neurons were active in both previous layers. The connections between the input layer and RoI detector layer, as well as the WTA mechanisms, were kept the same. The resulting 3-layer network occupies 68 neurocores on the chip and consists of 4.224 neurons and 81.716 synapses.

SpiNNaker

In collaboration with his co-supervisor at IMRA Europe, our intern Hugo Bulzomi deployed this network on SpiNNaker and applied it to a pedestrian detection task. Taking inspiration from the region proposal SNN introduced in [START_REF] Acharya | Spiking neural network based region proposal networks for neuromorphic vision sensors[END_REF], each neuron of the saliency detector is connected to its neighbourhood via excitatory synapses, in addition to being connected to all other neurons via exponential inhibitory connections. The weight adaptation rule 7.3 -7.3.3 Implementation on neuromorphic hardware 127 allows the active input regions to out-compete parts of the input with less consistent activity, such as noise. Additional noise-filtering layers such as the one described in [START_REF] Acharya | Spiking neural network based region proposal networks for neuromorphic vision sensors[END_REF] are then not needed.

Hugo added a clustering step to this model in order to detect objects in the visual scene and create their bounding boxes. The spikes of the saliency detector are aggregated over a fixed period of time. Since each neuron in the saliency detector corresponds to a patch of size S × S on the input, we can map their activation to the detection of objects over different parts of the input window. Patches whose corresponding neurons spiked enough times are considered to contain an object. Adjacent activated patches are finally grouped together to form bounding boxes. The overall network is described in Fig. 7.7.

Kraken

A third experiment was performed on ETH Zürich academic platform Kraken, which allows for the online adaptation of neuronal and synaptic parameters and achieved quicker than real-time saliency detection, thus overcoming both PyNN and Loihi's limits. However, the Sparse Neural Engine (SNE), Kraken's dedicated neuromorphic accelerator, is primarily designed to accelerate feed-forward neural networks using the framework PyTorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF]. This framework differs greatly from PyNN which requires a significant adaptation of the architecture's implementation to allow the deployment of the reference model on Kraken. In PyNN, each neuronal population is implemented first and then they are interconnected ; PyTorch however implements a neural network by initiating layers and connections together. PyTorch-defined networks are deployed on Kraken by converting them into an intermediate neural network representation format (ONNX). Executable C code is then auto-generated by using a dedicated Kraken deployment toolchain. The low-level generated C code is completely available to the programmer and, therefore, can be modified to include dynamic threshold adaptation rules as well as custom interconnection schemes among neurons belonging to different populations.

We worked conjointly with two other persons, Alfio di Mauro and Robin Hunzinker, from Magno's group to deploy the saliency detection algorithm to Kraken. The original synaptic interconnection schemes connecting the various neuron populations and performing the attention mechanism have been implemented as equivalent neural network layers. The latter accounts for the neuron synaptic connectivity supported by SNE, i.e., 1x1 and 3x3 convolutional and fully connected synaptic connectivity, and the exact LIF neuron model implemented in SNE. Specifically, the downscale connection between the input and the RoI detector, originally implemented as two populations of LIF neurons connected via an excitatory downscaling connection (i.e., a convolutional layer with no padding -see Fig. 7.5), is translated into a fully-connected synaptic connection with no bias and by setting the weights to O in location outside of the convolution kernel, i.e. where connections are not required. Indeed, the PyTorch linear layer is the most appropriate to implement All-to-All connections. The excitatory One-to-One connection between the input and the output layers (see Fig. 7.5) is translated into a convolutional layer with a kernel of size 3 × 3, padding of size 1 and no pooling nor bias. All the weights are null, except the kernel's central weight, which is set to 1, thus allowing only a connection between corresponding pre-and post-synaptic neurons. Additionally, exponential WTA mechanisms applied to the RoI detector and the output layer are implemented using linear layers with no bias. Since the inhibition is implemented in PyTorch by setting the weights to a negative value (the lower the value, the higher the inhibition), the exponential rule is adapted to enable a correct WTA behaviour as described in Eq. 7.4:

ω W T A = max( -e d w × h + ω W T A , 0) (7.4)
where the notation is similar to Eq. 7.1. d corresponds to the Euclidean distance between the active and target neuron subject to inhibition, w and h are the width and height of the layer, and w min is the lower bound of the weight ω W T A .

Multi spatiotemporal filtering -simultaneous saliency detection of multiple objects of interest

In this section, we introduce our saliency detection and multiple OoIs selection model, which is an extension of the visual attention model presented in the previous Section 7.3 to simultaneously detect and select multiple OoIs in an event-based visual scene using intrinsic SNN dynamics. The architecture of the model depends on the number n of OoIs to select in a scene: several lateral output layers are initiated, each connected to the input layer with an All-to-All connection and each selecting a different OoIs in the visual scene. This model is designed to detect n OoIshowever, in an effort to simplify the reader's comprehension, it is depicted in Fig. 7.8 under an architecture which would allow the detection of two OoIs at most. This new attentional selection of n OoIs features the same saliency detector, exponential WTA and weight adaptation rule as introduced in Section 7.2 ; however, we propose here a new dynamic threshold adaptation rule grounded on the activity of both the saliency detector and the lateral output layers. At each simulation timestep, for each neuron of each output layer, this dynamic rule will first identify the lateral neuronal activity at the same coordinates (see Eq. 7.5) and maximise the threshold if the corresponding lateral neurons are activated (see Eq. 7.7 and Eq. 7.8). The rule will then modify the threshold depending on the activation of the saliency detector in the corresponding neuronal regions (as described in Eq. 7.3.

The binary mask α n (t), applied to the neuron n, is calculated according to the activity of the lateral layer λ at the simulation step t as follows:

∀λ ∈ λ lateral layers , ∀n ∈ n neighbourhood , where f t λ corresponds to the firing time of the lateral layer λ's neuron at the same coordinates. δ λ is the delay applied to each output's influence on its lateral populations: as the output layers are implemented sequentially, i λ is the arbitrary identifier given to each layer and by which the δ λ is multiplied to calculate λ's delay. α n (t) is null if the corresponding neuron in at least one of the lateral layers is activated -thus hindering the selection of spikes emitted in other layers.

α n (t + 1) = 0 if t ≥ δ λ × i λ and f t λ > t δ 1 otherwise (7.5)
The neuronal threshold θ saliency (t + 1) is obtained according to the saliency detector activity as described in Eq. 7.3. It is then updated into θ n (t + 1) according to the lateral neuronal activity: first, the binary mask is applied to inhibit the neurons whose corresponding neurons are activated in lateral layers ; secondly, this inhibition is decayed over time based on the time elapsed since this lateral activation. In other words, θ n (t + 1) is calculated according to the result of:

θ n (t + 1) = f decay (f inhibition (θ saliency (t + 1))) (7.6)
The function f inhibition introduced in Eq. 7.6 inhibits the neuronal activity according to the newly calculated α n (t + 1), i.e. maximises the neuronal threshold:

f inhibition (θ) = θ max if α n (t + 1) = 0 θ otherwise (7.7)
where θ max is the upper bound of the neuronal threshold. If α n (t + 1) is null, it means that the corresponding neurons in at least one of the lateral layers λ is active and therefore the neuron of the current layer should not fire.

The function f decay introduced in Eq. 7.6 reduces the inhibition applied by f inhibition based on the time elapsed since this lateral activation: ∀λ ∈ λ lateral layers ,

f decay (θ) = θ -∆θ if f t λ < t -t δ θ otherwise (7.8)
where ∆θ is the positive threshold variation at each simulation step and f t λ (t) the firing time of the corresponding neuron in the lateral layer λ.

Fig. 7.9 illustrates the overall behaviour of these rules.

Experimental validation

In the following section, we aim to demonstrate the neuromorphic models' efficient saliency detection and attentional selection of one or multiple OoIs in an event-based visual scene by verifying their quantitative and qualitative accuracies.

Visual input data

To this end, we need an event dataset with a controlled number of OoIs, with known spatiotemporal coordinates. Additionally, as we wish to assess a qualitative aspect by evaluating the performance of a classification task on the output OoIs, the dataset must include labelled objects corresponding to the output OoIs.

Since (to the best of our knowledge) such a dataset does not exist, we artificially created one meeting the criteria described above by stitching together various samples from DVS 128 Gesture (see Section 3.3.1) according to three protocols described below.

Additionally, we use a fourth dataset for the experiments led using SpiNNaker that will be designated as "Prophesee Pedestrians" in the remainder.

Three-gesture dataset

A first combination of DVS 128 Gesture samples was performed to experimentally validate the model implementation on PyNN et Loihi. The samples from 3 classes were used in various combinations for the demonstration of this spatiotemporal attention model: "arm roll" (class 1), "hand clap" (class 2) and "right hand clockwise" (class 7). This small dataset will be referenced as "Three-gesture dataset" in the remainder.

The samples are separated into two sub-types, either "large" or "small", according to the number of events in the sample and the ratio of active pixels in the sensor. We arbitrarily decided that "small" gestures have an active region inferior to 2.5% of the sensor and a number of events inferior to 5000: according to Fig. 7.13, classes 1 and 2 are "small" and class 7 is "large". We differentiate between those two characteristics in order to assess the influence of event flow on the saliency detection. New samples with size 128×256 were generated by concatenating horizontally samples from both types. A varying time shift to the start of the second sample is added, using shifts of 0.5ms, 1ms, 5ms, 10ms, 20ms and 50ms, to demonstrate the accuracy of the filtering increasing with the time shift. We balance our dataset by allowing the first gesture to start either on the right or on the left.

The ground truth GT (x, y, t) for each experiment was defined as the event stream of the first gesture only and all other events are GT ′ (x, y, t).

Control and random symmetric combinations

Control dataset A second custom-made dataset called "control" was created by randomly selecting 50 DVS 128 Gesture samples, copying them n times and concatenating them horizontally by offsetting their x coordinates accordingly. This leads to a pool of 50 samples on which to detect one to n OoIs (i.e. the hand doing the gesture), whose spatiotemporal localisation is approximately known.

Random symmetric dataset A third custom-made dataset was similarly created by randomly selecting 50 × n DVS 128 Gesture samples and combining them together to produce a final pool of 50 samples. As the intrinsic activity measures of each sample differ according to their class and may affect the saliency detection (see Fig. 7.17), for each combination n -1 others are created by permuting the order of the samples in order to create a random but symmetric custom-made dataset of 50 × n elements.

The two datasets described above will be respectively referred to as "control" and "random symmetric" in the rest of this paper. An example of both custom-made datasets is shown in Annexes, in Fig. A.1, where n is arbitrarily set to 2 to facilitate the reader's comprehension. It is to be noted that the actual data used as input in the following sections have been spatially reduced by 4 using the event count method we introduce in Chapter 6, as seen in Annexes Fig. A.2 and Fig. A.3, due to the limitations of PyNN in terms of the maximum number of simultaneously simulatable neurons and connections. Furthermore, only the first 100ms (or 1s for any classification task) of each sample was used to reduce the computation time.

Shift Additionally, we introduce some variations within the two kinds of custom-made datasets presented above: the n samples are combined together according to a certain temporal shift, where the m+1 sample's temporal coordinate is shifted by a factor shif t to the m th sample. In the following experimental results, shif t varies between 0 (all the combined samples start simultaneously) and 5000µs.

Prophesee Pedestrian

The fourth dataset was provided by collaborators at IMRA Europe and used by Mr Hugo Bulzomi. This dataset contains multiple recordings of both interior and exterior scenes. Objects captured are pedestrians, using a Prophesee Gen 3 of resolution 640 × 480. From this event representation, an implementation of Firenet [START_REF] Scheerlinck | Fast image reconstruction with an event camera[END_REF] was used to generate grayscale video frames that were then annotated by hand with the bounding boxes of each pedestrian. Fig. 7.10 shows some examples from our dataset. Whereas some interior and simple exterior recordings involve relatively low amounts of noise, other exterior recordings in front of busy supermarkets feature large quantities of noise over the whole sensor.

Validation of the attention selection of one object of interest

We performed different experimental validations of the attention selection of one OoI and multi-OoIs during this PhD. This section describes the results concerning the case of one output, i.e. one OoI selected. The model was implemented on CPU using PyNN, and on two neuromorphic hardware Loihi and Kraken ; the following results compare those three implementations.

The different hyperparameters values used to implement this architecture on PyNN and Kraken are described in Tab. 7.1. The ones used to implement on Loihi were not communicated by our collaborators. The change is specifically stark in Kraken since the neural network weights are quantized to be simulated on Kraken. A first validation was performed using the original samples from the DVS 128 gesture dataset (see Sec. 3.3.1).

Parameters

PyNN Using the PyNN implementation of salient selection of one OoI, we assess the evolution of the quantitative and qualitative properties of event samples from this dataset before and after attentional filtering.

The quantitative properties we evaluated and present in Fig. 7.11 are: -the activated surface, i.e. the average percentage of pixels activated during a 50ms timewindow in the sample, over the total number of pixels in the sensor ; -the ratio of OoI's size to the original activated surface, which is the ratio between the averaged width and height of the OoI to the width and height of the activated surface of the sensor in the original data. The width and height are respectively calculated as the difference between the highest and lowest x and y coordinates of events in the identified OoI, accumulated over a 50ms time window. Each of those properties is calculated on the model's output compared to those of the original dataset, averaged for each class. The last plot in Fig. 7.11 presents the classification performance performed by the classifier implemented using PLIF (which will be referred to as "PLIF-based classifier" in the remainder of this chapter -see Sec. 4.3.1.2) on the original and output datasets.

We aim to implement an attentional selection of one or multiple OoIs that on one side significantly reduce the input data to handle, while on the other side maintaining a relatively good quality. Fig. 7.11 assures us that using our saliency detection model allows for an attentional selection of events answering to our needs, with a distribution of events in space reduced by 80% and a classification accuracy maintained at 70% of the original one's. Loihi On Loihi, the input event data is collected for 1ms and downsampled by a factor of 4 using an efficient bit-shift operation on Loihi's embedded x86 cores. Each input sequence is 1.5s long and thus requires 1500 simulation timesteps to be fully processed. On average, measurements showed that it takes 6ms to process 1ms of input data. This ratio of 1:6 is much closer to realtime than the one obtained on CPU. These numbers show that processing event-based data on neuromorphic hardware is more efficient in terms of latency. This is an important advantage not only for running offline simulations but also paves the way for potential real-time, closed-loop applications.

Kraken We describe here the experiments conducted on the Kraken platform (see Section 2.2), and we discuss the results of such explorations compared to the performance achieved by our model on PyNN and Loihi (see previous sections).

Table 7.2 provides the energy and latency breakdown for all the relevant algorithm functional modules benchmarked on the SNE accelerator. In our experiments, we used a 6% spiking activity, the highest spiking activity measured across DVS 128 Gesture. Table 7.3 reports the overall energy and execution latency for the entire Kraken platform, including the energy consumption of the RISC-V core (Fabric Controller) that supervises the algorithm execution on SNE. As multiple tiles are executed on the SNE accelerator to run a complete saliency detection, Table 7.3 also reports the SNE idle consumption, i.e. when the accelerator is waiting for weights to be loaded, or the Fabric Controller pre-processes inputs. Kraken achieves real-time simulation run-time as it requires 0.14s to execute the saliency detection algorithm on 1s of input spiking activity from an event camera, significantly lower than both Loihi and CPU run times. Moreover, compared to the algorithm implementation executed on Loihi and on CPU, it is possible with Kraken to fully implement on-site the dynamic weight and threshold adaptation rules presented in [START_REF] Gruel | Neuromorphic event-based spatiotemporal attention using adaptive mechanisms[END_REF], which are a crucial feature for online adaptation.

Layer

Energy (in J) Latency (in s)

Input to RoI detector 3.46 × 10 -4 1.26 × 10 -2

WTA on RoI detector 2.16 × 10 -5 7.86 × 10 -4

Input to Output 3.63 × 10 -4 1.26 × 10 -2

WTA on Output 3.12 × 10 -3 1.13 × 10 -1 Total 3.85 × 10 -3 1.39 × 10 -1 Fig. 7.12 compares the simulation time of the attentional algorithm on a CPU, Loihi and Kraken. We show that with Kraken, it is possible to achieve real-time saliency detection. Furthermore, we compare the quality of the detected OoI by comparing the classification accuracy obtained on those detected in DVS128 Gesture, performed by the PLIF-based classifier. The classification performance is 20% better on the OoIs detected by Kraken compared to PyNN: this significant increase both confirms the quality of the implementation of the original model on Kraken and can be explained by the greater number of events contained in Kraken's OoIs. Indeed, the PLIF-based classifier accumulates events in frames and performs better on a dataset rich in events. Figure 7.12 -Simulation time for an event input of 1s, detection latency (i.e. how long should be simulated before identifying the OoIs) and PLIF-based classification accuracy of the OoIs detected in DVS128 Gesture by our saliency selection model, compared between PyNN, Loihi and Kraken implementations. As Loihi's data comes from collaborators no longer working on this project, no classification accuracy could be measured on this hardware.

Three-gesture dataset validation

A second experiment was conducted on our custom-made three-gesture dataset. We first assess the quality of event filtering by comparing in Fig. 7.13 the number of events and the active region obtained on output data and on the original data. The values are evaluated according to different temporal shifts applied to the original samples. We observe a decrease in the number of events, as well as an influence of the time shift on the quality of the selection. Figure 7.13 -Mean number of events in and percentage of sensor activated by the event data from the three-gesture dataset. The values are computed for the original data ("ground truth data") and for the data output by the spatiotemporal mechanism after different time shifts, using the corresponding original sample as input. and "arm roll" (on the right), each gesture lasting 200ms ; while the right figure presents the simulation results on the same two gestures spatially inverted ("arm roll" on the left and "right hand clockwise" on the right). The first gesture to appear on the sensor is the one on the left in both cases. Such a permutation was tested in order to confirm that the saliency is not dictated by the spatial location of the OoI, but indeed by its timing and size (as hypothesised in the introduction of this chapter). The gesture on the left starts first, and overlaps over 100ms with the second one. One can clearly see that:

-the saliency detection (second plot from the top, identified by the label "ROI.d") clearly indicates the spatiotemporal RoIs ; -the output layer only s through the events corresponding to the OoI detected first by the saliency detection i.e. the first gesture performed, until the end of the corresponding 200ms ; -the evolution of the weights of the input to saliency detector connections follows Eq. 7.2, increasing at ω max = 2 when the saliency detector activates or decreasing to omega reset = 0.7 otherwise. -the evolution of the output thresholds follows Eq. 7.3, decreasing at θ reset = -65mV when the saliency detector activates or decreasing to θ max = 0mV in absence of activity.

Temporal filtering Both results produced using PyNN and Loihi show that this attention model filters the region of primary interest as a function of time. The output is not significantly influenced depending on the size of the different input RoIs: the first gesture is filtered out (see Fig. 7.14) and as seen in Fig. 7.15 and 7.16, the error stays low and decreases with the shift between the two samples, independently of the characteristics of the first and second occurring input gestures, whether the size or the spatial location vary. Adaptation rules Currently, Loihi does not support online threshold adaptation. Fig. 7.15 and 7.16 present the evolution of an error metric established by our collaborators at ETH Zürich to assess the quality of the OoI selection: assuming that a filtered OoI should have the same activation regions as the ground truth, they implemented a Root-Mean-Square Error (RMSE) evaluating the overlap between the active regions in the ground truth and the output of our model, weighted over 100ms time-windows. The lower the RMSE, the better the overlap and the OoI selection, and vice-versa. The results displayed in Fig. 7.15 and 7.16 were obtained by running the modified architecture described in Sec. 7.3.3, and not on the PyNN simulator due to the run-time limitations mentioned above. However, to demonstrate the effectiveness of the adaptation rules, we produced some limited results using shorter input data (400ms instead of 1.5s) on the PyNN simulator: the RMSE equals 0.03 on average in the context described in Fig. 7.14 Left, and 0.16 for Fig. 7.14

Right. This difference can be explained by the number of events in the OoI: in both cases, the OoI is the one detected first, i.e. the "arm roll" (class 7) for the "Left" context and "right hand clockwise" (class 2) for the "Right" context. As can be seen in Fig. 7.13, the "arm roll" has on average a higher number of events and active region, which facilitates its detection as OoI and explains the better RMSE value obtained compared to the "right-hand clockwise". These results point towards significant improvement using adaptation rules.

Control dataset validation

After verifying the performance of the saliency detection model with one OoI in the input data, we now wish to assess its behaviour and its bio-plausibility. Indeed, when confronted with a visual scene with multiple OoIs and asked to select only one, our hypothesis is that we expect a human being to always select either the one with the highest density of information (i.e. highest number of events and spatiotemporal density) or the one that comes first when the shift between the multiple OoIs is big enough. A bio-plausible saliency detection model would follow a similar behaviour ; we thus run our model on the pool of custom-made "random symmetric" samples (see Section III.A). Fig. 7.17 displays the results of this experiment: we can see that as expected, the Figure 7.17 -Performance of the attentional filtering according to the shift and diverse activity measures AM . The AM used here are the number of events N , the temporal density T D and the spatial density SD. The first plot presents the ratio of the values N , T D and SD averaged over the whole "random symmetric" dataset with no temporal shit, between the left and right concatenated samples. The colour of the blocks indicates the chance of selecting the right OoI using our model. According to this first plot, the right OoI is almost always selected first when its N , T D and SD are greater than the left OoI. The second, third and fourth plots detailed respectively the evolution of the detection performance of the first OoI (i.e. the one appearing first in the sensor) for a set of temporal shifts, according to the ratio of the N , T D and SD respectively between the first and second OoIs. The colour of the blocks indicates the chance of detecting the first OoI: it increases with the temporal shift as well as with the ratio. This means that the longer the timeshift between two OoIs as week as the greater the AM of the first OoI compared to the second, the higher the chance for the model to detect the first one as the most interesting. performance of detection of the first sample increases strongly with the shif t, reaching nearly 100% for a shif t of 1000µs and higher. On the other side, this performance is strongly degraded when the value of the activity measure of the first sample is smaller than the second one (i.e. where the ratio of the first sample's value to the second is smaller than 100%, highlighted by a green line on the figure). This is confirmed by comparing these results to the one obtained in case of no shift (plot on the left).

We can thus conclude that detecting the saliency in event data according to the event density is bio-plausible: the detected OoI corresponds either to the first object appearing in the scene (with at least a 1000µs delay compared to the others) or the one with the highest spatial density of events. We can thus indeed call such a model a spatiotemporal attention mechanism. Additionally,

Prophesee Pedestrian

We deployed our saliency model on a SpiNN-3 board and applied it to the dataset Prophesee Pedestrian in order to perform event denoising and pedestrian recognition. We compared our results with two other different approaches, Hynna's detection system and RPN-SNN. Hynna's detection system [START_REF] Singla | Hynna: Improved performance for neuromorphic vision sensor based surveillance using hybrid neural network architecture[END_REF] simply applies a connected component labelling algorithm over the input events and is meant for object detection in a similar use case. Inspired by [START_REF] Acharya | Spiking neural network based region proposal networks for neuromorphic vision sensors[END_REF], the RPN-SNN is closer to our approach but does not use the presented visual attention mechanisms. Instead, it features an additional refractory layer of the same size as the input window to filter out the noise.

Tab. 7.4 presents the measured precision and recall based on IoU between ground-truth bounding boxes and those predicted by our approach as well as the two models introduced above. One predicted bounding box can match one or several ground-truth bounding boxes as long as the IoU is over an arbitrary threshold.

Method

Precision Recall F1 Hynna's detection [START_REF] Singla | Hynna: Improved performance for neuromorphic vision sensor based surveillance using hybrid neural network architecture[END_REF] 0.252 0.896 0.393 RPN-SNN [START_REF] Acharya | Spiking neural network based region proposal networks for neuromorphic vision sensors[END_REF] 0.513 0.752 0.610 Ours 0.550 0.739 0.631 TABLE 7.4 -Recall, accuracy, and F1 score of compared methods over the whole dataset.

While this table alone suggests that our approach performs similarly to the RPN-SNN, these results do not differentiate performances in low and high noise conditions. To better evaluate whether the attention mechanisms we presented are able to efficiently filter out noise, we evaluated each video of our dataset separately and measured the noise ratio of each video. When the noise ratio comes close to 1, the corresponding data contains a large amount of noise and vice versa with 0. Fig. 7.18 presents performances relative to this ratio of noise. According to this figure, RPN-SNN performs best in relatively low-noise conditions: a ratio in [0, 0.4] yields better results than our approach. However, for higher noise values, our networks greatly surpass RPN-SNN both in terms of precision and recall. Tab. 7.5 presents a size comparison between our network and the RPN-SNN. It shows the mean spike count for both networks over time windows of 150ms with an 80 × 60 input window (640 × 480 input downscaled by a factor 8). Our network uses 26 times fewer neurons and approximately 4.3 times more synapses while producing almost 10 times fewer spikes. This trade-off is explained by the fact that we do not use an additional refractory layer to filter out noise and instead rely on visual attention mechanisms to ignore it, which requires fewer neurons and more synapses. A lower number of neurons and spike count, as shown in Tab. 7.5, should theoretically lead to lower energy requirements: neurons need to have their potential constantly updated whereas synapses are only used sparsely during the propagation of a spike. To validate this assumption, we will now present measurements directly taken on a neuromorphic hardware implementation of the presented networks.

We chose to implement our model on SpiNNaker to better assess the cost of our approach. The very small size of our network largely allowed it to fit on the small 4-chip SpiNN-3 board which can simulate up to 18K neurons. Tab. 7.2 presents energy consumption measurements for the different processes of simulation for a 150ms video segment that have been slowed down by a factor of 10 as the SpiNN-3 could not keep up with the flow of events. It should be stressed that the values from Tab. 7.6 are only meant to provide approximate energy measurements for the sake of comparison. Still, from this last table we can see that though our higher number of synapses increases the cost of building and loading the model, our lower neuron count leads to a lower energy consumption by the chips during runtime.

Validation of the simultaneous attention selection of multiple objects of interest

Fig. 7.19 presents the evolution of the performance of detection of two different OoIs depending on the tuning of synaptic and neuronal parameters: ω W T A which influences the number and size of detected RoIs (see Eq. 7.1), ∆θ which moderates the impact of salient activity on the output's thresholds (see Eq. 7.3) and t δ which delays the impact of the lateral output layers (see Eq.7.8). It highlights the importance of parameter tuning in SNN, and the difficulty to identify the correct parameters for each dataset -although we can conclude that it seems that ∆t has little impact on the selection performance. According to Fig. 7.19, in order to optimise the multi-OoI detection performance, this work was experimentally validated on PyNN using the hyperparameters defined in Table 7.7.

Parameters

Values We aim to assess the multi-OoIs selection performance as well as the quality of the output OoIs. Fig. 7.20 shows the evolution of the performance of multi-OoIs detection according to the temporal shift in the custom-made "control" datasets, as well as the detection latency and PLIFbased classification accuracy compared to the original (in orange). The drop in accuracy compared to the original can be explained by the smaller number of events contained in the multiple output OoIs (see Fig. A.3 in Annexes). Indeed, the PLIF-based classifier (Fang, Yu, Chen, Masquelier, et al., 2021) accumulates events in frames and therefore performs better on a dataset rich in events. In certain embedded systems performing autonomous navigation for example, the latency of the model's decision may be crucial to limit any risk of an accident. On Loihi, our attentional model selecting one OoI at the time rejects more than 50% of incoming unwanted events occurring only 20ms after activity onset. One might fear that extending this model to multiple OoIs might lead to a significant increase in decision latency -however, Fig. 7.21 demonstrates that our model maintains this latency performance, which is revised to the value of 14ms maximum without shift and decreases down to 5ms for an increasing shift.

Comparison with State-of-the-Art approaches

Table 7.8 compares our contributions with the neuromorphic saliency models implemented in [START_REF] Renner | Event-based attention and tracking on neuromorphic hardware[END_REF][START_REF] D'angelo | Event driven bioinspired attentive system for the icub humanoid robot on spinnaker[END_REF]. The data presented here were either retrieved directly from the information given by the authors or calculated from the description of each model. Those different metrics enhance our proposed model, whose implementation is resourceefficient while implementing additional features with lower latency.

Conclusion

We demonstrate in this chapter the significant benefits that might be derived from the application of a visual attention mechanism on event data, selectively focusing on relevant elements of sensory information thus improving the processing efficiency. This chapter described a neuromorphic algorithm for saliency detection in event data simultaneously selecting one or multiple OoIs. We highlighted the significant improvements to our results brought by adaptation rules. Incorporating such mechanisms in the design of future neuromorphic architectures will give rise to novel event-based attention applications.

Our proposed models are able to accurately detect and select n OoIs out of n initially present almost 50% of the time. The multi-OoI detection model selects at least one OoI with a delay of less than 15ms at most and reaching 5ms in the best cases. The quality of the OoIs selection is such that their classification reaches a performance of 73% the original data's while reducing the activated surface of the sensor by 1000. Our models can and should be implemented on neuromorphic hardware such as Loihi or Kraken to optimise the simulation time and energy consumption, as we demonstrate that it is amenable to resource-constrained embedded neuromorphic platforms and runs on the Kraken SoC platforms in 140ms. Applied to a pedestrian detection task, our largely surpasses the performances of the detection system of Hynna [START_REF] Singla | Hynna: Improved performance for neuromorphic vision sensor based surveillance using hybrid neural network architecture[END_REF]) and shows to be more robust to noise than the state-of-the-art [START_REF] Acharya | Spiking neural network based region proposal networks for neuromorphic vision sensors[END_REF]. Finally, our models achieve all the above with no training phase and a reduced number of neurons and connections compared to state-of-the-art salient detection methods.

Perspectives -Applications and deployments

In future works, we wish to validate this novel architecture to an increasing number of n OoIs as well as implement it on the neuromorphic hardware SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF], which allows for the implementation of dynamic adaptation rules, directly interfaced with an event camera. We believe in its usefulness in a realtime embedded task such as multi-object tracking or scene segmentation.

Additionally, we wish to validate the performance on our attention model on CV tasks with faster motion. Indeed, the results presented in this work were obtained on DVS 128 Gesture which, despite being a well-known dataset, requires a significant minimum observation time before decision (at least 1s according to (Fang, Yu, Chen, Huang, et al., 2021)), before which the latency of our attention model (at most 5ms) is negligible.

Perspectives -Towards bio-inspiration The review of attention's biological background presented in Section 5.1.1 allows us to develop some biologically-inspired proposals to adapt the concept of attention to computational models for purposes of responding to computer vision and multimedia tasks. Indeed, if those tasks are easily achieved by the human brain thanks to attention, the neural mechanisms involved deserve to be studied more thoroughly so as to profit from its efficiency and its robust performance.

Firstly, the forebrain presents a promising implementation of top-down attention. Featureoriented tasks with specific objectives are common in the domain of computer vision and multimedia (such as gesture recognition, object detection, etc.) and correspond fully to the forebrain's attention use.

An alternate perspective would be to exploit neuromodulator mechanisms. For instance, one could reproduce acetylcholine's modus operandi to enhance the capture of information from certain RoIs in visual scenes. Combined with a network mimicking the midbrain and its ability to detect the visual data relevant to the task at hand, this would lead to a higher information processing performance.

Another important avenue of investigation involves the exploitation of the link between saccadic eye movements and attention (Daucé et al., 2020), also understandable as the link between overt and covert attention. Such an attention could be achieved by implementing a retinocentric map of saliency, and then processing with a higher resolution on the place where the "focal gaze" is directed, i.e. where the saliency is higher. TABLE 7.8 -Comparison between the state-of-the-art and our contributions, i.e. models of saliency selection of one OoI (fourth column) and multi-OoIs (last column). In the notation, the input data is of size S × S, OL corresponds to the overlapping percentage described in [START_REF] D'angelo | Event driven bioinspired attentive system for the icub humanoid robot on spinnaker[END_REF] and div to the dividing factor between the input layer and the saliency detector in our models. A numerical value was calculated for each theoretical estimation, for S = 128, OL = 5% and div = 16.

n neurons 2S 2 S 2 × (1 + 4 OL 2 ) + 5 S 2 × (1 + 1 div 2 ) S 2 × (1 + 1 div 2 )
For S = 128: 32,768 19,010 17,408 17,408

n synapses S 2 × (2S 2 -1) S 2 × (4 + 20 OL 2 ) + 4 S 2 × (1 + S 2 div 4 -1 div 2 ) S 2 × (1 + S 2 div 4 -1 div 2 )
For S = 128:

Neuromorphic foveation

Foveation can be defined as the organic action of directing the gaze towards a visual region of interest, to acquire relevant information selectively. With the recent advent of event cameras, we believe that taking advantage of this visual neuroscience mechanism would greatly improve the efficiency of event-data processing. Indeed, applying foveation to event data would allow to comprehend the visual scene while significantly reducing the amount of raw data to handle. In this respect, we demonstrate in this chapter the stakes of neuromorphic foveation theoretically and empirically across several computer vision tasks, namely semantic segmentation and classification. We show that foveated event data has a significantly better trade-off between quantity and quality of the information conveyed than high or low resolution event data. Furthermore, this compromise extends even over fragmented datasets. The significant advantage of using foveation on embedded event data highlighted in this chapter reinforces the interest in developing a foveated camera such as the recent e-Fov DVS, which use will be similarly evaluated in future work.

Introduction

Traditional and neuromorphic computer vision models can have difficulties handling a great amount of data simultaneously while minimising their energy consumption, especially at a high temporal resolution. We explored in this work two potential remedies for such an issue: event data downscaling (see Chapter 6) and salient selection of OoIs in the event flow (see Chapter 7). However, we believe that the trade-off between information retention and data reduction with existing methods is not yet ideal and can be improved using neuromorphic foveation, a concept designed by Teresa Serrano-Gotarredona and Bernabe Linares-Barranco at IMSE (Sevilla, Spain). This work is a first step towards experimental validation of a neuromorphic foveated sensor. Figure 8.1 -Overview of the neuromorphic foveation concept as defined in this work. The results presented below correspond to the software neuromorphic foveation (red full arrows) as a validation for a future hardware neuromorphic foveation (orange dotted arrows). The multi-resolution event sample used as an example of the output of such a process is extracted from the DDD17 dataset [START_REF] Binas | DDD17: End-To-End DAVIS Driving Dataset[END_REF].

To demonstrate the interest in applying foveation to event data, we define in this work the concept of neuromorphic foveation (see Fig. 8.1), which consists of a feedback loop between an event camera and a neuromorphic saliency detector, merging events at multiple resolutions according to the detected RoIs. The detection of such RoIs relies on the visual attention mechanism described in Chapter 7.2: a set of coordinates in the event sensor is considered interesting if it is strongly activated in close temporal and spatial proximity. Such a process should be allowed by a foveated DVS as described in [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF] ; however, as this sensor is not yet available at the time of writing, we validate here the neuromorphic foveation concept with a first software implementation (red pathway in Fig. 8.1) before extending this feedback loop system to hardware (orange dotted pathway) in future works.

With the help of APROVIS3D collaborators, we studied the respective evolution of the amount of event data processed in a computer vision task and its accuracy when software neuromorphic foveation as described above is applied. In order to simulate the foveation, the event data will be processed at a higher or lower resolution, depending on the relevance of the spatial regions in the image at different coordinates. The proposed model goes beyond biology by allowing multiple RoIs of arbitrary size and shape.

The following sections establish a detailed outline of the different hardware and mechanisms involved in this work ; a complete description of the software neuromorphic foveation methodology ; and the experimental evaluation procedure. Our contributions are the following:

-A complete description of the binary foveation process and the different hyperparameters to facilitate the reproducibility of our neuromorphic foveation mechanism ; CHAPITRE 8 -Neuromorphic foveation -A complete comparison of the semantic segmentation performance between the original, reduced and foveated DDD17 dataset using various spatial event data downscaling, with results provided by PhD student Dalia Hareb (i3S) ; -The evaluation of the neuromorphic foveation on a second computer vision task, i.e. the classification of the DVS 128 Gesture dataset, with results provided by Antoine Grimaldi (INT) ; -An additional comparison of the foveation to another data pre-processing, where we study the semantic segmentation and classification performance on selected RoIs ; -A study of the overall qualitative and quantitative aspects of saliency detection.

To the best of our knowledge, this work and our results are the first to show that foveation offers a significantly better compromise between quantity and quality of the information than high or low resolution, especially on fragmented datasets. Furthermore, we demonstrate that our saliency detector is specifically efficient on data reduced using the event count method.

Our code is implemented in Python using the PyNN library (Davison et al., 2009) * . The results presented in this chapter were made publicly available on three occasions: a short abstract defining the concept and methodology was presented during the French national conference ORASIS 2021 [START_REF] Gruel | Stakes of foveation on event cameras[END_REF], an extended abstract with a first proof of concept was featured in the CVPR 2022 workshop "NeuroVision: What can computer vision learn from visual neuroscience ?" (Gruel, Hareb, Martinet, Linares-Barranco, & Serrano-Gotarredona, 2022). A journal paper was also submitted to the Biological Cybernetics special issue "What Can Computer Vision Learn from Visual Neuroscience ?" [START_REF] Gruel | Stakes of neuromorphic foveation: a promising future for embedded event cameras[END_REF].

Methodology -Binary foveation

In this work, we consider the foveation process akin to the combination of a sample's events in high resolution and low resolution using a mask, as presented by Fig. 8.2. This binary combination is a software simplification of the neuromorphic foveation concept as described above and discriminates the fovea (events in high resolution -purple region in Fig. 8.2) from the retinal periphery (low resolution ; i.e. spatially downscaled -yellow region in Fig. 8.2). The RoI is detected using the saliency detector described in Section 7.2 ; it is thus assimilated to the fovea. The saliency detection and binary foveation are performed using the hyperparameters described in Tab. 8.1.

As explained above, the software implementation of neuromorphic foveation used in this work is akin to a binary foveation process merging event data of two different resolutions together. As it is more difficult to produce higher resolution data of an original dataset, we chose to use four of the spatial event downscaling methods described in Section 6: the spatial funnelling, the event count and the linear and cubic log luminance. In order to remain consistent with a future conversion of this software foveation model into hardware, where the saliency is detected on aggregated pixels before refining the resolution at the relevant areas in order to minimise the bandwidth, we follow the process described in Fig. 8.2. The original dataset, corresponding thereafter to the denomina- tion "high resolution", is spatially reduced before being given as input to the saliency detector. A subsequent "multi-resolution merging" process then combines both datasets into foveated event data according to the detected RoIs.

Parameter

Values Figure 8.3 -Classification performance on DVS 128 Gesture according to the number of events in the dataset in high resolution (in blue), low resolution (in green), with only RoIs (in yellow) and after foveation (in red). The dashed line connects the results of high and low resolutions ; an ideal result would lie above this line, thus achieving a better trade-off between number of events and performance than the original and reduced data. The subplot on the left depicts the mean values and the corresponding confidence ellipsis ; it shows that the foveation applied to classification leads on average to a good trade-off between number of events and performance since the resulting accuracy lies in the upper left area. The mean values are plotted as an overlay in the subsequent subplots to assess the quality of the foveation and the reduction compared to the others. The tradeoff between classification performance and number of events per sample is overall better for the foveated data, especially using the log-luminance reconstruction methods.

SNN model

Experimental validation

To validate our proposed model, we apply two traditional computer vision tasks, semantic segmentation and classification, to foveated and spatially reduced datasets. All datasets are spatially downscaled by a dividing factor of 4. This section describes the datasets and the different models used to perform such tasks, as well as the comparative results. As presented in Tab. 3 and Fig. C.4 (in Annexes), the event data's properties were compared for sample in high resolution (original dataset), low resolution (spatially downscaled with factor 4) and foveated (binary combination of the previous two).

Performance on a classification task

We test the impact of the neuromorphic foveation on a classification task, which assigns a label to each sample of a dataset, using the HOTS model (see Section 4.3.1.3). Fig. 8.3 presents a comparison between the different versions of the DVS 128 Gesture dataset (Amir et al., 2017), i.e. in high (blue marks) and low (green marks) resolutions, after RoIs selection (yellow marks) and after foveation (red marks), according to the classification model HOTS's performance. Fig. 8.3 depicts the trade-off between the accuracy, the MIoU and the mean number per sample for each reduction method -the first subplot corresponding to the mean and confidence ellipse for all methods combined. We compare the classification performance on the foveated dataset and on the events belonging solely to the detected RoIs in order to validate the interest of foveation: indeed, the RoIs contain only the most interesting information in the visual scene (according to our defini-8. 3 -8.3.1 Performance on a classification task 153 Figure 8.4 -Semantic segmentation performance according to the number of events in the dataset in high resolution (in blue), low resolution (in green), with only RoIs (in yellow) and after foveation (in red). The dashed line connects the results of high and low resolutions ; an ideal result would lie above this line, thus achieving a better trade-off between number of events and performance than the original and reduced data. The subplot on the left depicts the mean values and the corresponding confidence ellipsis ; it shows that foveated data managed to keep on average the same performance as the original dataset while decreasing by two-thirds the mean number of events to reach the low resolution's value. The mean values are plotted as an overlay in the subsequent subplots to assess the quality of the foveation and the reduction compared to the others. This highlights the clear advantage of using the methods event count and linear compared to funnelling.

Figure 8.5 -Semantic segmentation performance evolution according to the percentage of total events in the dataset after processing for the event data in high resolution (in blue), low resolution (in green), only from RoIs (in yellow) and after foveation (in red). The different subsets are obtained by downscaling the respective datasets using the stochastic structural downscaling method for p ∈ [0. 2, 0.4, 0.6, 0.8, 1]. The subplot on the left depicts the mean values and how all four types of data behave similarly. According to the subsequent subplots, it is interesting to note that, except for the cubic method, the foveated data overall outperforms the high-resolution data from a 60% decrease and downwards. The funnelling method is the one maintaining the highest accuracy for the longest, either in the foveated or reduced dataset. 8.3 -8.3.2 Performance on a segmentation task 155 tion of saliency) but might miss contextual information helping to classify the data correctly. For example, in DVS 128 Gesture one might need to know about the subject shape to understand that the moving OoI detected by the model is the arm. Foveation however keeps the information provided by the non-interesting regions but in a reduced format, thus providing interesting information at a high resolution and contextual, less interesting information at a lesser resolution.

Performance on a segmentation task

To validate the neuromorphic foveation, we evaluate it using the semantic segmentation model Ev-SegNet (see Section 3.3.3). Fig. 8.4 and 8.5 present a comparison between the different versions of the DDD17 dataset [START_REF] Binas | DDD17: End-To-End DAVIS Driving Dataset[END_REF], i.e. in high (blue marks) and low (green marks) resolutions, after RoIs selection (yellow marks) and after foveation (red marks), according to the semantic segmentation model Ev-SegNet's performance. One could wonder how a semantic segmentation model would behave on data containing solely events within the identified RoIs (yellow marks in Fig. 8.4 and 8.5). Thanks to a study performed by Dalia, we know that Ev-SegNet assigns the class 0 (for background and sky) to segments with no events ; in the case of RoIs selection, it means that most of the scene is labelled as background and the other classes only appear in the RoIs, which explains the low performance of semantic segmentation on data post RoIs selection (Fig. 8.4) compared to the classification performance observed in Fig. 8.3. Fig. 8.4 depicts the trade-off between the accuracy, the MIoU and the mean number per sample for each reduction method -the first subplot corresponding to the mean and confidence ellipse for all methods combined. Fig. 8.5 shows the evolution of the segmentation performance for an increasingly sub-sampled input dataset for each pre-process. Each pre-process including spatial reduction (i.e. low resolution and foveated data) is detailed according to the spatial downscaling method used. Each dataset is reduced using the structural stochastic temporal downscaling method introduced in Sec. 6.3.2, with p ∈ [0.2, 0.4, 0.6, 0.8, 1] corresponding to the values in the x-axis of Fig. 8.5.

In those first two figures, the foveation is obtained by detecting the saliency on the data downscaled using the specified method, then merging it with the dataset reduced using the same method. This allows us to stay as close as possible to the hardware foveation design by simulating the feedback loop by using twice the same reduced data. Similarly to the classification results, we compare the semantic segmentation performance on the foveated dataset and on the events belonging to the detected RoIs in order to demonstrate the importance of keeping the information provided by the non-interesting regions but in a reduced format. Fig. 8.6 studies the qualitative aspect of saliency detection by presenting the semantic segmentation according to the mean number of events per sample. Each foveation method, i.e. each method producing data on which the saliency was detected, is indicated in different colours. The average values for all datasets foveated using one method are presented in opaque, while each specific dataset's performance and number of events are overlayed.

As we want to maximise the performance while minimising the number of events, the goal to be reached is situated in the upper left corner of this plot. Event count emerges as the method for the most efficient salience detection. The temporal density D t corresponds to the activation probability of pixels averaged over the whole sensor and is defined in Eq. 6.5 (in Chapter 6).

Quantitative assessment of the saliency detection

A contrario, the spatial density D s is the activation probability of the whole sensor over a limited time-window averaged over the temporal length of the sample, as described in Eq. 8.1.

D s = T t=t window P [t-tw,t]
N w (8.1) with t w and T respectively the length of the time window and the length of the sample in time.

The results in Fig. 8.7 are presented for t w = 50µs. N w corresponds to the number of successive time windows in the sample, as presented in Eq. 8.2.

N w = ⌈ T t w ⌉ (8.2)
The activation probability P [t-tw,t] is calculated as the ratio between the number of pixels activated during the time-window [t -t w , t] and the overall number of pixels in the sensor:

P [t-tw,t] = t T =t-tw w x=1 h y=1 δ(t x , t) • δ(t y , t) w • h (8.3)
with w and h respectively the width and height of the sensor, t x and t y the timestamp of any events occurring at coordinates (x, y) and δ the Kronecker delta function, which returns 1 if the variables are equal, and 0 otherwise.

Discussion

To validate our initial hypothesis, the foveation would have to produce a number of events significantly closer to the low resolution while allowing for a performance closer to the high resolution. In other terms, the foveated results should be above the dotted grey line on Fig. 8.3 and Fig. 8.4. We do observe a striking decrease in the number of events between pre-(high resolution) and post-processing (low resolution and foveation) of the dataset. Concerning the DVS128 Gesture dataset and its classification performance (Fig. 8.3), the average number of events per sample drops to 1% of the original metric after reduction and to 50% after foveation, while the performance decreases respectively by 63% and only by 25%. A similar behaviour is found with the DDD17 dataset (Fig. 8.4): on average, the spatial downscaling and the foveation keep 30% of the original events. The most important drop of events is seen with the linear method: the reduced data drops to only 1% of the original size, and the foveated data to 10%. Similarly, the foveation's semantic segmentation performance is averaged over all methods and is remarkably close to the high resolution's performance. Fig. 8.4 pictures an outlier, the cubic method. The trade-off between performance and data size displayed by the foveation is not as good as in other methods ; this can be explained by the fact that the method produces too sparse data to offer a coherent saliency detection. The plot depicts the average semantic segmentation performance according to the spatial downscaling method used to generate the data. The overlayed dots correspond to different reduction methods (with no indication of the corresponding methods) with which the multiresolution merging was achieved using the RoIs detected with one specific method. The goal is to minimise the mean number of events per sample while increasing the performance, thus best results are close to the subplot's upper left corner. Foveated data using RoIs detected on event count is the closest to this goal. Figure 8.7 -Quantitative study of the saliency detection, by comparing the RoIs' spatial and temporal density according to the mean number of RoIs detected per sample. While there is neither good nor bad result in this graph, it shows that the funnelling method yields a significantly higher temporal and spatial density, as well as the highest mean number of RoIs detected.

Additionally, we compare the performance of semantic segmentation and classification on foveated events to events belonging in the detected RoIs, pictured in yellow in Fig. 8.3, 8.4 and 8.5. For both tasks, the performance shows a significant decrease when achieved on RoIs events. The ratio of segmentation accuracy and MIoU to mean number of events is similar to the one obtained on reduced data, if not worse for the event count method. The classification performance does not show such an intense decline but is still significantly further away from the ideal compromise than that obtained on the foveated data. This difference is explained by the intrinsic distinction between the datasets and tasks: labelling a gesture in DVS 128 Gesture can be done by identifying the movement of the hands, i.e. by following the trajectory of the RoIs ; while segmenting a driving scene requires more information spread out on the whole scene and cannot rely solely on the data provided in the RoIs, although these greatly facilitate the task.

All in all, those observations combined do confirm our core thesis, that is, that neuromorphic foveation leads to the best trade-off between information quantity and quality, at least on a software level. Neuromorphic foveation is more relevant than attentional filtering to preserve event data while minimising its size.

Furthermore, it is interesting to note that when comparing the proportional decrease of the number of events in the dataset post-process in Fig. 8.5 while all three types of data show the same behaviour, the averaged foveated data outperforms the high-resolution data from a 60% decrease and downwards. This is explained by the fact that the majority of events kept in the foveated dataset provide relevant information to the semantic segmentation model, while a significant part of the events in the original dataset is not as useful.

The less important drop of the data size when applying foveation to the classification of DVS 128 Gesture (Fig. 8.3) compared to the one presented in Fig. 8.4 is explained by the inner properties of the DVS 128 Gesture dataset: the lower spatial density of this dataset compared to DDD17, due to a static setting and a non-moving camera, leads to the detection of more densely aggregated RoIs already containing most of the sample's event data. In other words, the principal OoI in DVS 128 Gesture is the hand, which will be the main subject of saliency detection due to its constant movement. This is confirmed in Fig. C.4 where the high resolution is only visible on the hand in the foveated sample. As the movement of the hand is the main cause of event production while recording the scene, the corresponding region contains the most events -thus the foveation on the DVS 128 Gesture dataset does not decrease the data size as much as when applied to a moving camera recording.

This theoretical reasoning is confirmed by the results presented in Fig. 8.7: the mean number of RoIs detected on DVS 128 Gesture and its corresponding temporal and spatial densities (right subgraph) are all significantly greater than those detected on DDD17 (middle sub-graph). Fig. 8.7 also highlights the important quantitative variations of the saliency detection according to the foveation method used: funnelling produces a great amount of RoIs, due to its lack of event drop ; while linear detects the least saliency. The choice of the method used to detect the salience is not trivial, each one having its interests and its disadvantages. One must thus take time to think upstream about the physical properties of the dataset and the desired intensity of the foveation. All in all, when applying the foveation to the classification of DVS 128 Gesture and semantic segmentation of DDD17, event count seems like a good trade-off between the three proposed methods.

The qualitative study of the saliency detection displayed in Fig. 8.6 using different methods (i.e. detecting saliency on data reduced using those methods) reinforces the interest in using the event count method for foveation. Indeed this figure highlights the overall advantage of event count, as its use leads to a minimised data quantity for a maximised semantic segmentation performance.

Conclusion

In this work, we demonstrate the stakes of foveation applied to event data for semantic segmentation and classification. Such a strategy does concurrently preserve the accuracy of event data processing and greatly reduces the amount of data needed for the task.

To meet our initial goal of finding the ideal process to reduce the number of events while maintaining the quality of the information transmitted, a final aspect can be discussed here to complete the results presented above: that of the generation time. Indeed, the spatial reduction of events as well as the neuromorphic detection of salience requires a non-negligible processing time. The first has been discussed in Chapter 6 and depends strongly on the selected reduction method. The second varies with the hist architecture used. Indeed, the SNN model can be run either on CPU, using an SNN simulator such as PyNN [START_REF] Davison | Pynn: a common interface for neuronal network simulators[END_REF], or on GPU with adapted simulators such as Norse (Pehle & Pedersen, 2021). In both cases, the simulation time increases with the number of neurons in the model and the size of the input data. A third option is to use neuromorphic chips, such as Human Brain Project's SpiNNaker (S. [START_REF] Furber | Spinnaker -a spiking neural network architecture[END_REF] or Loihi [START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF], which enable fast and low power simulations and which simulation time only depends on the input data size. All in all, it is very unlikely that these two processes (reduction and saliency detection) combined can be done in real-time.

However, we seek to convert this software neuromorphic foveation process into a hardware implementation using the foveated sensor introduced by [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF]. With such a sensor that implements spatial reduction and foveation electronically, we can disregard the generation time.

Further work will implement the hardware neuromorphic foveation feedback process by concurrently using the foveated DVS presented in [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF] and neuromorphic hardware, e.g. SpiNNaker, in order to record foveated event data in an embedded fashion.

CHAPTER 9

Towards embedded applications -First steps and perspectives Now that we have demonstrated the interest of using various approaches to pre-process event data for efficient embedded processing in previous chapters, we propose in this chapter to study original embedded applications. Two of the tasks addressed below are motivated by our involvement in the APROVIS3D project. The target use cases of this project are 1) the detection of the coastline by an unmanned aerial vehicle flying around an island of the Mediterranean Sea to allow for autonomous navigation and 2) the 3D reconstruction of this coastline. We propose various approaches to tackle these axes of research and present some preliminary results. We also take an interest in embedded lipreading, an advantageous neuromorphic task to demonstrate the criticality of the time dimension of event videos when properly exploited. The perspectives of this work include the development of a comprehensive dataset for coastline detection, the implementation of a demonstrator for embedded 3D reconstruction, the development and extensive experimental validation of a model of weight and delay learning and the implementation of SpiNN-3 of an existing unsupervised delay learning model.

Introduction

In the Contributions part of this manuscript, previous chapters focused on various approaches we implemented during this PhD to pre-process event data using SNN. We aimed to optimise the performance of embedded computer vision tasks by minimising the data size while maximising the information conveyed. Now that we demonstrated the benefits of the three proposed methodologies, i.e. event data downscaling in Chapter 6, visual attention in Chapter 7) and neuromorphic foveation in Chapter 8, we consider the embedded processing of event data applied to various applications.

Event data interfaced with an SNN model on embedded neuromorphic hardware allow for a large panel of energy-efficient, low-memory applications (due to both hardware architectures and intrinsic data characteristics -see Chapter 4). We already considered two common tasks in previous chapters, although it was for the purpose of validating various pre-processing methods and not meant to be embedded:

-the classification of either gestures or numbers, respectively applied to the datasets DVS 128 Gesture (see Sec. 3.3.1) and N-MNIST (see Sec. 3.3.2). We used three published SNN models: the PLIF and SLAYER benchmark classifiers as well as the HOTS model (see Sec. 4.3.1). -the semantic segmentation of a driving scene, applied to the DDD17 dataset using . During this work, we took an interest in two additional tasks for which the literature is less extensive. Through the supervision of interns and collaborations with other members of the APROVIS3D consortium, we have produced some first results that are described in this chapter.

The first application we address is motivated by a use case of APROVIS3D. Even though a high temporal resolution is not required for this task, it serves as a support for a first approach to exploiting the natural match between SNN and event data. As described above, this project aims to implement a neuromorphic architecture embedded in an unmanned aerial vehicle (UAV) flying over the Mediterranean Sea, which performs coastline detection for the autonomous navigation of the drone. Our collaborators at NTUA in Athens, Greece, have recorded RGB and event videos of the beach and the Mediterranean Sea as seen from a UAV, at different heights and with different speeds (see the example given in Fig. 9.1). Additionally, at UCA, we similarly recorded RGB videos of the beach and the Mediterranean Sea that we later translated into events to complete this dataset, which we will use as input to our approaches.

To tackle this task, we propose three methodologies and describe the first results below. A first approach would be to translate the ANN classifier HOTS to an embedded SNN for sea-ground segmentation: HOTS would differentiate between sea and ground patches in the original data, thus allowing to extrapolate the coastline. The "patches" are sampled and extracted from the latticed recorded data. This approach is innovative, but may be difficult to generalise: it depends on the environment and is based on certain conditions (rough seas, more movements in the water than in potential vegetation, etc). It is currently the object of a work group between NTUA, UCA and INT (in Marseille, France) in order to implement a demonstrator by the end of the APROVIS3D project (September 2023).

A second approach uses a delay learning model to differentiate between the ground and the sea by the movement of the events: those recorded on the ground will correspond to the ones generated by the ego-motion of the embedded camera only, while the additional motion of waves will disrupt the movement of those recorded on the sea. The implementation and application of the delay learning model introduced in [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF] have been the subject of multiple internships supervised during this PhD. We also propose as a perspective a potential learning rule inspired by the STDP and applied to delay -this proposition was presented as a poster during the French national conference JOBIM 2023 (Gruel & Martinet, 2022). Additionally, a complete review of "Precise Spiking Motifs in Neurobiological and Neuromorphic Data" was published in the MDPI -Brain Sciences journal (Grimaldi et al., 2022).

The third and final approach we propose to tackle coastline detection is a simple line detection model. This would allow us to remove the intermediate step of classifying patches into sea and ground and directly detect the coastline. We present here some first promising results on simple synthetic event data.

In this chapter, we also present various works implementing tentative first steps to answering the second task targeted by the project, i.e. the 3D reconstruction of the Mediterranean coastline. Two interns, Ms Huiyu Han and M. Noah Candaele, supervised during this PhD worked on this problem over a total period of 7 months. We present below a summary of the work accomplished during this period.

Finally, we address a third embedded application: neuromorphic lip-reading. During this PhD, we supervised two grouped internships on this subject, leading to the recording of an event-based lip-reading dataset and implementation of the first-ever end-to-end neuromorphic model for embedded lip-reading, which could directly benefit the development of portable assistive devices or surveillance equipment. This model and results presented below were made publicly available in an article presented during the CVPR 2023 Workshop on Event-based Vision [START_REF] Bulzomi | End-to-end neuromorphic lip-reading[END_REF].

Coastline detection

As presented above, the detection of the Mediterranean coastline is the main application task targeted by the APROVIS3D consortium and a demonstrator is to be produced by the end of the project. The following section describes the three lines of research conducted in order to answer this problem.

Adapting an ANN classifier into an embedded SNN classifier

A first attempt to solve this task is carried out by a dedicated work group composed of APRO-VIS3D members from NTUA in Athens, Greece, from INT in Marseille, France and from UCA.

At NTUA, PhD student Sotiris Aspragkathos and Prof. George C. Karras record event data and RGB videos of the sea and the coastline ground, separated and mixed. Once the data is acquired, at UCA, we handle the verification of the provided videos' quality and translate their formats into a more universal format. This modified dataset is then forwarded to PhD student Antoine Grimaldi and Prof. Laurent Perrinet at INT. Having implemented the HOTS classifier from previous work, they update this architecture to efficiently differentiate between patches of sea and ground.

Once it is properly adapted to this task, we at UCA will take care of implementing the HOTS network with the hyperparameters learned during training on the neuromorphic hardware SpiNN-3. The final topology will finally be embedded on the UAV set up by NTUA.

Input data

NTUA records event and RGB videos using an embedded DVS sensor provided by IMSE, as well as a CSL's DAVIS346 mono camera. This data could only be saved as an AEDAT2 file, using the object structure described in Tab. 4 in Annexes.

However, this format is no longer readable in Python because the associated libraries are no longer supported. Additionally, the HOTS implementation used by INT takes input data according to the Tonic format [START_REF] Lenz | Tonic: event-based datasets and transformations[END_REF]. This library stores and handles events according to a specific object structure, with specific intrinsic modules and attributes described below.

We thus implemented a pipeline to transform the AEDAT2 files into NPY files. Additionally, we also developed a pipeline to produce NPY files from RGB videos recorded in a similar context. This allows us to comparatively analyse the event data obtained from the DVS sensor and the standard RGB camera.

Pipeline -event data, from AEDAT2 to NPY As expressed previously, the AEDAT format has slowly evolved from version 2 to 4 over the last few years and current versions of Python libraries can no longer handle the AEDAT2 format. After some research, we found the AedatTools functions, implemented in Matlab and made publicly available by a Github user (qiaokaki, 2017). We leveraged these tools to implement a Matlab script translating an AEDAT2 file into a CSV file. A second Python script is then used to transform this CSV file into a structured Numpy array, saved as an NPY file. The two scripts described above can be found in the Annexes (Algorithms .1 and .2).

Pipeline -RGB data to event data When recording event data from various UAV flights, NTUA also recorded and made available to us corresponding RGB videos. In addition, the early recorded videos were not suitable for learning sea and ground patches only as these videos indiscriminately mixed the two scenes. While waiting for new event data to be recorded by NTUA, we recorded with the help of the robotic team of the I3S laboratory, some sea-only and ground-only scenes using a RGB camera. We then translated these videos into event data using the "Video-to-events" library (D. [START_REF] Gehrig | Video to events: Recycling video datasets for event cameras[END_REF]. This synthetic dataset, although imperfect in its synchronous aspect, allowed us 1) to have data on which to begin the HOTS training and 2) to compare these "ideal" data to the data recorded in the field by the embedded DVS sensor (see Sec. 9.2.1.1).

Tonic dataset structure Tonic wraps event data according to a specific structure (Lenz, 2019): an object is created for each dataset, inspired by TorchVision's dataset module (Linux Fundation, 2017). It requires methods to initialise the object, get a specific element of the dataset and output its length. The events from one sample of the dataset are stored as a structured Numpy array of shape (N, 4), with N the number of events and the four channels corresponding to the x and y coordinates, timestamp and polarity of each event.

We implemented our own wrapper to allow for the HOTS model to easily use the event data recorded by NTUA as input. A custom module was added to those required by Tonic: this function splits the input stream of events into patches defined spatially and temporally, themselves split into training and testing samples according to a user-defined ratio.

This wrapper can be applied to event data either recorded by NTUA's DVS sensor or produced from RGB videos, as long as the input files are of the format NPY or NPZ.

Validation of produced datasets Once both the DVS recordings and synthetic event data were wrapped according to the Tonic structure, we analysed both datasets to assess the quality of the data. However, the analysis of the first dataset provided by NTUA displayed many incoherences such as long periods of time (up to 10s) with no events and the absence of activation of certain pixels over the whole recording. According to Prof. Teresa Serrano-Gotarredona (a microelectronic researcher at IMSE, Spain and member of the APROVIS3D consortium), this behaviour is explained by the presence of the shadow of the UAV helix in the DVS sensor's field of view, which causes a high flow of events. The event rate produced by the DVS sensor has an upper bound ; additionally, the FPGA board handling the recording has a maximum speed due to the time needed to put a timestamp to the events, temporarily store them and send them to through the USB bus once the buffer is full. Those three elements combined lead to the saturation of the camera and intermittent communication of the events: the intermittent activation of rows of pixels is due to the delay in transmission of the output spikes ; the time periods of inactivation are caused by the higher rate of buffer refills than FPGA timestamping and communication through the USB bus, causing event loss.

Following this interpretation, Sotiris Aspragkathos recorded new data with an event rate lowered by reducing the sensor sensitivity and stimulus speed and contrast. After analysis, this new dataset is deemed adequate to be used as a support to train HOTS and perform coastline navigation.

The different datasets and analysis described above can be found in Annexes, in Sec. E. The first dataset recorded using a DVS sensor by NTUA and presenting issues is referred to as "NTUA_DVS_2022", and the second one recorded in similar conditions is referred to as "NTUA_DVS_2023". The dataset recorded by an RGB camera by UCA then translated into events is referred to as "UCA_synthetic_2022".

Perspectives We aim to record and make available to the public a more complete and improved dataset for the coastline segmentation task, from which we ultimately wish to infer segmentation offline. The data would be recorded using a more standard DVS (such as a DAVIS camera, whose characteristics will be specified when the dataset is opened to the public). We established the following requirements to accomplish this task:

-at least three levels of UAV altitude (for example, 5, 10 and 15 meters -knowing that the UAV does not usually fly higher than 20 meters in the project use case) ; -at least 3 levels of UAV speed (knowing that the UAV does not usually fly more than 5 m/s in the project use case) ; -different textures: sea, sand, grass, bushes, rocky coast, etc ; We would record three one-minute-long samples for each combination of requirements described above, thus collecting a total of at least 135 samples in the complete dataset. All samples would be labelled with the values corresponding to each requirement described above. We believe that such additions to the dataset would make segmentation more adaptive.

Adaptation on SpiNNaker

As HOTS has a relatively simple architecture, once trained to detect patches of ground and sea, this model will easily be implementable on an embedded SpiNN-3, using the Python library sPyNNaker (see Sec. 2.2.2). Each set of hyperparameters learned offline will be set to enable online inference, thus automatic coastline detection and autonomous navigation of the drone.

Delay learning

A second approach to coastline detection would be to classify smaller samples of the scene captured by the sensor according to the movement of the recorded events. Indeed, depending on whether the events recorded correspond to the ground or to the sea, their movement is quite different: as can be seen in Fig. 9.3, the events corresponding to the ground are generated by the sensor's ego-motion [START_REF] Chapel | Moving objects detection with a moving camera: A comprehensive review[END_REF] thus follow a vertical, north-south motion ; however the movement of the events corresponding to the sea is the sum between the movement generated by the ego-motion (vertical motion) and by the waves (horizontal motion when following the coastline). We thus propose to classify those samples using delay learning which performs particularly well for spatiotemporal pattern recognition (see Fig.9.2,9.3 and 9.4). Varying number of convolutions trained on noiseless data As stated precedently, while Hugo used an approach to visualise the learnt features similar to the authors, one goal of Kévin's internship was to develop new visualisation tools to assess the quality of the training: we can now visualise the actual evolution of the weight and delay values over the training phase (see Fig. 9.6). Additionally, we updated Hugo's code so that the model can be more modular and one can easily modify the number of convolution layers, as can be seen in Fig. 9.7. The variation in the number of convolution layers does not seem to influence the quality of learning according to the metrics displayed in Fig. 9.9 ; the accuracy is significantly high, meaning that the network is almost never wrong when predicting a given motion in the input data. However, the model quite often fails to recognise an existing motion, with a probability close to 0.5 according to the recall value. ). The precision is displayed in blue, the recall in orange and the F1-Score in green. Figure from [START_REF] Constantin | Synaptic delays for temporal pattern recognition[END_REF].

Defective lateral inhibition While showing a certain learning deficiency, the results presented in Tables 9.2a and 9.2b display a certain coherence: diagonals A and B, as well as C and D, seem to respectively produce similar clusters of good quality, specialising for the same trajectory independently of the motion direction. This behaviour of double specialisation can also be observed on noiseless data in certain simulation runs, such as in Fig. 9.8. It seems that it is common for two layers to be occluded: a layer specialised for one motion has weights correctly adjusted for its trajectory ; only the delays allow us to differentiate between the direction of this motion. This phenomenon is supposed to be avoided thanks to a lateral inhibition mechanism that appears to be defective in this architecture.

This issue could be addressed by implementing a more efficient lateral inhibition mechanism or by adjusting the parameters according to the number of convolution layers, the window size, the event density...

Tentative model of delay learning

After studying the neuromorphic state-of-the-art and especially the bio-plausible model introduced in [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF], as well as investigating the potential biological explanations for delay learning in (Grimaldi et al., 2022), we propose here a tentative learning rule for simultaneous weight and delay learning inspired from the traditional STDP learning rule, for recognition of spatiotemporal patterns. Here the weight would follow a traditional STDP rule (see Sec. 2.1.4.3) and the delay would be updated over time according to the following equation:

Perspectives

Our work on delay learning is still in progress, and several avenues are open to us to improve this model and apply it to our coastline detection application. Concerning the re-implementation of the model of [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF], a first step would be to achieve significantly accurate results on noisy input data, with or without noise in the training dataset, to more easily adapt the network to real-life data. A second would be to increase the number of patterns to be detected: the network performs well for 4 directions, but an application to real-life data would require the distinction between at least 8 directions. However such an experiment necessitates increasing the kernel size, thus asking to increase the computational capacities of the support machine.

Once those two steps are achieved, we could hope to apply this model to samples extracted from latticed real-life event data, either by inferring motions from the features learnt on synthetic data or by training our model directly on real-life data.

Finally, we aim to implement this model on an embedded SpiNN-3 board to achieve online classification to ultimately perform coastline detection and autonomous navigation.

The last two steps could similarly be followed with our original weight and delay learning rules, provided additional efforts have been lead towards the implementation and experimental validation of this model.

Line detection

As a third and final approach to coastline detection, we developed a small SNN for line detection, easily embedded on a SpiNN-3 to interact with an event camera. As this model is still under development, we have only produced a few first results which we present in the paragraph below.

Horizontal and vertical lines

The first network we implemented allows for the detection of vertical and horizontal lines, without (first version -see Fig. 9.11) or with (second version -see Fig. 9.12) the position in the sensor of the input lines.

For an input sensor of n × n, the first version of this network requires 2(n + 1) neurons and 2(n 2 + n) excitatory synaptic connections. Each input neuron is connected to one neuron in the vertical line detector and one in the horizontal line detector according to a low weight ω so that the latter is activated only if all the neurons of the same (vertical or horizontal) line are activated. Each of those two detectors of n neurons, arranged in linear ways according to a 2D orthogonal reference frame, are then respectively connected according to a strong weight ω to a neuron playing the role of overall supervisor. If one of the two neurons is activated, at least one line of the corresponding type is detected.

In our case, we need the detection of the coastline to allow for the autonomous navigation of the drone, therefore we need to know the position of each detected line. We implement a second version presented in Fig. 9.12 where the supervisor neurons from the first version are removed, and a WTA mechanism is added to the vertical and horizontal line detectors to obtain the exact position of the strongest line detected in the visual scene (as we wish to detect only the coastline). We obtain a network of 2n neurons and 2n 2 -n synaptic connections, of which n 2 are excitatory and n 2 -n inhibitory (WTA connections). We produced some initial promising results on static and moving lines, randomly oriented and positioned in the scene and with or without noise. The corresponding plots can be found in Sec. E.

Diagonal lines

The networks described above are light and perform well, but do not allow for the detection of diagonal lines, thus are not applicable for coastline detection. We propose here a second architecture extending the first one to detect diagonal lines.

As seen in Fig. 9.13a, instead of two detectors specialised either for vertical or horizontal lines, we propose four detectors, each detecting lines on half of the sensor: top detector for the upper part, bottom detector for the lower part, left detector for the left part and right detector for the right part. Each neuron of each part of the sensor is connected to at least one neuron of the corresponding detector. One example of the pattern of connections is given for the lower part, for one neuron of the bottom detector, in Fig. 9.13b: the neurons on a line starting from the border neuron at the same index as the detector neuron, and expanding according to a varying degree alpha so that a majority of line orientations is covered, are excitedly connected to the corresponding neuron in the detector.

This network theoretically allows for the detection of any line in the sensor, whether it is diagonal, vertical or horizontal (see examples given in Fig. 9.14), using only 4n neurons for an input sensor of dimension n × n neurons. Perspectives However different experiments demonstrated that the current pattern of excitatory connections is not sufficient to assure the activation of one detector by a line ending on the corresponding sensor border. Indeed, in the case of the connectivity presented in Fig. 9.13b, a horizontal line crossing the lower part of the sensor would activate the corresponding neuron in the bot- tom detector ; actually, as this pattern is similar to all neurons of the bottom detector, it would be entirely activated thus distorting the results.

We are currently working on implementing a lateral inhibition mechanism that would ensure that a detector neuron is activated if and only if all the neurons on the same line are activated.

3D reconstruction

As stated in the introduction of this chapter, the APROVIS3D project has a use case of 3D reconstruction of a visual scene from the stereo matching between the recording of two event cameras, using an SNN architecture. We investigated this question via the supervision of two internships:

-Ms Huiyu Han, a 5th-year engineering student in applied mathematics and modelling at the time, conducted a thorough state-of-the-art study (see Sec.

9.

3.1 Embedding the model on SpiNN-3 [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] originally reduced the size of their spiking stereo-matching model so that it can be embedded on the neuromorphic set-up they had available: indeed, to take as input the data produced by a sensor of size 128 × 128 with a disparity range of [0, 127], their network is composed of around 2 20 cells i.e. 3.17M neurons (see Eq. 4.5, where we do not consider the retina input as the corresponding neurons are not simulated stricto senso by the SpiNNaker boards), which requires 15 SpiNN-5 boards. The authors filter the input event and reduce the network size so that x max = y max = 106 and d max = 32. With those new parameters, the network contains only 315K neural micro-ensembles i.e. 945K neurons and can be run on the authors' locally available system of 6 interconnected SpiNN-5 boards.

In the framework of Huiyu's internship, our main goal was to further reduce this network and adapt it so that it can fit on a unique embedded SpiNN-3 board. In order to achieve this goal, our first approach was to calculate the maximum number of parameters such as x max = y max = d max + 1 and N n < 18K (see Sec. 2.2.2). However, with the set of parameters obtained (x max = y max = 22 and d max = 21) the board produces an unexpected resource error.

Indeed, this issue can be explained as follows: the SpiNNaker board places each population on a separate core, thus allowing for a maximum of 63 populations on SpiNN-3. The network structure originally implemented in [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] initialises one population per vertical column of blocker neurons and per vertical column of collector neurons ("columns" referring here to neurons sharing the same x coordinate). The number of populations can thus be calculated as twice the number of neural micro-ensembles n in each layer (see Eq. 4.3), with y max the number of neurons in each collector population and 2x max the number of neurons in each blocker population (since there are two blockers for each collector). With the current set of parameters, we have 552 populations that cannot be embedded on our SpiNN-3's 63 cores ; therefore we reduced our set of parameters to x max = y max = 7 and d max = 6 to implement the original network structure on SpiNN-3.

However, the setup described above is not optimal: we only place 7 neurons per population/core when each core could fit up to 256 neurons. We thus adjusted the network structure to place more neurons per population: our new structure initialises one population per horizontal rank of blocker neurons and per horizontal layer of collector neurons ("layer" referring here to neurons sharing the same y coordinate). The number of collector populations is now y max and the number of blocker populations 2y max , for a total of 3y max populations. Each population contains a number of neurons that equals the total number of neural micro-ensemble N nme described in Eq. 4.3. This new structure allows us to take input of size x max = y max = 21 and d max = 20 while maintaining the original pattern of excitatory and inhibitory connections.

A second approach to embed [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF]'s stereo-matching network on a SpiNN-3 was to spatially reduce the input data instead of cropping it so that it fits the parameters determined above.

Stereo-matching performance

We present below some initial results with both approaches on two datasets (two fans and pendulum) provided by [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] to reproduce their results and more innovatively on the DSEC dataset (see Sec. 3.3.4).

Performance on Dikov's datasets

The authors provide two datasets, presented in Section 4.3.2.2, and corresponding results that we wish to reproduce with our adaptations.

We present below a comparison between the results obtained with the original network as implemented in [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] and those obtained on EBRAINS after either downscaling the input data by 2 or on SpiNN-3 after downscaling and cropping the input data. The spatial downscaling is achieved using the event count method described in Sec. 6.2.2.1 using a factor 3. Table 9 (in Annexes) presents the performance of cooperative stereo-matching obtained on the "two fans" dataset, where the network activity is quasi-stationary in terms of disparities despite the high rotation speed of the blades thus produces good performance thanks to the reduced number of event. However, Table 10 (in Annexes) presents similar results obtained on the "pendulum" dataset, where the constant evolution of disparity produces a significant number of events thus asking for more resources (as proves the high accuracy obtained on EBRAINS compared to SpiNN-3).

Performance on DSEC

Finally, we adapted this cooperative stereo-matching network to a more complicated, realworld driving dataset: DSEC (see Sec. 3.3.4). As the spatial resolution is too important for event Manchester's 1 million core large scale SpiNNaker-1 system (available to us via EBRAINS), we spatially downscaled by 5 the first 300ms of one sample of this dataset. The initial disparity map we obtain is particularly sparse because the pixels spike less often than the network's simulation timesteps, thus not providing enough information at each timestep. To overcome this issue, each non-activated pixel is given the last computed disparity value at each timestep. Table 11 presents the disparity maps computed from ground truth and from the stereo-matching network results, with and without solving the non-activation issue mentioned above. We explain the surprisingly unsatisfying network results by the high speed of change of disparity at each pixel, compared to the dataset provided by the authors where each pixel's disparity changes much less often if at all. Additionally, we suspect that the parameters used in [START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] are not quite so generalisable and that a solution to this problem would be to find a method to adjust the parameters to this more complex dataset.

Live demonstrator

The main goal of Noah's tutorship was to take up the work done during Huiyu's internship and from there, create a live demonstrator interfacing the cooperative stereo-matching network implemented on SpiNN-3 with two locally available event cameras. A stereo event stream is recorded using two Prophesee GEN3.1 cameras, calibrated so that they record the same visual scene from two slightly different angles but synchronised in terms of focus, distortion, rotation, etc. If the calibration is not done correctly, the images acquired will not be aligned and the depth information will not be reliable.

The current implementation of sPyNNaker (see Sec. 2.2.2) provides different methods to retrieve records from an external camera, that is then stored in the chips' SDRAM to ensure that it is available for future access. However, sPyNNaker does not currently provide any out-of-the-box support for real-time interactions with simulations (such as interacting with an external device and receiving new data while a simulation is running), except the less-than-ideal solution of launching multiple successive runs and retrieving information in between. We overcome this problem using the external_devices module: it provides capabilities for real-time injection and retrieval of spikes during a running PyNN simulation, thus allowing for control over the simulation. This live input/output interface creates a database that is subsequently used by external receivers and/or senders to identify recently activated neurons. The overall behaviour adopted by this module for communication between a SpiNNaker board and an external device is presented in Fig. 9.15. Figure 9.15 -Communication between the SpiNNaker board and an external device as handled by the external_devices module, from [START_REF] Manchester | Simple data input and output with spinnaker -lab manual -github[END_REF].

The SpiNNaker board can thus receive event data in real-time but requires that the population of input neurons be implemented using the specific population type SpikeInjectors, which are sent using the send_spike() method (see Algo. .3 and .4 in Annexes). On the output side, the spikes can be extracted in real-time, at each timestep, using a "callback" function similar to a "listener" (see Algo. .5 in Annexes).

All in all, the module and population types described above could theoretically allow for the interfacing between our cooperative stereo-matching network on SpiNN-3 and our two event cameras to visualise the disparity map obtained from recorded data in real-time. For now, only a first step has been achieved: the real-time visualisation of the disparity map obtained from pre-recorded data, i.e. the "two fans" dataset (see Fig. G.7 in Annexes).

End-to-end neuromorphic lip reading

Human speech perception is intrinsically a multi-modal task since speech production requires the speaker to move the lips, producing visual cues in addition to auditory information. Lip reading consists in visually interpreting the movements of the lips to understand speech without the use of sound. The prospect of systems capable of reading lips is attractive for a variety of real-life applications, like assistive devices for persons with disabilities, security and surveillance applications, or automatic transcription of video content. It can either complement an audio-based speech recognition system or replace it when sound is not available. During this PhD, we supervised students in UCA master's degree in computer sciences on the subject of neuromorphic lip reading:

-M. Raphaël Pietrzak and Ms Juliette Sabatier (semester of Fall 2021) recorded and preprocessed a new event-based dataset for the classification of lip motions. plied to the I3S dataset as well as to the event-based dataset introduced in [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF]. We believe that the recording of an event-based lip-reading dataset and implementation of an end-to-end neuromorphic lip-reading system could directly benefit the development of portable assistive devices or surveillance equipment. As of now, SNNs are yet to be applied to automatic lip reading despite their attractiveness, supported by the arguments listed above. This innovative work is therefore the first to exploit its advantages for the specific task of lip-reading on event data.

i3S dataset

The goal of Raphaël and Juliette's internship initially was to develop and record an eventbased dataset addressing a specific task that could not be achieved by solely considering events accumulated into a frame. Indeed, sometimes the simple observation of such a frame easily allows any ANN to determine the class without having to take into account the sequence of the frames (see example presented by Fig. 9.16). All in all, our motivation to propose such an internship was to reinforce the significant interest in using SNNs to process event data due to the temporal aspect of their learning.

We decided to focus on a dataset allowing for the classification of precise movements, where the time value of event videos would be crucial and properly exploited. We finally settled on registering lip movements to perform a potential reading on the lips, as no similar dataset existed at that time (DVS-Lip was later published in June 2022, see Sec. 3.3.5). Compared to other visual tasks, automatic lip reading deals with especially subtle movements with very precise timing and is very sensitive to noise and other environmental factors. A model for automatic lip reading thus needs to excel at extracting both spatial and temporal information.

All in all, the dataset we recorded, which will be referred to as the "i3S dataset" * in the remainder, contains 45 words recorded from 7 participants with 3 repetitions of each word per participant, for a total of 945 samples representing 11 GB of data.

Desired features

The i3S dataset focuses only on lip movements and has a wide variety of words/sounds in addition to profiles whose characteristics differentiate them from each other.

We identified different features necessary to obtain a complete and useful dataset:

-Similar recording conditions: it is necessary that the conditions (position of the face, lighting, setting, etc) be identical so that learning is not distorted by unintended elements such as noise or an element of a class that stands out from the others due to an external factor. -Subjects' diversity: as this dataset records people, it is necessary to have diversity in the profiles in order not to introduce bias during learning. Recorded subjects were selected according to diversity in the following criteria: face shape, gender, and facial hair (hairless, beard, moustache, etc). We deemed the last criterion particularly important as it can hide part of the mouth and change the appearance of the movement to the camera. -High number and variety of classes: the more different scenarios and repetitions of the same scenario a dataset contains, the more complete and precise it will be in its learning. The i3S dataset is thus composed of 45 words with 2 to 4 syllables, each containing at least one sound associated with each element of the French phonetic alphabet. The complete list of words can be found in Sec. H.1 in Annexes. -Ease of use of the format: the format in which the data is saved should correspond to event data while being as easy as possible to use.

Figure 9.16 -Accumulated events from a DVS 128 Gesture sample. Here, one can easily guess the DVS 128 Gesture class presented: the polarity (green dots for positive events, blue for negative events) hints at movement going from right to left, and the lesser number of negative pixels at the bottom of the forearm suggests a rotational movement.

Dataset acquisition

The data was recorded using the EVK1 -Gen. 3.1 VGA camera from Prophesee, provided with the corresponding software Metavision and an extensive API to process the data. The event data is saved in raw format with "EVT2" encoding.

Only the bottom of the subjects' faces is recorded to remove any irrelevant disruptive information and allow for the anonymisation of people who have agreed to be registered for the dataset. To ensure the legality of the dataset, UCA's legal department assures that the GDPR does not apply given the tight framing on the lower part of the face and the retention of no personal data, as long as the participants sign a short legal form.

Each participant was asked to say each word three times to get different lip movements on the same word. The camera was placed at the level of their mouths and, using our instructions and the feedback they could see on our computer screen, they positioned themselves so that the bottom of their nose was visible at the top of the event video, about 10cm away from the lens. To stabilise themselves, the participants put both hands on the table. The lip movements for one word were performed 1s after the beginning of the recording, and an additional 1s was added before its stop.

Recorded data postprocessing

The camera parameters used to record the data were not optimal and did not totally prevent the formation of noise in the events. To solve this issue, the data was subsequently converted into Numpy arrays and then processed using the denoising function provided by the Tonic library [START_REF] Lenz | Tonic: event-based datasets and transformations[END_REF].

The operation of this function is simple: if an event occurs but no other event is detected within a pixel radius for a period of time given in the parameter, it is removed. Thus, with the right time parameter, the isolated events will be deleted and those participating in the movement kept.

Once processed, each sample is saved into a file tree structure adapted to the classification of spoken words: each repertory corresponds to one class, i.e. one word. Each class is subdivided into subclasses under the denomination "userX_Y", with X the subject number and Y their physical particularity (beard, moustache, etc). Each subclass contains the corresponding RAW, BIAS and NPY files, named "recording_Z.format" with Z the recording order. One example of such a tree file can be found in Sec. H.2 in Annexes.

Neuromorphic lip reading model

Under our supervision during the semester of Fall 2022, the intern Hugo Bulzomi proposed a neuromorphic model for lip reading taking as input the data produced by an event-based sensor capturing lip motion. The model classifies short sequences of events into word categories using an SNN architecture, thus leveraging the advantages of neuromorphic computing, including energy efficiency and low latency, making it suitable for real-time embedded scenarios. His work draws inspiration from Tan et al. [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF], who achieved state-of-the-art results with an original DNN architecture for an automatic lip reading task on event data (see Sec. 3.3.5). Additionally, the model takes as input the DVS-Lip dataset Tan et al. introduced in (Tan et al., 2022). To the best of our knowledge, the model described below is the first proposal of an end-to-end neuromorphic lip reading model.

Event data preprocessing

As we opted to use surrogate gradient descent to train this lip reading model, it became necessary to convert our data to a synchronous form. Finding efficient ways to represent event data is a difficult problem subject of many prior studies. Since we target the DVS-Lip dataset, we chose to use a method similar to the one described by [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF]. In their study, the authors converted the asynchronous events into a 3-dimensional array, i.e. a voxel grid. Eq. 9.4 and 9.5 from [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF] show how we can create a voxel grid of a specific length T where the polarity of each event is spread through the two closest spatiotemporal voxels:

t * k = T -1 t N -t 1 (t k -t 1 ) (9.4) V (t, y, x) = k p k max(0, 1 -|t -t * k |) (9.5)
where T is the number of frames we want to use (i.e. the time resolution of our grid), and t x is the timestamp of the x th event from the original video. The event data are discretised accordingly into a 3-dimensional grid of shapes (t, x, y). [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF] uses two different values for T : 30 for the low-rate branch of their network and 210 for their high-rate branch. However, the individual performances of each branch when not combined are very similar (accuracy of 69.57% and 69.49% respectively). Only when both are combined in a multi-grained network along with message flow module blocks does the overall accuracy increase to 72.10%. After some trials, we manually set T = 30 since using a higher value would tremendously slow the training processes for little improvement, and a lower value leads to an accuracy collapse.

Topology exploration

To the best of our knowledge, no prior work uses SNNs to classify dynamic scenes for lip reading. Moreover, the literature on dynamic classification with SNNs is scarce. The closest studies to our work used the DVS-Gesture dataset introduced by (Amir et al., 2017) along with reasonably simple topologies and different variations, like in [START_REF] Yao | Temporal-wise attention spiking neural networks for event streams classification[END_REF]. The DVS-Gesture dataset itself is comparatively easy to classify, we hope to contribute to the area of neuromorphic computer vision by showing that more difficult problems can be tackled with SNNs.

We tested several topologies of different levels of complexity. Fig. 9.17 presents two simple SNN topologies that we used ; we will refer to those models as SNN1 and SNN2. SNN1 takes inspiration from [START_REF] Yao | Temporal-wise attention spiking neural networks for event streams classification[END_REF], who originally designed this topology to classify DVS-Gesture ; while SNN2 takes inspiration from (L. [START_REF] Zhu | Event-based video reconstruction via potential-assisted spiking neural network[END_REF], where the authors use a spiking encoderdecoder architecture for event-video reconstruction. SNN2 is the encoder part of their model, where the residual layers have been replaced by two more convolution layers, yielding a higher accuracy. Batch normalisation layers have been added after each convolution since such layers have been shown to considerably facilitate the learning process [START_REF] Cordone | Object detection with spiking neural networks on automotive event data[END_REF]. Along with this simple architecture, we designed a spiking equivalent of the low-rate branch of MSTP. AS MSTP uses ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] as the backbone, we first implemented a spiking ResNet backbone (similarly to (Fang, Yu, Chen, Huang, et al., 2021)) and then applied the MSTP architecture. The topology of our spiking MSTP low-rate branch is presented in Fig. 9.18.

One key difference between our spiking adaptation of MSTP and the original model lies in the GRU layer employed near the architecture's output. RNNs are generally employed when informa-tion has to be extracted from temporal sequences. Their output depends on both their current input as well as their hidden state. More particularly, GRU allows each neuron to learn how much of the previous information needs to be forgotten and how much of the current new input should be memorised [START_REF] Chung | Empirical evaluation of gated recurrent neural networks on sequence modeling[END_REF]. Though a spiking LSTM layer (adaptable into a spiking GRU) has been proposed in [START_REF] Lotfi Rezaabad | Long short-term memory spiking networks and their applications[END_REF], we found it to give very underwhelming results in our case: our network exhibited a very slow and inefficient learning process, especially in the three original layers of bidirectional GRU used in MSTP. We ultimately decided to try to find alternatives for extracting temporal information to replace this GRU layer. Table 9.3 presents the parameters used to implement the various SNN lip reading models using the PLIF neuronal model (Fang, Yu, Chen, Masquelier, et al., 2021).

Parameters

Values Initial τ 2.0 Activation threshold 1.0V

Reset potential 0.0V

Experimental validation

Experimental results demonstrate the effectiveness of the proposed model in lip reading, achieving high accuracy in word classification.

Experimental setup

Experiments with SNNs were performed on the DVS-Lip dataset using 30 timesteps (T = 30 in Eq. 9.4) to help us identify the most promising spiking topology before comparing MSTP to this optimised SNN. The different models were implemented using the SpikingJelly library and the Adam optimiser with a learning rate of 1e -3 , and a cosine annealing scheduler to adjust the learning rate during training (similarly to [START_REF] Cordone | Object detection with spiking neural networks on automotive event data[END_REF]. All experiments were performed on a laptop with an Intel Core i9-12950HX CPU (2.5 GHz x 16), 62,5 GB RAM, with an NVIDIA RTX A5500 laptop GPU with 16 GB of VRAM.

Identification of the best topology

This paragraph presents the classification accuracy of the DVS-Lip dataset obtained with the three SNN models described above.

The choice of the surrogate function of the gradient descent can vastly influence how the networks perform. We tested three of the main existing surrogate functions: piecewise quadratic, ATan, and the Gaussian error surrogate function (Erf). Table 9.4 shows preliminary classification results on a subset of DVS-Lip, using the three functions mentioned above. As Erf's performance is slightly better than ATan and Piecewise's, we use this function in subsequent experiments.

Tab. 9.5 presents the performance on the whole DVS-Lip dataset of the three architectures described above. This second batch of experiments shows that the spiking MSTP obtains significantly better results than the other models (SNN1 and SNN2).

Ablation study and comparison to state-of-the-art

Up until this point, our Spiking MSTP did not use any recurrent layer and the original GRU was simply replaced by a spiking Fully Connected layer. Table 9.6 presents the results of different experiments with possible GRU replacement in order to identify whether we could reach a higher performance thanks to other temporal information extraction methods. The results, including the accuracy of the original MSTP published in [START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF], show that significant performance growth can be gained by using stateful synapses either as a spiking replacement for the GRU layer.

According to Table 9.6, the existing MSTP still outperforms our best SNN model. However, we observe in Table 9.7 that our model reaches 83% of the existing MSTP accuracy but requires an amount of memory 5 times smaller than the one used by the MSTP model. Furthermore, the main topological difference between our model and the low-rate branch of MSTP is the absence of the three layers of bidirectional GRU. If removed, the low-rate branch of the existing MSTP performs slightly worse than our Spiking MSTP with similar memory size. All in all, our model seems to be a promising option for embedded lip reading: thanks to its spiking nature, its lower accuracy makes a good trade-off for a more energy-efficient model.

Limitations and perspectives

Our lip reading model is amongst the first advanced deep spiking models applied to such a challenging task and manages to get promising results while keeping a relatively small size. Although the accuracy of the proposed model is lower than the current state-of-the-art, we demonstrate that SNNs can be used for complex video classification tasks since lip reading remains 9.4 -9.4.4 Further experiments to improve the data pre-processing and the SNN model itself may, in the near future, allow a state-of-the-art model to be achieved. In the absence of published deep and complex SNNs for similar tasks, we hope to provide a spiking baseline for future work in this area.

We believe that our model can still be improved, especially regarding the replacement of the GRU layer ; even though our stateful synapses allowed us to break the 60% accuracy line, we believe it to be the cause of the performance gap between MSTP and this model and can be bridged by identifying more efficient ways to extract information from the temporal component of the data. Furthermore, even though we believe surrogate gradient descent is the current best training method for supervised learning with SNNs, it brings many of the regular ANNs' limitations to SNNs while performing a lot of approximations during training. We thus hope for the development of other training methods in the future, allowing us to tap into the full potential of SNNs.

Finally, we wish to compare the results described above, obtained on the DVS-Lip dataset, to new experiments conducted on the cleaned-up i3S dataset.

Conclusion

This chapter thus concludes the Contributions part of this manuscript. We detailed the different contributions we initiated and the related perspectives:

-the implementation of three approaches to tackle APROVIS3D's task of coastline detection to allow the autonomous navigation of a drone ; -the extension of existing work to perform embedded 3D reconstruction using a stereo event stream ; -the development of a model of end-to-end neuromorphic lip-reading with the associated recording of an event-based dataset. Each of these axes of research has potential follow-ups, described in detail in this chapter. These include the development of a comprehensive dataset for coastline detection, the implementation of a demonstrator for embedded and self-sufficient 3D reconstruction, the development and extensive experimental validation of a model of weight and delay learning and the implementation of SpiNN-3 of an existing unsupervised delay learning model.

CHAPTER 10

Conclusion

As demonstrated multiple times in this manuscript, the embedded combined use of SNNs and event cameras is significantly attractive regarding biological inspiration, reduced energy consumption, decision latency, and memory use. It is gaining momentum in the field of embedded computer vision. However, their novelty leaves room for many improvements in terms of optimal preprocessing of data as well as how this data is processed, making the most of the singularity of these scientific concepts.

Summary of contributions

Within this PhD, we identified several challenges related to this broad research topic, which we tackled through two main questions:

1. How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for embedded data processing ?

2. How can we exploit the relevance of the SNN to process the specific chronology of event data in an embedded context ?

We answered the first axis of research with three potential solutions: event data could either 1) be spatially or temporally reduced, either online or offline ; 2) keep only salient elements and discard the rest ; 3) be foveated, as a bio-plausible compromise between the two previous solutions. We compared qualitatively and quantitatively the event data obtained after each preprocessing method to assess whether the trade-off between the amount of data (i.e. events) kept versus the relevance of information kept is optimal. We also evaluated the computation time of each method, as their ultimate goal is to be embedded and able to run online.

The first solution, i.e. event data reduction, was implemented and experimentally assessed as ten different downscaling methods, of which six influence the spatial resolution and four the temporal resolution. We identified preferable methods for runtime, post-processing classification accuracy and the number of events: SNN pooling for spatial downscaling and stochastic structural for temporal downscaling. However, we highlighted the potential limitations of event data downscaling. There is a high variation in optimising the trade-off mentioned earlier, which strongly depends on the chosen method, dividing factor and application task. Additionally, we provided insights into the human perception of gesture classification using event data, thanks to an original human performance assessment experiment.

As a second solution to this first axis of research, we proposed the detection and selection of salient elements in the event scene thanks to a bio-inspired visual attention mechanism. We introduced a neuromorphic model for visual attention on event data, with multiple variations enabling the selection of one or multiple OoIs in the visual scene. Based on event density, this adaptative model relies solely on intrinsic SNN dynamics and requires no training, merely a fine-tuning phase for a new type of data. It was designed on a CPU and deployed on the neuromorphic platforms Loihi, SpiNNaker and Loihi. Using this model, we managed to reduce the spatial repartition of events to 20% of the original one while maintaining a classification performance at 70% of the original one, with a detection latency under 15ms. However, the simultaneous salient detection is only effective at most 50% of the time ; this is not efficient enough for an embedded model, whose real-life applications favour a significantly lower margin of error.

This leads us to our third and last solution: neuromorphic foveation, which corresponds to a feedback loop between a foveated event camera and a neuromorphic saliency detector, merging events at multiple resolutions according to the salient regions detected in the visual scene. This solution is ultimately to be the subject of a hardware implementation ; however, in the context of this PhD, we have implemented a first software version to begin the experimental validation of such a concept. We have designed a "binary" version of this concept, akin to the combination of the events of each sample in high resolution and low resolution using a mask. Using the binary implementation, we applied neuromorphic foveation to two use cases, gesture classification and semantic segmentation of a driving scene. We demonstrated the significant benefits of this method for optimising the trade-off between event quantity and quality. Indeed, in the embedded scenario, we managed to keep the same performance as with the original dataset on average while decreasing the mean number of events by two-thirds to reach the low-resolution value. Additionally, we have experimentally shown that most events kept in the foveated dataset provide relevant information while a significant part of the events in the original dataset is not as valuable but still provides interesting contextual information specifically relevant for semantic segmentation.

Overall, the results obtained during this PhD demonstrate that neuromorphic foveation leads to a good trade-off between information quantity and quality, at least on a software level.

We have undertaken several lines of research to address our second problem introduced earlier while remaining within the framework of the APROVIS3D project. To exploit the relevance of SNNs to process the particular temporal aspect of event data, we identified three embedded application tasks relevant to this issue and to APROVIS3D: coastline detection, 3D reconstruction and neuromorphic lip reading.

We have laid the foundations for three possible approaches to the first task: ANN-to-SNN conversion, adaptative line detection and delay learning. While the two initial approaches fail to exploit the event data temporality and the task could have been solved using frames instead of single events, the last one is auspicious by its bio-plausibility. Using delay learning, the events are segmented not according to the global object they form but thanks to their specific spatiotemporal patterns.

The second task calls for the implementation of bio-inspired stereo-matching models. We implemented the first steps towards a live demonstrator for neuromorphic 3D reconstruction based on existing work adapted to run online on a SpiNNaker board. Temporal timing of events and spikes also intervene in this model: each network cell encodes a belief in matching a heterolateral pair of pixels, relying on the temporal timing of the stereo activation to prevent homolateral matching.

The third task, neuromorphic lip reading, falls further away from the APROVIS3D spectrum but highly benefits from the SNN's intrinsic temporal processing and event data's crucial chronology. We implemented an SNN architecture requiring event data as input -as a similar task could not easily be solved using frames instead of individual events -to perform lip reading. It 10.2 -Perspectives 191 leverages the advantages of neuromorphic computing, including energy efficiency and low latency, which makes it suitable for real-time embedded scenarios and produces promising results while keeping a relatively small size.

Perspectives

The results obtained during this PhD open the door to a multitude of potential developments, aiming to optimise embedded neuromorphic computer vision in the longer or shorter term.

Short term perspectives

Event data downscaling We believe that the subject of event data downscaling is quite promising. Indeed, other researchers started tackling this issue as well following the publication of our works [START_REF] Rizzo | Neuromorphic downsampling of event-based camera output[END_REF]. On our side, we believe we are still missing a final assessment: the complete evaluation of real-time data downscaling with the newer temporal methods. The diverse combinations of spatial and temporal methods could also bring a new dimension to the research field of event data downscaling.

Additionally, we believe that it would be highly beneficial for the scientific community to gain easier access to the methods we implemented and validated. They are currently available online on a GitHub repository ; one way to make them more accessible overall would be to integrate them into the Tonic library [START_REF] Lenz | Tonic: event-based datasets and transformations[END_REF], which positively encourages collaborative involvement and outsider contributions. This library already proposes an alternative version of the temporal and spatial funnelling methods (with no removal of duplicates) ; it could only benefit from the additional implementation of the eight methods we introduced.

Simultaneous saliency detection of multiple OoIs

We believe some SNN architectures implemented during this PhD can be improved. The simultaneous saliency detection model for multiple OoIs requires further parameter tuning to increase the multi-OoIs detection performance beyond the current maximum of 50%. Additionally, this model was solely assessed for the simultaneous detection of two OoIs while it could theoretically reach a greater number ; this should be the subject of further experiments on more complex datasets and scenarios with a greater number of salient objects. Finally, it should be applied to more complex scenarios (such as driving scenes) to assess its viability on an embedded system.

A second possible improvement is to update this model to enable an adaptative number of OoIs to be detected. In this ideal model, the number of output layers could be adapted to the number of OoIs present in the visual scene, i.e. the number of regions identified by the saliency detector, via a feedback regulation: for example, the activity of the n th output layer would give feedback on the existence of a n th OoI in the current scene, allowing the network to add or remove output layers appropriately. Such an implementation would require neurogenesis and synaptogenesis (i.e. dynamic instantiation of neurons and synapses), which is currently unavailable on most SNN simulators and neuromorphic hardware ; however, we could mimic this bio-inspired behaviour by initiating multiple output layers and inhibiting them according to their use. Additionally, the framework behind Kraken (Di [START_REF] Mauro | Kraken: A direct event/frame-based multi-sensor fusion soc for ultra-efficient visual processing in nano-uavs[END_REF] may allow such a behaviour to be implemented, even in an embedded scenario. This possibility is still to be further confirmed with experimental trials.

The scientific community is producing more and more results demonstrating the benefits of deploying SNN architectures on hardware. This will both facilitate on-board implementation and interfacing with event cameras as well as demonstrate the advantage of these neural networks over standard ANNs, which are still widely favoured in computer vision nowadays. Therefore, we wish to validate our SNN architectures, particularly our visual attention model variations, on neuromorphic hardware -further than the work begun in (Bulzomi, Gruel, et al., 2023). These models could be deployed on SpiNNaker (S. B. [START_REF] Furber | Overview of the spinnaker system architecture[END_REF], which has the advantages over Loihi of being academic, thus more accessible for students, and able to implement dynamic adaptation of parameters mid-run.

Delay learning In this thesis, we proposed 1) an extension of the delay learning model introduced by [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF] and 2) the first tentative steps towards a learning rule for simultaneous weight and delay learning inspired by the traditional STDP learning rule. Both those works are ongoing and are to be improved in order to promote delay learning, a mechanism we are convinced could be a significant contribution to the neuromorphic community.

Among others, the demonstration of SpiNNaker's capacity to simulate such a model would further support our argument ; as well as its application to a real-world use case. Indeed, we aim to apply delay learning to the coastline detection task targeted by APROVIS3D ; their deployment on software would ease their embedded application.

Medium term perspectives

Neuromorphic foveation In this work, we assessed the neuromorphic foveation concept by implementing a binary version (with only two resolutions) on software. However, this work draws inspiration from a hardware concept introduced in (Serrano- Gotarredona et al., 2022), not yet available for experimental use due to the sanitary crisis. This foveated event camera we wish to emulate is more flexible than our binary implementation. A more thorough conceptualisation of the neuromorphic foveation concept is required to achieve a more realistic implementation. We could implement an adaptative number of resolutions, where the spatial dividing factor depends on the activity in the saliency detector (see Chapter 8): the higher the spike rate is, the more interesting the detected object is and the higher the resolution must be ; in a region of the sensor, the dividing factor would thus be inversely proportional to the corresponding spike rate. This nested foveation model would output data with several levels of finesse, starting from the lower resolution and then refining salient areas in overlapping sub-regions. Now that the software implementation has been validated, we wish to integrate it on the foveated event camera created by IMSE [START_REF] Serrano-Gotarredona | Electronically foveated dynamic vision sensor[END_REF]. Indeed, this hardware deployment would enable real-time neuromorphic foveation thanks to the feedback loop provided by the saliency model deployed on an interfaced SpiNN-3.

Coastline detection As part of the APROVIS3D work group dedicated to embedded coastline detection, we still have to deploy a complete live demonstrator of an embedded SNN model applied to a live event stream and able to perform online coastline detection. This task is to be achieved in collaboration with INT and NTUA.

We introduced in this thesis a tentative lightweight SNN that could similarly solve this task. We believe this line of research would gain from being pursued further. 10. 3 -10.2.3 Long term perspectives 193 10.2.3 Long term perspectives Saliency mechanism We implemented a bottom-up saliency mechanism applied to event data in this thesis. We arbitrarily defined salient elements as regions in the sensor where events take place close in time and space. However, there exist many different ways to define saliency in the literature, even limiting ourselves to bottom-up visual attention. We believe it would be highly beneficial for the neuromorphic community to further research ways to implement visual attention in event data using SNNs, since, as we demonstrated in this thesis, this is a promising challenge for limiting the processing of event data to interesting elements of the scene.

Concerning the simultaneous detection of multiple OoIs: our proposed model adds to the complexity of the network by implementing a new layer for each new OoI to be detected. This strays farther from biology, where the human eye detects and processes one OoI at a time, quickly alternating between salient elements using saccades. SNNs implementing this mechanism would allow for the successive detection and processing of each OoI in the scene ; once processed, the OoI is discarded using a temporal WTA mechanism, and the network outputs another OoI. Such a mechanism could easily handle a dynamic scene by successively processing OoIs as they come while being more bio-plausible. However, this research direction raises questions at a higher level: in the neuromorphic visual attention field of research, should we aim to stay close to biology and benefit from its efficiency ? Or take advantage of the computational power of neuromorphic hardware which allows, among other things, massively parallel processing ?

Next steps

Overall, we believe we provided results with a significant impact on embedded neuromorphic computing, especially concerning the optimisation of data preprocessing. Concrete proof of this can be found in the fact that the work carried out during this thesis on visual attention led to another PhD thesis, the one of Mr Hugo Bulzomi, with an application in autonomous driving. Indeed, Hugo will start a CIFRE PhD in Autumn 2023 in collaboration with i3S and IMRA Europe (a branch of Aisin) ; we co-authored the first work on his thesis subject, published in MVA 2023 (Bulzomi, Gruel, et al., 2023).

We hope to have made a valuable contribution to the exploitation of the unique advantages of combining two innovative concepts that are SNNs and event-based cameras. We believe in the usefulness of neuromorphic computing for the future of AI, thanks to its low memory and energy consumption as well as its high adaptability. This field of research benefits greatly from the interdisciplinary nature of the researchers involved, ranging from biology and computer science to microelectronics, robotics and mathematics. This field of research benefits greatly from the interdisciplinary nature of the researchers involved, ranging from biology and computer science to microelectronics, robotics and mathematics. As a result, scientific knowledge is being shared, leading to a boom in neuromorphic research.

B Visualisation of different spatial downscaling methods.

Tables 1 and2 present one frame extracted respectively from the DVS 128 Gesture (Amir et al., 2017) and N-MNIST [START_REF] Orchard | Converting static image datasets to spiking neuromorphic datasets using saccades[END_REF] datasets.

Three examples are presented for each method, one corresponding to a spatial downscaling by a factor of 2 (first column), another by a factor of 4 (second column) and the last by a factor of 10 (third column). The corresponding number of events produced by each method from this sample is indicated below each frame. 3: Visual representation of the events and the different segments identified by Ev-SegNet in the sample rec1487417411_export_1467 from the DDD17 dataset, after various processes. The subplot on the left corresponds to the original data and in the middleleft to the same event data spatially reduced by 4 using the event count method. The middle-right subplot corresponds to the events contained in the RoIs detected on the data reduced using the event count method, and the right one to the sample foveated according to those RoIs (identified by the red bounding boxes). On the left, each frame corresponds to the accumulation of the events over 50ms, the sample's time-window.

Green and blue pixels correspond respectively to positive and negative events. 3: Visual representation of the events and the different segments identified by Ev-SegNet in the sample rec1487417411_export_1467 from the DDD17 dataset, after various processes. The subplot on the left corresponds to the original data and in the middleleft to the same event data spatially reduced by 4 using the event count method. The middle-right subplot corresponds to the events contained in the RoIs detected on the data reduced using the event count method, and the right one to the sample foveated according to those RoIs (identified by the red bounding boxes). On the left, each frame corresponds to the accumulation of the events over 50ms, the sample's time-window.

Green and blue pixels correspond respectively to positive and negative events. (Amir et al., 2017), after various process. The subplot on the left corresponds to the original data and in the middleleft to the same event data spatially reduced by 4 using the event count method. The middle-right subplot corresponds to the events contained in the RoIs detected on the dataset reduced using the event count method, and the right one to those RoIs (identified by the red bounding boxes). Each frame corresponds to the accumulation of the events occurring during the sample's first 10 ms. Green and blue pixels correspond respectively to positive and negative events.

H i3S dataset

Figure 1

 1 Figure 1.1 -Evolution of ImageNet image classification performance over time.

  (a) A spiking neuron receives spike trains as input and processes this information to produce a new sequence of activations. (b) Evolution of the neuron's membrane potential over time when activated by input spikes.

Figure 2 .

 2 Figure 2.1 -Behaviour of a spiking neuron.

Figure 2 . 2 -

 22 Figure 2.2 -Biological neuron.

Figure 2 . 3 -

 23 Figure 2.3 -Comparison of the neurocomputational properties of spiking models, from (E. M. Iz-hikevich, 2004). The x-axis label "# of FLOPS" is an approximate number of floating point operations (addition, multiplication, etc.) needed to simulate the model during a 1 ms period.

Figure 2 . 4 -

 24 Figure 2.4 -Comparison between three SNN coding schemes that can operate in a short time window, from (S. Thorpe et al., 2001).

Figure 3 . 2 -

 32 Figure 3.2 -Block diagram of an event camera pixel.

Figure 3 . 3 -

 33 Figure 3.3 -Illustration of event-based encoding of visual signals with respect to the predefined threshold ∆I, from (Lagorce et al., 2015).

  segmentation Various backgrounds 87.4M B (BAG, NPZ, TXT) GEN1 (de Tournemire et al., 2020) Automotive, pedestrian detection Cars, pedestrians unknown (DAT) HQF (Stoffregen et al., 2020) Acquisition of quality ground truth Camera motions, indoor and outdoor scenes 2.7GB (BAG) IROS2018 (Mitrokhin et al., 2018) Moving object detection and tracking Objects with varying degree of occlusions and brightness 200M B (TXT, PNG, BAG) . Z. Zhu, Thakur, et al., 2018) Optic flow and 3D reconstruction Stereo sequences of indoor and outdoor scenes, recorded from a driving car, a hexacopter, a motorcycle or a human hand 74.7GB (rosbag, HDF5)

  Fang et al. introduced during ICCV 2021 an SNN classifier using a new spiking neuron model called PLIF

Figure 4 .

 4 Figure 4.1 -Network topology of the cooperative stereo-matching network introduced in (Dikov et al., 2017).Figure from (Dikov et al., 2017). (a),(b) Connectivity in terms of excitatory and inhibitory connections: neurons/cells (dots) are framed into populations and interconnected by synaptic projections (lines). (c),(d) Inhibition and excitation pattern for an exemplary pair of retinal input spikes (c) leading to a match, i.e. spike, in the cooperative network (d) which in turn triggers internal inhibition and excitation according to Marr's physical constraints.

Figure 4 . 2 -

 42 Figure 4.2 -Topology and working principle of neural micro-ensembles to prevent homolateral self-matching, from[START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF].

  5.1 -5.1.1 Biological visual attention 65 the primary visual cortex (V1) ; it then journeys through the extrastriate areas V2, V3, V4 and V5 (see Fig. 5.1).

  5.1 -5.1.2 Neuromorphic visual attention 67 thus affecting the FEF and selective attention. A link has been established between this molecule and Attention Deficit Hyperactivity Disorder.

Figure 5 . 2 -

 52 Figure 5.2 -Biological foveation mechanism, adapted from[START_REF] Bear | The Human Eye[END_REF].

Figure 5 . 3 -

 53 Figure 5.3 -Visual representation of retinotopic foveation, from[START_REF] Perrinet | Implementing a retinotopic transform using 'grid_sample' from pytorch[END_REF]. "A picture (extract from the painting "The Ambassadors" by Hans Holbein the Younger) can be represented on a regular grid represented by vertical (red) and horizontal (blue) lines. Retinotopy transforms this grid, and in particular, the area representing the fovea (shaded gray) is over-represented. Applied to the original image of the portrait, the image is strongly distorted and represents more finally the parts under the axis of sight (here the mouth)."

Figure 5 . 4 -

 54 Figure 5.4 -Schematic showing the Jeffress model for sound localization as applied to chicken and barn owl, adapted from[START_REF] Kubke | Morphological variation in the nucleus laminaris of birds[END_REF]. Neurons located in the dorsal edge of the nucleus laminaris respond maximally to sounds originating from far contralateral space, and neurons located in more ventral positions respond maximally to sounds originating from the front.

Figure 5 . 6 -

 56 Figure 5.6 -Biological mechanism at the origin of delay learning via myelination.

Figure 5 . 7 -

 57 Figure 5.7 -Four samples of the dataset employed in (Nadafian & Ganjtabesh, 2020). The intensity of each dot shows the location of it over time (black indicates the most recent location).

Figure 5 . 8 -

 58 Figure 5.8 -Weights and delays of each convolution at different training epochs, from (Nadafian & Ganjtabesh, 2020). The size (intensity) of each circle represents the delay (weight) of its corresponding synapse, where the bigger circle illustrates lower synaptic delay (black indicates the higher synaptic weight).

Figure 6 . 3 -

 63 Figure 6.3 -Schematic illustration for linear log-luminance reconstruction. A sensor of size 2 × 2 outputs negative and positive events at each pixel across time (crosses in blue, yellow, red and green, each colour corresponding to one pixel), which are downscaled using log-luminance reconstruction with linear interpolation. The black line corresponds to the downscaled log-luminance values, which are computed as the average of the log-luminance values recorded by each of the 4 original pixels. In this example, the downscaling factor is set to 2.

Figure 6 . 4 -

 64 Figure 6.4 -Illustration of the constant strategy applied to linear log-luminance reconstruction. The input events and downscaling factors are similar to those used in Fig. 6.3.

Figure 6 . 5 -

 65 Figure 6.5 -Illustration of the interpolation strategy applied to linear log-luminance reconstruction. The input events and downscaling factors are similar to those used in Fig. 6.3.

Figure 6 . 6 -

 66 Figure 6.6 -Schematic illustration for linear log-luminance reconstruction. A sensor of size 2 × 2 outputs negative and positive at each pixel across time, which are downscaled using log-luminance reconstruction with cubic interpolation. In this example, the downscaling factor is set to 2.

Figure 6 . 7 -

 67 Figure 6.7 -Illustration of the constant strategy applied to cubic log-luminance reconstruction. The input events and downscaling factors are similar to those used in Fig. 6.6.

Figure 6 . 8 -

 68 Figure 6.8 -Illustration of the interpolation strategy applied to cubic log-luminance reconstruction. The input events and downscaling factors are similar to those used in Fig. 6.6.

Figure 6 .

 6 Figure 6.11 -Comparison of activity measures according to a varying synaptic weight ω for separate and mutual SNN downscaling methods applied to DVS128 Gesture.

Figure 6 .

 6 Figure 6.19 -Evolution of the accuracy performance of the PLIF-based classifier applied to DVS128 Gesture (first line) and N-MMNIST (second line). The dashed line separates each plotbetween what we aim for -a decreased number of events for a significantly good accuracy (upper left) -and what we do not aim for -a decreased accuracy for a great number of events (lower right). Globally most results are in the part that we aim for, with the notable exception of the "SNN pooling" results.

Figure 6 .

 6 Figure 6.22 -Interface presented to the participants during the human assessment experiment.

Figure 6

 6 Figure 6.23 -Participants performing the human assessment experiment.

Figure 6 .

 6 Figure 6.25 -Human accuracy, rate of "unknown" responses and human response time to classify event data downsized temporally. In the figure on the left, the dotted line at 25% of the accuracy corresponds to the chance level.

Figure 6 .

 6 Figure 6.26 -Human accuracy according to the resulting number of events per sample after spatial downscaling methods. The vertical dashed line corresponds to the number of events per sample in the original dataset.

Figure 6 .

 6 Figure 6.27 -Human accuracy according to the resulting number of events after temporal downscaling methods. The vertical dashed line corresponds to the number of events per sample in the original dataset.

Figure 7 .

 7 Figure 7.1 -Visual example of the difference between an RoI (outlined in red) and an OoI (outlined in purple), on a sample extracted from the dataset DVS128 Gesture.

Figure 7 . 2 -

 72 Figure 7.2 -SNN model used to detect saliency by event density.A) The blue and green dots and shapes in the input layer corresponds to input events: the dots represent noisy or non-saillant events and the "cloud" shapes illustrate events aggregated into saillant objects (OoIs). B) Precise depiction of the activating convolution between the Input and the Saliency detector layers as well as the soft exponential WTA on the Saliency detector. In this example, the sensor is of size 9 × 9 pixels and the convolution has a kernel size of S × S = 3 × 3. Each neuron is depicted as a circle, and the increased size in the Saliency detector layer simply allows better visualisation of the topology. The neurons are spatially positioned as a 2D rectangle. The red connections depict the lateral inhibition implemented by the soft exponential WTA, and the continuity of the lines corresponds to the intensity of the inhibition: neurons far from the activated one are highly inhibited (continuous lines), while those close are softly inhibited (dashed lines).

Figure 7 . 3 -

 73 Figure 7.3 -Bounded exponential WTA used as a radial basis function in the attention model.

Figure 7 . 4 -

 74 Figure 7.4 -Visual illustration of the behaviour of the weight adaptation rule.

Figure 7 . 5 -

 75 Figure 7.5 -Architecture of the spatiotemporal attention model filtering one OoI at a time.

Figure 7 . 6 -

 76 Figure 7.6 -Visual illustration of the behaviour of the threshold adaptation rule implemented in the saliency detection of one output.

Figure 7 . 7 -

 77 Figure 7.7 -Architecture of the saliency detection model adapted to SpiNNaker.

Figure 7 . 8 -

 78 Figure 7.8 -Architecture of the multi spatiotemporal attention. Here, the maximum number of OoIs (i.e. the number of output layers) has been set to 2 for clarity.

Figure 7 . 9 -

 79 Figure 7.9 -Visual illustration of the behaviour of the threshold adaptation rule implemented in the saliency detection of multiple outputs, according to the activity and the firing time of the saliency detector and lateral layers.

Figure 7 .

 7 Figure 7.10 -Samples extracted from the Prophesee Pedestrian dataset, where the events are accumulated over time. Yellow squares indicate pedestrians' bounding boxes.

Figure 7 .

 7 Figure 7.11 -Evolution of quantitative and qualitative properties of the DVS 128 Gesture dataset after attentional filtering, according to the dataset's various targets i.e. various labelled gestures. The values in red indicate the average value over all the targets for each property.

Figure 7 .

 7 Figure 7.14 -Simulation on PyNN of the attention model applied to two samples with a time shift of 100 ms. The Input, RoI.d and Output plots in the 3 first rows represent the neurons emitting a spike at time t (x axis) respectively in the input, saliency detector and output layers respectively. The Weight and Threshold Rules correspond to the evolution of the weights and thresholds in the output layer.

Fig. 7 .

 7 Fig.7.14 presents the results of PyNN simulations on two samples from the three-gesture dataset. The left figure presents the selection of one OoI in a sample displaying the gestures "right hand clockwise" (on the left) and "arm roll" (on the right), each gesture lasting 200ms ; while the right figure presents the simulation results on the same two gestures spatially inverted ("arm roll" on the left and "right hand clockwise" on the right). The first gesture to appear on the sensor is the one on the left in both cases. Such a permutation was tested in order to confirm that the saliency is not dictated by the spatial location of the OoI, but indeed by its timing and size (as hypothesised in the introduction of this chapter). The gesture on the left starts first, and overlaps over 100ms with the second one. One can clearly see that:-the saliency detection (second plot from the top, identified by the label "ROI.d") clearly indicates the spatiotemporal RoIs ; -the output layer only s through the events corresponding to the OoI detected first by the saliency detection i.e. the first gesture performed, until the end of the corresponding 200ms ; -the evolution of the weights of the input to saliency detector connections follows Eq. 7.2, increasing at ω max = 2 when the saliency detector activates or decreasing to omega reset = 0.7 otherwise. -the evolution of the output thresholds follows Eq. 7.3, decreasing at θ reset = -65mV when the saliency detector activates or decreasing to θ max = 0mV in absence of activity.

Figure 7 .

 7 Figure 7.15 -Size invariance of the attention model with selection of one OoI, run on Loihi. Error variation of different combinations between two "small" gestures (classes 1 and 2, x-axis) and one "large" gesture (class 7, y-axis left side), according to different time shifts (y-axis right size).Irrespective of the gesture size, the error decreases as the time shift increases.

Figure 7 .

 7 Figure 7.16 -Spatial invariance of the attention model with selection of one OoI run on Loihi, and evolution of the error as the time shift increases, the larger time shit, the better filtering from the attention model. Right: the RMSE behaves similarly when the first gesture (i.e. the detected OoI) is on the right or on the left. This confirms that the spatial location of the OoI does not influence its detection. Left: Average RMSE overall: the larger the time shit, the better the selection from the attention model.

  Fig. A.2 in Annexes allows for a visual assessment of the quality of the OoI selection on reduced input event data, whether the dataset used is "control" (Fig. A.2a in Annexes) or "random symmetric" (Fig. A.2b in Annexes).

Figure 7 .TABLE 7 . 5 -

 775 Figure 7.18 -Mean precision-recall comparison in different noise ratio intervals.

  Finally, Fig. A.3 in Annexes allows for a visual assessment of the quality of the OoI selection on reduced input event data, whether the dataset used is "control" (Fig. A.3a in Annexes) or "random symmetric" (Fig. A.3b in Annexes).

7. 5

 5 Figure 7.19 -Performance of simultaneous multi-OoIs detection according to the parameters ω W T A , ∆θ and ∆t. The impact of ∆t variation was observed for the parametrisations A (ω W T A = 0.5, ∆θ = 14), B (ω W T A = 0.5, ∆θ = 13), C (ω W T A = 0.5, ∆θ = 12) and D (ω W T A = 0.5, ∆θ = 11). The plot on the right highlights the lack of influence of ∆t on the simultaneous multi-OoI detection.

Figure 7 .

 7 Figure 7.20 -Multi-OoIs detection performance (top), the latency of the OoIs detection (middle) and classification accuracy of the selected OoIs (bottom) of the multi-OoIs selection model on a "control" dataset. The performance presented in the top figure corresponds to the chance of our model detecting two different OoIs (and not twice the same) over the complete "control dataset" for each temporal shift. The orange line on the bottom figure indicates the classification accuracy of the whole dataset. Since we classify between 11 classes, the chance level is around 9%.

Figure 7 .

 7 Figure 7.21 -Latency of the OoI selection our saliency detection model for one OoI (in blue) and multiple OoIs (in red) according to the shift. With the exception of the dip observed for a 50µs shift, the overall latency of OoI selection increases with the number of OoIs to be detected (from one in blue to two in red).

(

  [START_REF] Renner | Event-based attention and tracking on neuromorphic hardware[END_REF]) (D'Angelo et al., 2022) Salient selection of one OoI Salient selection of multi-OoI

Figure 8 . 2 -

 82 Figure 8.2 -Binary foveation of events using the corresponding high and low resolution (spatially downscaled by a dividing f actor) and based on a known RoI.

Fig. 8 .

 8 Fig. 8.7 presents the spatial and temporal density according to the mean number of RoI detected per method to discuss the quantitative aspect of saliency detection.

Figure 8 . 6 -

 86 Figure 8.6 -Qualitative study of the saliency detection, based on the semantic segmentation performance.The plot depicts the average semantic segmentation performance according to the spatial downscaling method used to generate the data. The overlayed dots correspond to different reduction methods (with no indication of the corresponding methods) with which the multiresolution merging was achieved using the RoIs detected with one specific method. The goal is to minimise the mean number of events per sample while increasing the performance, thus best results are close to the subplot's upper left corner. Foveated data using RoIs detected on event count is the closest to this goal.

Figure 9 .

 9 Figure 9.1 -Example of accumulated synthetic events obtained from the recording of the Mediterranean coastline by an RGB camera onboard a drone.

Figure 9 . 2 -

 92 Figure 9.2 -Delay learning changes the delay of the electrical spike journey in the pre-synaptic neuron's axon so that it recognises specific patterns in the visual input data.

Figure 9 . 3 -

 93 Figure 9.3 -Overall movements of the synthetic events obtained from the recording of the Mediterranean coastline by an RGB camera embedded on a drone. The top of the sensor corresponds to the direction of the UAV.

Figure 9 . 4 -

 94 Figure 9.4 -Overall movements of the synthetic events obtained from the recording of the Mediterranean coastline by an RGB camera embedded on a drone. The top of the sensor corresponds to the direction of the UAV.

Figure 9 . 5 -

 95 Figure 9.5 -Features learned by the delay learning model implemented in (Nadafian & Ganjtabesh, 2020), without or with additional noise in the input data used during training. Similarly to Fig. 5.8, the size and intensity of each circle represent respectively the delay and weight of its corresponding synapse, where the bigger circle illustrates lower synaptic delay and the colour black indicates the higher synaptic weight. On features learned on non-noisy data (left), the diagonal A corresponds to a south-east motion, B to a north-east motion, C to a north-west motion and D to a south-west motion. Figure from (Bulzomi, 2022).

Figure 9 . 6 -

 96 Figure 9.6 -Evolution of the features learned over time by the delay learning model implemented in[START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF], trained on noiseless input data. The delays are displayed in blue and weights in orange. The figure on the left presents the evolution of the delays and weights of the kernel corresponding to the sensor pixel of coordinates (x = 0, y = 4), i.e. within the pattern ; while the figure on the right presents the evolution of the delays and weights of the kernel corresponding to the sensor pixel of coordinates (x = 0, y = 0), i.e. outside the pattern. Figure from[START_REF] Constantin | Synaptic delays for temporal pattern recognition[END_REF].

Figure 9 . 7 -

 97 Figure 9.7 -Features learned at t = 15, 000µs by the delay learning model implemented in[START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF], without any additional noise in the input data used during training and using two (top) or four (bottom) convolution layers. The delays and weights are displayed separately: the first line corresponds to the delays, the second to the weights and each column to one convolution layer. The pattern learned by each layer is identified above each column. Figure from[START_REF] Constantin | Synaptic delays for temporal pattern recognition[END_REF].

Figure 9 . 8 -

 98 Figure 9.8 -Issue in the identification of patterns based on delay learning.

Figure 9 . 9 -

 99 Figure9.9 -Metrics (described in Eq. 9.2) to assess the quality of the classification performed by the features learned by the delay learning model implemented in[START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF], without any additional noise in the input data used during training and using two (left) or four (right) convolution layers (corresponding to the features displayed in Fig.9.7). The precision is displayed in blue, the recall in orange and the F1-Score in green. Figure from[START_REF] Constantin | Synaptic delays for temporal pattern recognition[END_REF].

  ∆δ =-α sin( ∆t τ ) if∆t ∈ [-τ π; τ π] 0 otherwise with ∆t = ∆t pre -t post and ∆t pre = t spikeA -t spikeB = t activationA + δ A -t activationB -δ B(9.3) where δ is the delay of the last spiking neuron between the pre-synaptic neurons A and B, τ is the time constant of the learning rule and α is the learning rate. When τ increases, the time window of the learning rule increases and the learning is applied to neurons whose spikes are farther apart (and vice versa). When α increases, the value of the ∆δ extrema increases and the learning is steeper. The behaviour of ∆δ and ∆ω are illustrated in Fig.9.10.

Figure 9 .

 9 Figure 9.10 -Behaviour of the tentative model of weight and delay learning we propose in this work. The plot on the left presents the evolution of the delay according to Eq. 9.3, and the plot on the right the behaviour implemented by a traditional STDP.

Figure 9 .

 9 Figure 9.11 -SNN architecture for vertical and horizontal line detection, with no knowledge of the line position.

Figure 9 .

 9 Figure 9.12 -SNN architecture for vertical and horizontal line detection, with knowledge of the line position.

Figure 9 .

 9 Figure 9.13 -SNN overall architecture (left) and pattern of connections for one detector (left) for the detection of diagonal lines.

Figure 9 .

 9 Figure 9.14 -Three examples of line reconstruction from various detector's neurons activations. Two neurons are activated in two of the detectors, thus giving the coordinates of the intersection points of the detected lines with the edges of the sensor.

Figure 9 .

 9 Figure 9.17 -Architecture of two lip reading SNN models.

Figure 9 .

 9 Figure 9.18 -Architecture of the overall spiking low-rate branch MSTP topology along with the associated ResBlock.

Table 1 :Table 2 : 2 :C

 122 Frames produced from the sample user14_led_0.npz, corresponding to the gesture "left hand clockwise" (class 6) from the DVS128 Gesture dataset, processed with the spatial downscaling methods implemented during this PhD. Each frame corresponds to the accumulation of the events occurring during the sample's first 5 ms. Green and blue pixels correspond respectively to positive and negative events. Frames produced from the sample user14_led_0.npz, corresponding to the gesture "left hand clockwise" (class 6) from the DVS128 Gesture dataset, processed with the spatial downscaling methods implemented during this PhD. Each frame corresponds to the accumulation of the events occurring during the sample's first 5 ms. Green and blue pixels correspond respectively to positive and negative events. (Continued) Frames produced from the sample user14_led_0.npz, corresponding to the number "4" from the N-MNIST dataset, processed with the spatial downscaling methods implemented during this PhD. Each frame corresponds to the accumulation of the events occurring during the sample's first 5 ms. Green and blue pixels correspond respectively to positive and nega-Frames produced from the sample user14_led_0.npz, corresponding to the number "4" from the N-MNIST dataset, processed with the spatial downscaling methods implemented during this PhD. Each frame corresponds to the accumulation of the events occurring during the sample's first 5 ms. Green and blue pixels correspond respectively to positive and negative events. (Continued) Visualisation of the stakes of neuromorphic foveation Table

Figure C. 4 -

 4 Figure C.4 -Visual representation of the events in the sample user25_led, corresponding to the gesture "left hand clockwise" (class 6) from the DVS128 Gesture dataset(Amir et al., 2017), after various process. The subplot on the left corresponds to the original data and in the middleleft to the same event data spatially reduced by 4 using the event count method. The middle-right subplot corresponds to the events contained in the RoIs detected on the dataset reduced using the event count method, and the right one to those RoIs (identified by the red bounding boxes). Each frame corresponds to the accumulation of the events occurring during the sample's first 10 ms. Green and blue pixels correspond respectively to positive and negative events.

5 -Figure F. 6 -

 56 Figure F.5 -Example of line detection on static lines produced by the line detection SNN. The neuronal activity is presented as a raster plot according to their index (y-axis) and time (x-axis).Static lines are generated during 10µs in input every 100µs, randomly horizontal or vertical (first plot). The corresponding layer, either the horizontal (second plot) or the vertical line detector (third plot), activates when it identifies the presence and type of line in the output. This activation is correlated with the corresponding membrane threshold (fourth and fifth plots). The orange and purple dotted lines indicate respectively the apparition of the horizontal and vertical static lines.

  How can visual neuroscience improve computer vision ? . . . . . . . . . . . . . 1.1.1 Why should computer vision draw inspiration from visual neuroscience ? 1.1.2 Which neuro-inspired tools should one use to improve computer vision ? 1.2 Scientific context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Our motivations and objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Neuromorphic data preprocessing for optimised online analysis . . . . . 1.3.2 Neuromorphic embedded processing of spatiotemporal patterns . . . . . 1.4 Our contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 How can we optimise the trade-off between quantity and quality of the visual information as a preprocess for any embedded data processing ? . . 1.4.2 How can we exploit the relevance of the spiking neuron to process the specific chronology of event data ? . . . . . . . . . . . . . . . . . . . . . 1.5 Manuscript outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Neuromorphic hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 Simulation on CPU/GPU . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.2 Human Brain Project's SpiNNaker . . . . . . . . . . . . . . . . . . . . . 2.2.3 Intel's Loihi . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.4 Center for Project-Based Learning's Kraken . . . . . . . . . . . . . . . . 2.3 Applications . .. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Operation of an event camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.1 Biological retina . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1.2 Silicon retina . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Event data representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Applications and corresponding datasets . . . . . . . . . . . . . . . . . . . . . . 43 3.3.1 DVS 128 Gesture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3.2 N-MNIST . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.3 DDD17 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.3.4 DSEC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.3.5 Lip reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.3.6 Overview of existing datasets . . . . . . . . . . . . . . . . . . . . . . . 47 3.4 Limits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

	xii	TABLE DES MATIÈRES
	3.3	Table of contents
	1 Introduction
	1.1 Notations
		State of the art
	3 Event-based cameras
	3.1 xi

2 Spiking neural networks 2.1 Neuronal dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Behaviour of a biological neuron . . . . . . . . . . . . . . . . . . . . . . 2.1.2 Behaviour of a spiking neuron . . . . . . . . . . . . . . . . . . . . . . . 2.1.3 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1.4 Network architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 3.2.1 Representations retaining temporal accuracy . . . . . . . . . . . . . . . 3.2.2 Representations losing temporal accuracy . . . . . . . . . . . . . . . . . 3.2.3 Tools for event data handling . . . . . . . . . . . . . . . . . . . . . . . . CHAPTER 1

TABLE 1

 1

.1 -Overview of publications published during this PhD thesis.

Table 2

 2

	.1: Overview of existing SNN simulators on CPU. (Continued)

Table 3 .

 3 1: Overview of existing event-based datasets.

	Name	Task	Classes	Size and format
	ASL DVS (Yuhuang, 2016)	Object recognition	Sign language for 24 letters (A to Y, excluding J)	19.9GB(MAT)
	ATIS Planes (Afshar et al., 2019)	High speed detection and unsuper-vised object extraction	Different plane models	322.8M B (MAT)
	Color Event Camera Dataset (Scheerlinck et al., 2019)	Multiple applications	Object and human motion, interior scenes, driving, calibrations with and without infrared	32.9GB (binary rosbag)
	DHP19 (Calabrese et al., 2019)	3D human pose recognition	33 movements	160GB (AEDAT)
	DND21 (S. Guo & Delbruck, 2022)	Background activity denoising	Driving and interior scenes, moving dots	44.8GB (AEDAT2)
	Driving Event Camera Dataset (Rebecq et al., 2019b)	Autonomous driving	Highway, street and parking with or without sun	124GB (rosbag)
	DSEC (M. Gehrig et al., 2021)	Autonomous driving and stereovi-sion	Driving in a variety of illumination conditions	152GB (H5)
	DVS09 (Delbruck, 2008b ; Lichtsteiner et al., 2008b)	Recognition, surveillance and tra-cking	Walking, hand motion, driving, eye tracking, mouse activity	450M B (DAT)
	DVS 128 Gesture (Amir et al., 2017)	Gesture classification	11 hand gestures	2.7GB (AEDAT, NPZ)
			Conversion of 4 existing dynamic RGB datasets,
	DVSACT16 (Hu et al., 2016 ; Hu, 2016)	Object tracking, action recognition and object recognition	i.e. VOT Challenge 2015 dataset, TrackingDataset, UCF-50 Action Recognition Dataset and Caltech-	45.1GB (AEDAT, HDF5)
			256 Object Category Dataset
	DVSFLOW16 (Rückauer & Delbruck, 2016) Optical flow	Synthetic and real-life motions	3GB (AEDAT)
	DVSMOTION20 (Almatrafi et al., 2020)	Optical flow	Camera motions (checkerboard, classroom, confe-rence room) and object motion (hands, cars)	849M B (AEDAT, MAT)
	DVSNOISE20 (Baldwin et al., 2020)	Denoising	Interior scenes	7.9GB (AEDAT, MAT)
				Continued on next page

CIFAR10-DVS (H.

Li et al., s. d.)

Object recognition Transportation, animals 7.81GB (AEDAT, DAT) Combined Dynamic Vision / RGB-D Dataset (Weikersdorfer et al., 2014) SLAM 5 scenarios 7.8GB (TSV, BVH) DDD17 (Binas et al., 2017) Autonomous navigation Scenes of highway and urban driving 447GB (AEDAT, HDF5) DDD20 (Hu et al., 2020) Autonomous navigation Scenes of highway and urban driving 931GB (AEDAT, HDF5) DET (Cheng et al., 2019) Lane extraction, autonomous navigation Labelled lines 44.31GB (BIN, BMP)

Table 3 .

 3 1: Overview of existing event-based datasets. (Continued)

	EBSSA (Cohen et al., 2017 ; Afshar et al., 2020)	Detection and tracking for space situational awareness	Daytime and nighttime recordings of spatial objects 12.6GB (MAT)
	EDFLOW (Liu & Delbruck, 2022)		Optical flow	Outdoor scenes	15.8GB (AEDAT)
	Event-based,	Direct	Camera	Tra-
	cking

TABLE 6 .

 6 1 -Hyperparameters used to initialise the separate and mutual SNN downscaling methods (see description of each variable in Section 2.1.3).

	6.2.4.4 Real-time data downscaling on SpiNN-3

TABLE 6 .

 6

	Methods	Number of Temporal Information Generation	Optical
		events	density	entropy	time	consistency
	Original fullscale data	194,398	1.95e -7	2.94e -6	-	-
	Spatial Funnelling	186,414	2.95e -6	3.65e -5	103.20 ms	*
	Tonic Downscale	194,398	5.40e -5	4.31e -4	3.20 ms	*
	SNN Pooling
	Separate handling	145, 680	142.27	0.15	648.86 s	**
	Mutual influence	113, 431	10.77	0.15	680.43 s	***
	Log-luminance
	Event Count	7,778	1.25e -7	1.88e -6	0.83 s	**
	Linear Estimation	624	8.58e -9	1.56e -7	2.63 s	***
	Cubic Estimation	435	8.92e -9	1.66e -7	2.94 s	****

2 -Features of the original DVS128 Gesture dataset and of the six spatial downscaling methods, for a downscaling factor of 4.

TABLE 6 .

 6 3 -Accuracy of the PLIF-based classifier applied to DVS128 Gesture reduced using the temporal structural downscaling methods.

	Reduction factor Deterministic structural Stochastic structural
	80	0.96875	0.97569
	60	0.96875	0.97569
	40	0.96875	0.98263
	20	0.96875	0.95486
	10	0.95833	0.95833
	5	0.95833	0.96181
	1	0.93055	0.95486
	0,5	0.88541	0.94097
	0,1	0.73611	0.73611
	0,05	0.65277	0.625
	0,01	0.45833	0.35763
	Reduction factor Asynchronous temporal funnelling Synchronous temporal funnelling
	50000	0.97916	0.96875
	100000	0.97569	0.97916
	500000	0.9548

TABLE 6 .

 6

4 -Accuracy of the PLIF-based classifier applied to DVS128 Gesture reduced using the temporal funnelling downscaling methods.

TABLE 7 .

 7 2 -SNE latency and energy consumption of the saliency detection functional modules.

	Kraken platform Current (in A) Simulation time Energy (in J)
	Fabric controller	9.5 × 10 -3	1.4 × 10 -1	8.62 × 10 -4
	SNE -active (avg) 44.4 × 10 -3	1.39 × 10 -1	3.85 × 10 -3
	SNE -idle	3.635 × 10 -2	3.6 × 10 -4	8.51 × 10 -6
	Total	NA	1.4 × 10 -1	4.72 × 10 -3

TABLE 7 .

 7 3 -Kraken platform energy consumption and latency.

TABLE 7 .

 7 6 -Energy used by the SpiNNaker implementation. The video has been slowed 10 folds to allow the SpiNN-3 to process every event.

TABLE 7 .

 7 of the saliency detector Values of the output layers

	Resting membrane potential	-65mV	-65mV
	Reset membrane potential	-100mV	-65mV
	Neuronal threshold	-25mV	-20mV
	Membrane time constant	2.5ms	25ms
	Refractory period	0.1ms	0.1ms
	Excitatory decay time	5ms	5ms
	Inhibitory decay time	5ms	5ms
	ω W T A	0.5	/
	∆θ	/	12mV
	t δ	/	50ms

7 -Hyperparameters used to implement the multi-OoI selection model.

TABLE 9 .

 9 1 -Number of each diagonal pattern per cluster and corresponding Gini index, with a network trained on noiseless data and tested on noisy and non-noisy input data. and seem to recognise the same trajectory. Despite those issues, the average Gini index reaches 0.25. Finally, we test this model on noisy input data and observe that the clustering has indeed suffered from the specialisation defects observed in the convolution filters, with an average Gini index of 0.605 (see Table9.2b).

		Diagonal A Diagonal B Diagonal C Diagonal D Gini
	Cluster	25	25	0	0	0.5
	Cluster	0	0	0	0	0
	Cluster	0	0	0	0	0
	Cluster	0	0	25	25	0.5
			(a) Without noise
		Diagonal A Diagonal B Diagonal C Diagonal D Gini
	Cluster	17	13	0	1	0.52
	Cluster	6	6	8	10	0.74
	Cluster	2	6	2	5	0.69
	Cluster	0	0	15	9	0.47
			(b) With noise

TABLE 9 .

 9 2 -Number of each diagonal pattern per cluster and corresponding Gini index, with a network trained on noisy data and tested on noisy and non-noisy input data.

  4.3.2), reimplemented the spiking stereo-matching method introduced by[START_REF] Dikov | Spiking cooperative stereo-matching at 2 ms latency with neuromorphic hardware[END_REF] on SpiNN-3 (see Sec. 2.2.2) and adapted to be tested on real data, i.e. the DSEC dataset (seeSec. 3.3.4). Her internship lasted from March to September 2021. -M. Noah Candaele, a 1st-year student in a master's degree in computer science, initiated the implementation of a demonstrator interfacing SpiNN and event-driven camera for realtime stereo matching. His internship conducted intermittently with his courses, spanned the duration of the semester of Fall 2022.

  Marcel Schweiker (semester of Fall 2022) re-implemented the ANN lip-reading model introduced in[START_REF] Tan | Multi-grained spatio-temporal features perceived network for event-based lip-reading[END_REF] and worked on its application to the i3S dataset, recorded by Raphaël and Juliette. -M. Hugo Bulzomi (semester of Fall 2022) developed an SNN model for lip-reading, ap-

	9.4 -9.4.1 i3S dataset	181
	-M.

TABLE 9 .

 9 3 -Neurons parameters of the lip-reading SNN model.

TABLE 9 .

 9 Limitations and perspectives 4 -SNN1 accuracy on a subset of 10 classes from DVS-Lip, using different surrogate functions. The classes correspond to the words: allow, allowed, America, American, benefit, benefits, billion, called, challenge, and change.

	Models Activation function Accuracy
	SNN1 Erf		0.546
	SNN1 Piecewise	0.531
	SNN1 ATan		0.534
	Models	Variation	Accuracy
	SNN1	Base model	0.395
	SNN2	Base model	0.514
	Spiking MSTP	No GRU	0.522

TABLE 9 .

 9 5 -SNN results on the entire DVS-Lip dataset.

	Model	Model	Accuracy
	MSTP	Original ANN (Tan et al., 2022)	0.721
	Spiking MSTP	Simple fc layer	0.522
	Spiking MSTP	Spiking bi-GRU	0.463
	Spiking MSTP	Spiking recurrent layer	0.476
	Spiking MSTP	Stateful synapse	0.602

TABLE 9 .

 9 6 -Spiking MSTP results on the entire DVS-Lip dataset, trying different substitutions for the 3-layer bidirectional GRU in the classification part.

	Model	Accuracy	Size
	Event Clouds MSTP	0.721	241.5MB
	MSTP low branch without GRU	0.591	47MB
	SNN1	0.395	26.7MB
	SNN2	0.514	88.9MB
	Spiking MSTP	0.602	47MB

TABLE 9 .

 9 7 -Size and accuracy comparison between our models and the state-of-the-art. very challenging even for humans. Moreover, this work provides valuable insights for future studies in this area. We demonstrate that surrogate gradient descent does provide a worthwhile option for supervised training of SNNs, thus giving our spiking MSTP the potential to have competitive results with those of regular deep ANNs.

TABLE 4 -

 4 E Analysis of DVS and RGB coastline dataset Structure of the AEDAT data as saved by IMSE's embedded DVS sensor. Each attribute is the mother of the following attributes on the same line.

			numEvents
			x
	data	polarity	y
			timeStamp
			polarity
		firstTimeStamp
		lastTimeStamp
		numEventsInFile
	infos	dateTime beginningOfDataPointer
		source
		deviceAddressSpace
		fileFormat
	source
	importParams	filePath

TABLE 5 -

 5 Events repartition over time, where the samples are temporally split into successive patches (each 1s long).

  Composition of the i3S dataset

	-addition	-école	-panier
	-appétit	-fillette	-papa
	-attirer	-hache	-parking
	-baguette	-huître	-pomme
	-ballon	-joli	-pyjama
	-beau	-jouet	-requin
	-bonne	-joyeux	-robe
	-carnaval	-lèvres	-ruisseau
	-cheval	-lundi	-sapin
	-chiffon	-maison	-serpent
	-coeur	-maman	-tentation
	-concombre	-manoir	-terminer
	-décider	-moto	-voiture
	-dessin	-musique	-xylophone
	-doudou	-neige	-zèbre
	H.2 Tree structure of the i3S dataset
	words/
	mot_1/
	user1_XXX/

H.1

†. This issue of processing event data on the fly might not be as great when accumulating events to process them as frames. However, we believe in the usefulness of retaining the high temporal accuracy of events for further process, which calls for pre-processing the data event per event.

Acknowledgements

(p(j/cluster)) 2 (9.1)

where n is the number of classes (here the number of patterns to classify) and p(j/cluster) is the frequency of class j in the cluster. The closer the Gini index is to 0, the "purer" is the cluster. We assess the final performance of the network as the average Gini index over each cluster populated by the network.

In a second phase, Kévin implemented three additional metrics to assess the convergence of the convolution kernels towards the recognition of input patterns: the precision, the recall and the F1-Score (see Eq. 9.2).

precision =

T P T P + F P recall = T P T P + F N F 1 Score =2 × precision × recall precision + recall (9.2) where T P corresponds to the true positive events, i.e. when a spike takes place when it is meant to happen ; F P corresponds to the false negative events, i.e. when a convolution layer spikes for an input pattern it is not specialised for ; F N corresponds to the false negative events, i.e. when a convolution layer does not spike for an input pattern it should recognise. The precision thus evaluates whether the actual spike activity matches with what it should have been, and the recall evaluates whether all the spikes that should have happened did happen. The overall performance of the network is therefore evaluated by comparing the expected activity (i.e. the input spikes) to the obtained activity (i.e. the output spikes) for each convolution layer. The metrics described in Eq. 9.2 require the initial step of linking each convolution layer to a specific input pattern. A layer is specialised once its delays and weights match an input pattern, with delays 1) not homogeneous but progressively increasing and 2) significantly lower than the delays of neurons external to this pattern.

Classification performance

Four convolutions trained on noiseless data Using the four initial convolution layers, we train the model on non-noisy synthetic input data after a long phase of manual tuning of various hyperparameters. We first test the trained model non-noisy data ; we obtain the features presented in Fig. 9.5a, which are quite similar to the results obtained by the authors. Each layer is specialised, with a perfect clusterisation according to the Gini index presented in Table 9.1a. Secondly, we test this trained model on noisy input data: in this case, the average Gini index increases to 0.24 (see Table 9.1b) and the clusters are less homogeneous, while still acceptable.

Four convolutions trained on noisy data We wish now to assess the classification performance of this model when trained on noisy input data, similarly to the experiments conducted in [START_REF] Nadafian | Bio-plausible unsupervised delay learning for extracting temporal features in spiking neural networks[END_REF]. When tested on noiseless data, the results show a less marked specialisation than when trained on noiseless data. Additionally, the convolution layers B and C in Fig. 9.5b seem to have specialised for a similar motion. According to Table 9.2a, two layers overpower the others G.2 "Pendulum" dataset TABLE 10 -Comparison of the performance of the cooperative stereo-network on original (first column) and downscaled (second and third columns) data from the "pendulum" dataset. The input data is spatially downscaled using the event count method by a factor 2 then fed to the network implemented on EBRAINS (second column) or downscaled similarly by a factor 2 then cropped before being fed to the network implemented on SpiNN-3 (third column). ) and the performance of the cooperative stereo-network pre (second column) and post (third column) adoption of the maintenance of the previously calculated disparity. The input data is spatially downscaled using the event count method by a factor of 5 and then fed to our cooperative stereo-matching network implemented on EBRAINS (second and third column).

List of figures

Spiking neural networks for embedded event-based vision

Amélie GRUEL

Abstract

In recent years, embedded computer vision has become omnipresent. It encompasses tasks such as detection, recognition and tracking of visual elements, with applications in robotics (autonomous driving), industries (assessment of product quality, automation of repetitive tasks), security, customer experience, social networks, etc. This widespread impetus only reinforces the need to overcome the challenges posed by this area of research, which requires gargantuan energy, high memory, and support for a wide range of algorithms. We believe a promising solution to these challenges can be found in the combined use of spiking neural networks (SNNs) and event-based cameras. SNNs consist of a bio-inspired artificial neural network aiming to mimic the dynamics of biological neurons by processing the information as spike trains. Event cameras are a novel type of bio-inspired visual sensor which generates asynchronous data according to pixel intensity changes. It is ideal for real-time applications, but the great amount of timed information is challenging to process using standard computer vision models. Moreover, event data make a natural match for SNNs in terms of biological inspiration, energy savings, latency and memory use for dynamic visual data processing.

However, the novelty of SNNs and event cameras leaves room for many improvements in terms of optimal preprocessing of data as well as how this data is processed, making the most of the particularities of these scientific concepts. In this thesis, we identified several questions related to this broad field of research, which we have condensed into two main topics. The first issue concerns the optimisation of the embedded preprocessing of event data acquired by an onboard camera to facilitate subsequent analysis. We propose three solutions: event data could either 1) be spatially or temporally reduced, either online or offline; 2) keep only salient elements and discard the rest; 3) be subject to foveation, as a bio-plausible compromise between the two previous solutions. We compared qualitatively and quantitatively the event data obtained after applying each preprocessing method to assess which method yields the best trade-off between the amount of data (i.e. events) kept versus the relevance of information maintained. The second challenge is the exploitation of the SNN relevance to processing the specific chronology of event data in an embedded context. Overall, we hope to have made a valuable contribution to the exploitation of the unique advantages of combining SNNs and event cameras for embedded computer vision, especially concerning event data preprocessing.