Leonardo Da

Florian Personal

Engert

Simplicity

A V A N T-P R O P O S

Au seuil de ce parcours scientifique, je souhaiterais, avec beaucoup d'humilité et tout en ayant conscience de mon peu de recul, évoquer plusieurs aspects de mon travail d'une manière plus personnelle. Il s'agira donc ici plus de mon expérience de la thèse, de ces trois années de recherche, souvent passées à courir après le temps, parfois à m'enthousiasmer devant un résultat bien meilleur qu'espéré, mais aussi à être exaspéré face à l'infructuosité de tel algorithme ou de l'échec de tel projet. Ainsi va la science, m'avait on dit.

La thèse est un peu comme un chemin sinueux, où l'on se perd facilement, où les bifurcations, si elles existent, font office de sentes cachées derrière un fourré. Se repérer devient alors un art, que l'on apprend à maîtriser au fil des années, jusqu'au moment de transmettre le peu que l'on a appris.

Enseigner durant ces années a été à la fois une échappatoire, mais aussi une source de satisfaction. Cette dure science de la transmission des connaissances, d'éveiller la curiosité de l'auditoire, continuera d'enrichir ma vision d'un domaine que l'on pense, à tort, cerner. Quoi de plus beau qu'un regard, peut-être naïf, mais aussi différent, sur un sujet particulier ? Avant tout, cette thèse aura été l'occasion de découvrir un domaine qui, s'il n'a pas encore acquis ses lettres de noblesse, se révèle être un champ dynamique, en plein essor, se basant sur des technologies arrivant à maturité, et véhiculant avec lui un ensemble de belles promesses, dont je ne peux que souhaiter l'accomplissement. J'aspire à continuer à travailler dans ce milieu, ainsi qu'à faire bon usage de l'introspection et de la patience que cette aventure m'a apporté.

Il va de soi que cette aventure n'aurait été possible sans le soutien indéfectible de nombreuses personnes, que je vais essayer de remercier ici, en tentant d'en omettre le moins possible. Tout d'abord, à mes parents, sans qui cette aventure n'aurait jamais eu lieu. Pour ce goût de la curiosité que vous m'avez inculqué, cet état d'esprit ne pouvant prendre pour argent comptant ce qui est parfois communément accepté.

Ensuite, à Elise, ma soeur. Tu as été présente tout au long de l'aventure, avec tes mots de soutien, tes sourires et ces heures, malheureusement bien trop peu nombreuses, passées à refaire le monde.

À Julien et Loïc, soutiens inébranlables depuis de bien longues années maintenant. Nous avons vécu de beaux moments, et de belles aventures nous attendent encore, je l'espère. Et bien entendu à Laura et Lucie, qui ont su me supporter et accepter que je leur emprunte leur compagnon pour de longues sorties en montagne. Les années passent, mais vous revoir me fait à chaque fois un bien fou.

À tous mes collègues, passés ou présents, qui ont su faire de cette dure quête un doux moment. Camille, Quentin, Francesco et Xavier L., mes premières racines dans l'équipe, qui se sont, au fil des années, envolées vers de nouvelles aventures. Je garderai toujours en mémoire nos discussions interminables autour de la machine à café. Merci pour votre accueil chaleureux au sein de cette équipe. Puis, à la nouvelle génération, avec laquelle j'ai traversé les eaux troubles de la recherche, en espérant trouver une issue à nos heures de réflexions. Bien entendu, à la Team Arkose, pour ces heures à respirer de la magnésie, s'esquinter les doigts sur des prises bien souvent trop petites, rigoler et profiter de belles journées pour aller tâter du rocher. Pour votre soutien lorsque mon corps n'a voulu suivre, et vos encouragements lorsqu'il m'a été possible de reprendre. En espérant vous retrouver aux détours d'un rocher très rapidement. À Paul, Kevin A. et Xavier B., qui sont devenus avec le temps bien plus que des collègues. Aux différents bars que nous avons fréquentés avec assiduité, à râler, refaire le monde ou tout simplement discuter. À la nouvelle génération, qui saura faire face à l'adversité et porter ce magnifique domaine encore plus haut. À Piotr Dudek, qui a pris du temps pour me faire découvrir le doux monde du semi-conducteur, du design au layout. À Garrick Orchard, pour ce projet un peu fou de sortir un papier après Telluride.

À Laurent Cabaret, pour son soutien lors de mon monitorat, sa confiance et la liberté qu'il m'a laissée. Enseigner à tes cotés a été un réel plaisir. À Jean-Pierre Barbot et Dominique Placko, deux de mes enseignants à l'École Normale, qui ont su éveiller ma curiosité sur l'électronique analogique, et pour ces nombreux projets encadrés avec vous. À Stéphane Laveau, et toute l'équipe de Prophesee, avec qui j'ai apprécié travailler, et passé des heures à bricoler des démonstrateurs de cette fabuleuse technologie.

A B S T R A C T

This thesis is about the implementation of neuromorphic algorithms, using, as a first step, data from a silicon retina, mimicking the human eye's behavior, and then evolve towards all kind of event-based signals. These eventbased signals are coming from a paradigm shift in the data representation, thus allowing a high dynamic range, a precise temporal resolution and a sensor-level data compression. Especially, we will study the development of a high frequency monocular depth map generator, a real-time spike sorting algorithm for intelligent brain-machine interfaces, and an unsupervised learning algorithm for pattern recognition. Some of these algorithms (Optical flow detection, depth map construction from stereovision) will be in the meantime developed on available neuromorphic platforms (SpiNNaker, TrueNorth), thus allowing a fully neuromorphic pipeline, from sensing to computing, with a low power budget.

R É S U M É P U B L I C A T I O N S

Haessig, G., Cassidy, A., Alvarez, R., Benosman Haessig, G., Gehere, K., and Benosman, R (2018). Spikes decoding spikes : an event-based framework for real-time unsupervised spike sorting. Submitted.

Haessig, G., Galluppi, F., Lagorce, X., and Benosman, R (2018). Neuromorphic networks on the SpiNNaker platform. In preparation.

Haessig, G., & Benosman, R. (2018, May). A sparse coding multi-scale precise-timing machine learning algorithm for neuromorphic event-based sensors. In : Micro-and Nanotechnology Sensors, Systems, and Applications X (Vol. 10639, p. 106391U). International Society for Optics and Photonics. This is a sea turtle. A (very) old animal, that is, every year, migrating from Africa to America, a 6000km journey that takes approximately 2 months. For years, scientists have been looking for the reason of this travel. They found that the beaches in both destinations are identical. Same climate. Same amount of food. And even same food. So why should this turtle take part in such a journey, costing time and energy, to reach a point that is similar to the starting one ? Biologists argued that the turtle is following the gulf stream to grab some plankton, explaining why the chosen path is not a straight one. But plankton can be found in closest places. For now, no plausible clues have been found.

I N T R O D U C T I O N

But... The sea turtle is a 110-million-years-old animal. Fossils have been found in both sides of the Atlantic Ocean, showing that this journey has been part of the turtle life for a long time. What was the shape of the world 110 million years ago ? Africa and America were so close that they formed an unique continent. At that time, the inferior Cretaceous (146million-100million years ago), the two identical beaches were probably a few miles apart. One might have sunrise, the other one sunset. Then, the continent drift slowly took these two places apart. Meanwhile, the turtle kept doing its routine, increasing slowly the distance required to reach the opposite coast, which is written in its genes. This may explain why, 110 million years later, this turtle is traveling across an ocean to reach its destination. 1 I do not claim this theory is the good one. Better ones might exist but this story shows us one important thing : sometimes, it is impossible to understand the reasons of a given behavior just by observing it. We have to go back to the origins of the phenomenon, put it in perspective, understand the growth and development of living systems in order to fully explain a phenomenon. This has to be remembered.

My first electrical engineering lecture was more than a decade ago. And, as in most of the following ones, it started with Moore's law. That the number of transistor in a given device will, and should, increase by a factor of 2 every 2 years [START_REF] Moore | Progress in digital integrated electronics[END_REF]. This has been true for a long time (factor 1.96 between 1971 and 2001), but nowadays, an inflection in this curve can be seen. Why ? This thesis will not be about electrical engineering, but during these three years, I found some hints that became, year after year, some kind of a motivation leading my work.

Since 2004, the frequency of Intel processors has leveled around 3 GHz [START_REF] Weste | CMOS VLSI design: a circuits and systems perspective[END_REF]. Nowadays, performance is given by the number of cores, and no more the clock frequency. Lot of cores means lot of transistors. Even if the energy required by a single transistor to switch is small, integration of billions of them in a single die made the power consumption a major consideration in design. Transistors also became smaller. This size reductions had the effect that the transistors are not exactly totally off, a (very) small current continuously leaking through it. Again, this is not a concern for a single unit. But with dies containing billion transistors, the equation changed. State of the art GPUs integrate 21 billion transistors (NVIDIA GV100 Volta, 12nm, 2017 2 ) or FPGAs with 50 billion (Xilinx Everest, 7nm, 2018 3 ). But that scaling, according to the self-fulfilling Moore's prophecy, couldn't last forever : transistors can not be smaller than atoms. This is physics, and it can not be bypassed.

So, thinking about new approaches is needed. The neuromorphic community, established by Carver Mead and Misha Mahowald in the late 80's [START_REF] Mead | A silicon model of early visual processing[END_REF], has been trying to find answers to these issues. They developed the first prototype of a silicon retina, mimicking the behavior of the human eye [START_REF] Mahowald | VLSI analogs of neuronal visual processing: a synthesis of form and function[END_REF]. Misha Mahowald received, in 1992, Caltech's Milton and Francis Clauser Doctoral Prize for her thesis' originality and "potential for opening up new avenues of human thought and endeavor". Since, the neuromorphic community is trying to find new approaches, some paradigm shift, to renew our way of seeing Perception and Computation.

But what does Neuromorphic mean ? " Neuromorphic engineering, also known as neuromorphic computing, is a concept developed by Carver Mead in the late 1980s, describing the use of very-large-scale integration (VLSI) systems containing electronic analog circuits to mimic neuro-biological architectures present in the nervous system. In recent times the term neuromorphic has been used to describe analog, digital, mixed-mode analog/digital VLSI, and software systems that implement models of neural systems (for perception, motor control, or multisensory integration).

" ( Wikipedia 4 )

The current trend is on Convolutionnal Neural Networks. State of the art dedicated implementations can tackle the problem of computation with 1pJ per addition and 5pJ per multiplication [START_REF] Frenkel | mm2 9.8-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28nm CMOS[END_REF]. But these networks need Random Access Memory (RAM) to all the values (membrane potential, weights). SRAM (Static RAM) access costs 10pJ, DRAM (dynamic RAM) 1nJ. Most of the energy is then spent on memory access, not on computation. Memory is then the bottleneck. I do not claim that CNNs can not solve problems, because they do [START_REF] Lecun | Convolutional networks for images, speech, and time series[END_REF] [START_REF] Esser | Convolutional networks for fast, energy-efficient neuromorphic computing[END_REF], but they are not the most efficient way of doing this. The brain is 30W. Lee Sedol is burning 30W for his brain. DeepMind, with its 1202 CPUs and 176 GPUs, can be estimated at 275kW [START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF], just for learning... With its 30W, Lee Sedol can also read, speak, interact with other people, and many other things that DeepMind is not able to do. This is why we are here, and why this field has a beautiful story to write. This dissertation is organized in two parts. First (Chapters 2,3 and 4), I will introduce new algorithms handling event-based signals. Chapter 2 presents an event-based approach to monocular depth estimation, Depth From Focus, using an event-based sensor and a liquid lens, allowing low power and high frequency depth map construction. Chapter 3 uses some sparse coding tools already existing in the literature and applies them to event-based machine learning algorithm, in order to drastically reduce the number of required prototypes, and thus allowing implementation in hardware with limited resources. Chapter 4 applies a modified version of the machine learning algorithm presented in the second chapter to mono-dimensional signals, for online unsupervised spike sorting tasks. The second part (Chapters 5 and 6) details some implementations of neuromorphic networks on two dedicated platforms : Chapter 5 details the implementation of a spiking neural network to compute optical flow on TrueNorth, IBM's neurosynaptic platform. Chapter 6 is about the implementation of the same optical flow network, and a stereovision network, on the SpiNNaker platform.

D E P T H F R O M D E F O C U S

Depth from defocus is an important mechanism that enables vision systems to perceive depth. While machine vision has developed several algorithms to estimate depth from the amount of defocus present at the focal plane, existing techniques are slow, energy demanding and mainly relying on numerous acquisitions and massive amounts of filtering operations on the pixels' absolute luminance value. Recent advances in neuromorphic engineering allow an alternative to this problem, with the use of event-based silicon retinas and neural processing devices inspired by the organizing principles of the brain. In this paper, we present a low power, compact and computationally inexpensive setup to estimate depth in a 3D scene in real time at high rates that can be directly implemented with massively parallel, compact, low-latency and low-power neuromorphic engineering devices. Exploiting the high temporal resolution of the event-based silicon retina, we are able to extract depth at 100Hz for a power budget lower than a 200mW (10mW for the camera, 90mW for the liquid lens and ∼ 100mW for the computation). We validate the model with experimental results, highlighting features that are consistent with both computational neuroscience and recent findings in the retina physiology. We demonstrate its efficiency with a prototype of a neuromorphic hardware system and provide testable predictions on the role of spike-based representations and temporal dynamics in biological depth from defocus experiments reported in the literature.

. The complexity of eyes' inner structure implies that any visual stimuli from natural scenes contains a wide range of visual information, including defocus. Several studies have shown that defocus is essential in completing some tasks and more specifically for depth estimation [START_REF] Held | Using Blur to Affect Perceived Distance and Size[END_REF][START_REF] Vishwanath | Retinal blur and the perception of egocentric distance[END_REF]. Although a large body of research on Depth From Defocus (DFD) exists since the early 60's, there is currently a gap between the information output from biological retinas and the existing literature both in the vision science and computer vision that uses images as the sole source of their studies. Although images are perfect to display static information, their use in acquiring dynamic contents of scenes is far from being optimal. The use of images implies a stroboscopic acquisition of visual information (unknown to biological systems) at a low sampling frequency. They are thus unable to describe the full dynamics of observed scenes. On the other hand, retinal outputs are massively parallel and data-driven: ganglion cells of biological retinas fire asynchronously according to the information measured in the scene [START_REF] Gollisch | Rapid Neural Coding in the Retina with Relative Spike Latencies[END_REF][START_REF] Berry | The structure and precision of retinal spike trains[END_REF] at millisecond precision. Recent neuroscience findings show that this temporal precision can also be found in other subcortical areas, like the lateral geniculate nu-cleus (LGN) [START_REF] Liu | Variability and information in a neural code of the cat lateral geniculate nucleus[END_REF][START_REF] Reinagel | Temporal Coding of Visual Information in the Thalamus[END_REF] and the visual cortex [START_REF] Mainen | Reliability of spike timing in neocortical neurons[END_REF]. The last decade has seen a paradigm shift in neural coding. It is now widely accepted that precise timing of spikes open new profound implications on the nature of neural computation [START_REF] Rieke | Spikes: Exploring the Neural Code[END_REF][START_REF] Maass | Computing with spiking neurons[END_REF]. The information encoded in the precise timing of spikes allows neurons to perform computation with a single spike per neuron [START_REF] Thorpe | Spike arrival times: A highly efficient coding scheme for neural networks[END_REF]. Initially supported by theoretical studies [START_REF] Thorpe | Spike-based strategies for rapid processing[END_REF], this hypothesis has been later confirmed by experimental investigations [START_REF] Johansson | First spikes in ensembles of human tactile afferents code complex spatial fingertip events[END_REF][START_REF] Panzeri | Population Coding of Stimulus Location in Rat Somatosensory Cortex[END_REF].

Here, we present a novel approach to the depth from defocus, inspired by biological retina ouput, which is compatible with ultra low latency and low power neuromorphic hardware technologies [START_REF] Chicca | Neuromorphic electronic circuits for building autonomous cognitive systems[END_REF]. In particular, we exploit advances made in both mixed signal Analog/Digital VLSI technology and computational neuroscience which enabled us to combine a new class of retina-like artificial vision sensors with brain-inspired spiking neural processing devices to build sophisticated real-time event-based visual processing systems [START_REF] Neftci | Synthesizing cognition in neuromorphic electronic systems[END_REF][START_REF] Indiveri | Neuromorphic architectures for spiking deep neural networks[END_REF][START_REF] Serrano-Gotarredona | CAVIAR: A 45K Neuron, 5M Synapse, 12G Connects AER Hardware Sensory-Processing-Learning-Actuating System for High-Speed Visual Object Recognition and Tracking[END_REF]. We show how precise timing of spiking retinas allows the introduction of a novel, fast and reliable biologically plausible solution to the problem of estimating depth from defocus directly from the high temporal properties of spikes.

Silicon retinas located at the core of the hereby presented system are a novel piece of hardware which do not sense scenes as a serie of frames. Conventional cameras wastefully record entire images at fixed frame rates(30-60Hz) that are too slow to match the temporal sub-millisecond resolution of human senses. Silicon retinas are asynchronous and clock-less, every pixel is independent from its neighbors and only reacts to changes caused by movements in a scene. Data are transmitted immediately and are scene driven, resulting in a stream of events with a microsecond time precision equivalent to conventional high-speed vision sensors, with the addition of being low power and sparse [START_REF] Posch | High-dr frame-free pwm imaging with asynchronous aer intensity encoding and focal-plane temporal redundancy suppression[END_REF]. This type of acquisition increases the sensor dynamic range and reduces power computation.

Spiking Neural Networks (SNNs [START_REF] Ghosh-Dastidar | Spiking neural networks[END_REF]) are computational models using neural stimulation. It has been shown that such networks are able to solve constraint satisfaction problems [START_REF] Binas | Spiking analog VLSI neuron assemblies as constraint satisfaction problem solvers[END_REF][START_REF] Mostafa | An event-based architecture for solving constraint satisfaction problems[END_REF], depth extraction from stereovision [START_REF] Osswald | A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems[END_REF][START_REF] Dikov | Spiking Cooperative Stereo-Matching at 2 ms Latency with Neuromorphic Hardware[END_REF] or flow computation [START_REF] Giulioni | Eventbased computation of motion flow on a neuromorphic analog neural platform[END_REF][START_REF] Haessig | Spiking Optical Flow for Event-based Sensors Using IBM's TrueNorth Neurosynaptic System[END_REF]. As they are mimicking real neurons behavior, they allow a massively parallel, low power calculation, which is highly suitable for embedded computation. The use of a SNN in this work is a natural choice to build a complete neuromorphic event-based system, from the signal acquisition to the final output of the depth information. This is advantageous because of the resulting low-power system promised by the spiking/neuromorphic technology. The developed architecture is particularly adapted on a variety of existing neuromorphic spiking chips such as the SpiNNaker [START_REF] Furber | The spinnaker project[END_REF], TrueNorth [START_REF] Merolla | A million spiking-neuron integrated circuit with a scalable communication network and interface[END_REF] or LOIHI [START_REF] Davies | Loihi: A Neuromorphic Manycore Processor with On-Chip Learning[END_REF] neural chips. More specific neuromorphic hardware, such as the 256 neurons ROLLS chip [START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses[END_REF], can also be used. When combined with an event-based camera, power as low as 100mW is proven to be sufficient to achieve a realtime optical flow computation [START_REF] Haessig | Spiking Optical Flow for Event-based Sensors Using IBM's TrueNorth Neurosynaptic System[END_REF]. We are showing with this work that a low-power ( 100mW), com-putationally inexpensive and realtime DFD system can be similarly achieved.

Among the multitude of techniques developed by vision scientists to estimate depth, those called depth from focus (DFF) or depth from defocus (DFD) have the great advantage of requiring only a monocular camera [START_REF] Gaganov | Robust shape from focus via Markov random fields[END_REF]. The DFF method uses many images, and depth clues are obtained from the sharpness at each pixel. This method is computationally expensive and the amount of data to process is substantial. On the other hand, DFD estimates the variance of spatially varying blur spots based on a physical model. This technique requires less images but at the cost of a greater error in positioning. Current methods that use DFD or DFF generate depth maps for static scenes only [START_REF] Suwajanakorn | Depth from focus with your mobile phone[END_REF] as they are limited by the frame rate of the camera driven at maximum of 25 fps. The computer vision and engineering community have described a number of algorithms for defocus computation [START_REF] Wandell | Common principles of image acquisition systems and biological Vision[END_REF][START_REF] Pentland | A new sense for depth of field[END_REF][START_REF] Moeller | Variational depth from focus reconstruction[END_REF]. However, they typically require multiple concurrent images [START_REF] Zhou | Coded aperture pairs for depth from defocus and defocus blurring[END_REF][START_REF] Watanabe | Rational filters for passive depth from defocus[END_REF][START_REF] Pentland | Simple range cameras based on focal error[END_REF], lightfield systems [START_REF] Tao | Depth from Combining Defocus and Correspondence Using light-Field Cameras[END_REF], specific lens apertures [START_REF] Zhou | Coded aperture pairs for depth from defocus and defocus blurring[END_REF][START_REF] Levin | Image and depth from a conventional camera with a coded aperture[END_REF], correlations [START_REF] Mateos-Pérez | Comparative evaluation of autofocus algorithms for a real-time system for automatic detection of Mycobacterium tuberculosis[END_REF], specific hardware [START_REF] Martel | High-Speed Depth from Focus on a Programmable Vision Chip Using a Focus Tunable Lens[END_REF] or light with known patterns projected onto the environment [START_REF] Pentland | Simple range cameras based on focal error[END_REF]. The use of images and luminance implies high computational costs of around 17ms to process a single frame [START_REF] Mateos-Pérez | Comparative evaluation of autofocus algorithms for a real-time system for automatic detection of Mycobacterium tuberculosis[END_REF].

These approaches cannot serve as conceivable models of defocus estimation in natural visual systems, as mammalian usually operate on a complete different data format and acquisition principles. Early studies [START_REF] Mather | Image blur as a pictorial depth cue[END_REF][START_REF] Mather | Blur discrimination and its relation to blurmediated depth perception[END_REF] show that the border between blurred and sharp regions can be used to establish the depth-order of objects. For example, an out-of-focus target with a blurry textured region and a blurry border was perceived to be located proximal to the plane of focus, while an out-of-focus target with a blurry region and a sharp border was perceived to be located distant to the plane of focus. Recent findings in neurosciences show that blur perception in human is a dynamic process that allows depth assessment. In particular, the retinal defocus blur provides information regarding the relative and/or absolute distance of objects in the visual field [START_REF] Grant | Accommodation and convergence in visual space perception[END_REF]. Recently [START_REF] Nguyen | Detection of the depth order of defocused images[END_REF], it has been demonstrated that subjects were able to detect the relative distance of two vertical edges, justifying that the retinal blur allowed the subjects to judge target distance differentially without any other depth cues. Other studies demonstrated that motor efference and/or sensory feedback related to the blur-driven accommodative response contain sufficient information to estimate the absolute distance of visual targets [START_REF] Fisher | Accommodation and apparent distance[END_REF]. In addition, information derived from image blur can be integrated by the visual system with other visual cues (e.g., retinal disparity, size, interposition, etc.), which would assist in enabling one to judge the depth order of objects over a range of distances [START_REF] Ciuffreda | Why two eyes?[END_REF][START_REF] Ciuffreda | Is one eye better than two when viewing pictorial art?[END_REF][START_REF] Mather | The use of image blur as a depth cue[END_REF][START_REF] Mather | Blur discrimination and its relation to blurmediated depth perception[END_REF][START_REF] Mather | Depth cue integration: stereopsis and image blur[END_REF]. The addition of blur information can improve the speed and accuracy in such a depth-ordering task [START_REF] Mather | Combining depth cues: effects upon accuracy and speed of performance in a depth-ordering task[END_REF].

.

. .

Event based cameras

Biomimetic neuromoprhic silicon event-based cameras are a novel type of vision sensor that are data driven. Unlike their frame-based counterparts, they are not controlled by artificially created timing and control signals (frame clock) with no relation to the source of the visual input. Events are generated when significant changes of the relative luminance occur at the pixel level as shown on Figure 2. The visual output is in the form of an address event (AER) and encodes the visual information in the time dimension at the microsecond time precision. As soon as a change of luminance is detected, the process of communicating the event off-chip is initiated. The process executes with low latency, of the order of a microsecond, ensuring that the time at which an event is read out from the camera inherently represents the time at which a contrast change is detected. Let e(x, y, p, t) be an event occurring at time t at the spatial location (x, y) T . A positive change of contrast will result in an "ON" event (p = +1) and a negative change of contrast will result in an "OFF" event (p = -1). The threshold n beyond which a change of contrast is high enough to trigger an event is tuned according to the scene. Smaller intensity fluctuations do not generate any event and are not recorded. The camera used in our setup is an asynchronous sensor which has a 640 × 480 pixels resolution [START_REF] Posch | High-dr frame-free pwm imaging with asynchronous aer intensity encoding and focal-plane temporal redundancy suppression[END_REF] with a high temporal resolution of 1µs. This array of fully autonomous pixels combines both a luminance relative change detector circuit and a conditional exposure measurement block (not used in the paper). When no change of luminance is detected, no events are generated and the static information is not recorded. This reduces the data load and allows high speed online processing at the native resolution of the sensor.

. .

Depth estimation

Geometrical optics relations

Let us consider a single thin convex lens of focal distance f with an infinite circular aperture. As shown on the Figure 3, the rays issued from a point z = d converge to form an image behind the lens onto the detector according to the geometrical optic formula:

m × m = f 2 (1)
In a stigmatic optical system, an object point is in focus when the image formed is a point as well. Any object out of focus forms a blurry spot on the detector. We can compute the distance of the camera to the objective as a function of the focal length of our optical system and the position of the object in focus: As this distance is fixed in our setup, increasing the focal value f results in an increase of the distance d at which an object is in focus and vice-versa. By tuning the focal value of the optical system, every plane of the 3D scene is successively in focus.

D cam/obj = f + m = f + f 2 m = f + f 2 d -f . ( 2
)
When an object is located before or after the focus point, it will form a blurry image of size s. Geometrical optic relations (Figure 3) give :

s = f 2 N × |x -d| (d -f)z , (3)
with f the focal value of the optical system, N the numerical aperture, d the position of the object when in focus and z(t) the variable position of the object over time. Due to the aberrations, diffraction phenomenon and nonidealities of the lenses, a Gaussian point spread function is commonly used to describe the defocus blur spot [START_REF] Lin | Depth recovery from motion and defocus blur[END_REF].

A real optical system has a finite aperture size, limited by the dimensions of the optics. The spatial resolution is also limited by the pixel size below which it is not possible to distinguish focus. This represents the circle of confusion, C, of the camera as shown on Figure 3. As a consequence, a range of several points will form an image "in focus" (the image spot size is smaller than the circle of confusion) on the detector for a given focal length. This range is called depth of field and the two limiting points are the values of z close and z far for which s = C, i.e.

z close/far = d 1 ± CN(d-f) f 2 , (4)
The depth of field is then given by computing the difference between z close and z far :

D o F = |zf -zc| = 2 × CNd(d -f) f 2 -(CN(d-f)) 2 f 2 . ( 5
)
The DoF of the lens is increasing with the distance of focus. Beyond a certain distance, called the hyper-focal, the whole scene appears in focus and differences in depth can no longer be distinguished. Ideally a DFD sensor should have an infinitely thin DoF for each focusing distance and an infinite hyper-focal. In practice one needs to minimize the DoF and increase the hyper-focal to have the best spatial resolution in depth on the longest distance possible.

Depth from the time of focus

The spread parameter σ(t) is proportional to the diameter s(t) of the ideal blur circle, i.e. σ(t) = αs(t). The resulting intensity onto the sensor, at a pixel (x i , y i ) is:

I i,j (x, y, t) = A. exp - r 2 i 2σ(t) 2 . ( 6
)
with r 2 i = (x -x i ) 2 + (y -y i ) 2 and A the amplitude. At the pixel level the evolution of the intensity will depend on how close to the camera the object is. As a function of time, the standard deviation in I can be used to determine the time t at which an event is triggered by the pixel, assuming σ is invertible i.e.:

t = σ -1 r 2 i 2 log A -log I i,j (x, y, t) (7)
We are dropping subscripts (i, j) for readability purpose as what we are describing is valid for any pixel. Hence, given the intensity at an arbitrary time t 0 , if the variations of its log reach some threshold ±n (described in the previous section), then: log I(x, y, t) I(x, y, t 0 ) = ±n and log I(x, y, t) = log I(x, y, t 0 ) ± n.

This gives the time when an event is emitted according to [START_REF] Bardow | Simultaneous optical flow and intensity estimation from an event camera[END_REF]:

t = σ -1 r 2 2(log A -log I 0 ∓ n) (9)
The sign of n is chosen according to the polarity of the spiking event, itself related to the sign of the intensity's derivative:

sign(n) = sign(p) = sign dI dt (10)
When the derivative is positive the polarity will be +1 (ON event) and -1 when negative (OFF event). Eq [START_REF] Barranco | Contour motion estimation for asynchronous event-driven cameras[END_REF] expresses when an event will be emitted w.r.t. n and to a reference event measured at t 0 . As we reach focus, the value of σ will be constant for small duration of time, therefore the derivative of I, ∂I ∂t is equal to 0, followed by a polarity change as shown in Fig. 4(c) and expressed in the temporal domain in Fig. 4(d) around 50ms. The detection of focus can then be determined by detecting the time t f of the polarity change that can be estimated from the average timing between the consecutive ON and OFF events. We can then estimate the size of the defocus blur s(t f ) according to [START_REF] Bardow | Simultaneous optical flow and intensity estimation from an event camera[END_REF] and deduce from (??), the depth information z(t f ) as:

z(t f ) = ∓df 2 /N S(t f )(d -f) ∓ f 2 /N . ( 11
)
The change of sign in z corresponds to the focal length that is the closest to the focus. Parameters d and f are controls of the liquid lens device.

Defocus with an event-based camera

Let us consider a small luminous object that will successively be out of focus, in focus and out of focus again. When a sweep of the focal length over its dynamic range is carried out, objects will successively appear out of focus, then in focus and out of focus again. The blurry spot around the object will therefore shrink until the object is sharp and grow again as shown in Fig. 4(a) and in Fig. 4(b) for a cross section of the blur spot. The size of the blur spot increases in connection to the distance respectively to the depth of field (DoF) location. When the object is in focus, the image spot will have its minimum size and the contrast will be maximum (sharp edges). . .

Liquid lens control

The optical system shown in Fig. 2(a) is composed of three components:

-an electrically focus-tunable liquid lens with a 10 mm clear aperture and focus range f ll ranging from 50 to 120 mm [START_REF] Blum | Compact optical design solutions using focus tunable lenses[END_REF]. -an offset lens with a focal f o = -150 mm. It acts as a relay imaging system between the focus-tunable lens and the objective and ensures a proper focus.

-an objective lens with focal length f ol = 35 mm, f ol /2 objective lens. This objective is a good compromise between large focal value, large clear aperture and low bulk (23.4mm length). It is used to form an image directly on the camera pixel array.

More details are given in Supplemental Data.

The thin lens approximation is given as follows:

d = f eq + f 2 eq D cam/obj -f eq , ( 12
)
where d is the position of the point in focus, f eq is the global optical system's equivalent focal value (liquid lens + offset lens + objective lens) and D cam/obj is the distance between the camera and the object. See Supplemental data for details.

The thin lens approximation assumes that the equivalent focal length f eq of our optical system is :

1 f eq = 1 f ll + 1 f ol + 1 f o (13)
with f ol and f o being respectively the objective lenses and the offset focal lengths.

. .

Spiking neural network

To estimate t f for each pixel, we are looking for the smallest time interval between two consecutive events of opposite signs. We implement a Spiking Neural Network (Figure 5.a) based on Leaky Integrate and Fire neurons ( [START_REF] Lapicque | Recherches quatitatives sur l'excitation electrique des nerfs traitee comme polarisation[END_REF]) to process the spikes from the output of the neuromorphic silicon retina. For every pixel, five neurons are required. Figure 5.a shows events generated by a circle going in and out of focus. At time t 1 , the stimulus in front of the receptive field generates a ON event (orange -Figure 5.c). The synaptic weight between the ON and B on neurons is not strong enough to trigger yet the B on neuron (Figure 5.d). As a second spike is generated by the same neuron at time t 2 , the B on neuron reaches its threshold value and spikes (Figure 5.d). A small inhibition link to the OUT neuron ensures that the OUT neuron won't fire now. After the focus, at time t 3 , we have a polarity inversion : the OFF neuron fires, thus exciting the output neuron that fires (Figure 5.e). The next OFF spike, at time t 4 , activates the B off neuron, thus preventing the OUT neuron to fire again in response to the future OFF spikes. Finally, the Sync neuron is triggered by the liquid lens, warning that the sweep is over and reseting the OUT neuron to its initial state. The depth can then be extracted as the timing between the OUT and Sync spikes. They directly connect to the output neuron, and also to two blocker neurons B on and B off that are inserted to avoid parasite firings of the output neuron due to a sequence of only ON or OFF polarity events. A synchronization with the liquid lens via the Sync neuron is added, in order to encode the depth in the length of the spike train. c-e) Simulation of the SNN with NEST. (c) The input spikes (ON and OFF events) and the output of the network (OUT and Sync). The point of focus is given by the OUT neuron, while the distance is encoded in the timing between the OUT and SYNC spikes. (d) Membrane potential for the two blockers neurons. After the first spike of its respective polarity, the blockers send a inhibition to the output neuron. (e) Membrane potential of the output neuron. Spikes from the same polarity do not allow the output neuron to reach its firing threshold, while a succession of ON and OFF events make the output neuron fire.

.

Results are obtained for a field of view of 15 • and a depth that ranges from 0.12 to 5.5 m. The distance upper bound corresponds to the hyperfocal distance of the global optical setup. The sparse nature of the data allows the algorithm to operate in real time at the native resolution of the sensor (640 × 480pixels).

The Spiking Neural Network previously described in section 2.2.4 was implemented using the PyNN framework [START_REF] Davison | PyNN: a common interface for neuronal network simulators[END_REF] and simulated using the NEST neural simulator [START_REF] Gewaltig | Nest (neural simulation tool)[END_REF]. All neurons are modeled as Leaky Integrate-and-Fire (LIF) neurons. Results are presented on Fig. 5. We set the dimension of the network to fit a region of 447 × 447 pixels, the network then using 999045 neurons. This amout is compatible with existing neuromorphic hardware implementation on the TrueNorth platform (1 million neuron [START_REF] Merolla | A million spiking-neuron integrated circuit with a scalable communication network and interface[END_REF]) or SpiN-Naker capability [START_REF] Furber | The spinnaker project[END_REF].

To better understand the possibilities and limits of the system, we performed a simulation on synthetic data generated with a controlled optical setup where all parameters can be tuned. The aim of this simulation is to study the algorithm without constraints from the physical setup. Fig. 6 shows three snapshots of the events generated during a sweep of a car. Fig. 6d) shows the reconstructed depth computed by the system. All the parameters being known we can estimate the relative error to the ground truth. We notice that most of the error is located at the front of the car on the grating where repetitive patterns are located. This is a known limitation of several vision algorithms such as stereo matching, which will be further discussed in section 2.3.1. Fig. 6 (e) displays the error repartition with a mean relative error of 10.4%. An example video on a car is available online [START_REF] Haessig | [END_REF].

The second experiment, the depth estimated from the DFD is assessed with our setup on a monitored scene where the ground truth is provided by a Microsoft Kinect sensor. The Kinect is taken as the reference similarly to previous studies [START_REF] Khoshelham | Accuracy analysis of kinect depth data[END_REF][69], reporting reconstruction precisions of few mm at 50cm to 3cm at 3m. Fig. 7 shows the setup and the depthmap computed for the presented neuromorphic technique with a comparison with the groundtruth depthmap: the error is increasing relative to depth. Up to 1.5m, the relative error is upper-bounded at 4% and increased up to 23% at 2m. This is however an expected result as the optical system's focal length is reaching the hyper-focal. The third experiment shows reconstruction for several objects with different textures and sizes. Fig. 8 shows for each object its corresponding depth map while the lens is sweeping through the object. . .

Remarks and limitations

The algorithm is inexpensive in computational power, in the presented experiments it is able to deal with more an average of 15 million events per second. A shaky scene viewed by the event-based camera will generate at most 2 million events per second. In the worst case, thus the current approach is 7 times faster than real time. However for most objects used it is more around 20 times faster than real time using an off-the-shelf laptop. This algorithm can be easily embedded on portable devices such as smartphones or autonomous vehicles as an ideal method for low power solutions to obstacle side-stepping or 3D scanners.

As pointed out during experiments, repetitive patterns can lead to incorrect depth estimation. Fig. 9 shows this situation for simulated data. If we consider two objects that are well separated (Fig. 9.f), the sweep of the liquid lens will produce an event stream (Fig. 9.l) with non overlapping spikes. Fig. 9.j is a snapshot of the sweeps' beginning. The four OFF edges are distinct. As the focus evolves, we reach the focus point for object 1 (Fig. 9.i). The two edges O 1L and O 1R of object 1 now generate ON events. After the focus point for object 2 (Fig. 9.h), the two other edges O 2L and O 2R now generate ON events. As the objects are in a sufficient relative distance, the edges O 1R and O 2L are not overlapping. If we consider two objects that are close each other (Fig. 9.a), the sweep of the liquid lens will now produce the event stream shown in Fig. 9.k. As the sweep starts (Fig. . In this paper we proposed a spiking neural network model that solves the depth from focus efficiently by exploiting an event-based representation amenable to neuromorphic hardware implementations. The network operates on visual data in the form of asynchronous events produced by a neuromorphic silicon retina. It processes these address-events in a data-driven manner using artificial spiking neurons computation units. This work introduces a valid explanation and a robust solution to depth estimation from defocus that has not been reported in the literature. The overall system matches recent existing literature of neuroscience, biological retinas and psychophysics studies on the role of defocus in the visual system. This network is nonetheless an abstract simplification of the depth estimation problem that must surely combine more complex information in biological systems. More importantly, this study should be coined depth from focus rather than from defocus as the neural structure developed aims at detecting the exact time of focus during a sweep. Although five decades of research tried to solve the problem of depth from defocus, the fundamental difference and novelty of this work is that the net- The two central plots show events in the x-time plane, smashing the y-dimension. Events are color coded with their polarity (red for OFF events, blue for ON events). The right one is a valid case, with no overlap.

The left one contains an overlap in the event stream, leading to wrong depth deductions in this case. 4 snapshots of events are presented for every case work proposed operates using exclusively precisely-timed temporal contrast events. These events are measured directly from the neuromorphic silicon retina, which models only the transient responses of retinal cells (i.e., of the Y-ganglion cells), without including the sustained ones, yet present in the system. While the sustained information is present in the silicon retina used, we show that this information is not necessary to provide depth estimation from defocus. Silicon retina transient responses produce single events. Their precise timing plays a crucial role in the estimation of blur and more importantly in determining when the observed object is in focus.

In contrast, the vast majority of computational models of depth from defocus are based on images that are known to be absent from the visual system and only rely on luminance information. Additionally, none of them use the precise timing of spikes. In these models, convolutions techniques are used to determine the level of blur. These methods are computationally expensive and meaningfully slower as several acquisitions are often needed to provide an accurate result. By contrast, the model we presented does not incorporate any notion of filtering or convolutions. These choices are based on the perception of spatial contrast, whereas the presented model solely responds to temporal contrast. Whether the brain is using such a technique to estimate depth from defocus is an open question. However due to the nature of precisely timed information output by biological retinas [120] convolutions algorithms cannot provide a viable explanation as the stroboscopic nature of image acquisition and luminance use is incompatible with neural systems. Instead, we show that the change of polarity at the pixel level contains sufficient information to estimate depth from defocus. Recent findings in physiology show that several mechanisms used by our methodology exist in the Nature. Biological retinas contain several types of ganglion cells, each informing the brain about a particular content of the visual scene, such as motion, edges or chromatic content. In a recent paper, a newly discovered ganglion cell type 'On-delayed' is described [START_REF] Mani | Circuit mechanisms of a retinal ganglion cell with stimulus-dependent response latency and activation beyond its dendrites[END_REF]. This cell has been shown to respond vigorously to increasing blur. Its degree of firing directly encodes the amount of high spatial frequencies contained in its receptive field. More importantly, this cell gets input from both ON and OFF polarities. While it is currently unknown how this defocus information is used by the brain, it is most likely that this information projects to the visual thalamus and cortex and also to midbrain structures where accommodation is controlled [START_REF] Baden | Visual Neuroscience: A Retinal Ganglion Cell to Report Image Focus?[END_REF].

We expect the most significant impact of our model to be in the field of artificial vision. Today's machine vision processing systems face severe limitations imposed both by the conventional sensors front-ends (which produce very large amounts of data with fixed sampled frame-rates), and the classical von Neumann computing architectures (which are a ected by the memory bottleneck and require high power and high bandwidths to process continuous streams of images). The emerging field of neuromorphic engineering has produced efficient event-based sensors, that produce low-bandwidth data in continuous time, and powerful parallel computing architectures, that have co-localized memory and computation and can carry out low-latency event-based processing. This technology promises to solve many of the problems associated with conventional computer vision systems. However, the progress so far has been chiefly technological, whereas related development of event-based models and signal processing algorithms has been comparatively lacking (with a few notable exceptions). This work elaborates on an innovative model that can fully exploit the features of event-based visual sensors. In addtion, the model can be directly mapped onto existing neuromorphic processing architectures. Results show that the full potential is leveraged when single neurons from the neural network are individually emulated in parallel. In order to emulate the full-scale network, however, effcient neuromorphic hardware device capable of emulating large-scale neural networks are required. The developed architecture requires few neurons per pixel and is implementable on a variety of existing neuromorphic spiking chips such as the SpiNNaker [START_REF] Furber | The spinnaker project[END_REF], TrueNorth [START_REF] Merolla | A million spiking-neuron integrated circuit with a scalable communication network and interface[END_REF] or LOIHI [START_REF] Davies | Loihi: A Neuromorphic Manycore Processor with On-Chip Learning[END_REF] neural chips.

This chapter introduces an unsupervised compact architecture that can extract features and classify the contents of dynamic scenes from the temporal output of a neuromorphic asynchronous event-based camera. Event-based cameras are clock-less sensors where each pixel asynchronously reports intensity changes encoded in time at the microsecond precision. While this technology is gaining more attention, there is still a lack of methodology and understanding of their temporal properties. This chapter introduces an unsupervised time-oriented event-based machine learning algorithm building on the concept of hierarchy of temporal descriptors called time surfaces. In this work we show that the use of sparse coding allows for a very compact yet efficient time-based machine learning that lowers both the computational cost and memory need. We show that we can represent visual scene temporal dynamics with a finite set of elementary time surfaces while providing similar recognition rates as an uncompressed version by storing the most representative time surfaces using clustering techniques. Experiments will illustrate the main optimizations and trade-offs to consider when implementing the method for online continuous vs. offline learning. We report results on the same previously published 36 class character recognition task and a 4 class canonical dynamic card pip task, achieving 100% accuracy on each.

. Neuromorphic event-driven time-based vision sensors operate on a very different principle than conventional frame-based cameras. Instead of acquiring static images of a scene, these sensors asynchronously record pixel intensity changes with a high temporal precision (around 1µs). The event format differs significantly from frames, and therefore conventional machine learning algorithms cannot be directly applied if one wants to fully use its potential and temporal properties in terms of power consumption, computational cost and low memory requirements. Previous notable work on object recognition using event-driven time-based vision sensors include real-time event-driven visual pattern recognition that recognizes and tracks circles of different sizes using a hierarchical spiking network running on custom hardware [START_REF] Serrano-Gotarredona | CAVIAR: a 45k neuron, 5M synapse, 12G connects/s AER hardware sensory-processing-learning-actuating system for high-speed visual object recognition and tracking[END_REF], a card pip recognition task on FPGAs, implementing different hierarchical spiking models inspired by Convolutional Neural Networks (CNNs) [START_REF] Pérez-Carrasco | Mapping from Frame-Driven to Frame-Free Event-Driven Vision Systems by Low-Rate Rate Coding and Coincidence Processing-Application to Feedforward ConvNets[END_REF] and new methods such as HFirst [START_REF] Orchard | HFirst: a temporal approach to object recognition[END_REF] and recently HOTS [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] , an unsupervised algorithm which fully considers the spatio-temporal aspects of event-based sensors. In this chapter, we introduce a compact hierarchical event-driven multi-temporal framework to learn spatiotemporal patterns of dynamic scenes extending the concept of hierarchically increasing spatiotemporal-scales time-surfaces introduced in [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] . A time-surface is a descriptor that provides a time context around an incoming event and describes the temporal activity in its surrounding. This descriptor, applied in a multilayer architecture requires a high number of features to correctly characterize a dynamic visual input and therefore high memory and computational costs. Ideally there must be a one to one relation between interesting spatio-temporal time-surfaces detected in the scene and the ones stored in each layer of the architecture. The aim of this work is to present a new formulation of the method in order to reduce the number of features using sparse coding to express any time-surface as a linear combination of elementary time-surfaces rather than selecting the most representative ones using clustering techniques (iterative K-means [START_REF] Barbakh | Online clustering algorithms[END_REF] in the original paper).

. -

Biomimetic event-based cameras are a novel type of vision sensor that are event driven. Unlike their frame-based counterparts, they are not controlled by artificially created timing and control signals (frame clock) with no relation to the source of the visual information. Events are generated when significant changes of the relative luminance occur at the pixel level as shown in Figure .10. The visual output is in the form of an Address Event Representation and encodes the visual information in the time dimension at the microsecond time precision. As soon as a change of luminance is detected, the process of communicating the event off chip is initiated. The process executes with low latency, of the order of a microsecond, ensuring that the time at which an event is read out from the camera inherently represents the time at which a contrast change was detected. The camera used in our setup is an Asynchronous Time-based Image Sensor (ATIS) [START_REF] Posch | A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[END_REF] which has a 304 × 240 pixel resolution. This array of fully autonomous pixels combines both a relative luminance change detector circuit and a conditional exposure measurement block. Only the change detector circuit will be used for our experiments. When no change of luminance is detected, no events are generated and the static information is not recorded. This reduces the data load and allows for high speed online processing.

. In this section, we will introduce the notion of time-surface, describing the spatio-temporal surrounding of an event, before showing that these so defined time-surfaces can be clustered using a sparse coding algorithm, thus allowing a drastic reduction of the total number of features used for classification, which directly impacts the hardware implementability of the algorithm, as discussed in next section. , a change detection event is generated, like at time t 0 . This triggers a second circuitry which measures the absolute gray level value. This value is coded in the timing between two events occurring at t 1 and t 2 . This timing encodes the time required to capture a given amount of light, and is then inversely proportional to the absolute gray level.

. .

Time-surface construction

We consider a stream of visual events (Figure 11.b), which we define as ev i = {x i , y i , t i , p i } where ev i is the i-th event, and consists of its spatial location (x i , y i ), its timestamp t i , and its polarity p i , with p i ∈ {-1, 1}, for respectively a decrease or increase in luminance. In [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] the notion of time-surface feature S i is introduced. It represents the previous activity around the spatial location of an incoming event ev i . Thus, for an incoming event ev i , we define a time context (Figure 11.d) as an array of the last activation time in the (2R + 1) 2 neighborhood (of radius R), centered at (x i , y i ). The time-surface S i (Figure 11.f) around an incoming event ev i is then obtained by applying an exponential decay (Figure 11.e) to each element of the time context [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF]) . For an incoming event ev i at location (x i ,y i ), the value of its associated timesurface is:

S i (u, v) = e -(t i -t u,v )/τ where u ∈ [[x i -R, x i + R]], v ∈ [[y i -R, y i + R]], t u,v
the timestamp of the most recent event that occurred at the respective pixel, t i the timestamp of the current event and τ the time constant of the exponential decay. As all the events are processed one after the other, the time difference t i -t u,v is necessary positive. Figure 11.b shows this full process of time-surface generation.

. . Training phase: Finding the patch of projection basis

Following the sparse coding algorithm introduced in [START_REF] Olshausen | Emergence of simple-cell receptive field properties by learning a sparse code for natural images[END_REF] , assuming that any time-surface S(x, y) can be approximated as a linear combination of N functions φ j (x, y) for j ∈ [ [1, N]] giving its estimation S (for clarity, we omit the i subscript for each event) :

S(x, y) = N j=1 a j φ j (x, y)
where the a j are real linear coefficients (a j ∈ R for j ∈ [[1, N]]). This relation is equivalent to the projection of a time-surface S onto a subspace defined by the elementary time-surfaces φ j . The feature estimation can be classically formulated as an optimization problem. The solution is given by minimizing the following residual error function E, set as the difference between the original surface and its reconstruction using the elementary time-surfaces linear summation (left member) and a measure of the sparseness of the coefficients (right member):

E = x,y   S(x, y) - N j=1 a j φ j (x, y)   2 reconstruction error + λ N j=1 | a j σ | sparseness of the (a j ) (14)
Where λ is a positive constant that determines the influence of the second term relative to the first, and σ a scaling constant.

The minimization is performed using a training dataset of events from the event-based camera [START_REF] Olshausen | Emergence of simple-cell receptive field properties by learning a sparse code for natural images[END_REF] . Elementary time-surfaces are initialized with random uniformly distributed values between 0 and 1, while the coefficients values are initialized with the product between the initial features and the time-surfaces computed from the training dataset. We use the conjugate gradient descent method [START_REF] Press | Numerical recipes in C[END_REF] to minimize the error E (i.e. maximize the similarity between S and S, by updating the coefficients a j , while maximizing the sparseness of the coefficients). Elementary time-surfaces are then updated by adding the residual error weighted by the coefficients obtained from the previous iteration of the minimization process:

φ j ←-E.a j .η + φ j
where η is the learning rate. All time-surfaces are computed offline from the learning dataset. Then, they are fed to the minimization algorithm until convergence. The number of features is computed iteratively and set as the one providing the lowest reconstruction error on the training time-surfaces set.

. . Building a hierarchical model

In the initial architecture described in [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] , each layer has a set of elementary time-surface prototypes matching time surfaces from the observed scenes and learned during the training phase. When an input time-surface matches a prototype, an event is produced and transmitted to the next layer. A single input event will produce at most one output event. In the present Figure 14 shows the response obtained at the output of a layer, and how it is sent to the next layers in order to build the proposed hierarchical model. Figure 14.a shows an object moving in front of an ATIS camera. Each event is captured from the stream sent by the event-based camera. In Figure 14.b, time-surfaces for each event are built by a convolution with an exponential kernel of time constant τ 1 , for a neighborhood of (2R 1 + 1) side length. As described in section II, time-surfaces are sent to Layer 1, to find a projection basis of N 1 elementary time-surfaces shown in Figure 14.c. Once the elementary time-surfaces projection basis has been extracted, the learning process of Layer 1 is finished. Now each incoming time-surface is directly projected onto all elementary time-surfaces basis. The projection coefficients (Figure 14.e) are defined as the least square solutions that minimize the error function E (Equation 14). The a j coefficients are constrained between -1 and 1. The projection coefficients are split into two groups (Figure 14.f-g), depending on whether their value is positive or negative, as detailed in [START_REF] Lagorce | Stick: Spike time interval computational kernel, a framework for general purpose computation using neurons, precise timing, delays, and synchrony[END_REF] . For each feature j of the basis, a linear decay is applied to the projection coefficients as shown in Figure 14.g, and a new event ev out is generated :

ev out = {x out , y out , t out , p out } where :        (x out , y out ) = (x in , y in ) t out = t in + α 1 -|a j | p out = j | j ∈ [[1, N 1 ]] (15)
were α is a timing scale factor describing the time span of the newly generated event-stream. The higher the coefficient, the higher the similarity between the basis and the surface input. We then encode the similarity level into the time domain, as a delay of the output event depending on the similarity between two surfaces. As shown in Figure 14.h, new events are convolved with an exponential decay of time constant τ 2 , for a neighborhood of (2R 2 + 1) side length, in order to generate new time-surfaces. In Figure 14.i, output delayed spikes, for positive coefficients, according to the results of convolutions with each elementary time-surfaces basis are sent to Layer (2, +) for training. The goal is now to determine the N 2 elementary time-surfaces of Layer 2. As shown in Figure 14.j, steps (d) to (h) are repeated. Then, the same steps (g) and (h) are applied to Layer (2, -) for negative coefficients.

Since the nature of the input and the output of our model is the same, events produced at the output of Layers (2, +) and (2, -) can be sent to the next layer, and the process from step (b) to (g) is repeated (Figure 14.l-k).

Layers are trained consecutively one after the other.

As in [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] , the main idea of the architecture is to gradually increase the complexity of spatial and temporal information. Information is integrated over larger and larger time scales. Each layer L i increases its number of elementary time-surfaces N i , the neighborhood radius R i , and the integration time constant τ i :

       τ i+1 = K τ τ i R i+1 = K R R i N i+1 = K N N i (16)
The network is only defined by a set of six parameters : the initial conditions (τ 0 , R 0 , N 0 ) and the evolution parameters (K τ , K R , K N ). Comparisons with [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] are provided in the experiments section.

. . Classification

The output of the last layer of the hierarchical structure is fed to a classifier for pattern classification. We use a simple classifier to show that the model provides sufficient discrimination without the need for complex classification methods. However, in case of larger databases, where a variety of descriptors are generated for the same class, the use of more advanced classification algorithms might become necessary. We compute an histogram from the output of the last layer of the hierarchical model that contains the total number of responses of each feature to the input pattern, independently from its spatial position. This is the signature of the observed object, and will be used for further comparisons. This method is the same as the one explained in [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] . Figure 13b shows an example of such an histogram.

Each learning example is presented to the model, in order to learn its signature. In a second step, during the testing phase, the signature of incoming patterns are computed and compared to the signatures of each learned examples. The closest one is then identified and finally, the recognized object is obtained by a majority vote from the results of all the last sub-layers.

We will use two types of distances between signatures : the Euclidean distance and the Bhattacharyya distance [START_REF] Bhattacharyya | On a measure of divergence between two multinomial populations[END_REF] . The histogram signature for classification, which is a weak classifier, is here chosen in order to prove the robustness of the proposed descriptor. Stronger classifiers exist (SVM, adaboost methods, ...), but their use in this context is outside the focus of this chapter. . .

Letters and digits dataset

The model has been used with the "Letters and Digit" dataset, provided by Orchard et. al [START_REF] Orchard | HFirst: a temporal approach to object recognition[END_REF] (Figure 12-a). It consists of a set of moving characters. The dataset used for the learning phase contains the representation of 26 characters from the roman alphabet and ten digits (A-Z, 0-9). The dataset used for the testing phase contains 12 representations of each character, in [A-Z] and [0-9]. The goal is to identify the character or the digit. For this experiment, we tested our method on a three layer architecture, we set empirically the number of elementary time-surfaces for each layer to be respectively:

N 1 = 6, N 2 = 9, N 3 = 12.
The neighborhood radius is empirically set to R = 2 throughout the whole architecture, while the integration times for the exponential decays are set to : τ 1 = 10ms, τ 2 = 15ms and τ 3 = 20ms. We were able to obtain 100% of accuracy in classification, both with the Euclidean and the Bhattacharyya distance.

. .

Flipped card deck

The second dataset we used is the "Flipped card deck" dataset, provided by Linares-Barranco et. al [START_REF] Pérez-Carrasco | Mapping from Frame-Driven to Frame-Free Event-Driven Vision Systems by Low-Rate Rate Coding and Coincidence Processing-Application to Feedforward ConvNets[END_REF] (Figure 12-b). It consists of a deck of cards whose corner is flipped in front an event-based camera so that only the suit symbol of the card is visible. The goal is to identify the cards' suit. The original dataset contains 40 samples, 10 from each suit. The learning and testing datasets were generated by randomly taking respectively 7 and 3 examples of each suit from the original dataset. For this experiment, we tested our method on a three layer architecture, with the same parameters used for the letters and digits experiment. We were able to obtain a 100% accuracy. We then decreased the number of features for each layer, in order to see the impact of such a modifications. For (N 1 , N 2 , N 3 ) = [START_REF] Amir | Cognitive computing programming paradigm: a corelet language for composing networks of neurosynaptic cores[END_REF][START_REF] Barbakh | Online clustering algorithms[END_REF][START_REF] Barranco | Contour motion estimation for asynchronous event-driven cameras[END_REF], the recognition rate drops to 83%, while decreasing the number of generated spikes by a factor of 2.5 (see Table 1). . This work exposes a new approach for learning spatiotemporal features from an event based camera. It allows a compact representation of information by reducing the number of prototypes in a hierarchical learning structure using sparse coding basis decomposition. This reduction induces an increase in the number of events generated by the system. Unlike the original model, we have shown that it is possible to reduce the number of prototypes without strongly impacting the precision. Our model increases the activity of each cell. This, however, is not a limitation since the available neuromorphic hardware can handle a large number of spikes (signal constraints) despite being limited to a small number of neurons (hardware constraints). Limitations are then shifted to maximal firing rate and bandwidth, not to mapping constraints due to a lack of available neurons. It is important to emphasize that we assumed a floating point precision, no consideration was given to precision limitations. Future work will focus on characterizing a loss of precision, both in number representation and in basis projection errors. Moreover, the addition of a refractory period to each prototype, in order to limit the firing rate of a single cell, will be studied. All of this could lead to a full hardware implementation. (f,g)

All the projection coefficients are split between positive and negative values. The projection coefficients are convolved with a linear decay, and the results are used to determine the spike times t out of the next emitted events ev out (cf. eq 15).

(h) Timestamps are sorted in ascending order and convolved with an exponential decay of integration time τ The same process can be repeated for the next layers, building in this way the proposed hierarchical model. Inspired from [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF] .

With the increase of multi-electrode array size, spike sorting algorithm are often overwhelmed by the quantity of data to process. It is then impossible to have meaningful results in real-time for applications such as closed loop experiments. Furthermore, there is almost always the need of an external human operation to control and validate results. Here, we show that neuromorphic computation can yield to very good results in real-time unsupervised spike-sorting tasks. Our results demonstrate that considering the time as the most valuable information in signals helps extracting coherent information from noisy data. Comparison between the proposed method and state-of-the-art algorithms shows that event-driven computation allows to significantly reduce computation time while increasing efficiency, reaching up to 90% of recognition rate on real data. We anticipate this work to open new horizons for embeddable real-time devices for closed-loops applications and low-cost performance analysis of in-vivo data.

. State of the art spike sorting methods relies on prior knowledge of the dynamic of the signal for discriminant features extraction, and aren't suitable for real-time applications. Thus, these methods, regardless their efficiency, couldn't be considered as bio-inspired ones. This work aims to show that it is possible to extract, in real time, the dynamic of the signal. It has been shown that an event-based approach for pattern recognition [START_REF] Chandrapala | The generative adaptive subspace self-Organizing Map[END_REF][117] [START_REF] Cohen | Skimming digits: neuromorphic classification of spike-encoded images[END_REF][59] [START_REF] Afshar | Racing to learn: statistical inference and learning in a single spiking neuron with adaptive kernels[END_REF] can yield to a better conceptualization of each new piece of information, with respect to past activity, both in spatial and temporal neighborhood. In this work, we will present a hierarchical event-based approach for spike sorting applications.

Spike sorting algorithms mainly rely on three steps : spike detection, feature extraction and classification. The first step often employs an automatic spike detection method (threshold detection in most of the case [START_REF] Rossant | Spike sorting for large, dense electrode arrays[END_REF]). During a second step, a set of features is computed, for example using mixture of Gaussian kernels (Khadir et Al. [START_REF] Kadir | High-dimensional cluster analysis with the masked EM algorithm[END_REF]), wavelets transformation [START_REF] Quiroga | Unsupervised spike detection and sorting with wavelets and superparamagnetic clustering[END_REF] or PCAs [START_REF] Rossant | Spike sorting for large, dense electrode arrays[END_REF]. Then, the extracted features are assigned to cell types by learning algorithms, such as (un)supervised clustering. In this chapter, I will first show a novel approach for features extraction, before explaining how it can be used for spike detection, allowing the whole pipeline to work in real-time. The performances of the algorithm were tested both with a synthetic benchmark [START_REF] Wild | Performance comparison of extracellular spike sorting algorithms for singlechannel recordings[END_REF] and in-vivo recordings [START_REF] Henze | Simultaneous intracellular and extracellular recordings from hippocampus region CA1 of anesthetized rats[END_REF], and then compared to state-of-the-art algorithms [START_REF] Rossant | Spike sorting for large, dense electrode arrays[END_REF][START_REF] Kadir | High-dimensional cluster analysis with the masked EM algorithm[END_REF][START_REF] Quiroga | Unsupervised spike detection and sorting with wavelets and superparamagnetic clustering[END_REF].

. Here I detail how, from an analog signal, the proposed algorithm generates events, extracts features, clusters them in a hierarchical model, and then classifies unknown examples.

. .

Model description

Level We will here give an overview of the proposed algorithm. Detailed explanations and equations can be found in the Method section.

When the input signal changes, our algorithm asynchronously generates events forming a spatio-temporal point cloud representing the signals' dynamical behavior. The sampled signal is first sent into a spike generator to (artificially) convert it in an event-based signal. As in silicon retinas [START_REF] Posch | A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[END_REF], the variation of the signal triggers an event generation. In a more detailed approach, the sampled signal if first filtered by a high-pass, 3 rd order, Infinite Impulse Response (IIR) filter for DC filtering and low frequencies noise rejection. Then, the filtered signal is compared to a bank of levels. Each time the signal crosses a level, a spike is triggered, both positive if the signal is increasing while crossing the level, or negative if the signal is decreasing (Figure 15-a). A linear interpolation between two samples is used for better time precision. Each event ev(i) can be seen as the following triplet :

ev(i) = {t i , c i , p i } , i ∈ N ( 17
)
where t i is the timestamp of the event, c i its channel (the crossed level) and p i its polarity, p i ∈ {-1, 1} (i.e. positive or negative).

Considering a stream of events, we can associate for each event a description of its spatio-temporal neighborhood. This description will be referred as context, generated by convolving an exponential kernel (Figure 15-b) to the most recent activity on the surrounding of the incoming event (Figure 15-c). These contexts are then clustered using an online iterative clustering method [START_REF] Barbakh | Online clustering algorithms[END_REF]. Once this learning phase done (i.e. the centers are determined and represent the dynamics of the learned signal), we extract the response of the network for different spikes. Then, events from unknown spikes are fed in the network, and their signatures compared to the learned ones for classification.

Input data is presented to the network. Contexts are built with the convolution with an exponential kernel of time constant τ 1 and considering a spatial neighborhood of side length (2R 1 + 1). Then, these contexts are clustered into N 1 centers. When a cluster is matched, an output event is produced, constituting the output of Layer 1. This output is of the same type as its input, as shown in Equation 17and 24. Thus, the same process applied to Layer 1 can be applied to layer 2 using different parameters for space-time integration (τ 2 , R 2 , N 2 ).

As stated before, each layer l is only characterized by 3 parameters : -τ l , the time constant of the exponential kernel, -R l , the size of the neighborhood, -N l , the number of centers to be learned by the clustering algorithm. The output of the last layer can be used as a feature for shape classification. The training of the recognition algorithm consists in two distinct steps. In the first one, learning data is presented in order to learn the centers computed as described in the previous section. Then, the same inputs are provided to the network and a histogram of centers activation (signatures) is built for each different class, using rather TSNE projection or offline KMeans clustering. These histograms represent the number of centers activated during the presentation of the example, independently of its spatial position, and are discriminant enough to allows proper classification.

. . Event generation

The sampled signal (48 kHz for the benchmark, 20 kHz for ex-vivo recordings) is first sent into a spike generator to (artificially) convert it in an eventbased signal. Then, the sampled signal if filtered by a high-pass, 3 rd order, Infinite Impulse Response (IIR) filter for DC filtering and low frequencies noise rejection. Then, the filtered signal is compared to a bench of levels.

Each time the signal crosses a level, a spike is triggered, either positive if the signal is increasing while crossing the level, or negative if the signal is decreasing (Figure 15-a). A linear interpolation between two samples is used for better time precision. Each event ev(i) can be seen as the following triplet :

ev(i) = {t i , c i , p i } , i ∈ N ( 18
)
where t i is the timestamp of the event, c i its channel (the crossed level) and p i its polarity, p i ∈ {-1, 1} (i.e. positive or negative).

. . Feature extraction and clustering

A spatio-temporal point cloud is formed with these events, representing the spike spatial distribution and dynamic behavior. Because this point cloud may contain information about the spikes dynamic and amplitude, we introduced the event context S i of the event ev(i) to convey information about surrounding activity just before time t i . Let N(c i ) be a windowing neighborhood with length 2R + 1 (in channels) around the event's channel c i :

N(c i ) c = {c i + c | abs(c) R} (19)
Then, this event's context S i is defined as :

S c i = exp t i -t c ref τ for {abs(c) R} (20)
where t c ref is the last spike time at the c th channel. When a context has been computed, it is compared to a bank of context, or centers. The most-closely matching center will then generate an output event. First, we do have a set of N initial contexts m k , k ∈ [ [1, N]] where m k takes the same form as S i in Equation 20. These contexts are initialized with the first N incoming contexts. More formally,

m k = S k k ∈ [[1, N]]
Then, they are clustered using an online Iterative Inverted Weight Kmeans (IIWKmeans, [START_REF] Barbakh | Online clustering algorithms[END_REF]). This clustering method was preferred to KNNs [START_REF] Cover | Nearest neighbor pattern classification[END_REF] or KMeans [START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF] because of its independence towards the initial conditions and its ability to work online. For each incoming context S, we define its nearest center m k * :

m k * = argmin k (||S -m k ||) (21)
where m k is the k th center. The update rule is then shown by:

m k * (t s + 1) = m k * -ζa ik * (S -m k * ) (22)
with:

a ik * = -(n + 1)||x i -m k * || n-1 -n||x i -m k * || n-2 j =k * ||S -m k * || (23)
ζ and n are two learning parameters able to constrain the update rule.

Once the learning is over (i.e. all the centers converged), each context can be associated with a particular center m k . So, the input stream of events is transformed into an output stream of centers' activation :

ev out = [c i , t i , k i ] (24)
where k i is the index of the matching center m k * . At this point, for noise (isolated context) rejection, it is possible to implement context rejection based on some thresholding among the distances and so prevent emitting a new event if the match between the context and the center was not strong enough.

. . Classification

The recognition by itself is done by comparing online signatures to the trained ones. The distance between two histograms (H 1 , H 2 ) used here is the Bhattacharyya [START_REF] Bhattacharyya | On a measure of divergence between two multinomial populations[END_REF] distance :

d b (H 1 , H 2 ) = -log i H 1 (i) card(H 1 ) H 2 (i) card(H 2 .
. .

Metrics

Noise level

The noise level n l was defined as the reciprocal value to the signal-to-noise ratio (SNR) [START_REF] Wild | Performance comparison of extracellular spike sorting algorithms for singlechannel recordings[END_REF] :

n l = SNR -1 = A signal A noise -1
where A signal is the root mean square value from all the extracted spikes, and A noise the one for the rest of the signal.

Performance rating

To compare our method to state of the art spike sorting algorithms, we used the Adjusted Mutual Information. This measure provides information about how good the clustering is. Unlike the recognition rate (trace of the confusion matrix divided by the sum of all the elements within the confusion matrix), this measure is adjusted by chance, meaning that a zero value stands for chance and a 1 value for a perfect classification. Details and properties can be found in [START_REF] Vinh | Information theoretic measures for clusterings comparison: Variants, properties, normalization and correction for chance[END_REF].

. . Benchmarking

The proposed algorithm was tested with a database of artificially generated data from simulating extracellular signals recorded with a single electrode. Nine real spikes where manually picked from extracellular tungsten micro-electrode recorded during Deep Brain Stimulation operation from the sub-thalamus nuclei. They are then corrupted with noise coming from a superimposition of activity of many distant neurons in the brain [START_REF] Wild | Performance comparison of extracellular spike sorting algorithms for singlechannel recordings[END_REF].

Results

In this section, we will present results of the proposed algorithm on the benchmarking database [START_REF] Wild | Performance comparison of extracellular spike sorting algorithms for singlechannel recordings[END_REF]. This database contains 32 sets of 9 artificially generated spikes, corrupted with noise. Figure 18-a shows the response among the centers of the fifth and last layer. The dynamic of the signal is well captured and clusters respond to local patterns of the input signal. Using high dimensionality reduction (TSNE, [START_REF] Maaten | Visualizing data using t-SNE[END_REF]) for data visualization, the projection of the signatures for some testing examples in two dimensions shows the ability of the algorithm to distinguish different types of spikes (Figure 18-b). Figure 17 presents the AMI score for the presented algorithm compared to state-of-the art methods. It has to be taken in account that the only previous method working in real time is the OSort algorithm. The HotsL3 and HotsL5 presented here are for the output of the third and fifth layer of the hierarchical model. The classification score after layer 5 overcomes all the other methods. After layer 3, only for intermediate noise level (0.15 -0.30) are we slightly below Waveclus. For noisy dataset (0.45 and more), the proposed method overcomes drastically all the others. This shows the very good immunity of the proposed method toward noise. Regarding the required time to process all these data, our algorithm performs real time for 1-3 layers on a standard desktop computer (Intel Core i7-4790 CPU @ 3.60GHz, 16GB of RAM, running Debian 8.5 Jessie). Figure 20 represents the computation time for 1-5 layers. Good results on semi-artificial dataset lead us to benchmark the proposed method on real data. Buzaki's lab developed new techniques for simultaneous recordings of in-vivo (patch clamp) and ex-vivo (multi electrode array) data [START_REF] Henze | Intracellular features predicted by extracellular recordings in the hippocampus in vivo[END_REF][START_REF] Harris | Accuracy of tetrode spike separation as determined by simultaneous intracellular and extracellular measurements[END_REF]. Data from the CA1 hippocampal region of anesthetizided rats [START_REF] Henze | Simultaneous intracellular and extracellular recordings from hippocampus region CA1 of anesthetized rats[END_REF], containing both intra and extracellular recordings, was used. The extracellular tetrodes [START_REF] Recce | The tetrode: a new technique for multiunit extracellular recording[END_REF] are made of four 13-um polyimide-coated nichrome wires. The intracellular glass micropipette filed mainly with potassium solution. The extracellular data contains responses from two different cells (type A and type B), whereas the in-vivo data target only one (type A). So, the intracellular data can be used as a ground truth to validate the sorting on extracellular data. Without fine tuning of the parameters, the use of the same parameters as in previous section leads up to 90% recognition rate.

Extension to in-vivo data

The dataset was split into 10 subsets, randomly picked, with around 30 % of type 1 cell. One subset is used for learning the clusters, and the 9 others to test the algorithm. The experience was run for 100 times. Table Table 3 shows the results of these runs. Our algorithm outperforms state-of-the art real-time spike, and is slightly lower than the best method reported on this dataset, which is not real-time. Computation time (on a Intel Core i7-4790 CPU @ 3.60GHz, 16GB of RAM, running Debian 8.5 Jessie) is about 25 seconds for one set of 80 seconds of data, which is more than 3 times faster than the real-time. Figure 20 shows the computation time versus the number of layer. We can see that the proposed method is able to handle in real-time a 3-layer architecture. .

We showed that a paradigm shift in the data representation and a fully event-based hierarchical processing pipeline can lead to very good recognition rates. As stated in [START_REF] Lagorce | Hots: a hierarchy of event-based time-surfaces for pattern recognition[END_REF], the hereby used "event-based spatiotemporal context" is speed dependent. This may be a problem for pattern recognition, where the task is to recognize a given shape, presented in various poses and motion, but helps us for cellular recordings, as each neuron fires with a spe- .

In the framework of this article, the input signal is first sampled and then converted into events. Intrinsically, this approach cannot reflect the signal's exact dynamic. The linear interpolation was a first step toward time precision, but we are still limited by the sampling frequency. This work was achieved during my first year of PhD. Meanwhile, Corradi et Al. [START_REF] Corradi | A neuromorphic event-based neural recording system for smart brain-machine-interfaces[END_REF] developed an event-based Multi Electrode Array (MEA), which directly converts the spike voltage in events. The BioAmp chip was developed at INI (Institute of Neuroinformatics, University of Zurich and ETH Zurich). As the only available part, at the beginning of the project, was the die, I had to design, in collaboration with Federico Corradi (INI) a PCB to bond the die on, and an interface board to connect the sensor to the FPGA for readout and control operations. The bonded die is showed in Figure 21. This chapter describes a fully spike-based neural network for optical flow estimation from Dynamic Vision Sensor data. A low power embedded implementation of the method which combines the Asynchronous Time-based Image Sensor with IBM's TrueNorth Neurosynaptic System is presented. The sensor generates spikes with sub-millisecond resolution in response to scene illumination changes. These spike are processed by a spiking neural network running on TrueNorth with a 1 millisecond resolution to accurately determine the order and time difference of spikes from neighboring pixels, and therefore infer the velocity. The spiking neural network is a variant of the Barlow Levick method for optical flow estimation. The system is evaluated on two recordings for which ground truth motion is available, and achieves an Average Endpoint Error of 11% at an estimated power budget of under 80mW for the sensor and computation.

. Estimation of optical flow is a computationally intensive task. Modern artificial systems achieve ever higher accuracy on the task at the expense of more complex algorithms and more powerful computing hardware such as GPUs. On the other hand, many biological systems are able to estimate optical flow both effectively and power-efficiently. Examples include insects such as bees and drosophilia, which rely heavily on optical flow for navigation [START_REF] Joesch | ON and OFF pathways in Drosophila motion vision[END_REF], but must estimate optical flow under severe size, weight, and power constraints. While the optical flow estimated by such insect may not rival modern computer vision approaches in terms of accuracy, the estimated flow is good enough for the insects to use in navigation, and impressive considering the tight constraints under which the estimates were generated.

In artificial systems, early optical flow estimation approaches relied on methods such as velocity-tuned filters in the frequency domain [START_REF] Watson | A look at motion in the frequency domain[END_REF][START_REF] Heeger | Model for the extraction of image flow[END_REF], phase-based methods [START_REF] Barron | Performance of optical flow techniques[END_REF], gradient based methods, and correlation based methods [START_REF] Sutton | Determination of displacements using an improved digital correlation method[END_REF]. The accuracy of these methods for optical flow estimation on sequences of images has long since been surpassed by even more computationally intensive methods which estimate flow at multiple scales [START_REF] Alvarez | A scale-space approach to nonlocal optical flow calculations[END_REF], and more recently by convolutional neural networks [START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF], which are mostly running on power inneficient GPUs or dedicated ASICs, which are less accurate, more general, but are more power efficient (PX4Flow [START_REF] Honegger | An open source and open hardware embedded metric optical flow cmos camera for indoor and outdoor applications[END_REF], optical mouse). Direction selective neurons are found as early as the retina in frogs [START_REF] Lettvin | What the frog's eye tells the frog's brain[END_REF], cats [START_REF] Barlow | The mechanism of directionally selective units in rabbit's retina[END_REF], rabbits, primates, and many other animals.

A key difference between optical flow estimation in biological and artificial systems lies in the format of the data from which optical flow is extracted. Both approaches start off with light from a scene focused on an image sensor, but most artificial systems convert this light signal into a sequence of static digital images from which optical flow must later be estimated. The retina works on a different principle, transducing the light signal into a continuous electrical signal, and later into discrete spikes for communication to the lateral geniculate nucleus and downstream visual cortex.

Not all artificial methods rely on static images though. Tanner and Mead [START_REF] Tanner | An integrated analog optical motion sensor[END_REF] introduced the first analog two dimensional optical flow chip, extracting a global motion from a given scene on dedicated hardware. Delbruck [START_REF] Delbrück | Silicon retina with correlation-based, velocity-tuned pixels[END_REF] also implemented such a network in VLSI. More recently, with the new event-based cameras, some spike based techniques have been proposed [START_REF] Benosman | Asynchronous frameless event-based optical flow[END_REF], but they require a dedicated computer for processing. An extensive review of motion estimation designs can be found in [START_REF] Orchard | Bioinspired visual motion estimation[END_REF].

Over the last decade, so-called silicon-retinae have matured to a level where they are now publicly available (described later in Section 5.2.2). These bio-inspired devices provide a spiking output more similar to the biological retina. Recently many papers have proposed different methods for processing data, including for estimating optical flow. Benosman et al. [START_REF] Benosman | Asynchronous frameless event-based optical flow[END_REF] proposed a plane fitting approach which estimates the motion of sharp edges. Later, Barranco et al. [START_REF] Barranco | Contour motion estimation for asynchronous event-driven cameras[END_REF] proposed a method which also estimates flow at edges, but their method estimates the magnitude of the spatial and temporal image gradients at the edge and then relies on a gradient-based method for optical flow estimation. Bardow et al. [START_REF] Bardow | Simultaneous optical flow and intensity estimation from an event camera[END_REF] apply a variational method which simultaneously estimates both the image grayscale values and optical flow from events. By enforcing spatial and temporal smoothness constraints, their method generates optical flow estimates even for image regions where no gradient information is available. Some optical flow estimation methods take bio-inspiration a step further and make use of biologically inspired computation using spiking neurons to extract optical flow. Examples include Brosch et al. implementing a phase based method [START_REF] Brosch | Event-based optical flow on neuromorphic hardware[END_REF], Conradt et al. implementing a Reichardt detector [START_REF] Richter | Bio-inspired optic flow detection using neuromorphic hardware[END_REF], and Orchard et al. [START_REF] Orchard | A spiking neural network architecture for visual motion estimation[END_REF] who simulated a method relying on synaptic delays.

In this chapter we propose a Spiking Neural Network variant of the Barlow & Levick model [START_REF] Barlow | The mechanism of directionally selective units in rabbit's retina[END_REF], using a silicon retina coupled with IBM's TrueNorth Neurosynaptic System.

The method exploits precise spike timing provided by the silicon retina to reliably extract motion direction and amplitude from a scene in real-time. Such methods promise to allow for low power visual motion estimation in real-time. TrueNorth is estimated to consume 70mW and the silicon retina consumes approximately 10mW (chip only, omitting FPGA for communication, which can be removed for dedicated applications). This setup can then easily be used in embedded devices such as drones or autonomous driving. Section 5.2 introduces both the model, IBM's TrueNorth Neurosynaptic System, and the silicon retina. Details of the model implementation are given in Section 5.3. Section 5.4 describes how the model was tested, and results of testing are described in Section 5.5, followed by discussion in Section 5.6. .

. . Direction Sensitive (DS) Unit

The basis of the network, introduced in [START_REF] Giulioni | Eventbased computation of motion flow on a neuromorphic analog neural platform[END_REF], is described in Fig. 23. The original model, inspired by the neural circuitry found in the rabbit's retina by Barlow and Levick [8] is based on inhibition-based direction-sensitive units, combined in motion detectors. Fig. 23 shows a simplified Direction Selective (DS) neuron which responds (spikes) to the detection of motion in its preferred direction (rightward). The DS neuron (gray) receives inputs from two neighboring ATIS pixels (bipolar cells). Each ATIS pixel (orange and blue) will generate an output spike when stimulated by the passing of an edge. For motion in the preferred direction (rightward), the edge will pass over the orange pixel first, and the blue pixel second.

When the edge passes over the orange pixel, a spike is generated and the DS neuron is excited. The excitatory input triggers a burst of spikes from the DS neuron. The burst continues until the edge passes over the blue pixel, at which point the blue pixel will generate an output spike which inhibits the DS neuron, causing the spike burst to end. The time-length of the spike burst encodes the time taken for the edge to pass from the orange pixel to the blue pixel, thus providing information on the edge velocity in the preferred direction.

For motion in the opposite direction (leftward) the inhibition from the blue pixel will arrive before the excitation from the orange pixel. Due to the initial inhibition, the later excitation from the orange pixel is not sufficient to drive the DS neuron membrane potential above threshold, and thus the DS neuron does not spike in response to leftward motion.

The full direction is obtained with a combination of four single DS units, as shown in Fig. 24.

In implementation, there is a possibility of receiving an isolated noise spike from the orange pixel, which would cause the DS neuron to begin bursting, and continue bursting indefinitely. To handle this case, a delayed copy of the spikes from the orange pixel are used to inhibit the DS neuron, thus limiting the maximum burst length to the length of delay used. A stimulus from left to right first excite (yellow cell) the output (gray), which is self-excitatory until it gets an strong inhibitory input (blue cell), as a movement in the other direction (right to left) doesn't produce any output. Thus, this unit is directive to a specified direction. More, the timing between the beginning and the end of the output spiking give an information of the (inverse) velocity. The excitatory input also copies a spike to the delay unit, to inhibit the DS cell in case the natural inhibition doesn't spike. Here, the delayed inhibition doesn't have any impact. Inspired by [START_REF] Giulioni | Eventbased computation of motion flow on a neuromorphic analog neural platform[END_REF].

. . Event-based Sensor

Conventional image sensors sample the visual scene at fixed temporal periods (framerate). All pixels acquire light in a synchronous fashion by integrating photons over a fixed time-period. When observing a dynamic scene, this framerate, no matter its value, will always be wrong because there is no relation whatsoever between the temporal dynamics of the scene and the acquisition period. This leads to simultaneous over-sampling and undersampling of different parts of the scene, motion blur for moving objects, and data redundancy for static background. Event-based sensors [START_REF] Posch | Retinomorphic event-based vision sensors: bioinspired cameras with spiking output[END_REF], also known as silicon retinae, are an alternative to fixed-frequency imaging devices. In these devices, a time-varying signal is sampled on its amplitude-axis instead of time-axis, leading to a non-uniform sampling rate that matches the temporal dynamics of the signal, as shown on Fig. 22. The Asynchronous Time-based Image Sensor (ATIS [START_REF] Posch | A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[END_REF]) is an QVGA array of asynchronous independent pixels, each combining a change detection unit and an absolute gray level measurement unit. Each time a pixel detects a significant change in luminance in its field of view, an event is generated. Each event can be represented as a triplet ev i = (x i , t i , p i ), ev i being the i -th event, where x i is the event spatial coordinates, t i its timestamp and p i its polarity, indicating if the change is an increase or decrease in luminance. In this work, only the change detection unit is used.

. . The TrueNorth Environment

A TrueNorth Chip [START_REF] Merolla | A million spiking-neuron integrated circuit with a scalable communication network and interface[END_REF] consists of a network of 64×64=4096 neurosynaptic cores with programmable connectivity, synapses, and neuron parameters. Connectivity between neurons follows a block-wise scheme. Each core has 256 input axons, with programmable connectivity to any of the 256 neurons in that core. Each neuron's output can be routed to a single axon anywhere on the chip. All communications to, from, and within chip are performed asynchronously [START_REF] Merolla | Deep neural networks are robust to weight binarization and other non-linear distortions[END_REF].

Each TrueNorth neurosynaptic core is made of 256 axons, 256×256 synapse crossbar and 256 neurons (Fig. 25 A). In this chapter, the NS1e board was used, containing 4096 cores, 1M + neurons, 268M + synapses, embedded in a board with a Zynq FPGA containing an ARM-core (Fig. 25 B). .

TrueNorth implementation of the model was achieved by carefully configuring a single tile-able TrueNorth core, thanks to the tools presented in [START_REF] Amir | Cognitive computing programming paradigm: a corelet language for composing networks of neurosynaptic cores[END_REF], to handle a small local region of the input space, and then tiling enough copies of these cores to cover the full input image.

The TrueNorth environment neuron's behavior is as following [START_REF] Cassidy | Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores[END_REF] :

Synaptic integration V t = V t-1 + N i=0 ω i x i (t) Leak V t = V t -l Threshold if V t > Θ spike V t = V r
where V t is the value of the membrane potential at timestep t, ω i the synaptic weight for the i -th synapse, l the leak, Θ the threshold, V r the reset potential, x i (t) the spike input to the synapse at time t and N the number of synapse for the considered neuron. The full neuron equations and parameters can be found in [START_REF] Cassidy | Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores[END_REF], figure 4. The leak is configured to, unless otherwise stated, leak towards zero. This behavior is detailed in [START_REF] Cassidy | Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores[END_REF].

Within each core, neurons are divided into three main modules, shown in different colors in Fig. 26. The first module (red) receives input spikes from ATIS and generates a copy of these spikes which can be used for further processing by the other two modules. The second module (blue) implements the delay required to generate the delayed inhibition for the DS neurons. The third module (green) implements the DS neurons. Each module is described further below. . .

Input Module

The input stage provides a copy of the input spikes which serve as inputs to the DS neuron module and delay module. The input stage is also used to enforce a refractory period which limits the minimum allowable time between two input spikes from the same pixel by blocking the second spike if it falls within τ r of the first.

A refractory period of τ r (ms) is implemented using a refractory neuron model (see Table 4). When at rest state (membrane potential of zero), the neuron should fire an output spike as soon as an input spike is received. This is achieved by enforcing

w e + l Θ (25)
where w e is the excitatory synaptic weight. After spiking, the neuron resets to a large negative voltage and slowly leaks back to zero, thus implementing a refractory period. The desired refractory period is achieved by setting the reset and leak values such that

τ r ≈ V r l V r = -(2 n -1), n ∈ [[0, 15]] (26)
where τ r is the desired refractory period, V r is the reset potential, and n the quantization of the desired value, which can only take on limited values due to constraints of the chip (here 15 bits). Input spikes received while the neuron potential is leaking towards zero will not cause an output spike, but they will shorten the refractory period. The actual refractory period achieved, τ r , is given by:

τ r = V r +w e s l = τ r + w e s l ( 27
)
where s is the number of spikes received during the refractory period τ r . To ensure that the achieved refractory period τ r is as close as possible to the desired refractory period, τ r , the term w e s/l must be minimized. The minimum value which still satisfies equation 25 occurs when l = 254, w e = 255.

Connectivity on TrueNorth is bound by certain constraints, one of which is that a spike can only be routed to one core. This constraint can be overcome by implementing multiple identical neurons on the same core. Each neuron will then generate an identical output, thereby providing multiple copies, each of which can be routed to different cores.

In the case of our TrueNorth model, there are four co-located DS neurons at every pixel location, and they each require input from one of the four neighboring pixels. Thus, the spikes from some pixels serve as input to neighboring DS neurons which reside on different cores. The input stage takes care of generating copies of the spikes from these pixels and routing them to the neighboring cores.

Each core only sends spikes to its neighboring cores lying to the South and East, and each core receives input spikes from its neighboring cores in the North and West.

The number of neurons required by the input module is therefore

N in (∆x, ∆y) = ∆x∆y + ∆x + ∆y, (28)
where ∆x and ∆y specify the pixel size of the image region in x and y direction processed by a single core. The input module implementing refraction is shown in red in Fig. 26.

. . Delay Module

The delay module implements the delay required by the delayed DS neuron inhibitory input. Delays of τ d milliseconds (here, τ d = 50ms) are implemented using the Delay neuron model of Table 4.

When at rest (zero membrane potential), the neuron will remain at rest until an input spike is received. An input spike will push the membrane potential above zero, and once above zero the neuron membrane potential will leak towards a positive threshold. The delay is implemented by the time taken to leak to the positive threshold. Once the threshold is reached, an output spike is emitted, and the neuron will be reset back to the rest state.

A disadvantage of this approach is that if two input spikes arrive from the same pixel at times t 1 and t 2 such that t 2 -t 1 < τ d , only a single delayed output spike will be generated, and the output spike will be generated with delay τ d -1 milliseconds after t 1 . To avoid such complications, we use the refractory period from the input stage to limit the probability of having two spikes occur close together. More specifically, we choose τ r such that τ r > τ d .

All input spikes from the region covered by the core must be delayed, and copies of the delayed spikes at the South and East boundaries must be generated for neighboring cores. Thus the total number of neurons required for the delay module, N delay (∆x, ∆y), is

N delay (∆x, ∆y) = N in (∆x, ∆y) = ∆x∆y + ∆x + ∆y. (29)
The delay module (blue in Fig. 26) receives inputs from the refractory input module (red), as well as input from the refractory input modules of the cores to the North and West. The delay module generates outputs which feed back to the same core, as well as output for use in processing of the cores to the South and East.

. .

DS Module

The DS module holds the DS neurons. The parameters for each neuron are shown in Table 4. There are four DS neurons per image location, so a core covering an image region of size ∆x by ∆y pixels will require

N DS (∆x, ∆y) = 4∆x∆y (30)
neurons.

The output neuron of the DS module should be self-excitatory until it receives a strong inhibitory spike, as shown in Figure 23. This behavior was achieved by setting the reset potential V r to be higher than the threshold Θ.

Then, when the membrane potential crosses the threshold, the cell enters a self-excitatory state until a strong inhibition event is received.

Sometimes, due to noise, the inhibitory spike can be missing. To prevent a cell to spike indefinitely, and second strong inhibition spike is generated and delayed (typically 100ms) as the excitation spike is generated. So, if the inhibition is not received, this delayed spike will reset the neuron, and thus putting it in a steady state. It has to be recalled that not all the noise events will trigger a flow computation, as a valid flow sequence consists of two spikes in the same neighborhood.

The DS neurons are shown in green in Fig. 26 and receive input from both the input refractory module, and the delay module of the current core, as well as from the cores to the North and West.

. . Parameters

TrueNorth neurons are a variant of linear leaky integrate and fire model with 23 tunable parameters, as mentioned in [START_REF] Cassidy | Cognitive computing building block: A versatile and efficient digital neuron model for neurosynaptic cores[END_REF]. Table 4 contains the neuron parameters for each population.

The total number of neurons N required by a core is N (∆x, ∆y) = N in (∆x, ∆y) + N delay (∆x, ∆y) +N DS (∆x, ∆y)

= 6∆x∆y + 2∆x + 2∆y N (∆x, ∆y) 256. (31)
The values ∆X = 6 and ∆Y = 6 maximize the image area processed by the core, subject to the constraint that the core only has 256 neurons. The number of axons used is less than the number of neurons, so neurons are the limiting factor.

285 identity cores are used to relay the ATIS input to the motion model (see Section 5.3.6). For the motion model to cover the 304×240 pixel input image requires another 304/6 × 240/6 ≈ 51 × 40 = 2040 cores. In total the TrueNorth model uses 2040 + 285 = 2325 cores to compute motion at all locations in the input image space. 240 neurons are used per core (a usage of 256 neurons per core is achievable if ∆X = 18 and ∆Y = 2, but each core would still only be processing 36 pixels, and thus such an arrangement is less efficient).

. . Interpreting the Result

To interpret the output spikes from the network at a particular pixel location, the outputs of all four of the DS neurons located at the pixel are combined. The same input spike will excite all four DS neurons, so they should all start bursting simultaneously (unless they are already inhibited). We use the notation t x + to denote the length of the burst from the neuron sensitive to motion in the positive x-direction. The velocity can then be calculated using

t x = (t x + -t x -) /δ x t y = t y + -t y -/δ y v x = t x / t 2 x + t 2 y v y = t y / t 2 x + t 2 y ( 32
)
where δ x and δ y represent the inter-pixel distance in the x and y direction respectively (in our case, δ x = δ y = 1). v x and v y indicate the component of velocity in the x and y directions respectively, in units of pixels per millisecond. If the inhibition is missing, the flow computation is discarded.

. . ATIS-TrueNorth link

The ATIS and TrueNorth chips both use variants of the Address Event-Representation (AER) protocol [START_REF] Boahen | Point-to-point connectivity between neuromorphic chips using address events[END_REF]. However, the two are not directly compatible, so the Opal Kelly XEM6010 board with Xilinx Spartan 6 FPGA which powers and configures ATIS is also used to convert the ATIS AER signals to TrueNorth AER signals.

The ATIS AER data consists of an absolute pixel address and a polarity, all of which are communicated in parallel when the AER request line is active.

The TrueNorth AER data consists of a relative core address, an axon address, and a 4 bit target time. The data is communicated in two phases, with half the bits being communicated in parallel during each phase.

The TrueNorth reset and 1kHz clock tick signals are shared with the Opal Kelly board so that it can keep track of the state of TrueNorth's internal 4 bit time. All events are communicated to TrueNorth with a target time 2ms later than the current time.

A one-to-one mapping from ATIS pixels to TrueNorth neurons is generated such that each TrueNorth core accepts events from a 16×16 pixel region, with polarity being ignored (all events are fed in the system, regardless their polarity). An array of 19×15 = 285 identity cores is instantiated at the physical core locations targeted by the ATIS interface. These identity cores generate a copy of the spikes they receive, which can then be routed to the rest of the TrueNorth model.

This interface arrangement uses an extra 285 cores, but allows the model to be rapidly changed by only reconfiguring TrueNorth, instead of having to reprogram the ATIS FPGA to target different cores and neurons for different TrueNorth models.

. . .

Sources of Visual Data

Two sources of visual data are used in this work: a recording of a black pipe rotating in front of a white background, originally presented in [START_REF] Orchard | A spiking neural network architecture for visual motion estimation[END_REF], and a recording of a rotating spiral [START_REF] Orchard | Bioinspired visual motion estimation[END_REF]. The pipe and spiral sequences are both ATIS recordings in which ground truth for motion can be predicted to quantify the accuracy of the output. The pipe and spiral recordings are shown in Fig. 27. This motion is modeled as described below.

. . Modeling Motion for the Rotating Pipe

We use a recording of just over half rotation of the pipe due to symmetry (the second half of the rotation will look almost identical to the first). A full rotation takes roughly 2.85 seconds, so we use a 1.5 second recording. The rotating pipe was modeled as having two parallel edges spaced a width of w apart from each other and rotating about a point centered between them. Motion of the pipe could then be modeled using

R(t) = cos (tω t ) -sin (tω t ) sin (tω t ) cos (tω t ) Location(l, t) = x c y c + R(t) ± w 2 l Direction(t, l) = tω t l 0 tω t + π l > 0 Speed(l) = |l|ω t l ∈ [-150, 150] t ∈ [0, π |ω t | ] ω t = 2.21rad/s (33)
where R(t) is a rotation matrix which varies with time (t) to model rotation of the pipe, ω t is the angular velocity of the pipe, Location(l, t) gives the pixel location, [x, y] T , of points on the pipe as a function of time, their position on the edge of the pipe l, and the location of the center of the pipe [x c , y c ] T . The speed of each point is dependent only on its position on the pipe l, while the direction depends on t and l because one half of the pipe is moving in the opposite direction to the other. The pipe has a finite length of 300 pixels, thereby limiting l. The recording is long enough for half a rotation of the pipe, thereby limiting t. The speed and direction given by [START_REF] Fisher | Accommodation and apparent distance[END_REF] are the components perpendicular to the edges of the pipe. The direction of the x and y axes are shown top left of Fig. 27. . .

Modeling Motion for the Rotating Spiral

The spiral stimulus does not exhibit the same symmetry as the pipe, so we use a recording of a full rotation of the spiral, which takes 0.5 seconds.

The shape of the spiral stimulus was parameterized by angle with a variable θ 0 from which the speed, direction, and location of motion can be modeled using

r(θ 0 ) = 2 θ 0 π Location(θ 0 , t) = x c y c + r(θ 0 ) cos (-θ 0 + tω t ) r(θ 0 ) sin (-θ 0 + tω t ) Speed(θ 0 , t) = 2 θ 0 π ln 2 π ω t 1 cos ( ln 2 πω t ) Direction(θ 0 , t) = -θ 0 + tω t + sin ( ln 2 πω t ) θ 0 ∈ [0, 20] t ∈ [0, π |ω t |] ω t = -12.57rad/s (34)
where r(θ 0 ) is the radial distance to the spiral from the center, [x 0 , y 0 ] T . Location(θ 0 , t) describes the [x, y] T location of the spiral edges. Speed(θ 0 , t) and Direction(θ 0 , t) give the speed and direction of motion respectively. The Cosine and Sine terms in the speed and direction equations account for the fact that the spiral is not perpendicular to the radial vector (second term in Location(θ 0 , t)).

. . Error Metrics

We use three main metrics to evaluate the results. Average Angular Error (AAE), Average Endpoint Error (AEE), and density estimation.

The first two error metrics (AAE and AEE) rely on ground truth motion being available from [START_REF] Fisher | Accommodation and apparent distance[END_REF] and [START_REF] Frenkel | mm2 9.8-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28nm CMOS[END_REF]. Sometimes sensor noise spikes result in flow being estimated where there is no image motion. The true flow is assumed to be zero for all locations which do not lie on an edge of the bar or spiral. For such locations, the error is equal to the flow estimate itself.

AAE is simply the average absolute error in angle between the computed flow V = (v 1 , v 2 ) and the ground truth Gt = (gt 1 , gt 2 ).

AAE = ( V, Gt) = atan gt 2 -v2 gt 1 -v 1 (35)
AEE (equation 23 from [START_REF] Baker | A database and evaluation methodology for optical flow[END_REF]) takes into account errors in both speed and direction. It is defined as the squared root of sum of the the squared errors between the estimated vector and ground truth.

AEE = (v 1 -gt 1 ) 2 + (v 2 -gt 2 ) 2 gt 2 1 + gt 2 2 (36)
Density (d) provides a measure of the ratio of optical flow estimates generated by the model to the number of input events fed to the model. A higher density is better so long as it is not accompanied by a significant decrease in accuracy. Many event-based methods quote this metric to avoid a situation where incredibly high accuracies are obtained by discarding all but the very best flow estimates.

d = N valid N all (37)
where N valid is the number of optical flow estimates, and N all is the total number of input events.

. Output spikes are sent from the board as UDP packets. A simple UDP receiver code for visualizing the output spikes was developed that captures UDP packets, decodes them and plots the results of the flow (direction and velocity). All the error analyses were done offline. The implemented network, as presented in Section 5.3, uses as parameters ∆X = 6 and ∆Y = 6, leading to a need of 2325 cores, representing 558000 neurons. Fig. 27 shows snapshots of the input data (rotating bar and rotating spiral). Feeding this network with these input gives results shown in Figs. 28-31. Fig. 28 shows a 3D representation of the optical flow estimates arising from the spiral data. Only the estimated direction of movement is shown, indicated by hue. Fig. 29 shows the AAE for the same spiral recording, in which the direction of movement is estimate with an AAE of ¯ 8.5 deg. This error is mainly located at the ends of the bar, where no neighborhood is available, thus giving incorrect measurements. Fig. 30 shows the AEE for the spiral data. The previously described network is able to extract the optical flow for the spiral with an average AEE of 11.3%. A video of the spiral results is available online at [128].

Fig. 31 shows similar angular error for the pipe data. The plots appear truncated because the bar was not entirely in the field of view of the camera. As for the spiral, most of the errors are located at the edges of the pipe, also due to a lack of neighboring events. The AAE is 8.7 • for this dataset. A video of the pipe results can be found online at [127].

Fig. 32 shows a snapshot of the output from data captured with the sensor hand-held in the lab. We provide this example in order to show the ability of the network to extract flow in a more complex scene, but ground truth and error estimates are not available. A video of this data is available online at [126].

The percentage of flow computations triggered by noise is 3.95%. We consider a flow computation to have been triggered by noise if the time and location of the flow estimate is not predicted by [START_REF] Fisher | Accommodation and apparent distance[END_REF] or [START_REF] Frenkel | mm2 9.8-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28nm CMOS[END_REF]. .

The model presented here is capable of estimating visual motion in realtime with good accuracy, but there are some limitations.

The method described uses 57% of the cores available on TrueNorth to estimate optical flow at every pixel over the full ATIS resolution of 304×240. To extend the method to operate at higher resolutions, such as the prototype 640×480 pixel DAVIS sensor, would require either using multiple TrueNorth chips, or only computing optical flow at every second pixel in the x and y directions. Our network is able to reconstruct the velocity vector with an average error of 11%, error which is mainly located at the edges of the spiral, where it is impossible to know the speed by our method. Of the 256 neurons available on each TrueNorth core, 16 are unused. Of the 240 remaining neurons, 24 (10%) are used to copy spike signals across the core boundaries to prevent motion estimation artifacts at the core boundary. Since copying the signal across the core boundary introduces a 1 tick delay, the inputs to each core must also be delayed 1 tick before processing, which uses another 6×6=36 of the 240 used neurons. The block connectivity of TrueNorth has many advantages for parallelizing local computations in each core, but at the cost of requiring spikes to be copied between neighboring blocks, which ends up using 36 + 16 = 52 neurons per core.

Another constraint is imposed by the time resolution of TrueNorth, which is operating at a 1ms tick interval. This time quantization results in each DS unit's output speeds being quantized to 1/n, n ∈ N pixels/ms. Such quantization can result in large error for large speeds (since the quantized bins are further apart at high speed), and speeds faster than 1 pixel/ms will be detected as 0. The maximum detectable speed could be increased by placing DS input pixels further apart. A spacing of k pixels between the DS unit input pixels would result in a maximum possible speed of k pixels/ms. However, this approach would aggravate the edge correspondence problem, because it becomes less likely that the two pixels are seeing the same edge.

Another possible method for increasing the maximum detectable speed is to reduce the tick period for TrueNorth. While this is definitely possible, keeping in mind that lowering the tick period may result in problems with spike delivery (routing issues) and may increase power consumption, a maximum speed of 1 pixel/ms was deemed good enough (objects shouldn't cross the visual field in less than 304ms -i.e. sensor width-. Proper lens selection may be done according to the desired application), and reducing the tick period of TrueNorth was not explored further.

The slowest speed which can be detected is 1/τ r , set by the length of the delay, τ r . Increasing the spacing of DS pixels to k would also increase this minimum detectable speed to k/τ r . There is a trade-off between the slowest detectable stimulus, and the minimum allowable time between two subsequent stimuli. For two nearby, fast moving stimuli to each be uniquely detected and measured, their onsets must be spaced as least τ r ms apart. Thus for fast moving stimuli it is desirable to have τ r be small, but for slow stimuli, we require τ r to be large.

Spike IO bandwidth of the TrueNorth chip also poses a constraint. The TrueNorth chip has four spike IO ports (North South East West), each capable of 40k spikes per millisecond. On the NS1e development platform used in this work, all outputs spikes are communicated to a Xilinx Zynq through one port. Although 40k spikes/ms is a lot, the model presented in this chapter does generate a lot of output spikes. The worst case scenario happens in complex scenes with lots of slowly moving edges. Edge density increases the number of DS units which respond, while slow moving edges generate a lot more spikes than fast moving edges. An isolated noise event at a single pixel is a particularly bad culprit for generating spikes since it will simulta-neously activate all four 4 DS units located at that pixel, and each of these DS units will generate a train of τ r spikes before self-inhibiting.

If too many output spikes are generated, the TrueNorth chip tick period will extend to allow time for the spikes to be communicated, which will distort the estimated flow (which assumes a 1ms tick). In the case where pre-recorded data is being run through the model (instead of live ATIS data), the TrueNorth Neurosynaptic System will input spikes in accordance with the slowed down tick, thereby still allowing the model to estimate speeds accurately, but the system will run slower than real-time. On-chip spike communication is much faster than off-chip spike communication, so in future a method which interprets velocity on-chip and more efficiently communicates the results off-chip can help to mitigate the IO constraint. The model described in this chapter can be classified as a token based method, where a token is generated whenever an edge passes over a pixel. The actual gradient of the edge is never estimated, the gradient just has to be large enough to trigger a spike from the ATIS pixel. Large gradients may result in multiple spikes in response to a single edge, but the refractory period of the first stage will eliminate these secondary spikes. Much like many other event-based optical flow methods [START_REF] Giulioni | Eventbased computation of motion flow on a neuromorphic analog neural platform[END_REF][START_REF] Benosman | Asynchronous frameless event-based optical flow[END_REF][START_REF] Barranco | Contour motion estimation for asynchronous event-driven cameras[END_REF], this method only estimates flow at edges, and it estimates normal flow (optical flow normal to the edge direction). However, some event-based methods do estimate gradients for use in optical flow computation [START_REF] Barranco | Contour motion estimation for asynchronous event-driven cameras[END_REF], or use global constraints to estimate optical flow for image regions where little or no gradient is present [START_REF] Bardow | Simultaneous optical flow and intensity estimation from an event camera[END_REF]. The data used for this chapter, specifically the spiral one, were chosen because of their wide spectrum in terms of directions, the full range is covered within on rotation, and speeds, as the linear speeds depends on the radius for a constant rotational speed. Currently the output spike signals must be interpreted off chip to extract the velocity information. Since 43% of the TrueNorth cores are unused, there remains room to develop on-chip interpretation of the spikes to extract velocity. However, the output of the TrueNorth chip is always in spiking format, so such development work would simplify, but not eliminate, the off-chip velocity extraction.

. A SNN method for normal optical flow computation from silicon retina data has been proposed. The method is capable of measuring the speed of edges in the range of 1/50 pixels/ms to 1 pixel/ms in real time. The light signal coming into the vision sensor is transduced into spikes by a silicon retina. These spikes are then fed into the SNN for processing, which provides an output estimate also encoded as spikes. This network, consuming 80mW (70mW for TrueNorth, 10mW for the ATIS, omitting FPGA used for communication, which can be removed for targeted applications), running real time and using 2325 cores, 558000 neurons, is able to extract optical flow with both a low Average Angular Error rate below 10 degrees, and Average Endpoint Error of 11% (for the spiral), for a density estimation of 51%.

N E U R O M O R P H I C N E T W O R K S O N S P I N N A K E R

This chapter introduces a pure, event-driven visual encoding approach that uses precise timing mechanisms to design new computation techniques in visual processing, with the use of a real-time neuromimetic platform specifically designed to efficiently exploit the eventdriver nature of the proposed algorithms. The research presented in this chapter resulted in a full event-driven visual processing system linking a neuromorphic retina [START_REF] Posch | A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[END_REF] directly to the SpiNNaker system by an Asynchronous Event Representation (AER) bus. I implemented event-driven early vision models (optical flow, same network as the one described in Chapter 5) and 3D stereovision [73] [111] in the SpiNNaker board using a precise timing mechanism. The work presented in this chapter is part of the work package WP11.3.5 : Asynchronous Computational Retina of the Human Brain Project, and specifically concerns the milestones 11. . Biomimetic event-driven time-based vision sensors are a novel class of vision devices that -like the biological retina -are driven by "events" happening within the visual scene. They are not like conventional vision sensors, which are driven by artificially created timing and control signals (e.g. frame clock) that have no relation whatsoever to the source of the visual information. This work aimed to introduce a whole neuromorphic processing framework implementing optical flow and stereo disparity detector. I relied only on the output of a circuit contained in the Asynchronous Time-Based Image Sensor (ATIS) pixels [START_REF] Posch | A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[END_REF] that detects relative changes in pixel log luminance over time. As soon as a change is detected, the process of communicating this change event off-chip is initiated. Off-chip communication executes with low latency (on the order of microseconds), ensuring that the time at which a change event is readout from the ATIS inherently represents the time at which the change was detected. This asynchronous low-latency readout scheme provides the high temporal resolution change detection data our features rely on. We discuss two types of change detection events: "ON" events and "OFF" events, which respectively indicate that an increase or decrease in log pixel intensity has been detected. This representation of the visual information fits the SpiNNaker massively parallel multi-core architecture. It benefits from SpiNNaker's ability to simulate millions of neurons, and makes use of its computing power to develop and map event-driven computation architecture. The flow is inspired by the neural circuitry found in the rabbit's retina by Barlow and Levick [8], based on inhibition-based direction-sensitive units, combined in motion detectors. A single direction sensitive (DS) unit provides information only if the object moves in its preferred direction, otherwise it stays silent. To extract a 2D time-of-travel vector we combined four DS units together in a single motion detector. A 64x64 pixels optical flow was implemented and runs real-time on the platform. We performed stereovision using the asynchronous high temporal resolution properties of the ATIS camera as developed in [START_REF] Rogister | Asynchronous event-based binocular stereo matching[END_REF], using a modified version of a human stereopsis from Marr & Poggio [START_REF] Marr | Cooperative computation of stereo disparity[END_REF]. The combination of spatial and temporal constraints fully uses the high temporal resolution of neuromorphic retinas. It allows us to produce an optimal stereo algorithm that is able to perform depth computation by detecting coactive pixels laying on the same epipolar lines.

. « The spiking neural network architecture (SpiNNaker) project aims to deliver a massively parallel million-core computer whose interconnect architecture is inspired by the connectivity characteristics of the mammalian brain, and which is suited to the modeling of large-scale spiking neural networks in biological real time. Specifically, the interconnexion allows the transmission of a very large number of very small data packets, each conveying explicitly the source, and implicitly the time, of a single neural action potential or spike. » [START_REF] Furber | The spinnaker project[END_REF].

. --

The ATIS camera used in this work is composed of two electronic boards. The first one, denoted the sensor board contains the ATIS chip itself with all its required components. The second one is an Opal Kelly XEM3010 FPGA board housing a XC3S1500-4FG320 Spartan-3 FPGA from Xilinx.

We designed an interface boards which can be plugged in between these two existing PCBs (see Fig. 36) in order to add the driving electronics and connectors required to interface the ATIS camera to the SpiNNaker system. This interface board contains the connector required for the SpiNNaker link (see [START_REF] Temple | AppNote 1 -SpiNN-3 Development Board[END_REF][START_REF] Plana | AppNote 8 -Interfacing AER devices to SpiNNaker using an FPGA[END_REF]) and the linking logic. A view of this PCB is presented Fig. 34, showing the different components sitting on the board.

FPGA connectors SpiNNaker connector

Voltage level selector

Power supplies

Sensor board connectors

Level shifter ATIS events occupy a routing space of 9 + 8 + 1 = 18 bits, as the xcoordinate can be mapped in the 0-303 range and the y-coordinate can be mapped between 0-239 and the polarity bit can assume values in the 0, 1 range accordingly to the light intensity change direction.

An original SpiNNaker event on the other side is represented by a 32 bit routing key/address (8+8 for the x,y chip coordinates, 5 bits for the core coordinate and 11 bits for the neuron id within a core). An ATIS events need therefore to be projected in the 32-bit routing space of SpiNNaker. This is done using the techniques described below, which derives from [START_REF] Plana | AppNote 8 -Interfacing AER devices to SpiNNaker using an FPGA[END_REF].

Both solutions rely on the idea to represent the spiking sensor as a virtual SpiNNaker chip, external to an original physical mesh and to convert the addresses to the same format used by SpiNNaker. By doing so the sensor can seamlessly be integrated in the network simulated on SpiNNaker; the sensor itself emits spikes which are no different than the ones produced by other SpiNNaker chips; its interconnection is therefore transparent to the system. In other words, spiking sensors are assigned fictional, virtual chips, which are not physically present on a SpiNNaker board. Sensors directly feed their spiking data into the SpiNNaker interconnect through the bespoken SpiNNaker link or through the SATA interface.

The mapping mechanism and the SpiNNaker packet structure are shown in Figure 35 (after [START_REF] Plana | AppNote 8 -Interfacing AER devices to SpiNNaker using an FPGA[END_REF]). Injecting spikes from a single ATIS silicon retina into SpiNNaker follows the same technique presented in [START_REF] Galluppi | Event-based neural computing on an autonomous mobile platform[END_REF], they both rely on using the link provided in chip 0,0 for both boards. A region of interest of 128 × 128 pixels is selected from the original ATIS sensor space, and are assigned a virtual chip 254,254 and injected in chip 0,0. During the mapping process extra neural applications, called Proxy, are responsible for the translation from the virtual routing key (254,254) to a key which is physically present in the same chip where the Proxy application is loaded, generally chip 0,0. After the packet translation, the AER packet containing the address of the event can be routed in the SpiNNaker system as any other MultiCast packet.

. --

. .

SpiNN3 board

Connecting two Atis on the 4-chip SpiNNaker board was developed with the same approach : the two available SpiNNaker links on the 4-chip board were used to inject spikes. Figure 36.b shows the two cameras connected to the SpiNN3 board. The models described in sections 6.5 and 6.6 require more than the 10k neurons available on the SpiNN3 platform (see Figure 38b). Moreover, the UDP link (over ethernet) for retrieving spikes has a bandwidth limited to 1Mbits/s. With our 16bit AER events, we can output 64kev/s, which is way lower than the event rate of a single camera (∼ 300 kev/s in standard cases). Then, we need to upgrade to a bigger board in order to achieve the desired resolution and live streaming of data.

. .

SpiNN5 board

Moving to the SpiNN5 platform (42 chips) required to develop a new interface for the two cameras : this board does not have a dual SpiNNaker link, but the SpiNN3 platform is not capable of simulating our networks.

As shown in Figure 38b, the number of neurons required to compute the disparity map is higher than the one a SpiNN3 board can handle. Work from [START_REF] Yousefzadeh | Fast predictive handshaking in synchronous FPGAs for fully asynchronous multisymbol chip links: Application to SpiNNaker 2-of-7 links[END_REF] was used in order to achieve this: an AERNode board receives events from both cameras, merge them and send them to the SpiNNaker board through the SATA bidirectional connection (Xilink High Speed LVDS connection protocol AURORA [START_REF] Inc | Aurora 8B/10B Protocol Specification[END_REF]). The AER protocol was used to inject events through SpiNN5 FPGAs [START_REF] Plana | AppNote 8 -Interfacing AER devices to SpiNNaker using an FPGA[END_REF]. The F1-L01 link (chip 0, 0, link West from FPGA F1 -Figure 38a) was used to inject the spikes in the platform. Such a bus reaches up to 2Mev/s. As the ethernet outgoing link was limited to 64 kev/s (one UDP packet per event per millisecond), the same link was used to send the events back to the AERNode Board [START_REF] Yousefzadeh | On multiple AER handshaking channels over high-speed bit-serial bidirectional LVDS links with flow-control and clock-correction on commercial FPGAs for scalable neuromorphic systems[END_REF], and then back to the computer through an AER-USB mini2 board [START_REF] Berner | A 5 Meps $100 USB2. 0 address-event monitor-sequencer interface[END_REF]. Figure 37 shows two ATIS sensors connected to the Node board and then going to the SpiNNaker platform via a SATA link. .

The full description of the model can be found in Chapter 5.

Our complete model comprises 64 × 64 × 4 direction sensitive units for a total of 64 × 64 motion detectors. Each motion detector receives input from a 2 × 2 macropixel of the original retinal resolution (128 × 128) through a subsampling population which acts as a robust edge detector. Figure 39 shows the output of the network. On the SpiNN3 platform, we can implement up to a 86 × 86 network, 213 × 213 on the SpiNN5 (see Figure 38b). .

Detecting temporal coincidence between two spikes is a widely used feature in spiking neural networks. As a consequence, we introduce a novel neuron model for SpiNNaker, as a dedicated core for this task instead of using standard integrate and fire neurons which would introduce an un-necessary overhead. Each neuron simulated by this core has two types of synaptic input and a time window. When an incoming spike is received on one input, the neuron will output a spike if another spike was received on its second input in the given time window. We added a refractory period to this process to limit the maximum firing rate of the neurons if required.

We perform stereovision using the asynchronous high temporal resolution properties of the ATIS camera as developed in [START_REF] Rogister | Asynchronous event-based binocular stereo matching[END_REF]. The combination of spatial and temporal constraints fully exploits the high temporal resolution of neuromorphic retinas. It allows us to produce a stereo algorithm that is able to perform stereo computation by detecting coactive pixels laying on the same epipolar lines. Figure 40 shows a simplified version of the network, reduced to the matching between to epipolar lines x l and x r . The input neurons connect to the synchrony detectors in an excitatory fashion. A Full description of the network can be found in [START_REF] Osswald | A spiking neural network model of 3D perception for event-based neuromorphic stereo vision systems[END_REF]. In our setup, as we are subsampling the input space by a factor of 4, we assume that the epipolar lines are lying on the horizontal axis of the camera, thus allowing us to skip the rectification process. Special care has been taken during the mechanical assembly to ensure the parallelism of the two sensors.

The hereby described networks required S x × S 2 y neurons, where S x and S y are respectively the x and y sizes of the subsampled population. 32768 neurons are required for a S x = 32, S y = 32 population. The SpiNN3 board can contain up to a 24 × 24 network, 44 × 44 for the SpiNN5 (see Figure 38b).

The same experiment is repeated by taking inputs from the two cameras and feeding them on the SpiNNaker in real-time. Figure 41 shows different snapshots of the network output, obtained by waving a pen in front of the cameras. We show the events coming from each silicon retina and the result of the disparity computation in a 32 × 32 plane. The color encodes the disparity value: a big disparity, or, in other words, a stimulus being close to the camera corresponds to red colors (Figure 41.c). Likewise, yellowish colors correspond to a lower level of disparity, representing an object further away (Figure 41.a).

. The research presented in this chapter introduces the first stages of a full neuromorphic pipeline, from an event-based camera to a fully asynchronous spiking neural network dedicated hardware. The first contribution consists in the introduction of the novel interface between our event-based sensors and the SpiNNaker platform. Most of the work found in the literature considered in pre-loading data on the platform (and thus preventing real-time applications), combined with a reduced network size in order to fit in a single board, as the bandwidth of the on-board ethernet is limited. To get beyond small, proof-of-concept models, we wanted to connect two 128 × 128 pixels sensor, with an event rate up to 1Meps for the two cameras. The developed architecture is able to both inject and retrieve up to 4Meps in the SpiNNaker, from two incoming AER nodes. A subsequent contribution The main issue that we faced was the instability of the SpiNNaker platform. As we are injecting and retrieving spikes from a virtual core, a single physical core sees all the events going through its router, making it randomly crashing. The problem may come either from the centralized message passing structure or the (legacy) software toolchain used, or both. Due to this instability, we weren't able to benchmark the provided algorithms. Some effort was put in order to move to the new software toolchain provided by University of Manchester, and first encouraging results were obtained. More, as the University of Dresden is designing the new SpiNNaker chip, I was able to try some implementations on the new SpiNNaker2 chip, which seems promising for the future as a single chip will have the computational abilities of the current SpiNN5 board. A stimulus far from the camera will produce a small disparity, encoded in yellowish colors, a close one will produce a big disparity, encoded in red color.

C O N C L U S I O N

This thesis entitled Neuromorphic computation using event-based sensors : from algorithms to hardware implementations, is made of a voluntary ambiguity. The neuromorphic computation using event-based sensors reflects that computation can be either done by the sensor itself, or with data coming from this kind of sensors. As previously stated, the event-driven sensors offer a huge dynamic range, an accurate temporal resolution and a sensor-level data compression. All along this manuscript, we tried to explore the precise timing of the event stream, avoiding binarization or events binning, both methods leading to the classical framework of computing vision : Frames.

In Chapter 2, we used the precise timing between two consecutive events with opposite polarity, generated by an event-based camera and a liquid lens, to extract, in real time, a depth map of the scene. The control of the liquid lens and the sensitivity of the sensor is the actual limitation for this algorithm. With available technologies, we were able to compute such a depth map at 100Hz, which is, as far as we know, the highest refresh rate reported so far for a monocular system. Chapter 3 presented an evolution of the original HOTS algorithm, introducing sparse coding methods to reduce the number of prototypes used for spatiotemporal contexts clustering. This can be of a huge advantage when thinking about hardware implementation, as the number of neurons available on state-of-the-art neuromorphic hardware is low. But this reduction of prototypes has an increasing impact on the number of spikes generated, which can be a problem on some specific hardware, such as the SpiNNaker platform : too much activity can crash some cores of the platform. This method can not be suitable, in its initial version, for online learning, as we need to get all the spatiotemporal contexts before running the optimization process.

The spike sorting method presented in Chapter 4 uses the timing of consecutive spikes to extract low level features in a hierarchical way. This work has proven its ability to tackle state-of-the-art algorithms, while running in real-time, and is a good candidate for hardware implementation. Further work will have to quantify the robustness of this algorithm to limited precision and noise. If it can be shown that this algorithm stays stable towards noise and quantization, I do believe this is a very good candidate for smarter brain-machine interfaces.

The 22nd edition of the Telluride Neuromorphic Workshop gave me the possibility to implement an optical flow algorithm on TrueNorth, IBM's Neurosynaptic plaform (Chapter 5). This method uses timing of spikes in a close surrounding to infer the direction of motion in the scene. To some extent, this work could be strengthened to noise and false alarm but shaping the excitation and inhibition neurons in circle, in the manner of the ganglion cells receptive field.

Chapter 6 was the first project I was involved in during my PhD. Here, again, the timing of the spikes is of a crucial importance for the disparity matcher, as we are looking for co-activation of two pixels among the same epipolar line in both cameras. Unfortunately, the SpiNNaker platform and the software pipeline were not reliable enough in order to be able to fully characterize the implemented networks. It was almost impossible to launch twice the same simulation without crashing a core of the SpiNNaker. This is, I guess, inherent to the architecture topology, as we are flooding a single core for inputing/outputting spikes to the platform. Wijekoon et. Al proposed an VLSI implementation of the Izhikevich neuron model [START_REF] Wijekoon | Integrated circuit implementation of a cortical neuron[END_REF]. This model does not fully replicate the behavioral equations, but reflects the trend of the neuron's state (membrane potential, synaptic current) to change towards a given input. There is something beautiful in this paper : that we do not always have to fit (and stick) to behavioral equations, but rather to analyze behavior and use available knowledge to build something that behaves in the same way. Thinking this way is an ability that I hope I acquired during this PhD, and I hope I will have the opportunity of applying this during the next years.

All among this thesis, we tried to be as fair as possible by comparing the proposed method to state-of-the-art algorithms. Unfortunately, due to the nature of the signals we are dealing with, it is a hard challenge to directly compare results coming from an event-based algorithm to their standard machine learning/computer vision counterparts. The performances of the presented methods in this manuscript were obtained with known/labeled datasets, when available. As the spike sorting community is willing to get more powerful analysis tools, there are a lot of synthetic or recorded labeled databases. This was a great help when developing/tweaking the algorithms. Unfortunately, this kind of database is not all the time available, especially in the neuromorphic community. A Real(ly) event-based dataset should, in my opinion, be developed, so as to unify the results and harmonize the comparison in between the proposed algorithms. To be completely fair, this database should contain the same stimulus, recorded both with an eventbased sensor and a regular sensor, thus allowing the whole community to compete, and then validate the superiority of event-based approaches.

The end of my PhD drove me towards thinking about brain-machine interfaces. Decoding, in real-time, cerebral activity is nowadays an open issue.

Prior work [START_REF] Wodlinger | Ten-dimensional anthropomorphic arm control in a human brain-machine interface: difficulties, solutions, and limitations[END_REF] has shown that it is possible to decode neural activity in the human brain, after a daily recalibration of the system that takes a significant amount of time, thus preventing this system to be widely used. Introducing a paradigm shift in the data representation can lead to a significant gain in both performances and power consumption. A last chapter could have been given, introducing some interesting thoughts about the possible VLSI implementation of the spike sorting algorithm. It has been a chance to discover this totally new field for me, and also a way to understand more thoroughly the behavior of such systems (including our silicon retina), and the reason why of all the biases in such sensors. Future work could be achieved in order to fully characterize the proposed hardware implementation. Moreover, there were some issues with the MEA chip, preventing us to get recordings to work with. Merging the MEA chip with some low-level computation, for embedded, smarter, brain-machine interfaces, could be an interesting project to conduct... This achieves more than 3 years of reflection. I had the chance to cover almost all the neuromorphic spectrum, from algorithms to hardware implementation and design. Algorithms are entering a maturity phase : the community now understands that there is a need (and benefit) to use the precise timing for computation purposes. Even if there is a theoretical gap, as we do not have any mathematical formulation as Shannon expressed one, there is hope to achieve major breakthrough in the next years or decades. Nevertheless, standard computers that we are using as computation platforms for our algorithms are not the most efficient ones, as most of the proposed algorithms work in a sequential manner, thus preventing to efficiently use the many core architecture which is nowadays present in all processors. This is why the mixed-signal community will be of a crucial importance. In the very beginning of my thesis, I struggled understanding the biases of the silicon retina. Later on, I understood that subthreshold transistors are very sensitive animals, corrupted by mismatch and noise, and that building a (working) silicon chip is a hard process. However, every new neuromorphic chip outperforms the previous one. Research is going its way, and tomorrow will see the growth of new architectures (either spiking or sequential ones), that will allow the community to build powerful embedded applications. All among these years, I had to go through the process of reading and writing papers. One would open the question of the reviewing scheme. Is this scheme not from an old and ancient time ? Do we really need peer-reviewed papers, with all the problems of conflicts it brings ? I was also amazed by the cost of publishing a paper. Is there an equity there ? Some small labs could have good ideas, but won't be able to publish them because they do not have the funding for. Where does this money go, as the reviewers are not paid ? This is maybe the most expensive layout I ever saw... Shouldn't all the papers be in open access ? Or even free to publish, with a voting system increasing the impact of the best ones ? Achieved on August 24, 2018.

  , R., & Orchard, G. (2018). Spiking Optical Flow for Event-based Sensors Using IBM's TrueNorth Neurosynaptic System. IEEE Transactions on Biomedical Circuits and Systems, (99), 1-11. Haessig, G., Berthelon, X., Ieng, S.H., and Benosman, R (2018). A Spiking Neural Network Model of Depth from Defocus for Event-based Neuromorphic Vision. Submitted.

Figure 1 -

 1 Figure 1 -Sea turtle. Flickr photo by Philippe Guillaume.

Figure 2 -

 2 Figure 2 -a) The neuromorphic silicon event based camera with the variable motorized focal lens controlled at 100Hz b) (left) Operating principle of event detection of an event-based camera: relative changes of the luminance greater than a predefined threshold n generate ON/OFF events when there is a positive/negative change of contrast. (right) Events output from the senors are shown as on the focal plane as a frame for purpose display, black dots represent OFF events while white dots represent ON events.

Figure 3 -

 3 Figure 3 -(left) Stigmatic optical system. (right) Principle of depth from defocus.

Figure 4 -

 4 Figure 4 -(a) Successive snapshots of a sphere when sweeping the focus range.The red line represents a line of pixels in the y direction. (b) Variations of the intensity profile along the red y-axis on the above snapshots. (c) Events corresponding to the sweeping of the focus range, in black are OFF events and in white ON events. (d) Representation of spikes among a single pixel, according to the driving current of the liquid lens. Here, the focus point is estimated to be at 22.6cm from the sensor.

Figure 5 -

 5 Figure 5 -Spiking neural network. a) Input data : a circle going in and out of focus, in front of a receptive field (a single pixel) b) Neural network for focus detection composed of two input neurons, ON and OFF.They directly connect to the output neuron, and also to two blocker neurons B on and B off that are inserted to avoid parasite firings of the output neuron due to a sequence of only ON or OFF polarity events. A synchronization with the liquid lens via the Sync neuron is added, in order to encode the depth in the length of the spike train. c-e) Simulation of the SNN with NEST. (c) The input spikes (ON and OFF events) and the output of the network (OUT and Sync). The point of focus is given by the OUT neuron, while the distance is encoded in the timing between the OUT and SYNC spikes. (d) Membrane potential for the two blockers neurons. After the first spike of its respective polarity, the blockers send a inhibition to the output neuron. (e) Membrane potential of the output neuron. Spikes from the same polarity do not allow the output neuron to reach its firing threshold, while a succession of ON and OFF events make the output neuron fire.

Figure 6 -

 6 Figure 6 -(a)-(c) Snapshots during a sweep of an object (d) Reconstructed depth scene for the car. The depth is also color-coded for clarity. (e) Distribution of the error. The mean relative error is at around 0.1% and a standard deviation of 0.12%.

Figure 7 -

 7 Figure 7 -(a) Depth map from the developed setup (raw data, no post-processing) (b) Conventional Image of scene for display purposes (c) Depth map from the Kinect used as reference. The yellow square corresponds to the field of view. (d) Relative error for this scene compared to Microsoft Kinect. The relative error increases with depth.

Figure 8 -

 8 Figure 8 -Snapshots of the event stream, and associated depth maps during a sweep (5ms) for multiple objects. Black and white dots are the OFF and ON events from the event-based silicon retina, as described in Section 2.2.1.Distance is color-coded.

  9.e), only the external edges of objects 1 and 2 (O 1L and O 2R ) generate OFF spikes. As the focus reaches object 1, object 1 generates ON spikes and object 2 OFF spikes. The two middle edges (O 1R and O 2L ) are now superimposed, with two different polarities, causing the failure of the algorithm (Fig.9.d).

Figure 9 -

 9 Figure 9 -Highlighting of the wrong depth measurements for two closeby edges.The two central plots show events in the x-time plane, smashing the y-dimension. Events are color coded with their polarity (red for OFF events, blue for ON events). The right one is a valid case, with no overlap.The left one contains an overlap in the event stream, leading to wrong depth deductions in this case. 4 snapshots of events are presented for every case

Figure 10 -

 10 Figure 10 -a) The ATIS sensor. b) Functional diagram of an ATIS pixel. Two types of asynchronous events, encoding change and brightness information, are generated and transmitted individually by each pixel in the imaging array. Each time the luminance of the considered pixels rises a level (a relative change from previous measurement), a change detection event is generated, like at time t 0 . This triggers a second circuitry which measures the absolute gray level value. This value is coded in the timing between two events occurring at t 1 and t 2 . This timing encodes the time required to capture a given amount of light, and is then inversely proportional to the absolute gray level.

Figure 11 -

 11 Figure 11 -Time-surface generation and decomposition over a basis. The ATIS camera (a) generates a stream of events (b) from an moving X character. The time context (d) of one event is extracted from the spatio-temporal domain (c) and convolved with and exponential decay (e), providing a time-surface (f). This time-surface is projected over a set of basis functions (g), which gives the resulting (h) reconstruction. Inspired from [59] .

.

Figure 12 -

 12 Figure 12 -Representation of the datasets used for the experiments: (a) Letters and digit dataset: consisting of the 26 characters of the roman alphabet and the digits from 0 to 9. (b) Flipped cards dataset: consisting of the four suits of a deck of cards (club, diamond, heart and spade). Inspired from [59] .

Figure 13 -Figure 14 - 2 × (2R 1 + 1 ) 2 . 1 prototypes (cf. eq 16 )

 13142112116 Figure 13 -Results for the character dataset. a) Output of the 3 layers network for three characters after learning. This output is color coded according to the responding basis (1 out of 24). We can see that some basis are specialized (red for horizontal edges, dark blue for curves, etc). b) Signature for two different representations of the same digit. The signatures are very similar between the same class and differ from class to class, allowing a good classification. c) 24 bases of the third layer of the model after learning on the letters and digits dataset. Because they are combination of responses of the previous layer, they are difficult to interpret as they are, but one can notice that they differ one from the other.

2 (cf. eq 16 ) 2 + 1 ) 2 .

 16212 to obtain new time-surfaces, of size N 1 × (2R layer (2, +)(i) and (2, -)(j) and a projection basis of N 2 (cf. eq 16) elementary time-surfaces is estimated. (k, l)

Figure 15 -

 15 Figure 15 -Generation of a context. (a) Level crossing for spike generation. The sampled signal (red dots) is interpolated. Each time the linearly interpolated signal (dashed red line) crosses a level (horizontal dark dashed-dotted lines), a spike is generated, either positive or negative depending of the rising or falling behavior of the signal. (b) Generation of a context. When a spike occurs at channel c i , all the values of the exponential decays among a spatial neighborhood are taken (i ∈ [[i -r, i + r]],r being the current radius value) and are our base features. (c) Exponential time decay among a spike. The value of the exponential kernel is color-coded : brighter pixels represent higher values than darker ones.

  Figure reffig:voronoi-b shows the noise-free input data. The training of the algorithm was done with one full set of the intermediate noise-level. The learned centers where then used for building signatures and recognition for all the 31 remaining sets. The parameters used for this benchmark are summarized in Table 2

  .

3 Figure 16 -

 316 Figure 16 -Output of the last layer. For all the 9 different shapes, the matching clusters among a spike is color-coded. Our algorithm learns and extracts local curvatures that can be used for classification.

Figure 18 -

 18 Figure 18 -Classification for the benchmarking data. (a) Voronoi diagram for the projected signatures in two dimensions and corresponding ground truth in color. Mostly, the signatures contain enough information to ensure good discriminant classification. (b) The associated temporal shapes.We can notice that our algorithm extracts signatures that are distinguishable one from each other all thereby allows good classification.

Figure 19 -

 19 Figure 19 -Results with data from ex-vivo rat retina. (a) TSNE projection with output of the last layer. It is possible to distinguish two major types of cells (c) Clustering is able to find several possible new cells, which can be merged together. (c) The Euclidian distance allows us to merge all the clusters to estimate the best number of classes (the darker the closer). Here, we have two different classes. (b) The two found classes with mean shape (dark) and some examples (color).

Figure 20 -

 20 Figure 20 -Average computation time for the proposed architecture. Y-axis represent the ratio of the computation time required and the duration of the recording. Baseline of 1 (red line) is real-time. Above is faster than real-time. The proposed method is able to handle in real time a 3-layer hierarchy.

Figure 21 -

 21 Figure 21 -One of the Bioamp chip, bonded on its PCB.

Figure 22 -

 22 Figure 22 -Working principle of the ATIS camera. (left) A significant relative change of the luminance (programmable threshold n) generates an ON (respectively OFF) event when there is an increase (resp. decrease) in luminosity. (right) Snapshot of events over 100 ms for a car passing in front of the camera. White dots represents ON events, black ones OFF events.

Figure 23 -

 23 Figure 23 -Direction Sensitive (DS) unit. A stimulus from left to right first excite(yellow cell) the output (gray), which is self-excitatory until it gets an strong inhibitory input (blue cell), as a movement in the other direction (right to left) doesn't produce any output. Thus, this unit is directive to a specified direction. More, the timing between the beginning and the end of the output spiking give an information of the (inverse) velocity. The excitatory input also copies a spike to the delay unit, to inhibit the DS cell in case the natural inhibition doesn't spike. Here, the delayed inhibition doesn't have any impact. Inspired by[START_REF] Giulioni | Eventbased computation of motion flow on a neuromorphic analog neural platform[END_REF].

Figure 24 -

 24 Figure24-Assembly of the 4 DS units (North, South, West, East). The direction of the movement is gathered by taking in consideration, for each pixel, a vertical and horizontal component. Inspired by[START_REF] Giulioni | Eventbased computation of motion flow on a neuromorphic analog neural platform[END_REF].

Figure 25 -

 25 Figure 25 -a) The TrueNorth topology. Each neurosynaptic core has 256 axons, 256×256 synapse crossbar and 256 neurons. Information flows from axons to neurons gated by binary synapses, where each axon fans out, in parallel, to all neurons thus achieving a 256-fold reduction in communication volume compared to a point-to-point approach. Network operation is governed by a discrete time step. In a time step, if the synapse value for a particular axon-neuron pair is non-zero and the axon is active, then the neuron updates its state by the synaptic weight corresponding to the axon type. Next, each neuron applies a leak, and any neuron whose state exceeds its threshold fires a spike [19]. b) IBM's NS1e board (4096 cores, 1 Million neurons and 256 Million synapses) and ATIS sensor with native link used for this chapter.

Figure 26 -

 26 Figure 26 -Arrangement of a TrueNorth core implementing the motion model. Refractory (red), Delay (Blue), and DS (green) neurons are shown horizontally along the bottom of the core. Input axons are shown vertically on the left of the core. Text labels indicate the direction inputs are arriving from and outputs are routed to. The number of neurons (horizontal axis) and axons (vertical axis) are explicitly indicated for each part of the model. Shaded regions indicate areas where connections (synapses) exist between axons on the left and neurons at the bottom.

Figure 27 -

 27 Figure 27 -The two recordings used in this chapter. (a) and (b) show images of a rotating pipe and spiral respectively from the sensor's viewpoint. The direction of rotation is shown by the green arrows inset. (c) and (d) show 10ms each of the pipe and spiral recordings. Red points indicate OFF-events (decreases in intensity) while blue points indicate ONevents (increases in intensity). Axes directions used in (33) and (34) are shown in the top left.

Figure 28 -

 28 Figure 28 -3D representation of network result on the Spiral data. For clarity purpose, only the direction is here represented. The direction of movement is color-coded in the HSV space. The individual isolated spikes are flow estimates generated in response to sensor noise. Video available online [128].

Figure 29 -

 29 Figure 29 -Rotating spiral. a) Absolute error. One can notice that the error is mainly located at the end of edges, due to the lack of adjacent active pixels. In some extend, it is impossible to compute a flow here. b) Top view of the output of the network.

Figure 30 -

 30 Figure 30 -Average Endpoint Error (AEE) for the spiral input. Our network isable to reconstruct the velocity vector with an average error of 11%, error which is mainly located at the edges of the spiral, where it is impossible to know the speed by our method.

Figure 31 -

 31 Figure 31 -Rotating pipe. a) Absolute error. One can notice that the error is mainly located at the end of edges, due to the lack of adjacent active pixels. In some extend, it is impossible to compute a flow here. b) Top view of the output of the network. Video available online [127].

Figure 32 -

 32 Figure 32 -Snapshot from a video showing optical flow output from TrueNorth for data captured while moving the ATIS sensor by hand in the lab. The leftmost image shows the scene from a similar angle captured using a cellphone camera. The middle shows the direction of motion detected, and the right shows the speed. Strong vertical and horizontal edges come out clearly, but the direction of motion differs because the network estimates the normal component of optical flow.

  3.5.1, 11.3.5.2, 11.3.5.5 and 11.3.5.7.

Figure 33 -

 33 Figure 33 -SpiNNaker chips and development boards. From [87].

Figure 34 -

 34 Figure 34 -Interface board used to feed data from an ATIS camera to SpiNNaker.The interface board sits in between the sensor board and the FPGA board driving the camera.

Figure 35 -

 35 Figure 35 -SpiNNaker packet structure and event to MC packet mapping.

Figure 36 -

 36 Figure 36 -Designed interface board plugged between an ATIS camera (a) and the SpiNNaker system, or two ATIS cameras (b) to the SpiNN3 platform.

Figure 37 -Figure 38 -

 3738 Figure 37 -Stereo setup. Two ATIS sensors, with AER connection to the Node Board, then packed in the Sata link to SpiNNaker

Figure 39 -

 39 Figure 39 -Output of the flow network for a Counter Clockwise rotation of a pen (a to f). Direction is encoded among the HSV map given in (g). The overall direction is the expected one, even if some noise events trigger false computations.

Figure 40 -

 40 Figure 40 -The proposed cooperative network for stereo matching, as described in[START_REF] Marr | Cooperative computation of stereo disparity[END_REF]. The output from the two epipolar lines x l and x r both connects to the synchrony detectors. The blue lines represent excitatory links, as the red ones inhibitory links. The final output encodes a representation of the original scene in disparity space. For the sake of visibility, only a horizontal line of retinal cells is considered. The corresponding coincidence and disparity detector units, hence, lie within a horizontal plane. Again, only few units are shown here, while in the complete network units would be uniformly distributed over the entire plane.

Figure 41 -

 41 Figure 41 -Results of disparity computation with real-time input from two ATIS.A stimulus far from the camera will produce a small disparity, encoded in yellowish colors, a close one will produce a big disparity, encoded in red color.

Figure 42 -

 42 Figure 42 -Chart of this thesis' words count, generated with www.wordclouds.com

Table 1 -

 1 Results and comparison with original HOTS model

	Algorithm		This work		HOTS [59]
	Dataset		Cards	Digits	Cards	Digits
	Number of Centers	3-6-9	6-9-12	6-9-12	8-16-32 8-16-32
	Recognition Rate	83%	100%	100%	100%	100%
	Number of spikes 7 599 450 18 500 730 74 440 535 52 410 513 383

Table 2 -

 2 Model parameters for the artificial benchmarking

		L 1	L 2	L 3	L 4	L 5
	τ(µs)	25	50	75	100	125
	R(px)	2	4	8	16	32
	N c	4	11	39	147	562
	ζ	5.10 -5 5.10 -5 5.10 -5 5.10 -5 5.10 -5

Table 3 -

 3 Recognition rate on in-vivo data. The bolted algorithms are the only one working in real-time. Our algorithm performs better than state of the art online spike-sorting, and are slightly below the known best algorithm reported on this dataset, which is not real-time.

	Method	minimum maximum	average
	Shahid et. Al [115]	N/A	N/A	66%
	Werner et. Al [134]	N/A	N/A	82.1%
	Gasthaus et. Al [38]	N/A	N/A	Fscore 0.91%
	Haessig et. Al (proposed method)	84.75%	90.22%	87.33% -Fscore 0.89%
	cific dynamic that we want to extract and distinguish. Without optimisations,
	the proposed method is able to handle up to a 3-layer architecture, defeat-
	ing state-of-the-art algorithms on publicly available datasets. This work is
	a major step towards efficient spike sorting, which is today a real challenge
	for research purpose, closed-loop applications or brain-machine interfaces
	[138].

Table 4 -

 4 Neuron parameters

	Parameter	Symbol Refractory DS neuron Delay
	excitatory weight (mV)	w e	255	150	1
	inhibitory weight (mV)	w i	0	50	0
	threshold (mV)	Θ	1	125	τ d
	leak (mV/ms)	l	-254	-1	1
	reset potential (mV)	V r	τ r .l	127	0
	negative floor (mV)	β	τ r .l	-50	0
	Delay (ms)	τ d		50
	Refractory period (ms)	τ r		50

Cette thèse porte sur l'implémentation d'algorithmes événementiels, en utilisant, dans un premier temps, des données provenant d'une rétine artificielle, mimant le fonctionnement de la rétine humaine, pour ensuite évoluer vers tous types de signaux événementiels. Ces signaux événementiels sont issus d'un changement de paradigme dans la représentation du signal, offrant une grande plage dynamique de fonctionnement, une résolution temporelle importante ainsi qu'une compression native du signal. Sera notamment étudiée la réalisation d'un dispositif de création de cartes de profondeur monoculaires à haute fréquence, un algorithme de tri cellulaire en temps réel, ainsi que l'apprentissage non supervisé pour de la reconnaissance de formes. Certains de ces algorithmes (détection de flot optique, construction de cartes de profondeur en stéréovision) seront développés en parallèle sur des plateformes de simulation neuromorphiques existantes (SpiNNaker, TrueNorth), afin de proposer une chaîne de traitement de l'information entièrement neuromorphique, du capteur au calcul, à faible coût énergétique.

https://www.nvidia.com

https://www.xilinx.com

https://en.wikipedia.org/wiki/Neuromorphic _ engineering