Amina Benchohra

Yanling Li

Lise-Marie Chamoreau

Benoit Baptiste

Erik Elkaim

Nathalie Guillou

David Kreher

Rodrigue Lescouëzec

Erik Elkaïm

The atypical hysteresis of [Fe(C6F5Tp)2]: Overlay of spincrossovers and symmetry-breaking phase transition

Spin-crossover (SCO) complexes represent a prominent class of molecular switches, [START_REF] Feringa | Molecular Switches[END_REF] which can be reversibly converted between two distinct electronic configurations, a high-spin (HS) state and a low-spin (LS) state, by various stimuli such as light, temperature, pressure, electric and magnetic fields. [START_REF] Halcrow | Spin-Crossover Materials: Properties and Applications[END_REF] This spin-state change can lead to drastic changes in the magnetic, optical, dielectric or mechanical properties, making these SCO complexes very interesting for the design of advanced materials. [START_REF] Koo | [END_REF][4][5] The spinstate switching is also attractive for the development of moleculebased electronic or spintronic devices. [6][7][8][9][10] In this context, light, neutral and sublimable Fe(II) SCO complexes are particularly investigated because they can be easily deposited onto surfaces through high-vacuum evaporation techniques. [11][12][13] In the solidstate, the spin crossover phenomenon can manifest itself in different manners depending on the interaction between the neighboring SCO units. It can be gradual when there is no (or weak) intermolecular coupling or abrupt, when efficient elastic interactions between SCO centers lead to cooperative Spin Transitions (ST). [14] For strongly cooperative systems hysteresis can occur. These transitions are of particular interest for information storage or sensing applications. [15,16] A possible synthetic strategy to increase the cooperativity consists in linking SCO centers through rigid bridging ligands or inorganic connectors. [7,15,17] Alternatively one can rely on supramolecular interactions (such as H-bondings) and electrostatic interactions that can be responsible for the large hysteresis observed in some monometallic species. [18][19][20] It should be noted that hysteresis can be associated with a symmetry-breaking structural phase transition (SPT). It is believed that local distortion during the spinstate change promotes displacement or motions in the SCO complexes. These motions would then propagate via intermolecular interactions through the crystal lattice giving rise to the concomitant structural phase transition. However, one cannot exclude that structural phase transition may also trigger the abrupt spin-transition. [21] In all cases, the supramolecular framework and the topology of the crystal packing are recognized to play a major role in the ST shape. The studies on polymorphs offer a unique way to enlighten the impact of the crystal packing on the transition profile. [22,23] The significant impact of polymorphism on SCO properties was mainly highlighted with polymorphs achieved by solution-mediated phase transitions (obtained from regular bench conditions). [24] In contrast, polymorph examples resulting from thermally initiated phase transformations are scarce. For example, recent reinvestigations of old SCO materials showed that an annealing could lead to new phases. Salmon et al. who reexamined the [Fe II (Tp)2] (Tp = hydrotris(pyrazolyl)borate) SCO properties, demonstrated that by heating the sample near its sublimation temperature it was possible to irreversibly transform a tetragonal polymorph (a kinetic phase) into a monoclinic one (a thermodynamic phase). [25] In that case, the conversion leads to a shift of the transition temperature and a moderate change of the SCO profile, where both phases l show a gradual transition with no hysteresis. A similar phenomenon was later observed for the related [Fe II (Tp * )2] complex. [26] In this context, we decided to conduct high-temperature investigations of the neutral [Fe II (C6F5Tp)2] SCO complex (C6F5Tp = pentafluorophenyl tris(pyrazol-1-yl)borate ligand). Herein, we detail the surprising effect of a high temperature treatment on [Fe II (C6F5Tp)2] crystals: a radical change in the crystal packing that is accompanied by an unprecedented changeover of the SCO properties, from a very gradual spin-equilibrium to one of the broadest hysteretic transition found in monometallic SCO complexes. Moreover, further in-depth single crystal and powder diffraction studies revealed the origins of the unprecedented "gradual and rounded" hysteresis loop observed after thermal treatment.

The synthesis of the [Fe II (C6F5Tp)2] complex was first reported by Wagner et al. together with its X-ray structure obtained from crystals grown at room temperature. However, the magnetic properties of this triclinic phase (noted phase 1) were not investigated. [27] We will first describe the magnetic properties of phase 1 (kinetic phase) and then those of the new phase obtained after thermal treatment: the phase 2 (thermodynamic phase) that otherwise undergoes a reversible structural phase transition between 2-LT and 2-HT. The χMT versus T curve of 1 (χM is the molar magnetic susceptibility) is shown in figure 1. The χMT measured at 300 K, ca. 0.06 cm 3 mol -1 K, indicates the occurrence of a low-spin Fe(II) state (S = 0). This agrees with the crystal data obtained at the same temperature (table 1), that show, metal-ligand distances typical of low-spin Fe(II) (as observed in [Fe II (Tp)2]) for the two Fe crystal sites that are present in phase 1. [25] Upon heating, the χMT product increases over a remarkably broad temperature range (ca. 200 K), reaching 3.56 cm 3 mol -1 K at 500 K. This value agrees with that expected for a complete spin-crossover, the expected χMT being ca. 3.6 cm 3 mol - 1 K (S = 2, g = 2.2). The gradual conversion is also coherent with the absence of significant supramolecular interactions in 1. The weak CH … π and CF … π interactions observed are detailed in the supporting information (SI). In order to better understand the broad SCO curve, the crystal structure was recorded at 430 K, a temperature corresponding approximately to the half spin-state conversion. Interestingly, the structural data show that the two crystallographic sites, Fe1 and Fe2, exhibit different spin states, with metal-ligand distances and distortion values that are typical of low-spin and high-spin environment, respectively. [25] As previously reported for similar bis-scorpionate Fe(II) complexes, the spin-state change is also reflected in the FeN-NB and FeN-NC torsion angles, which typically show larger distortion in the HS state (table 1). [22,28] In summary, the overall spin-crossover in phase 1 is better described as two consecutive gradual spincrossovers. The spin-crossover which occurs at lower temperature on the site 1 in comparison to the site 2, might be corelated to a weaker ligand field induced by a higher distortion as reflected by the distortion parameter, Σ, measured at 300 K (table1).

A closer look at the χMT vs T curve of phase 1 reveals an anomaly at high temperature, namely a sudden increase from 3.10 to 3.56 cm 3 mol -1 K between 495 and 500 K. This change is associated with the melting of phase 1, which was detected at ca. 493 K at ambient pressure. It is worth noting that the FT-IR and NMR solution spectra of the sample heated to 500 K (in the magnetometer or in the DSC apparatus) are identical to those of the fresh material (SI). Therefore, the change in the magnetic behavior is not associated with a chemical reaction but with a phase transition as confirmed by the structural study (see below).

The heating up to 500 K induces an irreversible change in the magnetic behavior: the new phase (phase 2) exhibits a transition with a pronounced sigmoidal shape and displays a reversible and remarkably broad and rounded hysteresis loop of 65 K width at the widest points (at scan rate of 2 K/min). The profile of the spin transition is very uncommon. For both heating or cooling ramp, it could be described as a gradual SCO behaviour, which seems to be accelerated from threshold temperatures. These threshold temperatures roughly correspond to temperature at which half of the complexes have undergone a spin-state change. In fact, the examination of the d(χMT)/dT curve (see SI) shows two maxima upon each thermal ramp, reflecting spin-conversion discontinuities. For instance, upon the second heating ramp, a first inflection point is measured at T= 461 K and a second one at T= 472 K. The height and the broadness of the hysteresis slightly vary with the experimental conditions (see SI) but it is always reproducible and it only appears after heating up to 500 K. In order to shed light on the unusual behavior of the [Fe II (C6F5Tp)2] complex, various single-crystal and powder X-ray diffraction experiments were carried out. First of all, we measured several crystals of the phase 1 upon heating. As mentioned above, a crystal structure of the phase 1 was obtained at 430 K. Beyond this temperature, the quality of the crystal data decreases and measurements above 490 K showed that the colorless crystals (in the high-spin state) shrink and melt. Therefore, we could not follow the phase transition in-situ. Nonetheless, we managed to study single crystals of the new phase obtained after the melting of The CCDC numbers can be found in the Supporting Information. Magnetic properties of both materials, before heating up to 500 K (blue) and after melting and recrystallizing in the magnetometer (orange).

phase 1 and recrystallization. Despite their poor crystallinity, a satisfactory structural model was obtained through XRD analysis using synchrotron radiation (SI). The structure of this new phase was measured at low-temperature (100 K) and is noted from now 2-LT. Two observations can be made on its crystal structure. Firstly, the approximate Fe-N distances (1.97 Å) and the distortion of the coordination sphere (Σ = 28.5°) measured at low temperature agrees with a low-spin state, coherently with the magnetic measurements. Secondly, the supramolecular arrangement of the complexes in the 2-LT crystal structure is completely reorganized in comparison with that of 1 (Figure 1). Views along b and c axes show an alternation of layers of {Fe II (Tp)2} core complexes and pentafluorophenyl groups (Figure 2). A close examination of the crystal packing did not reveal a significant enhancement of supramolecular interactions (see details in SI). Only very weak C-F … H, C-H … π and C-F … π interactions [29] are observed, with distances that are similar or even longer than those measured in phase 1. From these observations, we can conclude that the observed hysteresis does not arise from a strong cooperativity that would be mediated by the supramolecular interaction. Besides this experiment, several attempts were made to follow structural changes of this new phase by SC-XRD. However, the crystalline quality decreased upon heating preventing a reliable collection of data.

In order to overcome this problem, we then carried out powder Xray diffraction experiments on the sample recovered from DSC measurement after heating up 1 to 500 K. Satisfactory diffraction data were obtained by using synchrotron radiation. The powder diffraction patterns recorded in the 298-503 K range are depicted in Figure 3. First, the data obtained at room temperature display sharp peaks, which match well with those calculated from the crystal structure of 2-LT phase. Then, in order to understand its thermal behavior, Rietveld refinements of all the powder patterns recorded up to 503 K were carried out. The model of 2-LT phase determined from single crystal data was used as a starting point, treating the organic ligand as rigid body (see SI). Upon heating up to 383 K, the peaks remain thin and undergo some small shifts toward lower 2θ values, which is characteristic for thermal expansion (inset Fig. 3). From 393 K, the unit cell parameters show a different evolution, not simply due to the thermal effect but coherent with the occurrence of a gradual SCO (see SI), as seen in the χMT vs T curve. Although the standard deviations on the Fe-N distances are quite high, it seems that the average Fe-N distances tend to increase, which is also consistent with a spinstate change from LS Fe(II) to HS Fe(II) (SI). Then, nearby 473 K, the occurrence of a structural phase transition is evidenced (insert Fig. 3, details in SI). This is revealed by the appearance of new peaks on the powder patterns, whose intensity increases with rising temperature, while a decrease of the peaks ascribed to the 2-LT phase is concomitantly observed. The conversion from 2-LT to the high-temperature phase, 2-HT, is not complete at 503 K, as peaks of both phases coexist. Since the compound starts decomposing at higher temperature, it was not possible to obtain a pure 2-HT phase by further heating. Nevertheless, indexing of the 2-HT phase was made possible by excluding the diffraction peaks attributed to the 2-LT phase, which have been identified by a careful analysis of the thermal dependence of 2-LT unit cell parameters above 473 K. This analysis allows converging to an orthorhombic unit cell (space group Pbca), with a doubling of the a parameter compared to the monoclinic 2-LT one (and a doubling of the unit cell volume). The structural determination of the 2-HT phase was then undertaken by using the simulated annealing approach, considering (C6F5Tp) as a rigid body, the scattering contribution of the 2-LT phase being calculated together by the Rietveld method analysis. In the 473-503 K range, structural models of both phases were refined. together with an increase of the intermolecular Fe-Fe distances as a likely effect of the Fe-N lengthening. In summary, the powder Xray study supports the occurrence of a gradual nature of the spin conversion in the 2-LT phase, as observed by magnetometry below 470 K. Above that temperature, the material undergoes a phase transition to the higher symmetry 2-HT phase. This phase transition is reflected in the magnetic curve by the sudden increase of the χMT product near 473 K. In conclusion, we showed that [Fe II (C6F5Tp)2] SCO complex undergoes a first crystalline phase transition after a hightemperature melting and recrystallization from phase 1 to phase 2-HT. This irreversible transition leads to a profound reorganization of the crystal packing and a radical change in the SCO profile: from an extended spin-equilibrium to a broad and 'rounded' hysteresis. Despite this change, the spin-state change remains essentially non-cooperative, in agreement with the absence of significant intermolecular interactions. In fact, we demonstrated that the hysteresis opening results from a second reversible abrupt structural transition occurring between two phases which both show gradual spin transition at different temperatures. The structural phase transition is accompanied by a more abrupt spin-state change and it occurs when roughly over 50% of the SCO complexes underwent the gradual spin-state change. Upon cooling, the process seems perfectly symmetrical. Overall, these observations suggest that the molecular volume change that accompanied the gradual spin-state change would trigger the abrupt structural phase change once a threshold spinconversion ratio is achieved. To our knowledge such interplay between gradual SCO and a symmetry-breaking transition is rare. [30] The present phenomenon contrasts with the much well known "rectangle-like" hysteretic spin-transitions, generally associated to strong cooperativity, where both structural changes and spin-state change occur concomitantly in a sharp temperature range. Overall, this result provides a new example of the wealth and complexity of the spin-crossover behaviors.

Figure 2 .

 2 Figure 2. Perspective view of phase 2-LT phase along the c (top) and the b (bottom) axes.

Figure 1 .

 1 Figure 1. (top) Perspective view of the crystal packing in the phases crystallized at room temperature and high temperature (bottom).The CCDC numbers can be found in the Supporting Information. Magnetic properties of both materials, before heating up to 500 K (blue) and after melting and recrystallizing in the magnetometer (orange).

  Figure 3. [Fe II (C6F5Tp)2] (phase 2, obtained from melting of phase 1) (a) Thermal evolution of diffractograms in the 298 -503 K range (λ = 0.57039 Å); (b) zoom in the 2θ range 4.8 -8.15 ° is shown in inset; (c) Temperature dependence of the unit cell volume of the 2-LT phase in the range 298 -473 K.

Table 1 . Selected Structural Data of phase 1

 1

	T / K	Fe-N	Σ (Fe)	FeN-NB	FeN-NC
	(Range)		(average value)
	Fe1		Fe2	Fe1 Fe2 Fe1 Fe2 Fe1 Fe2
	300 1.952(3)	1.966(3)	22.0 28.2 2.8 2.6 170.3 168.2
	1.981(3)	2.001(3)
	430 1.967(4)	2.080(4)	28.9 56.6 3.6 5.6 169.4 165.3
	1.997(4)	2.131(4)
	490 1.99(1)	2.106(9)	38.0 72.6 4.8 7.9 168.3 164.0
	2.057(9)	2.14(1)

Angles (°) Distance (Å); Σ : sum of the deviation of the twelve pseudoorthogonal angles of the Fe coordination sphere.

Acknowledgements

In the framework of the CNRS ЯÉCIPROCS network, this work has been accepted for synchrotron beamtime by the Soleil scientific proposal committee (BAG proposals 20171078 and 20180913). The authors would like to thank Erik Elkaïm and Pierre Fertey for their help on CRISTAL beamline at SOLEIL and Kamel Boukheddaden for the useful discussions.

Funding

This work has been funded by the LabEx michem, by the CNRS and the Ministère de l'Education et de la Recherche.

COMMUNICATION

A rounded switch: The [Fe II (C6F5Tp)2] complex is converted through melting from a crystal phase showing a very gradual spin-crossover to another one showing a broad hysteretic spin-transition. The uncommon "rounded" hysteresis occurs in a noncooperative SCO system. It is due to a symmetry-breaking phase transition that is triggered when roughly ca. 50% of the SCO complexes are switched.


Milad Mozafari
email: milad.mozafari@ut.ac.ir

Mohammad Ganjtabesh

Abbas Nowzari-Dalini

Simon J Thorpe
email: simon.thorpe@cnrs.fr

Timothée Masquelier
email: timothee.masquelier@cnrs.fr

Bio-inspired digit recognition using reward-modulated spike-timing-dependent plasticity in deep convolutional networks

Keywords: Spiking neural networks, Deep architecture, Digit recognition, STDP, Rewardmodulated STDP, Latency coding

The primate visual system has inspired the development of deep artificial neural networks, which have revolutionized the computer vision domain. Yet these networks are much less energy-efficient than their biological counterparts, and they are typically trained with backpropagation, which is extremely data-hungry. To address these limitations, we used a deep convolutional spiking neural network (DCSNN) and a latency-coding scheme. We trained it using a combination of spike-timingdependent plasticity (STDP) for the lower layers and reward-modulated STDP (R-STDP) for the higher ones. In short, with R-STDP a correct (resp. incorrect) decision leads to STDP (resp. anti-STDP). This approach led to an accuracy of 97.2% on MNIST, without requiring an external classifier. In addition, we demonstrated that R-STDP extracts features that are diagnostic for the task at hand, and discards the other ones, whereas STDP extracts any feature that repeats. Finally, our ap-

Introduction

In recent years, deep convolutional neural networks (DCNNs) have revolutionized machine vision and can now outperform human vision in many object recognition tasks with natural images [START_REF] Rawat | Deep convolutional neural networks for image classification: A comprehensive review[END_REF][START_REF] Gu | Recent advances in convolutional neural networks[END_REF]. Despite their outstanding levels of performance, the search for brain inspired computational models continues and is attracting more and more researchers from around the world. Pursuing this line of research, a large number of models with enhanced bio-plausibility based on spiking neural networks (SNNs) have emerged. However, SNNs are not yet competitive with DCNNs in terms of recognition accuracy. If DCNNs work well, what is the reason for this increased interest in neurobiological inspiration and SNNs?

To begin with, energy consumption is of great importance. Thanks to the millions of years of optimisation by evolution, the human brain consumes about 20 Watts [START_REF] Mink | Ratio of central nervous system to body metabolism in vertebrates: its constancy and functional basis[END_REF] -roughly the power consump-1 tion of an average laptop. Although we are far from understanding the secrets of this remarkable efficiency, the use of spike-based processing has already helped neuromorphic researchers to design energy-efficient microchips [START_REF] Furber | Large-scale neuromorphic computing systems[END_REF][START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF].

Furthermore, employing embedded and real-time systems for artificial intelligence (AI) is important with the advent of small and portable computing devices. In recent years, specialized real-time chips for DCNNs have been released that are capable of fast simulation of pre-trained networks. However, online on-chip training of DCNNs with exact error backpropagation is not yet practical, due to the high-precision and time-consuming operations. In contrast, biologically inspired learning rules such as spike-timing-dependent plasticity (STDP) [START_REF] Gerstner | A neuronal learning rule for sub-millisecond temporal coding[END_REF][START_REF] Bi | Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type[END_REF] can be hardware-friendly, and appropriate for online on-chip training [START_REF] Yousefzadeh | Hardware implementation of convolutional stdp for on-line visual feature learning[END_REF].

These reasons, together with the natural ability of SNNs to handle spatio-temporal patterns, have led researchers to try a range of methods for applying SNNs to visual tasks. The use of hierarchically structured neural networks is a common approach, yet configuring other parameters such as the number of layers, neuron models, information encoding, and learning rules is the subject of much debate. There are shallow [START_REF] Masquelier | Unsupervised learning of visual features through spike timing dependent plasticity[END_REF][START_REF] Yu | Rapid feedforward computation by temporal encoding and learning with spiking neurons[END_REF] and deep [START_REF] Lee | Training deep spiking neural networks using backpropagation[END_REF][START_REF] O'connor | Deep spiking networks[END_REF] SNNs with various types of connectivity structures, such as recurrent [START_REF] Thiele | A wake-sleep algorithm for recurrent, spiking neural networks[END_REF], convolutional [START_REF] Masquelier | Unsupervised learning of visual features through spike timing dependent plasticity[END_REF][START_REF] Cao | Spiking deep convolutional neural networks for energy-efficient object recognition[END_REF][START_REF] Tavanaei | Bio-inspired spiking convolutional neural network using layer-wise sparse coding and STDP learning[END_REF], and fully connected [START_REF] Diehl | Unsupervised learning of digit recognition using spike-timing-dependent plasticity[END_REF]. Information encoding is the other aspect of this debate, where ratebased [START_REF] Merolla | A digital neurosynaptic core using embedded crossbar memory with 45pJ per spike in 45nm[END_REF][START_REF] Hussain | Improved margin multiclass classification using dendritic neurons with morphological learning[END_REF][START_REF] O'connor | Real-time classification and sensor fusion with a spiking deep belief network[END_REF] and temporal coding [START_REF] Masquelier | Unsupervised learning of visual features through spike timing dependent plasticity[END_REF][START_REF] Yu | Rapid feedforward computation by temporal encoding and learning with spiking neurons[END_REF][START_REF] Diehl | Unsupervised learning of digit recognition using spike-timing-dependent plasticity[END_REF][START_REF] Mostafa | Supervised learning based on temporal coding in spiking neural networks[END_REF] are two of the main options. Different learning techniques are also applied to SNNs, from backpropagation [START_REF] Cao | Spiking deep convolutional neural networks for energy-efficient object recognition[END_REF][START_REF] Diehl | Fast-classifying, high-accuracy spiking deep networks through weight and threshold balancing[END_REF][START_REF] Wu | Spatio-temporal backpropagation for training high-performance spiking neural networks[END_REF][START_REF] Liu | Mt-spike: A multilayer time-based spiking neuromorphic architecture with temporal error backpropagation[END_REF], tempotron [START_REF] Yu | Rapid feedforward computation by temporal encoding and learning with spiking neurons[END_REF][START_REF] Zhao | Feedforward categorization on AER motion events using cortex-like features in a spiking neural network[END_REF], and other supervised techniques [START_REF] Hussain | Improved margin multiclass classification using dendritic neurons with morphological learning[END_REF][START_REF] O'connor | Real-time classification and sensor fusion with a spiking deep belief network[END_REF][START_REF] Mostafa | Supervised learning based on temporal coding in spiking neural networks[END_REF][START_REF] Ponulak | Supervised learning in spiking neural networks with ReSuMe: sequence learning, classification, and spike shifting[END_REF][START_REF] Neftci | Event-driven contrastive divergence for spiking neuromorphic systems[END_REF], to unsupervised STDP and its variants [START_REF] Diehl | Unsupervised learning of digit recognition using spike-timing-dependent plasticity[END_REF][START_REF] Tavanaei | Acquisition of visual features through probabilistic spike-timingdependent plasticity[END_REF].

Regarding the pursuit of brain inspiration, STDP-based SNNs are the most biologically plausible ones. Using STDP, the network can successfully extract frequently occurring visual features. However, an unsupervised learning rule alone is not sufficient for decision-making, where external classifiers such as support vector machines (SVMs) and radial basis functions (RBFs), or supervised variants of STDP, are usually required. Digit recogni-tion is one of the well-known pattern recognition problems which has became a standard task to examine the newly proposed methods [START_REF] Kulkarni | Spiking neural networks for handwritten digit recognitionsupervised learning and network optimization[END_REF][START_REF] Hochuli | Handwritten digit segmentation: Is it still necessary?[END_REF][START_REF] Chang | Deep unsupervised learning with consistent inference of latent representations[END_REF]. There are several STDP-based SNNs that have been applied to MNIST dataset for digit recognition. For example, Brader et. al. [START_REF] Brader | Learning real-world stimuli in a neural network with spike-driven synaptic dynamics[END_REF] with 96.5%, Querlioz et. al. [START_REF] Querlioz | Immunity to device variations in a spiking neural network with memristive nanodevices[END_REF] with 93.5%, and Diehl and Cook [START_REF] Diehl | Unsupervised learning of digit recognition using spike-timing-dependent plasticity[END_REF] with 95% of recognition accuracies are successful models the use shallow structure in digit recognition. With a deeper structure, Beyer et. al. [START_REF] Beyeler | Categorization and decision-making in a neurobiologically plausible spiking network using a STDP-like learning rule[END_REF] achieved a not so good performance of 91.6%, however, Kheradpisheh et. al. [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF] trained a deep convolutional SNN (DCSNN) and increased the accuracy to 98.4%.

Researchers have started exploring the potential of using reinforcement learning (RL) in SNNs and DCNNs [START_REF] Dayan | Reward, motivation, and reinforcement learning[END_REF][START_REF] Daw | The computational neurobiology of learning and reward[END_REF][START_REF] Niv | Reinforcement learning in the brain[END_REF][START_REF] Lee | Neural basis of reinforcement learning and decision making[END_REF][START_REF] Steinberg | A causal link between prediction errors, dopamine neurons and learning[END_REF][START_REF] Schultz | Neuronal reward and decision signals: from theories to data[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF]. By RL, the learner is encouraged to repeat rewarding behaviors and avoid those leading to punishments [START_REF] Sutton | Introduction to reinforcement learning[END_REF]. Using supervised learning, the network learns at most what the supervisor knows, while with RL, it is able to explore the environment and learn novel skills (unknown to any supervisor) that increase the fitness and reward acquisition [START_REF] Silver | Mastering the game of go without human knowledge[END_REF].

We previously developed a shallow SNN with a single trainable layer [START_REF] Mozafari | First-spike-based visual categorization using reward-modulated stdp[END_REF], where the plasticity was governed by reward-modulated STDP (R-STDP). R-STDP is a reinforcement learning rule inspired by the roles of neuromodulators such as Dopamine (DA) and Acetylcholine (ACh) in modulation of STDP [START_REF] Frémaux | Neuromodulated spiketiming-dependent plasticity, and theory of three-factor learning rules[END_REF][START_REF] Brzosko | Sequential neuromodulation of hebbian plasticity offers mechanism for effective reward-based navigation[END_REF]. Our network made decisions about categories of objects solely based on the earliest spike in the last layer without using any external classifier. Together with rank-order encoding and at most one spike per neuron, our network was biologically plausible, fast, energyefficient, and hardware-friendly with acceptable performance on natural images. However, its shallow structure made it inappropriate for large and complex datasets with high degrees of variations.

In this research, we designed a 3-layer DCSNN with a structure adopted from [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF], mainly for digit recognition. The proposed network does not need any external classifier and uses a neuron-based decision-making layer trained with R-STDP. First, the input image is convolved with difference of Gaussian (DoG) filters at various scales. Then, by an intensity-to-latency encoding [START_REF] Gautrais | Rate coding versus temporal order coding: a theoretical approach[END_REF], a spike wave is generated and propagated to the next layer. After passing through multiple convolutional and pooling layers with neurons that are allowed to fire at most once, the spike wave reaches the last layer, where there are decision-making neurons that are pre-assigned to each digit. For each input image, the neuron in the last layer with the earliest spike time or maximum potential indicates the decision of the network.

We evaluated our DCSNN on the MNIST dataset for digit recognition. First, we applied R-STDP only to the last trainable layer and STDP to the first two, achieving 97.2% of recognition performance. Next, we investigated if applying R-STDP to the penultimate trainable layer is helpful. We found that when there are frequent distractors in the input and limitations on the computational resources, using R-STDP instead of STDP in the penultimate layer is beneficial.

The rest of this paper is organized as follows. A precise description of the components of the proposed network, synaptic plasticity, and decisionmaking is provided in Section 2. Then, in Sections 3 and 4, the performance of the network in digit recognition and the results of applying R-STDP to multiple layers are presented. Finally, in Section 5, the proposed network is discussed from different points of view and the possible future works are highlighted.

Methods

The focus of this paper is to train and tune a DCSNN by a combination of biologically plausible learning rules, i.e. STDP and R-STDP, for digit recognition. To this end, we modify the network proposed in [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF] by dropping its external classifier (SVM) and adding a neuron-based decision-making layer, as well as using R-STDP for synaptic plasticity in the higher layers.

The original implementation of the proposed network has done in C++/CUDA and the source code would be available upon request. We have provided a Python1 script which is a re-implementation of this work using our PyTorch2 -based framework called SpykeTorch (available on GitHub3 ) [START_REF] Mozafari | Spyketorch: Efficient simulation of convolutional spiking neural networks with at most one spike per neuron[END_REF].

Overall Structure

The proposed network has six layers, that are three convolutional layers (S1, S2, S3), each followed by a pooling layer (C1, C2, C3). To convert MNIST images into spike waves, they are convolved with six DoG filters and encoded into spike times by the intensity-to-latency encoding scheme. Plasticity of the afferents of convolutional layers is done by either STDP or R-STDP learning rules. The ultimate layer (C3) is a global pooling in which neurons are pre-assigned to each digit. These neurons are indicators for the network's decision. Figure 1 plots the outline of the proposed network.

Details of the input spike generation, functionality of each layer, learning rules, and decisionmaking process are given in the rest of this section.

Input Spike Waves

On each input image, On-and Off-center DoG filters of three different scales are applied with zero padding. Window sizes are set to 3 × 3, 7 × 7, and 13 × 13, where their standard deviations (σ 1 , σ 2 ) are (3/9, 6/9), (7/9, 14/9), and (13/9, 26/9), respectively. We keep the ratio of 2 between two sigmas in each scale. These values are chosen practically based on the statistics of input images.

From the output of the DoG filters, all the values below 50 are ignored and the remaining values are descendingly sorted, denoting the order of spike propagation.

In order to speed up running time of our parallel implementation of the network, we distribute the ordered input spikes equally into a fixed number of bins. All the spikes in each bin are propagated simultaneously. In te ns ity -t o -L at en cy en co d in g

Figure 1: Overall structure of the proposed DCSNN for digit recognition. The input image is convolved by six DoG filters (on-and off-center in three different scales) and converted into spike latencies based on intensity-to-latency encoding scheme. Generated spikes are processed through three spiking convolution-then-pooling layers (S1 -C1,S2 -C2, and S3 -C3). All of the spiking convolution layers are trainable, employing either STDP or R-STDP learning rules. The network makes its decision in C3, where neurons are pre-assigned to digits, and the decision is the digit assigned to the neuron with either the maximum internal potential or the earliest spike time. Regarding the decision, a reward/punishment (reinforcement) signal will be generated globally, which is received by R-STDP-enabled layers.

Convolutional Layers

Each convolutional layer (S-layer) in the proposed network contains several 2-dimensional grids of Integrate-and-Fire (IF) neurons, that constitute the feature maps. Each neuron in a S-layer has a fixed 3-dimensional input window of afferents with equal width and height, and a depth equal to the number of feature maps in the previous layer. The firing threshold is also set to be the same across all the neurons in a single layer.

In each time step, the internal potential of each IF neuron is increased by the incoming spikes within its input window using the magnitude of the corresponding synaptic weights. These neurons have no leakage and if a neuron reaches the firing threshold, it will emit a single spike, after which remains silent until the next input image is fed to the network. For each new input, the internal potentials of all neurons are reset to zero. Note that a weight-sharing mechanism is applied for all neurons in a single feature map.

Pooling Layers

Pooling layers (C-layers) in our network are employed for introducing position invariance and re-ducing information redundancy. Each C-layer has the same number of feature maps as its previous Slayer and there is a one-to-one association between maps of the two layers.

There are two types of pooling layers in our network: spike-based and potential-based. Both types have a 2-dimensional input window and a particular stride. Each neuron in the spike-and potentialbased C-layers, indicates the earliest spike time and the maximum potential of the neurons within its input window, respectively.

Decision-Making and Reinforcement Signal

As mentioned before, neurons in the final C-layer (C3) perform global pooling on their corresponding S3 grids. These neurons are labeled with input categories such that each C3 neuron is assigned to single category, but each category may be assigned to multiple neurons. Using the labeled neurons in C3, the decision of the network for each input is the label of the neuron with the earliest spike time or the maximum internal potential depending on the type of pooling layer. When the decision of the network has been made, it will be compared with the original label of the input image. If they match (or mismatch), a reward (or punishment) signal will be generated globally. This signal will be received by layers that employ R-STDP rule for synaptic plasticity (see the next section).

Plasticity

STDP is an unsupervised learning rule which functions based on the ordering of pre-and postsynaptic spikes [START_REF] Gerstner | A neuronal learning rule for sub-millisecond temporal coding[END_REF][START_REF] Bi | Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type[END_REF]. For each pair of pre-and post-synaptic spikes, if the ordering is pre-thenpost (or post-then-pre), then the synapse will be potentiated (or depressed). STDP is useful to learn repetitive spike patterns among a large set of incoming spike trains [START_REF] Masquelier | Spike timing dependent plasticity finds the start of repeating patterns in continuous spike trains[END_REF][START_REF] Gilson | STDP allows fast rate-modulated coding with poisson-like spike trains[END_REF][START_REF] Brette | Computing with neural synchrony[END_REF][START_REF] Masquelier | Stdp allows close-to-optimal spatiotemporal spike pattern detection by single coincidence detector neurons[END_REF][START_REF] Masquelier | Optimal localist and distributed coding of spatiotemporal spike patterns through stdp and coincidence detection[END_REF]. R-STDP is a reinforcement learning rule, in which the STDP rule takes the role of neuromodulators into account [START_REF] Frémaux | Neuromodulated spiketiming-dependent plasticity, and theory of three-factor learning rules[END_REF]. Environmental feedback can be encoded into the release of neurotransmitters. Unlike STDP, which is blind to these feedbacks, R-STDP modifies the polarity of weight changes based on the released neuromodulators. For instance, the release of DA can be considered as a rewarding feedback that encourages the neuron's behavior. Conversely, the release of ACh can signal punishment and result in discouraging the neuron's behavior [START_REF] Brzosko | Sequential neuromodulation of hebbian plasticity offers mechanism for effective reward-based navigation[END_REF]. For differences between STDP and R-STDP in action, we refer the readers to [START_REF] Mozafari | First-spike-based visual categorization using reward-modulated stdp[END_REF].

Here, we use either STDP or R-STDP to train each of the trainable layers. We define a general formula that can be employed for both rules as follows:

δ ij =        αφ r a + r + βφ p a - p if t j -t i ≤ 0, αφ r a - r + βφ p a + p if t j -t i > 0, or neuron j never fires (1)
∆w ij = δ ij (w ij )(1 -w ij ), (2)
where i and j refer to the post-and pre-synaptic neurons, respectively, ∆w ij is the amount of weight change for the synapse connecting the two neurons, and a + r , a - r , a + p , and a - p scale the magnitude of weight change. Furthermore, to specify the direction of weight change, we set a + r , a + p ≥ 0 and a - r , a - p ≤ 0. Our learning rule only needs the sign of the spike time difference, not the exact value, and uses an infinite time window. The other parameters, say φ r , φ p , α, and β, are employed as controlling factors which will be set differently for each learning rule. Moreover, we initially set the synaptic weights with random values drawn from a normal distribution with mean 0.8 and standard deviation 0.02 [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF].

To apply STDP to a S-layer, the controlling factors are φ r = 1, φ p = 0, α = 1, and β = 0. For R-STDP, the values of α and β depends on the reinforcement signal generated by the decision-making layer. If a "reward" signal is generated, then α = 1 and β = 0, whereas if a "punishment" signal is generated, then α = 0 and β = 1. The reinforcement signal can be "neutral" as well, for which α = 0 and β = 0. The neutral signal is useful when we have unlabeled data. Similar to our previous work [START_REF] Mozafari | First-spike-based visual categorization using reward-modulated stdp[END_REF], we apply adjustment factors by φ r = N miss N and φ p = N hit N , where N hit and N miss denote the number of samples that are classified correctly and incorrectly over the last batch of N input samples, respectively. Note that the plasticity rules are applied for each image, while φ r and φ p are updated after each batch of samples.

When implementing plasticity in each trainable layer, we employ a k-winner-take-all mechanism, by which only k neurons are eligible for plasticity. These neurons cannot belong to a same feature map because of the weight sharing strategy through the entire map. Choosing the k winners is first based on the earliest spike times, and then by selecting those with the highest internal potentials. During the training process, when a winner is indicated, a r × r inhibition window, centering the winner neuron, will be applied to all the feature maps, preventing them from being selected as the next winners for the current input image.

Additionally, we multiply the term w ij (1 -w ij ) by δ ij which not only keeps weights between the range [0, 1], but also stabilizes the weight changes as the weights converge. In this task, we train our proposed DCSNN using the 60000 images of handwritten digits from the MNIST dataset. Then the recognition performance of the network is tested over the 10000 unseen samples provided by the author of the dataset.

Network Configuration

The input layer consists of six feature maps corresponding to the six DoG filters (three scales with both on-and off-center polarities), each contains the spike latencies. After that, there are three Slayers, each followed by a C-layer with specific parameters that are summarized in Tables 123.

The first two S-layers have specific thresholds, however the last one has an "infinite" threshold. This means that S3 neurons never fire for an input image. Instead, they accumulate all the incoming spikes as their internal potential. When there are no more spikes to accumulate, a manual spike time, greater than the previous spike times, will be set for all of the S3 neurons.

C1 and C2 neurons perform spike-based local pooling, whereas C3 neurons apply potential-based global pooling which is consistent with the behavior of S3 neurons. Accordingly, the proposed network makes decisions based on the maximum potential among C3 neurons. In other words, the label associated to the C3 neuron with the maximum potential will be selected as the network's decision for each input image.

It is worth mentioning that in this task, we drop the weight stabilizer term from the plasticity equation for layer S3. Instead, we manually limit the weights between 0.2 and 0.8. This modification is done based on our experimental results, where we found that it is better to continue synaptic plasticity even in late iterations.

Training and Evaluation

We trained the network in a layer-by-layer manner. Layers S1 and S2 were trained by STDP for 10 5 and 2×10 5 iterations, respectively. In each iteration, an image was fed to the network and the learning rates were multiplied by 2 after each 500 iterations. We kept doing multiplication while a + r was less than 0.15.

Training of S3 was governed by R-STDP employing the reinforcement signal produced by layer C3. We trained S3 for 4 × 10 7 iterations in total and evaluated the recognition performance after each 6 × 10 4 iterations (i.e. presenting the whole set of training samples). Our network achieved 97.2% of performance, which is higher than most of the previous STDP-based SNNs evaluated on the MNIST dataset (see Figure 2). A comprehensive comparison of SNNs trained on MNIST dataset with various learning rules is provided in [START_REF] Tavanaei | Deep learning in spiking neural networks[END_REF]. It is worth mentioning that our network ranked as the second best even though it does not use an external classifier. This increases not only the biological plausibility, but also the hardware friendliness of the network [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF][START_REF] Mozafari | First-spike-based visual categorization using reward-modulated stdp[END_REF].

We also examined the application of R-STDP to the penultimate S-layer (S2) as well, but this did not improve the performance in this particular task. By analyzing the influence of R-STDP on S2, we hypothesized that it can help extracting target-specific diagnostic features, if there are frequent distractors among the input images and the available computational resources are limited. Our next task is designed to examine this hypothesis.

Comparison to Error Backpropagation

One of the advantages of using STDP in early layers is that it decreases the chance of overfitting by extracting common features. Unlike error backpropagation which is data-hungry, the proposed network performs well even if we reduce the number of training samples. In order to demonstrate this advantage, we compared the performance of our network to a supervised SNN proposed in [START_REF] Mostafa | Supervised learning based on temporal coding in spiking neural networks[END_REF] while training them with varying numbers of training samples. The chosen supervised SNN uses a similar latency coding scheme with exactly one spike per neuron. Moreover, it achieved a similar performance using the whole training set. As shown in Figure 3, it is clear that our network regained almost all of its generalization power even with 20% of the training samples. Conversely, the performance of the supervised SNN was tightly bound to the number of training samples which means achieving the highest accuracy requires the whole training set. Regarding the fact that humans learn with a few number of samples, the proposed network appears more realistic.

Task 2: R-STDP Can Discard Non-Diagnostic Features

The goal of this task is to distinguish two specific digits while other digits are presented as distractors. This task is conducted over a subset of MNIST images employing a smaller network, which allows us to analyze the advantages of applying R-STDP to more than one layer. This part of the paper is started by introducing a handmade problem and a simple two-layer network to solve it, for the sake of a better illustration. Then, we show the same outcome using images of the MNIST dataset.

Handmade Problem

We prepare a set of artificial images of size 9 × 3, each contains two oriented bars in its left and right most 3×3 regions. As shown in Figure 4a , we make all of the possible combinations of bars from four different orientations. We characterize three classes of target inputs by pair-wise combinations of three differently oriented bars (Figure 4b), leaving the other inputs as distractors. All of the inputs are fed to the network in a random order. We design a minimal two-layer spiking network that distinguishes the three target classes, if and only if it learns the three diagnostic oriented lines by the neurons in the penultimate S-layer, i.e. S1 (see Figure 4b). According to the target classes, the horizontal bar carries no diagnostic information and learning this bar is a waste of neuronal resources. To this end, we put three feature maps in layer S1 whose neurons have 3×3 input windows. After a spike-based global pooling layer, we added three neurons that have 1 × 1 × 3 input windows.

We tried both STDP and R-STDP learning rules for training S1, while the last layer was always trained by R-STDP in order to make decisions. Note that when we used R-STDP on two layers, the plasticity of both layers was postponed until arrival of the reinforcement signal, and thus they were trained simultaneously. Moreover, the reinforcement signal for non-target images was "neutral" regardless of the decision of the network. ) that solves a 3-class recognition task with layer S2 as the decision-making layer. Each class is the position invariant combination of two different bars. These classes are defined in a way that the horizontal bar carries no distinguishing information (distractor), while others are vital for final recognition (targets). Since there are only three feature maps in S1, the task is fully solvable if and only if the network learns all the three targets in this layer.

According to the results, the network with STDP-trained S1 failed most of the times to extract the correct orientations. In contrast, training S1 with R-STDP enabled the network to solve the problem without any failure. The obtained results agree with the nature of STDP, which extracts frequent features regardless of the outcome. Since the probability of the appearance of each oriented bar is the same, the chance of extracting all the diagnostic bars is 25% for STDP (the probability of discarding the non-diagnostic feature among the four is 1/4).

We implemented a similar task with the MNIST dataset. In each instance of the task, two digits are selected as the targets and the other digits marked as distractors. The goal is to distinguish the two target digits using a small number of features, while the network receives images of all digits.

Network Configuration

The number of layers and their arrangement is the same as the network in Task 1, however there are several differences. First, MNIST images are fed to the network without application of DoG filters. Second, we put fewer feature maps with different input window sizes and thresholds in all of the Slayers. Third, S3 neurons have specific thresholds, that allow them to fire, thus C3 performs a spikebased global pooling operation. In addition, for each category, there is only one neuron in C3. The values of parameters are summarized in Tables 123.

Training and Evaluation

We performed the task for all pairs of digits ( 102 = 45 pairs), and different number of feature maps in S2, ranging from 2 to 40. For each pair, the network was trained over 10 4 samples from the training set and tested over 100 testing samples of each target digit (200 testing samples in total). S1 was separately trained by STDP, while on S2 both STDP and R-STDP were examined. Plasticity in S3 was always governed by R-STDP.

For each particular number of feature maps in S2, we calculated the average recognition performance of the network for both cases of using STDP and R-STDP over all of the digit pairs. As shown in Figure 5, the results of the experiments clearly support our claim. As we have fewer neuronal resources, using R-STDP is more beneficial than STDP. However, by increasing the number of feature maps in S2, STDP also has the chance to extract diagnostic features and fills the performance gap. According to Figure 6, R-STDP helps S2 neurons to extract target-aware features. In contrast, using STDP, S2 neurons are blind to the targets and extract the same features for all the pairs (the same random seed is used for all of the tasks).

We acknowledge that the network could achieve higher accuracies if its parameters are tuned for each pair of digits. Besides, in order to get the results in a feasible time for large number of simulations, we had to limit the number of iterations and use faster learning rates which again, degrade the performance.

Discussion

With the development of DCNNs, AI is now able to solve sophisticated problems such as visual object recognition, in which humans used to be the masters [START_REF] Rawat | Deep convolutional neural networks for image classification: A comprehensive review[END_REF]. Most of the early DCNNs employ supervised learning techniques to be trained for a specific task. However, supervised learning limits the learner to the information provided by the supervisor. Recently, researchers have found that reinforcement learning could be the new game changer and started to combine DCNNs with reinforcement learning techniques, also known as deep reinforce-ment learning. This combination has introduced networks that are able to achieve super-human skills by finding solutions that have never been tested by a human before [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF][START_REF] Silver | Mastering the game of Go with deep neural networks and tree search[END_REF][START_REF] Silver | Mastering the game of go without human knowledge[END_REF].

Despite the revolutionary performance of DC-NNs in machine vision, SNNs have increasingly being used to solve complex vision tasks. SNNs are not yet as impressive as DCNNs, however, they possess a great spatio-temporal capacity by which they are able to fill the performance gap or even bypass DCNNs. Additionally, SNNs have been shown to be energy efficient and hardware friendly, which makes them suitable to be deployed in AI hard- As the number of feature maps increases, STDP has more chance to extract features for target digits, therefore approaches the performance of applying R-STDP. With respect to the number of feature maps, R-STDP shows a monotonic behavior in increasing the performance and confidence level, however, STDP is not as robust as R-STDP, specially in terms of confidence level.

ware [START_REF] Furber | Large-scale neuromorphic computing systems[END_REF][START_REF] Davies | Loihi: A neuromorphic manycore processor with on-chip learning[END_REF] for online on-chip learning.

In online learning, no training set is provided a priori, or the environment is so dynamic that the prior knowledge would not be enough to solve the task. The goal of an online learner is to improve its performance and adapt to the new conditions while it is experiencing various circumstances or observing different instances in an online manner. There are problems for which using online learning is a must such as, anomaly and fraud detection, analysis of financial markets, and online recommender systems, to mention a few [START_REF] Hoi | Online learning: A comprehensive survey[END_REF].

Therefore, online learning on hardware (online on-chip learning) needs fast and hardware-friendly algorithms. Because of high-precision and timeconsuming operations for supervised error backpropagation, DCNNs are hardly an appropriate choice for this type of learning. However, this has motivated researchers to modify the error backpropagation algorithm in order to make it more hardware-friendly [START_REF] Neftci | Event-driven random back-propagation: Enabling neuromorphic deep learning machines[END_REF]. Conversely, computation and communication with spikes can be extremely fast and applicable in low precision platforms [START_REF] Rueckauer | Conversion of continuous-valued deep networks to efficient event-driven networks for image classification[END_REF]. Moreover, biologically inspired learning rules such as STDP and R-RSTDP has proved to be quite efficient on real-time hardware [START_REF] Yousefzadeh | Hardware implementation of convolutional stdp for on-line visual feature learning[END_REF].

STDP has been successfully applied to SNNs for digit recognition [START_REF] Diehl | Unsupervised learning of digit recognition using spike-timing-dependent plasticity[END_REF], however, because of its unsupervised nature, a supervised readout is required to achieve high accuracies [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF]. Apart from the fact that supervised readouts (e.g. SVMs) are not biologically supported, they are not appropriate for hardware implementation.

In this paper, we presented a DCSNN for digit recognition which is trained by a combination of STDP and R-STDP learning rules. R-STDP is a biologically plausible reinforcement learning rule [START_REF] Frémaux | Neuromodulated spiketiming-dependent plasticity, and theory of three-factor learning rules[END_REF], appropriate for hardware implementation as well. This combination is inspired by the brain where the synaptic plasticity is mostly unsupervised, taking into account the local activity of pre-and post-synaptic neurons, but it can be modulated by the release of neuromodulators, reflecting the feedbacks received from the surrounding environment [START_REF] Gerstner | A neuronal learning rule for sub-millisecond temporal coding[END_REF][START_REF] Bi | Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type[END_REF][START_REF] Frémaux | Neuromodulated spiketiming-dependent plasticity, and theory of three-factor learning rules[END_REF][START_REF] Brzosko | Sequential neuromodulation of hebbian plasticity offers mechanism for effective reward-based navigation[END_REF].

The proposed network extracts regularities in the input images by STDP in lower layers and uses R-STDP in higher layers to learn rewarding behavior (accurate decision-making). R-STDP enables us to use a neuron-based decision-making layer instead of complex external classifiers. In this network, information is encoded in the earliest spike time of neurons. The input image is convolved with multiple DoG filters and converted into spike latencies by the intensity-to-latency encoding scheme. Spike propagation as well as plasticity are done in a layerwise manner except that all the R-STDP-enabled layers are trained simultaneously. When the spike wave reaches the decision-making layer, the label of the neuron with either the maximum potential or the earliest spike time will be selected as the network's decision. Comparing the decision to the original label of the input, a reward/punishment (reinforcement) signal is globally generated which modulates the plasticity in R-STDP-enabled trainable layers.

First, we applied our network to the whole MNIST dataset employing R-STDP only on the last trainable layer. Our network achieved 97.2% of recognition accuracy which is higher than most of the previously proposed STDP-based SNNs. We illustrated that unlike the supervised error backpropagation which is data-hungry, the proposed network achieves almost its ultimate recognition power even with a small portion of the training samples (∼ 20%). Next we investigated if applying R-STDP to more than one layer is helpful, by allowing the penultimate trainable layer to use R-STDP as well. Reviewing the results of a handful of different experiments revealed that if the input set is polluted with frequent distractors, applying R-STDP to more than one layer helps ignoring distractors and extracting diagnostic features using lower computational resources than a STDP-enabled layer.

The proposed network inherits bio-plausability, energy efficiency and hardware friendliness from its ancestors [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF][START_REF] Mozafari | First-spike-based visual categorization using reward-modulated stdp[END_REF], but is now able to solve harder and complex recognition tasks thanks to its deeper structure. Our network has two fundamental features that elevate its energy efficacy in comparison to the other deep networks, particularly DCNNs: (1) Communication with spikes: Using spike/nospike regime, the accumulation of the incoming spikes can be implemented by energy-efficient "accumulator" units. In the other, since DC-NNs use rate-based encoding, their hardware implementation needs energy-consuming "multiplyaccumulator" units. (2) At most one spike per neuron: SNN hardware can be implemented in an event-driven approach, where spikes are considered as events. Using at most one spike per neuron, while most neurons do not fire at all, results in minimal energy consumption. SNNs with single spike per neuron have got more popularity recently. Apart from the energy-efficiency, they have shown to be competitive if the parameter values are tuned well for the target task [START_REF] Falez | Multi-layered spiking neural network with target timestamp threshold adaptation and stdp[END_REF][START_REF] Vaila | Deep convolutional spiking neural networks for image classification[END_REF].

In order to improve the performance of the proposed network, we tested various configurations and values for parameters. Here, we survey our findings regarding the results of these tests:

Deciding based on the maximum potential in Task 1. We tried decision-making based on the earliest spike in the first step, but the results were not competing. By exploring the missclassified samples, we found that the digits with common partial features are the most problematic ones. For instance, digits 1 and 7 share a skewed vertical line (1 is mostly written skewed in MNIST). Let's say a neuron has learned to generate the earliest spike for digit 1. Obviously, it will generate the earliest spike for many samples of digit 7 as well, resulting in high rate of false alarms. Thus, we decided to omit the threshold and consider the maximum potential as the decision indicator. Since the network has to wait until the last spike, this approach increased the computation time, however it solved the aforementioned problem.

R-STDP is only applied to the last layer in Task 1. As we showed in Task 2 and our previous work [START_REF] Mozafari | First-spike-based visual categorization using reward-modulated stdp[END_REF], R-STDP can be better than STDP when computational resources are limited. In Task 1, the goal was to increase the performance of the network to an acceptable and competing level. Since MNIST is a dataset with high range of variations for each digit, we had to put enough feature maps in each layer to cover them. As a consequence, the large number of feature maps and the fact that there were no distractors in the inputs, STDP worked well in extracting useful intermediate features.

Learning in the ultimate layer is slower than the penultimate one in Task 2. In Task 2, S2 and S3 were trained simultaneously. If S3 modifies synaptic weights as fast as S2, each change in S2 may change a previously good behavior of S3 into a bad one, which then ruin S2, and this cycle may continue forever, destroying stability of the network.

Conclusion

In this paper, we proposed a bio-realistic DCSNN trained by the combination of STDP and R-STDP which decides on the category of an input image using neurons in its ultimate layer. Applying STDP to the lower layers helped the network extracting simple frequent features while the application of R-STDP to the higher layers guided the network toward extracting complex diagnostic features as the combinations of early features.

Using three trainable layers and at most one spike per neuron, the proposed network reached 97.2% of recognition performance on MNIST dataset and ranked as the second best among other STDP-based networks. We showed that, unlike the supervised counterpart, our network needs much less training data (∼ 20%) to reach its ultimate performance. Besides, employing R-STDP on multiple layers helps the network to avoid frequent nondiagnostic features and extract task-specific ones by neurons in the mid-layer as well as the last layer.

We believe that our DCSNN can be further improved from multiple aspects. As mentioned before, potential-based decision-making degrades the computational efficiency. One solution to overcome false alarms, due to common partial features, while having spike-based decision-making is to engage inhibitory neurons. Inhibitory neurons can suppress a neuron if it receives something more than the preferred stimulus (e.g. the horizontal line of digit 7 would inhibit those neurons responsible for digit 1, thus producing no false alarm). Another aspect would be performance improvement by designing a population-based decision-making layer. For instance, one can consider a group of earliest spikes instead of one and apply major voting, which may affect the training strategy as well. Another direction for future work would be investigation of the layer-wise application of R-STDP for intermediate layers where the reward/punishment signal can be generated based on other activity measures such as sparsity, information content, and diversity.

RIn

  ew ar d/ P un is hm en t S ig na l -w in d o w (r ea l va lu e) C o nv -w in d o w (s p ik in g ) P o o lin g -w in d o w

3 Task 1 :

 1 Solving 10-class Digit Recognition

Figure 2 :

 2 Figure 2: Recognition performance of several available SNNs trained by STDP-based learning rules on the MNIST dataset, as well as the proposed network.

Figure 3 :

 3 Figure 3: Comparison of recognition performances of the proposed network and a supervised SNN [20] while training them with restricted training set.

Figure 4 :

 4 Figure 4: (a) Input set for the handmade problem. It contains all combinations of bars in four different orientations. For each input, spikes are simultaneously propagated from the white tiles. A 3 × 3 gap is placed between the two oriented bars in order to prevent neurons seeing parts of the two bars at the same time. (b) A simple network with two trainable layers (S1 and S2) that solves a 3-class recognition task with layer S2 as the decision-making layer. Each class is the position invariant combination of two different bars. These classes are defined in a way that the horizontal bar carries no distinguishing information (distractor), while others are vital for final recognition (targets). Since there are only three feature maps in S1, the task is fully solvable if and only if the network learns all the three targets in this layer.

  -0.003 0.0005 -0.004 1 0

Figure 5 :

 5 Figure5: Comparison of recognition performance when STDP or R-STDP is applied to S2 in Task 2. Superiority of R-STDP over STDP in finding diagnostic features (and discarding distractors) is clearly shown when there are small number of feature maps in S2. As the number of feature maps increases, STDP has more chance to extract features for target digits, therefore approaches the performance of applying R-STDP. With respect to the number of feature maps, R-STDP shows a monotonic behavior in increasing the performance and confidence level, however, STDP is not as robust as R-STDP, specially in terms of confidence level.

Figure 6 :

 6 Figure 6: Reconstruction of features extracted by S2 neurons in Task 2, when the number of feature maps is limited by four. The lower part of the figure illustrates features that are extracted by applying R-STDP to S2 afferents, solving each of the pair-wise digit recognition tasks (corresponding to row and column indices). The existence of non-trained or semi-trained feature maps (dark or gray squares) is due to the fact that other features that are representative enough, win the plasticity competition for all images, leaving no chance for others to learn or converge. In the upper part, the only four features obtained by applying STDP to S2 are shown.

Table 1 :

 1 Values of the S-layer parameters used in each digit recognition task.

		Layer	Number of Feature Maps	Input Window (Width, Height, Depth)	Threshold
	Task 1	S1 S2 S3	30 250 200	(5, 5, 6) (3, 3, 30) (5, 5, 250)	15 10 ∞
	Task 2	S1 S2 S3	10 x ∈ {2, 4, ..., 20} ∪ {30, 40} 2	(5, 5, 1) (15, 15, 10) (1, 1, x)	5 40 0.9

Table 2 :

 2 Values of the C-layer parameters used in each digit recognition task.

		Layer	Input Window Stride (Width, Height)	Type
	Task 1	C1 C2 C3	(2, 2) (3, 3) (5, 5)	2 3 0	Spike-Based Spike-Based Potential-Based
	Task 2	C1 C2 C3	(2, 2) (14, 14) (1, 1)	2 0 0	Spike-Based Spike-Based Spike-Based

Table 3 :

 3 Values of parameters for the synaptic plasticity used in each digit recognition task.

https://www.python.org/

https://pytorch.org/

https://github.com/miladmozafari/SpykeTorch

Acknowledgment

This research received funding from Iran National Science Foundation: INSF (No. 96005286), the European Research Council under the European Union's 7th Framework Program (FP/2007-2013)/ERC Grant Agreement no. 323711 (M4 project), and has been supported by the Center for International Scientific Studies & Collaboration (CISSC) and French Embassy in Iran. The authors thank NVIDIA GPU Grant Program for supporting computations by providing a high-tech GPU.


Eric Thivant

Eunika Mercier-Laurent

Guilaine Talens

Developing a Knowledge Base on Climate Change for Metropolitan cities

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Studying climate change requires to take into consideration multiple factors that influence it. Finding and understanding the degree of importance of these factors as well as their interinfluences could provide indicators on how to apply these principles in growing metropolitan cities. Public, semi-public & private actors (research laboratories, government agencies, etc.), associations, NGOs, mobilize and participate in new discoveries in related fields and experiment; some of them involve citizens (Smart Cities, Enoll, Triple Helix, etc.).

The Metropolis of Lyon case Study:

French Metropolitan Cities have developed their PCETs program (Territorial Climate and Energy Plan) to meet the objectives set by the European Union for 2020 and the National Climate Plan, set out in the Grenelle I and Grenelle II laws in France. In 2007, as part of its Climate Energy Plan, the Metropolis of Lyon made a commitment to reduce CO2 emissions by 20% by 2020 (75% by 2050), to cut energy consumption by 20% and to consume 20% of renewable energies. This article propose defining and initiate an knowledge base on climate change and more specifically on the WEEE (waste electrical and electronic equipment). We focus on large Metropolitan cities such as Metropole of Lyon or Metropole of Montpellier in order to develop such a Knowledge base. It will contain best (and bad) practices, experiences, suited metrics, collaborative measures, projects, etc., using the Knowledge Management approach based on artificial intelligence techniques.

The Metropolis of

State of Art or related works on KM & Ontology

Concerning our state of art, we first define what is an ontology and then we concentrate on the SCO (Smart City Ontology).. Since 90s some researchers in "ingénierie des connaissances" field want to develop new SCB system and propose to use ontology to resolve SCB problems.

[GRUB 93:199] explains to us that "A body of formally represented knowledge is based on a conceptualization: the objects, concepts and other entities that are presumed to exist in some area of interest and the relationships that hold among them (Genesereth & Nilsson, 1987). A conceptualization is an abstract, simplified view of the world that we wish to represent for some purpose. Every knowledge base, knowledge-based system or knowledge-level agent is committed to some conceptualization, explicity or implicitly".

[GRUB 93:199] continue his thinking and define the concept of ontology : "An ontology is an explicit specification of a conceptualization. The term is borrowed from philosophy, where an ontology is a systematic account of Existence. For Knowledge-based systems, what exists is exactly that which can be represented. When the knowledge of a domain is represented in a declarative formalism, the set of objects, that can be represented is called the universe of discourse. This set of objects and the formalized relationships among them, are reflected in the representational vocabulary with which a knowledge-based program represents knowledge. Thus we can describe the ontology of a program by defining a set of representational terms."

Inside the ontology, "definitions associate the names of entities in the universe of discourse (e.g. classes, relations, functions, or other objects) with human readable text describing what the names are meant to denote, and formal axioms that constrain the interpretation and wellformed use of these terms."

The definition of Studer [START_REF] Studer | Knowledge Engineering: Principles and Methods[END_REF] based on Gruber [START_REF] Gruber | A translation approach to portable ontologies[END_REF] is:" An ontology is a formal, explicit specification of a shared conceptualization". 'Formal' refers to the fact that the ontology should be machine understandable, excludes natural language. 'Shared' reflects the notion, an ontology captures consensual knowledge, that is, it is not private to some individual, but accepted by a group".

Methodology

For our methodology we work on the state of the art for building such a knowledge base, taking into account the works of D. Boulanger (2005), E. Mercier-Laurent (2015, 2018) or DG.

Kayakutlu & E. Mercier-Laurent (2016) on that particular field, and we begin to list existing materials in order to implement in a near future a little knowledge base, on the recycling of products, particularly electronic products for these metropolitan cities.

For example, [ MER 11] has developed the concept of ontology of innovation… innovation has In order to build our ontology, we can use the following set of 5 criteria proposed by Gruber [START_REF] Gruber | Toward Principles for the Design of Ontologies Used for Knowledge Sharing[END_REF] : Clarity, Coherence, Extendibility, minimal encoding bias, Minimal ontological commitment. Clarity means that ". Coherence means "An ontology should effectively communicate the intended meaning of defined terms. Definitions should be objective."; For Coherence: "An ontology should be coherent: that is, it should sanction inferences that are consistent with the definitions. At the least, the defining axioms should be logically consistent".; Extendibility: "An ontology should be designed to anticipate the uses of the shared vocabulary. It should offer a conceptual foundation for a range of anticipated tasks, and the representation should be crafted so that one can extend and specialize the ontology monotonically". Minimal encoging bias "The conceptualization should be specified at the knowledge level without depending on a particular symbol-level encoding. An encoding bias results when a representation choice are made purely for the convenience of notation or implementation."; Minimal ontological commitment: "An ontology should require the minimal ontological commitment sufficient to support the intended knowledge sharing activities. An ontology should make as few claims as possible about the world being modeled, allowing the parties committed to the ontology freedom to specialize and instantiate the ontology as needed".

Presentation of the SCO (smart city Ontology)

Nicos Komninos*, Charalampos Bratsas, Christina Kakderi and Panagiotis Tsarchopoulos (2015) have developed a Smart City Ontology. This Ontology is specifically built for Smart Cities, because there are a lot of ontologies existing already and they want to propose a new on more clear… The authors have reviewed the existing ontologies for cities such as SOFIA, NOW, SCRIBE, etc. and then they define the buildings blocks of this ontology with respect to the most cited definitions of smart and structuring this ontology with the Protégé 5.0 editor, defining entities, class hierarchy and object properties and data type properties. review also the building blocks of smart or intelligent cities, which allows to define the class and properties of the SCO.

According to the authors: " The first version (v01) contains 10 superclasses, 708 entities, 422 classes, 62 object properties, 190 data properties, and 27 individ-uals from the software application class. Additionally, widely adopted extra ontologies are used, enriching the SCO, such as the Simple Knowledge Organization System (SKOS), a W3C recommendation designed for representation of thesauri, classification schemes, taxonomies, subject-heading systems, or any other type of structured controlled vocabulary [START_REF] Mercier-Laurent | Artificial Intelligence for Knowledge Management -5th IFIP WG 12.6 International Workshop, AI4KM 2017, Held at IJCAI 2017[END_REF]." You can see below in the graph the structure of the first level of the ontology. In [GYRAR 2018], smart city ontologies are presented and compared in order to be applied to IoT ontologies. The set of criteria are mainly focused on the reusability of the ontologies. The goal is to define a methodology to enrich smart city catalogs with the new ontologies found. It's implemented within the LOV4IoT ontology catalog. The aim of this catalog is to help developers in reusing existing smart city and IoT ontologies to build future applications.

ONTOLOGIES FOR CLIMATE CHANGE

[Pilegi and Lamia 2020] build an ontology to describe the climate change story from multiple perspectives, including scientific, social, political and technological ones.

In this article, several ontologies for climate are cited, ontologies to measure the impact of a changing climate on food and waterborne diseases, the impacts of agriculture on water resources, … but there is none ontology to capture the recycling of products.

Proposal of adding recycling in the SCO

Eco-design a key element for innovation

[Evbuomwan et al., 1996] defined the eco-design like "the process of establishing requirements based on human needs, transforming them into performance specification and functions, which are mapped and converted (subject to constraints) into design solutions (using creativity, scientific principles and technical knowledge) that can be economically manufactured and produced".

[ Lindhqvist, 1990] proposes the Extended Producer Responsibility (EPR) concept and change the classical view of PPP (the polluter pay for the pollution, from the consumer to the economic agent who produce the goods). According to the Organisation for Economic Co-operation and Development (OECD), EPR is "an environmental policy approach in which a producer's responsibility for a product is extended to the post-consumer stage of a product's life cycle". (Report of Development of Guidance on Extended Producer Responsibility (EPR), European Commission -DG Environment, 2014).

"The economic reasoning behind the EPR concept is to have producers internalise treatment and disposal cost so that they have an incentive to design products that last longer and are more easily treated after use".

Definitions of the EPR System (Development of Guidance on Extended Producer Responsibility)

EPR system or EPR scheme: Any system set up by one or several producers to implement the EPR principle. It can be an individual system (or individual compliance scheme) when a producer organises its own system, or a collective system (collective compliance scheme) when several producers decide to collaborate and thus transfer their responsibility to a specific organisation (a PRO).

Producer Responsibility Organisation or PRO: Entity set up in collective EPR schemes to implement the EPR principle in the name of all the adhering companies. We propose to add the recycling theme to this SCO ontology, with the following classes & proprieties from the WEEE domain, It takes different types of organizations, of products and ways of managing Electronics waste.

Conclusion

Our proposal for a new ontology of this domain, with different classes and proprieties is not yet finish. We need to continue our work with a case-based analysis & application in order to explain better our classes and the relationships between them.

  Lyon has launched a "territorial climate-energy plan" (PCET), i.e. a territorial sustainable development project, whose primary purpose is to combat climate change in 2007. The first plan was designed for the period 2010-2020. Then Lyon Metropole has voted a new plan in December 2019, called "Plan climate air energy territorial" (PCAET) for the period 2020-2030. "The metropolis of Lyon is working with its 145 partners to take effective action throughout its territory: the Climate and Territorial Energy Plan, which brings together local authorities, industrial and energy producers, companies in the tertiary sector, research laboratories and associations. » The Lyon Metropole has 3 objectives to achieve: -17% renewable energy in 2030 -43% CO2 emissions compared to 2000 -30% &energy consumption compared to 2000

Figure 1 :

 1 Figure 1: Ontology of Innovation [MER 11]

Figure 2 :

 2 Figure 2: An new view of the Smart City Ontology of Professor Nicos Komninos & al. (2020)

Fees:

  Tariff paid by a producer to have its products dealt with through a PRO.According to the WEEE Directive (2012/19/EU), there are 7 Categories of EEE :1. Large household appliances 2. Small household appliances 3. IT and telecommunications equipment 4. Consumer equipment and photovoltaic panels 5. Lighting equipment 6. Electrical and electronic tools (with the exception of large-scale stationary industrial tools) 7. Toys, leisure and sports equipment For each categories of EEE, we should develop our own view on WEEE.5.2 Ecosystem of the recycling circuitNatalia Alonso Movilla presents us the physical and financial flows of the WEEE industry below.

Figure 3 :

 3 Figure 3: Physical flows of WEEE Industry, from Mérot (2014) and modified by Natalia Alonso Movilla

Figure 4 :

 4 Figure 4: Flows of WEEE Indutry, from Mérot (2014) modified by Natalia Alonso Movilla's thesis

Figure 5 :

 5 Figure 5: process of dismantling from Natalia Alonso Movilla's thesis