domain: info phys

text:
Erwann Martin

Maxence Ernoult

Jérémie Laydevant

Shuai Li

Damien Querlioz

Teodora Petrisor

Julie Grollier
email: julie.grollier@cnrs-thales.fr

EqSpike: Spike-driven Equilibrium Propagation for Neuromorphic Implementations

Finding spike-based learning algorithms that can be implemented within the local constraints of neuromorphic systems, while achieving high accuracy, remains a formidable challenge. Equilibrium Propagation is a promising alternative to backpropagation as it only involves local computations, but hardware-oriented studies have so far focused on rate-based networks. In this work, we develop a spiking neural network algorithm called EqSpike, compatible with neuromorphic systems, which learns by Equilibrium Propagation. Through simulations, we obtain a test recognition accuracy of 97.6% on MNIST, similar to rate-based Equilibrium Propagation, and comparing favourably to alternative learning techniques for spiking neural networks. We show that EqSpike implemented in silicon neuromorphic technology could reduce the energy consumption of inference and training respectively by three orders and two orders of magnitude compared to GPUs. Finally, we also show that during learning, EqSpike weight updates exhibit a form of Spike Timing Dependent Plasticity, highlighting a possible connection with biology.

Introduction

Spike-based neuromorphic systems have, in recent years, demonstrated outstanding energy efficiency on inference tasks [START_REF] Merolla | A million spiking-neuron integrated circuit with a scalable communication network and interface[END_REF]. Implementing the training of deep neural networks in such systems remains, however, a considerable challenge, as backpropagation does not apply directly to spiking networks and requires spatially non-local computations that go against the principles of neuromorphic systems. A large number of neuromorphic systems use the unsupervised and biologically-inspired Spike Timing Dependent Plasticity (STDP) learning rule because its weight updates, based on the relative timing of pre-and post-synaptic spikes, are spatially local and can be achieved with compact circuits in several technologies [START_REF] Bi | SYNAPTIC MODIFICATION BY CORRELATED ACTIVITY: Hebb's Postulate Revisited[END_REF][START_REF] Masquelier | Unsupervised Learning of Visual Features through Spike Timing Dependent Plasticity[END_REF][START_REF] Bichler | Extraction of temporally correlated features from dynamic vision sensors with spike-timing-dependent plasticity[END_REF][START_REF] Linares-Barranco | On Spike-Timing-Dependent-Plasticity, Memristive Devices, and Building a Self-Learning Visual Cortex[END_REF][START_REF] Jo | Nanoscale Memristor Device as Synapse in Neuromorphic Systems[END_REF][START_REF] Pedretti | Memristive neural network for on-line learning and tracking with braininspired spike timing dependent plasticity[END_REF][START_REF] Serb | Unsupervised learning in probabilistic neural networks with multi-state metaloxide memristive synapses[END_REF][START_REF] Prezioso | Spike-timing-dependent plasticity learning of coincidence detection with passively integrated memristive circuits[END_REF][START_REF] Thakur | Large-Scale Neuromorphic Spiking Array Processors: A Quest to Mimic the Brain[END_REF][START_REF] Feldmann | All-optical spiking neurosynaptic networks with self-learning capabilities[END_REF]. Unfortunately, STDP weight updates generally do not minimize a global objective function for the network, and the accuracy of STDP-trained neural networks remains below state-of-the-art algorithms based on the error backpropagation [START_REF] Falez | Unsupervised visual feature learning with spike-timing-dependent plasticity: How far are we from traditional feature learning approaches?[END_REF]. Important research efforts therefore investigate how the error backpropagation algorithm can be mathematically modified to make it spatially local and appropriate for spiking neural networks [START_REF] Neftci | Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines[END_REF][START_REF] Sacramento | Dendritic cortical microcircuits approximate the backpropagation algorithm[END_REF][START_REF] Richards | A deep learning framework for neuroscience[END_REF][START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks[END_REF][START_REF] Kaiser | Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)[END_REF][START_REF] Bellec | A solution to the learning dilemma for recurrent networks of spiking neurons[END_REF][START_REF] Payeur | Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits[END_REF]. The derived learning rules are composed of three factors. The first two take into account, as usual, the behaviour of pre and post neurons, and the third allows for the introduction of an additional error factor. This third factor leads to implementations on neuromorphic chips that are less compact, and possibly less energy efficient, than two-factor learning rules such as STDP [START_REF] Payvand | Error-triggered Three-Factor Learning Dynamics for Crossbar Arrays[END_REF].

In this work, we propose a different approach to training spiking neural networks with high accuracy while using a local, two-factors learning rule compatible with neuromorphic implementations and scalable to complex tasks. Instead of starting from a non-local algorithm such as backpropagation and modifying it to make it local, we start from a rate-based algorithm called Equilibrium Propagation [START_REF] Scellier | Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation[END_REF] that is intrinsically local in space, and features key advantages for neuromorphic implementations [START_REF] Kendall | Training End-to-End Analog Neural Networks with Equilibrium Propagation[END_REF][START_REF] Zoppo | Equilibrium Propagation for Memristor-Based Recurrent Neural Networks[END_REF]. Equilibrium Propagation theoretically applies to any physical system whose dynamics derive from an energy function. By minimizing the energy of such a system on data patterns, it can be made to relax towards states of minimal error prediction with respect to targets [START_REF] Scellier | Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation[END_REF]. The weight updates of Equilibrium Propagation match those of Back-Propagation-Through-Time (BPTT) in recurrent neural networks with static inputs [START_REF] Ernoult | Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input[END_REF], and it reaches high accuracy on image benchmarks such as CIFAR-10 (Laborieux et al., 2020). Equilibrium Propagation uses the same set of weights for the forward and backward pass, a feature that is not biologically plausible, but is interesting for neuromorphic computing as it decreases the number of synaptic devices to update, thus reducing the overall power consumption. Contrarily to backpropagation, Equilibrium Propagation uses the same computations in the forward and backward phases, which is another highly desirable feature for neuromorphic systems as it greatly simplifies the circuits. Equilibrium propagation is, however, originally a rate-based algorithm.

Here, we design a spiking, hardware-friendly version of Equilibrium Propagation, called EqSpike, compatible with current neuromorphic technologies achieving online learning [START_REF] Schemmel | A wafer-scale neuromorphic hardware system for large-scale neural modeling[END_REF][START_REF] Furber | The SpiNNaker Project[END_REF][START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses[END_REF][START_REF] Davies | Loihi: A Neuromorphic Manycore Processor with On-Chip Learning[END_REF][START_REF] Frenkel | A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS[END_REF][START_REF] Ishii | On-Chip Trainable 1.4M 6T2R PCM Synaptic Array with 1.6K Stochastic LIF Neurons for Spiking RBM[END_REF][START_REF] Park | A 65-nm Neuromorphic Image Classification Processor With Energy-Efficient Training Through Direct Spike-Only Feedback[END_REF]. EqSpike is local in space and time: contrarily to backpropagation, neither error gradients nor activations need to be stored in external memories, and synapses can be directly updated through neural events. We simulate a fully connected network based on this architecture on the MNIST handwritten digits database. We obtain a test recognition accuracy of 97.6%, which compares favourably with spiking neural networks learning with backpropagation-derived methods, and on par with rate-based Equilibrium Propagation. We show that EqSpike can be implemented in silicon neuromorphic technology, and thus reduce the energy consumption of inference by up to three orders of magnitude and training by up to two orders of magnitude compared to graphics processing units (GPUs). Finally, we also show that during learning the weight updates of EqSpike exhibit a form of STDP, yielding insights to its link to biology.

EqSpike: a hardware-friendly spiking version of Equilibrium propagation

Equilibrium Propagation is an algorithm for training convergent recurrent neural networks. Input neurons are clamped to a static input and all the other neurons, bi-directionally connected through synapses, evolve dynamically in time to reduce the energy of the network [START_REF] Scellier | Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation[END_REF]. The algorithm functions in two phases: a free phase and a nudging phase. In the free phase, performing inference, the network is let to reach equilibrium (Fig. 1a). Once this is done, inputs are kept clamped, and output neurons are nudged towards the desired output (Fig. 1b). During this nudging phase, the prediction error at the output layer is converted into a "force" acting upon output neurons and propagating to the rest of the system through time until a second equilibrium is reached. For training, synaptic values are updated by probing the neuron states after [START_REF] Scellier | Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation[END_REF] or during [START_REF] Ernoult | Equilibrium Propagation with Continual Weight Updates[END_REF] the nudging phase through a learning rule that has been shown theoretically and numerically to match the updates of Back-Propagation Through Time, the state-of-the-art algorithm for such recurrent neural networks [START_REF] Ernoult | Updates of Equilibrium Prop Match Gradients of Backprop Through Time in an RNN with Static Input[END_REF]. It has been shown recently that Equilibrium Propagation also reaches accuracy within 1% of BPTT with convolutional architectures on the CIFAR-10 dataset [START_REF] Laborieux | Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias[END_REF].

The original version of Equilibrium Propagation uses a rate-based formulation where dynamical neurons evolve smoothly in time. For a network of leaky integrate and fire neurons described by an Hopfield-like energy function

= 1 2 ⁄ ∑ -1 2 ⁄ ∑ -∑
, where are the membrane potentials of neurons and their activation function, the Equilibrium Propagation learning rule is: ∆ ~ -, where the product is measured at equilibrium, at the end of the nudge phase and the free phase [START_REF] Scellier | Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation[END_REF]. This rule can be extended to the case when weights are continuously updated during the nudging phase [START_REF] Ernoult | Equilibrium Propagation with Continual Weight Updates[END_REF]:

~ + , (Eq.1)
where is the synaptic weight connecting neurons i and j, and , are the rates of the two neurons. The network dynamics thus directly compute the error derivative, encoded in the rate derivative of the post neuron and multiplied by the activation function of the pre neuron. The reformulation of this rate-based learning rule to a spiking neural network is therefore the following: each time neuron i spikes, the weight should be updated by a quantity proportional to the derivative of the rate of neuron j, (first term in Eq. 1), and reciprocally.

We propose here a simple strategy, compatible with current electronic hardware, to implement this learning rule. It is illustrated in Fig. 1c in the form of a circuit, including spike detection elements at the output of each neuron, as well as dedicated blocks that extract the rate derivative from the spike trains of each neuron in real-time, in order to update synapses accordingly.

We use leaky-integrate-and-fire (LIF) spiking neurons which output spike frequency as a function of input current approximates the hard sigmoid prescribed in the original formulation of Equilibrium Propagation [START_REF] Scellier | Equilibrium Propagation: Bridging the Gap between Energy-Based Models and Backpropagation[END_REF]. Their maximum frequency is !"# = 1/% &' &"( , where % &' &"( is the refractory time of the LIF neuron (see Supplementary Information for details and all parameter values).

The novelty compared to standard SNNs is the scheme that we propose for extracting the rate acceleration for each neuron, illustrated in Fig. 1e. A leaky-integrator with a leak factor ) *+ (without reset nor spikes), takes as input the spike train emitted by the neuron to which it is connected and outputs a slowly varying signal proportional to the rate of the neuron spike train: , *+ ~ /) *+ [START_REF] Navarro | Sodium channels implement a molecular leaky integrator that detects action potentials and regulates neuronal firing[END_REF]. To take the derivative, we delay this signal by a duration -, and subtract the actual value with the delayed value: , '."/ = , *+ 0 -, *+ 0 --≅ -23 45 2 ∝ -/) *+ . We then apply a low-pass filter for smoothing the variations. The filter is simulated using an average over 8 . simulation steps: 9 0 :::::: =

; < = >? ∑ 9 0 -@ A0 < = >? B; CD
, where A0 is the simulation time step.

The output of the filter, that approximates ̅ , which corresponds to an effective learning rate

J & = H & E F 45 = 1.5 10 -3 .
This approach is hardware-compatible as LIF neurons, leaky integrators, delays and low pass filters are circuit elements that can be efficiently implemented in CMOS technology (Mead and Ismail, 1989), and bidirectional synapses could be implemented with CMOS compatible emergent nanodevices such as memristors [START_REF] Ishii | On-Chip Trainable 1.4M 6T2R PCM Synaptic Array with 1.6K Stochastic LIF Neurons for Spiking RBM[END_REF][START_REF] Marković | Physics for neuromorphic computing[END_REF][START_REF] Wan | [END_REF]. The corresponding pseudo-code is given in Algorithm 1 (see Supplementary Information for details).

Algorithm 1: EqSpike learning procedure for one image Inputs: input image, Model K LM N , K P ' , K QM , Loss function, length Free phase % &'' , length Nudging phase % M R' , Parameters ) *+S , ) *+ , P , T, H & , -, 8 . , for 0 < % &'' : ∎ free phase for each neuron W:

Update membrane potential

) *+S , X if > P : Emit a spike (0 ) Update 0 , ) *+ for 0 ∈ [% &'' , % &'' + % M R' ]: ∎ nudging phase for each output neuron ]: Compute error gradient ∇_ Q Nudge neuron: Q ← Q -T ⋅ ∇_ Q for each neuron k: Update b ) *+S , X b if b > P : Emit a spike (0 b Update b 0 b , ) *+ Compute smoothed: ̅ b b 0 b , b 0 b --, … , 8 .
for each synapse I : ∎ Update synapses if neuron j emits a spike:

I ← I + H & ⋅ E F 45 ̅
if neuron i emits a spike:

I ← I + H & ⋅ E F 45 ̅ Return:
Trained weights for input image: and go to next image/next epoch.

Full network simulations: recognition rate on handwritten digits database

We now evaluate the performance of EqSpike on the MNIST handwritten digits classification task, using a fully connected network with one hidden layer (see Supplementary Information for details).

The obtained train (orange) and test (blue) accuracies are shown Fig. 2 as a function of the number of training epochs, with the deviation over six runs in shadow color. Table 1 compares the results to BPTT and the version of Equilibrium Propagation closest to our implementation, called Continual Eq-Prop [START_REF] Ernoult | Equilibrium Propagation with Continual Weight Updates[END_REF], trained with a batch size of one.

The test accuracy of EqSpike matches closely the accuracy of stochastic gradient descent through BPTT on the same network architecture, given the error margin. With a hidden layer of 300 neurons, EqSpikes reaches a test accuracy of 97.59%. Fully-connected spiking neural networks trained on MNIST without conversion from a non-spiking neural network typically achieve recognition rates in the 96-98% range [START_REF] Neftci | Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines[END_REF][START_REF] O'connor | Deep Spiking Networks[END_REF][START_REF] Lee | Training Deep Spiking Neural Networks Using Backpropagation[END_REF][START_REF] Mostafa | Supervised Learning Based on Temporal Coding in Spiking Neural Networks[END_REF][START_REF] Tavanaei | BP-STDP: Approximating backpropagation using spike timing dependent plasticity[END_REF]. EqSpike, with its local, two-factors online learning rule therefore reaches accuracies on MNIST comparable to those of the latest models investigated for training spiking neural networks on hardware platforms. We have chosen the MNIST dataset as a benchmark because it is a standard dataset for the neuromorphic community interested in training spiking neural networks online, due to the long simulation times. As EqSpike achieves results on MNIST equivalent to the baseline given by Equilibrium Propagation it has the potential, like Equilibrium Propagation, to adapt to convolutional architectures and perform with good accuracy on more complex image benchmarks [START_REF] Laborieux | Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias[END_REF], with the additional advantage of being compatible with current neuromorphic technologies.

Inference speed and energy

As EqSpike is derived from a rate-based algorithm, it is interesting for neuromorphic applications to quantify the number of spikes needed to achieve inference, and the time needed to reach high accuracy. Operation with fewer spikes is more desirable, as it reduces both execution time and energy consumption.

Fig. 3 shows the inference results as a function of the execution time multiplied by the maximum frequency of neurons (0 × !"# ). The orange line is the mean accuracy result over the whole test dataset, obtained by computing the spike rate of output neurons (as done for Fig. 2 and Table 1), through averaging in a time window % "e'&"R' of 100 simulation time steps (% "e'&"R' = 50/ !"# and considering that the neuron with highest frequency encodes the output. This method computes the rate accurately at the expense of having to wait the time % "e'&"R' and letting output neurons spike multiple times. Spiking neural networks also offer the possibility to accelerate the computation and reduce the energy consumption by determining the output class from the first output neuron to spike. The blue line in Fig. 3 is the accuracy as a function of time, averaged over all images in the test dataset, obtained by considering that the first output neuron to spike encodes the output.

The red, vertical dotted line in Fig. 3 indicates the average time of the first spike at the output over all presented images, corresponding to t ≅ 3.5/ !"# . At t ≅ 10/ !"# , the accuracy of single-spike inference (blue curve) reaches 95.11% ± 0.78%, within 1.4% of the precise rate computation (orange curve). This result shows that even though the algorithm is originally rate-based, a single spike at the output suffices in most cases to determine the correct class with good precision, a feature which is highly attractive for energy-efficient inference on neuromorphic chips. It means that inference can be achieved in 100 µs for electronic neurons with a firing rate of 100 kHz, available in neuromorphic chips working in accelerated time compared to biology [START_REF] Schemmel | A wafer-scale neuromorphic hardware system for large-scale neural modeling[END_REF], and 1 µs for electronic neurons with a firing rate of 10 MHz that can be produced, for example, with emerging nanotechnologies [START_REF] Li | High-endurance megahertz electrical self-oscillation in Ti/NbOx bilayer structures[END_REF]. The corresponding throughputs are respectively 10k and 1M images/s, on par with current spiking neural network implementations [START_REF] Pfeiffer | Deep Learning With Spiking Neurons: Opportunities and Challenges[END_REF][START_REF] Park | A 65-nm Neuromorphic Image Classification Processor With Energy-Efficient Training Through Direct Spike-Only Feedback[END_REF]. As the network operations are fully parallel, these orders of magnitudes will be conserved for wider networks. Simulations of rate-based Equilibrium Propagation on deeper networks indicate that the convergence time increases by a factor of about eight for a network with four hidden layers compared to a network of one hidden layer as here [START_REF] Laborieux | Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias[END_REF]. It should be noted that in the current implementation we present static inputs to the network, which means that input neurons need to integrate these signals before they emit the first spikes that will then propagate to the next layers. The speed of inference could be increased in the future by presenting inputs directly encoded in spikes, for example sourced from neuromorphic vision sensors [START_REF] Pfeiffer | Deep Learning With Spiking Neurons: Opportunities and Challenges[END_REF]).

An estimation of the energy consumption of a spiking neural network on a neuromorphic silicon chip can be performed by counting the number of synaptic operations involved. Synaptic operations (SynOps) are defined as the total number of spikes transiting through synapses of the network.

Frenkel et al show that a SynOp on a neuromorphic chip requires as little as 10 pJ [START_REF] Frenkel | A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS[END_REF]. The total number of synaptic operations needed for inference depends on the targeted recognition precision and, therefore, on the duration of inference (Fig. 3). For EqSpike, the recognition rate saturates at t ≅ 10/ !"# . The corresponding measured number of SynOps is about 150,000 in average. This is much less than expected if all neurons spiked. This is also a bit less but comparable to the number of SynOps needed at inference for Event-Driven Random BackPropagation [START_REF] Neftci | Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines[END_REF]. Considering 10 pJ/SynOps, each EqSpike inference could potentially consume 1.5 µJ. This means that testing the 10,000 images of the whole MNIST dataset could be achieved with a neuromorphic chip while consuming only 15 mJ, in other words, three orders of magnitude less than with a GPU (Joseph and Nagarajan, no date).

In our current EqSpike implementation, the input layer is the one leading to most spikes and SynOps: with only 16% of illuminated pixels in average in MNIST, the input layer emits 87.5% of all spikes and 98.6% of SynOps occur between the input layer and the hidden layer. In this work, we did not focus on reducing the number of spikes with encoding, but a better encoding of the input may reduce considerably the energy consumption. Kheradpisheh et al have shown that with a temporal encoding, the total number of spikes in the network before the first output spike can be reduced to 200 with a hidden layer four times larger than our network [START_REF] Kheradpisheh | Temporal Backpropagation for Spiking Neural Networks with One Spike per Neuron[END_REF]. In our case, 678 spikes in total are emitted in average before the first output spike. An adaptation of EqSpike to temporal encoding is not straightforward, but this number could potentially be decreased in the future by reducing the encoding frequency of the input.

Training speed and energy

Training with EqSpike requires performing the free phase and then the nudging phase, during which synaptic weights are updated. A way to speed-up the training, and reduce the total number of SynOps, is to perform the nudging phase only on poorly classified examples, and skip the updates when the accuracy is satisfactory. We apply this strategy inspired from (Park, Lee and Jeon, 2020) using the criteria that the nudging phase is performed only when the difference between the target rate and the actual rate b hb at the ouput is above 1%. Fig. 4a shows the number of presented examples per epoch as a function of epoch number. In the last 20 epochs only approximately 15% of the training dataset still require a nudging phase.

For MNIST, we thus perform the free phase on all the dataset (3×10 6 images for the 50 epochs), and the nudging phase on 489,000 images. Given the durations of each phase, we can estimate the training time to Ttraining ≅ 2.74×10 8 / !"# . For electronic neurons with a firing rate of 100 kHz [START_REF] Schemmel | A wafer-scale neuromorphic hardware system for large-scale neural modeling[END_REF], this leads to Ttraining ≅ 45 min, and for electronic neurons with a firing rate of 10 MHz [START_REF] Li | High-endurance megahertz electrical self-oscillation in Ti/NbOx bilayer structures[END_REF] to Ttraining ≅ 30s. As our networks feature a fully parallel nature, these training times would be the same for much wider networks, and increased by a factor of about eight only with four hidden layers [START_REF] Laborieux | Scaling Equilibrium Propagation to Deep ConvNets by Drastically Reducing its Gradient Estimator Bias[END_REF].

As EqSpike is derived from a rate-based approach, it is interesting to compare the actual spiking rates of neurons in the network during training to their maximum frequency !"# . For neuromorphic applications, low overall rates are indeed desirable. Fig 4b shows the average number of spikes emitted by each neuron for an image presentation in the training dataset, as a function of the epoch. We found that for the training conditions of Fig. 2, there are in average 36 spikes/neuron/image. This means that neurons in the network spike in average with a frequency of the order of 20% of !"# , well below !"# , which is promising for neuromorphic implementations. Again this number could be reduced in the future by optimizing the encoding of input at the first layer. Fig. 4c shows the numbers of synaptic operations needed for training as a function of recognition rate. The total number of synaptic operations after 50 epochs is of 4.23×10 12 , which is of the same order of magnitude as Event-Driven Random Back-Propagation [START_REF] Neftci | Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines[END_REF] for similar accuracy, and below training MNIST with BP based on ( 14). With 10pJ per SynOps [START_REF] Frenkel | A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS[END_REF] the training phase of EqSpike on a neuromorphic chip could consume as little as 42 J, again, two orders of magnitude less than with a GPU (Joseph and Nagarajan, no date).

Spike Timing Dependent Plasticity

We have shown that EqSpike transforms Equilibrium Propagation into an efficient algorithm for neuromorphic chips. We now highlight that it also brings Equilibrium Propagation closer to biological plausibility. Bengio et al have pointed out a connection between the Equilibrium Propagation learning rule of Eq. 1 and STDP [START_REF] Bengio | STDP-Compatible Approximation of Backpropagation in an Energy-Based Model[END_REF]. The STDP learning rule, illustrated in Fig. 5a, reinforces causality between the spikes of pre-and post-synaptic neurons in networks with unidirectional synapses. If the post-synaptic neuron spikes after the pre-, causality is observed, and the weight is increased. In the opposite scenario, the weight is decreased.

Let us consider a situation where the pre-synaptic neuron i spikes and the post-synaptic neuron j accelerates, as illustrated in Fig. 5b. According to the Eq-Prop learning rule, in a network with unidirectional synapses, ∝ = LQN L&' , a positive weight update should be applied.

Due to the acceleration of the post-neuron, there are less post-neuron spikes before the pre-neuron spike than after. Therefore, tpost -tpre is positive in average, yielding a positive weight update through STDP.

We have investigated if STDP-like weight updates did emerge during learning in our simulations. For this purpose, we monitored weight variations in synapses that connect input neurons and the hidden layer neurons. These synapses are unidirectional as input neurons are clamped to the input (see Fig. 1): their frequency does not vary. We used 100 images during the first epoch for the MNIST dataset.

The curve in Fig. 5c shows the average weight updates in the first layer as a function of the time difference between post-synaptic neuron spikes and the average time of pre-synaptic neuron spikes in a window of 200 time steps before the post-synaptic neuron spike. The obtained curve, centred on zero, indeed exhibits an STDP-like shape. It has been obtained by filtering out very low frequencies below 0.05. Quiet neurons in the free phase indeed induce large weight updates at the beginning of the nudging phase, due to the sudden acceleration from zero to non-zero frequency, inducing an additional noise in the curve. It should be noted that biological STDP curves being frequencydependent, they are also often obtained by focusing on a given range of frequencies [START_REF] Kirkwood | Experience-dependent modification of synaptic plasticity in visual cortex[END_REF]. Fig. 5c shows that the STDP amplitude and time-window vary with the strength of nudging T. In Equilibrium Propagation, weight changes are driven by neuron accelerations. As nudging applies a force that accelerates neurons, for larger T, larger weight changes are obtained, leading to a larger amplitude of the STDP-like curve. As higher accelerations are reached during the nudging phase, noticeable weight modifications occur for larger differences between tpost and tpre, corresponding to a wider time window for the STDP-like curve. These results confirm the possible connection between STDP and Equilibrium Propagation pointed out in [START_REF] Bengio | STDP-Compatible Approximation of Backpropagation in an Energy-Based Model[END_REF], despite the fact that individual spike timings are lost through averaging. They show that STDP-like behaviour can be obtained during learning in unidirectional synapses without a direct implementation of the original, causality-based rule. They also ask the question whether multilayer networks could be trained with the local STDP learning rule by using the nudging procedure of Equilibrium Propagation, possibly in the extended version called "Vector-field Equilibrium Propagation" in which synapses are unidirectional [START_REF] Scellier | Generalization of Equilibrium Propagation to Vector Field Dynamics[END_REF].

Discussion

Other versions of Equilibrium Propagation have been specifically designed to train spiking neural networks. O'Connor et al have designed a spiking network which, by construction, stochastically approximates the dynamics of its rate-based counter-part until reaching the same steady state with a minimal spikes communication budget between neurons (O'Connor, Gavves and Welling, 2019). Their technique, successfully tested against MNIST, comes at the cost of combining predictive coding, sigma-delta modulation and adaptive step sizes at the neuron level. In comparison, our technique simply requires LIF neurons, a spike-count and a low-pass filter to implement the learning rule.

Mesnard et al proposed a version of Equilibrium Propagation to train spiking networks that also makes use of low-pass filters to estimate firing rates. However, they only demonstrate their approach on a non-linear toy problem, and the implemented learning rule is not local in time [START_REF] Mesnard | Towards deep learning with spiking neurons in energy based models with contrastive Hebbian plasticity[END_REF]. In contrast, we demonstrate the effectiveness of our fully event-based implementation on MNIST.

More generally, several spike-based approaches to backpropagation have been proposed. One method consists in smoothing the spikes as a function of time so that gradients can be backpropagated through time [START_REF] Huh | Gradient Descent for Spiking Neural Networks[END_REF][START_REF] Bellec | A solution to the learning dilemma for recurrent networks of spiking neurons[END_REF]. Another technique consists in gating the spikes propagating the error signals by surrogate derivatives, as done in SpikeGrad [START_REF] Thiele | SpikeGrad: An ANN-equivalent Computation Model for Implementing Backpropagation with Spikes[END_REF]. Finally, Event-driven Random Backpropagation uses firing rates in the backward pass, as we do in this paper, and achieves similar performance on MNIST [START_REF] Neftci | Event-Driven Random Back-Propagation: Enabling Neuromorphic Deep Learning Machines[END_REF]. In comparison with these approaches however, our implementation of Equilibrium Propagation does not require to know the activation function equation in order to compute its derivatives. One other interesting approach is S4NN, which employs latency coding, where each neuron can spike at most once, and the output is encoded as the first neuron to spike [START_REF] Kheradpisheh | Temporal Backpropagation for Spiking Neural Networks with One Spike per Neuron[END_REF]. In contrast with rate-based coding, latency coding has the potential to save important energy in neuromorphic implementations. However, there is no clear indication of whether S4NN could scale to harder visual tasks, while rate-based Equilibrium Propagation was shown to train deep ConvNets on CIFAR-10 (Laborieux et al., 2020). One intrinsic limitation of EqSpike, however, may stem from the necessity for the system to reach a steady state before the gradient computation phase. This prevents, for now, the classification of time-varying inputs. Payeur et al recently proposed a spike-based approach where top-down error signals are encoded as spike bursts and are multiplexed with bottom-up feedforward signals so that the backward pass and the forward pass can occur simultaneously with minimal disruption [START_REF] Payeur | Burst-dependent synaptic plasticity can coordinate learning in hierarchical circuits[END_REF]. While they show that the rate-based counterpart of their algorithm works on CIFAR-10 and ImageNet, it comes at the cost of employing dendritic network topologies and specialized synapses. Indeed, as noted in the introduction, most backpropagation-derived local learning rules involve a third factor for supervision [START_REF] Payvand | Error-triggered Three-Factor Learning Dynamics for Crossbar Arrays[END_REF]. In this regard, we believe that our EqSpike implementation of Equilibrium Propagation, with only two factors, achieves an optimal trade-off between circuitry complexity and performance.

Finally, Zoppo et al [START_REF] Zoppo | Equilibrium Propagation for Memristor-Based Recurrent Neural Networks[END_REF] and Kendall et al [START_REF] Kendall | Training End-to-End Analog Neural Networks with Equilibrium Propagation[END_REF] have proposed using Equilibrium Propagation for training neuromorphic hardware. However, their implementations remain either rate-based or current-based, and are therefore not directly compatible with spiking neuromorphic chips. EqSpike, on the other hand, could be trained directly on reconfigurable neuromorphic systems with online learning such as SpiNNaker [START_REF] Furber | The SpiNNaker Project[END_REF] and Loihi [START_REF] Davies | Loihi: A Neuromorphic Manycore Processor with On-Chip Learning[END_REF].

Limitations of the study

A limitation of EqSpike is that, with a wrong choice of γ jk or τ combined with low nudging times, fast or brief rate changes cannot be detected, and the corresponding weight modification cannot be applied. This does not seem to be a problem with our experimentations as demonstrated above but could lead to a lower accuracy on more complex problems.

The circuits to compute the rate derivative could be further miniaturized by extracting directly from the membrane potential if neuron models with smoothly varying membrane potential are used [START_REF] Gerstner | Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition[END_REF].

EqSpike could also be sped up by building on dedicated hardware in analog or digital CMOS [START_REF] Schemmel | A wafer-scale neuromorphic hardware system for large-scale neural modeling[END_REF][START_REF] Qiao | A reconfigurable on-line learning spiking neuromorphic processor comprising 256 neurons and 128K synapses[END_REF][START_REF] Thakur | Large-Scale Neuromorphic Spiking Array Processors: A Quest to Mimic the Brain[END_REF][START_REF] Frenkel | A 0.086-mm$^2$ 12.7-pJ/SOP 64k-Synapse 256-Neuron Online-Learning Digital Spiking Neuromorphic Processor in 28-nm CMOS[END_REF][START_REF] Park | A 65-nm Neuromorphic Image Classification Processor With Energy-Efficient Training Through Direct Spike-Only Feedback[END_REF]. Emerging nanotechnologies such as memristive synapses and nanoscale spiking oscillators are compelling candidates to scale up neuromorphic hardware due to their small size, their speed and their low energy consumption [START_REF] Marković | Physics for neuromorphic computing[END_REF][START_REF] Milo | Memristive and CMOS Devices for Neuromorphic Computing[END_REF][START_REF] Sebastian | Memory devices and applications for in-memory computing[END_REF][START_REF] Wang | Resistive switching materials for information processing[END_REF][START_REF] Xi | In-Memory Learning With Analog Resistive Switching Memory: A Review and Perspective[END_REF]. These technologies are typically prone to imperfections such as the device-to-device variability, cycle-to-cycle variability or the non-linearity in the conductance-tovoltage response, which are known to considerably jeopardize learning in memristive neural networks [START_REF] Ishii | On-Chip Trainable 1.4M 6T2R PCM Synaptic Array with 1.6K Stochastic LIF Neurons for Spiking RBM[END_REF][START_REF] Zhang | Neuro-inspired computing chips[END_REF]. Our paper implicitly assumes that the underlying memory technology at use would be linear, deterministic and identical across different synapses. The learning rate J & in EqSpike is indeed about 10 times smaller than typically used in standard gradient descent. This is due to the fact that the batch size is one, and also that a low J & is needed to avoid too large modifications accumulated over all the nudging phase. For low precision device as synapse, a first possible solution is to update the synaptic devices less frequently (not at each spike), but with a higher value. Another strategy is to adapt to EqSpike the training schemes used for binary neural networks [START_REF] Hubara | Binarized Neural Networks[END_REF][START_REF] Rastegari | XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks[END_REF][START_REF] Hirtzlin | Hybrid Analog-Digital Learning with Differential RRAM Synapses[END_REF]. Further study should be done to propose a fully end-to-end circuit to implement EqSpike and investigate its resilience to the memristive device imperfections mentioned.

Conclusion

In this work we present a new algorithm for spiking neural networks, EqSpike, compatible with neuromorphic systems, and achieving good performance on MNIST. We show that EqSpike implements the learning rule of Equilibrium Propagation locally and autonomously. The gradients are computed by the dynamics of the system and the weights are modified by a spike and the addition of only one block to the neuron. This can lead to spiking neuromorphic systems that do not need an external circuit to compute the error gradients given by backpropagation and learn autonomously, simply by presenting inputs and nudging the outputs according to errors. Our method obtained results on MNIST close to backpropagation through time and Equilibrium Propagation, two state-ofthe-art algorithms. Moreover, because EqSpike is based on Equilibrium Propagation, the performance on more complex task, like CIFAR-10, could be similar. The number of synaptic operations to obtain these results in MNIST show we can obtain theoretically two orders of magnitude less energy consumption than a GPU for training and the same magnitude of time with high frequency neurons. The inference, after training the network, can be accelerated by waiting for the first output spike rather than compute the highest rate, with only a small loss of accuracy.

Finally, we show that the weight updates of EqSpike share similarity with STDP during learning, raising the question of a possible biological plausibility of the algorithm. In average, the modification of weight is proportional to the spike timing. This could permit to implement synapses in neuromorphic hardware by emergent nano-devices with a STDP-like behavior, to obtain a lower power consumption and higher surface density. [START_REF] Bi | SYNAPTIC MODIFICATION BY CORRELATED ACTIVITY: Hebb's Postulate Revisited[END_REF]. b) Illustration of the link between Eq-Prop and STDP learning rules; illustration reproduced from [START_REF] Bengio | STDP-Compatible Approximation of Backpropagation in an Energy-Based Model[END_REF]. c) STDP-like curve during EqSpike learning.

  then multiplied by the coefficient H & . The corresponding weight updates are ∆I = H & E F 45

Figure 1 -

 1 Figure 1 -EqSpike: spike-driven Equilibrium Propagation a) Schematic of the free phase in Equilibrium Propagation. b) Schematic of the nudging phase in Equilibrium Propagation. c) Illustration of the weight update implementation in EqSpike. d) Spiking rate of the neuron as a function of the amplitude of the input signal. e) Schematic of the rate acceleration computation.

Figure 2 -

 2 Figure 2 -Recognition accuracy during training (orange) and test (blue) as a function of the number of epochs for MNIST, averaged over six runs.

Figure 3 -

 3 Figure 3 -Inference time: recognition accuracy on MNIST on the test dataset as a function of time multiplied by the maximum neuron frequency fmax. Orange line: recognition accuracy computed from the output neuron showing the highest rate. Blue line: recognition accuracy computed from the output neuron spiking first. Red, vertical dotted line: average time of first output spike.

Figure 4 -

 4 Figure 4 -Training performance. a) Number of presented images in the nudging phase per epoch versus epoch number. b) Number of spikes/neuron/image occurring during the two phases (nudge+free), as a function of the epoch. c) SynOps: number of spikes during both phases (nudge+free) as a function of the recognition accuracy.

Figure 5 -

 5 Figure 5 -STDP. a) Illustration of the STDP learning rule; reproduction with data from[START_REF] Bi | SYNAPTIC MODIFICATION BY CORRELATED ACTIVITY: Hebb's Postulate Revisited[END_REF]. b) Illustration of the link between Eq-Prop and STDP learning rules; illustration reproduced from[START_REF] Bengio | STDP-Compatible Approximation of Backpropagation in an Energy-Based Model[END_REF]. c) STDP-like curve during EqSpike learning.

Table 1 -Comparison between BPTT, C-EP and EqSpike, with

 1

the same initialization procedure. Batch size = 1.

Acknowledgements

This publication has received funding from the European Union's Horizon 2020 research innovation programme under grant agreement 732642 (ULPEC project) and 876925 (ANDANTE project)". E. Martin received funding from the ANRT grant 2018/0884.

Data and code availability

The data generated by this study is available on reasonable request.

Author contributions

JG, TP and DQ have devised and supervised the study, EM has performed the spiking simulations with the help of ME, JL and SL. ME and JL have performed the benchmark rate-based simulations. All authors have contributed to writing the article.

Declaration of interests

The authors declare no competing interests.

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, Julie Grollier (julie.grollier@cnrs-thales.fr).

Materials availability

N.A.