Romain Zimmer

Thomas Pellegrini

Srisht Fateh Singh

Timothée Masquelier

Technical report: supervised training of convolutional spiking neural networks with PyTorch

Recently, it has been shown that spiking neural networks (SNNs) can be trained efficiently, in a supervised manner, using backpropagation through time. Indeed, the most commonly used spiking neuron model, the leaky integrate-andfire neuron, obeys a differential equation which can be approximated using discrete time steps, leading to a recurrent relation for the potential. The firing threshold causes optimization issues, but they can be overcome using a surrogate gradient. Here, we extend previous approaches in two ways. Firstly, we show that the approach can be used to train convolutional layers. Convolutions can be done in space, time (which simulates conduction delays), or both. Secondly, we include fast horizontal connections à la Denève: when a neuron N fires, we subtract to the potentials of all the neurons with the same receptive the dot product between their weight vectors and the one of neuron N. As Denève et al. showed, this is useful to represent a dynamic multidimensional analog signal in a population of spiking neurons. Here we demonstrate that, in addition, such connections also allow implementing a multidimensional send-on-delta coding scheme. We validate our approach on one speech classification benchmarks: the Google speech command dataset. We managed to reach nearly state-of-the-art accuracy (94%) while maintaining low firing rates (about 5Hz).

INTRODUCTION

Current Artificial Neural Networks (ANN) come from computational models of biological neurons like McCulloch-Pitts Neurons [McCulloch and Pitts, 1943] or the Perceptron [Rosenblatt, 1958]. Yet, they are characterized by a single, static, continuous-valued activation. On the contrary, biological neurons use discrete spikes to compute and transmit information, and spike timing, in addition to the spike rates, matters. SNNs are, thus, more biologically realistic than ANNs. Their study might help understanding how the brain encodes and processes information, and lead to new machine learning algorithms.

SNNs are also hardware friendly and energy efficient if implemented on specialized neuromorphic hardware. These neuromorphic, non von Neumann architectures are highly connected and parallel, require low-power, and collocate memory and processing. Thus, they do not suffer from the so-called "von Neumann bottleneck" due to low bandwith between CPU and memory [Backus, 1978]. Neuromorphic architectures have also received increased attention due to the approaching end of Moore's law. Neuromorphic computers might enable faster, more power-efficient complex calculations and on a smaller footprint than traditional von Neumann architectures. (See [START_REF] Schuman | A survey of neuromorphic computing and neural networks in hardware[END_REF] for a survey on neuromorphic computing and neural networks in hardware).

Neuroscientists have proposed many different, more or less complex, models to describe the dynamics of spiking neurons. The Hodgkin-Huxley neuron [HODGKIN and HUXLEY, 1952] models ionic mechanisms underlying the initiation and propagation of action potentials. More phenomenological models such as the leaky integrate-and-fire model with several variants e.g. the quadratic integrate and fire model, adaptive integrate and fire, and the exponential integrate-and-fire model have proven to be very good at predicting spike trains despite their apparent simplicity [START_REF] Jolivet | Integrate-and-fire models with adaptation are good enough: Predicting spike times under random current injection[END_REF]. Other models such as Izhikevich's neuron model [Izhikevich, 2003] try to combine the biological plausibility of Hodgkin-Huxley-type dynamics and the computational efficiency of integrate-and-fire neurons. See [Burkitt, 2006] for a review of the integrate and fire neuron models.

However, these models have been designed to fit experimental data and cannot be directly used to solve real life problems.

LITERATURE REVIEW

Various models of spiking neural networks for machine learning have alreay been proposed.

Recurrent Spiking Neural Networks (RSNNs) have been trained to generate dynamic patterns or to classify sequential data. They can have one or more populations of neurons with random or trainable connections. The computational power of recurrent spiking neural networks has been theoretically proven in [START_REF] Maass | On the computational power of circuits of spiking neurons[END_REF]] and models such as liquid state machines [START_REF] Maass | Real-time computing without stable states: A new framework for neural computation based on perturbations[END_REF] and Long short-term memory Spiking Neural Networks (LSNNs) [START_REF] Bellec | Long short-term memory and learning-to-learn in networks of spiking neurons[END_REF] have been proposed. Feed forward spiking neural networks have also been studied. [START_REF] Rueckauer | Conversion of continuous-valued deep networks to efficient event-driven networks for image classification[END_REF] de-rives a method to convert continuous-valued deep networks to spiking neural networks for image classification. However, these models only use rate coding.

Spiking neural networks can also be trained directly using spike-timing-dependent plasticity (STDP), a local rule based on relative spike timing between neurons. This is an unsupervised training rule to extract features that can be used by an external classifier. For instance, [START_REF] Kheradpisheh | Stdp-based spiking deep convolutional neural networks for object recognition[END_REF] have built a convolutional SNN trained with STDP and used a Support Vector Machine (SVM) for classification. More recently, [START_REF] Mozafari | First-spike-based visual categorization using rewardmodulated stdp[END_REF], Mozafari et al., 2019] proposed a reward modulated version of the STDP to train a classification layer on top of the STDP network and thus, does not require any external classifier. These networks usually use latency coding with at most one spike per neuron. The label predicted by the network is given by the first spike emitted in the output layer. Backpropagation has also been adapted to this sort of coding, by computing gradients with respect to latencies [Mostafa, 2017, Comsa et al., 2019, Kheradpisheh and Masquelier, 2019]. The "at most one spike per neuron" limit is not an issue with static stimuli (e.g. images), yet it is not suitable for dynamic stimuli like sounds or videos.

Backpropagation through time (BPTT) [Mozer, 1995] cannot be used directly to train spiking neural network because of their binary activation function (see 5.2). The same problem occurs for quantized neural networks [START_REF] Hubara | Quantized neural networks: Training neural networks with low precision weights and activations[END_REF]. However, the gradient of these functions can be approximated. For instance, [START_REF] Bengio | Estimating or propagating gradients through stochastic neurons for conditional computation[END_REF] studies various gradient estimators (e.g. straight-through estimator) for stochastic neurons and neurons with hard activation functions. Binarized networks with with performances similar to standard neural networks have been developed [START_REF] Courbariaux | Binarynet: Training deep neural networks with weights and activations constrained to +1 or -1[END_REF]. They use binary weights and activations, whereas only activations are binary in this project. Yet, their encoding cannot be sparse as they use {-1, 1} as binary values.

These ideas can also be used to train spiking neural networks. [START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks[END_REF] gives an overview of existing approaches and provides an introduction to surrogate gradient methods, initially proposed in ref. [START_REF] Bohte | Error-Backpropagation in Temporally Encoded Networks of Spiking Neurons[END_REF], Esser et al., 2016, Wu et al., 2018, Bellec et al., 2018, Shrestha and Orchard, 2018, Zenke and Ganguli, 2018]. Moreover, [START_REF] Kaiser | Synaptic plasticity dynamics for deep continuous local learning[END_REF] proposes a Deep Continuous Local Learning (DECOLLE) capable of learning deep spatio-temporal representations from spikes by approximating gradient backpropagation using locally synthesized gradients. Thus, it can be formulated as a local synaptic plasticity rules. However, it requires a loss for each layer and these losses have to be chosen arbitrarily. Another approach has been proposed by [START_REF] Huh | Gradient descent for spiking neural networks[END_REF]: they replaced the threshold by a gate function with narrow support, leading to a differentiable model which does not require gradient approximations.

The encoding method used in this project to represent signals with spikes (See Spiking neural networks and event-based sampling) is very similar to the matching pursuit algorithm proposed by S. Mallat [Mallat and Zhifeng Zhang, 1993]. This algorithm adaptively decomposes a signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions. Starting with the raw signal, waveforms are greedily chosen one at a time in order to maximally reduce the approximation error. At each iteration, the projection of the signal on the selected waveform is removed. The algorithm stops when the energy of the remaining signal is small enough. [START_REF] Bourdoukan | Learning optimal spike-based representations[END_REF] used a similar idea to represent efficiently a signal with the activity of a set of neurons. The potential of each neuron depends on the projection of the signal on the direction of the neuron. And each neuron compete to reconstruct the signal. However, for this project, the goal is to classify an input signal and not to reconstruct it. Thus, the goal is to find the most interesting direction for classification and not the ones that best reconstruct the signal. The link between send-ondelta and integrate-and-fire event-based sampling schemes has already been highlighted by [START_REF] Moser | On quasi-isometry of threshold-based sampling[END_REF]. And [Moser, 2016] proposes a mathematical metric analysis of integrate-and-fire sampling. However, they use negative spikes if the "potential" goes under the opposite of the threshold and only consider 1 dimensional input signals.

INTEGRATE AND FIRE NEURON MODELS

LEAKY INTEGRATE AND FIRE (LIF)

In the standard Leaky Integrate and Fire (LIF) model, the sub-threshold dynamics of the membrane of the i t h neuron is described by the differential equation [START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks[END_REF]

τ mem dU i d t = -(U i -U rest ) + R I i (3.1)
where U i (t ) is the membrane potential at time t , U r est is the resting membrane potential, τ mem is the membrane time constant, I i is the current injected into the neuron and R is the resistance. When U i exceeds a threshold B i , the neuron fires and U i is decreased. The -(U i -U rest ) term is the leak term that drives the potential towards U rest .

NON-LEAKY INTEGRATE AND FIRE (NLIF)

If there is no leak, the model is called Non-Leaky Integrate and Fire (NLIF) and the corresponding differential equation is

τ mem dU i d t = R I i (3.2)
Without loss of generality, we will take R = 1 and U r est = 0 in the following.

INPUT CURRENT

The input current can be defined as the projection of the input spikes along the preferred direction of neuron i given by W i , the i t h row of W.

I i = j W i j S in j (3.3)
where S j (t ) = k δ(tt j k ) if neuron j fires at time t = t j 1 , t j 2 , .... Or, it can be a be governed by another differential equation, e.g. a leaky integration of these projections

τ syn d I i d t = -I i + j W i j S in j
where τ syn is the synapse time constant.

In the former, the potential will rise instantaneously when input spikes are received whereas it will increase smoothly in the latter.

SPIKING NEURAL NETWORKS AND EVENT-BASED SAMPLING

RATE VS TEMPORAL CODING

Standard Deep Learning (DL) is based on rate coding models that only consider the firing rate of neurons. The outputs of standard DL models are thus real-valued. However, there is evidence that precise spike timing can play an important role in the neural code [START_REF] Gollisch | Rapid neural coding in the retina with relative spike latencies[END_REF], [START_REF] Moiseff | Neuronal and behavioral sensitivity to binaural time differences in the owl[END_REF], [START_REF] Johansson | First spikes in ensembles of human tactile afferents code complex spatial fingertip events[END_REF]. Furthermore, computing with sparse binary activation can require much less computing power than traditional real-valued activation. We wanted to create a model based on precise spike timing with an efficient and sparse encoding of the information. However, there is no commonly accepted theory on how real neurons encode information with spikes. Thus, we based our approach on even-based sampling theory and explained how it can be related to spiking neural networks.

SEND-ON-DELTA

Most of the time, the input signal is real-valued and has to be encoded as spike trains. This can be done using an event-based sampling strategy. In this work, the Send-On-Delta (SoD) sampling strategy [Miskowicz, 2006] is used.

The SoD strategy is a threshold-based sampling strategy. The sampling is triggered when a significant change is detected in signal x, i.e. the :

t k = inf t ≥t k-1 {t , |x(t ) -x(t k-1 )| ≥ ∆} (4.1)
where t k is the time of the k t h sampling event. Changes in the signal can be either an increase or a decrease (See Fig. 4.1). This strategy is used by event-based cameras and is very efficient to remove temporal redundancy as sampling will only occur if the input signal changes.

SEND-ON-DELTA WITH INTEGRATE AND FIRE NEURONS

This encoding scheme can be achieved by two NLIF neurons with lateral connections and whose input is the derivative of the signal.

Let,

I (t ) = w ẋ(t ) and U (t + k ) = 0 with t + k (resp. t - k
) the time just after (resp. before) the k t h spike has been emitted and w a scaling factor.

Then, for the IF model we have, If the threshold is B = w 2 , the next spike is emitted at t + k+1 such that

U (t ) = w(x(t ) -x(t k ))
t k+1 = inf t ≥t k {t , U (t ) ≥ w 2 } = inf t ≥t k {t , sign(w)(x(t ) -x(t k )) ≥ |w|}
Depending on the sign of w, the neuron will detect an increase or a decrease of at least |w| in the signal since the last emitted spike, provided that the potential is reset when a spike is emitted, i.e. U i (t + k ) = 0 for all k. To achieve a send-on-delta sampling, two IF neurons are needed. One "ON" neuron with w ON > 0 and one "OFF" neuron with w OFF < 0, and their potentials have to be reset when any of them fires. Indeed, the reference value of the signal must be updated when sampling occurs. This can be done by adding lateral connections between the "ON" and "OFF" neurons.

Let t k be the k t h time that any of the "ON" and "OFF" neurons fires. And suppose that the "ON" neuron fires at time t k+1 . We have

U ON (t - k+1 ) = w ON (x(t k+1 ) -x(t k )) = w 2 ON (4.2) =⇒ U OFF (t - k+1 ) = w OFF (x(t k+1 ) -x(t k )) = w OFF w ON (4.3)
Applying the same reasoning for a spike emitted by the "OFF" neuron, we find that the weight of the lateral connection between the "ON" and the "OFF" neurons must be -w OFF w ON in order to reset the potential of both neurons when any of them fires (see Fig. Note that, the reset is equivalent to an update of the input signal for each neuron. For example, if the "ON" neuron fires at time t k+1

U ON (t - k+1 ) -w 2 ON = w ON (x(t k+1 ) -x(t k ) -w ON ) U OFF (t - k+1 ) -w OFF w ON = w OFF (x(t k+1 ) -x(t k ) -w ON )
If an increase of at least w ON is detected, then the reference signal is increased by w ON for each neuron.

In this case, the deltas for the detection of an increase or a decrease are different. To have the same delta, w OFF should be equal to -w ON .

MULTI-DIMENSIONAL SEND-ON-DELTA

The previous results only apply to 1-dimensional signal. For a m-dimensional signal, each dimension can be tracked independently. The number of neurons required is thus 2*m. However, tracking each dimension independently is not efficient if the coordinates of the signal are correlated. Thus, we propose a multi-dimensional generalization of send-on-delta and the corresponding network architecture, inspired by [START_REF] Bourdoukan | Learning optimal spike-based representations[END_REF].

Instead of detecting changes in each dimension separately, we can detect changes in the projection of the signal along a given sampling direction. In this case, rather than simply increasing or decreasing the reference value of the signal when sampling occurs, it has now to be moved in the sampling direction (see Fig. 4.

3).

Let us consider a population of n neurons. Each neuron i ∈ [[0, n-1]] has its own "preferred" direction w i .

After the k t h spike emitted by any of these neurons, we have for for neuron i The threshold of the i t h neuron is set to w i 2 and the weight of lateral connections between neurons i and j to -< w j , w i >. Just after the reset, we have

U i (t ) =< w i , x(t ) -x(t k ) >
U j (t + k+1 ) =< w j , x(t k+1 ) -x(t k ) -w i >
Thus, if neuron i fires, w i is added to the signal reference value of all neurons.

Note that, if w i and w j are orthogonal the weight of lateral connections is 0 and neurons i and j are independent. If w j and w i are collinear, then the potential is reset to exactly 0. In particular, this is the case for neuron i .

Interestingly, the multi-dimensional generalization of SoD yields the same networks architecture as in [START_REF] Bourdoukan | Learning optimal spike-based representations[END_REF] for optimal spike-based representations. The only difference is that in [START_REF] Bourdoukan | Learning optimal spike-based representations[END_REF], the threshold is set to w i 2 2 instead of w i 2 as they want to minimize the distance between the signal and the samples. It can be interpreted very easily in the context of event-based sampling. If sampling is associated with an update of the reference signal of ±∆, then sampling reduces the reconstruction error as soon as the signal deviates by more than ∆ 2 from the reference. LIF neurons can be used as well with the same architecture. The sampling scheme would be equivalent to SoD with leak. Depending on the leak, only abrupt changes would be detected.

DEEP SPIKING NEURAL NETWORKS

Based on [START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks[END_REF] and the results of section 4, we implemented a spiking neural network in PyTorch. Spiking Layers can be Fully-Connected or Convolutional and with or without lateral connections. The neural network can be a standard feed forward network or a pool of neurons with recurrent connections.

A PyTorch based implementation of the different layers is available at http://github. com/romainzimmer/s2net.

LIF NEURONS AS RECURRENT NEURAL NETWORKS CELLS

The differential equations of LIF models can be approximated by linear recurrent equations in discrete time (See Appendix A.1). Introducing the reset term R i [n] corresponding to lateral connections, the neuron dynamics can now be fully described by the following equations.

U i [n] = β(U i [n -1] -R i [n]) + (1 -β)I i [n]
(5.1)

I i [n] = j W i j S in j [n] (5.2) R i [n] = (W • W T • S out [n]) i (5.3) S out i [n] = Θ(U i [n] -B i ) (5.4) B i = W i 2 (5.5)
where β = exp(-∆t τ mem ) and Θ is the Heaviside step function. Thus, LIF neurons can be modeled as a Recurrent Neural Network (RNN) cells whose state and output at time step n are given by (U [n], I [n]) and S[n] respectively [START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks[END_REF].

In practice, we used a trainable threshold parameter b i for neuron i , such that

S i [n] = Θ( U i [n] B i + -b i ) = Θ( U i [n] W i 2 + -b i )
with b i initialized to 1 and = 10 -8 . We normalize with W i 2 as the scale of the surrogate gradient is fixed.

SURROGATE GRADIENT

The RNN model can be implemented with traditional Deep Learning tools. However, one major issue has to be addressed regarding the threshold activation function.

The derivative of the Heaviside step function is 0 everywhere and is not defined in 0. Thus, no gradient can be back-propagated through it. To solve this problem, [START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks[END_REF] propose to approximate the derivative of the Heaviside step function. Thus, Θ ≈ sig σ (5.6) where σ ∈ R + and sig σ : x → 1 1+e -σx

FEED-FORWARD MODEL

We designed a feed-forward model composed of multiple spiking layers and a readout layer.

The input of a spiking layer is a spike train except for the first layer whose input is a multidimensional real-valued signal . Each layer outputs a spike train except for the readout layer that outputs real values that can be seen as a linear combination of spikes.

FULLY-CONNECTED SPIKING LAYER

For the fully-connected spiking layer, the input current at each time step is a weighted sum of the input spikes emitted by the previous layer at the given moment (or a weighted sum of the input signal if it is the first layer). The state and the output of the cells are updated following the above equations.

CONVOLUTIONAL SPIKING LAYER

For the convolutional spiking layer, the input current at each time step is given by a 1D (0D + time),2D (1D + time) or 3D (2D + time) convolution between a kernel and the input spike train. Note that convolution in time can be seen as propagation delays of the input spikes.

In this layer, lateral connections are only applied between neurons that have the same receptive field, i.e. locally between the different channels. For the i t h receptive field at time step n, when lateral connections are used, the reset term for channel p is

R i ,p [n] = l < Wk , Wl > S i ,l [n -1]
Where Wp is the vectorized form of W p , the kernel corresponding to the p t h channel and the sum is over the different channels.

READOUT LAYER

For the readout layer, [START_REF] Neftci | Surrogate Gradient Learning in Spiking Neural Networks[END_REF] proposed to use non-firing neurons. Thus, there is no reset nor lateral connections in this layer. For classification tasks, the dimension of the output is equal to the number of labels and the label probabilities are given by the softmax of the maximum value over time of the membrane potential of each neuron.

In practice, we have found that using time-distributed fully connected layer and taking the mean activation of this layer over time as output makes training more stable, at least with the datasets we have used. Thus, the output is the mean over time of a linear combination of input spikes.

RECURRENT MODEL

In the recurrent model, a pool of neurons with recurrent connections (output is fed back to the neurons) is used instead of stacking multiple layers. The input current for this model can also be computed using convolutions. And in this case, recurrent connections are only applied locally, i.e. between the channels for a given receptive field.

PENALIZING THE NUMBER OF SPIKES

A simple way to penalize the number of spikes is to apply a L1 or L2 loss on the total number of spikes emitted by each layer.

However, due to the surrogate gradient, some neurons will be penalized even if they haven't emitted any spike.

As S k [n] ∈ {0, 1}, the number of spikes for a given layer is

#spikes = 1 K N n k S k [n] = 1 K N n k S 2 k [n]
where K is the number of neurons and N is the number of time steps.

Replacing S k [n] by S k [n] 2 is a simple way to ensure that the regularization will not be applied to neurons that have not emitted any spikes, i.e. for which S k [n] = 0. Indeed,

d S 2 k [n] dU k [n] = 2 * S k [n] * si g σ (U k [n])
which is 0 when S k [n] = 0.

EXPERIMENTS

During this project, we mainly worked with feed-forward convolutional models on the Speech Commands dataset [Warden, 2018] as the goal was to compare spiking neural networks to standard deep learning models.

SPEECH COMMANDS DATASET

The Speech Commands dataset is a dataset of short audio recordings (at most 1 second, sampled at 16 kHz) of 30 different commands pronounced by different speakers for its first version and 35 for the second. All experiments were conducted on the first version of the dataset.

The task considered is to discriminate among 12 classes:

• 10 commands: "yes", "no", "up", "down", "left", "right", "on", "off", "stop", "go"

• unknown

• silence

The model is trained on the whole dataset. Commands that are not in the classes are labelled as unknown and silence training data are extracted from the background noise files provided with the dataset. See Table 6.1.

Authors of [Warden, 2018] also provide validation and testing datasets that can be directly used to evaluate the performance of a model.

PREPROCESSING

Log Mel filters together with their derivatives and second derivatives are extracted from raw signals using the python package LibROSA [McFee et al., 2015]. For the FFT, we used a window size of 30 ms and a hop length of 10 ms, which also corresponds to the time step of the simulation δ t . These are typical values in speech processing. Then, the log of 40 Mel filter coefficients were extracted using a Mel scale between 20 Hz and 4000 Hz only as this frequency band contains most of speech signal information (see Fig 6.1).

Finally, the spectrograms corresponding to each derivative order are re-scaled to ensure that the signal in each frequency has a variance of 1 across time and are considered as 3 different input channels.

ARCHITECTURE

For this task, we used 3 convolutional spiking layers with the same lateral connections as in the multi-dimensional send-on-delta architecture. The readout layer is a time distributed fully connected readout layer. Each convolutional layer has a β trainable parameter controlling the time constant of the layer, C channels and one threshold parameter b per channel. Kernels are of size H along the "time" axis and W along the "frequency" axis. [Warden, 2018]).

tional layers have a stride of 1 and dilation factors of d H and d W along "time" and "frequency" axes respectively. See Table 6.2 for details. The model was trained using the Rectified-Adam optimizer [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF] with a learning rate of 10 -3 , for 30 epochs with 1 epoch of warm-up, a weight decay of 10 -5 . For each layer l = 1, ..., L, a regularization loss L r (l ) was added with a coefficient 0.1 * l L to enforce sparse activity.

L r (l ) = 1 2K N n k S 2 k [n]
Gradient values were clipped to [-5, 5], β to [0, 1] and b to [0, +∞[

RESULTS

This network achieves 94% accuracy on this task with a mean firing rate of roughly 5Hz. Thus, the activation of the network is very sparse. Moreover, the trade-off between sparsity and performance can be controlled by the regularization coefficient (see Fig. 6.2).

In comparison, standard deep learning models achieve an accuracy of 96-97% [de Andrade et al., 2018].

These experimental results were obtained using lateral connections. However, we managed to get similar performances without them. Thus, we would like to further explore the impact of these connections for other tasks. Furthermore, we have found that recurrence through time is not necessarily useful if the discretization time step is large and processing spikes at each time step independently is enough.

DISCUSSION

We proposed a generalization of the send-on-delta sampling scheme to multi dimensional signal and showed that it can be achieved by a spiking neural network with lateral connections. We designed a deep spiking neural network with binary sparse activation. This network can be trained using backpropagation through time with surrogate gradient methods and achieves comparable performance to standard deep learning models on the speech recognition task we worked on.

These results show the potential of spiking neural networks. Although PyTorch is not particularly suitable for the development of spiking neural networks, it is a very popular library in the deep learning community and we hope that this work will help to develop interest in spiking neural networks.

For future work, we would like to test our model on other tasks and especially on event data such as Dynamic Vision Sensor camera data. We would also like to continue to explore the relationships between event-based sampling theory and spiking neural networks. In discrete time with sampling rate 1 h , (E) can thus be approximated by the recurrent equation:

u[n] = βu[n -1] + (1 -β)i [n]
with β = e -h τ .

Figure 4

 4 Figure 4.1: Send-on-delta sampling strategy. Blue dots represent sampling due to a significant increase, red dots represent sampling due to a significant decrease.

  4.2).

Figure 4 .

 4 Figure 4.2: Architecture for SoD encoding with two IF neurons and spike train generated for the example presented in figure 4.1

Figure 4 . 3 :

 43 Figure 4.3: Left: independent tracking of each dimension, Right: tracking along a given direction

Figure 5 .

 5 Figure 5.1: Approximation of the derivative of the Heaviside step function

Figure 6 .

 6 Figure 6.1: Example of Mel filters extracted for the word "off".

Figure 6 . 2 :

 62 Figure 6.2: Example of spike train for one channel of the first layer for the word "off".

Table 6 .

 6 1: Number of recordings in the speech commands dataset (extracted from

	All convolu-

  The scale of the surrogate gradient was set to 10.

	Conv. layer number C H W d H d W
	1	64 4	3	1	1
	2	64 4	3	4	3
	3	64 4	3	16	9
	Table 6.2: Parameters of each convolutional layer for the speech commands dataset
	6.1.3. TRAINING AND EVALUATION

  u K (t + h) ≈ e -h τ u K (t ) + i (t + h) τ = e -h τ u K (t ) + (1e -h τ )i (t + h)

	t +h	e -t +h-s τ	d s
	t

A. APPENDIX A.1. DISCRETE TIME APPROXIMATION

Let's consider the following differential equation (E) and it's homogeneous equation (H) :

The solutions of (E) can be found using the variation of parameters method.

The solution of (H) has the following form:

Let's consider a solution of (E) of the form

Injecting this solution in E , yields the following equivalent equation

is a particular solution of (E) and all the solutions of (E) can be written as:

For sufficiently small h,


Karine Lan
email: karine.lan@utt.fr

Hing Ting

Dimitri Voilmy

Guillaume Dessinger

Vincent Gauthier

Jean-Louis Dillenseger

Guy Carrault

Jean-Marc Laferté

Alain-Jérôme Fougeres

J-L Dillenseger

J- M Laferte

A-J Fougeres

Co-designing a falls detection device: combining concerns for human motion and elders needs

Keywords: Falls, Ambient Assisted Living, Elderly empowerment, Human Motion Analysis, Participatory design

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

To address the major public health problem of falls among the elderly (WHO, 2011), the interdisciplinary project PRuDENCE proposes a new low-cost device. Based on depth and/or thermal sensors [START_REF] Halima | Sensors fusion for head tracking using Particle filter in a context of falls detection[END_REF][START_REF] Zoetgnandé | Robust low resolution thermal stereo camera calibration[END_REF], the device makes it possible, on the one hand, to detect falls and prolonged presence on the ground and, on the other hand, to prevent the risk of falling by analysing the activity of individuals. In addition to the detection performance characteristics demonstrated during tests in nursing homes, the device proposes to contribute effectively to the overall objective of preventing loss of autonomy.

Previous work

Among the growing applications and research interest around Human Motion Analysis (cf. Bandera & al., 2019 for details), are new application fields, like Ambient Assisted Living technologies. In the context of an ageing population, AAL technologies can play an important role in helping elderly people achieve healthy ageing and maintain their autonomy. Indeed, falls or risk of falling, manipulation issues or motion impairments are among the key causes of autonomy loss among the elderly population. Thus, motion evaluation becomes an important feature for these systems. Among possible Human Motion Analysis solutions are autonomous systems to monitor the status and activities of a person, without interfering with them. Autonomous tools -that evaluate human motion, whether in their daily life activities, performing motion clinical tests or in case of falls detection -will have to be precise, and yet, will avoid imposing any constraints on the elderly user. E.g., no special environments, markers or garments. Ideally, the point of view from which motion sequences are captured should not be imposed. This is to allow the performing of these evaluations, from any monitoring camera, in the person's environment, e.g. her home [START_REF] Bandera | A new paradigm for autonomous human motion description and evaluation: Application to the Get Up & Go test use case[END_REF], especially in the use case of falls. The PRuDENCE falls detection device features these characteristics.

Research questions, methods and insights

The project's research questions focus, in a complementary way, both on i) technical viability and detection performance as part of the sensors' development and calibration [START_REF] Halima | Sensors fusion for head tracking using Particle filter in a context of falls detection[END_REF][START_REF] Zoetgnandé | Robust low resolution thermal stereo camera calibration[END_REF], and ii) on use and utility, adopting a living lab approach (Lan Hing Ting et al., 2020), that is iterative, participatory [START_REF] Blomberg | Ethnography: positioning ethnography within participatory design[END_REF], and human-centred [START_REF] Bannon | Reimagining HCI: toward a more human-centered perspective[END_REF]. From a technical perspective, the development research included the evaluation of the performance of the developed depth and thermal sensors in detecting falls and prolonged presence on the ground. Using the same protocol developed for previous research [START_REF] Bandera | A new paradigm for autonomous human motion description and evaluation: Application to the Get Up & Go test use case[END_REF], the experiments of Motion Capture were made at the Movement analysis platform of our Living Lab. Human movements -like falls, accelerations, daily activities -were captured simultaneously in 3 ways : i) the developed sensors ii) a precise a Vicon Nexus 1.8.5 MoCap system based on markers, iii) external video cameras. While the video analysis allowed saving and contextualization of the data capture session for later use, as a way of "visualizing" and making sense of the data, the sensors' data have been compared with the Vicon's data in order to characterize capture errors from the former. This comparison was useful to inform the improvements made to the detection algorithms. Results from this Human Capture experiment show that the proposed sensors are adequate to be used in this scenario of falls detection. From a human-centred design perspective, a living lab approach was used in order to ensure the relevance of the design ideas. Coherent with a human-centred approach, aiming not only at designing a potentially life-saving device, but one which also adds to the dignity and empowerment of the elders concerned [START_REF] Bannon | Reimagining HCI: toward a more human-centered perspective[END_REF], the first step has been a qualitative needs analysis. The ethnographic observations and in-depth interviews focused on understanding people's gait problems -their concerns, activities, values, daily constraints, aspirations -in 2 contexts: independent living at home, and fall prevention training in a rehab centre. More generally, following the participatory approach adopted, different stakeholders with different perspectives were actively involved in the different design and evaluation tasks: elderly people themselves and their carers; health professionals (general practitioners, geriatricians, pharmacists) and rehabilitation professionals (physiotherapists, Adapted Physical Activity Teachers). In particular, observations of multi-professional "falls prevention workshops" with patients at risk of falls, allowed to understand the multifactorial causes of falls, as well as their consequences in all their complexity. The knowledge gained made it possible to usefully "inform" the design of the system and to improve its quality in a reflexive manner. Based on these insights and the sensors prototypes, the next step has been to examine usefulness and acceptability issues during participatory workshops. Two of them brought together elders, caregivers (informal and professionals), clinicians, and researchers/engineers developing and building the sensors, and one was organized among rehabilitation professionals. The objective was to enrich and inform the design process with different point of views, competencies, perspectives, to prospectively and collaboratively build functionalities and scenarios that are desirable, useful, technically feasible, at a reasonable cost, and last but not least, which succeeds in empowering elders at risk of falling.

Understanding values to inform design

Falls prevention at the rehabilitation hospital participating in this research, rely mainly on patients' awareness and knowledge of the multifactorial causes of falls, and that they learn the techniques to get up by themselves without exterior help in case they fall. Discussions about elders empowerment and responsibility for themselves during workshops reveal stakeholders' reluctance about some assistive technologies. For them, too much materialization of assistance is neither educational nor preventive, and seriously limits the elders capacity to act. If one constantly needs a third party, whether an assistive technology or a person, is this really autonomy? Therefore, the values that emerged as important for stakeholders concerning the PRuDENCE device as an assistive technology, are: Individual freedom, independence, free will, while avoiding relational dependency. The device must combine the following characteristics: Adaptable/configurable (the user is in "control"), non-intrusive, discreet, efficient, durable (no false alarms, breakdowns), accessible -both from financial and usability perspectives, e.g. voice interaction -reassuring, accountable. Interestingly concerning the workshop's issue of "visualization", what is considered desirable in this falls prevention device is precisely the fact that it is "transparent". Contrary to wearable devices or connected objects associated to follow-up apps with visual interfaces (Lan Hing Ting et al., 2020), the device is discretely installed in the home (small white boxes that match the environment). Its main perceived advantage is that "you don't have to think about it", users say. Thus, the absence of "interaction" -having no visual feedback or monitoring on the human movement data that the sensors are capturing with the device -is precisely what is expected and desired (especially when one knows that false alerts can be common with these types of sensors). However, the sine qua non condition is the trust in the visualization performance of the sensors, on their side, the sensors are actually detecting the human activity or sudden movement efficiently, so that falls can be detected successfully. In the case a user falls, research is underway concerning the interaction design possibilities that would be in line with the important values that stakeholders expressed, mainly voice interaction and reassurance effect.

Conclusion

In this paper, we discussed the central importance of achieving both technical performance and putting elders needs at the centre of all design considerations. This way, a falls detection device like this will succeed in being useful for falls prevention and elders' empowerment, contributing to elders' autonomy, decision-making, and patient education.

Acknowledgements

The authors warmly thank "Les Amis du Living Lab" and our colleagues from CRRF COS Pasteur -specially Dragos Ghitiu, Anaïs Seguin, Adeline Delancret, Caroline Halm and Mattieu Detroy -as well as the patients for their participation in this research.