Hammouda Elbez

Mazdak Fatahi

Progressive Layer-based Compression for Convolutional Spiking Neural Network

Keywords: Spiking Neural Network, Neuromorphic Computing, Compression, STDP, Surrogate Gradient, SpiNNaker

Spiking neural networks (SNNs) have attracted interest in recent years due to their low energy consumption and the increasing need for more power in real-life machine learning applications.

Having those bio-inspired networks on neuromorphic hardware for extra-low energy consumption is another exciting aspect of this technology. Furthermore, many works discuss the improvement of SNNs in terms of performance and hardware implementation. This paper presents a progressive layer-based compression approach applied to convolutional spiking neural networks trained either with Spike Time Dependent Plasticity (STDP) or Surrogate Gradient (SG). Moreover, we study the effect of this approach when used with SpiNNaker. This approach, inspired by neuroplasticity, produces highly compressed networks (up to 90% compression rate per layer) while preserving most of the network performance, as shown by experimental results on MNIST, FMNIST, Caltech face/motorbike, and CIFAR-10 datasets.

INTRODUCTION

In recent years, the use of neural networks in real-life applications has risen significantly due to the progress in the field. However, the current progress increased the complexity of the used models, resulting in more resource-hungry models that the Von Neumann architecture cannot guarantee. Spiking neural networks (SNNs) are considered a promising alternative to overcome Moore's law limitation, rise to the energy demands of modern network models, and provide a bio-inspired solution for lower energy consumption.

SNN is inspired by brain functionality and uses spikes to communicate, guarantee low energy consumption, and the ability to process natural signals. Deploying SNNs using neuromorphic hardware is another promising way to have those benefits on more optimized architectures, making it possible to use such technology with energy-constrained applications.

Nowadays, complex and deep architectures are often necessary for better accuracy regarding neural network performance, which explains the rising complexity of the recent models, such as MobileNetV2 [START_REF] Sandler | Mobilenetv2: Inverted residuals and linear bottlenecks[END_REF], ResNet152 [START_REF] He | Deep residual learning for image recognition[END_REF], and GPT-3 [START_REF] Brown | Language models are few-shot learners[END_REF].

Moreover, in SNN, increasing the size of the network helps improve performance and process complex data. As we can see in the existing works, some of them raised the number of neurons in the network to get a better accuracy. (Diehl and Cook, 2015), and other works preferred to use multiple layers, each with a group of neurons to get a better performance [START_REF] Lee | Training deep spiking neural networks using backpropagation[END_REF]Diehl et al., 2015;[START_REF] Kheradpisheh | STDP-based spiking deep convolutional neural networks for object recognition[END_REF]. However, the number of neurons increases using large networks, and so does the internal activity and complexity. Therefore, it is more challenging to analyze the network behavior, especially with the use of spikes for communication and the asynchronous nature. Moreover, the increase in the network size implies an increase in the required resources to run, which will prevent their deployment on hardware using technologies like memristive crossbars [START_REF] Merolla | A digital neurosynaptic core using embedded crossbar memory with 45pj per spike in 45nm[END_REF][START_REF] Strukov | The missing memristor found[END_REF], SpiNNaker [START_REF] Furber | The SpiNNaker Project[END_REF], or Loihi [START_REF] Davies | Loihi: A Neuromorphic Manycore Processor with On-Chip Learning[END_REF].

In terms of hardware implementation, one of the techniques that we can use to overcome the complexity issue is pruning. Pruning compresses the network by reducing one or multiple network components, which results in a smaller size that can fit the hardware-limited resources and decreases computational operations. Pruning is inspired by the activity of the human brain in the early stages [START_REF] Huttenlocher | Synaptic density in human frontal cortex -developmental changes and effects of aging[END_REF][START_REF] Cun | Optimal brain damage[END_REF][START_REF] Hassibi | Optimal Brain Surgeon: Extensions and performance comparisons[END_REF], where the brain losses neurons during the process of learning. The changes in synapse strength (known as neuroplasticity) happen not only in the early stages but throughout a person's lifespan. In biology, we have synaptotrophins and synaptotoxins responsible for synapse creation and elimination respectively [START_REF] Sanes | Development of the vertebrate neuromuscular junction[END_REF]. We can identify three types of pruning when working with neural networks: Filter pruning [START_REF] He | Filter pruning via geometric median for deep convolutional neural networks acceleration[END_REF][START_REF] Huang | Learning to prune filters in convolutional neural networks[END_REF][START_REF] Li | Pruning filters for efficient convnets[END_REF], weights pruning [START_REF] Carreira-Perpinan | learning-compression" algorithms for neural net pruning[END_REF][START_REF] Liu | Frequency-domain dynamic pruning for convolutional neural networks[END_REF], and neuron-based pruning [START_REF] Yu | NISP: pruning networks using neuron importance score propagation[END_REF].

In the case of spiking neural networks, we can use pruning to reduce the size of the network. Shi et al. [START_REF] Shi | A Soft-Pruning Method Applied During Training of Spiking Neural Networks for In-memory Computing Applications[END_REF] presented a pruning method for SNNs on emerging non-volatile memory (eNVM) devices by exploiting the output firing characteristics of neurons. This technique is used during training and can maintain 90% classification accuracy on MNIST with up to 75% of the network pruned. Cho et al. [START_REF] Cho | A 2048-neuron spiking neural network accelerator with neuro-inspired pruning and asynchronous network on chip in 40nm cmos[END_REF]) applied a distance-based pruning on a CMOS SNN chip, which decreases spikes activity by 52%. Rathi et al. [START_REF] Rathi | STDP-Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy-Efficient Recognition[END_REF] combined weight quantization and pruning during the learning phase.

Using a two-layer SNN of 6400 neurons and a static pruning threshold, they obtain a highly compressed network able to preserve a good performance, which is based on the Spike Time Dependent Plasticity (STDP) learning rule [START_REF] Bi | Synaptic Modifications in Cultured Hippocampal Neurons: Dependence on Spike Timing, Synaptic Strength, and Postsynaptic Cell Type[END_REF]. In the work of Chen et al. [START_REF] Chen | Fast and efficient deep sparse multi-strength spiking neural networks with dynamic pruning[END_REF], the authors used a three-phase prune process. The first two involve removing quiet neurons, and the third one concerns the removal of weak synapses. They used the prune operation as a part of the CNN to SNN conversion to reduce computational operations by 85%. Finally, Saunders et al. [START_REF] Saunders | Locally connected spiking neural networks for unsupervised feature learning[END_REF] used a two-layer network of 900 neurons, and applied pruning once after the learning phase. Therefore, removing half of the synapses while preserving 90% network accuracy. For more deep spiking neural networks, we usually train using a global learning rule such as Surrogate Gradient [START_REF] Neftci | Surrogate gradient learning in spiking neural networks: Bringing the power of gradient-based optimization to spiking neural networks[END_REF]or local learning rule like the Spike Time Dependent Plasticity (STDP). However, the pruning mechanism remains the same. Chen et al. [START_REF] Chen | Pruning of deep spiking neural networks through gradient rewiring[END_REF] proposed a gradient rewiring technique (Grad R), an algorithm for learning weights and connectivity in a deep spiking neural network. As a result, the authors minimized the loss in terms of performance. Furthermore, they revealed a remarkable structure refining capability in SNN since they had a 3.5% loss in accuracy when using 0.73% connectivity. Nguyen et al. [START_REF] Nguyen | Connection pruning for deep spiking neural networks with on-chip learning[END_REF] presented connection pruning applied to a deep SNN, which is trained using STDP on FPGA. The approach consists of two stages: dynamic pruning during the on-chip learning and post-learning pruning after each layer.

Using a weight update history value, the author calculated it using a proposed formula and compared it to a predefined threshold to prune. As a result, they achieved 2.1x speed-up and 64% energy saving during the on-chip learning. In the work of Faghihi et al. [START_REF] Faghihi | A synaptic pruning-based spiking neural network for hand-written digits classification[END_REF], a synaptic pruning-based SNN was presented, which uses a modified learning rule combined with a synaptic pruning method. Moreover, the prune operation is based on a defined threshold µ, resulting in a sparse neural connection between two layers that uses a few-shot-based classification method.

Progressive Layer-based Compression for Convolutional SNN

By looking at the literature, we can see that the existing works focus on when to prune (at the end or during network activity), what to prune (neurons or synapses), and how to treat the pruned element (hardpruning or soft-pruning). Our contribution is the proposition of a novel technique for pruning threshold selection which is dynamic as opposed to existing works. The new threshold depends on the pruning rate of the previous prune operation. This work extends the previous work [START_REF] Elbez | Progressive compression and weight reinforcement for spiking neural networks[END_REF] applied only on shallow networks to Convolutional Spiking Neural Networks (CSNN) by using a layer-based progressive pruning to get highly compressed layers. The compression rate increases when going more profound in the network (up to 98% compression rate). Moreover, the network performance is preserved compared to the baseline in the best-case scenario or records less than 3% accuracy loss in the worst-case scenario. We evaluated the efficacy of this approach by applying it to MNIST, FMNIST, Caltech face/motorbike, and CIFAR-10 datasets and analyzing the result when used with SpiNNaker.

We can resume our contribution in six points: 1) the extension of the progressive pruning and weight reinforcement techniques for convolutional spiking neural networks by adapting the first formula for multi-epoch training. 2) the application of progressive compression on networks trained with Spike Time Dependent Plasticity (STDP) or Surrogate Gradient (SG). 3) the study of the maximum pruning threshold value (α) effect on the network performance and compression rate. 4) proposing a layer-based version of this approach for more compression by setting the initial alpha value based on the depth of the actual layer. 5) testing this approach for the first time on SpiNNaker by deploying the compressed and baseline network on the board and estimating the reduced energy. 6) evaluating this technique on multiple datasets instead of only MNIST, which is the case in the previous work. Moreover, we share an opensource repository that contains the required code to reproduce the experiments using the csnn-simulator, Norse, and SpiNNaker1

MATERIALS AND METHODES

We observe the benefit of compressing a spiking neural network when we want to implement it on hardware.

Due to limited resources, reducing the elements of a neural network can enable the deployment of more extensive networks, which is impossible without compression. Moreover, by reducing the network size, we also reduce the resources needed for the deployment. This section describes our approach, the network topology, the different datasets used in the experiments, and the SpiNNaker board.

The Progressive Compression

Progressive Compression involves two processes: Progressive Pruning (PP) and Dynamic Synaptic Weight Reinforcement (DSWR). Progressive Pruning (PP) eliminates connections between neurons from one layer to another. We perform this operation after each batch (group of inputs) during the training phase [START_REF] Elbez | Progressive compression and weight reinforcement for spiking neural networks[END_REF]. Using a dynamic pruning threshold T n , n ∈ N, which we calculate using equation [START_REF] Deng | Comprehensive snn compression using admm optimization and activity regularization[END_REF].

T n+1 = T n + α * (C rn /C n ) n ∈ N (1)
α is a constant representing the initial threshold. T n and T n+1 are the old and new threshold for the next batch, respectively. C n represents the total number of synapses, and C rn the remaining synapses between the two layers at batch n. In [START_REF] Elbez | Progressive compression and weight reinforcement for spiking neural networks[END_REF], the authors applied this approach to single-layer neural networks using the MNIST dataset, which proved effective in compressing the network. Therefore, in this work, we will extend this work and study the effect of this approach on convolutional spiking neural networks trained with STDP or SG.

Progressive Layer-based Compression for Convolutional SNN Dynamic Synaptic Weight Reinforcement (DSWR) is another process combined with the pruning operation, which concerns the maintained synapses after pruning. We can see this process in biology and the human brain as a part of the synaptogenesis process [START_REF] Sanes | Development of the vertebrate neuromuscular junction[END_REF]. Moreover, by reinforcing the preserved synapses, we speed up their convergence toward one feature. The equation used to determine the amount of reinforcement depends on the currently calculated threshold, and it is done based on equation (2).

W n+1 = W n + β * T n , n ∈ N, W ∈ [0, 1] (2)
β is a constant we define based on experiments. W n and W n+1 are the concerned connection's current and new weights, respectively. In our work, for STDP-based networks, we keep the two constants α and β the same as in [START_REF] Elbez | Progressive compression and weight reinforcement for spiking neural networks[END_REF]) (α = 0.05 and β = 0.1). Those values were fixed using Pareto front multiobjective optimization [START_REF] Deb | Multi-objective optimisation using evolutionary algorithms: An introduction[END_REF] based on network accuracy and compression rate.

When applying the two formulas on a single-layer network, we trained for one epoch and used it after each batch of 10k inputs (in the case of MNIST). However, for deeper networks, we usually train for multiple epochs, and following the same method will cause the pruning threshold to be updated numerous times and destroy the network connectivity. Therefore, instead of compressing after each batch, we would compress after each epoch and add a constraint on the max possible pruning threshold value. If we reach this value, we do not update the threshold value. This approach is represented in Figure 1, and as a result, equation (3) represents the adapted version of the Progressive Pruning formula for STDP-based networks. θ is the maximum pruning threshold, which is used to decide if we will compute a new threshold or not (θ = 0.3 in this work). Moreover, the reinforcement part stays the same, and we apply it after each prune operation.

Progressive Layer-based Compression for Convolutional SNN

In the case of SG-Based networks, since the weights in the network are not all positive, which is the case in STDP-Based ones. We need to update the Progressive Pruning formula in a way it can also support negative weights, otherwise, the network will be destroyed by setting all negative weights to zero. From equation (3), we will create two formulas (equation ( 4)) with θ + and θ -, which represent the positive maximum pruning threshold and negative one, respectively. Positive and negative initial threshold (α + = 0.005 and α -= -0.005). Finally, T + e and T - e represent the positive and negative threshold values.

T + e+1 = T + e + α + * (C re /C e ) e ∈ N, T + e+1 ≤ θ + (4) T - e+1 = T - e + α - * (C re /C e ) e ∈ N, T - e+1 ≥ θ -
Since the range of the weights in the SG-Based networks using Norse is not similar, the selection of the positive θ + and negative θ + maximum pruning threshold is computed using equation 5. If we apply this equation on a network with positive weights W ∈ [0, 1], we will get θ + = θ = 0.3, which is the initial threshold used with STDP-Based networks.

θ + = θ * max(W L ) (5) θ -= θ * min(W L )
For applying the Dynamic Synaptic Weight Reinforcement (DSWR) with SG-Based networks, we also have β + = 0.1 and β -= -0.1, and the reinforcement is applied based on equation ( 6). After a couple of epochs, the two thresholds became stable (around 0.06 and -0.06) since θ + = 0.0595 and θ -= -0.0599.

W + n+1 = W + n + β + * T + e , W > 0 (6) W - n+1 = W - n + β - * T - e , W < 0

Network Topology

Our experiments use two approaches for training SNN: STDP and SG. This section presents the network topology used for each of the two approaches.

STDP-based networks:

For the STDP-based networks we use the same topology presented by Falez et al. [START_REF] Falez | Improving Spiking Neural Networks Trained with Spike Timing Dependent Plasticity for Image Recognition[END_REF], which is composed of multiple pairs of convolutional and max pooling layers. Those pairs of layers are followed by a dense layer and a support vector machine (SVM) for classification and decision-making. We use integrate-and-fire (IF) neurons [START_REF] Burkitt | A review of the integrate-and-fire neuron model: I. homogeneous synaptic input[END_REF] in the different layers of the model. IF neuron model adds the input spikes into the membrane potential v(t) until a threshold v th (t) is reached, resulting in the neuron firing and sending an output spike. Then, the membrane potential is reset to a defined value (0 in this work). This neuron model is represented by the following equation ( 7) [START_REF] Falez | Improving Spiking Neural Networks Trained with Spike Timing Dependent Plasticity for Image Recognition[END_REF].

C m ∂v(t) ∂t = i ε v i f s (t -t i ), f s (x) = 1, if x ≥ 0 0, otherwise v(t) = v r when v(t) ≥ v th (t) (7)
C m represents the membrane capacitance, and v i is the spike voltage of the i-th spike. Also, ε and f s represent the set of incoming spikes and the kernel of spikes, respectively. Finally, t i is the timestamp of the i-th spike.

What makes the neural network useful is being able to learn. In our work, we use the Spike Time Dependent Plasticity (STDP) learning rule, where the synaptic weight update depends on the spike time from neurons on both ends of the synapse (pre-neuron and post-neuron). The STDP we use is defined in equation ( 8).

∆ w =    η w e - tpre-t post τ STDP , if t pre ≤ t post -η w e - tpre-t post τ STDP , otherwise (8)
τ STDP is the time constant for the STDP learning window, and η w is the learning rate. t pre and t post represent the spiking time of pre-synaptic and post-synaptic neuron, respectively. We can see from the equation that Progressive Layer-based Compression for Convolutional SNN the update on the synaptic weight ∆ w can be either positive or negative, depending on which spike came first. In this work, the synaptic weights are between 0 and 1.

For the multi-layer network to work properly, two additional mechanisms are used. First, the Winner-takesall (WTA) represents an inhibition mechanism to prevent neuron domination when learning and prevent multiple neurons in one layer from learning the same feature, which improves the network performance.

Besides WTA, we need to add a homeostasis mechanism. Since we are using multi-layer SNN we will use the same neuron threshold adaptation technique presented in [START_REF] Falez | Improving Spiking Neural Networks Trained with Spike Timing Dependent Plasticity for Image Recognition[END_REF], which trains the neurons to fire at a given time t obj to maintain the homeostasis. This technique is applied each time a neuron fires or gets inhibited. Every neuron's threshold is updated, so its firing time converges toward t obj . The neuron threshold adaptation is presented by equations ( 9) and ( 10).

∆ 1 th = -η th (t -t obj ) ∆ 2 th = η th , if t i = min(t 0 , ..., t N ) -η th l d (n) , otherwise (9) v th (t) = max(th min , v th (t -1) + ∆ 1 th + ∆ 2 th ) (10)
η th represents the threshold learning rate, and l d is the number of neurons in competition in the layer. Furthermore, t and t i are the spike timestamp of the neuron and the firing time of neuron i, respectively.

Finally, th min is the minimum possible neuron threshold value.

In our experiments, the different hyperparameters we used are: Moreover, we use default parameters for the SVM part of the network, which delivers good performance.

Surrogate Gradient-based networks:

For the Surrogate Gradient-based experiments, we use Norse [START_REF] Pehle | Norse -A deep learning library for spiking neural networks[END_REF] for the simulation.

Norse is a Python library that expands PyTorch with primitives for bio-inspired neural components, which allows us to train multilayer spiking neural networks using Surrogate Gradient.

We used an architecture similar to the STDP-based network (multiple pairs of convolutional and max pooling layers). However, the only difference is replacing the SVM part with a LILinearCell, which consists of a group of cells for a leaky-integrator (LI) with an additional linear weighting. We can see in Figure 3 (B) that when using Norse, each convolution or Dense layer is followed by a LIFCell, which consists of a group of leaky integrate-and-fire (LIF) neurons to process the output of that layer before going to the next one.

Moreover, we use latency coding [START_REF] Rullen | Rate coding versus temporal order coding: What the retinal ganglion cells tell the visual cortex[END_REF]) (SpikeLatencyLIFEncoder) before introducing the input to the network.

For training the network, we use the SuperSpike method [START_REF] Zenke | SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks[END_REF], which is a voltagebased global learning rule that can be interpreted as a nonlinear Hebbian three-factor rule. The learning

∆w k ij = r ij t k+1 t k e i (s) Error signal α *   σ (U i (s)) Post ( * S j )(s) Pre   ds. ( 11
)
t k is the spike time (k = 1,2,...). The error signal represents the third factor in this rule, with e i (s) = α * ( Ŝi -S i ). S i is the spike activity of pre-synaptic neuron i, Ŝi represents the target spike train for a given stimulus, and α is a normalized smooth temporal convolution kernel. r ij is the learning rate for synapse ij.

σ (U i (s)) represents the derivative of a continuous auxiliary function σ of the membrane potential U i (s),

and is the postsynaptic potential (PSP) shape.

For the SG-based experiments, the different hyperparameters we used are:

1. Latency coding: T = 35 2. LIFCell: V th = 0.25 3. SuperSpike: α = 80, Optimizer = Adam, Learning rate = 0.001

Datasets

Using convolutional SNN, we can test the compression effect on the network using different datasets. In our experiments, we use MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], composed of 28x28 pixel images of handwritten digits with labels from 0 to 9. MNIST contains 60,000 training images and 10,000 test images. FMNIST [START_REF] Xiao | Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms[END_REF] is similar to MNIST in terms of type, data dimensions, and dataset size. However, FMNIST contains clothes with greyscale images instead of handwritten digits. Caltech face/motorbike [START_REF] Kheradpisheh | STDP-based spiking deep convolutional neural networks for object recognition[END_REF] contains modified data from Caltech-100 by using only two classes (Face/Motor). Caltech face/motorbike images are converted to greyscale and resized to 160 pixels in height while preserving the aspect ratio, and it contains 400 train and 396 test images. Finally, CIFAR-10 (Krizhevsky and Hinton, 2009) consists of colored images composed of 32x32 pixel images of objects with ten classes. CIFAR-10 contains 50000 train images and 10000 test images.

Progressive Layer-based Compression for Convolutional SNN

SpiNNaker Board

Modeling large neural networks on Von Neumann architecture requires a lot of computing resources and power consumption [START_REF] Sharp | Power-efficient simulation of detailed cortical microcircuits on spinnaker[END_REF]. SpiNNaker [START_REF] Painkras | Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation[END_REF] is one of the neuromorphic architectures [START_REF] Basu | Spiking neural network integrated circuits: A review of trends and future directions[END_REF] that was proposed to overcome the limitations and provide the requirements for spiking neural networks.

SpiNNaker is a biologically inspired, massively parallel computing system optimized for modeling and simulating large-scale real-time networks. In this work, we use the SpiNN-5 (SpiNNaker 103)

board [START_REF] Painkras | Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation[END_REF]) [START_REF] Furber | Overview of the SpiNNaker System Architecture[END_REF], which consists of 48 SpiNNaker chips. Each chip contains 18 ARM cores with a 32 kB ITCM (instruction tightly coupled memory) and a 64 kB DTCM (data tightly coupled memory) per core. Moreover, a 128 MB SDRAM is shared between the 18 cores. To imitate the high connectivity of the brain, the cores are interconnected by an asynchronous Network-on-Chip (NoC) through a multicast packet-routing mechanism. In addition, SpiNN-5 uses three Xilinx Spartan-6

FPGAs for high-speed serial links.

A 100 MB Ethernet controller handles the connection between the SpiNNaker board and the computer.

We use it to load data to the SpiNNaker memory to perform a real-time simulation. Furthermore, the sPyNNaker [START_REF] Rhodes | spynnaker: A software package for running pynn simulations on spinnaker[END_REF] is a software package used to define models in PyNN script [START_REF] Davison | Pynn: a common interface for neuronal network simulators[END_REF] and translates models into a suitable form for SpiNNaker.

RESULTS AND DISCUSSION

This section describes our experiments on the image classification task and presents the results using the csnn-simulator and Norse.

STDP-based networks

In the case of STDP-based networks, training is done layerwise by training each layer for 100 epochs. For the network architecture, the number of layers used varies from one dataset to another, but they all use SVM for decision-making. Therefore, we use one convolutional layer for CIFAR-10, while two convolutionalmax pooling layers and one dense layer are used with MNIST, FMNIST, and Caltech face/motorbike. More details about the different architectures are presented in Table 1. Note: w = width, h = height, n = number

In Table 2, we can see the effect of compressing the network using our approach on different datasets.

We observe, in particular, the network classification rate, the compression rate on each layer, the duration of the training and testing phase, number of spikes per layer, and number of synaptic updates per layer.

Finally, the comparison is made between the compressed network and the baseline (the same model without compression). By analyzing Table 2, we can see that in terms of the STDP-based networks performance, we have a small loss when compressing on some datasets and a considerable one on Face/Motor. Moreover, for the compression rate, we can observe that compression is not that high (19%) for a single-layer network (CIFAR-10). On the other hand, for multi-layer networks, we can see the compression rate for each layer.

However, it is interesting that despite using different datasets, the compression rate is higher at the first layer, decreases when going deeper (MNIST & FMNIST), and is more stable in the case of Face/Motor. Furthermore, we can see a slight change in the simulation time when we use compression due to applying it during the training and the SVM (not concerned with compression) training part, which is time-consuming.

Finally, for the layer activity (spikes & synaptic updates), we do not see a difference in spikes activity in the last layer. However, some compressed layers have more activity than the baseline, which differs from what we expected. Furthermore, for the synaptic updates, we can see an apparent decrease in the activity of all layers of the compressed network, which is expected due to the compression. In Table 2, we set the max pruning threshold θ = 0.3 for the experiments. The selection of θ can impact the network compression since it defines the highest possible pruning threshold value, and if we set it too high (θ = 1), the compression will destroy the network. Moreover, the value of θ may vary from one application to another and from one dataset to another. In Table 3, we explore the effect of increasing θ up to 0.7 on the network performance and compression rate. The experiments are applied using the same network architectures and the same datasets.

In Table 3, we can see that by increasing the maximum pruning threshold value θ, the compression rate is stable with a small increase in the network performance, which is visible on all the datasets. Therefore, for the STDP-based networks, we do not see a clear improvement in compression by increasing the maximum pruning threshold value θ. Regarding the network performance, we can see that the best-recorded accuracy across the experiments does not always use a specific θ value and changes from one dataset to another.

To understand why the compression rate is low on the STDP-based networks even when increasing the max pruning threshold and the accuracy is lower, we need to take a look at the threshold mechanism used in the network. The threshold mechanism is essential in the network for learning, and it helps the network maintain activity in the different layers by updating the threshold of the neurons. Therefore, any other mechanism that may affect the spike activity in the network may negatively impact the network by changing the neuron threshold, the synaptic weights, or the input value. In our work, the progressive compression, when applied, will reduce the number of synapses and reinforce the remaining synapses, affecting the synaptic weights and input value and conflicting with the existing threshold adaptation mechanism, which explains the low compression rate in the STDP-based networks, and the loss in accuracy. Moreover, in Table 2, we noticed that the spikes activity increases in the compressed network. Such an increase is now justified due to the threshold mechanisms reducing the threshold as a reaction to the compression.

In Figure 4 (A), we can see how the thresholds of neurons behave when we apply the compression. Therefore, it is visible that when the compression is applied (the red dotted line), there is a visible change in the thresholds, which is the threshold adaptation mechanism reaction that we mentioned before. In Figure 4 (B), we can see how the synaptic weights of one neuron are being updated due to the compression and the threshold adaptation. As we can see in the figure, the weights are initially between zero and one.

However, once we compress the network after each epoch, we can see how the weights are being removed (going to zero), and at the same time, the other weights are going to one. This behavior remains the same

Surrogate Gradient-based networks

In the Surrogate Gradient-based networks, we train the network for 100 epochs. We use a similar architecture with parameters similar to STDP-based networks, with one additional dense layer to replace the SVM. Regarding datasets, we use MNIST, FMNIST, CIFAR-10, and Caltech face/motorbike. Table 4 presents more details about the used architectures.

(0, 0) (0, 0) (0, 0) (0, 0) - - stride (w, h) (1, 1) (2, 2) (1, 1) (2, 2) - - Face/Motor filters (w, h, n) (5, 5, 32) (7, 7) (17, 17, 64) (5, 5) 128 2 padding (w, h) (3, 3) (3, 3) (9, 9) (2, 2) - - stride (w, h) (1, 1) (6, 6) (1, 1) (5, 5) - - Note: w = width, h = height, n = number
Table 5 shows the compression effect on different datasets using SG-based networks. We observe, in particular, the network classification rate, the compression rate on each layer, the simulation time, and the number of trainable parameters. Moreover, we can compare the compressed network (θ = 0.3) and the baseline based on those points.

We can see in Table 5 a slight decrease in the network accuracy when compressing. Compared to the baseline, we can see that for CIFAR-10 and FMNIST, we have a more significant loss (2%) and a minor loss for Face/Motor and MNIST. Moreover, in terms of the compression rate, we can see that compression varies from one layer to another, getting bigger once we move more profoundly in the network, which is observed in all the datasets. Furthermore, we can see that we have a slightly short simulation time in the In Table 6, we explore the effect of increasing/decreasing the maximum/minimum pruning threshold value θ + and θ -by increasing the value of θ (used in equation 5) up to 0.7 on the SG-based network performance and compression rate. Moreover, we apply the experiments using the same network architecture.

We can see from Table 6 that by increasing the value of θ, the compression rate is growing, and some layers are compressed more than 80%. Furthermore, we can see that the compression rate increases when going more profound in the network, which is also the case in STDP-based networks. In terms of the network accuracy, we can see a slight improvement in some cases (Face/Motor and FMNIST) and a small decrease (less than 2%) in others (CIFAR-10 and MNIST). Therefore, for the SG-based networks, we see a clear compression improvement by increasing the θ value. Finally, we can see that the best-recorded accuracy across the experiments does not use a specific θ value. The selection of this value may also depend on the used dataset and architecture.

In Table 7, we compare our work (STDP-based and SG-based) with existing works regarding accuracy and compression rate. Although in our work, we focus on providing a pruning technique that reduces the loss in performance and does not improve the state-of-the-art (SOTA) performance, we can see that the network's performance depends on the training mechanism and the network architecture. Our work reports accuracy close to SOTA with some datasets (MNIST and Face/Motor) and worst in others (FMNIST and CIFAR-10), which may be due to the small size of the network or the lack of hyperparameters tuning.

Moreover, let's compare the two types of networks used in our work. We can see that SG-based networks do better in network performance and compression rate than STDP-based networks. Finally, We can see that different pruning techniques have been used in the existing works, and most of them report a highly compressed network.

Layer-based Progressive Compression

From the experimental results that we presented in Table 2, Table 3, Table 5, and Table 6 we can see that in the case of multi-layer SNN, we get a higher compression rate in the deeper layers compared to the first one. Moreover, in the previous experiments, we used the same α value (α = 0.05 for STDP-based networks, and α + = 0.005, α -= -0.005 for SG-based networks), which represents the initial pruning threshold value across all layers. Therefore, we test in this section a layer-based progressive compression by studying the effect of having an increasing α when going more deep in the network on the performance and compression rate. In our experiments, We increase the α value (α = 0.05 for STDP-based networks, and α + = 0.005, α -= -0.005 for SG-based networks) each time we go to the next layer in the network, and we test on the same multi-layer architectures used with MNIST, FMNIST, and Caltech face/motorbike.

We can see the evolution of the pruning threshold on three different network layers in Figure 5. For the STDP-based networks (Figure 5 (A)), we can see that the pruning threshold value does not increase after crossing the θ value set to 0.7 for all layers, and the time required to cross the threshold rises when going deeper in the network. Moreover, the last possible pruning threshold value equals or exceeds θ. For the SG-based networks (Figure 5 (B)), we can see that for each layer, we have two thresholds, the maximum pruning threshold θ is different from one layer to another due to the weights range being different from one layer to another. Moreover, we can see that the first layer is the first to cross θ, while the last layer threshold is still increasing, which is the opposite of the STDP-based networks.

In Table 8, we run the same experiments using a fixed α and layer-based α for ten times, and we record the results in terms of compression per layer and network performance.

We can see in Table 8 that using a layer-based α allows a higher compression rate compared to a fixed α in both cases (STDP-based and SG-based). Moreover, the performance is maintained and slightly improved in the case of STDP-based networks. On the other hand, a slight loss in accuracy is recorded for the SG-based networks (around 2%), which can be due to the high rate of compression recorded in some internal layers. Finally, we can see for the STDP-based networks that the compression did not increase even with a layer-based α due to the issue we mentioned in Section 3.1.

Compressed Network On The SpiNNaker Board

To evaluate the effectiveness of the proposed approach in neuromorphic implementation, we transfer the learned weights of a baseline and compressed STDP-based network from csnn-simulator to SpiNNaker to observe the network activity in both cases. We use a network of two fully-connected layers of 50 and 128 neurons, which we train for 10 epochs, and We use the MNIST dataset for this experiment.

The trained weights are transferred to the PyNN model without additional adaptation (neuron model or other hyperparameters) compared to the original network used in csnn-simulator. Moreover, it is worth mentioning that for the sake of simplicity, the transfer learning to SpiNNaker concerns only the synapses in this work. Therefore, using the same neuron type and looking for an optimized configuration and hyperparameters will be discussed in future work. (four digits) for the two layers of the network. Therefore, we see a difference in spikes activity in the two use cases, with a drop in the spikes activity of the output layer for the compressed network (especially the first input). However, in the first layer, the difference between the two use cases is not very visible. Hence, Figure 6 shows the effect of the compression approach on the network activity compared to the baseline when using the SpiNNaker board. In Table 9, we compare the number of spikes per layer for the baseline and the compressed network using the MNIST test set (10k digits) and report the estimated energy in both cases. In terms of spikes activity, since spikes generation depends on many factors but essentially on synapses for propagation, we observe a drop in the number of spikes in second layer (almost 16%) when compressed. Moreover, regarding the energy consumption of SpiNNaker, based on literature [START_REF] Painkras | Spinnaker: A 1-w 18-core system-on-chip for massively-parallel neural network simulation[END_REF][START_REF] Stromatias | Optimising the overall power usage on the spinnaker neuromimetic platform[END_REF][START_REF] Sugiarto | High performance computing on spinnaker neuromorphic platform: A case study for energy efficient image processing[END_REF][START_REF] Van Albada | Performance comparison of the digital neuromorphic hardware spinnaker and the neural network simulation software nest for a full-scale cortical microcircuit model[END_REF][START_REF] Stromatias | Power analysis of large-scale, real-time neural networks on spinnaker[END_REF], a significant fraction of the total power for different stages of simulation is spent on the idle mode. Moreover, the reported energy per synaptic event for LIF neurons equals 8 nJ. Therefore, compressing saves approximately 3.2 × e -3 Joule, which means 11.068 uW less power consumption (for simulation time = 290 s).

This paper presents the progressive compression for convolutional spiking neural networks, which we train using STDP+SVM or Surrogate gradient. The proposed approach, an extension of the PP & DSWR for shallow networks, is tested with complex architecture on a classification task with multiple datasets.

We also test the resulting network on the SpiNNaker board by transferring the final weights. Using this approach, we got an average layer compression of more than 70% in some datasets when using SG-based networks, with some layers highly compressed than others (more than 80%). Moreover, we discuss the low compression rate recorded when using STDP-based networks due to the combination of the threshold adaptation mechanism and the progressive compression, which did not help the network maintain a reasonable classification and compression rate.

Furthermore, the layer-based approach discussed in this work provides extra compression (up to 98%) without a significant loss in the network performance (less than 3%). For some datasets, we record a tiny improvement in the network performance. Finally, the tests we conducted on the SpiNNaker board by analyzing the two use cases (baseline and compressed) show a noticeable decrease in the spikes activity when we apply the compression, which will allow the implementation of bigger models in a resource-constrained architecture.

Regarding compression in neural networks, we can use different techniques targeting synapses and other network components. Therefore, the work we presented, which concerns the synapses, can easily be combined with other methods (neuron compression, weight quantization, etc.) to improve the compression even more. Finally, as future works, a detailed parameters exploration for the different parameters in the model or the two formulas (α & β) can improve the compression, testing this approach on networks trained with other surrogate gradient methods, more complex datasets, and different tasks (other than image classification). For SpiNNaker, a more profound analysis of the compression effect and parameters exploration is needed when training onboard, with a clear report on the energy and the performance of the resulting network.

Figure 1 .

 1 Figure 1. Compression flowchart for Convolutional SNN

Figure 2

 2 Figure2shows how the threshold value changes during training (20 epochs) using our approach in both cases (STDP-Based and SG-Based). Moreover, we can also follow the evolution of the compression rate. After a couple of epochs, we can see in the case of the STDP-Based network (Figure2 (A)) that the threshold value is stable (around 0.32) due to θ = 0.3. Nevertheless, the compression keeps increasing due to the learning and the reinforcement applied in the network. Since we have two pruning thresholds for the SG-Based network, we can see in Figure2(B) how the thresholds change while the compression increases.

Figure 2 .

 2 Figure 2. Compression rate and pruning threshold evolution for (A) STDP-Based network and (B) SG-Based network

1 .

 1 Difference-of-Gaussian: DoG in = 1.0, DoG out = 4.0,DoG size = 7.0 2. STDP: η w = 0.1, τ STDP = 0.1 3. Neuron Threshold Adaptation: η th = 1.0, th min = 1.0, t obj CIFAR-10 = 0.95, t obj MNIST = t obj FMNIST = 0.75, t obj Face/Motor = 0.80

Figure 3 .

 3 Figure 3. The network topology for (A) STDP-based networks (B) Surrogate Gradient-based networks

Figure 4 .

 4 Figure 4. The threshold adaptation and progressive compression impact on (A) the thresholds of the neurons and (B) single neuron weights activity

Figure 5 .

 5 Figure 5. Layer-based pruning threshold activity using convolutional SNN for (A) STDP-based networks and (B) SG-based networks

Figure 6 .

 6 Figure 6. (A) spikes accumulated activity in the output layer. (B) and (C) represent spikes per layer for four inputs in the case of baseline and compressed STDP-based network, respectively

Table 1 .

 1 The architectures used in the experiments of STDP-based networks

	Dataset	Parameters			Architecture
			Conv1	Pool1	Conv2	Pool2	Fc1
		filters (w, h, n) (5, 5, 128)	-	-	-	-
	CIFAR-10	padding (w, h)	(0, 0)	-	-	-	-
		stride (w, h)	(1, 1)	-	-	-	-
		filters (w, h, n) (5, 5, 32)	(2, 2)	(5, 5, 128)	(2, 2)	(4, 4, 1024)
	MNIST & FMNIST	padding (w, h)	(0, 0)	(0, 0)	(0, 0)	(0, 0)	(0, 0)
		stride (w, h)	(1, 1)	(2, 2)	(1, 1)	(2, 2)	(1, 1)
		filters (w, h, n) (5, 5, 32)	(7, 7)	(17, 17, 64)	(5, 5)	(5, 5, 128)
	Face/Motor	padding (w, h) (5/2, 5/2) (7/2, 7/2) (17/2, 17/2) (5/2, 5/2) (5/2, 5/2)
		stride (w, h)	(1, 1)	(6, 6)	(1, 1)	(5, 5)	(1, 1)

Table 2 .

 2 Accuracy, compression, and layers activity (spikes & synaptic updates) for STDP-based networks

		Accuracy ± std Compression/layer Simulation time	Spikes/layer	Synaptic updates/layer
	CIFAR-10	baseline compressed 53.45 ± 0.28 54.63 ± 0.37	-19.91	2:21:34 2:13:33	5x10 6 5x10 6	7.5x10 8 1.6x10 7
	98.43 ± 0.10 compressed 87.82 ± 0.49 Face/Motor baseline	-44.12|46.00|45.77	0:41:51 0:36:31	39996|40000|40000 39995|40000|40000	1999810|3.69x10 8 |6.40x10 7 429385|5.16x10 7 |8.22x10 6
	MNIST	baseline compressed 96.55 ± 0.11 98.18 ± 0.07	-42.43|27.17|13.16	2:29:56 2:48:16	3982386|5954200|6x10 6 1.99x10 8 |4.76x10 9 |1.22x10 10 3988197|5950494|6x10 6 4.13x10 7 |5.67x10 7 |1.24x10 8
	FMNIST	baseline compressed 83.50 ± 0.31 84.65 ± 0.21	-50.25|27.17|13.17	3:50:05 3:52:29	5604907|5999531|6x10 6 2.80x10 8 |4.79x10 9 |1.22x10 10 5609543|5999693|6x10 6 1.35x10 7 |5.51x10 7 |1.24x10 8

Table 3 .

 3 Max pruning threshold θ effect on the accuracy and compression for STDP-based networks

					Max pruning threshold θ
			0.3	0.4	0.5	0.6	0.7
	CIFAR-10 Accuracy ± std Compression/layer	53.45 ± 0.28 19.91	53.01 ± 0.33 19.80	53.49 ± 0.28 19.97	53.58 ± 0.45 19.61	53.71 ± 0.41 19.91
	Face/Motor	Accuracy ± std Compression/layer 44.12|46.00|45.77 41.56|47.67|47.26 42.93|48.73|47.49 41.62|49.41|47.24 42.87|49.85|47.56 87.82 ± 0.49 89.29 ± 0.30 89.79 ± 0.61 90.50 ± 0.47 90.95 ± 0.37
	MNIST	Accuracy ± std Compression/layer 42.43|27.17|13.16 42.12|27.22|13.16 41.43|27.22|13.17 41.75|27.18|13.16 43.00|27.29|13.16 96.55 ± 0.11 96.64 ± 0.19 96.88 ± 0.19 96.78 ± 0.18 96.80± 0.34
	FMNIST	Accuracy ± std Compression/layer 50.25|27.17|13.17 50.68|27.25|13.17 50.56|27.17|13.17 50.31|27.23|13.10 50.18|27.25|13.17 83.50 ± 0.31 83.41 ± 0.11 83.44 ± 0.23 83.46 ± 0.19 83.61 ± 0.40

Table 4 .

 4 The architectures used in the experiments of SG-based networks

	Dataset	Parameters	Architecture
		Conv1 Pool1	Conv2	Pool2 Fc1 Fc2
		filters (w, h, n) (5, 5, 32) (2, 2) (5, 5, 128) (2, 2) 1024 10
	MNIST, FMNIST, CIFAR-10	padding (w, h)

Progressive Layer-based Compression for Convolutional SNN

Table 5 .

 5 Accuracy, compression, and layers activity (spikes & synaptic updates) for SG-based networks CIFAR-10 and Face/Motor). However, for MNIST and FMNIST, we have a longer simulation time. Finally, for the trainable parameters, we can see that we have a considerable decrease in trainable parameters for all the datasets due to the compression.

		Accuracy ± std	Compression/layer	Simulation time Trainable params
	CIFAR-10	baseline compressed 57.82 ± 0.16 33.42|55.36|72.71|67.65 59.87 ± 0.45 -	1:30:08 1:22:23	3,391,840 952,210
	97.60 ± 1.11 compressed 96.17 ± 2.94 26.88|55.92|51.14|61.56 -Face/Motor baseline	0:30:28 0:26:32	1,036,320 461,144
	MNIST	baseline compressed 98.69 ± 0.14 41.25|85.31|73.31|66.22 99.05 ± 0.13 -	0:55:14 1:08:13	2,210,592 1,008,487
	FMNIST	baseline compressed 83.21 ± 0.47 28.25|55.83|60.32|64.02 85.63 ± 0.37 -	0:55:46 1:19:17	2,210,592 901,983
	case of some datasets (

Table 6 .

 6 Positive and negative pruning threshold effect on the accuracy and compression for SG-based networks, by updating θ value

	θ value

Table 7 .

 7 Accuracy and compression compared to existing works

			Training	Architecture	Accuracy ± std Pruning technique Compression
		Our work	STDP + SVM	1 conv layer	53.71 ± 0.41	PP & DSWR	19.91
	CIFAR-10	Our work	Surrogate Gradient	2 conv-pool + 2 fc layer	57.82 ± 0.16	PP & DSWR	57.29
		(Deng et al., 2021)	Surrogate Gradient	7 conv + 2 fc layer	89.15	ADMM-based	50.00
		(Chen et al., 2021)	Surrogate Gradient	6 conv + 2 fc layer	92.54	Grad R	71.59
		(Nguyen et al., 2021)	STDP + SVM	3 conv-pool layer	95.70	Static threshold	92.83
	Face/Motor	Our work	STDP + SVM	2 conv-pool + 1 fc layer	90.95 ± 0.37	PP & DSWR	46.76
		Our work	Surrogate Gradient	2 conv-pool + 2 fc layer	97.93 ± 0.83	PP & DSWR	53.38
		(Zhang et al., 2022)	Back-propagation	2 conv-pool + 3 fc layer	99.50	-	-
		Our work	STDP + SVM	2 conv-pool + 1 fc layer	96.88 ± 0.19	PP & DSWR	27.27
	MNIST	Our work	Surrogate Gradient	2 conv-pool + 2 fc layer	98.69 ± 0.14	PP & DSWR	66.52
		(Chen et al., 2021)	Surrogate Gradient	2 fc layer	98.59	Grad R	74.29
		(Diehl et al., 2015)	ANN-SNN Conversion	2 conv-pool + 1 fc layer	99.14	-	-
		Our work	STDP + SVM	2 conv-pool + 1 fc layer	83.61 ± 0.40	PP & DSWR	30.20
	FMNIST	Our work	Surrogate Gradient	2 conv-pool + 2 fc layer	83.66 ± 0.56	PP & DSWR	71.88
		(Ranjan et al., 2020)	Back-propagation	2 conv + 1 pool + 2 fc layer	89.00	-	-
		(Zhang et al., 2022)	Back-propagation	2 conv-pool + 3 fc layer	90.1	-	-

Table 8 .

 8 Layer-based compression compared to having a fixed α

	Accuracy ± std	Compression/layer

Table 9 .

 9 Spikes activity and energy estimation on SpiNNaker # of Spikes (L1) # of Spikes (L2) Energy (J)

	Baseline	999, 786	2, 559, 907	28.48 × e -3
	Compressed	999, 827	2, 158, 839	25.27 × e -3

https://archive.softwareheritage.org/swh:1:rev:d8ead5b48684e309a58ceba04664b4849e9ae5c2

https://archive.softwareheritage.org/swh:1:dir:fa5f52f0d1c769c1d4ebc691fd2dea61d3bbce5f

ACKNOWLEDGMENTS

This work was supported in part by IRCICA (Univ. Lille, CNRS, USR 3380 -IRCICA, F-59000 Lille, France) under the Bioinspired Project. Experiments presented in this paper were carried out using the Grid'5000 experimental testbed, being developed under the INRIA ALADDIN development action with support from CNRS, RENATER and several Universities as well as other funding bodies (see https://www.grid5000.fr).

FUNDING

This work was partially funded by the European CHIST-ERA APROVIS3D project, and the Luxant-ANVI industrial chair (I-Site and Metropole Européene de Lille).

CONFLICT OF INTEREST STATEMENT

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

AUTHOR CONTRIBUTIONS

H.E. and M.F. contributed to formulating the study. H.E. conducted the experiments on csnn-simulator and Norse. M.F. implemented the networks on the SpiNNaker board. In addition, both contributed to writing the paper.