Ghouthi Boukli Hacene

Carlos Lassance

Vincent Gripon

Matthieu Courbariaux

Yoshua Bengio

Ghouthi Boukli Hacene

Attention Based Pruning for Shift Networks

are published or not. The documents may come   L'archive ouverte pluridisciplinaire

Introduction

Convolutional Neural Networks (CNNs) are the state-of-the-art in many computer vision tasks, such as image classification, object detection and face recognition [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF]. To achieve top-rank accuracy, CNNs rely on the use of a large number of trainable parameters, and considerable computational complexity. This is why there has been a lot of interest in the past few years towards the compression of CNNs, so that they can be deployed onto embedded systems. The purpose of compressing DNNs is to reduce memory footprint and/or computational complexity while mitigating losses in accuracy.

Prominent works to compress neural networks include binarizing (or quantifying) weights and activations [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF], [START_REF] Soulié | Compression of deep neural networks on the fly[END_REF], [START_REF] Rastegari | Xnor-net: Imagenet classification using binary convolutional neural networks[END_REF], [START_REF] Lin | Neural networks with few multiplications[END_REF], Li et al. [2016a], [START_REF] Zhou | Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients[END_REF], Han et al. [2015a], [START_REF] Wu | Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions[END_REF], pruning network connections during or before training Han et al. [2015a,b], Li et al. [2016b], [START_REF] Ardakani | Sparsely-connected neural networks: towards efficient vlsi implementation of deep neural networks[END_REF], using grouped convolutions [START_REF] Chollet | Xception: Deep learning with depthwise separable convolutions[END_REF], [START_REF] Andrew G Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF], [START_REF] Sandler | Mobilenetv2: Inverted residuals and linear bottlenecks[END_REF], introducing new components [START_REF] Zhang | Interpretable convolutional neural networks[END_REF], [START_REF] Juefei-Xu | Local binary convolutional neural networks[END_REF], using convolutional decomposition [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], or searching for a lightweight neural network architecture Iandola et al. [2016].

Recently, the authors of [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF] have proposed to replace the generic convolution operator with the combination of shifts and simpler 1x1 convolutions. In their work, the shifts are hand-crafted and all the parameters to be trained are in the 1x1 convolutions. These operations are particularly well suited to embedded devices [START_REF] Boukli Hacene | Quantized guided pruning for efficient hardware implementations of convolutional neural networks[END_REF], [START_REF] He | Soft filter pruning for accelerating deep convolutional neural networks[END_REF].

In this paper, we introduce the Shift Attention Layer (SAL), which can be seen as a selective shift layer. Indeed, the proposed layer starts with a vanilla convolution and learns to transform it into a shift layer throughout the training of the network function. It uses an attention mechanism [START_REF] Vaswani | Attention is all you need[END_REF] that selects the best shift for each feature map of the architecture, what can be seen as 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

L C

(1) In panel (3), we depict the corresponding obtained shift layer: for each slice along the input feature maps (C), the cell with the highest attention is kept and the others are disregarded. As a consequence, the initial convolution with a kernel size S has been replaced by a convolution with a kernel size 1 on a shifted version of the input X. As such, the resulting operation in panel ( 3) is exactly the same as the shift layer introduced in [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF], but here the shifts have been trained instead of being arbitrarily predetermined.

S X Y d L C (2) S X Y d L Shifted X Y d C (3)
a pruning technique. We demonstrate it is able to significantly outperform original shift layers [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF] and other pruning techniques at the cost of requiring more parameters during training, but ends up with less parameters for inference. It is thus of particular interest for implementing the inference process on resource-limited systems.

The outline of the paper is as follows. In Section 2 we explain the proposed method. In Section 3 we present related work. In Section 4 we present experiments on challenging computer vision datasets. Finally, In Section 5 we discuss future work and conclude. All the code used to run the experiments in this paper is available online: https://hiddenduringreviewing.

Methodology

In this section, we first review the classical spatial convolution operation and see how it can be related to the shift operation defined in [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF]. We then introduce our proposed method.

Convolution/Shift operation

Let us consider the 1d case (other cases can be easily derived). Let us denote by X ∈ R C×L an input tensor of a convolutional layer, where C is the number of channels and L is the dimension of each feature map (a.k.a. spatial dimension). We denote W ∈ R D×C×S the corresponding weight tensor, where D is the number of output channels, S the kernel size, and Y ∈ R D×L the output tensor. Disregarding padding (i.e. border effects), the convolution operation is defined through Equation (1) and depicted in Figure 1, panel (1).

y d, = C c=1 S =1 x c, + -S/2 w d,c, , ∀d, . (1)
A shift layer is obtained when connections in W are pruned so that exactly one connection remains for each slice W d,c,• , ∀d, c. We denote d,c the index such that w d,c, d,c is not pruned. Then the shift operation is defined in Equation (3) and depicted in Figure 1: panel (3).

y d, = C c=1 x c, + d,c -S/2 w d,c, d,c (2)
= C c=1 xc, wd,c , (3)
where xc, = x c, + d,c -S/2 and wd,c = w d,c, d,c . We observe that the shift operation boils down to shifting the input tensor X then convoluting with a kernel of size 1 (in the equation, the kernel W is indexed only by the input and output feature maps). In the original work [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF], the authors proposed to arbitrarily predetermine which shifts are used before training the architecture. To the contrary, in this work, we propose a method that learns which shifts to perform during the training process.

Shift Attention Layer (SAL)

The idea we propose is to enrich a classical convolution layer with a selective tensor A which aims at identifying which connections should be kept in each slice of the weight tensor. As such, we introduce A ∈ R D×C×L a tensor containing as many elements as weights in the weight tensor. Each value of A is normalized between 0 and 1 and represents how important the corresponding weight in W is (c.f. for each training iteration do T ← αT

for d := 1 to D do for c := 1 to C do A d,c,• ← A d,c,• sd(A d,c,• ) A d,c,• ← Sof tmax(T A d,c,• ) end for end for W A ← W • A (• is the pointwise multiplication)
Compute standard convolution as described in Equation 1 using input tensor X and weight tensor W A instead of W . Update W and A via back-propagation. end for Note that contrary to the vanilla shift layers where the same number of shifts is performed in every direction, at the end of the training process the resulting shift layer can have an uneven distribution of shifts (c.f. Section 4). This comes with a price, which corresponds to the memory needed to retain which shift is performed for each input feature map. In order to be fair, we thus reduce the number of feature maps in the networks we use in the experiments so that the total memory is comparable.

In the next section we present related works, thus our proposed method can easily be compared to previously introduced methods.

Related Work

Let us introduce related works aiming at reducing complexity and memory footprints of CNNs.

In a first line of works, authors have proposed to binarize weights and activations [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF][START_REF] Courbariaux | Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1[END_REF], [START_REF] Rastegari | Xnor-net: Imagenet classification using binary convolutional neural networks[END_REF], and then replace all multiplications by low cost multiplexers. Other works have proposed to use K-means to quantize weights Han et al. [2015b], [START_REF] Ullrich | Soft weight-sharing for neural network compression[END_REF], [START_REF] Wu | Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions[END_REF] and thus reduce the model size of CNNs.

Another line of work relies on network pruning Han et al. [2015a]. For example Li et al Li et al. [2016b] use the sum of absolute weights of each channel to select and prune redundant ones. In the same vein, He et al [START_REF] He | Soft filter pruning for accelerating deep convolutional neural networks[END_REF] proposed a novel Soft Filter Pruning (SFP) approach to prune dynamically filters in a soft manner. Using the reconstruction error of the last layer before classification, Yu et al [START_REF] Yu | Nisp: Pruning networks using neuron importance score propagation[END_REF] introduce the Neuron Importance Score Propagation (NISP) method. The main idea is to estimate the importance of each neuron in each layer in the backward propagation of the scores obtained from the reconstructed error. Huang et al [START_REF] Huang | Learning to prune filters in convolutional neural networks[END_REF] propose a reinforcement learning based method, in which an agent is trained to maximize a reward function in order to improve the accuracy when pruning selected channels. Yamamoto et al [START_REF] Yamamoto | Pcas: Pruning channels with attention statistics[END_REF] use a channel-pruning technique based on attention statistics by adding attention blocks to each layer and updating them during the training process to evaluate the importance of each channel. Compared to these works, the method we propose can be seen as a structured pruning technique in which most of the weights of a convolution are pruned but one per slice.

For mobile applications with limited resources, specific neural network architectures have been introduced. Iandola et al [START_REF] Forrest N Iandola | Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size[END_REF] propose a fire module to define SqueezeNet, a very small neural network with an acceptable accuracy. In Howard et al. [2017], [START_REF] Sandler | Mobilenetv2: Inverted residuals and linear bottlenecks[END_REF], the authors use depthwise separable convolutions to define MobileNet and build lightweight CNNs.

Other approaches focus on a decomposition of the convolution. Simonyan et al [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] reduce the number of parameters of VGG by replacing a 5 × 5 convolutions by two 3 × 3 convolutions each. In [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], the authors decompose 7 × 7 convolutions into 1 × 7 and 7 × 1 convolutions. Our proposed work builds upon Wu et al [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF], where the authors introduce a shift operation, that we call shift layers throughout this article, as an alternative to spatial convolutions. The authors deconstruct 3 × 3 spatial convolutions into the concatenation of a shift transform on the input and 1 × 1 convolutions. As such, they are able to considerably reduce the complexity and memory usage of CNNs. In Hacene et al. [2018], the authors show it is possible to exploit this procedure for specific hardware targets. In [START_REF] Jeon | Constructing fast network through deconstruction of convolution[END_REF], the authors propose an active shift layer (ASL) that improves the accuracy of the shift operation method, and replaces the fixed handcrafted integer parameters of the shift layers by learned floating point parameters. Consequently, the result architecture requires to perform interpolations and does not fall into the original shift layer formulation.

In this contribution, we introduce Shift Attention Layers (SALs), a novel pruning-shift operation method that ends up replacing convolutions with shift layers by the end of training. Contrary to [START_REF] Jeon | Constructing fast network through deconstruction of convolution[END_REF], the proposed methodology result in a network that is exactly as described in [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF], with the main difference that shifts are learned instead of being predetermined. We demonstrate in this paper that it results in better accuracy for the exact same complexity and number of parameters.

Experiments

In this section we present the benchmark protocol used to evaluate the proposed method, and then compare the obtained performance with a CNN baseline, pruning methods and vanilla shift layers.

Benchmark Protocol

We evaluate using three object recognition datasets: CIFAR10, CIFAR100 and ImageNet ILSVRC 2012. To be comparable with the relevant literature, we use the same architectures from previous papers [START_REF] Yamamoto | Pcas: Pruning channels with attention statistics[END_REF], [START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF], [START_REF] Jeon | Constructing fast network through deconstruction of convolution[END_REF], that is Resnet-20/56 on CIFAR10 and Resnet-20/50 on CIFAR100 using the following parameters: the training process For the evaluations using CIFAR10 or CIFAR100 [START_REF] He | Identity mappings in deep residual networks[END_REF], the training batch size is 128, the initial/final softmax temperatures are 6.7/0.02, and the temperature is multiplied at each step by α = 0.99994, 0.99996 when using 300, 400 epochs respectively. For ImageNet ILSVRC 2012, we used Resnet-w32 and Resnet-w64 defined in [START_REF] Jeon | Constructing fast network through deconstruction of convolution[END_REF] using the following parameters: the total number of epochs is 90, the training batch size is 1024, the learning rate starts at 0.1 and is divided by 10 after each 30 epochs, initial/final softmax temperatures are 6.7/0.016 so that the temperature update at each step is α = 0.99995. We also used standard data augmentation defined in [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF].

Let us point out that the values of temperatures were obtained by using a grid search. The fact the final temperature is not zero means that the tensors A may contain nonbinary values. This is why we binarize A using a hard max to obtain the corresponding shift layers before evaluating on the test set.

Results

The first experiment consists in comparing the accuracy, memory and number of operations of the proposed method compared to other shift layer based methods and pruning methods. We run tests using CIFAR10 and CIFAR100. Table 1 shows that our method achieves a better accuracy with fewer parameters than the baseline and other shift-module based methods. Tables 2 and3 show that the proposed method is comparable or better in term of accuracy and number of parameters/floating point operations (FLOPs) with other pruning methods.

As mentioned in Section 2, the number of shifts in each direction can be uneven at the end of the training procedure. As such, in a second run of experiments, we look at the proportions of each shifts (relative position in kernel) depending on the layer depth in the architecture. In Figure 2, a heat-map represents the distribution of kept weights at different relative positions through W d,c,•,• , ∀d, c slices, for the 4 first layers (first row), and the 4 last layers (second row), of Resnet-20 trained on CIFAR10, where attention tensors A values are initially drawn uniformly at random. We observe that at the end of the training process, first layers seem to yield a uniform distribution of kept weights. To the contrary, for the last layers, there is a clear asymmetry that favours corners. This interestingly suggests that shift-layers would benefit from a non regular number of shifts in each direction. 77.6%(78) 3.9 (16.9) 251( 1308) To see how much the initialization of A is related to the distribution of kept weights, we then perform another experiment where A is initialized uniformly at random but then the centre value A d,c, S/2 , S/2 is changed to the maximum over the corresponding slice max(A d,c,•,• ). We observe in Figure 3 that almost all kept weights in the first layer are slice centres. Subsequent layers yield an almost uniform distribution, and we observe the same kept weights distribution for the last layers as in the previous experiment. We also plot a heat-map of kept weights in Resnet-56 trained on CIFAR10, and where attention tensor A values are initially drawn uniformly at random. Figure 4 shows that for the first layers, the number of kept weights is more important in the centre row than at other positions. However, we see on the last layers that there is more kept weights in the corners than other positions.

-1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 0.0604 0.3172
For further results, we run an experiment in which we replace all 3 × 3 Resnet-20 kernels by 5 × 5 kernels, and train the network on CIFAR10. We observe in Figure 5 that the weights of the center in first layers are more important that other positions. We also see that on the last layers the weight distribution is still not uniform, and the weights on the corners are more important in the last layer.

From all these experiments, we consistently observe that in deeper layers, the method tends to keep more weights in corner positions than others, and this independently from initialization process or neural network architecture. This finding interestingly questions the hyperparameters used by the corresponding architectures. It clearly seems the network is more interested in locality in the initial layers than it is in the last layers. Based on this finding, we modified the vanilla shift layer method, using an equivalent uneven distribution of shifts as the one found in our experiments. As such, shifts are predetermined but not uniform. We obtained an accuracy of 94.8% on Resnet-20 and CIFAR10, to be compared to the 93.17% accuracy from Table 1. Interestingly, this accuracy is even better than the results obtained using the method in [START_REF] Jeon | Constructing fast network through deconstruction of convolution[END_REF]. On the other hand, the obtained accuracy remains lower than that of the proposed method, suggesting that selecting the shifts during the learning process is still more efficient than having a good choice of predetermined shift proportions.

In a third experiment, we observe the effect of initial and final temperature choices on accuracy. Figure 6: left represents the evolution of accuracy of Resnet-20/56 trained on CIFAR10 and Resnet- -1 0 1 -1 0 1 It shows that the accuracy decreases when the final temperature becomes too high. Note that when the final temperature is large, obtained values in A at the end of the training process can be far from binary. In all cases, we round the values in A to the nearest integer before computing the accuracy. This experiment shows that final temperature values need to be small enough so the softmax can push the highest value to 1 and the other values to 0. Figure 6: right shows the behaviour of the accuracy of Resnet-20/56 trained on CIFAR10 and Resnet-20/50 trained on CIFAR100 when initial temperature is changed and final temperature is fixed at 0.02. We see an interesting region between 10 and 6.7 in which the accuracy is better. It is worth mentioning that the choice of initial and final temperature is sensitive with respect to the obtained accuracy. Throughout our experiments, we observed that a too slow decrease in temperature causes the architecture to get stuck in local minima that are poorly fitted to the ending rounding operation. To the contrary, a too fast decrease in temperature prevents the learning procedure from finding the best shifts and boils down to an accuracy that is very similar to that of vanilla shift layers.

-1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 0.0604 0.3172
-1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 -1 0 1 0.0658 0.1787
-2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2 -2 -1 0 1 2
In the fourth experiment, we compare the accuracy, memory usage and FLOPs of SAL against vanilla Shiftnet and standard CNN on ImageNet ILSVRC 2012. Table 4 shows that SAL is able to obtain better accuracies than vanilla Shiftnet and standard CNN for the same memory and FLOPs budget. As last experiment, we propose to compare SAL with some other compression methods introduced in Section 3. To get a good approximation of memory needed to implement a neural network on a limited resources embedded system, we consider both weights and activations when one input data is processed, thus the memory footprint of a DNN will be the memory needed to store both weights and activations. In our evaluation we compare SAL with Binary Connect (BC) [START_REF] Courbariaux | Binaryconnect: Training deep neural networks with binary weights during propagations[END_REF] and Binary Weight Network (BWN) [START_REF] Rastegari | Xnor-net: Imagenet classification using binary convolutional neural networks[END_REF] applied to Resnet-20, MobileNetV2 Sandler et al. [2018] andSqueezenet Iandola et al. [2016]. We also perform the comparison with another version of SAL denoted SAL2, in which we keep two weights per kernel instead of one. We use Resnet-20 with different number of weights and activations as baseline. Figure 7 shows that the baseline outperforms MobileNetV2, Squeezenet and applying BC on Resnet-20. It also shows that SAL and SAL2 outperform all other methods.

Conclusion

In this contribution, we proposed a novel attention-based pruning method that transforms a convolutional layer into a shift layer. The resulting network provides interesting improvements in terms of memory and computation time, while keeping high accuracy. Compared to existing methods, we showed that the proposed method results in improved accuracy for similar memory budgets as existing shift-layer based methods, using challenging vision datasets. The proposed method is thus an interesting alternative to channel-pruning methods.

Figure 1 :

 1 Figure1: Overview of the proposed method: we depict here the computation for a single output feature map d, considering a 1d convolution and its associated shift version. Panel (1) represents a standard convolutional operation: the weight filter W d,•,• containing SC weights is moved along the spatial dimension (L) of the input to produce each output in Y d . In panel (2), we depict the attention tensor A on top of the weight filter: the darker the cell, the most important the corresponding weight has been identified to be. At the end of the training process, A should contain only binary values with a single 1 per slice A d,c,• . In panel (3), we depict the corresponding obtained shift layer: for each slice along the input feature maps (C), the cell with the highest attention is kept and the others are disregarded. As a consequence, the initial convolution with a kernel size S has been replaced by a convolution with a kernel size 1 on a shifted version of the input X. As such, the resulting operation in panel (3) is exactly the same as the shift layer introduced in[START_REF] Wu | Shift: A zero flop, zero parameter alternative to spatial convolutions[END_REF], but here the shifts have been trained instead of being arbitrarily predetermined.

  Figure 1: panel (2)) with respect to the task to be solved. At the end of the training process, A becomes binary, with only one nonzero element per slice A d,c,• , corresponding to the weights in W that should be kept. More precisely, each slice A d,c,• is normalized using a softmax function with temperature T . The temperature is decreased smoothly along the training process. Note that in order to force the mask A to be selective, we first normalize each slice A d,c,• so that it has 1 standard deviation (sd). Algorithm 1 summarizes the training process of one layer. At the end of training, the selected weight in each slice W d,c,• corresponds to the maximum value in A d,c,• . Algorithm 1 SAL algorithm of one layer Inputs: Input tensor X, Initial softmax temperature T , Constant α < 1.

Figure 2 :

 2 Figure 2: Heat maps representing the average values in A for various layers in the Resnet-20 architecture trained on CIFAR10. In this experiment, values in A are initialized uniformly at random. The first row represents the 4 first layers and the second row the 4 last layers of Resnet-20.

Figure 3 :

 3 Figure 3: Heat maps representing the average values in A for various layers in the Resnet-20 architecture trained on CIFAR10. In this experiment, values in A are initialized uniformly at random but the centre value that takes the maximum over the corresponding slice. The first row represents the 4 first layers and the second row the 4 last layers of Resnet-20.

Figure 4 :

 4 Figure 4: Heat maps representing the average values in A for various layers in the Resnet-56 architecture trained on CIFAR10. In this experiment, values in A are initialized uniformly at random. The first row represents the 4 first layers and the second row the 4 last layers of Resnet-56.

Figure 5 :Figure 6 :

 56 Figure 5: Heat maps representing the average values in A for various layers in the Resnet-20 architecture with 5 × 5 kernels trained on CIFAR10. In this experiment, values in A are initialized uniformly at random. The first row represents the 4 first layers and the second row the 4 last layers.

Figure 7 :

 7 Figure 7: Evolution of accuracy when applying compression methods on different DNN architectures trained on CIFAR10.

Table 1 :

 1 Comparison of accuracy and number of parameters between the baseline architecture (ResNet20), ShiftNet, ASNet, and SANet (the proposed method) on CIFAR10 and CIFAR100.

	CIFAR10	CIFAR100
	Accuracy Params (M) Accuracy Params (M)

consists of 300 epochs for Resnet-20 and 400 epochs for Resnet-50/56, the learning rate starts at 0.1 and is divided by 10 after each 100 epochs.

Table 3 :

 3 Comparison of accuracy, number of parameters and number of floating point operations (FLOPs) between baseline architecture (Resnet-50), SANet (the proposed method) , and some other pruning methods on CIFAR100. Note that the number between () refers to the result obtained by the baseline used for each method.

			CIFAR100
		Accuracy	Params (M) FLOPs (M)
	Pruned-B Li et al. [2016b]	73.6%(74.46)	7.83(17.1)	616(1409)
	Pruning PCAS Yamamoto and Maeno [2018] 73.84%(74.46) 4.02(17.1)	475(1409)
	SANet (ours)

Table 4 :

 4 Comparison of accuracy, number of parameters and FLOPs between a standard CNN, SAL and vanilla Shiftnet on ImageNet ILSVRC 2012.

	Top-1 Top-5 Params (M) FLOPs (M)