Marios Krestenitis

Nikolaos Passalis
email: passalis@csd.auth.gr

Alexandros Iosifidis

Moncef Gabbouj
email: moncef.gabbouj@tuni.fi

Anastasios Tefas
email: tefas@csd.auth.gr

Human Action Recognition using Recurrent Bag-of-Features Pooling

HAL is

Introduction

Computer vision is one of the most active and continuously expanding research fields, while with the advent of deep learning (DL) many powerful visual information analysis methods for high dimensional data have been recently proposed [START_REF] Lecun | Deep learning[END_REF]. The typical pipeline of a visual information analysis approach involves at least the following two steps: a) feature extraction, in which lower level information is extracted from small spatial or temporal segments of the data, and b) feature aggregation, in which the extracted information is fused into a compact representation that can be used for the subsequent tasks, e.g., classification, retrieval, etc. DL unified, to some extent, these two steps by employing deep trainable feature extraction layers, e.g., convolutional layers, that are combined with naive pooling operators, e.g., max or average pooling, to lower the complexity of the model and provide translation invariance. Indeed, the outstanding performance of Convolutional Neural Networks (CNNs) in complex and challenging image analysis tasks, has confirmed their ability to extract meaningful feature vectors with high discriminative power [START_REF] Lecun | Deep learning[END_REF]. However, these powerful feature vectors are crushed through the pooling layers of the network, that usually implement the pooling operation in a less sophisticated manner. As we will demonstrate through this paper, this can lead to significant information loss, especially in cases where the informative content of the data is sequentially distributed over the spatial or temporal dimension, e.g., videos, which often requires extracting fine-grained temporal information, which is discarded by these pooling approaches.

The aforementioned limitations can be better understood through the following example. Consider the task of activity recognition in videos, where the action of sitting down must be distinguished from the action of standing up. Feature vectors can be extracted from every video frame or a sequence of them by using a deep CNN. However, the pooling layers, as the weak point of the network, dull the expressiveness of the extracted feature vectors and produce less discriminative representations by pooling over the time dimension. For example, assume that a sequence of feature vectors is extracted from a given video instance of action sitting down. Let that sequence, notated as S 1 = [a 1 , a 2 , a 3 ], be composed of three feature vectors a 1 = [0, 0, 1] T , a 2 = [0, 1, 0] T , and a 3 = [1, 0, 0] T , which are the feature vectors that correspond to the sub-actions standing above a chair, bending knees and sitting on a chair, respectively. Similarly, consider the same feature vector sequence, but in reverse order, i.e., S 2 = [a 3 , a 2 , a 1 ], that represents a video instance of the activity standing up. Also, let s i denote the aggregated representation extracted for the i-th video. Note that when average or max pooling is applied over both sequences, then the same representation is extracted for both videos, i.e., s

1 = s 2 = [max i [a i ] 1 , max i [a i ] 2 , max i [a i ] 3 ] T = [1, 1, 1]
T for max pooling (where the notation [x] i is used to refer to the i-th element of vector x)

or s 1 = s 2 = 1 3 3 i=1 a i = [ 1 3 , 1 3 , 1 3 ]
T for average pooling. Therefore, even though the employed CNN was capable of perfectly recognizing the fundamental subactions from still frames, the resulting deep model cannot discriminate between the two actions due to the employed pooling layer. Therefore average and max pooling layers are not capable of capturing the fine-grained spatial or temporal interactions between the feature vectors extracted from a given sequence. Note that in other cases, instead of employing a pooling layer, the extracted feature map may be flatten to a vector and fed to the subsequent fully connected layer. However, this approach makes it impossible for the network to handle inputs of arbitrary size, while it significantly reduces the invariance of the network to temporal shifts (the features must always arrive at the exact same moment).

In this paper, we propose a novel stateful recurrent pooling approach that is capable of overcoming these limitations. The proposed method builds upon the well-known Bag-of-Feature (BoF) model [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF], which is capable of creating a constant-length representation of a multimedia object, e.g., video, audio, etc., by compiling a histogram over its quantized feature vectors. Therefore, every object is represented using a fixed-length histogram over the learned codewords.

The codewords/dictionary can be either learned using generative/reconstruction approaches [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF], or by employing discriminative dictionary learning methods [START_REF] Iosifidis | Discriminant bag of words based representation for human action recognition[END_REF], which usually better fit classification tasks. This scheme found application in numerous computer vision tasks, including scene recognition [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF], texture classification [START_REF] Zhang | Local features and kernels for classification of texture and object categories: A comprehensive study[END_REF], etc. BoF can be also combined with deep neural networks to provide more powerful trainable pooling layers that can better withstand distribution shifts, while handling inputs of various sizes [START_REF] Passalis | Learning bag-of-features pooling for deep convolutional neural networks[END_REF]. Despite its remarkable success in various tasks and its ability to handle variable size inputs, the main drawback of BoF-based methods is the loss of spatial and temporal information, as well as their inability to capture the geometry of input data [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF].

These drawbacks severely limit the ability of BoF to process temporal or sequential data, such as video data, since it is not capable of capturing the temporal succession of events. To overcome this limitation, we suggest that the quantization process should take into account the order in which the features arrive, allowing for forming temporal codewords that also capture the interrelation between the feature vectors. As a result, a BoF method employing a stateful recurrent quantizer would be able to quantize the vector a 2 -"bending knees" (given in the previous examples) into a different codeword depending on whether it was following the vector a 1 -"standing above a chair" or the vector a 3 -"sitting on a chair". In this way, it would be possible to extract a representation that can discriminate the standing up action from the sitting down action.

The proposed method is inspired by the BoF model, but employs a stateful trainable recurrent quantizer, instead of a plain static quantization approach to overcome the limitation of existing BoF formulations. In this way, the proposed method harness the power of a novel powerful recurrent quantization formulation in order to capture the temporal information of input data, which is crucial in classification tasks, such as activity recognition, while still maintaining all the advantages of the BoF model. In this work, the proposed Recurrent BoF (abbreviated as "ReBoF") layer is used between the last feature extraction layer and the fully connected layer of a network. Therefore, instead of using other naive pooling layers, that can lead to significant loss of temporal information, the extracted feature vectors are quantized to a number of codewords in a recurrent manner, enabling us to encode the fine-grained temporal information contained in the original feature vectors. The resulting network can be trained in an end-to-end fashion using plain gradient descent and back-propagation, since the proposed ReBoF formulation is fully differentiable. This allows for building powerful deep learning models for various visual information analysis tasks, as thoroughly demonstrated in this paper, while at the same time keeping the overall space and time complexity low compared to competitive approaches. In this way, the proposed method holds the credentials for providing fast and efficient human-centric perception methods for various embedded and robotics applications [START_REF] Lane | Squeezing deep learning into mobile and embedded devices[END_REF]. To the best of our knowledge, in this paper we propose the first stateful recurrent Bag-of-Features model that is capable of effectively modeling the temporal dynamics of video sequences. It is also worth noting that existing BoF formulations, e.g., [START_REF] Passalis | Learning bag-of-features pooling for deep convolutional neural networks[END_REF], merely provide models that fully ignore the temporal information.

The rest of the paper is structured as follows. In Section 2 the proposed Recurrent BoF (ReBoF) method is analytically derived. Then, the experimental evaluation of the ReBoF method is provided in Section 3. A thoroughly discussion on how ReBoF could be used for practical applications, along with its limitations, are provided in Section 4, while conclusions are drawn and future work is discussed in Section 5.

Proposed Method

In this Section, we first briefly introduce the regular (non-recurrent) Bag-of-Features model. Then, we derived the proposed Recurrent Bag-of-Features formulation, draw connections with the regular Bag-of-Features model and discuss how it can be used for video classification tasks.

Bag-of-Features

Let X = {x i } N i=1 be a set of N videos to be represented using the standard BoF model. From each video N i feature vectors are extracted:

x ij ∈ R D (j = 1, ..., N i ),
where D is the dimensionality of each feature vector. BoF provides a way to efficiently aggregate these features into a fixed-length histogram. To this end, each feature vector is first quantized into a predefined number of codewords, by employing a codebook V ∈ R N K ×D , where N K is the number of codewords. This codebook is usually learned by clustering all feature vectors into N K clusters [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF]. Clustering algorithms, such as k-means, can be used to this end, with each centroid, v k ∈ R D (k = 1, ..., N K ), corresponding to a codeword. Then, the quantized feature vectors of each object are accumulated to form the final histogram. Even though several feature quantization approaches have been proposed [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF], this work focuses on using a soft quantization approach that allows for learning the codebook using regular back-propagation, along with the rest of the parameters of the model [START_REF] Passalis | Neural bag-of-features learning[END_REF]. This can significantly improve the discriminative power of the model, since the codebook is adapted for the task at hand.

More specifically, each feature vector x ij , extracted from the i-th object, is quantized by measuring its similarity with each of the N k codewords as:

[d ij ] k = exp -v k -x ij 2 σ ∈ [0, 1], (1)
where σ controls the fuzziness of the quantization process. Then, for each feature vector we obtain a fuzzy membership vector, after normalizing the observed similarities as:

u ij = d ij d ij 1 ∈ R N K . (2)
Finally, the final histogram is extracted by accumulating all the normalized membership vectors as:

s i = 1 N i Ni j=1 u ij ∈ R N K . (3)
The histogram s i has unit l 1 norm, regardless the number of the extracted feature vectors, and provides an efficient representation of the feature vectors extracted from the corresponding video. This histogram is then fed to a multilayer perceptron (MLP) to classify the video. Note that the extracted histogram can be also used in other tasks, such as regression or retrieval.

Proposed Recurrent BoF

The histogram extracted using the regular BoF formulation, as described previously, discards any spatial or temporal information encoded by the feature vectors. To overcome this limitation, in this work we propose using a recurrent stateful quantizer, which allows for capturing and effectively encoding the temporal information expressed by the order in which the feature vectors arrive to the model. Note that in the case of video, we assume that the feature vector x ij corresponds to the j-th timestep of the i-th video sequence. Before deriving the proposed recurrent quantization approach, it is worth examining, from a probabilistic perspective, the quantization process involved in the BoF model. Using Kernel Density Estimation [START_REF] Bhattacharya | A probabilistic representation for efficient large scale visual recognition tasks[END_REF], we can estimate the probability of observing the feature vector x ij , given an input object x i , as:

p(x ij |x i ) = N K k=1 [s i ] k K(x ij , v k ) ∈ [0, 1], (4)
where the histogram s i ∈ R N K separately adjust the density estimation, while K(•) is a kernel function. Then, a maximum likelihood estimator can be used to actually calculate the histogram:

s i = arg max s Ni j=1 log N K k=1 [s] k K(x ij , v k ) . (5)
The involved parameters (histogram) can be estimated as [START_REF] Bhattacharya | A probabilistic representation for efficient large scale visual recognition tasks[END_REF], where

s i = 1 N i Ni j=1 u ij , as shown in
[u ij ] k = K(x ij , v k ) N K l=1 K(x (t) ij , v l ) ∈ [0, 1]. ( 6
)
Using this formulation, we can re-derive the regular BoF with soft-assignments. Also, note that we can also replace the Gaussian kernel used in [START_REF] Bhattacharya | A probabilistic representation for efficient large scale visual recognition tasks[END_REF], which typically requires tuning the width σ, with an easier to use hyperbolic kernel, which does not require tuning any hyper-parameter. The hyperbolic kernel also proved to be stabler and easier to use when the proposed method was combined with deep neural networks. Therefore, we can now measure the similarity between each feature vector and the codewords in order to quantize the feature vectors as:

[d ij ] k = σ(x T ij v k ) ∈ R N K , (7)
where

σ(x) = 1 1 + exp(-x) , (8)
is the logistic sigmoid function. Note that this formulation still ignores the temporal information, since it provides no way to encode the current state of the quantizer. Therefore, we extend [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF] in order to take into account the temporal information, as expressed by the histogram extracted until the current step, as:

[d ij ] k = σ(Vx ij + V h (r t s i,j-1 )) ∈ R N K , (9)
where V h ∈ R N K ×N k is a weight matrix that is used to transfer the gated histogram vector into the quantization space, s i,j-1 is the histogram extracted from previous quantizations (state) and r j ∈ R N K is the output of a reset gate, introduced to ensure the long-term stability of the model. The additional parameters introduced in this formulation are learned during the training process using back-propagation. The proposed method also employs a reset gate, inspired by the GRU model [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF], to further increase the stability of the learning process. Therefore, the reset gate is defined as:

r ij = σ(W r x ij + U r s i,j-1 ) ∈ R N K , (10)
where W r ∈ R N K ×D and U r ∈ R N K ×N K are the weight matrices used to implement the reset gate.

Then, the l 1 normalized membership vector is computed similarly to the regular BoF model as:

u ij = d ij d ij 1 . (11)
Note that the initial state s i,0 is initialized to:

s i,0 = 1 N K 1, (12)
where N K is the number of codewords and 1 ∈ R N K is a vector of all ones. This ensures that quantizer's output will be always a properly normalized membership vector. Therefore, the histogram is recurrently updated as:

s i,j = (1 -z ij ) s i,j-1 + z j u ij ∈ R N K . ( 13
)
The update gate z i,j , which controls how much the current histogram will be updated, is defined as:

z ij = σ(W z x ij + U z s i,j-1 ) ∈ R N K , (14)
where W z ∈ R N K ×D and U z ∈ R N K ×N K are parameters of the update gate. Finally, to compile the final histogram we average all the intermediate histograms as:

s i = 1 N i Ni j=1 s i,j ∈ R N K . (15)
Back-propagation can be directly used to learn all the parameters of the proposed ReBoF model. Note that ReBoF allows for capturing their temporal information, since it is capable of recursively processing the input feature vectors. First, the proposed recurrent stateful quantizer is employed to quantize the input feature vectors, as described by ( 9) and [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF]. Then, these vectors are employed to update the state of the quantizer, as descibed by [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], and allowing for compiling the resulting histogram. It is worth noting that ReBoF, similarly to all BoF-based models, is capable of processing varying-length input sequences.

ReBoF provides a significant advantage over existing BoF formulations, since it allows to capture the temporal information of sequential input data. This alows the ReBoF model to effective tackle challenging video analysis problems, e.g., video retrieval, activity recognition, etc. Therefore, to apply ReBoF we: a) use a convolutional neural network to extract a feature vector x ij from each frame of a video, and b) feed the extracted feature vectors to the ReBoF model in order to extract a compact representation for each video. This allows ReBoF to process videos of arbitrary duration, while creating fixed-length compact representations of them. Note that the whole architecture can be trained in an end-to-end fashion, since the proposed ReBoF formulation is fully differentiable.

Experimental Evaluation

The proposed method was evaluated using three video activity recognition datasets, the UTKinect-Action3D [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] dataset, the UCF101 dataset [START_REF] Soomro | UCF101: A dataset of 101 human actions classes from videos in the wild[END_REF] and a more challenging dataset, the Complex UCF101, which was designed to evaluate the ability of the methods to capture the temporal dimension of video sequences, as it will be described below. The UTKinect-Action3D [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] consists of 10 types of human activities in indoor settings. Samples for each action are collected from 10 different people that perform every activity two times. The provided RGB frames were used for all of the conducted experiments. Since there is no official training/testing split provided, a 50%-50% subject-based split strategy was employed, i.e., the videos of the first five subjects were included in the training set and the rest of them were used to form the testing set. Hence, a quite challenging setup was created, as the activities belonging in the testing set were performed from unseen subjects. The UCF101 dataset [START_REF] Soomro | UCF101: A dataset of 101 human actions classes from videos in the wild[END_REF] is widely used for benchmarking action recognition models. The dataset contains 13, 320 action instances belonging in 101 classes, that can be grouped in five generic categories. For all the experiments conducted in this paper, the first evaluation split of the dataset was used.

We also created a challenging and more complex dataset based on the UCF101 dataset to better demonstrate the ability of the proposed method to capture the temporal dynamics of video data. To this end, we compiled a dataset by mixing instances from different activities of the UCF101 dataset together. More specifically, 10 activities of UCF101 (split 1) were selected to be joined together. Every action of this subset was joined with each one of the remaining, leading to 90 complex actions. One can further understand the significance of encoding the temporal information of these instances by considering that a sample of action "A" combined with one of action "B" (let name this complex activity class "AB") must be separated from samples of complex activities from class "BA". Note that "AB" and "BA" videos contain the same set of frames (for a specific instance of "A" and "B"), but in a different order. Hence, 114 samples were selected for each class, as this was the minimum number of instances contained in the selected initial classes. The selected 10 action classes were the following: Ap-plyEyeMakeup, ApplyLipstick, Billiards, BoxingPunchingBag, BoxingSpeedBag, Haircut, Hammering, TableTennisShot, TennisSwing, and Typing. Then, the ith sample of initial class "A" is combined with the i-th sample of initial class "B" and so on, leading to 7, 380 training and 2, 880 testing data equally balanced among the 90 classes. The compiled dataset is called "Complex UCF" in the rest of this paper. For the UTKinect-Action3 and UCF101 datasets, every video instance was uniformly sampled in time in order to extract a predefined number of frames, denoted by N f . The number of extracted frames was set to N f = 30 for the UTKinect-Action3D dataset and to N f = 40 for the UCF101 dataset. Shorter videos were looped over as many times as necessary to ensure that each video contains at least N f frames, following the methodology described in the relevant literature [START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF]. Then, an Inception V3 model [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], pretrained in ImageNet, was used to extract a feature representation from each frame from the last average pooling layer of the network. Therefore, from each video, a sequence of N f feature vectors was extracted. This sequence was then fed to the proposed ReBoF layer, followed by a hidden fully connected layer with 512 neurons and dropout with rate 0.5, as well as by the final classification layer. The ReLU activation function was used for the hidden layer, while the cross-entropy loss was used for training the model. The network was trained using the Adam optimizer and a learning rate of 10 -5 , apart from the pretrained feature extractors which were kept frozen. Furthermore, note that we also experimentally found out that scaling the extracted histogram by N K significantly improved the convergence of the proposed method, especially when training from scratch. This scaling ensures that the gradients from the fully connected layers will not diminish as they are back-propagated to the previous layers. The network was trained for 800 epochs for the UTKinect-Action3D dataset and for 500 epochs for the UCF101 (the training/evaluation procedure was also repeated 3 times and the mean and standard deviation is reported). For the Complex UCF101 a slightly different setup was used. First, a 16-frame sliding window, with overlap of 4 frames, was applied on every activity instance of UCF101 dataset, while a pretrained 3D ResneXt-101 [START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF] was used to extract a feature vector from each window. The features were extracted from the last average pooling layer of the network. Therefore, a 32-length sequence (N f = 32) of 2048-dimensional feature vectors were extracted for each video action. The training process stopped when 99.9% accuracy was achieved in training set (for the average pooling baseline, the network was trained for 50 epochs). The evaluation of ReBoF and GRU methods was repeated 3 times and the mean accuracy and standard deviation on the test set are reported.

The performance of the proposed method was also compared to two other established pooling methods. The same feature extractors and classification block was used to ensure a fair comparison between the methods. First, global average pooling (denoted by "Average Pooling" in the rest of the paper), over the temporal dimensions of the input sequence, was used instead of the proposed ReBoF method. Furthermore, a more powerful recurrent aggregation model, a GRU [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF], was also employed to aggregate the extracted features.

First, the proposed method was evaluated on the UTKinect-Action3D, while the results are reported in Table 1. The proposed method greatly outperforms the other two evaluated methods, leading to the highest accuracy (54.64%) using 512 codewords. For the GRU also 512 units were employed, since at this point GRU achieved its best accuracy, which is however significantly lower (47.71%) compared to the proposed ReBoF method. Both ReBoF and GRU outperform the plain Average Pooling, since they are capable of effectively modeling the temporal dynamics of the input video sequences allowing for better discriminating between similar activities, such as stand up and sit down.

Moreover, in Fig. 1 the effect of using a wider range of codewords and number of GRU units in the classification accuracy on the UTKinect-Action3D dataset is evaluated using the two methods that achieve the highest performance (GRU and ReBoF). In all cases, the proposed method leads to higher accuracy compared to the GRU method. Furthermore, the proposed method allows for reducing the size of the extracted representation, since it outperforms the best GRU model (512 units) using just 128 codewords. This allows for reducing the size of the extracted representation and, as a result, the number of parameters in the subsequent fully connected layer. Both methods achieve their best performance for 512-dimensional representations. After this point, the accuracy for both models drops, mainly due to overfitting phenomena.

Again, similar conclusions can be drawn from the evaluation results on the UCF101 dataset, which are reported in Table 2. Note that even though UCF101 is a less challenging dataset, in terms of temporal dependence, the proposed method still outperforms the rest of the evaluated methods, achieving the highest accuracy (72.02%) for 1024 codewords. Again, all the methods were tuned to use the best number of codewords/representation length to ensure a fair comparison. Note that the proposed method outperforms the GRU while using representations with half the size of the ones used by the GRU. The effect of using different number of codewords and recurrent units is also evaluated in Fig. 2. Again, the proposed method outperforms the GRU method regardless the number of used codewords, while it achieves comparable accuracy using 2 to 4 times smaller representations.

Finally, the proposed method was also evaluated on the Complex UCF dataset. The results are provided in Table 3. As expected, Average Pooling fails to overpass the 50% test accuracy, since "AB" activities cannot be distinguished from those of "BA", due to employing global averaging, which completely discards the temporal information. On the other hand, ReBoF again achieves the highest accuracy (89.29% when 1024 codewords are used). It is worth noting that even Fig. 1: UTKinect-Action3D Evaluation: Effect of using different number of codewords/recurrent units for the ReBoF and GRU methods though spatio-temporal information is already encapsulated in the extracted feature vectors, since 3D kernels were used in feature extraction block, GRU again achieves lower accuracy compared to the ReBoF for both representation sizes.

Discussion

In this section, some further practical details regarding the employment of Re-BoF method are presented, providing more insight into the proposed method compared to our previous works, e.g., [START_REF] Krestenitis | Recurrent bagof-features for visual information analysis[END_REF]. A more thorough inspection of equations ( 9), ( 10), ( 13) and ( 14) implies that ReBoF shares some similarities with GRU units [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF], since both ReBoF and GRUs use a similar update-reset gate structure. Therefore, the RBoF method can be implemented by modifing an existing (and optimized) GRU implementation by a) replacing the output activation function (in order to ensure the quantization of the feature vectors), b) setting the initial state s i,0 (to ensure that the histogram vector will maintain a unit l 1 norm) and c) appropriately initializing the codebook. This allows us to simply modify existing and highly optimized GRU implementations to provide highly efficient ReBoF implementations. As far as the number of employed codeworks, a rule of thumb, based on the current work's analysis, could be to initialize ReBoF with 512 codewords, since it can provide a balanced trade-off among feature space dimensionality and model accuracy.

Furthermore, the advantages of employing ReBoF architecture in the corresponding framework for video analysis might be limited under certain circumstances. In case that temporal information is not crucial to distinguish different Fig. 2: UCF101 Evaluation: Effect of using different number of codewords/recurrent units for the ReBoF and GRU methods video instances, the employment of ReBoF is not guaranteed that will lead to significantly higher performance compared to a simpler architecture designed to analyze video instances by a single or a small set of frames. Yet, any method that discards temporal information is incapable to cope with tasks where crucial information is contained over temporal dimension of data. This is clearly demonstrated via the experimental results on UCF101 and Complex UCF101 datasets. In the first case, the employment of ReBoF slightly improves the model accuracy, since performing classification of the videos of UCF101 can considered a not so challenging task in terms of temporal dependence, given that in many cases we can achieve quite high recognition accuracy just using one frame from the videos. On the contrary, in the second case of Complex UCF101, enclosing temporal information is crucial to distinguish different video activities and thus, the employment of ReBoF leads to significant performance improvements compared to methods that discard temporal information. Finally, it should be noted that feeding to the ReBoF layer feature vectors that a priori enclose temporal information might lead to improvements to some extent however, this could also disproportionately increase overall the complexity of the models.

Conclusions

A novel stateful Recurrent Bag-of-Features (ReBoF) model was proposed in this paper. ReBoF employs a stateful trainable recurrent quantizer, instead of a plain static quantization approach, as the one used in existing BoF formulations. This allows ReBoF to capture the temporal information of the input data, which is crucial in classification tasks, such as activity recognition, while still maintaining all the advantages of the BoF model. ReBoF can be directly used in DL models and the resulting architecture can be trained in an end-to-end fashion using backpropagation, allowing for building powerful deep learning models for various visual information analysis tasks.

ReBoF opens several interesting future research directions. First, ReBoF can be also used for encoding the spatial information, instead of merely the temporal one. For example, the spatial information encoded in the feature vectors extracted from static images can be encoded by manipulating the extracted feature map as a sequence of feature vectors. This allows for overcoming one longstanding limitation of regular BoF formulation that led to the need of using complicated spatial segmentation schemes [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF]. Furthermore, activity recognition can be further enhanced by combining a ReBoF layer over the spatial dimension, followed by a ReBoF layer over the temporal dimension. In this way, the spatiotemporal information can be more properly encoded by creating a spatiotemporal histograms. Finally, using ReBoF for other tasks, such as video retrieval and hashing [START_REF] Tang | Discriminative deep quantization hashing for face image retrieval[END_REF], is expected to further boost the performance of existing methods by extracting compact, yet more discriminative representations.

Table 1 :

 1 UTKinect-Action3D Evaluation

	Method	# Codewords /	Test
		GRU Units	Accuracy (%)
	Average Pooling	-	40.83
	GRU	512	47.71
	ReBoF	512	54.64

Table 2 :

 2 UCF101 Evaluation

	Method	# Codewords /	Test
		GRU Units	Accuracy (%)
	Avg. Pooling	-	70.32 ± 0.43
	GRU	2048	71.04 ± 0.20
	ReBoF	1024	72.02 ± 0.68

Table 3 :

 3 Complex UCF101 Evaluation

	Method	# Codewords /	Test
		GRU Units	Accuracy (%)
	Average Pooling	-	48.95
	GRU	512	88.86 ± 2.04
	ReBoF	512	89.25 ± 1.08
	GRU	1024	88.62 ± 1.02
	ReBoF	1024	89.29 ± 0.89


Nikolaos Passalis
email: passalis@csd.auth.gr

Alexandros Iosifidis
email: alexandros.iosifidis@eng.au.dk

Moncef Gabbouj
email: moncef.gabbouj@tuni.fi

Anastasios Tefas
email: tefas@csd.auth.gr.

Hypersphere-based Weight Imprinting for Few-shot Learning on Embedded Devices

Keywords: Weight Imprinting, Few-shot Learning, Edge TPU, Embedded Deep Learning

Weight Imprinting (WI) was recently introduced as a way to perform gradient descent-free few-shot learning. Due to this, WI was almost immediately adopted for performing fewshot learning on embedded neural network accelerators that do not support back-propagation, e.g., Edge TPUs. However, WI suffers from many limitations, e.g., it cannot handle novel categories with multimodal distributions and special care should be given to avoid overfitting the learned embeddings on the training classes, since this can have a devastating effect on classification accuracy (for the novel categories). In this paper, we propose a novel hypersphere-based WI approach that is capable of training neural networks in a regularized, imprintingaware way effectively overcoming the aforementioned limitations. The effectiveness of the proposed method is demonstrated using extensive experiments on three image datasets.

I. INTRODUCTION

Deep Learning (DL) has achieved remarkable results on a wide range of difficult problems [START_REF] Lecun | Deep learning[END_REF], from image and video analysis to natural language processing and visual questioning answering. However, DL models are especially computationally intensive, both during the training and inference. Even though the use of Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs) has led to significant performance improvements, both in terms of training/inference speed and energy consumption, they remain too bulky and energyintensive to be used in most embedded applications, where significant energy and weight constraints exist [START_REF] Dundar | Embedded streaming deep neural networks accelerator with applications[END_REF]. This led to the development of embedded neural network accelerators, specifically designed to accelerate only the inference process, e.g., Edge TPUs. Even though these devices led to tremendous improvements in terms of operations/Watt, most of them suffer from the same limitation: they cannot be used to further train the network using back-propagation. This limits their usefulness under open-world settings, where the models must be able to continuously adapt to emerging categories that were not seen during the training, which is especially challenging for several robotic perception scenarios [START_REF] Murata | Learning to perceive the world as probabilistic or deterministic via interaction with others: A neuro-robotics experiment[END_REF], as well as for a wide range of different multimedia applications [START_REF] Yan | Efficient parallel framework for hevc motion estimation on many-core processors[END_REF], [START_REF] Yan | A fast uyghur text detector for complex background images[END_REF], [START_REF] Yan | Cross-modality bridging and knowledge transferring for image understanding[END_REF], [START_REF] Yan | Stat: spatial-temporal attention mechanism for video captioning[END_REF]. Therefore, the models should be able to draw connections and generalize their knowledge to new novel classes using only a few labeled examples, usually acquired during their interaction with the world. This problem is known as lowshot learning or few-shot learning [START_REF] Gidaris | Dynamic few-shot visual learning without forgetting[END_REF], [START_REF] Qiao | Few-shot image recognition by predicting parameters from activations[END_REF], [START_REF] Cai | Memory matching networks for one-shot image recognition[END_REF], [START_REF] Hariharan | Low-shot visual recognition by shrinking and hallucinating features[END_REF]. Note that zero-shot learning [START_REF] Socher | Zero-shot learning through cross-modal transfer[END_REF], [START_REF] Yu | Transductive zero-shot learning with adaptive structural embedding[END_REF], [START_REF] Ji | Attribute-guided network for cross-modal zero-shot hashing[END_REF], is a related extreme case of the same problem, in which no labeled samples are available for each category.

It is worth noting that even though several methods have been recently proposed for few-shot learning, only a few of them are suitable for inference-only neural network accelerators. Among them, Weight Imprinting (WI) was recently proposed as a way for performing gradient descent-free fewshot learning [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. WI allows for directly expanding the set of categories which a neural network can recognize by directly imprinting a new weight vector in the last layer of the network, without requiring back-propagating through the network. This is done by simply calculating the average embedding vector w i of the new (i-th) category using a few training samples and then calculating its similarity with the embeddings φ(x), extracted through the penultimate layer of the network, where x is the input to the network. Therefore, the probability that the input x belongs to the class i can be readily calculated, without performing any additional learning, as:

p i (x) = exp cw T i φ(x) l exp cw T l φ(x) , (1)
where c is a trainable scaling factor (fixed during the inference process). Note that the embedding vectors are normalized to have unit l 2 norm, and, as a result, w T i φ(x) equals to the cosine similarity between the two vectors. Also, using the scaling factor c ensures that the cosine similarity will range between -c and c, allowing for effectively training the network without imposing a strict lower-bound on the cross-entropy loss [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. This process found an immediate application on edge accelerators, e.g., Edge TPUs, since it can be readily applied to any neural network simply by extending the last fully connected layer.

WI, despite its immediate adoption, suffers from many limitations. First, it assumes that the distribution of the new categories will be unimodal. This assumption is mostly true for the distribution of classes presented to the network during the training process. However, this is not always the case for novel categories for which the network has not been optimized [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. Furthermore, the process of imprinting can negatively affect the accuracy of the network for the existing categories, if there is a significant overlap between the class boundaries. Results reported on the MNIST dataset using the setup described in Section III. Classification accuracy (%) on all the classes (novel and training) is reported.

WI does not provide any efficient mechanism for detecting if adding a new category will negatively impact the existing ones. Finally, the impact of the scaling factor c was not thoroughly discussed in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. We experimentally found out that the initial value of c can significantly affect the behavior of the model in some cases. For smaller initial values of c the embeddings tend to gather closely around the class prototypes, while for larger initial values of c the embedding vectors are spread around each class prototype w i . This behavior is illustrated in Fig. 1 (please refer to Section III for more details on the setup used for this experiment). This does not only affect how the embeddings are distributed through the space, but it has also a significant impact on the classification accuracy, as shown in Table I. Using larger initial values of c (up to a point) leads to better classification performance. Therefore, we observe that the embeddings that maintained larger variance around the prototypes allowed for performing better weight imprinting later on. These observations hint to a direct connection between maintaining the variance of the embeddings around the prototypes and the generalization abilities of a representation/model on unknown classes. This is not a surprising result, since it is well known that overfitted representations almost always lead to worse generalization (after a certain point) [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF]. This naturally leads us to the following question: Is it possible to design a representation in which the variance around the prototypes will be deliberately controlled to achieve the perfect balance between overfitting and underfitting instead of relying on early-stopping, implicit regularization or other heuristics to maintain enough variance? Also note that maintaining the variance will allow more information about the in-class similarities/dissimilarities to be encoded in the resulting representation. Fig. 2. Hypersphere-based Weight Imprinting: The representation space is constructed in a way that natively supports imprinting, spreads the embeddings in hyperspheres with radius r (the in-class variance is better preserved) and allow for detecting when the imprinting process cannot be performed safely (e.g., potential overlap between the novel class and the j-th class).

Motivated by the aforementioned observations, in this paper we propose a novel weight imprinting method for fewshot learning that can overcome the limitations of WI by: a) learning a regularized representation that maintains the variance around the prototypes in a structured way, while natively supporting weight imprinting and few-shot learning, b) providing a way to directly handle novel categories with multimodal distributions, and c) allowing for detecting beforehand if imprinting a new category will significantly affect the performance of the model for the rest of the categories. An open-source implementation of the proposed method, along with code that can be used to reproduce the conducted experiments, will be available online at [START_REF]Pytorch-based impleemnetaiton of hypersphere-based weight imprinting[END_REF].

To the best of our knowledge, this is the first work that provides a structured way to perform imprinting-aware neural network training, while at the same time proposing a simple, yet efficient classification scheme for few-shot learning. Compared to the weight imprinting method proposed in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], the proposed method models each class using a hypersphere, instead of just using a softmax-based classification formulation. This allows for learning a more regularized feature space that leads to better generalization on unknown classes by maintaining the variance around each class prototype, as well as being able to handle multimodal classes by using multiple prototypes per class.

The rest of this paper is structured as follows. First, the proposed method is derived and discussed in Section II. Then, the proposed method is evaluated and compared to regular WI under different scenarios in Section III. Finally, conclusions are drawn in Section IV.

II. PROPOSED METHOD

The proposed method, called Hypersphere-based Weight Imprinting (HWI), learns a carefully designed feature space to more effectively support weight imprinting. To this end, it employs a centroid-based loss which uniformly distributes the embedding vectors within a radius r around each prototype (centroid). Furthermore, to ensure that the prototypes are discriminative enough it is required that the minimum distance between two prototypes is at least ρ > 2r. This process is illustrated in Fig. 2. After learning a representation that fulfills the aforementioned requirements we can directly classify a new sample, perform gradient descent-free fewshot learning, detect and handle multimodal novel classes and detect intrusion to the existing classes that can lower the performance of the model. Neural network training: First, we will describe the proposed imprinting-aware training process. Let φ(x) ∈ R m be the output of a neural network, where m is the dimensionality of the embeddings extracted from the network, when presented an input sample x. Also, let w i be the prototype vector for the i-th class used during the training process and X i be the set of samples that belong to the class i. Then, to ensure that the embeddings will be uniformly distributed around each class prototype w i ∈ R m we define the appropriate class-induced loss as:

L c = 1 N N C l=1 x∈X l (||φ(x) -w l || 2 -αr) 2 , ( 2
)
where N is the total number of training samples, N C is the total number of training classes, ||•|| 2 denotes the l 2 norm of a vector and α ∈ [0, 1] is a number drawn uniformly from the range [0...1]. During the optimization a different random value is drawn for α for each sample and iteration, leading to a uniform distribution of the embeddings within a radius r from each w i . Even though this process does not ensure that the full space around w i will be occupied, it ensures that the embeddings will be sampled uniformly at various radiuses around the corresponding center, significantly improving the generalization abilities of the representation, as we will also demonstrate later in Section III. Note that by setting r = 0, the loss L class degenerates to the regular center loss [START_REF] Wen | A discriminative feature learning approach for deep face recognition[END_REF]. Furthermore, to further model the uncertainty regarding the class prototypes, we can use Gaussian noise to corrupt the prototypes as wi = w i + N (0, σ). The effect of varying the radius r is demonstrated in the toy example of Figure 3. For radius equal to r = 0 the model compresses the embeddings around the prototypes (whether a collapse will happen or not depends on the ability of the neural network to overfit the data), while for larger values the embeddings are spread around the prototypes as desired, ensuring that the in-class variance is maintained. At the same time, each prototype i is required to be at a distance of at least ρ from each other prototype (to ensure that there is no overlapping between the hyperspheres that enclose the embeddings of each class). To this end, we also define the prototype loss as:

L p = 1 N C (N C -1) N C i=1 N C j=1,i =j max(0, ρ -||w i -w j || 2 ).
(3) Fig. 3. Visualizing the embeddings learned for different radiuses r using the MNIST dataset and the setup described in Section III. The data were visualized by projecting them into 2 dimensions using PCA.

Let W denote the parameters of the model φ(x), along with the prototype vectors w i . The proposed method employs gradient descent to optimize these parameters in order to minimize both the class-induced loss and the prototype loss: L = L c + γL p , where γ is a hyper-parameter that alters the weight of the prototype loss (set to 1 for all the experiments conducted in this paper). Therefore, at each training iteration, a batch of data is sampled and the parameters of the model are updated as:

∆W = - ∂L ∂W. ( 4
)
Classifying a sample: To classify an input sample we can directly choose the class that corresponds to the prototype with the smaller distance to the extracted embedding φ(x).

The network can be used in a similar fashion as a one trained using the softmax activation simply by using a final classification layer that calculates the membership value for each prototype/class probabilities as:

p i (x) = 1 1 + ||φ(x) -w i || 2 . ( 5
)
Performing few-shot learning: To perform few-shot learning we can simply augment the final classification layer with an additional prototype vector w n calculated as:

w n = 1 |X n | x∈Xn φ(x), (6)
where X n is the set that contains the training samples for a novel category. Note that similarly to regular WI, no gradient descent-based optimization is required for extending the classifier to support novel classes. However, as we further demonstrate in Section III, the regularized nature of the learned feature space leads to significantly better performance compared to regular WI.

Detecting and handling multimodal novel categories: There are several ways to detect if the distribution of a novel class is indeed multimodal, including, but not limited to, the bandwidth test [START_REF] Silverman | Using kernel density estimates to investigate multimodality[END_REF], and the runt test [START_REF] Hartigan | The runt test for multimodality[END_REF]. Any of these approaches can be combined with the proposed method to allow for detecting whether the distribution of a class is multimodal. Furthermore, the proposed hypersphere-based formulation also provides a straightforward way to discover multimodal classes: the embedding vectors extracted for a novel category are clustered and the distance between the cluster centroids is measured. If we detect centroids that are at distance greater than r from each other, then a hypersphere with radius of r cannot enclose the embeddings of the novel class. To address this, we can simply add one or more prototypes (according to the number of centers that are at distance greater than r) to model the distribution of the novel class. In this way, one class can be represented using more than one prototype. On the other hand, if the centers of the clusters are within a radius of r, then we assume that proposed classification scheme can directly handle the distribution of the novel class (even though there is no guarantee that the distribution is not multimodal). The proposed way of handling multimodal classes is straightforward to implement when the proposed hypersphere imprinting approach is used and allows for improving the accuracy of the proposed method, as further demonstrated in Section III.

Detecting intrusions: We can directly detect if a novel prototype w n will intrude an existing class simply by measuring the distance between the prototype and each other prototype as:

d i = ||w n -w i || 2 . If min i (d i ) < r
, then at least one hypersphere of an existing class will be intruded by the novel class. This can be regarded as a hint that imprinting will fail and cannot be used to support the specific novel class. In this case, the model should be probably re-trained off-line using another few-shot learning method.

III. EXPERIMENTAL EVALUATION

The proposed method was evaluated using three different image datasets, the MNIST dataset [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], the Caltech-UCSD Birds 200-2001 (CUB-200-2011) dataset [START_REF] Wah | The caltech-ucsd birds-200-2011 dataset[END_REF], and the Animals with Attributes (AwA2) dataset [START_REF] Xian | Zero-shot learninga comprehensive evaluation of the good, the bad and the ugly[END_REF]. For the MNIST dataset the first 5 classes were used to train the model, while the remaining 5 classes were used for fewshot learning and evaluation. For the CUB-200-2011 dataset, we followed the evaluation setup proposed in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. For the AwA2 dataset [START_REF] Xian | Zero-shot learninga comprehensive evaluation of the good, the bad and the ugly[END_REF] the first 40 classes were used for training the model, and the rest of the classes were employed for evaluating the performance of the proposed method. The performance evaluation was repeated 5 times using different training samples for the novel classes (except for the hyperparameter evaluation experiments) and the mean and standard deviation is reported. For the MNIST dataset the employed neural network was composed of a 3 × 3 convolutional layer with 32 filters, followed by a 2 × 2 max pooling layer, another 3×3 convolutional layer with 64 filters, an additional 2×2 max pooling layer and a fully connected layer with 256 neurons. An InceptionV1 architecture [START_REF] Szegedy | Going deeper with convolutions[END_REF], pretrained on the Imagenet dataset was used for the CUB-200-2011 dataset. The last fully connected layer of the InceptionV1 architecture was discarded and replaced with a fully connected layer with 256 hidden neurons, following the approach used in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. For the AwA2 dataset we used a pretrained ResNet101 to extract feature vectors (as described in [START_REF] Xian | Zero-shot learninga comprehensive evaluation of the good, the bad and the ugly[END_REF]) that were then fed to two fully connected layers with 2048 and 512 neurons respectively. The relu activation function was used for all the layers. The models were trained using the Adam optimizer with a learning rate of 10 -3 for 20 training epochs for the MNIST dataset and of 10 -4 for 20 training epochs for the AwA2 dataset. For the CUB-200-2011 dataset, the added fully connected layers were pre-trained for 10 epochs using a learning rate of 10 -3 , followed by 10 additional full training epochs using a learning rate of 10 -4 . First, we evaluated the effect of altering the minimum distance ρ between the different class prototypes w i , while keeping the radius fixed to r = 0. The results are presented in Table II. Increasing the minimum distance between the class prototypes seems to have a positive effect on the classification accuracy, both for the novel split (denoted by "Novel") and the combined split of novel and training classes (denoted by "All"). This was expected since the learned representation is not capable of perfectly collapsing the embeddings to the corresponding prototypes, even though the radius was set to r = 0. Therefore, keeping a quite large margin between the different prototypes helps to reduce the risk of wrongly classifying a sample.

Next, we also evaluated the effect of altering the radius r on the learned representation. In Section II it was conjectured that spreading the training embeddings in a hypersphere of radius r will have a positive regularization effect on the learned representation by allowing the model to capture and better model the in-class variations. Indeed, as demonstrated in Table III, the classification accuracy increases by more than 14% for the novel split, and by 1.8% for the combined split. This confirms our hypothesis that collapsing the embeddings to the class centers, without any form of regularization, can significantly reduce the classification accuracy, especially when dealing with classes that were not seen during the training process and using powerful models that can overfit the training data. Next, we evaluated the proposed method using a 1-shot, 2shot and 5-shot evaluation protocol on the MNIST dataset. The results are reported in Table IV. Two variants of the proposed method were evaluated: HWI-, where r = 0 and σ = 0 were used, and HWI, where r = 5 and σ = 0.05 were used. The proposed method was also compared to the plain Weight Imprinting (WI) approach [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], using an initial scaling value of c = 10 (following the results of Table I). Again, it was confirmed that using the proposed variance preserving approach improves the performance over simply using a center-based loss, allowing for outperforming the regular WI method. It is worth noting that the accuracy for all the evaluated methods is relatively low compared to the state-of-the-art, since neither WI or the proposed method perform any kind of optimization according to a discriminative objective.

The proposed approach for handling multimodal novel classes, abbreviated as "HWI-M", was also evaluated using an additional multimodal split of the MNIST dataset. This split was compiled by merging two succeeding classes into one, e.g., "0" and "1" were merged into a new class, "2" and "3" into another, and so on. Then, the three first classes (digits 0 to 5) were used for training and the remaining two of them (6 to 9) for evaluating the few-shot learning performance. The evaluation results are reported in Table V. The employed threshold was used to detect whether a class distribution is multimodal (by clustering the training data into two clusters and measuring the distance between the resulting centroids). If the distance of the resulting centers was greater than the specified threshold, then two prototypes were used per novel class. Again, note that the proposed variance-preserving variant of HWI greatly outperforms the HWI-variant. Using the multimodal variant HWI-M allows for further improving the accuracy of the proposed method, outperforming all the other evaluated approaches.

The proposed method was also evaluated using the CUB-200-2011 dataset, which allows for a direct comparison with the original weight imprinting approach, as presented in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. Results for Generator + Classifier approach ("Generator") and Matching Networks ("Match. Nets") are as reported in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. "WI + FT" refers to results obtained in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF] using the proposed combined weight imprinting and fine-tuning approach. The experimental results are reported in Table VI. The proposed method (ρ = 30, r = 15) was also compared to three other baseline few-shot learning approaches, the Generator + Classifier approach, as proposed in [START_REF] Hariharan | Low-shot visual recognition by shrinking and hallucinating features[END_REF], the Matching Networks [START_REF] Vinyals | Matching networks for one shot learning[END_REF], as well as the Weight Imprinting + Fine-tuning approach (which requires further optimization of the network for the novel classes compared to plain weight imprinting), as proposed in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF]. Note that the proposed method outperforms the rest of the evaluated approaches. Note that despite using the same network architecture as the one proposed in [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], i.e., an InceptionV1 model [START_REF] Szegedy | Going deeper with convolutions[END_REF], our implementation also leads to slightly better accuracy for the original WI approach, possibly due to the different initialization of the network. However, the proposed HWI still outperforms the plain WI, regardless the used setup. Similar results are also obtained when the multimodal split of the CUB-200-2011 dataset is used, as reported in Table VII. This split was compiled by merging each set of 10 successive classes of the original dataset, leading to 10 classes that are used for training the models and 10 classes for evaluating the imprinting performance. The proposed multimodal-aware imprinting approach again leads to higher accuracy over all the evaluated methods, confirming its ability to handle multimodal novel classes. Finally, we also evaluated the performance of the proposed method using the AwA2 dataset. The results are reported in Table VIII. As before, the proposed method leads to significant performance improvements over the plain WI method, while it still outperforms the HWI-methods. The smaller differences between HWI-and HWI can be possibly attributed the to the smaller learning capacity of the employed network (the risk of overfitting the representation is higher when more powerful networks are employed). Note that slightly different parameters were used for the HWI method in this experiment: ρ = 20, r = 10, and σ = 0.

To demonstrate that the proposed method is also robust to novel prototypes/distractors that are at a distance of at least r from the existing prototypes, we also performed one additional experiment. We randomly generated 10 additional prototypes, each one having a distance between δr and δr + 1 from the closest prototype, where δ is a hyper-parameter that controls the intrusion (for δ > 1 there will be no class intrusion). The results are in Table IX. The goal of this experiment was to demonstrate that the existing prototypes can be effectively "protected" from distractors, if they exist in a distance greater than r from the prototypes. Indeed, for δ = 1 the accuracy of the model has not been reduced, confirming the robustness of the proposed method to distractors (given that the appropriate distance from the existing prototypes is maintained to avoid intrusions). For smaller values of δ, even though the existing hyperspheres are intruded by the distractors, the effect on the classification accuracy is minimal.

IV. CONCLUSIONS

In this paper we proposed a novel hypersphere-based weight imprinting approach that maintains all the advantages of regular WI [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], i.e., it is able to readily extend a pretrained neural network to classify samples from novel categories simply by adding new weight vectors in the final classification layer without requiring to perform any form of back-propagation to this end. At the same time, the proposed method was capable to overcome significant limitations of WI by being able to learn regularized representations that provide better generalization for classes which were not seen during the training and provides a straightforward way to directly handle novel categories with multimodal distributions. The proposed method was extensively evaluated on three image datasets, outperforming the regular WI approach.

Fig. 1 .

 1 Fig. 1. Weight Imprinting: Visualizing the embeddings for two different initial values of c using the MNIST dataset and the setup described in Section III. The data were visualized by projecting them into 2 dimensions using PCA.

TABLE I EFFECT

 I OF PARAMETER c ON WEIGHT IMPRINTING

	c	1-shot	2-shot	5-shot

1 43.19 ± 3.0 43.26 ± 3.2 43.02 ± 0.6 10 57.51 ± 5.8 58.35 ± 6.4 64.49 ± 3.2 20 54.59 ± 4.0 57.10 ± 2.9 62.30 ± 2.9

TABLE II EVALUATING

 II THE EFFECT OF THE HYPER-PARAMETERS ρ ON THE 2-SHOT LEARNING ACCURACY OF THE PROPOSED IMPRINTING METHOD

	Min. Distance ρ	Novel Split	Combined Split
	1	56.28	77.79
	2	56.24	77.42
	5	58.53	78.66
	10	61.02	80.06
	20	53.82	76.51
		TABLE III
	EVALUATING THE EFFECT OF THE HYPER-PARAMETERS r ON THE 2-SHOT
	LEARNING ACCURACY OF THE PROPOSED IMPRINTING METHOD
	Radius r	Novel Split	Combined Split
	0	61.02	80.06
	4	64.51	80.59
	5	69.57	81.55
	6	66.86	78.26

TABLE IV MNIST

 IV : EVALUATING THE ACCURACY OF IMPRINTING METHODS ON THE COMBINED NOVEL AND TRAINING CATEGORIES.

	Method Split	1-shot	2-shot	5-shot
	WI	All	57.51 ± 5.8	58.35 ± 6.4	64.49 ± 3.2
	HWI-	All	73.27 ± 4.7	79.46 ± 1.5	82.88 ± 1.8
	HWI	All	73.66 ± 3.9	79.09 ± 6.0	84.23 ± 1.3

TABLE VIII AWA2

 VIII : EVALUATING THE ACCURACY OF IMPRINTING METHODS ON THE COMBINED NOVEL AND TRAINING CATEGORIES SPLIT. HWI 56.14 ± 2.70 70.16 ± 2.63 77.85 ± 1.84 55.60 ± 3.06 69.69 ± 3.33 77.02 ± 2.81 0.5 55.73 ± 3.02 70.02 ± 2.85 77.73 ± 1.95 0.8 56.02 ± 2.78 70.06 ± 2.79 77.75 ± 1.93 1 56.10 ± 2.73 70.16 ± 2.64 77.83 ± 1.86 No distractors 56.10 ± 2.73 70.16 ± 2.64 77.83 ± 1.86

	Method	1-shot	2-shot	5-shot
	WI	51.03 ± 3.71	61.33 ± 2.73	75.23 ± 1.85
	HWI-	54.55 ± 3.31	68.44 ± 3.17	76.95 ± 2.12
		TABLE IX
	CLASS INTRUSION ROBUSTNESS EVALUATION ON THE AWA2 DATASET
	δ	1-shot	2-shot	5-shot
	0.1

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR). This publication reflects the authors' views only. The European Commission is not responsible for any use that may be made of the information it contains.