Marios Krestenitis

Nikolaos Passalis
email: passalis@csd.auth.gr

Alexandros Iosifidis
email: alexandros.iosifidis@eng.au.dk

Moncef Gabbouj
email: moncef.gabbouj@tuni.fi

Anastasios Tefas
email: tefas@csd.auth.gr

Recurrent Bag-of-Features for Visual Information Analysis

Keywords: Bag-of-Features, Recurrent Neural Networks, Pooling Operators, Activity Recognition

Deep Learning (DL) has provided powerful tools for visual information analysis. For example, Convolutional Neural Networks (CNNs) are excelling in complex and challenging image analysis tasks by extracting meaningful feature vectors with high discriminative power. However, these powerful feature vectors are crushed through the pooling layers of the network, that usually implement the pooling operation in a less sophisticated manner. This can lead to significant information loss, especially in cases where the informative content of the data is sequentially distributed over the spatial or temporal dimension, e.g., videos, which often require extracting fine-grained temporal information. A novel stateful recurrent pooling approach, that can overcome the aforementioned limitations, is proposed in this paper. The proposed method is inspired by the well-known Bag-of-Features (BoF) model, but employs a stateful trainable recurrent quantizer, instead of plain static quantization, allowing for efficiently processing sequential data and encoding both their temporal, as well as their spatial aspects. The effectiveness of the proposed Recurrent BoF model to enclose spatio-temporal information compared to other competitive methods is demonstrated using six different datasets and two different tasks.

Introduction

A vast variety of visual information analysis techniques has been proposed in the field of computer and robotics vision, as one of the most active and continuously expanding research fields. The pipeline of a typical visual information analysis method consists of two fundamental information processing steps: a) feature extraction, which typically extracts lower level information from small spatial or temporal segments of the data, and b) feature aggregation, which fuses the information extracted during the previous step into a compact representation that can be used for various tasks, e.g., classification, retrieval, etc. For example, handcrafted feature vectors can be used to describe local image regions [START_REF] Nanni | Handcrafted vs. non-handcrafted features for computer vision classification[END_REF], while the set of the extracted features can be then aggregated into a more compact representation using feature aggregation methods, such as the Bag-of-Feature model [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF] and Vectors of Locally Aggregated Descriptors (VLAD) [START_REF] Jegou | Aggregating local image descriptors into compact codes[END_REF].

With the advent of deep learning (DL) these two steps were, to some extent, unified and replaced with deep trainable feature extraction layers, e.g., convolutional layers, that are combined with naive pooling operators, e.g., max or average pooling, to lower the complexity of the model and provide translation invariance. Indeed, the outstanding performance of Convolutional Neural Networks (CNNs) in complex and challenging image analysis tasks, has confirmed their ability to extract meaningful feature vectors with high discriminative power for a wide variety of different computer vision tasks [START_REF] Nanni | Handcrafted vs. non-handcrafted features for computer vision classification[END_REF]. However, these powerful feature vectors are crushed through the pooling layers of the network, that usually implement the pooling operation in a less sophisticated manner, often leading to significant information loss, as further discussed in [START_REF] Passalis | Learning bag-of-features pooling for deep convolutional neural networks[END_REF][START_REF] Ma | Ts-lstm and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition[END_REF]. This is even more evident in cases where the informative content of the data is sequentially distributed over the spatial or temporal dimension, e.g., videos, which often require extracting fine-grained temporal information, which is discarded by these pooling approaches.

The aforementioned limitations can be better understood through the following example. Consider the task of activity recognition in videos, where the action of standing up must be distinguished from the action of sitting down. By employing a deep CNN, robust feature vectors can be extracted from every video frame or a sequence of them. However, the pooling layers, as the weak point of the network, dull the expressiveness of the extracted feature vectors and produce less discriminative representations by pooling over the time dimension. For example, assume that a sequence of feature vectors are extracted from a given video instance of action sitting down. Let that sequence, notated as S 1 = [a 1 , a 2 , a 3 ], be composed of three feature vectors a 1 = [0, 0, 1] T , a 2 = [0, 1, 0] T , and a 3 = [1, 0, 0] T , which are the feature vectors that correspond to the sub-actions standing above a chair, bending knees and sitting on a chair, respectively. Similarly, consider the same feature vector sequence, but in reverse order, i.e., S 2 = [a 3 , a 2 , a 1 ], that represents a video instance of the activity standing up. Also, let s i denote the aggregated representation extracted for the i-th video.

Note that when average or max pooling is applied over both sequences, then the same representation is extracted for both videos i.e., s

1 = s 2 = [max i [a i ] 1 , max i [a i ] 2 , max i [a i ] 3 ] T = [1, 1, 1] T
for max pooling (where the notation [x] i is used to refer to the i-th element of vector x) or

s 1 = s 2 = 1 3 3 i=1 a i = [ 1 3 , 1 3 , 1 3 ]
T for average pooling. Therefore, even though the employed CNN was capable of perfectly recognizing the fundamental sub-actions from still frames, the resulting deep model cannot discriminate between the two actions due to the employed pooling layer. Based on this observation, in this work we characterize the average and max pooling layers as weak pooling layers, since they are not capable of capturing the fine-grained spatial or temporal interactions between the feature vectors extracted from a given sequence. Note that in other cases, instead of employing a pooling layer, the extracted feature map can be flattened to a vector and fed to the fully connected layer. However, this approach makes it impossible for the network to handle inputs of arbitrary size, while it significantly reduces the invariance of the network to temporal or spatial translations (the features must always arrive at the exact same moment or position).

Before delving into the details of the proposed pooling method that is capable of overcoming the aforementioned limitations, it is worth revisiting a well-known feature aggregation model, the Bagof-Features (BoF) model [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF], also known as Bag-of-Visual-Words (BoVW). BoF has been extensively used in a wide range of machine learning and computer vision problems [START_REF] Silva | Graph-based bag-of-words for classification[END_REF][START_REF] Passalis | Learning bag-of-embedded-words representations for textual information retrieval[END_REF][START_REF] Ghalyan | Estimation of ergodicity limits of bag-of-words modeling for guaranteed stochastic convergence[END_REF]. Initially inspired by the Bag-of-Words (BoW) model, which was designed for text analysis, BoF creates a constantlength representation of a multimedia object, e.g., video, audio, time-series, etc., by compiling a histogram over its quantized feature vectors. Assuming a set of objects is given, the usual framework of BoF is as follows:

1. At first, a feature extraction procedure takes place, where every input object is translated to a set of feature vectors. This set defines the features space, where every object is projected into.

2.

A set of representative feature vectors is learned, which is known as dictionary or codebook.

Every feature vector of the codebook is called codeword, while the learning process is called dictionary learning.

3. Every feature vector of an input object is quantized based on the learned codebook. This step is referred as feature encoding or feature quantization.

4.

A histogram is created by counting the number of feature vectors that were quantized to each codeword.

Following this pipeline, every object can be represented as a fixed-length histogram over the learned codewords. The codewords/dictionary can be either learned using generative/reconstruction approaches [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF][START_REF] Bhattacharya | A probabilistic representation for efficient large scale visual recognition tasks[END_REF], or by employing discriminative dictionary learning methods [START_REF] Iosifidis | Discriminant bag of words based representation for human action recognition[END_REF][START_REF] Iosifidis | Multidimensional sequence classification based on fuzzy distances and discriminant analysis[END_REF], which usually better fit classification tasks. Note that BoF can be also combined with deep neural networks to provide more powerful trainable pooling layers that can better withstand distribution shifts, while handling inputs of various sizes [START_REF] Passalis | Neural bag-of-features learning[END_REF].

Despite the remarkable results in various tasks and the ability to handle variable size inputs, the main drawback of BoF-based methods is the loss of spatial and temporal information, i.e. their inability to capture the geometry of input data [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF]. These drawbacks severely limit the ability of BoF to process temporal or sequential data, such as video data, since it is not capable of capturing the temporal succession of events. Therefore, BoF-based methods are still considered as weak pooling layers, since the quantization of feature vectors to codewords does not take into account the features' sequential order and thus the temporal information is entirely discarded. To overcome this limitation, the quantization process should take into account the order in which the features arrive, allowing for forming temporal codewords that also capture the interrelation between the feature vectors. As a result, a BoF method employing a stateful recurrent quantizer would be able to quantize the vector a 2 (bending knees) into a different codeword depending on whether it was following the vector a 1 (standing above a chair) or the vector a 3 (sitting on a chair), allowing for extracting a representation that can discriminate between the standing up action from the sitting down action.

In this paper, a novel stateful recurrent pooling approach that can overcome the aforementioned limitations is proposed. The proposed method is inspired by the BoF model, but employs a stateful trainable recurrent quantizer, instead of a plain static quantization approach. In this way, the proposed method harness the power of a novel powerful recurrent quantization formulation in order to capture the temporal information of input data, which is crucial in classification tasks, such as activity recognition, while still maintaining all the advantages of the BoF model. The proposed Recurrent BoF (abbreviated as "ReBoF") layer is used between the last feature extraction layer and the fully connected layer of a network. Therefore, instead of using weak pooling layers, the extracted feature vectors are quantized to a number of codewords in a recurrent manner, enabling to encode the fine-grained temporal information contained in the original feature vectors. The resulting network can be trained in an end-to-end fashion using plain gradient descent and back-propagation, since the proposed ReBoF formulation is fully differentiable. This allows for building powerful deep learning models for various visual information analysis tasks, as thoroughly demonstrated in this paper. Furthermore, the proposed layer enables the network to operate with arbitrary sized inputs, while it can also reduce the size of the fully connected layer by extracting a compact, yet powerful constant length representation. Finally, it is worth noting that ReBoF can be also used for encoding the spatial information, instead of merely the temporal one. For example, the spatial information encoded in the feature vectors extracted from static images can be encoded by manipulating the extracted feature map as a sequence of feature vectors. This allows for overcoming one long-standing limitation of regular BoF formulation that led to the need of using complicated spatial segmentation schemes [START_REF] Lazebnik | Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories[END_REF].

The rest of the paper is structured as follows. In Section 2 the related work is presented and compared to the proposed approach. Next, the ReBoF model is elaborately described in Section 3.

The experimental evaluation of ReBoF over six different datasets, along with comparisons to other related techniques, are presented in Section 4. Finally, conclusions are drawn in Section 5.

Related Work

Over the years, Bag-of-Features has been a widely used method for image analysis tasks. Its ability to provide better scale and position invariance of local keypoint features attracted significant research attention [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF]. Moreover, its capability to handle variable sized images made BoF a common choice in image classification and retrieval tasks. The remarkable results in these research areas among with its simplicity motivated researchers to employ BoF in more complex tasks, such as video activity recognition and video semantic retrieval [START_REF] Jiang | Towards optimal bag-of-features for object categorization and semantic video retrieval[END_REF][START_REF] Iosifidis | View-invariant action recognition based on artificial neural networks[END_REF]. Video-based human activity recognition consists one of the most challenging problems in computer vision community, that gains increasing interest over the years. Early approaches were focused on crafting discriminative features to describe each activity, such as space-time trajectories of human joints [START_REF] Yacoob | Parameterized modeling and recognition of activities[END_REF]. This is not an easy task, since the extracted low-or mid-level features must be robust to noise and intra-class variations, such as viewpoint and illumination changes, different motion speeds, etc. Moreover, there are human actions that are associated with specific objects or tools, e.g., cooking, hammering, etc., distinct human poses, e.g., sitting, playing guitar, or executed in a distinguishing environment, e.g., surfing, playing tennis. These cases highlight the need for high-level descriptors that are also capable of capturing the high-level semantic information of the video. The proposed ReBoF model goes beyond these methods, since it not only provides an end-to-end trainable model that allows for extracting representations tailored exactly to the task at hand, but it is also capable of modeling the temporal information of the input feature streams.

Previous efforts tried to tackle some of these issues by exploiting the ability of BoF to process objects of various lengths and providing an orderless representation of the input local keypoint features, extracted from RBG streams, optical flow and joint annotations. Feature descriptors, such as HoG/HoF [START_REF] Laptev | Learning realistic human actions from movies[END_REF], and MBH [START_REF] Jiang | Towards optimal bag-of-features for object categorization and semantic video retrieval[END_REF], were used to perform feature extraction. Furthermore, Dollar et al. proposed a feature extractor ensemble [START_REF] Dollár | Behavior recognition via sparse spatio-temporal features[END_REF], which among with other local features, such as Dense Trajectories (DTs), improved Dense Trajectories (iDTs), Space Time Interest Points (STIPs), and skeleton joint and body shape features [START_REF] Jalal | Robust human activity recognition from depth video using spatiotemporal multi-fused features[END_REF], were combined with the BoF model to better capture the spatio-temporal information of a video. However, these methods employed BoF just as a post-processing step for aggregating the pre-computed features, while the selection of codewords is usually performed in a fully unsupervised way, leading to less discriminative representations and restricting the accuracy of these approaches.

On the other hand, the abrupt expansion of DL in a variety of computer vision problems has also been imported in activity recognition tasks [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF][START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]. One of the initial approaches was to use a neural network, which was pretrained on an image classification task, to extract features from each frame and then merge them to classify the whole video [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF]. Donahue et al. proposed to feed the features extracted from each frame to a recurrent layer composed of Long Short-Term Memory (LSTM) units [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF], allowing for encoding the temporal dimension of the sequence. Later, more powerful methods employed convolutional networks with 3D kernels (C3D) [START_REF] Ji | 3d convolutional neural networks for human action recognition[END_REF][START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF] or fused the RGB frame-based features with features from a stack of optical flow frames, through a two-stream network [START_REF] Feichtenhofer | Convolutional two-stream network fusion for video action recognition[END_REF], allowing for effectively capturing the motion features of the video. Li et al. [START_REF] Li | Spatio-temporal deformable 3d convnets with attention for action recognition[END_REF] proposed a deformable C3D network enhanced with an attention submodule in order to capture the temporal and spatial irregularities of video streams. Current state-of-the-art methods use Inflated 3D Convolutional (I3D) networks [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF], that are capable of effectively combining both the RGB and optical flow streams. In spite of the impressive results, all these methods use powerful feature extractors combined with weak pooling layers.

The limitations of these approaches led to the development of more advanced pooling layers.

Earlier works extended traditional feature aggregation methods into differentiable models that can be combined with DL models and trained in an end-to-end fashion, e.g., Bag-of-Features was extended to Neural BoF pooling [START_REF] Passalis | Neural bag-of-features learning[END_REF], while Vector of Locally Aggregated Descriptors (VLAD) was extended into NetVLAD pooling [START_REF] Arandjelovic | Netvlad: Cnn architecture for weakly supervised place recognition[END_REF]. However, these approaches still suffer from the same limitations, since they fail to capture the temporal information contained in the input feature stream. It is worth noting that even more advanced pooling approaches, such as attention-based pooling [START_REF] Girdhar | Attentional pooling for action recognition[END_REF], which dynamically adjusts the feature aggregation on each temporal segment, and learnable pooling with context gating [START_REF] Miech | Learnable pooling with context gating for video classification[END_REF], cannot extract compact representations that can capture the temporal information. More recently proposed methods attempt to overcome this limitation by further encoding the spatial information either by employing pyramid-based pooling schemes [START_REF] Wang | Temporal pyramid pooling-based convolutional neural network for action recognition[END_REF],

or by further extending the VLAD model to capture spatio-temporal information [START_REF] Xu | Sequential video vlad: training the aggregation locally and temporally[END_REF]. However, these approaches either do not employ memory or use VLAD-based features of significantly higher dimensionality compared to those typically used in BoF-based models. On the other hand, the proposed method is capable of equipping BoF with memory (by employing a context-aware recurrent quantizer), overcoming all these limitations, while keeping the ability of the BoF model to extract compact, yet discriminative, histogram-based representations.

This work aims to bridge the gap between earlier BoF approaches and the recently proposed DL models by employing a powerful recurrent BoF formulation, which can be readily combined with DL architectures, as well as effectively process sequential data. The proposed method manages to successfully adapt the BoF model to temporal tasks by overcoming the limitations of other recurrent models, e.g., LSTMs and GRUs, that often fail to correctly model the temporal information extracted from videos, as also noted in [START_REF] Miech | Learnable pooling with context gating for video classification[END_REF]. It is worth noting that previous methods successfully managed to combine neural networks with BoF, where the codewords are learned via end-to-end training of the model [START_REF] Passalis | Neural bag-of-features learning[END_REF]. However, they were mainly limited to image analysis problems, while suffering from spatial information loss, due to BoF layer architecture. Furthermore, it has been shown that BoF can be modeled as a (stateless) recurrent neural network that simply sums over the quantized features [START_REF] Richard | A bag-of-words equivalent recurrent neural network for action recognition[END_REF]. In contrast to these approaches, the proposed method employs a stateful recurrent quantizer that exploits the sequential nature of the extracted features. Therefore proposed ReBoF model retains the valuable assets of BoF models, while, at the same time, it can effectively encode the spatio-temporal characteristics of the data. To the best of our knowledge, ReBoF is the first method that provides a stateful recurrent BoF formulation, which can capture the spatio-temporal dynamics of the data and be enclosed in a convolutional network, providing an end-to-end trainable DL model for several visual information analysis tasks.

Proposed Method

The proposed recurrent BoF model is analytically derived in Section 3.2, after briefly introducing the standard Neural BoF model in Section 3.1. Then, the suggested methodology to apply ReBoF in two use cases, i.e., video and image classification, is described in Section 3.3

Standard BoF

Assuming that a set X = {x i } N i=1 of N objects is given, N i feature vectors are extracted from each object and notated as

x ij ∈ R D (j = 1, ..., N i ),
where D is the dimensionality of the feature space. For example, for image classification tasks each of the N images is represented by feature vectors extracted using a deep convolutional network or hand-crafted feature descriptors [START_REF] Nanni | Handcrafted vs. non-handcrafted features for computer vision classification[END_REF].

BoF aims to aggregate these extracted feature vectors into a fixed-length histogram. This is achieved using a two-stage procedure, where during the first stage each feature vector is quantized using a predefined number of codewords and then, during the second stage, the N i quantized feature vectors of each object are accumulated to form the final histogram. Note that for the first stage, a codebook V ∈ R N K ×D must be employed, where N K is the number of codewords. This codebook is usually learned by clustering all feature vectors into N K clusters [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF]. Common clustering algorithms, such as k-means, can be used in this step, with each centroid,

v k ∈ R D (k = 1, ..., N K ),
corresponding to a codeword.

Several feature quantization approaches have been proposed [START_REF] Sivic | Video google: A text retrieval approach to object matching in videos[END_REF][START_REF] Passalis | Neural bag-of-features learning[END_REF]. In this work, we focus on a soft quantization approach that allows for learning the codebook using regular back-propagation, along with the rest of the parameters of the model [START_REF] Passalis | Neural bag-of-features learning[END_REF]. This can significantly improve the discriminative power of the extracted representation, since the codebook is fined-tuned for the task at hand.

The feature vector x ij , extracted from the i-th object, is quantized by measuring its similarity with each of the N k codewords as following:

[d ij ] k = exp -v k -x ij 2 σ ∈ [0, 1], (1)
where σ controls the fuzziness of the quantization assignment (also known as the width of the Gaussian kernel). Then, the fuzzy membership vector of each feature vector is obtained after normalizing the observed similarities:

u ij = d ij d ij 1 ∈ R N K . (2)
Finally, the normalized membership vectors are accumulated in a histogram for every object:

s i = 1 N i Ni j=1 u ij ∈ R N K . (3)
Note that the histogram s i has unit l 1 norm, regardless the number of the extracted feature vectors, and provides an aggregated representation of the feature vectors extracted from the corresponding object. Then, the compiled histogram can be fed to a multilayer perceptron (MLP) to classify the object or used in other tasks, such as regression or retrieval.

Recurrent BoF

Note that the representation extracted using the standard BoF formulation described by (1),

(2) and (3) completely discards any spatial or temporal information encoded by the input feature vectors. To overcome this limitation, BoF is extended by employing a recurrent stateful quantizer that can capture and effectively encode the temporal information. Note that in case of sequential 215 data, the feature vector x ij corresponds to the j-th timestep of the i-th sequence. As we will demonstrate in Subsection 3.3, this is without loss of generality, since the same approach can be also used to capture part of the spatial information of the input.

Before deriving a recurrent quantization approach, it is worth revisiting the quantization process involved in the BoF model from a probabilistic perspective. The probability of observing each feature vector x ij , given an input object x i , can be trivially estimated using Kernel Density Estimation [START_REF] Bhattacharya | A probabilistic representation for efficient large scale visual recognition tasks[END_REF] as:

p(x ij |x i ) = N K k=1 [s i ] k K(x ij , v k ) ∈ [0, 1], (4)
where K(•) is a kernel function and the histogram (image-specific parameters) s i ∈ R N K separately adjust the density estimation. Then, the histogram can be calculated using a maximum likelihood estimator:

s i = arg max s Ni j=1 log N K k=1 [s] k K(x ij , v k ) . (5)
It can be easily shown [START_REF] Bhattacharya | A probabilistic representation for efficient large scale visual recognition tasks[END_REF], that these parameters can be estimated as

s i = 1 N i Ni j=1 u ij , where [u ij ] k = K(x ij , v k ) N K l=1 K(x (t) ij , v l ) ∈ [0, 1], (6)
giving rise to the regular BoF with soft-assignments, as described in the previous subsection. This formulation allows for replacing the Gaussian kernel used in [START_REF] Nanni | Handcrafted vs. non-handcrafted features for computer vision classification[END_REF], which requires tuning the width σ of the kernel, with an easier to use hyperbolic kernel that involves no hyper-parameters. The main reason for this choice is that the Gaussian kernel proved to be quite unstable, when employed in a recurrent quantizer. On the other hand, the proposed hyperbolic-based formulation was significantly stabler and easier to use. Therefore, a hyperbolic (sigmoid) kernel is used to compute the similarity between each feature vector and the codewords:

[d ij ] k = σ(x T ij v k ) ∈ R, (7)
where σ(x) = 1 1+exp(-x) is the logistic sigmoid function. Note that this formulation still ignores the temporal information, since it provides no way to encode the current state of the quantizer. Therefore, in order to take into account the temporal information the current state, as expressed by the histogram compiled using the feature vectors fed to the network until the current time-step, [START_REF] Passalis | Learning bag-of-embedded-words representations for textual information retrieval[END_REF] is extended to:

d ij = σ Vx ij + V h (r ij s i,j-1 ) ∈ R N K , (8)
where V h ∈ R N K ×N K is weight matrix that is used to transfer the gated histogram vector into the quantization space and is learned during the training process, s i,j-1 is the histogram extracted from previous quantizations (states) and r ij ∈ R N K is the output of a reset gate, introduced to ensure the long-term stability of the model. The reset gate is inspired by the GRU model [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF] and is defined as:

r ij = σ(W r x ij + U r s i,j-1 ) ∈ R N K , (9)
where W r ∈ R N K ×D and U r ∈ R N K ×N K are the weight matrices used to implement the reset gate.

Then, similarly to standard BoF approach, the l 1 normalized membership vector is computed as in (2):

u ij = d ij d ij 1 . ( 10
)
Note that in order to ensure that quantizer's output is always a properly normalized membership vector, the initial state s i,0 is set equal to s i,0 = 1 N K 1, where N K is the number of recurrent codewords and 1 ∈ R N K is a vector of all ones. Hence, the fixed-length histogram is recurrently updated as follows:

s i,j = (1 -z ij ) s i,j-1 + z ij u ij ∈ R N K , (11)
where the update gate z ij controls how much the current histogram, which also encodes the state of the quantizer, will be updated and it is defined as:

z ij = σ(W z x ij + U z s i,j-1 ) ∈ R N K , (12)
where W z ∈ R N K ×D and U z ∈ R N K ×N K are parameters of the update gate. Finally, the total histogram is compiled by averaging the intermediate state histograms as:

s i = 1 N i Ni j=1 s i,j ∈ R N K . (13)
All the parameters of the ReBoF model can be learned using regular back-propagation. Note that ReBoF is capable of recursively processing the input feature vectors, while capturing their temporal information. First, the feature vectors are quantized using the proposed recurrent stateful quantizer, as described by ( 8) and [START_REF] Iosifidis | Discriminant bag of words based representation for human action recognition[END_REF]. Then, the quantized vectors are used to appropriately update the state of the quantizer, as given by [START_REF] Iosifidis | Multidimensional sequence classification based on fuzzy distances and discriminant analysis[END_REF], and finally compile the resulting histogram. It is worth noting that ReBoF, similarly to all BoF-based models, is capable of processing varying-length input sequences.

The pipeline of the proposed ReBoF pooling method is summarized in Fig. 1. The input feature vectors x ij are first fed to the reset gate which controls how much the previous histogram s i,j-1

will contribute to the quantization process. Note that recurrent connections are plotted in red in Fig. 1. Then, the proposed recurrent quantizer is employed to extract the membership vector ReBoF for video analysis: As presented in 3.2, the advantage of ReBoF is its ability to capture the temporal information of sequential data. This can be readily exploited for tackling challenging video analysis problems, e.g., activity recognition, video retrieval, etc. Assuming that every frame of the video is described by an extracted feature vector x ij and that the i-th video is composed of N i frames, then ReBoF can be directly applied by feeding to it the sequence of . . .

u ij ,

Feature Extraction

Framelevel Pooling ... the frame-based feature vectors. This allows ReBoF to process videos of arbitrary duration, while creating fixed-length compact representations of them. This process is illustrated in Fig. 2a, where a feature vector is extracted from every frame of a video. Note that usually a high-dimensional tensor is first extracted from every video frame when convolutional feature extractors are used. This tensor is first aggregated into a feature vector, using a frame-level pooling approach (ReBoF can be also used for this task, as discussed below), and then the feature vectors sequence is fed into the ReBoF model to extract a video-level compact representation. This representation can be then used for the task at hand, e.g., classification.

ReBoF Classifier

Input Video

ReBoF for image analysis: Moreover, ReBoF can be utilized also for processing nonsequential input data, while preserving crucial spatial information. For example, in case of image classification, let X ∈ R W ×H×C be the 3D feature map extracted from a CNN layer, with width, height and number of channels equal to W , H and C respectively. Several ways to "scan" the image can be employed to flatten the feature map into a set of vectors. In the simplest case considered in this paper, the image is scanned from up to down and left to right. Therefore, a total of N i = W • H feature vectors are extracted from each image. Then, this flattened sequence of feature vectors can be fed into the ReBoF model. This process is illustrated in Fig. 2b. Note that since the feature vectors are always produced following the same scanning pattern, the model can learn the spatial interactions between the extracted feature vectors, effectively capturing the spatial information.

ReBoF for spatio-temporal analysis: Finally, it is possible to combine two ReBoFs in order to capture the spatio-temporal content of the input video. Similarly to the image analysis approach, the first ReBoF model can be used to create a histogram for every video frame, i.e., perform the frame-level pooling, as shown in Fig. 2b. Then, a time series of histograms is compiled and fed to the second ReBoF that captures the temporal information of the video (as shown in Fig. 2a) and thus leads to a compact spatio-temporal representation that is subsequently fed to the classifier.

Experimental Evaluation

In this section the proposed ReBoF model is evaluated on six different datasets. The experimental procedure along with the results are presented and discussed in the following two main subsections. In the first one, the proposed method is validated over three video datasets for activity recognition, while in the second one, evaluation is conducted over three datasets for image classification.

Video Activity Recognition

Datasets: The proposed ReBoF model is initially validated in 3 different video datasets for activity recognition. The specific task was selected to demonstrate the ability of ReBoF to effectively capture the temporal dimension of video sequences, which is expected to contain essential information for classification tasks.

The first dataset is the UTKinect-Action3D [START_REF] Xia | View invariant human action recognition using histograms of 3d joints[END_REF] that consists of 10 types of human activities in indoor settings. Samples for each action are collected from 10 different people that perform every activity two times. The demonstrated actions are: walk, sit down, stand up, pick up, carry, throw, push, pull, wave hands, and clap hands. For each video instance, three streams were recorded: RGB, depth and skeleton joint locations. The RGB images were used for all of the conducted experiments.

Since there is no official training/testing split provided, a 50%-50% subject-based split strategy was employed, i.e., the videos of the first five subjects were included in the training set and the rest of them were used to form the testing set. Hence, a quite challenging setup was created, as the activities belonging in the testing set were performed from unseen subjects.

The second database is the well-known UCF101 dataset [START_REF] Soomro | UCF101: A dataset of 101 human actions classes from videos in the wild[END_REF], that is widely used for benchmarking action recognition models. The dataset contains 13, 320 action instances belonging in 101 classes, that can be grouped in five generic categories, namely: 1. Human-Object Interaction, 2. Body-Motion Only, 3. Human-Human Interaction, 4. Playing Musical Instruments, and 5. Sports. UCF101 also contains three different evaluation splits. For all the experiments conducted in this paper, the first split was used.

We also created a challenging and more complex dataset based on the UCF101 to better demonstrate the ability of the proposed method to capture the temporal dynamics of video data. To this end, we compiled a dataset by mixing instances from different activities of the UCF101 dataset together. More specifically, 10 activities of UCF101 (split 1) were selected to be joined together.

Every action of this subset was joined with each one of the remaining, leading to 90 complex actions. One can further understand the significance of encoding the temporal information of these instances by considering that a sample of action "A" combined with one of action "B" (let name this complex activity class "AB") must be separated from samples of complex activities from class "BA". Note that "AB" and "BA" videos contain the same set of frames (for a specific instance of "A" and "B"), but in a different order. A typical example of such two complex actions would be the action sequences of "sitting to a chair" (AB) and "rising from a chair" (BA), where A is the action of getting closer to the chair and B the action of actually being on the chair. A method that does not capture the temporal information cannot discriminate between these two actions, since the videos would contain the same frames, but in the reverse order. Hence, 114 samples were selected for each class, as this was the minimum number of instances contained in the selected initial classes. The selected 10 action classes were the following: ApplyEyeMakeup, ApplyLipstick, Billiards, BoxingPunchingBag, BoxingSpeedBag, Haircut, Hammering, TableTennisShot, TennisSwing, and Typing. These classes were selected to offer a challenging dataset of complex activities, where every primary action has familiar content with at least one of the rest. The i-th sample of initial class "A" is combined with the i-th sample of initial class "B" and so on, leading to 7, 380 training and 2, 880 testing data equally balanced among the 90 classes. The compiled dataset is called "Complex UCF" in the rest of this paper.

Evaluation Setup: The video analysis approach described in Subsection 3.3 was employed in order to evaluate the proposed method over the first two datasets. Every video instance was uniformly sampled in time in order to extract a predefined number of frames, denoted by N f . The number of extracted frames was set to N f = 30 for the UTKinect-Action3D dataset and to N f = 40 for the UCF101 dataset. Shorter videos were looped over as many times as necessary to ensure that each video contains at least N f frames, following the methodology described in the relevant literature [START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF].

Every frame was fed to an Inception V3 model [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], pretrained in ImageNet, and a feature representation was extracted from each frame from the last average pooling layer of the network.

Therefore, every frame is represented by a 2048-dimensional feature vector, building a sequence of N f features for every video sample. This sequence is then fed to the ReBoF layer, which is followed by a fully connected layer block composed of two fully connected layers. The first one is composed of 512 neurons using the ReLU activation function and dropout with rate of 0.5, while the output (classification) layer has as many neurons as the classes of each dataset and employs the softmax activation function. The cross-entropy loss is used during the training. The resulting network was trained from scratch using the Adam optimizer and a learning rate of 10 -5 , apart from the pretrained feature extractors which were kept frozen. We also experimentally found out that scaling the extracted histogram by N K improved the convergence of the proposed method, especially when training the whole architecture from scratch. This scaling ensures that the gradients from the fully connected layers will not diminish as they are back-propagated to the previous layers. For the UTKinect-Action3D dataset the models were trained for 800 epochs, while for the UCF101, all models were trained for 50 epochs and the training/evaluation procedure was repeated 3 times.

The performance of the proposed method is compared with three other pooling methods, that also use the exact same feature extraction and classification layer blocks (in order to ensure a fair comparison). First, global average pooling over the temporal dimensions of the input sequence is used in place of the proposed ReBoF layer. This method is called "Average Pooling" in the rest of this paper. For the second method, a recently proposed state-of-the-art variant of the BoF method that is adapted for use with DL architectures, the Linear BoF [START_REF] Passalis | Training lightweight deep convolutional neural networks using bag-offeatures pooling[END_REF], is employed. Furthermore, a powerful recurrent aggregation model, the GRU [START_REF] Cho | Learning phrase representations using rnn encoder-decoder for statistical machine translation[END_REF], which is used to aggregate the features, while also capturing the temporal dimension of the data, is also compared to the proposed method. To ensure a fair comparison, we use the same number of codewords or hidden units for the Linear BoF or GRU model, respectively, as the number of codewords used in our method, except otherwise stated.

A slightly different setup was used for the third dataset (Complex UCF101). A 16-frame sliding window, with overlap of 4 frames was applied on every activity instance of UCF101 dataset, creating 16-frame clips. A 3D ResneXt-101, pretrained on UCF101, as deployed in [START_REF] Hara | Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?[END_REF], was used for feature extraction. Every clip was then fed to the network and features were extracted from the last average pooling layer of the network. After this step, every video clip is represented by a 2048dimensional vector. Note that contrary to the previous setup, feature vectors already enclose some spatio-temporal information, since 3D convolutions were used during the feature extraction. Then, for every activity instance of class "AB", 16 sequential feature vectors were selected from each one of the classes "A" and "B", starting from the beginning of the video. If a subsequence of features is shorter, then it was looped over. The two subsequences were then stacked, forming a 32-length sequence (N f = 32) of feature vectors for the action "AB" of the ComplexUCF dataset.

Similar to the previous setup, the performance of ReBoF is compared to the "Average Pooling", "Linear BoF" and "GRU" methods. All methods share the same classification block as before and the networks were trained from scratch, excluding the weights of feature extractors which were pretrained. In case of Average Pooling, the network was trained for 50 epochs, while in the rest, the training process stopped when 99.9% accuracy was achieved in training set. The evaluation of ReBoF and GRU methods is repeated 3 times and the mean accuracy and standard deviation on the test set are reported.

Experimental Results: The experimental evaluation for the UTKinect-Action3D dataset is provided in Table 1. The proposed method outperforms the other three evaluated methods, providing the highest accuracy (54.64%) using 512 codewords. GRU also achieves its best accuracy (47.71%) for the same number of codewords, while for the Linear BoF method the highest performance (44.47%) is achieved for 256 codewords. Both ReBoF and GRU significantly outperform the Average Pooling and Linear BoF methods, since they are capable of effectively modeling the temporal dynamics of the input video sequences and distinguishing between similar activities, such as stand up and sit down.

Moreover, in Fig. 3 the effect of using a wider range of codewords and number of GRU units in the classification accuracy on the UTKinect-Action3D dataset is evaluated using the two methods that achieve the highest performance (GRU and ReBoF). In all cases, the proposed method leads to higher accuracy compared to the GRU method. Furthermore, the proposed method allows for reducing the size of the extracted representation, since it outperforms the best GRU model (512 units) using just 128 codewords. This allows for reducing the size of the extracted representation and, as a result, the number of parameters in the subsequent fully connected layer. Both methods achieve their best performance for 512-dimensional representations. After this point, the accuracy for both models drops, mainly due to overfitting phenomena.

Similar conclusions can be drawn from the evaluation results on the UCF101 dataset, which are reported in Table 2. The experiments were repeated three times and the mean accuracy and standard deviation on the test set are reported. Even though UCF101 is a less challenging dataset, in terms of temporal dependence, the proposed method still outperforms the rest of the evaluated methods, achieving the highest accuracy of 72.02% for 1024 codewords. The effect of using different number of codewords and recurrent units is also evaluated in Fig. 4. Again, the proposed method outperforms the GRU method regardless the number of used codewords, while it achieves comparable accuracy using 2 to 4 times smaller representations. Next, the evaluation on the Complex UCF dataset is provided in Table 3. Note that the Average Pooling and Linear BoF methods fail to overpass the 50% test accuracy barrier, since "AB" activities cannot be distinguished from those of "BA", due to discarding crucial temporal information. Note that the Average Pooling and Linear BoF methods obtain significantly lower accuracy on this dataset, since capturing the temporal information plays a less crucial role on the other two datasets, in which many videos that belong to certain actions can be categorized just by one frame, leading to a smaller gap between the Average Pooling method and the ones that capture the temporal information, i.e., GRU and ReBoF. Again, ReBoF achieves the highest accuracy equal to 89.29% for 1024 codewords, outperforming the GRU method. Moreover, qualitative results on Complex UCF101 dataset are presented in Fig. 5 in order to further illustrate the ability of ReBoF to encode efficiently the temporal information of video data. ReBoF is compared with the rest of the methods using the confidence scores for the topk (k = 1, 2, 3) predictions. The corresponding labels are reported for each prediction. Fig. 5 clearly demonstrates the inefficiency of Average Pooling and Linear BoF methods to distinguish the combination of activities "TennisSwing + ApplyLipstick" ("AB") from the inverted combination of activities "ApplyLipstick + TennisSwing" ("BA"), due to the complete loss of the temporal information. GRU method presents to some extent better performance since the correct predicted label is found on top-2 ranking. Yet, the model is overconfident over the wrong class, probably due to the difficulty of distinguishing the second part ("ApplyLipstick") of the complex action from a similar one ("ApplyEyeMakeup"). On the other hand, the proposed ReBoF method is more robust to such phenomena, more efficiently capturing the temporal details and managing to accurately classify the input sample, contrary to rest of the methods. It should be also noted that the inverted complex action "ApplyLipstick+TennisSwing" is not present in the first top-3 ranking and thus, the confidence score for the specific class is at least lower than 0.01, signifying the effectiveness of ReBoF's recurrent quantizer to enclose the temporal characteristics of the video sequence.

Finally, in Table 4 ReBoF is compared to other similar approaches reported in the literature on UCF101 dataset. Activity recognition based on a single RGB frame of the video achieved 67.4% accuracy across all three splits of UCF101. LRCN model presented in [START_REF] Donahue | Long-term recurrent convolutional networks for visual recognition and description[END_REF] improved the results by employing an LSTM to capture the temporal information of the data. The "slow fusion" scheme presented in [START_REF] Karpathy | Large-scale video classification with convolutional neural networks[END_REF] aimed to conceive the temporal information by applying a CNN network over a time window of the RGB stream, yet the reported results are lower compared to the case of single frame network, failing to exploit the temporal information contained in the video input. On the other hand, the proposed ReBoF formulation manages to effectively capture the spatial information of video streams, outperforming the rest of the evaluated methods. It is worth noting that other approaches that use C3D networks, such as the I3D network [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF], can further boost the action recognition performance. However, these more powerful architectures usually also require using strated in Table 4 (e.g., training 3D-ConvNets from scratch leads to lower recognition accuracy), while at the same time these architectures also significantly increase the computational complexity of the model compared to the proposed method.

Image Classification

Datasets: ReBoF is also able to capture the spatial information of the extracted feature vectors, as discussed in Section 3.3. To this end, ReBoF is also evaluated on three image datasets: the MNIST database of handwritten digits (MNIST) [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF], the Fashion MNIST database of fashion images [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF],

and the CIFAR-10 database of color images with varying content [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF].

The MNIST dataset is widely used in the computer vision field and contains 60, 000 training and The ReLU activation function is used for all the hidden layers, while the network is trained to minimize the cross-entropy loss. The network is trained from scratch in an end-to-end fashion for 100 epochs using the Adam optimizer and learning rate equal to 0.001.

The proposed method is also compared to using an average pooling baseline, as well as to the Linear BoF method, similarly to the activity recognition task. In the average pooling method, the output of the employed feature extraction block is fed to a global average pooling layer and then forwarded to the employed classification layer block. In this case, no dropout is applied to the feature extractor's output. In the Linear BoF method, the ReBoF block is substituted by a Linear BoF, while the rest of the network is kept the same to ensure a fair comparison between the methods. Both networks were trained from scratch, using the same training parameters as in the case of ReBoF.

Evaluation Results: The proposed method is compared with two other evaluated methods in Table 5. ReBoF outperforms both the Average Pooling and Linear BoF methods in all the evaluated cases, regardless the used dataset. For the MNIST and Fashion MNIST datasets the highest performance is achieved when 512 codewords are used, while for the CIFAR-10 dataset for 128 codewords. Similarly, for the Linear BoF method the best results for the MNIST dataset are achieved when 64 codewords are employed, whereas in the case of Fashion MNIST and CIFAR-10 when 128 codewords are used. The performance improvements are marginal for the simpler MNIST and Fashion MNIST datasets. However, significant accuracy improvements are observed for the more complex CIFAR-10 dataset, confirming the ability of the proposed method to capture the spatial information encoded in the extracted feature vectors. This is also confirmed in the results reported in Fig. 6, where the effect of varying the number of codewords for the ReBoF method is examined. ReBoF achieves remarkable results in the MNIST and Fashion MNIST datasets, even for a small number of codewords. It is worth noting that when a very large number of codewords is used, i.e., 1024 or 2048, the accuracy begins to decrease due to overfitting. The effect of the number of codewords on the model's accuracy can be also clearly demonstrated for CIFAR-10, which contains larger and more complex images. In this case, the best accuracy is achieved when 128 codewords are used.

Also, the ability of ReBoF method to capture spatial patterns from images is further demonstrated in Fig 7, where qualitative results are presented for the MNIST dataset. In case of Average

Pooling method, the prediction model is overconfident over the wrong digit in all of the three evaluated cases. Linear BoF presents a more balanced distribution over the prediction scores, yet it also fails in some cases to classify the input image correctly. On the contrary, ReBoF, which is capable of capturing the spatial information contained in the feature vector sequence extracted from each MNIST digits, leads to the best results, accurately identifying each digit, while also providing the highest confidence on the correct label.

Conclusions

In this paper a novel Recurrent Bag-of-Features formulation has been proposed, designed to effectively process sequential input data. ReBoF was inspired by the BoF model, but employs a stateful trainable recurrent quantizer, instead of a plain static quantization approach. This enables ReBoF to harness the power of a powerful recurrent quantizer that is able to capture the temporal

Input

Average Pooling Linear BoF ReBoF Ranking information of input data, which is crucial in classification tasks, such as activity recognition, while still maintaining all the advantages of the BoF model. ReBoF can be directly used between the last feature extraction layer and the fully connected layer of a network, while the resulting architecture can be trained in an end-to-end fashion using back-propagation, allowing for building powerful deep learning models for various visual information analysis tasks. The performance of ReBoF model was extensively evaluated in various activity recognition tasks using three video datasets. In all cases, the proposed method outperformed the rest of the evaluated methods, demonstrating the ability of ReBoF to capture the temporal information and increase the classification accuracy. Furthermore, ReBoF is also capable of encoding the spatial information, as it was also demonstrated in image classification tasks using three more datasets.

ReBoF opens several interesting future research directions. First, the proposed approach for activity recognition can be further enhanced by combining a ReBoF layer over the spatial dimension, followed by a ReBoF layer over the temporal dimension. In this way, the spatio-temporal information can be more properly encoded by creating a space-time histogram, further improving the performance over the applications presented in this paper. Second, ReBoF can be employed to replace the intermediate weak pooling layers of CNNs, aiming to increase the accuracy of the models. Finally, employing the proposed method for other tasks, such as video retrieval and hashing, is expected to boost the performance of existing methods by extracting compact, yet more discriminative representations.

  210

3. 3 .

 3 Using Recurrent BoF for Visual Information Analysis TasksReBoF can be directly used in any DL architecture without any modification between the last feature extraction layer and the first fully connected one. The extracted representation s i depends only on the number of used codewords N K and, as a result, ReBoF is capable of processing inputs of variable size without any modification. Three different ways to employ ReBoF in DL models are presented: a) video analysis using CNN frame-based feature extractors, b) image analysis using CNNs and c) spatio-temporal video analysis.

Figure 2 :

 2 Figure 2: Using Recurrent BoF for two different tasks: (a) video analysis and (b) image analysis

Figure 3 :

 3 Figure 3: UTKinect-Action3D Evaluation: Effect of using different number of codewords/recurrent units for the ReBoF and GRU methods

Figure 4 :

 4 Figure 4: UCF101 Evaluation: Effect of using different number of codewords/recurrent units for the ReBoF and GRU methods

Figure 5 :

 5 Figure 5: Qualitative results on the Complex UCF101 dataset: For a given video sequence the first three top-k (k = 1, 2, 3) prediction scores among with the corresponding predicted label are reported for each of the evaluated methods. In the first column (top-1 results) red and green color is used to denote false and correct predictions, respectively.

Figure 6 :

 6 Figure 6: Test accuracy (%) of ReBoF in three datasets for different number of codewords

  which is then fed to the update gate. The update gate controls how much the previous histogram will be updated based on the output of the recurrent quantizer u ij and the previous histogram s i,j-1 , leading to the updated histogram s ij . The final histogram representation of an Pipeline of the proposed ReBoF pooling approach. Figure depicts the main information processing steps involved in ReBoF. Red lines denote recurrent connections/information flow, while blue lines denote feed-forward connections/information flow. Figure best viewed in color.

			Recurrent Bag-of-Features
	Input Features	Recurrent Quantizer Codebook Normalization	Update	⋮	Accumulator	Output Histogram
	0			Gate	⋮	, -1
	⋮			⋅
						⋮
					⋮	,
		⋮	⋮
	⋮
	Reset Gate Control		Update Gate Control
	,			,
		⋅
	⋮	Reset		⋮
		Gate
	Figure 1:

input object is obtained by incrementally accumulating the extracted histogram vectors into the final one s i ∈ R N K . It is worth noting that compared to other Neural BoF approaches

[START_REF] Passalis | Neural bag-of-features learning[END_REF]

, the proposed one is the first one that employs a recurrent quantizer by incorporating an additional reset gate and update gate. The reset gate is used to equip the quantizer with (resetable) memory, allowing for performing context-aware quantization, while the update gate allows for dynamically updating the histogram only when appropriate (e.g., ignoring features that are not relevant to the current context).

Table 1 :

 1 UTKinect-Action3D Evaluation

	Method	# Codewords -GRU Units Test Accuracy (%)
	Average Pooling	-	40.83
	Linear BoF	256	44.47
	GRU	512	47.71
	ReBoF	512	54.64

Table 2 :

 2 UCF101 Evaluation

	Method	# Codewords -GRU Units Test Accuracy (%)
	Average Pooling	-	70.32 ± 0.43
	Linear BoF	1024	71.11 ± 0.35
	GRU	2048	71.04 ± 0.20
	ReBoF	1024	72.02 ± 0.68

Table 3 :

 3 Complex UCF101 Evaluation

	Method	# Codewords -GRU Units Test Accuracy (%)
	Average Pooling	-	48.95
	Linear BoF	1024	43.88
	GRU	512	88.86 ± 2.04
	ReBoF	512	89.25 ± 1.08
	GRU	1024	88.62 ± 1.02
	ReBoF	1024	89.29 ± 0.89

Table 4 :

 4 Comparison with other methods on UCF101 (split 1)

	Method

  10, 000 testing grayscale images of handwritten digits. The size of each image is 28 × 28 pixels and the data are distributed over 10 different classes, one for each digit (0 to 9). The Fashion MNIST is an MNIST-like dataset that contains labeled fashion grayscale images. Similarly to MNIST, it consists of 60, 000 training and 10, 000 testing images of size 28 × 28 pixels. The CIFAR-10 dataset contains totally 60, 000 colour images of size 32 × 32 pixels. The dataset is separated in 50, 000 images for training and 10, 000 images for testing. Data are equally divided in 10 classes of various generic categories, e.g., airplane, cat, ship, etc. ReBoF layer, and 3. a classification layer block. The feature extraction block is composed of two convolutional sub-blocks. The first sub-block is composed of two convolutional layers that use 32 3 × 3 filters, followed by a 2 × 2 max pooling layer. The second sub-block consists of two additional convolutional layers that use 64 3 × 3 filters, followed by a 2 × 2 max pooling layer. In addition, dropout with rate 0.25 is applied to the input of second block. The output of the last convolutional block is used to extract feature vectors for each image. The final feature map is scanned from left to right and from top to bottom to extract feature vectors that are sequentially fed to the ReBoF block, after applying dropout with rate of 0.25. The extracted features are recurrently quantized in N K -dimensional vectors and accumulated in a N K -length histogram. The output of the ReBoF layer is then fed to the classification layer block that is composed of a hidden fully connected layer with 512 neurons and an output layer with 10 neurons, while dropout with rate of 0.2 and 0.5 is applied to the input of the first and second fully connected layers, respectively.

	Evaluation Setup: The proposed method is evaluated in the aforementioned image datasets,
	following the approach presented in Section 3.3. The ReBoF model is combined with a convolutional
	neural network, creating a model that consists of three main layer blocks: 1. a feature extraction
	block, 2. a

Table 5 :

 5 Image classification evaluation

	Method	MNIST Fashion MNIST CIFAR-10
	Average Pooling	99.44	92.80	79.71
	Linear BoF	99.46	92.97	80.25
	ReBoF	99.50	93.28	82.25
	(Classification accuracy is reported (%))

Acknowledgments

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR). This publication reflects the authors' views only. The European Commission is not responsible for any use that may be made of the information it contains.


Nikolaos Passalis
email: passalis@csd.auth.gr

Alexandros Iosifidis

Moncef Gabbouj
email: moncef.gabbouj@tuni.fi

Anastasios Tefas
email: tefas@csd.auth.gr

ROBUST HYPERSPHERE-BASED WEIGHT IMPRINTING FOR FEW-SHOT LEARNING

Keywords: Weight Imprinting, Few-shot Learning, Edge TPU, Embedded Deep Learning

Performing fast few-shot learning is increasingly important in a number of embedded applications. Among them, a form of gradient-descent free learning known as Weight Imprinting was recently established as an efficient way to perform few-shot learning on Deep Learning (DL) accelerators that do no support back-propagation, such as Edge Tensor Processing Units (Edge TPUs). Despite its efficiency, WI comes with a number of critical limitations. For example, WI cannot effectively handle multimodal novel categories, while it is especially prone to overfitting that can have devastating effects on the accuracy of the models on novel categorizes. To overcome these limitations, in this paper we propose a robust hypersphere-based WI approach that allows for regularizing the training process in an imprintingaware way. At the same time, the proposed formulation provides a natural way to handle multimodal novel categories. Indeed, as demonstrated through the conducted experiments, the proposed method leads to significant improvements over the baseline WI approach.

I. INTRODUCTION

Deep Learning (DL) has achieved remarkable results on a wide range of especially challenging problems [START_REF] Lecun | Deep learning[END_REF], which span from image understanding and visual questioning answering problems [START_REF] Xu | Ask, attend and answer: Exploring question-guided spatial attention for visual question answering[END_REF], to natural language processing [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], and video analysis [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF]. Despite the success of DL on the aforementioned areas, DL models require powerful hardware both during the training process, as well as during the inference [START_REF] Cheng | Model compression and acceleration for deep neural networks: The principles, progress, and challenges[END_REF], [START_REF] Passalis | Heterogeneous knowledge distillation using information flow modeling[END_REF]. The development of advanced Graphics Processing Units (GPUs) [START_REF] Vasilache | Fast convolutional nets with fbfft: A gpu performance evaluation[END_REF], as well as Tensor Processing Units (TPUs) [START_REF] Norman P Jouppi | In-datacenter performance analysis of a tensor processing unit[END_REF], allowed for accelerating both the training and inference processes, as well as lowering the energy requirements of DL. However, most of these accelerators remain too bulky for many embedded applications, including robotics [START_REF] Sünderhauf | The limits and potentials of deep learning for robotics[END_REF] and Internetof-Things (IoT) applications [START_REF] Li | Learning iot in edge: Deep learning for the internet of things with edge computing[END_REF], leading to developing extremely energy efficient hardware for accelerating only the inference process, such as Edge TPUs [START_REF] Hu | Deephome: Distributed inference with heterogeneous devices in the edge[END_REF]. These platforms can lead to tremendous improvements in terms of operations/Watt. However, being designed to support inferenceonly operations, they are unable to support back-propagation, rendering impossible to perform on-device training. These limitations severely restrict their applications on open-word settings, which frequently occur in many applications, e.g., robotics [START_REF] James | Taskembedded control networks for few-shot imitation learning[END_REF], where DL models should be able to promptly adapt to emerging categories that were not seen during the training. This problem is known as few-shot learning or lowshot learning [START_REF] Lake | One shot learning of simple visual concepts[END_REF], [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], [START_REF] Hariharan | Low-shot visual recognition by shrinking and hallucinating features[END_REF], where models are required to learn new categories and generalize the already encoded knowledge using just a few labeled examples.

Despite the large number of few-shot learning methods proposed in the literature [START_REF] Lake | One shot learning of simple visual concepts[END_REF], [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], [START_REF] Hariharan | Low-shot visual recognition by shrinking and hallucinating features[END_REF], [START_REF] Wang | Low-shot learning from imaginary data[END_REF], just a few of them support inference-only DL accelerators. Perhaps the most widely known is the Weight Imprinting (WI) approach [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], which was recently proposed as the default way to perform training on the Google's Edge TPU accelerator. WI allows for extending the classes a DL model can recognize by performing gradient descent-free learning. To this end, an appropriate vector prototype is calculated for each novel class and, then, this prototype vector is imprinted in the last classification layer of a DL model. Note that this process does not require back-propagating any gradients through the network, while the required calculations can be efficiently performed using an embedded CPU, rending the whole process extremely efficient.

However, WI suffers from significant limitations, that hinders its application in many real-world cases. First of all, WI implicitly assumes that the distribution of the features extracted from the DL model for novel categories will be unimodal. While this assumption usually holds for the distribution of classes for which the DL model was trained, there is no guarantee that this will hold for novel categories, which were never presented to the network during the training process. Furthermore, the impact of the training process on the actual performance on the network during few-shot learning has not been adequately examined in existing literature. More specifically, as we also experimentally demonstrate in this paper, there seems to be a direct connection between maintaining the variance of the embeddings around the prototypes and the generalization abilities of a representation/model on unknown classes. This is not a surprising result, since it is well known that overfitted representations almost always lead to worse generalization (after a certain point) [START_REF] Srivastava | Dropout: a simple way to prevent neural networks from overfitting[END_REF], [START_REF] Passalis | Entropy optimized featurebased bag-of-words representation for information retrieval[END_REF], [START_REF] Passalis | Variance-preserving deep metric learning for content-based image retrieval[END_REF]. This naturally leads us to the following question: Is it possible to design a representation in which the variance around the prototypes will be deliberately controlled to achieve the perfect balance between overfitting and underfitting instead of relying on early-stopping, implicit regularization or other heuristics to maintain enough variance? Also note that maintaining the variance will allow more information about the in-class similarities/dissimilarities to be encoded in the resulting representation.

The main contribution of this paper is a novel weight imprinting method, which is capable of overcoming the aforementioned limitations. To this end, the proposed method learns a regularized embedding by maintaining the variance around the prototypes in a structured way. The proposed approach provides an effective a way to directly handle novel categories with multimodal distributions, as well as natively supports few-shot learning. To the best of our knowledge, in this paper we propose the first variance-preserving approach for performing imprinting-aware few-shot learning.

The rest of this paper is structured as follows. First, the proposed method is derived and discussed in Section II. Then, the proposed method is evaluated and compared to regular WI under different scenarios in Section III. Finally, conclusions are drawn and future research directions are discussed in Section IV.

II. PROPOSED METHOD

The proposed method aims to learn a carefully designed feature space to more effectively support weight imprinting. To this end, a centroid-based loss, which uniformly distributes the embedding vectors within a radius r around each prototype (centroid), is employed. Furthermore, to ensure that the prototypes are discriminative enough it is required that the minimum distance between two prototypes is at least ρ > 2r. This process is illustrated in Fig. 1. After learning a representation that fulfills the aforementioned requirements we can directly classify a new sample, perform gradient descentfree few-shot learning, detect and handle multimodal novel classes and detect intrusion to the existing classes that can lower the performance of the model.

A. Embedding Extractor Training

First, we will describe the proposed imprinting-aware training process. Let φ(x) ∈ R m be the output of a neural network, where m is the dimensionality of the embeddings extracted from the network, when presented with an input sample x. Also, let w i be the prototype vector for the i-th class used during the training process and X i be the set of samples that belong to the class i. Then, to ensure that the embeddings will be uniformly distributed around each class prototype w i ∈ R m we define the appropriate class-induced loss as:

L c = 1 N N C l=1 x∈X l (||φ(x) -w l || 2 -αr) 2 , ( 1
)
where N is the total number of training samples, N C is the total number of training classes, ||•|| 2 denotes the l 2 norm of a vector and α ∈ [0, 1] is a number drawn uniformly from the range [0...1]. During the optimization a different random value is drawn for α for each sample and iteration, leading to a uniform distribution of the embeddings within a radius r from each w i . Even though this process does not ensure that the full space around w i will be occupied, it ensures that the embeddings will be sampled uniformly at various radiuses around the corresponding center, significantly improving the generalization abilities of the representation, as we will demonstrate later in Section III. Note that by setting r = 0, the loss L class degenerates to the regular center loss [START_REF] Wen | A discriminative feature learning approach for deep face recognition[END_REF]. Furthermore, to further model the uncertainty regarding the class prototypes, we can use Gaussian noise to corrupt the prototypes as:

wi = w i + N (0, σ). (2)
At the same time, each prototype w i is required to be at a distance of at least ρ from each other prototype (to ensure that there is no overlapping between the hyperspheres that enclose the embeddings of each class). To this end, we also define the prototype loss as:

L p = 1 N C (N C -1) N C i=1 N C j=1,i =j max(0, ρ -||w i -w j || 2 ).
(3) Therefore, the model parameters, along with the prototypes, are optimized during the training to minimize the following loss:

L = L c + γL p , (4)
where γ is a hyper-parameter parameter that alters the weight of the prototype loss (set to 1 for all the experiments conducted in this paper).

B. Classification and Few-shot Learning

To classify an input sample we can directly choose the class that corresponds to the prototype with the smaller distance to the extracted embedding φ(x). The network can be used in a similar fashion as a one trained using the softmax activation simply by using a final classification layer that calculates the membership value for each prototype/class probabilities as:

p i (x) = 1 1 + ||φ(x) -w i || 2 . (5)
It is worth noting that the probability distribution can be smoothed or sharpened by appropriately transforming it, e.g., using the softmax function with appropriately tuned temperature, if needed.

To perform few-shot learning we can simply augment the final classification layer with an additional prototype vector w n calculated as:

w n = 1 |X n | x∈Xn φ(x), (6)
where X n is the set that contains the training samples for the novel category. Note that similarly to regular WI, no gradient descent-based optimization is required for extending the classifier to support novel classes. However, as we further demonstrate in Section III, the regularized nature of the learned feature space leads to significantly better performance compared to regular WI. Fig. 1. Weight Imprinting using Hyperspheres: The representation space is constructed in a way that natively supports imprinting and spreads the embeddings in hyperspheres with radius r (the in-class variance is better preserved). Note that the proposed method also allow for detecting when the imprinting process cannot be performed safely (e.g., note the potential overlap between the second novel class and the j-th class).

C. Detecting and handling multimodal novel categories

There are several ways to detect if the distribution of a novel class is indeed multimodal. In this paper, we propose using a quite simple, yet effective way: we propose directly detecting multimodal categories by clustering the embedding vectors extracted for a novel category. If we detect centers that are at distance greater than r (or any other user-defined threshold) from each other, then a hypersphere with radius of r cannot enclose the embeddings of the novel class. To address this, we can simply add one or more prototypes (according to the number of centers that are at distance greater than r) to model the distribution of the novel class. In this way, one class can be represented using more than one prototype. On the other hand, if the centers of the clusters are within a radius of r, then we assume that proposed classification scheme can directly handle the distribution of the novel class (even though there is no guarantee that the distribution is not multimodal).

III. EXPERIMENTAL EVALUATION

The proposed method was evaluated using two different image datasets, the MNIST dataset [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] and the Animals with Attributes (AwA2) dataset [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF], in order to measure its performance under two different settings. For the MNIST dataset the first 5 classes were used to train the model, while the remaining 5 classes were used for few-shot learning and evaluation. For the AwA2 dataset [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF] the first 40 classes were used for training the model, and the rest of the classes were employed for evaluating the performance of the proposed method. The performance evaluation was repeated 5 times using different training samples for the novel classes (except for the hyper-parameter evaluation experiments) and the mean and standard deviation is reported. For the MNIST dataset the employed neural network was composed of a 3 × 3 convolutional layer with 32 filters, followed by a 2 × 2 max pooling layer, another 3×3 convolutional layer with 64 filters, an additional 2 × 2 max pooling layer and a fully connected layer with 256 neurons. For the AwA2 dataset we used a pretrained ResNet101 to extract feature vectors (as described in [START_REF] Xian | Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly[END_REF]) that were then fed to two fully connected layers with 2048 and 512 neurons respectively. The relu activation function was used for all the layers [START_REF] Glorot | Deep sparse rectifier neural networks[END_REF]. The models were trained using the Adam optimizer with a learning rate of 10 -3 for 20 training epochs for the MNIST dataset and of 10 -4 for 20 training epochs for the AwA2 dataset [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF].

First, we evaluated the effect of altering the minimum distance ρ between the different class prototypes w i , while keeping the radius fixed to r = 0. The results are presented in Table I. Increasing the minimum distance between the class prototypes seems to have a positive effect on the classification accuracy, both for the novel split (denoted by "Novel") and the combined split of novel and training classes (denoted by "All"). This was expected since the learned representation is not capable of perfectly collapsing the embeddings to the corresponding prototypes, even though the radius was set to r = 0. Therefore, keeping a quite large margin between the different prototypes helps to reduce the risk of wrongly classifying a sample.

Next, we also evaluated the effect of altering the radius r on the learned representation. In Section II it was conjectured that spreading the training embeddings in a hypersphere of radius r will have a positive regularization effect on the learned representation by allowing the model to capture and better model the in-class variations. Indeed, as demonstrated in Table II, the classification accuracy increases by more than 14% for the novel split, and by 1.8% for the combined split. This confirms our hypothesis that collapsing the embeddings to the class centers, without any form of regularization, can significantly reduce the classification accuracy, especially when dealing with classes that were not seen during the training process and using powerful models that can overfit the training data.

Next, we evaluated the proposed method using a 1-shot, 2- shot and 5-shot evaluation protocol on the MNIST dataset.

The results are reported in Table III. Two variants of the proposed method were evaluated: "Proposed-", where r = 0 and σ = 0 were used, and "Proposed", where r = 5 and σ = 0.05 were used. The proposed method was also compared to the plain Weight Imprinting (WI) approach [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], using an initial scaling value of c = 10. Again, it was confirmed that using the proposed variance preserving approach improves the performance over simply using a center-based loss, allowing for outperforming the regular WI method on all the evaluation splits and few-shot learning setups. It is worth noting that the accuracy for all the evaluated methods is relatively low compared to the state-of-the-art, since neither WI or the proposed method perform any kind of optimization according to a discriminative objective.

To demonstrate the ability of the proposed method to handle multimodal classes we also evaluated the proposed approach using an additional multimodal split of the MNIST dataset. This split was compiled by merging two succeeding classes into one, e.g., "0" and "1" were merged into a new class, "2" and "3" into another, and so on. Then, the three first classes (digits 0 to 5) were used for training and the remaining two of them (6 to 9) for evaluating the few-shot learning performance. The evaluation results are reported in Table IV. The employed threshold was used to detecting whether a class distribution is multimodal (by clustering the training data into two clusters and measuring the distance between the resulting centroids). If the distance of the resulting centers was greater than the specified threshold, then two prototypes were used per novel class. Again, note that the variance-preserving variant of the proposed method greatly outperforms the baseline variant ("Proposed-"), as well that using the proposed multimodal handling approach (with a correctly tuned threshold) allows for significantly improving the performance of the proposed method, outperforming all the other evaluated approaches. Finally, we also evaluated the performance of the proposed method using a more challenging dataset, the AwA2 dataset. The results are reported in Table V. We report evaluation results only for the novel categories, since due to the small number of training samples, all the data from the training categories were used for training the model. As before, the proposed method leads to significant performance improvements over the plain WI method, while it still outperforms the HWImethods. The smaller differences between HWI-and HWI can be possibly attributed to the the smaller learning capacity of the employed network (the risk of overfitting the representation is higher when more powerful networks are employed). Note that slightly different hyper-parameters were used for the HWI method in this experiment: ρ = 20, r = 10, and σ = 0.

IV. CONCLUSIONS

In this paper we proposed a novel hypersphere-based weight imprinting approach that maintains all the advantages of regu- lar WI [START_REF] Qi | Low-shot learning with imprinted weights[END_REF], i.e., it is able to readily extent a pretrained neural network to classify samples from novel categories simply by adding new weight vectors in the final classification layer without requiring to perform any form of back-propagation to this end. At the same time, the proposed method was capable to overcome significant limitations of WI by being able to learn regularized representations that provide better generalization for classes which were not seen during the training and provide a straightforward way to directly handle novel categories with multimodal distributions. The proposed method was extensively evaluated on two image datasets, outperforming the regular WI approach. However, it is worth noting that the imprinting process is still quite behind traditional gradient descent-based learning approaches.

There are several interesting future research directions. First, even though the proposed clustering-based multi-modality detection led to adequate results, several most sophisticated methods can be used for detecting whether the distribution of a novel class is indeed multimodal. Also, to better spread the in-class samples in the volume of the hypersphere we employed a stochastic process that randomly distributes the samples in various radiuses. However, this process ignores the actual geometry of the data. More advanced methods that take into account the actual manifold of the data, e.g., by using data from a previous layer and then distilling the extracted information into the used model [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], [START_REF] Passalis | Learning deep representations with probabilistic knowledge transfer[END_REF], [START_REF] Passalis | Heterogeneous knowledge distillation using information flow modeling[END_REF], can be employed. Finally, the proposed hypersphere-based loss showed a great potential for improving regular training tasks as well. Therefore, it can be potentially successfully used even for regular classification tasks, further improving the accuracy of the corresponding models.

TABLE I EVALUATING

 I THE EFFECT OF THE HYPER-PARAMETERS ρ ON THE 2-SHOT LEARNING ACCURACY OF THE PROPOSED IMPRINTING METHOD ON BOTH THE NOVEL CATEGORIES ("NOVEL" SPLIT) AND THE COMBINED NOVEL AND TRAINING CATEGORIES ("ALL" SPLIT).

	Min. Distance ρ	Novel	All
	1	56.28	77.79
	2	56.24	77.42
	5	58.53	78.66
	10	61.02	80.06
	20	53.82	76.51
		TABLE II
	EVALUATING THE EFFECT OF THE HYPER-PARAMETERS r ON THE 2-SHOT
	LEARNING ACCURACY OF THE PROPOSED IMPRINTING METHOD ON BOTH
	THE NOVEL CATEGORIES ("NOVEL" SPLIT) AND THE COMBINED NOVEL
	AND TRAINING CATEGORIES ("ALL" SPLIT).
	Radius r	Novel	All
	0	61.02	80.06
	4	64.51	80.59
	5	69.57	81.55
	6	66.86	78.26

TABLE III MNIST

 III : EVALUATING THE ACCURACY OF IMPRINTING METHODS ON BOTH THE NOVEL CATEGORIES ("NOVEL" SPLIT) AND THE COMBINED NOVEL AND TRAINING CATEGORIES ("ALL" SPLIT).

	Method	Split	1-shot	2-shot	5-shot
	WI	Novel	52.44 ± 7.4	66.95 ± 4.3	72.51 ± 2.6
	Proposed-Novel	47.46 ± 9.9	59.98 ± 2.6	66.60 ± 3.7
	Proposed	Novel	57.93 ± 4.2 68.72 ± 5.1 75.20 ± 3.5
	WI	All	57.51 ± 5.8	58.35 ± 6.4	64.49 ± 3.2
	Proposed-All	73.27 ± 4.7	79.46 ± 1.5	82.88 ± 1.8
	Proposed	All	73.66 ± 3.9	79.09 ± 6.0	84.23 ± 1.3

ACKNOWLEDGMENTS

This project has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 871449 (OpenDR). This publication reflects the authors' views only. The European Commission is not responsible for any use that may be made of the information it contains.