Baptiste Barreau
email: baptiste.barreau@bnpparibas.com

Laurent Carlier
email: laurent.carlier@bnpparibas.com

Damien Challet
email: damien.challet@centralesupelec.fr

DEEP PREDICTION OF INVESTOR INTEREST: A SUPERVISED CLUSTERING APPROACH A PREPRINT

Keywords: investor activity prediction, deep learning, neural networks, mixture of experts, clustering

We propose a novel deep learning architecture suitable for the prediction of investor interest for a given asset in a given time frame. This architecture performs both investor clustering and modelling at the same time. We first verify its superior performance on a synthetic scenario inspired by real data and then apply it to two real-world databases, a publicly available dataset about the position of investors in Spanish stock market and proprietary data from BNP Paribas Corporate and Institutional Banking.

Introduction

Predicting investor activity is a challenging problem in Finance. The basic problem can be stated as follows: given many thousands of assets and many thousands of investors, predict which investors will be interested in buying/selling which assets in the next (short) time period. What makes this problem difficult is the large heterogeneity of both investors and assets, compounded by the non-stationary nature of markets and investors and the limited time over which predictions are relevant.

Ad-hoc methods are surprisingly efficient at clustering investors according to their trades in a single asset [START_REF] Tumminello | Identification of clusters of investors from their real trading activity in a financial market[END_REF]. In addition, clusters of investors determined for several assets separately have a substantial overlap [START_REF] Baltakys | Multilayer aggregation with statistical validation: Application to investor networks[END_REF], which shows that one may be able to cluster investors for more than a few assets at a time. The activity of a given cluster may systematically depend on the previous activity of some clusters, which can then be used to predict the investment flow of investors [START_REF] Challet | Statistically validated lead-lag networks and inventory prediction in the foreign exchange market[END_REF]. Here, we leverage deep learning to train a single neural network on all the investors and all the assets of a given market and give temporal predictions for each investor.

The heterogeneity of investors translates into a heterogeneity of investment strategies [START_REF] Tumminello | Identification of clusters of investors from their real trading activity in a financial market[END_REF][START_REF] Musciotto | Long-term ecology of investors in a financial market[END_REF]: for the same set of information, e.g., financial and past activity indicators, investors can take totally different actions. Take for instance the case of an asset whose price has just decreased: some investors will buy it because they have positive long-term price increase expectations and thus are happy to be able to buy this asset at a discount; reversely, some other investors will interpret the recent price decrease as indicative of the future trend or risk and refrain from buying it.

Formally, in our setting, a strategy f is a mapping from current information x to expression of interest to buy and/or sell a given asset, encoded by a categorical variable y: f : x → y. We call here D = {f k : x → y} k the set of all the investment strategies that an investor may follow. Unsupervised clustering methods suggest that the number of different strategies that describe investors' decisions is finite [START_REF] Musciotto | Long-term ecology of investors in a financial market[END_REF]. We therefore expect our dataset to have a finite number K of clusters of investors, each following a given investment strategy f k . Consequently, we expect D to be such that |D| = K, i.e. D = {f k : x → y} k=1,••• ,K . Alternatively, D can be thought of as the set of distinguishable strategies, which may be smaller than the total number of strategies and which may therefore be considered as an effective set of strategies. At any rate, a suitable algorithm to solve our problem therefore needs to be able to infer the set of investment strategies D.

A simple experiment shows how investors differ. We first transform BNP Paribas CIB bonds' Request for Quotation (RFQ) database, along with market data and categorical information related to the investors and bonds, into a dataset of custom-made, proprietary features describing the investors' interactions with all the bonds under consideration. This dataset is built so that each row can be mapped to a triplet (Investor, Financial Product, Date). This structure allows us, for a given investor and at a given date, to provide probabilities of interest in buying and/or selling a given financial product in a given timeframe. As the final decision following an RFQ is not always known, we consider the RFQ itself as the signal of interest in a product. Consequently, we consider a given day to be a positive event for a given investor and a given financial product when the investor actually signalled his interest in that product in a window of 5 business days around that day. The reason is twofold: first because bonds are by essence illiquid financial products and second because this increases the proportion of positive events.

At each date, negative events are randomly sampled in the (Investor, Financial Product) pairs that were observed as positive events in the past and that are not positive at this date. Using this dataset, we conduct an experiment to illustrate the non-universality of investors, i.e. the fact that investors have distinct investment strategies. The methodology of this experiment is reminiscent of the one used in [START_REF] Sirignano | Universal features of price formation in financial markets: Perspectives from deep learning[END_REF] to study the universality of equity limit order books.

Figure 1: Universality matrix of investors' strategies: the y-axis shows investors' sector used to train a gradient boosting model while the x-axis shows investors' sector on which predictions are made using the model indicated on y-axis. Scores are average precision, macro-averaged over classes and expressed in percentages.

We use a dataset constructed as described above with five months of bonds' RFQ data. We split this dataset into many subsets according to the investors' business sector, e.g. one of these subset contains investors coming from the Insurance sector only. We consider here only the sectors with a sufficient amount of data samples to train and test a model. The remaining sectors are grouped together under the Others flag. Note that this flag is mainly composed of Corporate sectors, such as car industry, media, technology, telecommunications. . . For each sector, some of the latest data is held out, and a gradient boosting model is trained on the remaining data. This model is then used for prediction on the held-out data of the model's underlying sector, and for all the other sectors as well. For comparison purposes, an aggregated model using all sectors at once is also trained and tested in the same way.

Because classes are unbalanced, we compute the average precision score of the obtained results, as advised by [START_REF] Davis | The relationship between Precision-Recall and ROC curves[END_REF], macro-averaged over all the classes, which yields the universality matrix shown in Fig. 1. The y-axis labels the sector used for training, and the x-axis is the section on which the predictions are made.

We observe that some sectors are inherently difficult to predict, even when calibrated on their data only -this is the case for Asset Managers of Private Banks and Pension Funds. On the contrary, some sectors seem to be relatively easy to predict, e.g. Broker Dealers and, to some extent, Central Banks. Overall, we note that there is always some degree of variability of the scores obtained by a given model -no universal model gives good predictions for all the sectors of activity. Thus follows the non-universality of clients. In addition, it is worth noting that the aggregated model Figure 2: Global architecture of an ExNet obtained better performance on some sectors than the models trained on these sectors' data only. As a consequence, a suitable grouping of sectors would improve predictions for some sectors. This observation is in agreement with the above K-investment strategies hypothesis.

Following on these hypotheses, this work leverages deep learning both to uncover the structure of similarity between investors, namely the K clusters, or strategies, and to make relevant predictions using each inferred clusters. The advantage of deep learning lies in the fact that it allows to solve both of these tasks at once, and thereby unveils the structure of investors that most closely corresponds to their trading behaviour in a self-consistent way.

Related work

This work finds its roots in mixture-of-experts research, which began with [START_REF] Jacobs | Adaptive mixtures of local experts[END_REF], from which we keep the basic elements which drive the structure presented in Section 3, and more particularly the gating and expert blocks. A rather exhaustive history of the research performed on this subject can be found in [START_REF] Yuksel | Twenty years of mixture of experts[END_REF].

The main inspiration for our work is [START_REF] Shazeer | Outrageously large neural networks: The sparsely-gated mixture-of-experts layer[END_REF], which, although falling within the conditional computation framework, presented the first adaptation of mixture of experts for deep learning models. We build on this work to come up with a novel structure designed to solve the particular problem presented in Section 1. As far as we know, the approach we propose is new. We use an additional loss term to improve learning of the strategies, reminiscent of the one introduced in Liu and Yao [1999].

Experts Network

We introduce here a new algorithm, inspired by [START_REF] Shazeer | Outrageously large neural networks: The sparsely-gated mixture-of-experts layer[END_REF], which we call the Experts Network (ExNet). The ExNet is purposely designed to be able to capture the hypotheses formulated in Section 1, i.e. to capture a finite, unknown number K of distinct investment strategies D = {f k : x → y} k=1,••• ,K .

Architecture of the network

The structure of an ExNet, illustrated in Fig. 2, comprises two main parts: a gating block and an experts block. Their purposes are the following:

-The gating block is an independent neural network whose role is to learn how to dispatch investors to n experts defined below. This block receives a distinct, categorical input, the gating input, corresponding to an encoding of the investors and such that the i-th row of the gating input corresponds to the investor indexing the i-th row of the experts input. Its output consists in a vector of size n which contains the probabilities that the input should be allocated to the n experts, computed by a softmax activation. -The experts block is made of n independent sub-networks, called experts. Each expert receives as input the same data, the experts input, corresponding to the features used to solve the classification or regression task at hand, e.g. in our case the features outlined in Section 1 -for a given row, the intensity of the investor's interest in the financial asset considered, the total number of RFQ done by the investor, the price and the volatility of the asset. . . As investors are dispatched to the experts through the gating block, each expert will learn a mapping f : x → y that most closely corresponds to the actions of its attributed investors. The role of an expert is therefore to retrieve a given f k , corresponding to one of the K underlying clusters of investors which we hypothesized.

The outputs of these two blocks are combined through f (x|a) = n i=1 p(i|a)f i (x), where a denotes the investor related to data sample x and p(i|a) is the probability that investor a is assigned to expert i. Our goal is that K experts learn to specialize to K clusters. As K is unknown, retrieving all clusters requires that n ≥ K, i.e. n should be 'large enough'. We will show below that the network ability to retrieve the K clusters is not impacted by high values of n; using large n values therefore ensures that the n ≥ K condition is respected and only impacts computational efficiency. The described architecture corresponds in fact to a meta-architecture. The architecture of the experts is still to be chosen, and indeed any kind of neural network could be used. For the sake of simplicity and computational ease, we use here rather small feed-forward neural networks for the experts, all with the same architecture, but one could easily use experts of different architectures to represent a more heterogeneous space of strategies.

Both blocks are trained simultaneously using gradient descent and backpropagation, with a loss corresponding to the task at hand, be it a regression or classification task, and computed using the final output of the network only, f (x|a). One of the most important features of this network lies in the fact that the two blocks do not receive the same input data. We saw previously that the gating block receives as input an encoding of the investors. As this input is not time-dependent, the gating block of the network can be used a posteriori to analyse how investors are dispatched to experts with a single pass of all investors' encodings through this block alone, thereby unveiling the underlying structure of investors interacting in the considered market.

For a given investor a, the gating block computes attribution probabilities of investor a to each expert

p(x|a) = Sof tmax (W experts * x),
where x is a trainable d-dimensional embedding of the investor a, W experts is a trainable n×d-dimensional matrix where the i-th row corresponds to the embedding of the corresponding expert, and we define Sof tmax(x) k = e x k / i e xi .

Disambiguation of investors' experts mapping

The ExNet architecture is similar to an ensemble of independent neural networks, where the weighted average is given by the gating block of the network. We empirically noticed that ExNets may assign equal weights to all experts for all investors without additional penalization. To avoid this behaviour, and thereby to help each investor follow a single expert, we introduce an additional loss term

L entropy = - 1 |B| i∈B n j=1 p j (x|a i ) log p j (x|a i ),
where B is the current batch of data considered, n is the number of experts, and p j (x|a i ) is the attribution of investor a i to the j-th expert. This loss term corresponds exactly to the entropy of the probability distribution over experts of a given investor. Minimising this loss term will therefore encourage distributions over experts to peak on one expert only.

Helping experts specialize

Without a suitable additional loss term, the network has a tendency to let a few experts learn the same investment strategy, which also leads to more ambiguous mapping from investors to experts. Thus, to help the network finding different investment strategies and to increase its discrimination power regarding investors, we add a specialization loss term, which involves cross-experts correlations, weighted accordingly to their relative attribution probabilities. It is written as:

L spec = n i=1 n j=1,j =i w i,j ρ i,j with w i,j = p i p j n i=1 n j=1,j =i p i p j if i = j, 0 else,
and

p i = a∈A p a i a∈A 1 p a i =0
.

Here, i, j ∈ {1, • • • , n}, ρ i,j is the batch-wise correlation between experts i and j outputs, averaged over the output dimension, and p i is the batch-wise mean attribution probability to expert i, with p a i the attribution probability of investor a to expert i, computed on the current batch of investors only. The intuition behind this weight is that we want to avoid correlation between experts that were confidently selected by investors, i.e. to make sure that the experts that matter do not replicate the same investment strategy. As the size of the investors clustering around a given expert should not matter in this weighing, we only account for the nonnegative probabilities for all the considered investors in these weights. In some experiments, it was found useful to rescale

L spec from [-1; 1] to [0; 1].
This additional loss term is reminiscent of [START_REF] Liu | Simultaneous training of negatively correlated neural networks in an ensemble[END_REF]. As a matter of fact, in ensembles of machine learning models, negatively correlated models are expected to perform better than positively correlated ones. This can also be expected from the experts of an ExNet, as negatively correlated experts better span the space of investment strategies. As the number of very distinct strategies grow, we can expect to find strategies that more closely match the ones the investors use in the considered market, or the basis functions on which investment strategies can be decomposed.

Uncovering structure from gating

Up to this point, we only discuss gating input related to investors. However, as seen above, being able to retrieve the structure of attribution of inputs to experts only requires to use categorical data as input to the gating part of the network after the training phase. We can therefore perform gating on whatever is thought to be suitable -for instance, it is reasonable to think that bonds investors have different investment strategies depending on the bonds' grades, or depending on the sector of activity of the bonds' issuers. Higher-level details about investors could also be considered, for instance because investment strategies may depend on factors such as the sector of activity of the investor, i.e. whether it is a hedge fund, a central bank or an asset manager, or the region of the investor. The investor dimension could even be totally forgotten, and the gating performed on asset related categories only.

Gating allows one to retrieve the underlying structure of interactions of a given category, or set of categories. One can therefore purposely set categories to study how they relate in the problem one wants to study. This may however impact performance of the model, as chosen categories do not necessarily have distinct decision rules.

Note also that the initialization of weights in the gating network has a major impact on the future performance of the algorithm. To find relevant clusters, i.e. clusters that are composed of unequivocally attributed categories and that correspond to the original clusters expected in the dataset, categories need to be able to explore many different clusters' configurations before the exploitation of one relevant configuration. To allow for this exploration, the gating block must be initialized so that all the expert weights are fairly evenly initially distributed. In our implementation, we therefore use a random normal initialization scheme for the d-dimensional embeddings of the categories and of the experts.

Limitations of the approach

Our approach allows us to treat well a known, fixed base of investors. However, it cannot easily deal with new investors, or, at a higher level, new categories as seen in Section 3.4, as embeddings for these new types of element would need to be trained from scratch. To cope with such situations, we therefore recommend to use sets of fixed categories to describe the evolving ones. For instance, instead of performing gating on investors directly, one can use investors' categories such as sector, region,. . . , that are already present in the dataset and on which we can train embeddings. Doing so improves the robustness of our approach to unseen categories. Note that this is reminiscent of one of the classic problems of recommender systems, known in the literature as the cold start problem.

Experiments

Before testing the ExNet architecture on real data, we first check its ability to recover a known strategy set, to attribute correctly traders to strategies, and finally to classify the interest of traders on synthetic data. We then show how our methodology compares with other algorithms on two different datasets: a dataset open-sourced1 as part of the experiments presented in Gutiérrez-Roig et al. [2019], and a BNP Paribas CIB dataset. Source code for the experiments on synthetic data and the open-source dataset is provided and can be found at https://github.com/BptBrr/deep_prediction.

Synthetic data

Generating the dataset

Taking a cue from BNP Paribas CIB bonds' RFQ database, we define three clusters of investors, each having a distinct investment strategy, which we label as 'high-activity', 'low-activity' and 'medium-activity'. Each cluster contains a different proportion of investors, and each trader within a cluster has the same activity frequency: the 'high-activity' cluster accounts for roughly 70% of the dataset samples, while containing roughly 10% of the total number of investors. The 'low-activity' cluster accounts for roughly 10% of the samples, while containing roughly 50% of the total number of investors. The 'medium-activity' cluster accounts for the remaining number of samples and investors. In all the clusters, we assume that investors are equally active.

We model the state of investors as a binary classification task, with a set of p features, denoted by X ∈ R p , and a binary output Y representing the fact that a client is interested or not in the considered asset. Investor a belonging to cluster c follows the decision rule given by Y a = (σ(w T a X) > U ) ∈ {0, 1}, where w a = w c + b a ∈ R p , w c ∈ R p being the cluster weights and b a ∼ N (0, α) ∈ R an investor-specific bias, X i ∼ N (0, 1) for i = 1, • • • , p, U is distributed according to the uniform distribution on [0, 1], and σ is the logistic function.

The experiment is conducted using a generated dataset of 100, 000 samples, 500 investors and p = 5 features. This dataset is split into train/validation/test sets, corresponding to 70/20/10% of the whole dataset. α is set to 0.5, and the cluster weights are taken as follows:

-High-activity cluster: w high = (5, 5, 0, 0, -5) -Low-activity cluster: w low = (-5, 0, 5, 0, 5) -Medium-activity cluster: w medium = (-5, 0, 5, 5, 0)

These weights are chosen so that the correlation between the low-and medium-activity clusters is positive, but both are negatively correlated with the the high-activity cluster. In this way, we build a structure of clusters, core decision rules and correlation patterns that is sufficiently challenging to demonstrate the usefulness of our approach.

Results

We examine performance of our proposed algorithm, ExNet, against a benchmark algorithm, LightGBM [START_REF] Ke | LightGBM: A highly efficient gradient boosting decision tree[END_REF]. LightGBM is a popular implementation of gradient boosting, as shown for example by the percentage of top Kaggle submissions that use it. This algorithm is fed with both the experts input of the ExNet and an encoding of the considered investors, used as a categorical feature in the LightGBM algorithm. For comparison purposes, experiments are also performed on a LightGBM model fed with experts input and an encoding of the investors' underlying clusters, i.e. whether the investor belongs to the high-, low-or medium-activity cluster, called LGBM-Cluster.

ExNets are trained using the cross-entropy loss, since the problem we want to solve is a classification one. The network is optimized using Nadam [START_REF] Dozat | Incorporating Nesterov momentum into Adam[END_REF], a variation of the Adam optimizer [START_REF] Kingma | A method for stochastic optimization[END_REF]] using Nesterov's Accelerated Gradient [START_REF] Nesterov | A method for solving the convex programming problem with convergence rate o(1/k 2 )[END_REF], reintroduced in the deep learning framework by [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF], and Lookahead [START_REF] Zhang | Lookahead optimizer: k steps forward, 1 step back[END_REF]. For comparison purposes, experiments are also performed on a multi-layer perceptron model fed with the experts inputs concatenated with a trainable embedding of the investors, called Embed-MLPthis model therefore differs from a one-expert ExNet in that this ExNet does not use an embedding of the investor to perform its predictions. All neural network models presented here used the rectified linear unit, ReLU(x) = max(x, 0), as activation function [START_REF] Nair | Rectified linear units improve restricted boltzmann machines[END_REF].

LightGBM, ExNet and Embed-MLP results are shown in Table 1. They were obtained using a combination of random search [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF] and manual fine-tuning. LightGBM-Cluster results used the hyperparameters found for LightGBM. These results correspond to the model which achieved best validation accuracy over all our tests. The LightGBM and LightGBM-Cluster shown had 64 leaves, a minimum of 32 samples per leaf, a maximum depth of 10, a learning rate of 0.005 and a subsample ratio of 25% with a frequency of 2. ExNet-Opt, the ExNet which achieved the best validation accuracy, used 16 experts with three hidden layers of sizes 64, 128 and 64, a dropout rate [START_REF] Srivastava | Dropout: A simple way to prevent neural networks from overfitting[END_REF] of 40%, loss weights λ spec = 7e -3 and λ entropy = 1e -3 , a batch size of 1024, and a learning rate of Table 1 contains the results for all the tested implementations. As the binary classification considered here is balanced, we use the accuracy as evaluation metric. This table reports results on train, validation and test splits of the dataset, and a view of the test results on the three different clusters generated. As the generation process provides us with the probabilities of positive events, it is also possible to compute metrics for a model that would output these probabilities, denoted here as perfect model, which sets the mark of what good predictive performance is in this experiment.

We see here that the LGBM implementation fails to completely retrieve the different clusters. LGBM focused on the high-activity cluster and mixed the two others, leading to poorer predictions for both of these clusters and here particularly for the medium-activity one. In comparison, LGBM-Cluster performed significantly better on the mediumand low-activity clusters. Embed-MLP better captured the structure of the problem, but appears to mix the mediumand low-activity clusters as well, albeit getting better predictive performance. ExNet-Opt, found with random search, captured well all clusters and obtained the best overall performances.

Moreover, the ExNet-n experiment shows how the algorithm behaves as n increases. ExNet-1 successfully captured the largest cluster in terms of samples, i.e. the high-activity one, partly ignoring the two others, and therefore obtained poor overall performance. ExNet-2 behaved as the LGBM experiment, retrieving the high-activity cluster and mixing the remaining two. ExNet-3 perfectly retrieved the three clusters, as expected. Even better, the same holds for ExNet-10 and ExNet-100: this is because the ExNet algorithm, thanks to the additional specialization loss, is not sensitive to the number of experts even if n K, as long as there are enough of them. Thus, when n ≥ K, the ExNet is able to retrieve the K initial clusters and to predict the interests of these clusters satisfactorily.

Further analysis of specialization

The previous results show that as long as n ≥ K, the ExNet algorithm is able to capture the investment strategies corresponding to the underlying investor clusters efficiently. One still needs to check that the attribution to experts is working well, i.e. that the investors are mapped to a single, unique expert. To this end, we retrieved from the gating block the attribution probabilities to the n experts of all the investors a posteriori. For comparison, we also analyse the investors' embeddings of Embed-MLP. The comparison of the final embeddings of ExNet-Opt and the ones trained in the Embed-MLP algorithm is shown in Fig. 3.

To visualize embeddings, we use here the UMAP algorithm [START_REF] Mcinnes | Umap: Uniform manifold approximation and projection for dimension reduction[END_REF], which is particularly relevant as it seeks to preserve the topological structure of the embeddings' data manifold in a lower-dimensional space, thus keeping vectors that are close in the original space close in the embedding space, and making inter-cluster distances meaningful in the two-dimensional plot. The two-dimensional map given by UMAP is therefore a helpful tool for understanding how investors relate to each other according to the deep learning method. In these plots, high-activity investors are shown in blue, low-activity investors in red and medium-activity investors in green. We can see in Fig. 3 that the Embed-MLP algorithm did not make a totally clear distinction between the low-and medium-activity clusters, contrarily to the ExNet which separated these two categories with the exception of a few low-activity investors mixed in the medium-activity cluster. The ExNet algorithm was therefore completely able to retrieve the original clusters.

The attribution probabilities to the different experts of ExNet-Opt are shown in Fig. 4. We see in this figure that the attribution structure of this ExNet instance is quite noisy, with three different behaviours clearly discernable. The first group of investors correspond to the low-activity cluster, the second group to the medium-activity cluster and the last one to the high-activity cluster. Attributions are here very noisy, and investors of a same cluster are not uniquely mapped to an expert. It is however possible to achieve a more satisfactory experts attribution, as one can see in Fig. 5 with the plots of ExNet-100. This comes from the fact that the ExNet-100 instance used a higher level of λ entropy than ExNet-Opt -at the expense of some performance, we are able to obtain far cleaner attributions to the experts. We see here on the left plot that all investors were attributed almost entirely to one expert only, with for each of the corresponding experts mean attribution probabilities p > 99%, even with an initial number of experts of 100, i.e. the n K setting. One can directly see on the UMAP plot three well-defined, monochromatic clusters. We can also see here that a low-activity investor got mixed in the medium-activity cluster, and that two separated low-activity clusters appear -these separated clusters originate from the fact that some low-and medium-activity investors were marginally attributed to the expert corresponding to the other cluster, as appearing on the experts distribution plot. The ExNet-100 therefore solved the problem that we originally defined, obtaining good predictive performance on the three original clusters and uniquely mapping investors to one expert only, thereby explicitly uncovering the initial structure of the investors, a feature that an algorithm such as Embed-MLP is unable to perform.

IBEX data

Constructing the dataset

This experiment uses a real-world, publicly available dataset published as part of Gutiérrez-Roig et al. [2019] (https: //zenodo.org/record/2573031) which contains data about a few hundred private investors trading 8 Spanish equities from the IBEX index, from January 2000 to October 2007. For a given stock and each day and each investor, the dataset gives the end-of-the-day position, the open, close, maximum and minimum prices of the stock as well as the traded volume.

We focus here on the stock of the Spanish telecommunication company Telefónica, TEF, as it is the stock with the largest number of trades. Using this data, we try to predict, at each date, whether an investor will be interested into buying TEF or not. An investor is considered to have an interest into buying TEF when ∆p a t = p a t -p a t-1 > 0, where p a t is the position of investor a at time t. We only consider here the buy interest as the sell interest of private investors can be driven by exogenous factors that cannot be modelled, such as a liquidity shortage of an investor, whereas the buy interest of a investor depends, to some extent, on market conditions. We thus face a binary classification problem which is highly unbalanced: on average, a buy event occurs with a frequency of 2.7%.

We consider a temporal split of our data in three parts: training data is taken from January 2000 to December 2005, validation data from January 2006 to December 2006 and test data from January 2007 to October 2007. We restrict our investor perimeter to investors that bought TEF more than 20 times during the training period. We build two kinds of features:

-Position features. Position is shifted such that at date t corresponds p t-1 , and is normalized for each investor using statistics computed on the training set. This normalized, shifted position is used as is as feature, along with moving averages of it with windows of 1 month, 3 months, 6 months and 1 year. -Technical analysis features. We compute all the features available in the ta package [START_REF]Technical Analysis Library using Pandas[END_REF], which are grouped under 5 categories: Volume, Volatility, Trend, Momentum and Others features. As most of these features use close price information, we shift them such that features at a date t only use information available up to t -1.

We are left with 308 rather active investors and 63 features.

Results

ExNet and LightGBM are both trained using a combination of random search [START_REF] Bergstra | Random search for hyper-parameter optimization[END_REF] and hand fine-tuning. Because of the class imbalance of the dataset, the ExNet is trained using the focal loss [START_REF] Lin | Focal loss for dense object detection[END_REF], an adaptive re-weighting of the cross-entropy loss. Other popular techniques to handle class imbalance involve undersampling the majority class and/or oversampling the minority one, such as SMOTE [START_REF] Chawla | Smote: synthetic minority over-sampling technique[END_REF]. The γ parameter of this loss is treated as an hyperparameter of the network, and is also randomly searched. We also used the baseline buy activity rate of each investor in the training period as a benchmark. The LightGBM shown in Table 2 used 16 leaves with a minimum of 128 samples per leaf, a maximum depth of 4, a learning rate of 0.002, a subsample ratio of 35% with a frequence of 5, a sampling of 85% of columns per tree, with a patience of 50 for a maximum number of trees of 5000. The ExNet shown used 4 experts with two hidden layers of size 32 and 32 with a dropout ratio of 50%, embeddings of size d = 32, an input dropout of 10%, λ spec = 7.7e -4 and λ entropy = 4.2e -2 , a focal loss of parameter γ = 2.5, a batch size of 1024, a learning rate of 7.8e -4 and was trained using Nadam and Lookahead, with an early stopping of patience 20. As can be seen on this table, both algorithms beat the historical baseline, and the ExNet achieved overall better test performance. While the precision of LightGBM is better in the training set, it is clearly inferior to that of ExNet in the validation set, a sign that ExNet is less prone to overfitting than LightGBM. Figure 6 gives a deeper view of the results obtained by the ExNet. Three distinct behaviours appear in the left plot. Some of the investors were entirely attributed to the blue expert, some investors used a combination of the blue expert and two others, and some used combinations of the light blue and red experts. These three clusters are remarkably spaced in the UMAP plot on the right. It therefore appears that the ExNet retrieved three distinct behavioural patterns from the investors interacting on the TEF stock, leading to an overall better performance than the LightGBM who was not able to capture them, as the experiments performed in Section 4.1 show.

Experts analysis

We saw in Section 4.2.2 that the ExNet algorithm retrieved three different clusters. Let us investigate in more details what these clusters correspond to. First, the typical trading frequency of the traders attributed to each of these three clusters are clearly different. The investors that were mainly attributed to the blue expert in Fig. 6, corresponding to the blue cluster on the right of the UMAP plot, can be understood as 'low-activity' investors, trading on average 2.1% of the time. The blue cluster on the left-hand side of the UMAP plot can be understood as medium-activity investors, buying on average 5.5% of the days; the red cluster on the left of the plot is made of high-activity investors (13.9%). The ExNet therefore retrieved particularly well three distinct behaviours, corresponding to three different activity patterns.

To get a better understanding of these three clusters, we can try to assess these clusters' sensitivity to the features used in the model. We use here permutation importance, a widespread method in machine learning, whose principle was described for a similar method in [START_REF] Breiman | Random forests[END_REF]. The idea is to replace a given feature by permutations of all its values in the inputs, and assess how the performance of the model evolves in that setting. Here, we applied this methodology to the six groups of features: we performed the shuffle a hundred times and averaged the corresponding performance variations. For each of the three clusters, we pick the investor who traded the most frequently, and apply permutation importance to characterize the behaviour of the cluster. Results are reported in Table 3.

Feature 3: Percentage of average precision variation when perturbing features of the group given in the first column for the three clusters appearing in Fig. 6, using permutation importance. Cluster 1 corresponds to the cluster of low-activity investors, cluster 2 to the medium-activity ones and cluster 3 to the high-activity ones.

We call in this table cluster 1 the low-activity one, cluster 2 the medium-activity one and cluster 3 the high-activity one. We see that the three groups have different sensibilities to the groups of features that we use in this model. While all clusters are particularly sensitive to position features, the respective sensitivity of groups to the other features vary: leaving aside cluster 2 that only looks sensitive to position, cluster 1 is also sensitive to volume, volatility and trend, whereas cluster 3 is also sensitive to volume and momentum. The clusters therefore not only encode the activity rate, but also the type of information that a strategy needs, and by extension the family of the strategies used by traders, which validate the intuition that underpins the ExNet algorithm.

BNPP CIB data

The previous experiments proved the ability of the network to retrieve the structure of investors with a finite set of fixed investment strategies, and the usefulness of our approach on a real-world dataset. We now give an overview of the results we obtain on the BNPP CIB bonds' RFQ dataset specified in Section 1 for the non-universality of clients study.

As a reminder, assets considered are corporate bonds. The data used ranges from early 2017 to the end of 2018 with temporal train/val/test splits, and is made of custom proprietary features using clients-related, assets-related and trades-related data. Our goal is, at a given day, to predict the interest of a given investor into buying and/or selling a given bond; each row of the dataset is therefore indexed by a triplet (Investor, Bond, Date). Targets are constructed as previously explained in Section 1. In the experiment conducted here, we consider 1422 different investors interacting around a total of more than 20000 distinct bonds.

The left-hand plot of Fig. 7 shows the distribution over experts for all the 1422 considered investors. We see three different patterns appearing : one which used the brown expert only, another one the green expert only and a composite one. These patterns lead to four clusters on the right-hand plot. In this plot, as previously done in the IBEX experiment, each point corresponds to an investor, whose color is determined by the expert to which she is mainly attributed, with colors matching the ones of the left-hand plot. We empirically remark that these clusters all have different activities: the larger brown cluster is twice more active than the green one, the two smaller clusters having in-between average activities. The ExNet therefore retrieved distinct behavioural patterns, confirmed by a global rescaled specialization loss below 0.5, hence negatively correlated experts.

Obtaining finer clusters could be achieved in multiple ways. A higher-level category could be used as gating input: instead of encoding investors directly, one could encode their sector of activity, in the fashion of the non-universality of clients experiment. With an encoding of the investors, running an ExNet on the investors of one of the retrieved clusters only would also lead to a finer clustering of the investors -a two-stage gating process could even directly lead to it, and will be part of further investigations on the ExNet algorithm. Note however that these maps (and ExNets) are built from measures of simultaneous distance, hence, do not exploit lead-lag relationships -how ExNets could be adapted to a temporal setting to retrieve lead-lag relationships will be worthy of future investigations as well.

Conclusion

We introduced a novel algorithm, ExNet, based on the financial intuition that in a given market, investors may act differently when exposed to the same signals, and cluster around a finite number of investment strategies. This algorithm is able to perform both prediction, be it regression or classification, and clustering at the same time. The fact that these operations are trained simultaneously leads to a clustering that most closely serves the prediction task, and a prediction that is improved by the clustering. Moreover, one can use this clustering a posteriori, independently, to gain knowledge as to how individual agents behave and interact with each other. To help the clustering process, we introduced two additional loss terms that penalize the correlation between the inferred investment strategies and the entropy of the investors' allocations to experts. Thanks to an experiment with simulated data, we proved the usefulness of our approach, and we discussed how the ExNet algorithm performs on an open-source dataset of Spanish stock market data and on data from BNP Paribas CIB. Further research on the subject will include how such architectures could be extended and staged, and how they could be adapted to retrieve lead-lag relationships in a given market.

On a final note, the ExNet architecture introduced in this article can be applied wherever one expects agents to use a finite number of decision patterns, e.g. in e-shopping or movie opinion databases [START_REF] Bennett | The Netflix Prize[END_REF].

Acknowledgements

This work was conducted under the French CIFRE PhD Program, in collaboration between the MICS Laboratory at CentraleSupélec and BNP Paribas CIB Global Markets. We thank Sarah Lemler, Frédéric Abergel and Julien Dinh for helpful discussions and feedback on early drafts of this work.

Figure 3 :

 3 Figure 3: UMAP visualization of investors embeddings for both Embed-MLP and ExNet algorithms. Colors on the right plot correspond to investors' original clusters: high-activity is shown in blue, medium-activity in green and low-activity in red.

Figure 4 :

 4 Figure 4: Distribution over experts of all investors for ExNet-Opt, obtained with random search. Each column shows the attribution probabilities of a given investor, where colors represent experts.

Figure 5 :

 5 Figure 5: Distribution over experts of all investors and UMAP visualization of investors embeddings for the ExNet-100 algorithm. Each column of the left plot shows the attribution probabilities of a given investor, where colors represent experts. Colors on the right plot correspond to investors' original clusters: high-activity is shown in blue, medium-activity in green and low-activity in red.

Figure 6 :

 6 Figure 6: Distribution over experts of all investors and UMAP visualization of investors embeddings for the ExNet algorithm on the TEF dataset. Each column of the left plot shows the attribution probabilities of a given investor, where colors represent experts -same colors are used on the right plot.

Figure 7 :

 7 Figure 7: Distribution over experts of all investors and UMAP visualization of investors embeddings for the ExNet algorithm on the BNPP CIB Bonds' RFQ dataset. Each column of the left plot shows the attribution probabilities of a given investor, where colors represent experts -same colors are used on the right plot.

Table 2 :

 2 Experimental results on IBEX data: average precision scores, expressed in percentage.

	Algorithm Train Val Test
	Historical 9.68 4.55 2.49
	LGBM	22.22 7.53 5.35
	ExNet-4	18.37 8.63 6.45

Data is available at https://zenodo.org/record/2573031


Chairi Youssef
email: chairi@univ-brest.fr

Abedrrabba Sarra
email: sarra.abedrrabba@arkane-tech.fr

Merlet Thomas
email: thomas.merlet@thalesgroup.com

Rozenn Allanic
email: allanic@univ-brest.fr

El Oualkadi
email: aeloualkadi@uae.ac.ma

Ahmed Labtic

Reklaoui Kamal
email: k.reklaoui@uae.ma

Quendo Cédric
email: quendo@univ-brest.fr

Le Gouguec
email: legouguec@univ-brest.fr

Ku-Band Lightweight Aluminium Waveguides Fabricated by Direct Metal Laser Sintering Process

Keywords: 3D metal printing, Additive manufacturing, Direct metal laser sintering, Lightweight, Waveguide. I

This paper attempts to establish a good base for weight-reduction technique of microwave devices using 3D printing techniques. Experimental studies to minimize the weight of radiofrequency waveguides printed using additive manufacturing techniques are presented. Two types of Kuband rectangular waveguides (perforated and full-metal) have been printed using direct metal laser sintering (DMLS) process. The insertion losses increased by ~1.2 dB/m after applying 1 mm grids which reduces the weight of Ku-band rectangular waveguide by ~17.5%. Comparison between these measurements and those of a waveguide manufactured by conventional method was used to determine process loss. The insertion losses of full-metal waveguide manufactured using DMLS process increase by ~0.7 dB/m compared to the conventional one. Finally, RF measurements of waveguides manufactured using DMLS process show a good agreement with the simulation results.

INTRODUCTION

Additive manufacturing (AM) technology, commonly known as 3D printing, is a production method of rising popularity in the academic research as well as in the manufacturing industry. AM techniques stay until now an opportunity for the majority of companies and at the same time a risk to manage. Regardless of the AM barriers, this new technology provides new freedom degrees for designers that are unmatched by other manufacturing methods, without the need of expensive tooling or developing thereof. This opens up for numerous opportunities especially in aeronautics such as testing ideas at low cost [START_REF] Zhu | A Lightweight 3-D Printed Dual-Band High-Gain Slotted Spherical Antenna[END_REF][START_REF] Haumant | Ultra-Wideband Transition From Coaxial Line to Two Parallel Lines Manufactured Using Additive Manufacturing Technology[END_REF], lightweight parts [START_REF] Abedrrabba | Subarray Optimization for Compactness Improvement of a 3D Printed Adaptive Phased Array Antenna Operating in Kuband[END_REF][START_REF] Li | Lightweight low-cost Ka-band 3-D printed slotted rectangular waveguide bandpass filters[END_REF], manufacturing of complex structures that previously would have been impossible [START_REF] Guo | An X-band lightweight 3-D printed slotted circular waveguide dual-mode bandpass filter[END_REF]. Using direct metal laser sintering (DMLS) process known recently by powder bed fusion process according to additive manufacturing technology standards (ASTM), the object is built layer by layer from a metallic, ceramic or composite powder bed. A laser source is responsible for the consolidation by local heating.

Waveguide components have been used in communication systems for many years because of their great advantages such as: low loss and high-power handling capabilities compared to planar devices [START_REF] Pozar | Microwave Engineering[END_REF].

By using AM techniques, there are three methods to minimize the weight of microwave devices. The first one introduces grids in electrically non-active areas so as not to affect radiofrequency (RF) function of the device i.e. areas with a low current line density [START_REF] Rojas-Nastrucci | Meshed rectangular waveguide for high power, low loss and reduced weight applications[END_REF]. The second method is by metal plating on plastic 3D printed structure to form an electrically conductive surface [START_REF] Li | Lightweight low-cost Ka-band 3-D printed slotted rectangular waveguide bandpass filters[END_REF][START_REF] Huang | Lightweight Perforated Waveguide Structure Realized by 3-D Printing for RF Applications[END_REF]. However, the third method uses a monolithic structure that allows RF functions to be nested considering design freedom provided by AM techniques which gives to the designers the ability to miniaturize the RF structures at design stage [START_REF] Peverini | Integration of an H-Plane Bend, a Twist, and a Filter in Ku/K-Band Through Additive Manufacturing[END_REF], [START_REF] Mouthaan | A 3D Printed Ku-band Waveguide Array with Integrated Waveguide Filter[END_REF]. In contrast to the second method, the third method is valid for all AM techniques. Otherwise, the impact of the position in printing chamber as well as the environment during printing on RF structures with small RF functions such as radiating slots or coupling iris are significant, because the limitation in the resolution of DMLS process [START_REF] Guennou-Martin | Design and manufacturing of a 3-D conformal slotted waveguide antenna array in Ku-band based on Direct Metal Laser Sintering[END_REF].

In this paper, an alternative solution offering an optimized orientation of the grids regarding the overall orientation of the RF devices during DMLS process is presented. This is experimentally demonstrated on 50 mm Ku-band rectangular waveguides. Also, two perforated waveguides with cubic grids have been printed as well as a full-metal waveguide. The RF performances of commercial waveguide have been compared, firstly, to simulation results in order to determine measurements errors except those due to air gaps, and, secondly, to the measurements of printed full-metal waveguide in order to determine the insertion losses due to DMLS process linked to both the surface roughness and dimensional accuracy. Afterwards, the measurement graphs of the full-metal waveguide have been compared to those of perforated one in order to see the impact of the grids on insertion losses. The mechanical strength is not discussed in this paper since the parts will not be mechanically stressed in the operating environment [START_REF] Manfredi | From Powders to Dense Metal Parts: Characterization of a Commercial AlSiMg Alloy Processed through Direct Metal Laser Sintering[END_REF].

II. WAVEGUIDE DESIGN AND FABRICATION

A. Design

In this paper, a full-metal waveguide and two perforated waveguides with different grids size (C1 = 1 mm and C2 = 1.5 mm) have been designed. The orientation of grids for perforated waveguides has been chosen in order to avoid printing problems during DMLS process. However, the grids respect the 45° rule for AM techniques. Taking into account this rule during the design process minimizes overhanging zones which avoid material collapsing of zones without printing supports. This proposed orientation of grids is more effective for complex RF structures with surfaces oriented vertically to the printing platform.

Overall, the sizes of grids were selected in order to keep enough material between two grids so that waveguides can hold during DMLS process.

B. Fabrication

The proposed metallic waveguides have been manufactured layer by layer from a metallic powder bed using ProX300 3D printer, under argon atmosphere. The printing material is an aluminum alloys AlSi7Mg0.6 with a layer thickness of 40 µm. It is a typical metallic alloy for applications that require a combination of good electrical conductivity and lightweight. The geometry of the proposed lightweight Ku-band waveguides is shown in Fig. 1. The wall thickness of waveguides is equal to 1 mm and all have a length of 50 mm. The cross-section dimensions of printed waveguides are standardized like commercial Ku-band waveguides (WR-62 according to electronic industries alliance standards) as well as dimensions of printed flanges. The grid size was chosen taking into account the accuracy of Prox DMP 300B machine from 3D Systems used for the fabrication of the waveguides. Typically, the accuracy of Prox DMP 300B is equal to ±0.1 mm [START_REF]3D Systems[END_REF].

The aluminum alloy conductivity is of 25.10 6 S/m. All printed waveguides have undergone a surface treatment by a pressure blasting machine to improve the surface quality. Both waveguides have been printed using the same design orientation and process parameters. Nevertheless, the position upon the printing platform and the environment of the printed waveguide (e.g., surrounding prototypes) are not the same. The waveguide with C2 = 1.5 mm grids blew up during the DMLS manufacturing process.

III. WAVEGUIDE SIMULATION AND MEASURMENT

A. Simulation results

In this study, the effects on RF performances of perforated waveguides were analyzed by using HFSS TM . The waveguides have been powered during simulations by two wave ports with a characteristic impedance adapted to the input impedance of the designed waveguide and have been meshed using an automatic adaptive meshing technique. A set of Ku-band (WR-62) rectangular waveguides (one solid waveguide and two perforated waveguides) are simulated. The grids are uniformly distributed over all the surfaces of perforated waveguides in order to avoid high concentration of current lines at specific area. The reflection coefficient and the insertion loss of fullmetal waveguide and perforated waveguides with different size grids (C1 = 1 mm and C2 = 1.5 mm) are shown in Fig. 2. Therefore, the insertion losses increase from ~0.26 dB/m for full-metal waveguide to ~0.31 dB/m for waveguide with 1.5 mm size grids. The difference in the insertion losses between the two perforated waveguides is still negligible compared to the insertion losses of full-metal waveguide. The reflection coefficients are lower than -40 dB among Ku-band for all simulated waveguides. According to the simulation results, adding perforations in order to reduce weight of RF devices is very promising.

B. Experimental results

The reflection coefficient and the insertion losses are measured using an Agilent N5230C network analyzer after applying a Ku-band standard TRL (Through-Reflect-Line) calibration set. The reflection coefficient and the insertion loss of measured 3D printed waveguides are shown in Fig. 3. The insertion losses of measured conventional waveguide manufactured using conventional methods are ~0.6 dB/m which shows that insertion losses are multiplied by 2 compared to the simulation. This increase is due to the inevitable measurement errors of misalignment of SMA connectors and waveguide flanges.

Afterwards, the insertion losses of printed full-metal waveguide manufactured by DMLS process are ~1.3 dB/m, which are two times greater than the insertion losses of conventional waveguide. Thus, we can understand that the DMLS process increases the insertion losses mainly because of surface roughness and skin depth at Ku-band which is very low compared to the surface roughness as well as the air gaps between SMA connectors and waveguide flanges during RF measurement. A change in the electrical conductivity across the printed waveguide because of process parameters during printing process and powder quality can also increase the insertion losses.

The measured insertion losses after applying grids of 1 mm (C1 in Fig. 1.) to a solid waveguide are 2.5 dB/m, which means an increase of insertion losses by ~1.2 dB/m. The results of perforated waveguide with 45° oriented grids show more losses than full-metal waveguide, and this is unavoidable, because a part of metal is taken compared to the solid waveguide. However, the total weight of waveguide is reduced from ~30 g to ~24.75 g, corresponding to a ~17.5% weight reduction. Otherwise, insertion losses remain low, but they can be multiplied by more than 2 between a perforated waveguide and a solid waveguide depending on the frequency. The reflection losses are lower than -30 dB across the Ku-band for all measured waveguides printed using DMLS process. Fig. 4. shows photographs of the conventional waveguide (WR-62) as well as the 3D printed waveguide that exploded during DMLS process attached to the printing plate and after electrical discharge machining technique which allows to separate the manufactured prototypes from the printing plate. Thus, we can also conclude that there is an upper limit for the size of grids that must be respected in order to avoid manufacturing defects according to the exploded waveguide with 1.5 mm grids size (C2 in Fig. 1.) shown in Fig. 4. (c). Furthermore, this limit must be much lower than the size of radiating slots in order to avoid additional radiation losses. However, the length of radiating slots is determined according to the half-wavelength at the operating frequency as in [START_REF] Zhu | A Lightweight 3-D Printed Dual-Band High-Gain Slotted Spherical Antenna[END_REF].

𝐿𝑒𝑛𝑔𝑡ℎ 𝑜𝑓 𝑠𝑙𝑜𝑡𝑠 ≅ 𝜆(𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑛𝑔 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦) 2 [START_REF] Zhu | A Lightweight 3-D Printed Dual-Band High-Gain Slotted Spherical Antenna[END_REF] IV. CONCLUSION This paper presents the design of lightweight and lowcost Ku-band rectangular waveguides. The effect of grids on waveguide electrical performances have been discussed. The insertion losses increase from ~1.3 dB/m to ~2.5 dB/m for a grid that reduces the weight by ~17.5%. However, adding grids to waveguides produces a decrease in the electrically conductive surfaces and that has to be taken into account if we want to apply grids to complex RF structures to avoid important insertion losses. Furthermore, this work attempts to indicate the attractive potential of AM techniques in decreasing the lead time of mass producing of waveguides and also some similar RF structures such us filters and slotted waveguide antennas.

Fig. 1 .

 1 Fig. 1. Structural illustration of Ku-band waveguides designed on HFSS TM . (a) Full-metal waveguide. (b) Perforated waveguide with C1=1 mm. (c) Shape of cubic grids for both perforated waveguides.

Fig. 2 .

 2 Fig. 2. Simulated S-parameters of the perforated waveguides and the fullmetal waveguide. (b) Insertion loss. (b) Reflection coefficient.

Fig. 3 .

 3 Fig. 3. Simulated and measured S-parameters of the perforated waveguide and full-metal waveguide as well as conventional waveguide. (a) Insertion loss. (b) Reflection coefficient.

Fig. 4 .

 4 Fig. 4. Photographs of conventional waveguide and 3D printed waveguide. (a) Conventional waveguide. (b) Orientation and support structure. (c) Exploded waveguide during DMLS process with grids size of C2 = 1.5 mm.

ACKNOWLEDGMENT

The authors would like to thanks Thales T3DM, Ltd. for manufacturing waveguides as well as Thales LAS France SAS, Ltd. This work is carried out as part of the CIFRE-France/Morocco program. The authors would like to thank also the TECHYP platform (the High Performance Computing Cluster of the Lab-STICC) thanks which the devices could be simulated. This publication is supported by the European Union through the European Regional Development Fund (ERDF), and by the Ministry of Higher Education and Research, Brittany Brest Métropole, through the CPER Project SOPHIE / STIC & Ondes.


Yu Liu
email: yu_liu@etu.u-bourgogne.fr

Fan Yang
email: fanyang@u-bourgogne.fr

Dominique Ginhac
email: dominique.ginhac@ubfc.fr

ACDnet: An Action Detection network for real-time edge computing based on flow-guided feature approximation and memory aggregation

Introduction

In past years, human action detection has been an active area of research driven by numerous applications: autonomous vehicles, video search engines, and human-computer interaction, etc. As it aims not only to recognize actions of interest in a video, but also to localize each of them, action detection poses more challenges when compared to video classification. The task becomes even more difficult in practical applications when detection is to be performed in an online setting and at real-time speed. For instance, time-critical scenarios such as autonomous driving demand instant detection in order for machines to react immediately. Other use cases which seek for mobile or largescale deployment, such as service robots and distributed unmanned surveillance, require detection or scene meta-data extraction at low-end edge devices. In general, edge devices (e.g., embedded systems) have limited computational power and are only compliant with resource-efficient detection algorithms.

Following the success of Convolutional Neural Network (CNN) in diverse computer vision tasks, modern action detectors are mainly based on CNN. In particular, fast object detectors have been widely adopted to spatially localize action instances at each frame [START_REF] Singh | Online realtime multiple spatiotemporal action localisation and prediction[END_REF], [START_REF] Zhao | Dance with flow: Two-in-one stream action detection[END_REF]). Naturally, effective temporal modeling plays an imperative role for identifying an action. To reason both spatial and temporal context, [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF] pioneered the two-stream CNN framework which aggregates spatial and temporal cues from separate networks and input modalities (RGB and optical flow). Such an approach has motivated many state-of-the-art methods in the field of action recognition and detection. Alternatively, 3D CNN [START_REF] Carreira | Quo vadis, action recognition? a new model and the kinetics dataset[END_REF]) which performs spatio-temporal feature learning on stacked frames has also been increasingly explored to tackle video analysis tasks.

Despite recent advances in action detection, existing methods are inherently sub-optimal in two aspects. First, consecutive video frames exhibit high appearance similarity. Extracting frame features without taking into account this inter-frame similarity introduces redundancy. Moreover, the increased system complexity associated with employing two-stream or 3D CNN models is not proportionally reflected in the detection accuracy. In contrast, the above inevitably raises computational requirements associated with motion extraction and 3D convolution operation, prohibiting practical deployment on edge devices.

This work focuses on action detection solutions more pertinent to the criteria of realistic applications. To address the aforementioned limitations, we first exploit the temporal coherence among nearby video frames to enhance detection efficiency. This is embodied by performing feature approximation at the majority of frames in a video, mitigating re-extraction of similar features from neighboring frames. Furthermore, we hypothesize that a less expensive framework can effectively extract meaningful temporal contexts. Here, we adopt a multiframe feature aggregation module, which recursively accumulates 2D spatial features over time to encapsulate long temporal cues. Such feature aggregation implicitly models temporal variations of actions and facilitates understanding degenerated frames with limited visual cues.

To the best of our knowledge, this is the first attempt applying feature approximation and aggregation techniques to achieve efficient action detection which can benefit resource-limited devices. To summarize, our contribution is three-fold:

• We propose an integrated detection framework, ACDnet, to address both detection efficiency and accuracy. It combines feature approximation and memory aggregation modules, leading to improvement in both aspects.

• Our generalized framework allows for smooth integration with state-of-the-art detectors. When incorporated with SSD (single shot detector), ACDnet could reason spatiotemporal context well over real-time, more appealing to resource-constrained devices.

• We conduct detailed studies in terms of accuracy, efficiency, robustness and qualitative analysis on public action datasets UCF-24 and JHMDB-21.

Related work

Recent advancements in action detection are largely led by building upon successful cases in object detection and action recognition. Here, we briefly review these relevant topics.

Object detection based on CNN methods can be grouped into two families. The two-stage approach such as Faster R-CNN by [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] and R-FCN by [START_REF] Dai | R-fcn: Object detection via region-based fully convolutional networks[END_REF] first extracts potential object regions from images, on which it performs object classification and bounding box regression on features corresponding to each proposed location. Such a sequential pipeline imposes a bottleneck to real-time inference. Alternatively, single-stage detectors such as YOLO proposed by [START_REF] Redmon | Yolo9000: better, faster, stronger[END_REF], or SSD in [START_REF] Liu | Ssd: Single shot multibox detector[END_REF], remove the intermediate region proposal, directly achieving bounding box regression and classification in a single forwardpass. Bypassing the intermediate bottleneck enables real-time detection at the cost of minor accuracy drop.

A number of research focuses on video object detection instead of the image domain. Popular approaches such as [START_REF] Han | Seq-nms for video object detection[END_REF] and [START_REF] Kang | T-cnn: Tubelets with convolutional neural networks for object detection from videos[END_REF] exploit videos' temporal consistency by associating detection boxes and scores from multiple frames. Similarly but on the feature level, [START_REF] Hetang | Impression network for video object detection[END_REF] and Zhu et al. (2017a) aggregate multiple frame features to enhance detection accuracy. On the other hand, [START_REF] Zhu | Deep feature flow for[END_REF] leverage the temporal redundancy among video frames to improve detection efficiency. Their framework propagates features from a sparse set of key frames to successive ones by motion to avoid re-extracting similar object features. In a similar spirit, [START_REF] Liu | Mobile video object detection with temporally-aware feature maps[END_REF] propagate frame-level information across frames using a recurrent-convolutional architecture.

Action recognition is typically treated as a classification task on trimmed videos [START_REF] Yao | A review of convolutional-neural-networkbased action recognition[END_REF]). In addition to spatial features, reasoning temporal information across multiple frames is also crucial. Among different temporal modeling techniques, the two-stream architecture in [START_REF] Simonyan | Two-stream convolutional networks for action recognition in videos[END_REF] demonstrates state-of-art performance. Its framework consists of two feed-forward pathways, with one CNN learning spatial features from RGB stream and the other one learning motion features from optical flow stream. The two streams are trained and run inference independently to aggregate complemented features [START_REF] Feichtenhofer | Convolutional two-stream network fusion for video action recognition[END_REF]. Even though such a framework can exploit existing 2D CNN backbones, fine-grained optical flow is expensive to extract. Thus, flow images are typically pre-computed, which do not conform to the online workflow demanded in real-world scenarios.

Recently, 3D CNNs have been increasingly explored (Carreira and Zisserman (2017), [START_REF] Li | Large-scale gesture recognition with a fusion of rgb-d data based on optical flow and the c3d model[END_REF]) along with the release of large-scale action dataset Kinetics. They utilize 3D kernels to jointly perform spatio-temporal feature learning from stacked RGB frames, achieving comparable and even superior modeling capability than two-stream CNN. However, these models inherently suffer from higher number of parameters and computational cost than their 2D counterparts, making their deployment on resource-constrained devices impractical.

Efficient spatio-temporal modeling. To alleviate the high computational cost associated with flow extraction, several studies seek alternative motion representations that are easier to compute. These include feature-level displacement [START_REF] Jiang | Stm: Spatiotemporal and motion encoding for action recognition[END_REF], Sun et al. (2018b)), or simply taking the RGB difference between adjacent frames [START_REF] Wang | Temporal segment networks: Towards good practices for deep action recognition[END_REF]). On the other hand, to reduce the complexity of 3D CNN, decoupled architectures such as P3D [START_REF] Qiu | Learning spatio-temporal representation with pseudo-3d residual networks[END_REF]) and R(2+1)D [START_REF] Tran | A closer look at spatiotemporal convolutions for action recognition[END_REF]) have been studied. Alternatively, TSM proposed by [START_REF] Lin | Tsm: Temporal shift module for efficient video understanding[END_REF] handles temporal convolution as channelshifting operators to fuse spatial features from different time steps. Their approach has demonstrated effectiveness on edge devices such as Jetson Nano and Galaxy Note8.

Spatio-temporal action detection simultaneously addresses action localization and classification in time and space. Leading approaches often leverage CNN object detectors as the core building block. The extension mainly consists of adopting the two-stream framework, fusing complementary detection results from both spatial and temporal stream to acquire frame-level detection, as demonstrated in [START_REF] Singh | Online realtime multiple spatiotemporal action localisation and prediction[END_REF]. For temporal localization, detection at each frame is then linked over time to construct action tubes [START_REF] Peng | Multi-region two-stream r-cnn for action detection[END_REF]). Beyond detection at the frame-level, [START_REF] Kalogeiton | Action tubelet detector for spatio-temporal action localization[END_REF] and [START_REF] Li | Actions as moving points[END_REF] adopt a clip-based approach, which exploits stacking multiple frame features to capture temporal cues on top of the two-stream architecture. In this case, actions are regressed and inferred directly on action cuboids.

Inspired by the latest adoption of 3D CNN in action recognition, more recent studies incorporate 3D CNN as the backbone [START_REF] Yang | Step: Spatio-temporal progressive learning for video action detection[END_REF], Sun et al. (2018a), [START_REF] Girdhar | Video action transformer network[END_REF], [START_REF] Gu | Ava: A video dataset of spatio-temporally localized atomic visual actions[END_REF], [START_REF] Wei | P3d-ctn: Pseudo-3d convolutional tube network for spatio-temporal action detection in videos[END_REF]). In addition, various ways of fusing spatial and temporal context have also been investigated. Besides aggregating at the detection level (e.g., union of detection results), others perform feature-level fusion. These include the use of 1 × 1 convolution [START_REF] Ali | Real-time end-to-end action detection with twostream networks[END_REF]), attention model [START_REF] Köpüklü | You only watch once: A unified cnn architecture for real-time spatiotemporal action localization[END_REF]) or conditional normalization [START_REF] Zhao | Dance with flow: Two-in-one stream action detection[END_REF]). Such approaches allow a part of CNN layers to adaptively learn from the fused features.

ACDnet

Our objective is to perform detection in an online manner for every incoming frame of a video. The proposed ACDnet which consists of the feature approximation and aggregation module, is summarized in Figure 1.

Feature approximation by motion guidance

Video content varies slowly over consecutive frames. This phenomenon is more so reflected in the corresponding CNN feature maps which capture high-level semantics. Intuitively, the shared appearances among neighboring frames can help to propagate essential information for a given task. The practice of propagation in [START_REF] Zhu | Deep feature flow for[END_REF] has established success to enhance object detection efficiency in videos, which motivates our feature approximation module.

Within the approximation scheme, the heavier feature extraction sub-network, N f eat , only operates on a sparse set of key frames during inference. The features of successive non-key frames are obtained by spatially transforming those from their preceding key frames via two-channel flow fields. The workflow can be summarized by the following equations. Let M i→k be the two-channel flow field capturing relative motion from the current frame I i to its previous key frame I k (horizontal and vertical direction). Then, feature approximation (also referred as feature propagation) is realized according to inverse warping:

F i = W(F k , M i→k ) (1)
where F k is the key frame feature, and F i is the newly warped feature corresponding to I i . W denotes the inverse warping operation to sample the correct key frame features and assign them to the warped ones. Inverse warping is necessary to ensure every location p at the warped feature can be projected back to a point p + ∆p at the key frame feature, where ∆p = M i→k (p). Concretely, the warping operation W is performed as:

f c i (p i ) = p k G(p k , p i + ∆p) f c k (p k ) (2)
In Equation 2, f c i and f c k denote the c th channel of feature F i and F k , respectively; G denotes the bilinear interpolation kernel. Every location p i in the warped feature map undergoes this warping scheme to sample features from key frames, independently for each feature channel c. The warping operation is much lighter compared to layers of convolution for feature extraction. Consequently, by applying feature approximation on a dense set of non-key frames, computation is greatly reduced.

Previous methods on action-based tasks typically acquire motion features from accurate optical flow using non-learningbased algorithms. However, computing flows in such a way imposes a bottleneck to real-time and online detection due to high consumption of time or requiring to pre-compute flow results. In contrast, ACDnet integrates a fast flow estimation subnetwork, N f low , to predict flow fields. In our case, optical flow serves to spatially transform CNN features; it does not need to capture fine-grained motion details and has the same height and width as the corresponding feature. Using such a learningbased flow estimator also allows it to be jointly trained with all other sub-networks specific to the task of action detection.

In detail, the flow sub-network take (I k , I i ) as input, and generates a pair of motion field and position-wise scale map. Given that H, W, and C denote height, width and channel of F k , then the flow field M i→k is of size H × W × 2 , and the scale map is H × W × C, whose dimension matches that of F k to be warped. After the inverse warping described by Equation 1, the warped feature F i is refined by multiplying the scale map in an elementwise way. Any F k and F i would be fed to the shared detection sub-network, N det , to obtain final detection. This workflow is illustrated in Figure 1 (a) and (b).

Memory feature aggregation

Propagating features across frames reduces the computation cost associated with feature extraction. However, since most features are now approximated and heavily dependent of the quality of the precedent key frame features, we adopt a memory aggregation module as inspired by [START_REF] Hetang | Impression network for video object detection[END_REF] to enhance the feature representation at key frames. Given incoming video frames, the core of memory aggregation is to reinforce features of a target frame by recursively incorporating supportive and discriminating context from the past. This allows implicit spatio-temporal modeling without explicitly extracting motion features. In addition, in cases when the current frame is deteriorated, an action can still be inferred with the supportive visual cues from memory. Figure 1 (c) gives an example when such memory aggregation could be useful.

Memory aggregation shares the same warping operation used for feature approximation. ACDnet takes a sparse and recursive approach to aggregate memory features only at key frames, due to similar appearances shared among nearby frames. Given two succeeding key frames I k1 and I k2 , where I k2 is the more recent one in time, memory aggregation follows Equation 3:

F k2 aggregated = w k1 ⊗ F k1 + w k2 ⊗ F k2 (3)
where F k1 = W(F k1 , M k2→k1 ), is the warped feature of I k1 to spatially align its position with that of I k2 . The position-wise weights w k1 and w k2 both have the same height and width as F k1 and F k2 . These weights are normalized and determine the importance of memory feature at each location p with respect to the target frame feature map (w k1 (p)+w k2 (p) = 1). All channels of a feature share the same spatial weights.

The weights w k1 and w k2 are adaptively calculated based on the similarity of memory and target features. We estimate feature similarity by first projecting them into an embedding space via a few convolution layers, and then computing the cosine similarity between the embedded features. Finally at the current key frame, the weighted sum of the memory and current features will be fed to the detection sub-network and passed along as the new memory.

Training procedure

ACDnet follows a three-frame training scheme, as depicted in Figure 2. From each training mini-batch, frame I i and two precedent video frames (I k and I mem ) are selected, whose features simulate key frame and memory features respectively. The offset between I i and I k is a random number from 0 to T k→i , and the offset between I k and I mem is fixed at T mem→k .

The feature maps F mem and F k are first extracted from I mem and I k respectively. Two sets of flow fields, namely, the relative motion between I k -I mem , and I i -I k are also estimated. The former flow is used to propagate F mem to F k to simulate the occurrence of memory feature aggregation following Equation 3. Finally, the fused feature is warped with the second flow (simulating feature approximation) following Equation 1, which will be the final feature map for N det . Under this training mode, only the groundtruth of I i is needed to determine losses, which are back-propagated to update all sub-networks.

Adaptation for multi-feature scale detector

Workflows of feature approximation and aggregation are generic for video-based tasks. ACDnet further employs SSD, an one-stage detector to fulfill the objective of high-speed action detection potentially for embedded vision systems. In particular, the SSD300 model is chosen due to its superior speed. In SSD, a set of auxiliary convolutional layers are progressively added after the base network (e.g., VGG16 in a standard SSD) to extract features at multiple scales. This creates multiple feature maps where the detector makes prediction for objects of various sizes. Consequently, adopting the described framework in SSD requires feature approximation and memory aggregation to be handled for features at all scales.

To enable multi-level feature approximation, we duplicate N f low 's flow prediction layer into several branches. The number of branches matches that of the feature maps; the branches' outputs are also progressively resized via average pooling according to sizes of SSD's feature maps. Then, each branch reconstructs a pair of flow field and scale map in accordance with the dimension of SSD feature (refer to Figure 3). To cope with multi-level feature approximation and aggregation, Equation 1 and 3 are also generalized to take place at each feature level independently. Note that the standard SSD300 applies detection at six feature scales. Nevertheless, we only use the first five of them, as the dimension of the last feature map becomes a 1D vector resulting from progressive resizing, which is not feasible for feature approximation governed by 2D spatial warping.

Experimental results

Experimental Setup

Dataset. Our proposed methods are evaluated on two popular action datasets: UCF-24 and JHMDB-21. The former one 2013) consists of 928 short videos divided into three splits, with 21 action categories in daily life. Each video is trimmed and has a single action instance. We report our experimental results on the average of three splits for this dataset.

Network architectures. ACDnet incorporates the following sub-networks: SSD300 (with VGG16 backbone), FlowNet [START_REF] Dosovitskiy | Flownet: Learning optical flow with convolutional networks[END_REF]) and feature embedding. Feature embedding contains five branches for measuring feature similarity at five different scales. Each embedding branch has a bottleneck design of three 1 × 1 convolution layers interleaved with ReLU non-linearity, where the number of filters corresponds to the number of channels at each feature level l: f eat l channel /2, f eat l channel /2 and f eat l channel × 2, respectively. FlowNet is modified to also generate five sets of flow fields and position-wise scale maps, each pair being used for warping and refining designated features. We initialize the weights of the first two branches of flow generation layers using FlowNet's pre-trained weights. Considering the later three flow outputs are spatially much smaller than that of the original FlowNet, we randomly initialize the weights of those branches. We apply different hyperparameters on the two datasets. UCF-24 is trained for 100K iterations; the learning rate is initialized as 0.0005 and reduced by a factor of 0.1 after the 80K th and 90K th iterations. Weights of VGG16's first two convolution blocks are frozen. For JHMDB-21, due to its smaller training and testing size, we observe that detection accuracy tends to fluctuate significantly between successive epochs. Hence, we empirically train this dataset for 20K iterations with learning rate initialized as 0.0004 and reduced by a factor of 0.5 after the 8K th and 16K th iteration. During its training, the first three convolution blocks of VGG16 are frozen. In addition, all layers of FlowNet until the flow generation layers (the five branches at the end of our modified model) are also frozen to further reduce the risk of overfitting.

All sub-networks are trained jointly (also evaluated) on an NVIDIA Quadro P6000 GPU using a training batch size of 8. For the rest of hyperparameters and data augmentation methods, we follow the same setup as the original SSD by [START_REF] Liu | Ssd: Single shot multibox detector[END_REF]. The weights of VGG16 and FlowNet are pre-trained using ImageNet and the Flying Chair dataset respectively.

Ablation study

Our proposed architecture has been evaluated in terms of accuracy, efficiency and robustness over several network configurations. The standard frame-level mean average precision (F-mAP) and frame-per-second (FPS) have been used as the evaluation metrics. Specifically, FPS is measured based on the complete detection pipeline, including data loading and model inference using batch size of 1.The Intersection-over-Union threshold is set to 0.5 throughout all experiments. For brevity, we refer to feature approximation and memory feature aggregation as FA and MA respectively when presenting their results.

Accuracy. F-mAP results of different configurations are reported in Table 1. From both datasets, we observe a decrease of accuracy when only feature approximation is included. However, the accuracy drop can be compensated by the addition of memory aggregation, which exceeds the accuracy of the standalone SSD. Figure 4 shows some examples of how the memory aggregation module benefits detection. Overall, we remark that aggregating multipleframe features over time, even in a sparse manner, improves models' abilities to more confidently discriminate among different actions.

We also examine the effect of separate branches of positionwise scale maps designed for refining visual features. Our results indicate that such refinement mildly improves detection accuracy. The scale maps serve as implicit attention maps which reinforce feature responses associated with moving actors (elaborated in Figure 5).

Even though similar result patterns can be seen from both datasets, the benefit of memory aggregation appears less prominent in JHMDB-21. This could result from the fact that each video clip in JHMDB-21 is much shorter (40 frames or less). As MA is performed sparsely at every 10 th frame, its impact is limited to 2-3 aggregation per clip. Furthermore, we observe that motions in several JHMDB-21 clips are relatively small. In these clips, key frames far apart still appear fairly identical, limiting additional visual cues to be propagated.

Efficiency is evaluated on UCF-24 by simultaneously inspecting accuracy, run time and number of parameters of various configurations. Here, we assume the use of scale map refinement if applicable. As shown in Table 2, ACDnet (SSD, FA, MA) outperforms the stand-alone SSD in both speed and accuracy. This suggests it is relevant to handle inter-frame redundancy, and that long-range memory fusion is effective for collecting more discriminating features. Regarding the number of required parameters, the increase in ACDnet (SSD, FA) from stand-alone SSD is associated with the addition of FlowNet, which can be replaced by much lighter architectures in the future. Likewise, the increase with the addition of MA module corresponds to the extra embedding layers for measuring feature similarity at various scales. In terms of run time, the speed drop with MA is incurred by the additional operations at key frames (except for the first one), where flow estimation, feature extraction, similarity measure and aggregation all take place.

To examine how our generic architecture performs on a different detection framework, we conduct the same experiments while incorporating ACDnet with R-FCN, a state-of-the-art two-stage detector. The run time improvement brought by feature approximation is more significant with R-FCN, due to it using a much deeper backbone for feature extraction. The number of additional parameter needed to carry out memory aggregation is less too for R-FCN, as it is designed to perform prediction on a single-scale feature (needing only one branch for the embedding and flow sub-networks). Overall, when taking into account run time, memory consumption and obtained accuracy together, our results still strongly favor the SSD-based ACDnet.

Robustness. Concerning the robustness of our models trained with a fixed duration T mem→k and T k→i (at 10), we evaluate their performances under various key frame intervals (k) during inference. Figure 6 displays F-mAP results on both datasets, with k ranging from 2 to 20. Both models (with and without MA) express an overall steady drop in accuracy on the two datasets as k increases. This is reasonable, as the ability of flow fields to correctly encode pixel correspondence diminishes under large motions. However, even when k is large, ACDnet with MA still retains decent accuracy which outperforms or is comparable with the best cases of the other configurations.

Run time is also inspected under the same setting, as shown in Figure 7. It can be observed that ACDnet (FA, MA) exceeds the speed of SSD starting around k = 8, while the FAonly model is consistently faster. Larger key frame intervals intuitively should lead to further speed boost, as higher ratio of features are approximated. Interestingly, we observe that this pattern is neatly presented when k ≤ 10. After that, the run time of the examined models begin to saturate. This phenomenon is associated with two factors. On the one hand, as key frame interval increases, the ratio between approximated and real features become less significant. On the other hand, larger key frame intervals introduce more motion which could compromise the quality of approximated features. This results in an increase of low-confident predictions, which take longer for SSD's non-maximum suppression to filter.

Comparison with state-of-the-art

We compare the complete ACDnet (with FA and MA) against state-of-the-art methods in Table 3. Since our proposed framework targets lightweight action inference for realistic deployment rather than solely obtaining superior accuracy, only topperforming works which take into account both accuracy and run time are considered for fair comparison. With this in mind, recent research such as the works of [START_REF] Wei | P3d-ctn: Pseudo-3d convolutional tube network for spatio-temporal action detection in videos[END_REF] and [START_REF] Gu | Ava: A video dataset of spatio-temporally localized atomic visual actions[END_REF] demonstrate impressive accuracy but are excluded from our comparison, as they utilize heavier configurations and omit speed analysis. Alongside performances, comprehensive summary of each method's backbone is also reported for clearer comparison. It should be noted that methods such as ACT, STEP and MOC perform clip-based detection. In other words, they take clips of multiple RGB frames with the support of stacked flow images at once (e.g., five flow for each RGB), predicting action tubelets spanning these RGB frames. In contrast, methods such as YOWO gather supportive contextual cues from multiple frames to augment the target one. These particular attributes are summarized Table 3 column 4. As shown in Table 3, ACDnet outperforms the others in terms of run time. This is ascribed to the feature approximation module and our less complex architecture overall. The other methods either adopt two-stream or 3D CNN architectures to capture complemented spatial and temporal features, which raise computation. In addition, preparation of accurate flow using Brox [START_REF] Brox | High accuracy optical flow estimation based on a theory for warping[END_REF]) or FlowNet2 [START_REF] Ilg | Flownet 2.0: Evolution of optical flow estimation with deep networks[END_REF]) is particularly expensive; as a result, all methods employing a second flow stream do not take into account optical flow acquisition when measuring run time (except ROAD using a fast flow estimator by [START_REF] Kroeger | Fast optical flow using dense inverse search[END_REF]). In contrast, flow generation in ACDnet is fast and can be carried out in an online setting as it does not aim to encode fine-grained motion features.

In terms of accuracy, ACDnet retains competitive performance on UCF-24. On the other hand, its performance on JHMDB-21 is less impressive compared to the other methods. As opposed to UCF-24, whose classes of sports activities are visually more distinctive, we observe that JHMDB-21 contains more classes that share similar visual context (for example, Sit v.s. Stand, and Run v.s. Walk, etc.). Figure 8 demonstrates 3, ACDnet applies detection on frames far fewer than other methods, which limits its ability to model detailed variations of visual cues over time. In addition, JHMDB-21 consists of short clips for which sparse memory aggregation can only take place minimally. The above factors result in ACDnet's less satisfactory accuracy on JHMDB-21. This visual ambiguity could generally be mitigated when examining more frames at once, as demonstrated by all clip-based methods.

Similarly, 3D-CNN-based method such as YOWO also proves effective to learn spatio-temporal features when taking 16 consecutive frames. However, such an approach inevitably raises the computation time; not only from model inference, but also data loading, which is excluded in their reported speed performance. Furthermore, ACDnet achieves comparable accuracy when YOWO employs lighter 3D CNN variants, implying the necessity of deeper models to effectively reason temporal context. In conclusion, our experimental results verify ACDnet's competitive capability to efficiently infer actions with strong visual cues, but its sparse spatio-temporal modeling scheme does not capture temporal cues as effectively as the more expensive two-stream / 3D CNN. On the other hand, ACDnet is compact and can achieve inference speed far beyond real-time requirement. This not only permits more seamless deployment potentially on resource-constrained devices, but also can afford to further adopt a clip-based framework or lightweight 3D CNN to improve its accuracy.

Conclusions and Future works

In this paper, we present ACDnet, a compact action detection network with real-time capability. By exploiting temporal coherence among video frames, it utilizes feature approximation on frames with similar visual appearances, which significantly improves detection efficiency. Additionally, a memory aggregation module is introduced to fuse multi-frame features, enhancing detection stability and accuracy. The combination of the two modules and SSD detector implicitly reasons temporal context in an inexpensive manner. ACDnet demonstrates real-time detection (up to 75 FPS) on public benchmarks while retaining decent accuracy against other best performers at far less complex settings, making it more appealing to edge device deployment in practical applications. Our future works include further investigation in cost-effective architectures for Table 3. State-of-the-art comparison. *For any key frame, ACDnet (FA, MA) fuses the accumulated key frame feature from the past with the current one (considered 2RGB implicitly). For any non-key frame, its feature is approximated based on the preceding key frame feature (considered 1RGB).

Method

Fig. 1 .

 1 Fig. 1. Illustration of ACDnet inference pipeline. (a) At the initial frame, features are obtained from the feature extraction sub-network (N f eat ). (b) For non-key frames (dense), the flow sub-network (N f low ) estimates a pair of flow field and position-wise scale map between the non-key frame and its preceding key frame. The resulted flow field is used to propagate appearance feature, which is then refined by the scale map via element-wise multiplication. (c)At key frames (sparse), new features are extracted. They are then aggregated with those from the past key frames (memory features) via N f low and the aggregation sub-network (N aggr ). The fused features will be used for detection (N det ) and also passed along as the updated memory.

Fig. 2 .

 2 Fig. 2. Training procedure. Each mini-batch consists of three frames (I mem , I k , and I i ) and the groudtruth of I i .

Fig. 3 .

 3 Fig. 3. Flow estimation sub-network adapted for multi-scale feature approximation and aggregation. The depicted design corresponds to the architecture of SSD300 and FlowNet.

  Training. Images are resized to 300 × 300 for training and inference. Training is conducted by stochastic gradient descent. To address data imbalance among different actions, from each training video clip of UCF-24, 15 frames spanning the whole video are evenly sampled as the training set. Since video clips of JHMDB-21 are generally short (≤ 40), we evenly sample 10 frames from each clip for training. Specifically, both T mem→k and T k→i are set to 10 during training. These chosen values correspond to the key frame interval used during inference, which is also fixed at 10 in our experiments unless specified.

Fig. 4 .

 4 Fig. 4. Examples where ACDnet (FA, MA) improves the baseline SSD.Green / Red boxes correspond to correct / incorrect detection respectively.

Fig. 5 .

 5 Fig. 5. Position-wise scale maps produced by our modified FlowNet. The scale map (bottom row) only reinforces activation (top row) associated with the actor by up-scaling, without altering activation in other feature regions.

Fig. 6 .

 6 Fig. 6. F-mAP under varied key frame intervals.

Fig. 7 .

 7 Fig. 7. FPS under varied key frame intervals.

Fig. 8 .

 8 Fig. 8. Examples of false detection in JHMDB-21. (a) Correct action: Jump. (b) Correct action: Sit. (c). Correct action: Stand; ACDnet incorrectly predicts two actions (Stand and Run).

Table 1 .

 1 F-mAP results for different configurations.

	ACDnet
	SSD
	FA
	Scale map
	MA
	F-mAP
	UCF-24	67.32 65.84 67.23 68.06 70.92
	JHMDB-21 47.90 46.65 46.69 49.37 49.53

Table 2 .

 2 Performance of different configurations on UCF-24.

	F-mAP FPS # params

  For a fully integrated and resource-efficient vision system, lightweight alternatives of the current sub-networks will be explored, and we will precisely customize candidate solutions for embedding them on edge devices such as NVIDIA XavierGPU. video recognition, in: IEEE CVPR, pp. 2349-2358.

		+2-stream Flow	+3D CNN	#frames:#det.	UCF24 JHMDB21 FPS
	ROAD, Singh et al. (2017)	Kroeger		(1RGB+1FL):1	65.66	-	28
	ACT, Kalogeiton et al. (2017)	Brox		(6RGB+30FL):6	69.5	65.7	25
	TS-YOLO, Ali and Taylor (2018)	FlowNet2-SD		(1RGB+1FL):1	71.67	-	25
	STEP, Yang et al. (2019)	Brox	3× 3D conv	(6RGB+30FL):6	75	-	21
	YOWO(a), Köpüklü et al. (2019)		3D-ResNet101	16RGB:1	80.4	74.4	34
	YOWO(b)		3DShuffleNetV2	16RGB:1	71.4	55.3	-
	YOWO(c)		3DMobileNetV2	16RGB:1	66.6	52.5	-
	MOC, Li et al. (2020)	Brox		(7RGB+35FL):7	78	70.8	25
	ACDnet (FA, MA)			2(1) * RGB:1	70.92	49.53	75
	spatio-temporal modeling and performing temporal localiza-
	tion.

Acknowledgments

This work was supported by the H2020 ITN project ACHIEVE (H2020-MSCA-ITN-2017: agreement no. 765866).

//github.com/dginhac