Julien Girard
email: julien.girard2@cea.fr

Youssef Tamaazousti
email: youssef.tamaazousti@cea.fr

Hervé Le Borgne
email: herve.le-borgne@cea.fr

Céline Hudelot
email: celine.hudelot@centralesupelec.fr

Learning finer-class networks for universal representations

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The state-of-art performances in visual recognition obtained by Convolutional Neural Networks (CNNs) are subject to the availability of a large set of annotated training data to learn the model. Since it is rarely the case for many practical tasks of interest (target-tasks), one usually adopts a transfer-learning approach [START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF][START_REF] Popescu | Scalable domain adaptation of convolutional neural networks[END_REF][START_REF] Sharif Razavian | Cnn features off-the-shelf: an astounding baseline for recognition[END_REF] which relies on a CNN pre-trained on a source task with sufficient annotated data (often ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]) then further truncated to provide the representations of the samples of target-task. Then, even with few annotated data, this last can usually be learned with a linear classifier. Such approaches raise the question of the similarity of the source-task on which the representation has been learned and the target-task on which it is used. Although this similarity is not easy to formalize, one has the intuition that the closer the both tasks the better the representation will be adapted to the target-task. This consideration leads to several methods that tend to obtain more universal representations [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF][START_REF] Conneau | Supervised learning of universal sentence representations from natural language inference data[END_REF][START_REF] Kokkinos | Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory[END_REF][START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF][START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF], that is to say that are more adapted to a large set of diverse target-tasks, in a transfer-learning scenario. c 2018. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.

The general idea of these methods is to diversify the classification problem of the sourcetask in order to obtain more features, able to adequately represent new target-datasets, from more domains, in a larger context. All these approaches vary the problem by creating new categories having an existing label. However, most of them studied the effect of adding categories extracted from ImageNet, either generic categories [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Mettes | The imagenet shuffle: Reorganized pre-training for video event detection[END_REF][START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF] or specific ones [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF][START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF][START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF][START_REF] Zhou | Learning deep features for scene recognition using places database[END_REF], that are at the bottom of a hierarchy such as ImageNet, except [START_REF] Joulin | Learning visual features from large weakly supervised data[END_REF][START_REF] Vo | Effective training of convolutional networks using noisy web images[END_REF][START_REF] Phong | Harnessing noisy web images for deep representation[END_REF] that use web annotations with noisy labels. In general, the usage of specific categories tends to provide better performances than generic ones [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF][START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], although combining them can significantly boost the universalizing capacity of the CNN [START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]. Yet, even for the most specific categories, it is plausible that it exists a variety of semantics within the class that is not explored (e.g., one could imagine to split the object-class according to the different poses of the object). Clearly, the limiting point is the availability of such finest annotation (e.g., poses, contexts, attributes) for existing specific classes.

In this article, we argue that exploring finer classes than the most specific existing ones, can significantly increase the diversity of the problem, therefore improve the universality of the representation learned in the internal layers of the CNN. The main difficulty is the lack of annotation below the most specific levels. We propose to rely on unsupervised learning (clustering) to determine these finer categories within each specific category. Our contribution is three-fold. First, we show that the use of finer categories rather than the most specific ones to learn CNNs, improves the universality of the resulting representation, even when the finer classes are determined randomly within each specific class. Second, the usage of a K-means based approach leads to slightly better results although the resulting clusters are strongly imbalanced. To fix this, our core contribution splits and merges the specific categories to automatically determine better balanced finer classes, leading to better results. Last, we show that CNNs learned with our approach provide a better complementary to standard CNN representations than those learned on generic categories.

Let note that if the target-task has enough data, the representation can be adapted to the target-task by fine-tuning. This is nevertheless out of the scope of this work, because it is a complementary process to the transfer-learning in itself, that will always improve the performances, and especially, because fine-tuning modifies the representations, which leads to a bias that hides the real ability of a universalizing method [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF]. Hence, in this paper, we are only interested into studying the universality of the representations, independently of many possible refinements of a full adaptation method on each target-task.

Previous works [START_REF] Chen | S-cnn: Subcategory-aware convolutional networks for object detection[END_REF][START_REF] Dong | Subcategory-aware object classification[END_REF][START_REF] Dong | Looking inside category: subcategory-aware object recognition[END_REF][START_REF] Xiang | Subcategory-aware convolutional neural networks for object proposals and detection[END_REF] exploited sub-categories in the context of visual recognition. In [START_REF] Dong | Subcategory-aware object classification[END_REF][START_REF] Dong | Looking inside category: subcategory-aware object recognition[END_REF], an object instance affinity graph is computed from intra-class similarities and inter-class ambiguities then the visual subcategories are detected by the graph shift algorithm. The process is nevertheless quite computationally demanding and applied to object detection on small target datasets only. In [START_REF] Chen | S-cnn: Subcategory-aware convolutional networks for object detection[END_REF][START_REF] Xiang | Subcategory-aware convolutional neural networks for object proposals and detection[END_REF], subcategories that are learned from extrapolated feature maps and fine-tuned on a target-dataset, are used within a CNN to improve region proposal for object detection. To the best of our knowledge, our paper is the first to propose the usage of subcategories determined by unsupervised learning on a source-task, in order to improve universality of representations. More related to universality, [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF][START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF][START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] added annotated data from more domains as well as domain-specific neurons to an initial set of domain-agnostic ones. Contrary to them, our method only modifies the source-problem at zero cost of annotation. Our work is closer to the approach of [START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF] that proposes to relabel specific categories into generic ones (that match the upper categorical-levels), to learn an additive CNN with the same architecture. Nevertheless, our approach is interested into the "opposite way", that is, creating finer classes than those at the bottom of a hierarchy (ImageNet), for which no annotation exists and thus their method can not be applied. We evaluated our proposal on the problem of universality, that is, in a transfer-learning scheme using multiple target-tasks (i.e., ten classification benchmarks from multiple domains, including actions, food, scenes, birds, aircrafts, etc.). In particular, in comparable settings (using ILSVRC as source-task and two architectures, AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and DarkNet [START_REF] Redmon | Yolo9000: Better, faster, stronger[END_REF]), we showed that our method outperforms state-of-the-art ones.

Proposed Method

We propose a new universalizing method that consists in training a network on a set of categories that are finer than those of the finest-level of a hierarchy (e.g., ImageNet hierarchy or any set of categories). In Sec. 2.1, we start by describing its general principle as well as two baselines that splits them either randomly or by clustering their features. With such baseline, the number of finer classes must be a priori fixed, thus we propose a "bottom-up clusteringbased merging" approach that determines a better splitting automatically (Sec. 2.2). Furthermore, we propose to combine the features learned on the specific categories and those learned on the finer ones to get an even more universal representation (Sec. 2.3).

FiNet: Network Trained on Finer-Classes

The leaf nodes of the ImageNet hierarchy represents the finest or most specific categories that are annotated. More generally, this is the case for the set of categories of any classification dataset. To go towards our goal of automatically obtaining finer categories (without annotations) from the finest ones, a baseline approach consists in using a random partitioning of the specific categories or a simple clustering-based approach of their image-features. The first baseline, randomly assigns every image of a specific category to one of K i clusters. The second one, first, learns a CNN (noted SpeNet) on the specific categories, uses one of its layer as features-extractor for every image, then determines K i clusters using K-means on these vectors. A more sophisticated way is our final method that is presented in Sec. 2.2. Note that, in all cases, the splitting is performed on specific categories, that already contains quite similar samples/vectors. Once the finer classes obtained, we train another network (denoted FiNet) on the same images used to train SpeNet, but labeled among the obtained finer-classes. The whole set of finer-classes forms the new finest-level of the hierarchy. Our general principle is illustrated in Fig. 1 and presented more formally below.

Let us consider a semantic hierarchy with hyponymy relations, that is to say a set of categories organized according to "is-a" relations (e.g., ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF] hierarchy). This hierarchy denoted H = (C , E) is a directed acyclic graph of levels of nodes, with C being all the nodes and E the set of directed edges between the nodes. Each node c i ∈ C corresponds to the i-th category at level in H and n is the number of categories at level . A hierarchy-edge (c i , c +1 j ) ∈ E indicates that class c i subsumes class c +1 j . Let us also consider an initial dataset D N containing a set of N images labeled among the categories at level and let us denote N i the number of images labeled among the i-th category at level . Note that, N = ∑

n l i=1 N i . Each image I j i ∈ D N of the dataset, is associated to a given category c j for j ∈ [n ] 1 . Let us denote X ,L
i ∈ R d the representation of an image I i extracted from layer L of the network trained on D N (i.e., SpeNet). Let also

X i = {X ,L j } N i
j=1 being the set of features extracted from all the images belonging to the category c i .

In order to construct the K i finer-categories {c +1 j } n +1 j=1 of each category of the previous level c i , we apply a clustering algorithm (e.g., K-means, MeanShift or BUCBAM presented in Sec. 2.2) on X i (that builds K i centroids) where the feature vector X ,L j of each image I j ( j ∈ [N i ]) is assigned to the nearest centroid (hard-coding [START_REF] Liu | In defense of soft-assignment coding[END_REF]), which forms the K i finer classes. This process is applied for all i ∈ [n ] which gives the n +1 = K i × n finer classes, that forms the nodes of the finer level + 1. This latter results in a new dataset D +1

N , for which each image

I j i ∈ D +1 N is associated to a given category c +1 j for j ∈ [n +1 ].
Note that by construction, every c i subsumes all its finer-categories {c +1 j } K i j=1 , thus we have

(c i , c +1 j ) (i, j)∈[n ]×[K i ] .
The whole process results in a new hierarchical level + 1 that forms the new hierarchy

H +1 = (C +1 , E ) with C +1 = C ∪ {c +1 j, j∈[K i ] } i∈[n ]
. E corresponds to the union of E and the edges that connect each category c i to its K i finer ones c +1 j . It is important to point out that, depending on the clustering algorithm, K i will depend on c i or be the same for all categories. This is discussed in the next section.

The new dataset D +1 N is used to train (softmax cross-entropy loss minimized by SGD) the FiNet network, which has n +1 neurons on its last layer. FiNet is then used as featuresextractors for the images I i of the target-tasks:

X +1,L i = Φ +1 L (I i ).

Bottom-Up Clustering-Based Merging

We empirically observed (see Fig. 3) that clustering approaches with a fixed K i for each category (e.g., Kmeans) usually leads to a FiNet that gives better universality results than approaches that adapt K i to each category (e.g., Affinity-Propagation). Indeed, this latter tends to provide a set of finer classes with many clusters containing few images and a couple of clusters containing a large number of images, leading to an undesirable imbalanced dataset that penalizes the network training. Even if the use of fixed-K i clustering methods leads to more balanced data, it remains sub-optimal since it sets the same amount of clusters for all specific categories (∀i ∈ [n ], K i = K), while this may depend on the content of each category. Furthermore, in fixed-K i clustering methods, the K value is an hyper-parameter that is cross-validated on the target-tasks, which are not accessible during the learning on the source-task, in the context of universality [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]. Hence, the cross-validation of K should be performed only with the source-task, which is not trivial (optimal K on the source-task not necessary optimal one for the target-tasks). To overcome the drawbacks of fixed and adapted-K i clustering methods, we proposed an hybrid one called "Bottom-Up Clustering-BAsed Merging" (BUCBAM). It roughly starts with the clusters obtained by a fixed-K i clustering and automatically sets the amount of clusters for each specific category by enforcing a more balanced resulting set of finer-classes. Specifically, it consists in three main steps (illustrated on the left of Fig. 2): (i) splitting the specific categories into K clusters (with a large K);

(ii) attaching small clusters to the closest bigger ones (to avoid imbalanced data); and (iii) merging the most similar ones, with respect to a proposed similarity criteria. Formally, BUCBAM starts with a large amount of K finer-classes per category c j with j ∈ [n ] and K being the same for all c j . Let us assume we have K(∈ N * ) finer-categories {c +1 i } i∈[K j ] obtained from the category c j of the previous level through a fixed-Ki clustering method. Let denote

X +1 i = {X ,L i } i∈[N +1 i
] the whole set of features extracted from the images of a given category c +1 i through the SpeNet Φ L . Note that, N +1 i corresponds to the amount of images in each c +1 i2 . The goal of BUCBAM is to get an amount of clusters K j depending on the images of each category c j . To do so, it first prunes out the small clusters (i.e., all the c i such that ∀i ∈ [K j ], Card(c i ) < S ∈ N * , with S N i ), by re-assigning their samples I k,k∈[N i ] (that were assigned to c i ) to the category of the closest feature vector

X i l,l∈[N i ] = N (X m k,k∈[N m ]
), with m = i and N (•) being a function that provides the closest vector (e.g., k-NN algorithm with Euclidean distance) in the set of features {X j } belonging to the other and large clusters (i.e., all c m with Card(c m ) S). Pruning small clusters for all categories c j , results in a set of K P j finer-classes {c i } i∈K P j per class c j . The last step of BUCBAM is to merge the similar clusters. To do that, a classifier Ψ i is trained for each cluster c i -using features of c i samples as positives and same amount of samples from a diverse class c d as negatives -and evaluated on the images of all other clusters. The diverse category c d is created by randomly picking elements equiprobably from all the categories

{c i } i∈[n ]
. The evaluation of the classifiers provides a similarity matrix M j ∈ [0, 1] K P i ×K P i for each category c i . This last is used to merge similar clusters and let dissimilar ones disjoint. More precisely, a first strategy is to consider clusters c i and c m symmetrically similar

(BUCBAM-SS) if: Ψ i (X m k,k∈[N m ] ) > S H and Ψ m (X i k,k∈[N i ]
) > S H , with m = i and S H ∈ [0, 1] a high score (close to 1). Another strategy is to consider, clusters asymmetrically similar (BUCBAM-AS) if only one constraint is respected and the other is greater than S M , with S M = S H /2 a medium score. In both cases (that are illustrated in Fig. 2), dissimilar clusters are desirably let disjoint. Merging similar clusters for all classes c j , results in a set of K M j finer-classes {c i } i∈K M j per category c j .

SpeFiNet: Combining Specific and Finer Features

Following the approach of [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF] -which roughly consists in training initial features on an initial set of categories, then learning new features on new set of categories and finally combining initial and new features -, we propose to learn the new features with our FiNet (rather than a network trained on generic categories [START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]) and combine them with the features of the initial SpeNet to get a representation even more universal. This method is denoted SpeFiNet in the following. Formally, the final SpeFiNet representation combines specific and finer features and is computed for an image I i of a target-task as:

X i = F {Z(X i ), Z(X +1 i )}
, where F is a fusion operator, and Z is a normalization function. In practice for the normalization and fusion, we respectively choose the L-infinite norm (L-∞) and the concatenation. To the best of our knowledge, we are the first to propose to combine a SpeNet (trained on specific categories) and a FiNet (trained on finer categories) to get more universal representations.

Experimental Results

Universality

Universalizing methods are evaluated in a transfer-learning scheme on multiple target-tasks [START_REF] Cer | Universal sentence encoder[END_REF][START_REF] Conneau | Supervised learning of universal sentence representations from natural language inference data[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]. More precisely, a source-task is used to train a network that acts as a representation extractor on the data of the target-tasks. Each target-task is trained with a simple predictor on top of the representations extracted from the samples of the target-task. Note that, fine-tuning the representations on the target-tasks could always improve performances but induces a bias avoiding correct evaluation of universality [START_REF] Conneau | Senteval: An evaluation toolkit for universal sentence representations[END_REF][START_REF] Huh | What makes imagenet good for transfer learning?[END_REF][START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]. Hence, following the literature, simple predictors that do not modify the representations learned on the source-task are used. In particular, here for the target-tasks, we used a classification task with datasets from multiple visual domains (presented below) and for the predictor, we used a one-versus-all SVM classifier for each class. Even if [START_REF] Conneau | Senteval: An evaluation toolkit for universal sentence representations[END_REF][START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]] initiated a work around universality evaluation, it seems to remain an open problem. Hence here, since we only have benchmarks that are evaluated in terms of accuracy and precision, we evaluate universalizing methods in terms of average of their performances on the multiple benchmarks.

Datasets

For the source-task, we used ILSVRC [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] and ILSVRC* (half of the former, detailed in [START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF]). For the target-tasks, we used ten datasets from multiple domains, including general objects (VOC07 [START_REF] Everingham | The pascal visual object classes challenge[END_REF], NWO [START_REF] Chua | Nus-wide: A real-world web image database from national university of singapore[END_REF], CA101 [START_REF] Fei-Fei | One-shot learning of object categories[END_REF], CA256 [START_REF] Griffin | Caltech-256 object category dataset[END_REF]), scenes (MIT67 [START_REF] Quattoni | Recognizing indoor scenes[END_REF]), actions (stACT [START_REF] Yao | Human action recognition by learning bases of action attributes and parts[END_REF]), birds (CUB [START_REF] Wah | The caltech-ucsd birds-200-2011 dataset[END_REF]), plants (FLO [START_REF] Nilsback | Automated flower classification over a large number of classes[END_REF]), food (FOOD [START_REF] Bossard | Food-101 -mining discriminative components with random forests[END_REF]) and airplanes (AIRC [START_REF] Maji | Fine-grained visual classification of aircraft[END_REF]). The characteristics of all the datasets are detailed in supplementary material. All the methods are trained on the data of ILSVRC* with an AlexNet network and compared in terms of average (Avg) performance on the set of eight target-tasks used in [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]. For each benchmark, we highlight the best score in bold and the second is underlined. Methods marked with † are obtained with a parameter cross-validated (on the target-tasks), while our BUCBAM method automatically set this parameter on the source-task. Note that MuCaLe-Net, MulDiP-Net and SpeFiNets use representations which dimension is twice other method's.

Implementation Details

Our method consists in the combination of a SpeNet and FiNet. For both networks, we used two architectures, namely the classical AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] and the deeper DarkNet [START_REF] Redmon | Yolo9000: Better, faster, stronger[END_REF]. They are respectively trained on the images of ILSVRC* and ILSVRC. SpeNet is thus respectively trained to recognize C = 483 and C = 1, 000 specific categories. In contrast, FiNet is trained to recognize a set of K i ×C finer-classes (K i depends on the splitting method), for which we used four variants: (i) random splitting with K i ∈ {2, 4, 8, 16} fixed, denoted Random-K (ii) K-means clustering with K i ∈ {2, 4, 8, 16} fixed, denoted Cluster-K (iii) BUCBAM splitting with asymmetrically similar clusters merging, denoted BUCBAM-AS and (iv) BUCBAM splitting with symmetrically similar clusters merging, denoted BUCBAM-SS. Note that, the BUCBAM methods leads to a K i depending on the content of each category. In Sec. 3.2, we provides some statistics of the resulting dataset of each method, including the total amount of finer-classes. In Cluster-K and BUCBAM methods, we extract features from the penultimate layer to represent the samples of each class, which results in features of 4096 dimensions for AlexNet and 1000 for DarkNet. Specific to BUCBAM, the K, S and S H parameters are respectively set to 32, 15 and 0.8. Indeed, K has to be large, and we found that as long as K is larger than 20 our method provides the same splitting result. S = 15 ensures to train a network with at least 15 images per class. We obtained similar results with S = 50. The parameters S H is not critical since similar clusters generally provides very high (close to 1.0) classification scores. All the methods are trained on the data of full ILSVRC with a DarkNet network and compared in terms of average performance (Avg) on the set of ten target-tasks presented in Sec 3. For each benchmark, we highlight the best score in bold and the second is underlined. Methods marked with † are obtained with a parameter cross-validated on the target-tasks. BUCBAM automatically set this parameter on the source-task.

Comparison to the State-of-the-Art

We compare the results obtained by our method with those of the literature, in particular, all the methods re-implemented and reported in [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]. For fair comparisons, we followed their training configuration, and trained our method with an AlexNet network and ILSVRC* as source-task. Moreover, instead of using our more diverse set of ten target-datasets, we used the eight ones used in their paper. The results are reported on Table 1. We first observe that our methods are always better than the reference method used in [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF][START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF][START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF][START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF][START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF], namely SpeNet. In particular our best method (BUCBAM) exhibits a boost of 6 points on average, compared to SpeNet. Let also note that SpeFiNet is always better than FiNet, itself better than SpeNet, regardless the splitting method. More precisely, the BUCBAM splitting method is significantly better than the best Cluster one, without the high cost of cross-validation of the K parameter. Compared to state-of-the-art methods, ours achieves the best performances, that is, almost 2 points of improvement compared to the most competitive MulDiP-Net method [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF], while it surpasses all other methods by more than 3 points. A last salient result is the fact that a SpeFiNet (whatever the splitting method) is significantly better than MuCaLe-Net [START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF] which has been trained on the best generic categories (manually obtained from categorical-levels [START_REF] Tamaazousti | Diverse concept-level features for multi-object classification[END_REF][START_REF] Tamaazousti | Mucale-net: Multi categorical-level networks to generate more discriminating features[END_REF]). This latter clearly demonstrates than combining features trained on specific categories with those trained on finer categories is better than combining them with those trained on generic categories.

Furthermore, since [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF] reported better results with a deeper network (DarkNet) trained on the full ILSVRC, we also implemented our method in the same configuration. Since MulDiP-Net provides the best results of the literature on the problem of universality, we only compare to them for this setting. The results are reported on Table 2. While the improvement is only slightly better, our method still beats the competitive MulDiP-Net. Moreover, a salient observation is that our method tend to be much better than theirs on the fine-grained classification benchmarks, which are more challenging. As in the previous setting, SpeFiNet is better than FiNet which is itself better than SpeNet. We also compared to the GenNet (which is the generic sub-component of MulDiP-Net [START_REF] Tamaazousti | Learning more universal representations for transferlearning[END_REF]) and we observe that FiNet-BUCBAM is better by 0.9 points. Figure 3: Comparison of our methods to multiple splitting baselines. On the left, we illustrate the average results of each method on the ten target-tasks, were for each splitting method, we illustrate the FiNet in dark color and SpeFiNet in light color. On the right, we plot a diagram for the SpeFiNet-BUCBAM, FiNet-BUCBAM and SpeNet methods, their performances on each target-task. Best view in color

Analysis and Comparison to Baselines

In this section, we perform an in-depth analysis of our method through an ablation study, a comparison to baselines and visualization of some statistics. In particular, in supplementary we compared our method to multiple clustering baselines, namely Spectral Clustering and Affinity propagation. The former provides a fixed set of clusters per category, while the latter leads to a dynamic set of clusters. A summary of results is presented on the left of Fig. 3, where we plot, a bar for each method, that represents its average performance on the set of 10 benchmarks described in Sec 3. We also tested the Mean-Shift algorithm with many different bandwidth values, but it always led to many clusters with one or two images, and one cluster containing all the remaining images. This setting providing very low results, we did not report them. From these results, we observe that our BUCBAM method is better than all the baselines including other existing algorithms. Rather than the average performances, in the diagram on the right of Fig. 3, we illustrated the detailed results (on the ten benchmarks) of the SpeFiNet-BUCBAM, FiNet-BUCBAM and SpeNet methods. We clearly observe that the diagram of our SpeFiNet-BUCBAM overlaps FiNet-BUCBAM, which itself overlaps the reference SpeNet method. In addition, we provide in supplementary material the detailed results of all the methods on all the target-tasks.

In Figure 4, we visualize some of the clusters obtained by each splitting method (random, clustering and BUCBAM). To do so, we highlight three clusters for two specific categories (two blocks of three rows of five images). On the left, the clusters are determined from a random distribution within the full specific category, leading to clusters that contain its full diversity. On the contrary, with the K-means clustering (middle), the clusters exhibits a more coherent aspect. For example, for the goldfish category, the c 1 3 cluster report close-up views of fish that are rather seen on their profile. We have a similar behaviour for the banjo category with cluster c 3 1 and c 3 2 . With the proposed BUCBAM method (right), the clusters are even more specific than in the K-means case. For instance, for the goldfish category, we clearly identify a cluster that represents "many golfishes" (c 1 1 ), "on goldfish in a close-up view" (c 1 2 ) and some images on which the fish tank is visible (c 1 3 ). Also for the banjo class, we also clearly observe that our method identified a cluster that represents "person playing banjo" c 3 1 and even "person playing banjo in a concert" c 3 3 . Importantly, while the clustering method tend to results in duplicate clusters (e.g., c 1 2 with c 1 3 ; c 3 1 with c 3 2 ; etc.), ours tend to provide only dissimilar results, thank to our merging process.

In supplementary material, we also provided some statistics of our method and baselines (i.e., histograms of average amount of clusters per category, histograms of intra-class variance of the clusters) and more visualizations of the obtained clusters, through the visualization of some images in some clusters and the features of each image of the clusters in a 2D dimensional space, after performing a PCA on their full features. This highlights the clear interest of our method, in terms of cluster relevancy and the balance of resulting data, compared to the random and clustering baselines.

Conclusions

In this paper, we tackled the problem of universality of representations with a new method relying on categories that are finer than the most specific ones of the ImageNet hierarchy. These last being the finest that are annotated, we proposed a method that automatically add a hierarchical-level to the ImageNet hierarchy. A network trained on the categories of such finer-level provides a more universal representation than with the upper levels. In practice, it leads to significantly better results in a transfer-learning scheme, on 10 publicly available datasets from diverse domains.

We also showed that a K-means and, surprisingly, a random partitioning of the leaf nodes of ImageNet already gives interesting results, although below than the proposed approach. It nevertheless suggests that the general principle highlighted in this article could be fruitful to design new CNN-based representations that are more universal in a transfer-learning context. Furthermore, it should be noted that our principle is neither limited to the ImageNet hierarchy nor to the classification task. Indeed, it could be applied to any hierarchy or dataset and on other tasks, such as detection, segmentation or keypoint estimation, as considered in [START_REF] Wang | The more you look, the more you see: towards general object understanding through recursive refinement[END_REF].

Figure 1 :

 1 Figure 1: Illustration of our splitting principle that determines a finer-level +1 containing n +1 finer-classes (blue nodes in A) from the finest-level of a hierarchy (here, ImageNet), containing n specific categories that the leaf gray nodes in (A). Each finer-class c +1 i (i ∈ [n +1 ]) is related to a specific-class c j ( j ∈ [n ]) through "is-a" relations, since the c +1 i classes are obtained from each c j class. In (B), we focus on the particular specific class c n l ( ) and its image-representations ( ), to determine the finer-classes {c +1 n l -3 , c +1 n l -2 , c +1 n l -1 , c +1 n l }. In (b), it is splitted into K (here K=4) groups ( ) corresponding to the finer-classes c +1 j .

Figure 2 :

 2 Figure 2: On the left, given image-representations ( ) labeled among a specific class ( ), BUCBAM first performs a clustering with many clusters ( ), then attaches small clusters to bigger closest ones ( ) and lastly, merges similar ones ( ), w.r.t certain strategies. On the right, we describe the latter merging strategies. Given the similarity matrix M j (high values: blue, low: white) for one specific class c j , BUCBAM-SS merges only clusters that are reciprocally highly similar to each other, respecting constraint (a) with high scoring values in (M j ) k,l and (M j ) l,k ). BUCBAM-AS also merges asymmetrically similar clusters (those that respect constraints (a), (b) and (c)). In both cases, dissimilar clusters (d) are let disjoint.

Figure 4 :

 4 Figure 4: Illustration of some finer categories obtained from two specific categories (two row blocks) with the different methods: random split (left), Kmeans clustering with K=16 (middle) and our BUCBAM proposal (right). In every block, a line (from the three) shows the five most representative images of a cluster at the new finest-level. Best view in PDF.

Table 1 :

 1 Comparison of our methods (bottom) to the state-of-the-art (top).

	Method	VOC07 CA101 CA256 NWO MIT67 stACT CUB FLO mAP Acc. Acc. mAP Acc. Acc. Acc. Acc.	Avg
	SpeNet (REFERENCE)	66.8	71.1	53.2	52.5	36.0	44.3	36.1	50.5	51.3
	SPV spe A [1] SPV gen G [22, 35]	66.6 67.7	74.7 73.0	54.7 54.3	53.2 50.5	37.4 37.1	45.1 44.9	36.0 36.8	51.9 50.3	52.4 51.8
	AMECON [5]	61.1	58.7	40.6	45.8	24.3	32.7	26.1	36.4	44.5
	WhatMakes [16]	64.0	69.4	50.1	45.6	33.7	41.9	15.0	42.8	45.3
	ISM [42]	62.5	68.8	50.7	28.5	37.9	42.6	34.0	50.0	46.9
	GrowingBrain-RWA [41]	69.1	74.8	55.9	50.4	40.0	48.4	38.6	56.1	54.2
	FSFT [36]	67.5	73.9	55.0	44.6	40.4	47.1	38.7	56.8	53.0
	MuCaLe-Net [34]	69.5	76.0	56.8	54.7	41.3	48.5	35.6	54.8	54.6
	MulDiP-Net [36]	69.8	77.5	58.3	47.9	43.7	50.2	37.4	59.7	55.6
	FiNet, Random †	66.4	72.4	53.2	51.0	39.7	46.9	35.7	55.9	52.6
	FiNet, Cluster †	66.0	73.2	54.6	50.9	40.7	47.2	36.4	55.6	53.1
	FiNet, BUCBAM	65.3	75.4	56.0	48.6	41.6	49.4	37.8	59.8	54.2
	SpeFiNet, Random †	69.8	75.7	57.5	54.6	41.2	50.0	39.8	58.3	55.9
	SpeFiNet, Cluster †	68.6	77.9	58.1	53.9	41.3	50.5	40.8	60.1	56.4
	SpeFiNet, BUCBAM	69.1	78.3	59.3	54.0	42.7	52.0	41.8	61.7	57.4

Table 2 :

 2 Comparison of our methods (bottom) to the state-of-the-art (top).

	Method	VOC07 CA101 CA256 NWO MIT67 stACT CUB FLO AIRC FOOD Avg. mAP Acc. Acc. mAP Acc. Acc. Acc. Acc. Acc. Acc.
	SpeNet (REF.)	82.7	91.0	78.4	70.5	64.8	72.2	59.5	80.0	49.2	47.6	69.6
	GenNet [34]	83.2	91.5	78.1	73.2	64.4	72.6	52.5	78.9	48.5	46.2	68.9
	MulDiP-Net [36]	84.1	92.7	80.1	73.9	66.4	74.5	61.2	82.1	53.5	49.3	71.8
	FiNet, Cluster †	82.5	91.8	78.8	70.0	65.8	73.2	60.9	81.9	51.9	47.8	70.5
	FiNet, Cluster(K = 16)	81.4	91.5	77.4	69.5	64.6	72.2	58.6	81.5	52.3	47.5	69.6
	FiNet, BUCBAM	81.3	91.0	77.0	69.7	64.3	72.2	59.1	81.6	52.9	48.9	69.8
	SpeFiNet, Cluster †	83.7	92.5	79.8	71.9	66.7	74.8	63.6	83.1	54.5	49.5	72.0
	SpeFiNet, Cluster(K = 16)	83.3	92.2	79.6	71.9	66.6	74.1	62.5	83.0	55.1	49.8	71.8
	SpeFiNet, BUCBAM	83.2	92.2	79.6	71.7	66.6	74.5	62.7	83.4	56.1	50.0	72.0

For simplicity, we omit the power indices , + 1 and L in the following.