Youssef Tamaazousti

Hervé Le Borgne

Céline Hudelot

Mohamed-El-Amine Seddik

Mohamed Tamaazousti

Learning More Universal Representations for Transfer-Learning

Keywords: Deep-Learning, Universal Representations, Universality Evaluation, Transfer-Learning, Visual Recognition. !

A representation is supposed universal if it encodes any element of the visual world (e.g., objects, scenes) in any configuration (e.g., scale, context). While not expecting pure universal representations, the goal in the literature is to improve the universality level, starting from a representation with a certain level. To do so, the state-of-the-art consists in learning CNN-based representations on a diversified training problem (e.g., ImageNet modified by adding annotated data). While it effectively increases universality, such approach still requires a large amount of efforts to satisfy the needs in annotated data. In this work, we propose two methods to improve universality, but pay special attention to limit the need of annotated data. We also propose a unified framework of the methods based on the diversifying of the training problem. Finally, to better match Atkinson's cognitive study about universal human representations, we proposed to rely on the transfer-learning scheme as well as a new metric to evaluate universality. This latter, aims us to demonstrates the interest of our methods on 10 target-problems, relating to the classification task and a variety of visual domains.

INTRODUCTION

H Umans are able to recognize a scene and the objects that composed it with disconcerting ease in comparison to a machine. According to Atkinson's cognitive study [START_REF] Atkinson | The developing visual brain[END_REF], such capabilities result from the development of a powerful internal representation in their infancy, which they re-use later in life to solve multiple problems. In analogy, machines based on neuralnetworks also perceives the data with representations that they use to solve tasks. However, while vision in a human works well for multiple problems, machines are only able to solve one at a time. This latter motivated some recent works, that wanted to "mimic" such abilities of humans by learning universal models [START_REF] Kokkinos | Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory[END_REF], [START_REF] Wang | The more you look, the more you see: towards general object understanding through recursive refinement[END_REF] or universal representations [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF]. By the former, we mean models that are able to solve every possible task (recognition, detection, segmentation, etc.), and by the latter, we mean representations that are able to encode every possible element (materials, objects, scenes, etc.), in every possible configuration (scale, occlusion, context, etc.). Note that, without a good encoding of the data, a task can not be solved efficiently, thus a first step toward universal models is to get universal representations. We are far from expecting a representation that could encode any information, thus the actual purpose of our work is to improve the universality of a given representation, at a minimal cost.

Given a reference universal representation, a straightforward way to increase its universality, is undoubtedly the use of more training data (tasks and domains). For instance, [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] proposed to simultaneously learn a wide set of different visual domains, by better considering the additional data, through the use of scaling parameters that learns the statistics of each domain. In the same vein, Subramanian et al. [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF] considered multiple • Y. Tamaazousti, is at the CSAIL of MIT, USA. E-mail: ytamaaz@mit.edu • H. Le Borgne, M.E.A. Seddik and M. Tamaazousti are at the CEA, LIST,

France. E-mail: firstname.lastname@cea.fr • C. Hudelot is at the MICS laboratory of CentraleSupélec (University of Paris-Saclay). E-mail: celine.hudelot@centralesupelec.fr Manuscript received September 2, 2018 Fig. 1. A universal representation is a set of features learned on a source-problem (SP), that exhibit good performances when it is transfered on many target-problems (TPs). A universal representation R A is obtained by a reference method (A) that consists in training a network N on a SP (a). When it is transfered on a large set of TPs (e), it leads to a high aggregated performance P A on it. In this paper, we propose to build a more universal representation R B by: (i) normalizing and combining the features of (A) with those of a network (yellow N block) learned after a source-problem variation (SPV) (c) -that, transforms an initial SP (a) into a new one (b) -and (ii) re-training both networks on their initial SP through focused self fine-tuning denoted FSFT (d). Since our method (B) is more universal than (A), it has a higher universality score according to the proposed evaluation-metric.

datasets with different tasks and proposed to find which of the available multi-task learning algorithms and set of tasks provides the best universal level. Note that, this approach of adding more data is limited by availability of data. Hence, here we are interested by increasing universality from a fixed set of data (tasks and domains). Conneau et al. [START_REF] Conneau | Supervised learning of universal sentence representations from natural language inference data[END_REF], [START_REF] Conneau | What you can cram into a single vector: Probing sentence embeddings for linguistic properties[END_REF], [START_REF] Conneau | Very deep convolutional networks for text classification[END_REF] have also the same goal. However, while their approach consists to find the best algorithm, we argue that we could structure the training-data in a way that provides different tasks to learn more or better features.

arXiv:1712.09708v5 [cs.CV] 3 Sep 2018

The overall contribution of this article (illustrated in Fig. 1) consist in a method that increases universality from a fixed set of data. In the vein of MuCale [START_REF] Tamaazousti | Mucalenet: Multi categorical-level networks to generate more discriminating features[END_REF], we proposed MuLDiP-Net that: (i) variates the initial source-problem (SP); (ii) uses its resulting SPs to train new features; and (iii) combines all the features to form a more universal representation. By construction, MulDiP-Net learns new discriminative features without any additional images or annotations. Regarding the variation, we propose a formalism that consist to start from a set of specific categories (e.g., rottweiler, pit-bull) and group them in more generic ones (e.g., dog), according to the upper-levels of a given hierarchy (e.g., WordNet, hierarchy obtained by clustering the specific categories, or categorical-levels relating to Human-categorization [START_REF] Jolicoeur | Pictures and names: Making the connection[END_REF], [START_REF] Rosch | Principles of categorization[END_REF]). The proposed formalism is quite general and makes MuCale a particular case of our method, that groups categories according to categorical-levels only. We also introduce a new method named focused self fine-tuning (FSFT), based on self-training [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] of CNNs on the same SP. This approach increases the universality without any additional data, and more interestingly, at the same network-capacity. FSFT can also be used to reduce the dimension of representations. Combined to MulDiP-Net, it thus improves the performances while reducing the network-capacity.

We also address the problem of universality evaluation. Rebuffi et al. [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF] proposed an interesting Visual Decathlon Challenge (VDC) to measure universality on multiple domains. Furthermore, they proposed a VDC metric that coherently aggregates the scores on the multiple domains. However, they restricted their evaluation in a classical end-to-end scheme: learn the representation on the train-set and evaluate on the test-set. Such scheme do not completely match with the claim of Atkinson 1 , that is, learning on one environment and use the learned representation on different problems. Hence, we propose to go further by evaluating universality in a transfer-learning (TL) scenario [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF], [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF]. TL, naturally fits with [START_REF] Atkinson | The developing visual brain[END_REF], with the source-problem that corresponds to the "infancy learning environment" and the target-problems to those "solved later in life". Also, while the VDC metric respects some desirable criteria, we argue that more are required for universality, and thus proposed a new metric that respects more of them.

The present manuscript differs in many ways from our previous work on this topic [START_REF] Tamaazousti | Mucalenet: Multi categorical-level networks to generate more discriminating features[END_REF]. In particular, it includes the following new contributions: [START_REF] Ahmed | Network of experts for large-scale image categorization[END_REF] a new universalizing method based on selftraining of neural-networks [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] (Sec. 3); (2) a more general formalism of the MulDiP-Net approach based on the principle of source-problem variation by grouping (Sec. 4), allowing the use of multiple variations and thus making [START_REF] Tamaazousti | Mucalenet: Multi categorical-level networks to generate more discriminating features[END_REF] a special case of our approach; (3) the formalization of FSFT as a dimensionality reduction method and its combination with MulDiP-Net (Sec. 4.5), making this latter more performing while significantly reducing its network-capacity; (4) the evaluation of universality in a TL scheme, that better fits the claim of [START_REF] Atkinson | The developing visual brain[END_REF] and the highlight of four desirable criteria for universality evaluation as well as the proposition of a new metric (Sec. 5); and (5) more extensive and detailed experimental results, including a comparison to the stateof-the-art based on ten available target-problems (Sec. 6), as well as an in-depth analysis of our approach including an ablation study and the impact of some important component of our approach (Sec. 6.3 and supplementary material).

1. In all the document, the "claim of Atkinson" states for: "the ability of humans to develop a universal and powerful internal representation of images in the early years of their development and re-use it (almost) as is later in life for solving multiple kind of different problems" [START_REF] Atkinson | The developing visual brain[END_REF].

RELATED WORK

Learning Universal Representations

With UberNet, Kokkinos [START_REF] Kokkinos | Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory[END_REF] considered a unified architecture that is trained end-to-end to tackle several vision tasks (universal model) and addressed resulting technical challenges. In the same vein, Subramanian et al. [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF] used multiple datasets with different NLP tasks and proposed to find which of the available multitask learning algorithms and set of tasks are the best to learn universal representations. Alternatively, Bilen and Vedaldi [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF] rather considered that a universal representation should be able to address multiple visual domains. They proposed to learn a compact representation able to perceive a wide set of domains, using scaling parameters to learn the statistics of each. It has been extended in [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] by respectively using sequential and parallel residual adapters. All these works use more data, and it is undoubtedly the most straightforward way towards universality, but it is limited by its availability. It is thus interesting to improve universality from a fixed set of data, which is our goal as well as that of Conneau et al. [START_REF] Conneau | Supervised learning of universal sentence representations from natural language inference data[END_REF], [START_REF] Conneau | Very deep convolutional networks for text classification[END_REF]. However, in contrast to their approach, that consists to find the best task and algorithm for universality, ours is to structure the data in a way that automatically provides different tasks to learn more or better features.

All the above works consider multiple source-problems (e.g., tasks [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF], domains [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF] or problems with same images but different categories as our work) to learn the universal representation. It is important to note that learning the representation with one network (using multi-task, multi-label or recursive [START_REF] Wang | The more you look, the more you see: towards general object understanding through recursive refinement[END_REF] learning) is just a way to do it. In this paper we proposed to rely on multiple networks through independent learning, and the advantages and drawbacks of these methods will be detailed in Sec. 2.2.2.

Approaches for Universalizing Methods

Many works can be considered as universalizing methods, in the sense that they diversify and increase feature detectors, either by modifying the source-problem (SP) on which a network is learned, or relying on ensemble models that learn several networks on different problems. We propose a unified view of these approaches, using the notion of Source-Problem Variation (SPV, Sec. 4.1).

Learning one Network on a Modified SP

To diversify an initial problem, a first approach consists in adding new categories and corresponding annotated images. Three kinds of categories are added: specific [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF], [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Zhou | Learning deep features for scene recognition using places database[END_REF] (e.g. rottweiler), generic [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], [START_REF] Mettes | The imagenet shuffle: Reorganized pre-training for video event detection[END_REF], [START_REF] Tamaazousti | Vision-language integration using constrained local semantic features[END_REF] (e.g. dog) and noisy [START_REF] Joulin | Learning visual features from large weakly supervised data[END_REF], [START_REF] Phong | Harnessing noisy web images for deep representation[END_REF]. In some cases, the categories added contain data from multiple domains [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF]. These methods slightly increase the capacity of the network by adding parameters specific to each domain. In all cases, this approach can be quite powerful to increase the universality, but at a high cost since it requires many additional data and corresponding annotations. Another limitation lies in the learning methodology. Indeed, the network is often trained jointly on the generic and specific data, resulting into a mix of generic and specific features in the intermediate layers. Since a softmax loss is often used, it considers generic and specific categories as mutually exclusive, that is not the case. At the opposite, we propose to group the categories according to their level and separately learn the features on each to respect the real-world semantics. Alternatives to this proposal are the works of [START_REF] Chami | Amecon: Abstract meta concept features for text-illustration[END_REF], [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF] that respectively group hierarchically or by clustering. However (as shown in the experiments), for the sake of universality, it is preferable to group at categorical-levels as we propose.

Learning a Set of Networks on a Set of SPs

The methods of the second approach [START_REF] Ahmed | Network of experts for large-scale image categorization[END_REF], [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], [START_REF] Venkatesh N Murthy | Deep decision network for multi-class image classification[END_REF], [START_REF] Ouyang | Factors in finetuning deep model for object detection with long-tail distribution[END_REF], [START_REF] Wu | Deep convolutional neural network with independent softmax for large scale face recognition[END_REF], [START_REF] Yan | Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition[END_REF] give an answer to the problem that in joint training, different kinds of features are undesirably mixed. Indeed, all the works in this approach use an ensemble-model on different source problems and a sequential training procedure. They train a network on an initial problem (that contains specific or generic categories) and they fine-tune it on another problem or on a set of smaller problems. As a consequence, even if the scope of the new representation is increased by this process, all the features learned on the new problems are biased toward those of the model previously learned on the initial problem. Thus, due to their sequential learning procedure, these approaches do not combine different types of knowledge (specific and generic categories) but only consider one of them, that of the last problem used for training. A consequence of the latter point is that, they need many models (i.e., more than 10) to get significant diversity in the set of features, which is very costly. The proposed MulDiP-Net method, provides an answer to this limitation by carrying independent network training (one network per problem).

From the point-of-view of the strategy used to vary the source problem, all the methods [START_REF] Ahmed | Network of experts for large-scale image categorization[END_REF], [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], [START_REF] Ouyang | Factors in finetuning deep model for object detection with long-tail distribution[END_REF], [START_REF] Yan | Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition[END_REF] from this approach re-label specific categories into non-semantic generic ones, that is to say categories that do not exist in the real world) [START_REF] Chami | Amecon: Abstract meta concept features for text-illustration[END_REF] and capture the common properties among many object classes independently of an actual common semantics. These generic categories are built using hierarchical clustering on low/mid-level features (obtained from a network trained on the initial SP) of images among the initial set of categories. As a consequence, these methods are dependent to the visual low/mid-level features that can lead to irrelevant categories when low/mid-level features fail to capture the dissimilarity between different categories. For the sake of universality, it is preferable to rely on a grouping process that uses explicit human categorization expertise in order to reflect semantically-real relations between categories.

Universality Evaluation

Inspired by studies on the visual brain (claim of [START_REF] Atkinson | The developing visual brain[END_REF]), authors of [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] evaluated universality of representations as their ability to simultaneously cover a large range of visual domains. Their evaluation consists in learning and testing on the same problem and only the visual domain differs. Moreover, since many problems are considered, they proposed a metric to aggregate the scores in each task, that respects their proposed criterion of significance. While such criterion is undoubtedly useful for universality evaluation, it should be noted that their classical evaluation framework does not completely match with [START_REF] Atkinson | The developing visual brain[END_REF]. In contrast to our transfer-learning scheme, their scheme do not consider the terms "re-used later in life" which means that multiple and different problems should be solved by a universal representation. Moreover, in addition to their criterion of significance and the coherent aggregation considered in [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF], we proposed a total of six additional criteria important for universality evaluation, as well as two metrics that respect almost all the criteria. Nevertheless, while such TL scheme has been already used in the NLP community [START_REF] Conneau | Senteval: An evaluation toolkit for universal sentence representations[END_REF], [START_REF] Conneau | Supervised learning of universal sentence representations from natural language inference data[END_REF], [START_REF] Conneau | Very deep convolutional networks for text classification[END_REF], [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF] for universal representations, to the best of our knowledge, we are the first to propose it in the vision community and more importantly, to link it to the claim of [START_REF] Atkinson | The developing visual brain[END_REF].

Cognitive Studies in Computer Vision

A last line of work deals with the inspiration from cognitive studies in computer vision [START_REF] Deng | Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition[END_REF], [START_REF] Mathews | Choosing basic-level concept names using visual and language context[END_REF], [START_REF] Ordonez | Predicting entry-level categories[END_REF], [START_REF] Tamaazousti | Diverse concept-level features for multi-object classification[END_REF]. Generally, their goal is to output basic-level concepts of an image from a set of predicted finer ones. An exception is the work of [START_REF] Tamaazousti | Diverse concept-level features for multi-object classification[END_REF], that is closest to ours since they consider categorical-levels in their representation. As in our work, their system reflects the psychological hint stating that, even if humans tend to categorize objects at the subordinate-level, they are still aware of the other categorical-levels [START_REF] Rosch | Principles of categorization[END_REF]. However, the key difference is how we integrate that hint as well as the purpose of its consideration. Indeed, our goal is to diversify the features learned in CNNs, while they aim at solving the problem of generic categories that output low scores because of their high intra-class variance, in order to force their beneficial consideration. Moreover, we opt for an integration at three levels (data, learning and representation), while they do it only after the computation of their semantic representation [START_REF] Alexandru | Large-scale image mining with flickr groups[END_REF], [START_REF] Tamaazousti | Constrained local enhancement of semantic features by content-based sparsity[END_REF].

FOCUSED SELF FINE-TUNNING

We propose a new method that takes advantage of the principle of the re-training of neural networks on the same problems, and thus does not need more data [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF], [START_REF] Mettes | The imagenet shuffle: Reorganized pre-training for video event detection[END_REF], [START_REF] Tamaazousti | Vision-language integration using constrained local semantic features[END_REF], nor increasing the network capacity [START_REF] Ahmed | Network of experts for large-scale image categorization[END_REF], [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], [START_REF] Tamaazousti | Mucalenet: Multi categorical-level networks to generate more discriminating features[END_REF], [START_REF] Wang | Growing a brain: Fine-tuning by increasing model capacity[END_REF]. Our approach relates to the work of [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] who proposes an extensive study of the effect of different self-training methods (i.e., re-training a neural network on the same problem it was trained originally). They explored two learning-strategies: (i) frozen-based re-training (that we call Frozen Self-Training and denote FrST) and (ii) fine-tuning based re-training (that we call Self Fine-Tuning and denote SFT). More precisely, both methods re-train an initial network (that we denoted initNet), with Θ a =(θ a 1 , θ a 2 ) being its set of trained weights. Both methods consist in three steps: (i) the weights of FrST and SFT are splitted into two sets (θ b 1 and θ b 2 ), with θ b 1 containing the weights of the first L layers and θ b 2 , the weights of the last ones (ii) the two sets of weights are initialized differently -in FrST and SFT, the first layers are initialized with the weights of the pre-trained initNet and the last layers are initialized randomly -; and finally (iii) the re-training of the weights -FrST retrains only last layers and "frozes" the first ones, while SFT retrains them all with the same learning-rate. Their extensive study leads to some interesting conclusions that motivate our approach. In particular, they showed that: (i) FrST hurts the performances of initNet because"initNet contained fragile co-adapted features on successive layers" (i.e., features that interacted with each other in a fragile way during initNet learning such that this co-adaptation could not be relearned by the randomly initialized upper layers alone) and (ii) SFT slightly increases the performance because "it aims to recover co-adapted features that were trained by the initNet". Simply said, it is important to preserve some knowledge acquired during the learning of the original network, but it is a sub-optimal to completely focus the training on the last layers. As a consequence, we propose a new method called Focused Self Fine-Tuning (FSFT) that can be seen as an hybrid view of the two previous ones. As in [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF], the re-training principle consists in dividing the weights learned on the initNet into two sets, initializing them differently (first with the pre-trained initNet, others randomly), jointly minimizing them but with different learningrates. An illustration is given in Fig. 2.

More formally, let us consider a source training database D S containing N images x i with their associated labels y i . Let us also consider a network N that was trained on D S by minimizing a loss function L with a certain optimization algorithm. Let note Θ a =(θ a 1 , θ a 2 ) being its set of trained weights splitted in two sets, namely θ a 1 that contains the weights of the first L layers and θ a 2 that contains those of the remaining layers. FSFT consists to retrain the network N by minimizing the same loss-function L as initNet, on the same training database D S , which is expressed by: arg min

Θ b L((Ψ b (x, Θ b = (θ b 1 , θ b 2 )); y), (1)
where Ψ b (x, Θ b ) is the predicted probability vector for images x using learnable weights

Θ b =(θ b 1 , θ b 2 )
. The set of weights θ b 1 is initialized with those of the first layers of initNet (i.e., θ b 1 =θ a 1 ) and θ b

2 is initialized randomly. The individual weights w b ij , that forms the full sets θ b j (with j ∈ {1, 2}), are updated through:

w b ij ← w b ij -η j ∂L ∂w b ij , ∀w b ij ∈ θ b j , (2)
where, η j respectively corresponds to the learning-rate of the first layers (when j=1) and last ones (when j=2). More specifically, η 1 =α×η 2 , with α ∈ [0, 1] is a parameter that can be set by cross-validation. To summarize, our FSFT method preserves some knowledge acquired during the learning of the initNet (by initializing the weights of its first layers, with the firsts of initNet).

It also focuses the training on the last layers (by completely training the last layers, while allowing, through factor α, some slight change of the first ones).

MULTI DISCRIMINATIVE-PROBLEM NETWORK

The previous method is efficient without needing additional data nor capacity but always solves the same problem. To learn more universal representations, one must solve different problems.

In this section, we propose another universalizing method that combine different but complementary features learned on different problems. It consists in three components: (i) source problem variation (SPV) (Sec. 4.1) -that define a new source problems (SPs)with a special emphasis on variation with grouping (Sec. 4.2); (ii) independent training of networks on a set of multiple SPs obtained by the SPV (Sec. 4.3); and (iii) extraction of features from the set of trained networks followed by a particular combination to form the more universal representation (Sec. 4.4). Since each SP is a discriminative model solved by a neural network, the method is named "Multi Discriminative Problem Network" (MulDiP-Net, illustrated in Fig. 3). In Sec. 4.5, we use the FSFT (Sec. 3) as a dimensionality reduction method in the MulDiP-Net.

SPV: Source-Problem Variation

A source problem

D S k = {(x k i , y k i )} i= 1,N k , consists in a set of N k pairs (x k i , y k i ) with x k
i being a training image and y k i its associated label. The images x k i are labeled according a label set

Y k = {c k 1 , . . . , c k C k } of C k categories.
By solving this source problem (SP), the CNN learns features that discriminate between the images of the different categories of the given SP. For instance, if we consider a network that solves a SP containing images of lemons and green-apples, the network will learn different features than one that solves a SP containing lemons and strawberries's images. Hence, changing the SP to be solved by a CNN can lead to a change in the set of learned features. Motivated by this assumption 2 , we propose the principle of Source Problem Variation (SPV) that is a variation function ϑ k 0 (•) that transforms an initial source problem D 0 into a new one D k,k>0 . Such a function has the following form:

ϑ k 0 : {R S I × {0, 1} C0 } N0 → {R S I × {0, 1} C k } N k (3) D 0 → D k = {(x k i , y k i )} i= 1,N k , with the following constraints: ∀i ∈ 1, N k , ∀j ∈ 1, N 0 , ∃(x k i , y k i ), x k i = x 0 j or y k i = y 0 j ∀i ∈ 1, N k , ∀j ∈ 1, N 0 , ∃(x k i , y k i ), x k i = x 0 j or y k i = y 0 j ( 4
Y k = {c k 1 , . . . , c k C k } containing C k categories.
Regarding the constraints, the first ensures that at least one element of D 0 has to be variated compared to the elements of D k and the second one warrants that at least one element of D 0 has to be in D k . With these constraints, taking a dataset completely different than the initial one is not a SPV and in contrast, changing all the data (images and labels) without keeping any element from the initial SP is also not a SPV.

In practice, the variations can be of many nature and act both on the images {x 0 i } i∈ 1,N0 and/or the labels {y 0 i } i∈ 1,N0 ). We consider three types of variations: (i) adding images or categories to the initial SP; (ii) splitting the initial SP ; (iii) grouping categories (thus images) of the initial SP (Fig. 4). This principle, although not explicitly described as a SPV, have been already applied in the literature. For instance, variations on the image set while preserving the set of categories have been proposed in [START_REF] Herranz | Scene recognition with cnns: objects, scales and dataset bias[END_REF], [START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF]; adding variations has been used by [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] (by adding data labeled among specific categories) and [START_REF] Mettes | The imagenet shuffle: Reorganized pre-training for video event detection[END_REF], [START_REF] Tamaazousti | Vision-language integration using constrained local semantic features[END_REF] (by adding data labeled among generic categories); splitting variations on the set of categories has been performed in [START_REF] Ahmed | Network of experts for large-scale image categorization[END_REF], [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF], [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF], [START_REF] Wu | Deep convolutional neural network with independent softmax for large scale face recognition[END_REF], [START_REF] Yan | Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition[END_REF] and finally, variation of the categories by grouping them has been explored in [START_REF] Chami | Amecon: Abstract meta concept features for text-illustration[END_REF], [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF], [START_REF] Salvador | Learning cross-modal embeddings for cooking recipes and food images[END_REF]. However, note 2. Assumption empirically validated by [START_REF] Zhou | Learning deep features for scene recognition using places database[END_REF]. Indeed, they have shown that a CNN trained on scenes learns different features (i.e., object detectors) than one trained on objects (which learns object-part detectors). that a SPV can be applied for different goals and as explained in the beginning of this section, our goal is to learn different features than those learned on the initial SP, but complementary when combined together. Thus, with respect to our goal, the SPV has to be associated to a learning procedure and learn new neurons.

Beyond the stated aim, our work introduce a new type of SPV based on the grouping of categories and their associated images.

SPV Based on Grouping

Grouping-SPV consists in the grouping of images through the grouping of their initial categories. In practice, it can be done in many ways (e.g., randomly, based on clustering or semantically). Here, motivated by knowledge on human categorization, we propose a semantic grouping, through categorical-levels. The advantage of the semantic-grouping lies in its semantic aspect, which aims to get a new SP that is highly different than the initial one but relevant, which is not necessarily the case with random and clustering-based grouping. Compared to other SPVs, the grouping one has important advantages. Indeed, compared to adding SPV (which needs more annotated data, costly to obtain) grouping SPV does not need more annotated data, and compared to splitting SPV (which decreases the performances of the networks, since it decreases considerably the amount of training data), grouping SPV maintains exactly the same amount of images. Semantic-grouping SPV consists in the grouping of specific categories into generic ones, according to a semantic knowledge. This semantic knowledge is generally represented in the form of hierarchies. Here, we focus on a particular semantic knowledge, named categorical-levels [START_REF] Jolicoeur | Pictures and names: Making the connection[END_REF], [START_REF] Rosch | Principles of categorization[END_REF], [START_REF] James | Object categories and expertise: Is the basic level in the eye of the beholder? Cognitive Psychology[END_REF] that consists in a hierarchy of categories mostly used by Humans to categorize objects. Let us consider a semantic hierarchy with hyponymy relations (i.e., a set of categories organized according to "is-a" relations).

In practice here we use categorical-levels, but hierarchical-levels (i.e., levels of the ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF] or WordNet) or clustering-levels could be used (more details in last paragraph of this section). Formally, the starting semantic hierarchy (ImageNet) is a directed acyclic graph H = (V, E) consisting of a set V of nodes and directed edges E ⊆ V × V. Each node v ∈ V is a label and (v i , v j ) ∈ E is a hierarchy-edge indicating that label v i subsumes label v j . Let us also consider an initial source-problem D S 0 containing N 0 images labeled among C 0 specific categories belonging to C S 0 = {c 0 1 , c 0 2 , . . . , c 0 C0 }, such that C 0 ⊂ V. We now consider a categorical-level defined according to human cognition. Let us note B cat L a set of categories that belong to a categorical-level L (i.e, subordinate level for L=0, basic for L=1 and superordinate for L=2). It is important to note that the categories of B cat L do not correspond to a given level of the ImageNet hierarchy H. Hence, consider that all c cat L i ∈ B cat L are mapped into certain nodes of the hierarchy H. Our purpose is thus, to group the categories of C S 0 into G generic categories. This latter is equivalent to get the partitioning of C S 0 into G subsets i.e., C S 0 = G i=1 G i . To do so, we define a partitioning function according to a categorical-level B cat L as:

P cat L : V → 2 C S 0 (5) c cat L i → C S 0 ∩ D H c cat L i ,
where

D H (c cat L i
) is the set of all descendant nodes of the categories c cat L i according to H. Using the P cat L function, we obtain G generic categories, with G C 0 . We can now define our re-labeling function relative to a given categorical-level B cat L by:

R B cat L : 2 C S 0 → B cat L ( 6
)
C i → B cat L ∩ A V (LCA H (C i ))
, where A V (•) is a function that outputs, for all c i categories the set of all its ancestors in V and is defined as

A V (c i ) = {δ j H (c i )} ∞ j=1
, with δ H (•) being a deductive function that associates to a category v i of V its direct ancestor, that is to say, the category directly above v i according to H and δ n H (•) its corresponding iterated function (i.e δ H (•) composed with itself n times and for which we assume that the image of the root node of H is itself).

Simply said, while Eq. ( 5) partitions the set of specific categories into "unnamed" generic categories (i.e., do not have association to a humanly understandable word), Eq. ( 6) re-labels them into existing (and thus "named") categories of the categoricallevel B cat L . Importantly, images are also automatically re-labeled according the same process than the initial categories they belong. To recap, our ϑ G grouping SPV (illustrated in Figure 5) is the combination of the partitioning and re-labeling functions.

While the above process is applied on the re-labeling of the specific categories into generic ones belonging to a categoricallevel B cat L , it can also be applied to re-label into generic categories belonging to hierarchical-levels B h L of H or clustering-levels (hierarchies constructed based on data, through clustering). For the former, we need the following two assumptions about H: descendants of leaf-nodes are themselves (i.e., D H (c 0 i ) = c 0 i ) and if a leaf-node is at a certain hierarchical-level B h L with L = 0, its least common ancestor is itself (i.e., if

c 0 i ∈ B h L , LCA(c 0 i ) = c 0 i ).
For the latter, no assumptions are needed and practical details to construct the clustering-hierarchy are given in Sec. 6.1.3.

In summary, in MulDiP-Net, we consider the initial specific SP as well as those obtained by three grouping SPVs, namely, categorical, hierarchical and clustering-based ones. This latter, results in a set of SPs denoted D Ω . consists in a set of N k images x k i labeled among C k categories c k j . This latter forms a set of training-data which is used to train each network. It is worth noting that, we can have as many architectures and learning procedures (one per network) as the number of SPs in D S Ω , but here we focus on the special case where they are all the same (N k =N 0 3 , P k =P 0 for k>0). Note also that, our method does not depend on a particular architecture or learning procedure and can thus directly benefit from the advances in this domain.

MulDiP-Net Training

S k = {(x k i , y k i )} i∈ 0,N k
Specifically, here we used common CNN architectures (AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and DarkNet [START_REF] Redmon | Yolo9000: Better, faster, stronger[END_REF]) and follow the classical learning procedure for P 0 , that is to say, a random initialization of the weights (with a Gaussian distribution), optimizing a softmax loss-function with stochastic gradient descent (SGD). We have as many cost functions to minimize as the number of SPs in D S Ω and each cost function L k (Θ) is minimized independently. Since the SPV functions are based on grouping, all the SPs contain the same images as the initial one thus each cost function is minimized on the same set of training images, but with different labels. There is thus no cost in terms of additional data, while the cost to annotate is dramatically reduced since it is reported on each category and not each image. On ILSVRC for instance, it means 1, 000 annotations instead of 1.2 million. At convergence, we obtain a set N * Ω = {N * 0 , . . . , N * ω } of ω+1 networks. Such an "ensemble-like" learning method could be limited in terms of network capacity, but it exists solutions to decrease this capacity while maintaining high performance, as detailed in Sec. 3.

MulDiP-Net Representation

Let us consider a MulDiP-Net (i.e., a set N * Ω of ω+1 trained networks) and an image I τ i of a target-task τ . To extract the representation R Ω i,τ from I τ i , we perform the following two steps: (i) extraction of the features of I τ i through each subnetwork N * k of MulDiP-Net and (ii) normalization and combination of these features into a relevant representation (Fig. 3). For the extraction 3. In all the document, N indicates the architecture of a network, while N * indicates a trained network.

of the features, let recall that an architecture is a composition of classifiers Ψ and features-extractor Φ (i.e., N = Ψ • Φ). Thus, to get the features from an image, the classifier as well as some of the last layers are discarded, and the K th first layers of network N * k filter the images (e.g, {conv1, conv2} when K=2 for AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]). Hence, the features-extractor function (denoted φ K k (•)) outputs, for I τ i , a vector (if K points to a fully-connected layer) or a tensor of activations (if K indicates a convolutional layer). In the latter case, the tensor is flattened or pooled in order to get a vector. Formally, the MulDiP-Net representation for the query image I τ i is thus computed as:

R Ω i,τ = F k∈ 0,ω Z(φ K k (I τ i )) , (7)
where F is the fusion operator among the ω+1 input vectors, and Z is a normalization function that returns normalized representations. In practice for the normalization function, we choose the L-infinite norm L-∞ and for the fusion functions, we used the concatenation k∈ 0,ω (reasons discussed in next paragraph).

Our goal was to combine features trained separately on specific and generic labels in order to learn more universal representations at near-zero cost of annotation, because we hypothesized that their learned features can be complementary. However, because of the different training-data, the networks have different behaviors after training. Indeed, once they are trained, they react differently on the same input images. More precisely, the features learned on specific data tend to fire with higher values than those learned on generic data. Thus, naively concatenating these sub-representations would restrict the fused representation to the activations of the dominating features (those learned with specific labels) with an additional noise from dominated features (those learned with generic labels), which will result to a degradation of performance. Hence, the independent normalization step is crucial, because it aims to homogenize the scales of the sub-representations. The same problem of dominating values has been observed in [START_REF] Liu | Parsenet: Looking wider to see better[END_REF], [START_REF] Wang | Growing a brain: Fine-tuning by increasing model capacity[END_REF], who solved it with a similar normalization process.

FSFT as a Dimensionality Reduction Method

The more SPs we consider in MulDiP-Net, the more universal representation we get. However, because of its concatenation fusion, the dimensionality of the fused representation linearly increases with the amount of SPs considered. Also, MulDiP-Net is an ensemble-like method since it is implemented such that it contains as many networks as the number of SP considered. This point is generally translated in terms of the amount of parameters in the final model. Indeed, while the MulDiP-Net aims at building more universal representations, it contains ω+1 times more parameters than a standard network. In summary, the actual form of MulDiP-Net faces two problems: (ii) the high capacity of its resulting model; and (ii) the high dimensionality of its resulting representation. To alleviate these drawbacks, we proposed to rely on the FSFT method (introduced in Sec. 3) and even more, on its use as a dimensionality reduction method. To do so, we roughly re-train each of the subnetworks of MulDiP-Net on the same initial SP, but with the last layers (on which FSFT focuses the retraining) containing much less neurons. Compared to the classical FSFT, the "FSFT as a Dimensionality Reduction method" (FSFT-DR) results in: (i) a final model that has much lower capacity and (ii) a representation with a lower dimensionality, at a nearzero decrease of performance, but that still significantly increases Comparison of all the universality evaluation-metrics -Avg baseline, RG [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF], VDC [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF] and ours, namely BC and mNRG -according all the criteria mentioned (and highlighted in bold) in Sec. 5.

the performances compared to the initial network (experiments of Sec. 6.3). Obviously such a behavior is also observed on the MulDiP+FSFT-DR (FSFT-DR applied on each subnetwork of MulDiP-Net). More formally, let us consider a features-extractor φ k trained on a certain SP noted D S k with a network architecture N k that has a penultimate layer of dimensionality dim(φ k (•))=n. FSFT-DR consists to re-train (as depicted in Eq. 2 about the FSFT re-training) the last layers of φ k (which results in a new features-extractor denoted φ k ) on the same source-problem D S k but with a different network-architecture N k that has a penultimate layer of a dimensionality dim(φ k (•))=n lower than those of N k (i.e., n <n). Note that in the classical formulation of FSFT n =n. Note also that, while by construction FSFT-DR reduces the dimensionality, it also reduces the number of parameters of the final network N k (since N k contains less neurons and thus less weights than N k ). It is all the more true since FSFT-DR is applied on the last layers, which are generally fully-connected ones and thus contain the majority of the parameters of the CNN. For instance, AlexNet contains 62.35 millions parameters with 3.75 millions in the convolutional layers and 58.6 millions in the fully-connected (FC) ones (about 94% of the total). Regarding the application of FSFT-DR on the MulDiP-Net method, let first consider a given set N * Ω = {N * 0 , . . . , N * ω } of ω+1 trained networks. FSFT-DR is applied on each subnetwork N * k∈ 0,ω by setting the last layer of φ k to be of size n = n/(ω + 1) , with n being the dimensionality of the original sub-representation φ k . The rest of the pipeline remains the same than in MulDiP-Net, that is, extraction of the penultimate layer from each subnetwork and merging them after independent normalization. In summary, compared to MulDiP-Net, MulDiP+FSFT-DR (i) contains much less parameters and (ii) results into a smaller representation than the classical MulDiP-Net, while being much more performing.

EVALUATION OF UNIVERSALIZING METHODS

Universality is motivated by the claim of Atkinson's cognitive study [START_REF] Atkinson | The developing visual brain[END_REF]. Inspired by this, authors of [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] evaluated the universality of representations by their ability to simultaneously cover a large range of domains, like objects, faces, animals, etc. However, their evaluation scheme does not completely match with [START_REF] Atkinson | The developing visual brain[END_REF] since learning and testing are conducted on the same problem and only the image distribution differs (learn on train images and evaluate on test ones). However, in [START_REF] Atkinson | The developing visual brain[END_REF], the terms "reused later in life" mean that universality implies to work well on many but different problems than those used to learn the representation. Hence, in this paper, we propose to consider this through the scheme of transfer-learning (TL) that naturally fits with it. Indeed, in TL, the source-task is used to learn the representation (analogically, the universal one developed in the early years) and the target-tasks are used to evaluate it (analogically, the problems solved later in life). Moreover, to better match the claim of [START_REF] Atkinson | The developing visual brain[END_REF] (re-use "as-is"), we propose to not modify the representation (do not fine-tune it) for each target-task, but rather learn a simple taskpredictor on top of it. Nevertheless, when humans develop their representation, they do not have access to future problems, thus the target-tasks should not be available during the learning of the representation. What makes our evaluation framework unique is that it lies in a TL scheme, closer to [START_REF] Atkinson | The developing visual brain[END_REF] than those of [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] which lies in a classical end-to-end learning scheme.

Evaluating universality requires to aggregate the scores of a method on multiple target-problems. This step is not straightforward, since metrics of each problem could be different (e.g., error-rate, mAP, F1-score) or even be increasing (e.g., recall) or decreasing (e.g., median rank). Thus, naively averaging the scores would result into incoherences. To get coherent-aggregation, Subramanian et al. [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF] proposed to average the gain over the scores of a reference universal representation across the set of multiple problems: relative gain (RG) computed as:

U RG i = 1 P P j=1 s ref j -s i j , with s ref j
and s i j being respectively the performance of a reference method M ref and the method to evaluate M i on the problem P j . Such aggregation is coherent, but depends on a reference method that needs to be arbitrarily and manually chosen. Rebuffi et al. [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF] went further by identifying one additional criterion for universality: the metric should better rewards significant gain compared to the reference scores (called significance criterion). To do so, they proposed the Visual Decathlon Challenge (VDC), which is computed as: U V DC i = P j=1 α j max{0, E max j -E j } γj , with E max j and E j respectively being the error-rate of the actual method M i and the best reachable one on the problem P j , α j is just a normalizing factor, and γ j 1 models the significance.

In addition to the properties already put on evidence, we introduce four more criteria. The first criteria (merit-bonus) indicates that the metric should reward proportionally with the score of the reference method. For instance a jump of 1% is more rewarding if it starts from a reference at 90% than one at 10%. The second criteria (penalty for damage) is motivated by the fact that a method could improve the performance on some tasks but decrease it on others. In such case, the metric should penalize them. The third proposed criteria (penalty malus) lies with the fact that, the aggregated improvement (over the reference) of some methods, could be lower than the aggregated decrease. Such method should also be penalized. A last important criterion is that the metric should be independent to outlier methods, in the sense that it should detect methods that are well suited to a given benchmark (gain a lot of points) that could compensate (when aggregated) a insignificant gain on the other benchmarks.

To respect these constraints, we propose the median normalized relative gain (mNRG):

U mN RG i = median j∈ 1,P (s j - s ref j )/(s max j -s ref j ).
The numerator is the simple RG metric, thus it respects the same criteria (coherent aggregation and penalty for damage because negative scores are allowed). The denominator acts as a normalization and naturally handles the merit-bonus and penalty malus criteria. Finally, instead of an average to aggregate, we used the median to make our metric independent to outliers.

Note that in VDC, negative values are not allowed (because of the max), thus it does not respect the penalty for damage nor the penalty malus. In addition, the significance criterion is modeled through a power γ, which not only prevents the merit-bonus, but even exhibits the inverse behavior. Indeed, VDC will reward more a method that gains 3% starting from a reference at 10%, than one that gains 2.5% starting from 90%.

Finally, none of the above metrics is independent to a reference method, thus we propose an alternative metric that we call Borda Count (BC), which is based on a voting method and the ordinal scale, with each benchmark considered as an independent voter. First, each voter can use its own measure to estimate the performance of the methods, which provides coherent aggregation. Second, it becomes insensitive to some outliers such as the exceptional fit of a method to a particular benchmark. Moreover, we consider that reporting the number of times a method M 1 is better than a method M 2 can be a more reliable information than the average difference of scores (as long as these score are actually comparable). Note that, BC lacks the consistency with time since its score differs with the chosen comparison methods. Formally, let us consider M methods to rank, relying on the information provided by P problems. Each method is then ranked according to each problem, resulting into a rank r j i with M ∈ 1, i and P ∈ 1, j . It is converted into a score M -r j i that is itself averaged to give the final score of the method: S i = P j=1 M -r j i . All the metrics are compared in Table 1 according to all the mentioned criteria. None of them address all the criteria but the proposed mNRG is the most complete of them.

EXPERIMENTAL RESULTS

Experimental Settings

Transfer-Learning Protocol

As mentioned above, the evaluation of all the methods is carried in a transfer-learning scheme on multiple target-problems. The methods are trained with standard architectures: AlexNet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] (default), VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and DarkNet [START_REF] Redmon | Yolo9000: Better, faster, stronger[END_REF]). By default, we train on ILSVRC* (see Sec. 6.1.2) because it is smaller than ILSVRC, making the process faster. The network on the source-problem is used to extract the image signatures on the target-problem. Here we focus on the classification task for the target-problems and learn each of their classes with a one-vs-all linear SVM classifier. Note that, as mentioned in Sec. 5, to match the claim of [START_REF] Atkinson | The developing visual brain[END_REF], there is no fine-tuning. The cost parameter of the classifiers is optimized on each dataset through cross-validation with the usual train/val splits. The performances of each target-problem are evaluated with standard splittings and metrics, namely the mean Average Precision (mAP) for multi-label datasets and Accuracy (Acc.) for mono-label ones. To evaluate the universality, we use the proposed mNRG evaluation-metric (Sec. 5).

Source and Target-datasets

For the source-problem, we used two subsets of ImageNet [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]: ILSVRC and ILSVRC* that contains half of the former. The characteristics of the datasets are presented on top of Table 2. Regarding the target-problems, we used ten image classification benchmarks from different visual domains (actions, scenes, objects, birds, plants, etc.). Specifically, five of them contain coarse categories -Pascal VOC 2007 (VOC07) [START_REF] Everingham | The pascal visual object classes challenge[END_REF], Pascal VOC 2012 (VOC12) [START_REF] Everingham | The pascal visual object classes challenge[END_REF], Caltech-101 (CA101) [START_REF] Fei-Fei | One-shot learning of object categories[END_REF], Caltech-256 (CA256) [START_REF] Griffin | Caltech-256 object category dataset[END_REF] and Nus-Wide Objects (NWO) [START_REF] Chua | Nus-wide: A real-world web image database from national university of singapore[END_REF] -, three contain fine categories -Stanford Cars (CARS) [START_REF] Krause | 3d object representations for fine-grained categorization[END_REF], CUB-200 Birds (CUB) [START_REF] Wah | The caltechucsd birds-200-2011 dataset[END_REF] and Flowers-102 (FLO) [START_REF] Nilsback | Automated flower classification over a large number of classes[END_REF] -, one contain scenes -MIT Indoor 67 (MIT67) [START_REF] Quattoni | Recognizing indoor scenes[END_REF] -and one contain actions -Stanford Actions (stACT) [START_REF] Yao | Human action recognition by learning bases of action attributes and parts[END_REF]. Their characteristics are presented at the bottom of Table 2. For all the benchmarks, we follow standard protocols (i.e., common splits and metrics). For VOC12 and stCAR, we used the official evaluation-servers.

Datasets (1) (2) (3) (4) (5)

Implementation Details

For the SPV, we started with ILSVRC (and ILSVRC*) as initial source-problem and variated at generic levels with three generic grouping methods: categorical, hierarchical and clustering. For the two first methods, we used the ImageNet hierarchy in the functions of partitioning (Eq. ( 5)) and re-labeling (Eq. ( 6)). Practically, in the first method, a part of the categorical-level categories of ILSVRC is obtained from the list released in [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF] and the other part is re-labeled by ourselves as depicted in [START_REF] Tamaazousti | Mucalenet: Multi categorical-level networks to generate more discriminating features[END_REF]. This latter results into 480 generic categories for the 1, 000 specific ones of ILSVRC (200 generic for the 483 specific of ILVRC*). For the hierarchical-method, we follow the bottom-up approach of [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF] and re-labeled the categories to higher levels of the ImageNet hierarchy. For the clustering method, we follow [START_REF] Chami | Amecon: Abstract meta concept features for text-illustration[END_REF] and clustered the data of the categories with a Kmeans algorithm (K from 50 to 300 with steps of 50). Regarding the extraction and combination of the features (Eq. 7), we used one label-set per grouping SPV (i.e., basic-level, 7 th hierarchical and the clustering with K=100). The latter choice results in a set of four SPs (including the initial one). For the extraction of the representations of images of target-tasks, we always use the penultimate layer from each subnetwork. Regarding the normalization step before combining the representations, we used the infinite-norm (L ∞ ). In the FSFT methods, we choose the following settings: L=6, meaning that we focus the retraining on the two last layers; η 2 =10 -2 , as the learning-rate used to train the original network; and α=0.1, meaning that we train the last layers 10 times faster than the firsts. Note that, for FSFT we also use the penultimate layer as representation of the target-images.

Comparison to the State-Of-The-Art

We compare the proposed FSFT and MulDiP+FSFT to state-ofthe-art methods that could be used as universalizing methods:

• REFERENCE [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF]: A CNN trained on the initial sourceproblem, that contains specific categories (here 483). Since it is a classical method that works quite well on many problems, we use it as reference to evaluate universality of other methods. • SPV spe A [START_REF] Azizpour | Factors of transferability for a generic convnet representation[END_REF], [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Zhou | Learning deep features for scene recognition using places database[END_REF]: A method that consists in a adding SPV followed by the training on the obtained SP. Specifically, we added 100 specific categories (randomly obtained from the leaf nodes of ImageNet) with their 100K images. Simply said, the method is trains a network on 583 specific classes.

• SPV gen A [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF], [START_REF] Mettes | The imagenet shuffle: Reorganized pre-training for video event detection[END_REF], [START_REF] Tamaazousti | Vision-language integration using constrained local semantic features[END_REF]: Same as the previous method but with a generic adding SPV that adds 100 generic categories (with their 100K images). This results in training a network on 583 specific and generic classes. The generic ones were obtained from random internal-nodes of the ImageNet hierarchy.

• WhatMakes [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF]: A grouping SPV followed by the training of a network on the obtained SP. Specifically, the grouping SPV corresponds to a relabeling of specific categories into internal hierarchical-levels of the ImageNet hierarchy. We performed it for all the levels and report the results for the best one (6 th level starting from the leaf-node's one).

• AMECON [START_REF] Chami | Amecon: Abstract meta concept features for text-illustration[END_REF]: Similar to the previous method but differs by its kind of grouping SPV. Indeed, the grouping is performed by clustering. Specifically, all the images of each specific categories are used to compute the mean features (obtained through the f c7 layer of the pre-trained reference network) for the categories. Then, a Kmeans algorithm is used to cluster this set of category mean features. We applied this method with different amount of clusters (K from 50 to 300 with a step of 50) and report the best results (K=100).

• ISM [START_REF] Wu | Deep convolutional neural network with independent softmax for large scale face recognition[END_REF]: This method trains an ensemble-model with N networks, one for each SP obtained from a splitting SPV.

Here, we applied the method with half splitting (we split the initial SP in two balanced subsets). Note that, we chose two subsets to limit the method to a maximum of two networks for fair comparisons. Once the networks trained, we normalize and concatenate the features extracted from each of them. • GrowBrain-WA [START_REF] Wang | Growing a brain: Fine-tuning by increasing model capacity[END_REF]: This recent method consists to finetune a trained network on the same source-problem it was trained originally, by growing the network capacity (wider or deeper). The best setting is the width augmented (WA) growing that consists to add 2, 048 neurons to the f c7 layer. We also implemented their normalization and scaling step for the new and old layers, because they are crucial to make this method performing. The final representation corresponds to the 6, 192-dimensional f c7 layer. GrowBrain-RWA is an extended version that performs a recursive growing of the network capacity. The best setting they report is to add 1, 024 neurons on the f c6 layer and 2, 048 on the f c7 one. • MuCaLe-Net [START_REF] Tamaazousti | Mucalenet: Multi categorical-level networks to generate more discriminating features[END_REF]: It consists to perform a normalization step followed by the concatenation of the features extracted from two CNN-models, one trained on data labeled according specific categories and one according categorical-levels. It results in a 8, 192-dimensional representation.

[37], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] explicitly tackled the universal representations problem and could be used for comparison. However, it is important (top). The comparison is carried in a transfer-learning scheme on ten target-datasets. Methods are compared in terms of their individual scores on each benchmark (with standard metrics) and especially their aggregated scores, with our nMRG (blue scores in last column). The universalizing methods are compared to a reference one for which we colored its scores in red. In each column, we highlight the highest score in bold, and the second one is underlined. All the methods have been learned with the same AlexNet architecture on the same initial SP (ILSVRC*).

Method VOC07 VOC12 CA101
to note that their goal was to improve universality by adding data from multiple domains, while our goal is to improve universality from a fixed set of data (one domain), making their methods not comparable to ours, since they use more annotated data.

As depicted in Sec. 5, the methods are evaluated in terms of the proposed mNRG score, in a transfer-learning scheme on a set of ten target-datasets from different domains. The results of the comparison are presented in Table 3. As expected, the methods that consist to add images and their annotations (SPV spe A and SPV gen A ) as well as those that increase the capacity of the network (GrowBrain) aim to learn more universal representation compared to the reference method (positive mNRG score). Surprisingly, the ISM method is not as performing as reported in their context. This might be due to the fact that it is designed for very large sourceproblems, and the half-million images used here are not sufficient, highlighting a clear limitation of their method. The same behavior can be observed with AMECON and WhatMakes. The former could be because the specific categories we used (leaf node of Ima-geNet) are not as specific as the ones (captions) used in their paper. The latter was rather a study that highlighted that a network trained on data labeled among generic categories is almost as performing as one trained on specific categories, through the evaluation on three target-tasks distributed among three domains (general objects, actions and scenes). However, we clearly observe here that on more domains, and especially fine-grained objects, such behavior is not observed anymore (highly negative mNRG score). Finally, we observe that our MulDiP+FSFT method significantly performs better than all other methods (highest mNRG score), meaning that it is clearly the most performing universalizing method. Note that, it also significantly outperforms MuCaLe-Net (by 2 points of mNRG), which clearly highlight the interest of combining MulDiP-Net with the proposed FSFT method.

Another salient observation is that our FSFT only is quite powerful, especially because it outperforms the methods that consist to add data and their annotations (SPV spe A and SPV gen A ) as well as one that increase the network-capacity (GrowBrain-WA). It is worth noting that FSFT does need more data neither more capacity. Thus, it increases universality at zero cost of capacity and annotation, which is quite promising.

Comparison with Baseline Methods

Here, we take further experiments to analyse the performances achieved by the proposed methods (MulDiP-Net and FSFT), through their comparison to several baseline-methods. All the baselines are described below, illustrated in Fig. 6 and the results are reported in Table 4. Note that, in this section (and supplementary), the comparisons are conducted on eight of the ten target-datasets presented in Sec. 6.1 (VOC12 and stCAR removed because evaluation-servers limited to 1 run per day needed).

We first assess whether the gain of universality obtained by our MulDiP-Net method is caused by the ensemble-model component.

To do so, we compare it to a baseline that consists in an ensemblemodel with two network trained on the same specific SP but with different random initializations of the weights (Ensemble). While Ensemble is significantly increasing universality compared to the reference, our MulDiP-Net is provides much better results. This latter, means that the performances of our method does not come from the ensemble-model aspect only, but also by the combination of features learned with data labeled among generic and specific categories. Hence, another baseline is to compare our method without the ensemble-model, but by jointly training a network on the two SPs (generic and specific). Indeed, we tested three variants: (i) training the set of SPs with a sum of softmax losses, i.e., one for each SP (Multi-Task); (ii) training the set of SPs with a multi-label loss layer (e.g., hinge loss), where the labels for each image contain both annotations, namely generic and specific (Multi-Label); and (iii) recursively training the SPs by training the network on the generic SP, then continuing the training on the specific SP (Recursive). Globally, the first two baselines, which belongs to the joint-training approach, strongly hurts performance compared to the reference method. The Recursive method is almost as performing as the reference one and we assume this is due to catastrophic forgetting [START_REF] Robert | Catastrophic forgetting in connectionist networks[END_REF], i.e., when the network is trained on the second specific problem, it forgets the features learned on the former generic one. In summary, the latter baselines clearly demonstrate the utility of the independent-learning (in our method) compared to the joint one. In other words, the growing capacity of the ensemble-model aspect is crucial to make our method benefiting from multiple SPs.

Method

For our FSFT method, we mainly compare it to the two methods (namely, Frozen Self Training (FrST) and Self Fine-Tuning (SFT)) studied by [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF], that also re-train a network on the same problem. More precisely, FrST consists in retraining the last layers only with the previous layers being "frozen", while FST re-trains all the layers with the same learning-rate. As highlighted in [START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF], we observe a performance drop of FrST compared to the reference. As mentioned, in Sec. 3, this is due to the fragile coadaptation neurons learned in the original network. A slight drop of performance is also observed for FST, meaning that fine-tuning does not always recovers all the co-adapted neurons. In contrast, our FSFT method increases performance, even when we compact its representation to 2, 048 (FSFT DR ), clearly highlighting its capacity to recover co-adapted neurons of the original network and even the training of others.

Finally, we also assess the utility of combining MulDiP-Net with FSFT and even considering FSFT as a dimensionality reduction method (FSFT DR ), compared to the reference and especially the MulDiP-Net method. As shown above, MulDiP+FSFT performs much better than MulDiP-Net, but more surprisingly, when MulDiP-Net is combined with FSFT DR , it is almost as performing as the former, at a much lower capacity. Hence, by using FSFT as a dimensionality reduction method (FSFT DR ), we not only get a jump of performance (compared to MulDiP-Net), but also significantly alleviate the higher capacity produced by the ensemble-model aspect of MulDiP-Net.

Let note that supplementary materials are available and roughly contain: (i) a comparison of our methods to the stateof-the-art according all the universality evaluation-metrics of the literature; (ii) an evaluation of the impact of more and different grouping SPVs used in our MulDiP-Net; and (iii) the evaluation of MulDiP-Net with more training data and deeper architectures.

CONCLUSION

In this paper, we proposed four contributions: (i) a new challenge of learning more universal representations from a fixed set of data (domains and tasks); (ii) the evaluation of universality in a more suitable scheme (transfer-learning), as well as a new metric respecting most of the highlighted desirable criteria; (iii) a new method based on the re-training of networks, by focusing the training on some parameters; and finally (iv) a new method based on a general formalism of source problem variation and training of multiple networks. We demonstrated the effectiveness of our universalizing methods, in a transfer-learning scheme, through our evaluation-metric, on ten target-datasets from different domains. An in-depth analysis has also been conducted to highlight some important insights of our methods. We hope that our contributions will further support the creation of other methods to get more universal representations and open doors for many less explored aspects of transfer-learning such as, learning the source-problem, using Human knowledge or even through more realistic multimodal and dynamic environments such as HOME [START_REF] Brodeur | Home: A household multimodal environment[END_REF].

APPENDIX

In the Sec. 5 of the main paper, we proposed the mNRG universality evaluation metric and used it for the comparison of our methods with state-of-the-art. However, since our metric is novel, it is important to also perform the same comparison according the metrics of the literature, and compare the advantages and drawbacks of each metric. Such comparison is provided in Table 5 and should be analyzed with Table 3 of the main paper, which contain the detailed results on each benchmark. First of all, from the results we see that our MulDiP+FSFT method is the best universalizing method regardless the evaluation-metric. Moreover, still regardless the evaluation-metric, our FSFT is quite promising, since it significantly outperforms SP V spe A , SP V gen G and GrowBrain-WA at zero cost of annotation and without adding any additional parameter.

Regarding the comparison of the metrics, we can observe that our mNRG metric respects some of the properties highlighted in Sec. 5 of the main paper (e.g., merit bonus, penalty for damage, penalty malus), which is not the case for the baseline Avg, the RG [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF], the VDC [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], the BC and the aNRG (being our method with an average operator instead of the median). For instance, compared to Avg and RG that gives almost the same universality scores to GrowBrain-WA and FSFT, our mNRG is able to give more points to FSFT since it gives significantly better results than GrowBrain-WA on seven of the ten datasets. Moreover, beside the significantly better results on the seven benchmarks, mNRG gives only +0.5 compared to GrowBrain-WA, since it penalizes its loose of performance on the NWO dataset.

We also observe that the VDC do not penalize the methods that performs less than the reference on some benchmarks (e.g., it gives to our MulDiP+FSFT almost twice the universality-score than MuCaLe-Net, while compared to the former, the latter never decreases performance in any of the benchmarks), while our mNRG is able to penalize it (MulDiP+FSFT outperforms MuCaLe-Net by only 2.1 points in terms of our mNRG metric). This is even more visible on the ISM method that should clearly give negative universality scores. Indeed, compared to the reference, ISM gives lower results on nine of the ten benchmarks (higher results only on the MIT67 benchmark), but since VDC do not perform penalty for damage, it gives 0.0 points on the the nine benchmarks and 0.9 points on the MIT67 one, which undesirably results in a positive universality-score. Moreover, VDC perform neither penalty for damage nor penalty malus, and as a consequence, is unable to say which method between AMECON and WhatMakes performs worse. A metric that has such ability (say method A is worse than B, even if they are both lower than the reference) could be interesting in a case, were for example, the methods A and B have some practical advantages compared to the reference and we would like to know which of these practically advantageous methods should be used as a reference for improving universality. Finally, regarding the VDC metric, we observe that compared to the scores (around 2000) reported in their paper, the scores reported in this experiment are much lower (around 100). It is important to note that, this is due to the fact that our evaluation scheme (transfer-learning: training the representation in the source-problem and evaluate on target-problem unseen during training) is much more challenging than theirs (end-toend learning: training the representation in the source-problem and evaluate on the test-set of the same source-problem). Simply said, while we do not have access to the target-problems during the learning of the representation, they have, making it easier.

We also observe that compared to aNRG, our mNRG is able to decrease the SPV spe A to a similar universality score than SPV gen G , since the former seems to be well suited on some datasets like CA101 and CA256 (compared to other absolute improvements). Finally, while not visible here, by construction, the Avg do not provide coherent aggregation. For the same reason, our BC do not provide the same results according the comparison methods, making it not consistent with time. Note however that, as the best universality metric (our mNRG), BC has some good advantages like penalty for damage or independence to outliers.

Our MulDiP-Net method is based on a grouping SPV using categorical-levels. Here, we assess what is the impact of using different grouping methods. In particular, we compared it to grouping based on hierarchical-levels [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF] of WordNet, clustering ones [START_REF] Chami | Amecon: Abstract meta concept features for text-illustration[END_REF] and also random. For every grouping method (Random, Clustering, WordNet and our Categorical), we also compare MulDiP-Net to each of its subnetworks alone -i.e., the one trained on specific classes (Net-S) and the one trained on the generic ones (Net-G). In the main paper, we always used only two levels for fair comparisons, but as depicted in Sec. 4.4 of the main paper, our method could benefit from multiple levels. Thus, we implemented MulDiP-Net with more levels, namely Multi-3: initial specific SP and generic SPs obtained from from categorical and Wordnet grouping SPVs; and Multi-4: same as Multi-3 with an additional clustering-based grouping SPV. The results are presented in Fig. 7.

From the results, a first observation is that, whatever the grouping SPV, the Net-G is much less performing than Net-S, which contradicts the work of [START_REF] Huh | What makes imagenet good for transfer learning?[END_REF] (limited to few domains on target-datasets). Even if below than Net-S, our categorical one is the best grouping SPV, clearly highlighting the interest to introduce a grouping inspired by cognitive studies. Second, whatever the grouping, MulDiP-Net always performs better than its subnetworks (Net-G and especially the reference Net-S), which demonstrates the interest of combining specific and generic knowledge, in the way we do it. Third, in MulDiP-Net, while the best results are achieved with our categorical grouping (confirming its interest), it is worth noting that, the performance of random grouping is very close to Net-S, which highlights the utility of semantic grouping SPV. Finally, it is clearly observable that, the more levels we use in MulDiP-Net, the better performance we get.

Increasing network capacity (wider or deeper layers) can be a very efficient universalizing method, since it can learn to perceive more elements or configuration through its new features. However, it is important to note that, it is not easy to modify the architecture (many costly experiments are needed to set all the hyper-parameters as well as the architecture itself) and no certainty of convergence is promised. In all cases, our contribution is orthogonal to this domain, and our aim here is to demonstrate this orthogonality. To do so, we implemented the reference, as well as our MulDiP-Net method with three popular architectures, namely the basic AlexNet (5 convolutional and 2 fully-connected layers), the deep and wide VGG-16 (16 convolutional and 2 fully-connected layers) and the fast and very-deep DarkNet-20 (20 convolutional layers followed by average pooling). Another important question is whether our approach of learning from a fixed set of training data could benefit from more data if they are available (adding-data approach. Thus, in this experiment, instead of using ILSVRC* (containing half-million images and 483 categories) as the initial source-problem, we used the whole ILSVRC which contains 1.2M images and 1K categories. The results of these experiments are presented in Table 6.

Four observations can be made. First, even with twice more data than in Table 3, MulDiP-Net still significantly increases universality compared to the reference. This demonstrates the orthogonality of our approach with the works that adds more data (domains [START_REF] Bilen | Universal representations: The missing link between faces, text, planktons, and cat breeds[END_REF], [START_REF] Rebuffi | Learning multiple visual domains with residual adapters[END_REF], [START_REF] Sylvestre-Alvise Rebuffi | Efficient parametrization of multi-domain deep neural networks[END_REF] or tasks [START_REF] Subramanian | Learning general purpose distributed sentence representations via large scale multi-task learning[END_REF]). Second, the deeper architecture do not learn the more universal representation (Net-S with VGG-16 is better than Net-S with DarkNet-20). This clearly highlights that, compared to diversifying the source-problem, naively increasing the capacity is not safe for improving universality. Third, we clearly observe that MulDiP-Net outperforms its subnetworks regardless the architecture, which demonstrates that our approach could benefit from the field of network architectures. Last but not least, we can observe that Net-G is always below Net-S, except for DarkNet. This is surprising since one could have the intuition that the finer categories we use for training, the better results we get. However, it seems that this depends on the architecture, or maybe on the ratio between the number of units in the representation and the number of classes used for training.

Fig. 2 .

 2 Fig. 2. Focused Self Fine-Tunning (FSFT). An initial network (gray branch) denoted initNet (first layer's weights denoted θ a 1 , last ones θ a 2 and the classifier Ψ a ) is trained on a source-database, by minimizing the loss-function L. Once initNet trained, another network (black branch) denoted FSFT (first layer's weights denoted θ b 1 , last ones θ b 2 and the classifier Ψ b ) is trained on the same source-database and solves the same problem (minimize same loss L). The weights θ b 1 (black block filled in gray) of the first layers of FSFT are initialized with the firsts of initNet (θ a 1 ) and weakly updated through training. The weights of the last layers θ b 2 (black blocks filled in white) are randomly initialized and fully trained.

  ) where S I = W × H × D corresponds to the size of images (i.e., width W , height H and depth D with D = 3 for RGB images), (x k i , y k i ) is an element of the new SP D k , with each training image x k i labeled according the label-set

Fig. 3 .

 3 Fig. 3. Illustration of our MulDiP-Net method. Let consider an initial source problem (SP) D S 0 consisting in a set of images and their associated labels (x 0 i , y 0 i ) i∈ 1,N . MulDiP-Net consists in three phases: (i) variation (ϑ) of the initial SP (D 0 ) into new ones (D k ); (ii) training networks (φ and Ψ) on the whole set of SPs ({D 0 , D k }); and (iii) normalization (Z) followed by combination (F) of the features extracted from each trained network in order to form a more universal representation. More precisely, phase (i) is a source problem variation (SPV) (ϑ k 0 ) applied on the initial SP (D S 0 ), which outputs a new SP (D S k,k>0). After applying K SPV functions, we get a set of K+1 SPs containing the new SPs and the initial one (only one variation illustrated here, thus K=1). Phase (ii) consists in the learning of one network for each of the K+1 SPs, resulting in a set of K+1 trained networks: {N k } k= 0,K . Each network N k (that is a composition of a features-extractor φ k and a classifier Ψ k ) is trained by minimizing the loss function L k computed using the output of the predictor Ψ k and the ground-truth of the SP D S k . Phase (iii) consists in the extraction of the more universal representations R i,τ from images I τ i of a target-task τ . Specifically, it passes the images I τ i into the trained networks and gets features (through the extractors φ k ) that are independently normalized (Z) and fused (F) in order to output the final representation R i,τ .

  To describe the training of our Multi Discriminative Problem Network (MulDiP-Net), let us first consider a set of SPsD S Ω = {D S 0 , D S 1 , • • • , D S ω }, with D S 0 being the initial SP and D k>0 , being the ω SPs obtained from ω different SPVs functions applied on the initial SP (as depicted in the previous section). MulDiP-Net consists in training one network per SP D S k,k∈ 0,ω , with a network architecture N k and a learning procedure P k , ∀k ∈ 0, ω (i.e., including the initial SP). Indeed, each SP D

Fig. 5 .

 5 Fig.5. Illustration of our grouping SPV. Given a set of specific categories (leaf nodes of the hierarchy H in white diamonds) and a set of generic categories (here c 1 i ) at a certain level (here categorical denoted B cat L ), our grouping SPV ϑ G consists in three steps: computation of all descendants of c 1 i according to the H (1); computation of the descendants that belong to the initial set of categories (2), producing a group G 0 j ; and relabeling of the categories (as well as their images) of the latter group (3) into the categories of B cat L . The first two points correspond to Eq. (5), while the last one corresponds to Eq. (6).

Fig. 6 .

 6 Fig. 6. Illustration of the different baseline methods.

Fig. 7 .

 7 Fig. 7. Impact of the different grouping SPV and more levels considered in our MulDiP-Net. Net-S (red dashed line) is used as reference.

TABLE 1

 1

	Criterion	Avg	RG	VDC	BC	mNRG
	Coherent aggr.
	Significance
	Merit bonus
	Penalty malus
	Penalty for damage
	Indep. to outliers
	Indep. to reference
	Time consistency

TABLE 3

 3 Comparison of our methods (bottom) to state-

of-the-art universalizing-methods

TABLE 4

 4 Comparison of our universalizing-methods with baseline methods. Methods are compared in terms of their mNRG score (last column in blue). All the methods have been learned with the same AlexNet architecture on the same initial SP (ILSVRC*). Methods marked with * are ours.

TABLE 5

 5 Comparison to state-of-the-art, according different universality evaluation metrics (those mentioned in Sec. 5 of the main paper). Note that, for a set of 11 methods and 10 datasets, the best achievable BC score is 110, while the worse is 10.

	Method	Network	VOC07 CA101 CA256 NWO MIT67 stACT CUB FLOW mAP Acc. Acc. mAP Acc. Acc. Acc. Acc.	mNRG
	Net-S (Ref.)	AlexNet	71.7	79.7	62.4	58.3	46.9	51.2	36.3	58.4	0.0
	Net-G	AlexNet	71.5	77.4	60.4	57.8	42.8	49.3	19.5	52.4	-7.7
	MulDiP-Net	AlexNet	74.4	82.5	65.2	60.8	47.4	54.2	36.1	62.5	+7.4
	Net-S	VGG-16	86.1	88.8	78.0	71.8	66.7	73.5	69.8	78.9	+44.8
	Net-G	VGG-16	85.7	87.6	76.9	70.3	65.8	72.2	67.0	75.0	+38.9
	MulDiP-Net	VGG-16	87.5	92.0	80.9	72.6	68.9	75.0	71.5	81.9	+55.3
	Net-S	DarkNet-20	82.7	91.0	78.4	70.5	64.8	72.2	59.5	80.0	+38.9
	Net-G	DarkNet-20	83.2	91.5	78.1	73.2	64.4	72.6	52.5	78.9	+40.6
	MulDiP-Net DarkNet-20	84.1	92.7	80.1	73.9	66.4	74.5	61.2	82.1	+47.1

TABLE 6

 6 MulDiP-Net performances with different network architectures and more training data. To compute the mNRG scores (last column in blue), we used the Net-S of AlexNet as reference. All the methods have been learned on the same initial SP (whole ILSVRC).