Maxime Bucher

Stéphane Herbin

Frédéric Jurie

Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification

Keywords: zero-shot learning, attributes, semantic embedding

This paper addresses the task of zero-shot image classification. The key contribution of the proposed approach is to control the semantic embedding of images -one of the main ingredients of zero-shot learning -by formulating it as a metric learning problem. The optimized empirical criterion associates two types of sub-task constraints: metric discriminating capacity and accurate attribute prediction. This results in a novel expression of zero-shot learning not requiring the notion of class in the training phase: only pairs of image/attributes, augmented with a consistency indicator, are given as ground truth. At test time, the learned model can predict the consistency of a test image with a given set of attributes, allowing flexible ways to produce recognition inferences. Despite its simplicity, the proposed approach gives state-of-the-art results on four challenging datasets used for zero-shot recognition evaluation.

Introduction

This paper addresses the question of zero-shot learning (ZSL) image classification, i.e., the classification of images belonging to classes not represented by the training examples. This problem has attracted much interest in the last decade because of its clear practical impact: in many applications, having access to annotated data for the categories considered is often difficult, and requires new ways to increase the interpretation capacity of automated recognition systems.The efficiency of ZSL relies on the existence of an intermediate representation level, effortlessly understandable by human designers and sufficiently formal to be the support of algorithmic inferences. Most of the studies have so far considered this representation in the form of semantic attributes mainly because it provides an easy way to describe compact yet discriminative descriptions of new classes.

It has also been observed [START_REF] Mahajan | A joint learning framework for attribute models and object descriptions[END_REF][START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] that attribute representations as provided by humans may not be the ideal embedding space because it can lack the informational quality necessary to conduct reliable inferences: the structure of the attribute manifold for a given data distribution may be rather complex, redundant, noisy and unevenly organized. Attribute descriptions, although semantically meaningful and useful to introduce a new category, are not necessarily isomorphic to image data or to image processing outputs.

To compensate for the shortcomings induced by attribute representations, the recent trends in ZSL studies are aiming at better controlling the classification inference as well as the attribute prediction in the learning criteria. Indeed, if attribute classifiers are learned independently of the final classification task, as in the Direct Attribute Prediction model [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], they might be optimal at predicting attributes but not necessarily at predicting novel classes.

In the work proposed in this paper, we instead suggest that better controlling the structure of the embedding attribute space is at least as important than constraining the classification inference step. The fundamental idea is to empirically disentangle the attribute distribution by learning a metric able to both select and transform the original data distribution according to informational criteria. This metric is obtained by optimizing an objective function based on pairs of attributes/images without assuming that the training images are assigned to categories; only the semantic annotations are used during training. More specifically, we empirically validated the idea that optimizing jointly the attribute embedding and the classification metric, in a multi-objective framework, is what makes the performance better, even with a simple linear embedding and distance to mean attribute classification.

The approach is experimentally validated on 4 recent datasets for zero-shot recognition, i.e. the 'aPascal&aYahoo', 'Animals with Attributes', 'CUB-200-2011 and 'SUN attribute' datasets for which excellent results are obtained, despite the simplicity of the approach.

The rest of the paper is organized as follows: Section 2 presents the related works, Section 3 describes the proposed approach while the experimental validation is given in Section 4.

Related work

Visual features and semantic attributes

Image representation -i.e. the set of mechanisms allowing to transform image raw pixel intensities into representations suitable for recognition tasks -plays an important role in image classification. State-of-the art image representations were, a couple of years ago, mainly based on the pooling of hard/soft quantized local descriptors (e.g. SIFT [START_REF] Lowe | Distinctive image features from scale-invariant keypoints[END_REF]) through the bag-of-words [START_REF] Csurka | Visual categorization with bags of keypoints[END_REF] of Fisher vectors [START_REF] Sánchez | Image classification with the Fisher vector: Theory and practice[END_REF] models. However, the work of Krizhevsky et. al. [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] has opened a new area and most of the state-of-the-art image descriptors nowadays rely on Deep Convolutional Neural Networks (CNN). We follow this trend in our experiments and use the so-called 'VGG-VeryDeep-19' (4096-dim) descriptors of [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF].

Two recent papers have exhibited existing links between CNN features and semantic attributes. Ozeki et al. [START_REF] Ozeki | Understanding Convolutional Neural Networks in Terms of Category-Level Attributes[END_REF] showed that some CNN units can predict some semantic attributes of the 'Animals with Attributes' dataset fairly accurately. One interesting conclusion of their paper is that the visual semantic attributes can be predicted much more accurately than the non-visual ones by the nodes of the CNN. More recently, [START_REF] Escorcia | On the Relationship between Visual Attributes and Convolutional Networks[END_REF] showed the existence of Attribute Centric Nodes (ACNs) within CNNs trained to recognize objects, collectively encoding information pertinent to visual attributes, unevenly and sparsely distributed across all the layers of the network.

Despite these recent findings could certainly make the performance of our method better, we don't use them in our experiments and stick to the use of standard CNN features, with the intention of making our results directly comparable to the recent zero-shot learning papers (e.g. [START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF]).

Describing images by semantic and non-semantic attributes

Zero-shot learning methods rely on the use of intermediate representations, usually given as attributes. This term can, however, encompass different concepts. For Lampert et al. [START_REF] Lampert | Attribute-Based Classification for Zero-Shot Visual Object Categorization[END_REF] it denotes the presence/absence of a given object property, assuming that attributes are nameable properties (color or presence or absence of a certain part, etc.). The advantage of so-defined attributes is that they can be used easily to define new classes expressed by a shared semantic vocabulary.

However, finding a discriminative and meaningful set of attributes can sometimes be difficult. [START_REF] Parikh | Interactively building a discriminative vocabulary of nameable attributes[END_REF][START_REF] Duan | Discovering localized attributes for fine-grained recognition[END_REF] addressed this issue by proposing an interactive approach that discovers local attributes both discriminative and semantically meaningful, employing a recommender system that selects attributes through human interactions. An alternative for identifying attribute vocabulary without human labeling is to mine existing textual description of images sampled from the Internet, such as proposed by [START_REF] Berg | Automatic attribute discovery and characterization from noisy web data[END_REF]. In the same line of thought, [START_REF] Ba | Predicting deep zero-shot convolutional neural networks using textual descriptions[END_REF] presented a model for classifying unseen categories from their (already existing) textual description. [START_REF] Elhoseiny | Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions[END_REF] proposed an approach for zero-shot learning where the description of unseen categories comes in the form of typical text such as an encyclopedia entries, without the need to explicitly define attributes.

Another drawback of human generated attributes is that they can be redundant or not adapted to image classification. These issues have been addressed by automatically designing discriminative category-level attributes and using them for tasks of cross-category knowledge transfer, such as in the work of Yu et al. [START_REF] Yu | Designing category-level attributes for discriminative visual recognition[END_REF]. Finally, the attributes can also be structured into hierarchies [START_REF] Verma | Learning hierarchical similarity metrics[END_REF][START_REF] Rohrbach | Evaluating knowledge transfer and zero-shot learning in a large-scale setting[END_REF][START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF] or obtained by text mining or textual descriptions [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Ba | Predicting deep zero-shot convolutional neural networks using textual descriptions[END_REF][START_REF] Elhoseiny | Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions[END_REF].

Beside these papers which all consider attributes as meaningful for humans, some authors denoted by attributes any latent space providing an intermediate representation between image and hidden descriptions that can be used to transfer information to images from unseen classes. This is typically the case of [START_REF] Wang | A Discriminative Latent Model of Object Classes and Attributes[END_REF] which jointly learn the attribute classifiers and the attribute vectors, with the intention of obtaining a better attribute-level representation, converting undetectable and redundant attributes into discriminative ones while retaining the useful semantic attributes. [START_REF] Farhadi | Describing objects by their attributes[END_REF] has also introduced the concept of discriminative attributes, taking the form of random comparisons. The space of class names can also constitute an interesting embedding, such as in the works of [START_REF] Mensink | COSTA: Co-Occurrence Statistics for Zero-Shot Classification[END_REF][START_REF] Norouzi | Zero-Shot Learning by Convex Combination of Semantic Embeddings[END_REF][START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF][START_REF] Fu | Zero-shot object recognition by semantic manifold distance[END_REF] which represent images as mixtures of known classes distributions.

Finally, it worth pointing out that the aforementioned techniques are restricting attributes to categorical labels and dont allow the representation of more general semantic relationships. To counter this limitation, [START_REF] Parikh | Relative attributes[END_REF] proposed to model relative attributes by learning a ranking function.

Zero shot learning from semantic and attribute embedding

As defined by [START_REF] Palatucci | Zero-shot learning with semantic output codes[END_REF], the problem of zero-shot learning can be seen as the problem of learning a classifier f : x → y that can predict novel values of y not available in the training set. Most of existing methods rely on the computation of a similarity or consistency function linking image descriptors and the semantic description of the classes. These links are given by learning two embeddings -the first from the image representation to semantic space and the second from the class space to the semantic space -and defining a way to describe the constraints between the class space and the image space, the two being strongly interdependent.

DAP (Direct Attribute Prediction) and Indirect attribute prediction (IAP), first proposed by [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], use the between layer of attributes as variables decoupling the images from the layer of labels. In DAP, independent attribute predictors are used to build the embedding of the image, the similarity between two semantic representations (one predicted from the image representation, one given by the class) being given as the probability of the class attribute knowing the image. In IAP, attributes form a connecting layer between two layers of labels, one for classes that are known at training time and one for classes that are not known. In this case, attributes are predicted from (known) class predictions. Lampert et al. [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] concluded that DAP gives much better performance than IAP.

However, as mentioned in the introduction, DAP has two problems: first, it does not model any correlation between attributes, each being predicted independently. Second, the mapping between classes and the attribute space does not weight the relative importance of the attributes nor the correlations between them. Inspired by [START_REF] Weston | WSABIE: scaling up to large vocabulary image annotation[END_REF], which learns a linear embedding between image features and annotations, [START_REF] Akata | Label-Embedding for Image Classification[END_REF] and [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] tried to overcome this limitation. The work of Akata et al. [START_REF] Akata | Label-Embedding for Image Classification[END_REF] introduced a function measuring the consistency between an image and a label embedding, the parameters of this function is learned to ensure that, given an image, the correct classes rank higher than the incorrect ones. This consistency function has the form of a bilinear relation W associating the image embedding θ(x) and the label representation φ(y) as S(x, y; W ) = θ(x) t W φ(y). Romera et al. [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF] proposed a simple closed form solution for W , assuming a specific form of regularization is chosen. In comparison to our work none of the two papers ( [START_REF] Akata | Label-Embedding for Image Classification[END_REF], [START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF]) use a metric learning framework to control the statistical structure of the attribute embedding space.

The coefficients of the consistency constraint W can be also predicted from a semantic textual description of the image. As an example, the goal of [START_REF] Elhoseiny | Write a Classifier: Zero-Shot Learning Using Purely Textual Descriptions[END_REF] is to predict a classifier for a new category based only on the learned classes and a textual description of this category. They solve this problem as a regression function, learnt from the textual feature domain to the visual classifier domain. [START_REF] Ba | Predicting deep zero-shot convolutional neural networks using textual descriptions[END_REF] builds on these ideas, extending them by using a more expressive regression function based on a deep neural network. They take advantage of the architecture of CNNs and learn features at different layers, rather than just learning an embedding space for both modalities. The proposed model provides means to automatically generate a list of pseudo-attributes for each visual category consisting of words from Wikipedia articles.

In contrast with the aforementioned methods, Hamm et al. [START_REF] Hamm | Probabilistic Zero-shot Classification with Semantic Rankings[END_REF] introduced the idea of ordinal similarity between classes (eg. d('cat','dog') < d('cat', 'automobile')), claiming that not only this type of similarity may be sufficient for distinguishing cat and truck, but also that it seems a more natural representation since the ordinal similarity is invariant under scaling and monotonic transformation of numerical values. It is also worth mentioning the work of Jayaraman et al. [START_REF] Jayaraman | Zero-shot recognition with unreliable attributes[END_REF] which proposed to leverage the statistics about each attribute error tendencies within a random forest approach, allowing to train zero-shot models that explicitly account for the unreliability of attribute predictions.

Wu et al. [START_REF] Wu | Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts[END_REF] exploit natural language processing technologies to generate event descriptions. They measure their similarity to images by projecting them into a common high-dimensional space using text expansion. The similarity is expressed as the concatenation of L 2 distances of the different modalities considered. Strictly speaking, there is no metric learning involved but a concatenation of L 2 distances. Finally, Frome et al. [START_REF] Frome | DeViSE: A Deep Visual-Semantic Embedding Model[END_REF] aim at leveraging semantic knowledge learned in the text domain, and transfer it to a model trained for visual object recognition by learning a metric aligning the two modalities. However, in contrast to our work, [START_REF] Frome | DeViSE: A Deep Visual-Semantic Embedding Model[END_REF][START_REF] Wu | Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts[END_REF] do not explicitly control the quality of the embedding.

Zero-shot learning as transductive and semi-supervised learning

All the previously mentioned approaches consider that the embedding and the consistency function have to be learned from a set of training data of known classes, used in a second time to infer predictions about the images of new classes not available during training. However, a different problem can be addressed when images from the unknown classes are already available at training time and can hence be used to produce a better embedding. In this case, the problem can be cast as a transductive learning problem i.e. the inference of the correct labels for the given unlabeled data only, or a semi-supervised learning problem i.e. the inference of the best embedding using both labeled and unlabeled data.

Wang and Forsyth [START_REF] Wang | Joint learning of visual attributes, object classes and visual saliency[END_REF] proposed MIL framework for jointly learning attributes and object classifiers from weakly annotated data. Wang and Mori [START_REF] Wang | A Discriminative Latent Model of Object Classes and Attributes[END_REF] treated attributes of an object as latent variables and captured the correlations among attributes using an undirected graphical model, allowing to infer object class labels using the information of both the test image and its (latent) attributes. In [START_REF] Mahajan | A joint learning framework for attribute models and object descriptions[END_REF], the class information is incorporated into the attribute classifier to get an attribute-level representation that generalizes well to unseen examples of known classes as well as those of the unseen classes, assuming unlabeled images are available for learning. [START_REF] Socher | Zero-Shot Learning Through Cross-Modal Transfer[END_REF] considered the introduction of unseen classes as a novelty detection problem in a multi-class classification problem. If the image is of a known category, a standard classifier can be used. Otherwise, images are assigned to a class based on the likelihood of being an unseen category.

Fu et al. [START_REF] Fu | Transductive multi-view embedding for zero-shot recognition and annotation[END_REF] rectified the projection domain shift between auxiliary and target datasets by introducing a multi-view semantic space alignment process to correlate different semantic views and the low-level feature view, by projecting them onto a latent embedding space learnt using multi-view Canonical Correlation Analysis. More recently Li et al. [START_REF] Li | Semi-Supervised Zero-Shot Classification With Label Representation Learning[END_REF] learned the embedding from the input data in a semi-supervised large-margin learning framework, jointly considering multi-class classification over observed and unseen classes. Finally, [START_REF] Kodirov | Unsupervised Domain Adaptation for Zero-Shot Learning[END_REF] formulated a regularized sparse coding framework which used the target domain class label projections in the semantic space to regularize the learnt target domain projection, with the aim of overcoming the projection domain shift problem.

Zero-shot learning as a metric learning problem

Two contributions, [START_REF] Mensink | Metric learning for large scale image classification: Generalizing to new classes at near-zero cost[END_REF] and [START_REF] Kuznetsova | Exploiting view-specific appearance similarities across classes for zero-shot pose prediction: A metric learning approach[END_REF], exploit metric learning to Zero shot class description. Mensink et al. [START_REF] Mensink | Metric learning for large scale image classification: Generalizing to new classes at near-zero cost[END_REF] learn a metric adapted to measure the similarity of images, in the context of k-nearest neighbor image classification, and apply it in fact to One Shot Learning to show it can generalize well to new classes. They don't use any attribute embedding space nor consider ZSL in their work. Kuznetsova et al. [START_REF] Kuznetsova | Exploiting view-specific appearance similarities across classes for zero-shot pose prediction: A metric learning approach[END_REF] learn a metric to infer pose and object class from a single image. They use the expression zero-shot to actually denote a (new) transfer learning problem when data are unevenly sampled in the joint pose and class space, and not a Zero-Shot Classification problem where new classes are only known from attribute descriptions.

As far as we know, zero-shot learning has never been addressed explicitly as a metric learning problem in the attribute embedding space, which is one of the key contributions of this paper.

Method

Embedding consistency score

Most of the inference problems can be cast into an optimal framework of the form:

Y * = arg min Y∈Y S(X, Y)
where X ∈ X is a given sample from some modality, e.g. an image or some features extracted from it, Y * is the most consistent association from another modality Y, e.g. a vector of attribute indicators or a textual description, and S is a measure able to quantify the joint consistency of two observations from the two modalities. In this formulation, the smaller the score, the more consistent the samples. One can think of this score as a negative likelihood. When trying to design such a consistency score, one of the difficult aspects is to relate meaningfully the two modalities. One usual approach consists in embedding them into a common representational space A1 where their heterogeneous nature can be compared. This space can be abstract, i.e. its structure can be obtained from some optimization process, or semantically interpretable e.g. a fixed list of attributes or properties each indexed by a tag referring to some shared knowledge or ontology, leading to a p-dimensional vector space. Let ÂX (X) and ÂY (Y) be the two embeddings for each modality X and Y , taking values in X and Y and producing outputs in A.

In this work, it is proposed to define the consistency score as a metric on the common embedding space A. More precisely, we use the Mahalanobis like description of a metric parametrized by a linear mapping W A :

d A (A 1 , A 2 ) = (A 1 -A 2 ) T W A 2 ,
assuming that the embedding space is a vector space, and define the consistency score as:

S(X, Y) = d A ( ÂX (X), ÂY (Y)) = ( ÂX (X) -ÂY (Y)) T W A 2 .
The Mahalanobis mapping W A can be interpreted itself as a linear embedding in an abstract m-dimensional vector space where the natural metric is the Euclidean distance, and acts as a multivariate whitening filter. It is expected that this property will improve empirically the reliability of the consistency score (1) by choosing the appropriate linear mapping.

We are now left with two questions: how to define the embedding? How build the Mahalanobis mapping? We see in the following that these two questions can be solved jointly by optimizing a unique criterion.

Embedding in the attribute space

The main problem addressed in this work is to be able to discriminate a series of new hypotheses that can only be specified using a single modality, the Y one with our notations. In many Zero-Shot Learning studies, this modality is often expressed as the existence or presence of several attributes or properties from a fixed given set. The simplest embedding space one can think of is precisely this attribute space, implying that the Y modality embedding is the identity: ÂY (Y) = Y with A = Y. In this case, the consistency score simplifies as:

S(X, Y) = ( ÂX (X) -Y) T W A 2 (1)
The next step is to embed the X modality into Y directly. We suggest using a simple linear embedding with matrix W X and bias b X , assuming that X is in a d-dimensional vector space. This can be expressed as:

ÂX (X) = max(0, X T W X + b X ). ( 2
)
We use a reLu-type output normalization to keep the significance of the attribute space as property detectors, negative numbers being difficult to interpret in this context.

In the simple formulation proposed here, we do not question the way new hypotheses are specified in the target modality, nor use any external source of information (e.g. word vectors) to map the attributes into a more semantically organized space such as in [START_REF] Socher | Zero-Shot Learning Through Cross-Modal Transfer[END_REF]. We leave the problem of correcting the original attribute description to the construction of the metric in the common embedding space.

Metric learning

The design problem is now reduced to the estimation of three mathematical objects: the linear embedding to the attribute space W X of dimensions d × p, a bias b X of dimension p, and the Mahalanobis linear mapping W A of dimensions p × m, m being a free parameter to choose.

The proposed approach consists in building empirically those objects from a set of examples by appling metric learning techniques. The training set is supposed to contain pairs of data (X i , Y i ) sampling the joint distribution of the two modalities: X i is a vector representing an image or some features extracted from it, while Y i denotes an attribute-based description. Notice that we do not introduce any class information in this formulation: the link between class and attribute representations is assumed to be specified by the use case considered.

The rationale behind the use of metric learning is to transform the original representational space so that the resulting metric takes into account the statistical structure of the data using pairwise constraints. One usual way to do so is to express the problem as a binary classification on pairs of samples, where the role of the metric is to separate similar and dissimilar samples by thresholding (see [START_REF] Bellet | A Survey on Metric Learning for Feature Vectors and Structured Data[END_REF] for a survey on M.L.). It is easy to build pairs of similar and dissimilar examples from the annotated examples by sampling randomly (uniformly or according to some law) the two modalities X and Y and assigning an indicator Z ∈ {-1, 1} stating whether Y i is a good attribute description of X i (Z i = 1) or not (Z i = -1). Metric learning approaches try to catch a data-dependent way to encode similarity. In general, the data manifold has a smaller intrinsic dimension than the feature space, and is not isotropically distributed.

We are now given a dataset of triplets {(X i , Y i , Z i )} N i=1 , the Z indicator stating that the two modalities are similar, i.e. consistent, or not2 . The next step is to describe an empirical criterion that will be able to learn W X , b X and W A . The idea is to decompose the problem in three objectives: metric learning, good embedding and regularization.

The metric learning part follows a now standard hinge loss approach [START_REF] Shalev-Shwartz | Online and batch learning of pseudometrics[END_REF] taking the following form for each sample:

l H (X i , Y i , Z i , τ ) = max 0, 1 -Z i (τ -S(X i , Y i ) 2 ) . (3)
The extra parameter τ is free and can also be learned from data. Its role is to define the threshold separating similar from dissimilar examples, and should depend on the data distribution. The embedding criterion is a simple quadratic loss, but only applied to similar data:

l A (X i , Y i , Z) = max(0, Z i ). Y i -ÂX (X i ) 2 2 . ( 4
)
Its role is to ensure that the attribute prediction is of good quality, so that the difference Y -ÂX (X) reflects dissimilarity due to modality inconsistencies rather than bad representational issues.

The size of the learning problem (d × p + p + p × m) can be large and requires regularization to prevent over fitting. We use a quadratic penalization:

R(W A , W X , b X ) = W X 2 F + b X 2 2 + W A 2 F (5)
where . F is the Frobenius norm.

The overall optimization criterion can now be written as the sum of the previously defined terms:

L(W A , W X , b X , τ ) = i l H (X i , Y i , Z i , τ ) + λ i l A (X i , Y i , Z i ) +µR(W A , W X , b X ) (6)
where λ and µ are hyper-parameters that are chosen using cross-validation. Note that the criterion (6) can also be interpreted as a multi-objective learning approach since it mixes two optimal but dependent issues: attribute embedding and metric on the embedding space.

To solve the optimization problem, we do not follow the approach proposed in [START_REF] Shalev-Shwartz | Online and batch learning of pseudometrics[END_REF] since we also learn the attribute embedding part W X jointly with the metric embedding W A . We use instead a global stochastic gradient descent (see section 4 for details).

Application to image recognition and retrieval

The consistency score ( 1) is a versatile tool that can be used for several image interpretation problems. Section 4 will evaluate the potential of our approach on three of them.

Zero-shot learning

The problem can be defined as finding the most consistent attribute description given the image to classify, and a set of exclusive attribute class descriptors

{Y * k } C k=1
where k is the index of a class:

k * = arg min k∈{1...C} S(X, Y * k ) (7)
In this formulation, classifying is made equivalent to identifying between the C classes the best attribute description. A variant of this scheme can exploit a voting process to identify the best attribute among a set of k candidates, inspired from a k-nearest neighbor approach.

Few-shot learning Learning a metric in the embedding space can conveniently be used to specialize the consistency score to new data when they are available. We study a simple fine tuning approach using stochastic gradient descent on criterion (6) applied to novel triplets (X, Y, Z) from unseen classes only, starting with the model learned with seen classes. This makes "few-shot learning"possible. The decision framework is identical to the ZSL one.

Zero-shot retrieval The score (1) can also used to retrieve the data from a given database that have at least a consistent level λ with a given query defined in the Y (or A) modality:

Retrieve(A, λ) = {X ∈ X / S(X, A) < λ}
The performance is usually characterized by precision-recall curves.

Experiments

This section presents the experimental validation of the proposed method. The section first introduces the 4 datasets evaluated as well as the details of the experimental settings. The method is empirically evaluated on three different tasks as described in section 3: Zero-Shot-Learning (ZSL), Few-Shot Learning (FSL) and Zero-Shot Retrieval (ZSR). The ZSL experiments aim at evaluating the capability of the proposed model to predict unseen classes. This section also evaluates the contribution of the different components of the model to the performance, and makes comparisons with state-of-the-art results. In the FSL experiments, we show how the ZSL model can serve as good prior to learning a classifier when only a few samples of the unknown classes are available. Finally, we evaluate our model on a ZSR task, illustrating the capability of the algorithm to retrieve images using attribute-based queries.

Datasets and Experimental Settings

The experimental valuation is done on 4 public datasets widely used in the community, allowing to compare our results with those recently proposed in the literature: the aPascal&aYahoo (aP&Y) [START_REF] Farhadi | Describing objects by their attributes[END_REF], Animals with Attributes (AwA) [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], CUB-200-2011 (CUB) [START_REF] Wah | The Caltech-UCSD Birds-200-2011 Dataset[END_REF] and SUN attribute (SUN) [START_REF] Patterson | The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding[END_REF] datasets (see Table 1 for few statistics on their content). Theses datasets exhibit a large number of categories (indoor and outdoor scenes, objects, person, animals, etc.) and attributes (shapes, materials, color, parts, etc.)

These datasets have been introduced for training and evaluating ZSL methods and contain images annotated with semantic attributes. More specifically, each image of the aP&Y, CUB and SUN datasets has its own attribute description, meaning that two images of the same class can have different attributes. This is not the case for AwA where all the images of a given class share the same attributes. As a consequence, in the ZSL experiments on aP&Y, CUB and SUN, the attribute representation of unknown classes, required for class prediction, is taken as their mean attribute frequencies.

In order to make comparisons with previous works possible, we use the same training/testing splits as [START_REF] Farhadi | Describing objects by their attributes[END_REF] (aP&Y), [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] (AwA), [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF] CUB and [START_REF] Jayaraman | Zero-shot recognition with unreliable attributes[END_REF] (SUN).

Regarding the representation of images, we used both the VGG-VeryDeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] and AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] CNN models, both pre-trained on imageNet -without fine tuning to the attribute datasets -and use the penultimate fully connected layer (e.g., FC7 4096-d layer for VGG-VeryDeep-19) for representing the images. Very deep CNN models act as generic feature extractors and have been demonstrated to work well for object recognition. They have been also used in many recent ZSL experiments and we use exactly the same descriptors as [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF].

One of the key characteristics of our model is that it requires a set of image/attributes pairs for training. Positive (resp. negative) pairs are obtained by taking the training images associated with their own provided attribute vector (resp. by randomly assigning attributes not present in the image) and are assigned to the class label '1 (resp. '-1). In order to bound the size of the training set we generate only 2 pairs per training image, one positive and one negative.

Our model has three hyper-parameters: the weight λ, the dimensionality of the space in which the distance is computed (m) and the regularization parameters µ. These hyper-parameters are estimated through a grid search validation procedure by randomly keeping 20% of the training classes for cross-validating the hyper-parameters, and choosing the parameters giving best accuracy for these so-obtained validation classes. The parameter are searched in the following ranges: m ∈ [20%, 120%] of the initial attribute dimension, λ ∈ [0.05, 1.0] and µ ∈ [0.01, 10.0. τ is a parameter learned during training.

Once the hyper-parameters are tuned, we take the whole training set to learn the final model and evaluate it on the test set (unseen classes in case of ZSL).

The optimization of W A and W X is done with stochastic gradient descent, the parameters being initialized randomly with normal distribution. The size of the mini-batch is of 100. As the objective function is non-convex, different initializations can give different parameters. We addressed this issue by doing 5 estimations of the parameters starting from 5 different initializations and selecting the best one on a validation set (we keep a part of the train set for doing this and fine-tune the parameters on the whole train set when the best initialization is known).We use the optimizer provided in the TensorFlow framework [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. Using the GPU mode with a Nvidia 750 GTX GPU, learning a model (W A Table 2: Zero-shot classification accuracy (mean ± std). We report results both with VGG-verydeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] and AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] features for fair comparisons, whenever it's possible (AwA images are not public anymore preventing the computation of their AlexNet representations). and W X ) takes 5-10 minutes for a given set of hyper-parameters. Computing image/attribute consistency takes around 4ms per pair.

Zero-Shot Learning Experiments

The experiments follow the standard ZSL protocol: during training, a set of images from known classes is available for learning the model parameters. At test time, images from unseen classes are processed and the goal is to find the class described by an attribute representation most consistent with the images. Table 2 gives the performance of our approach on the 4 datasets consideredexpressed as multi-class accuracy -and makes comparisons with state-of-the-art approaches. The performances of previous methods are taken from [START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF][START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Zhang | Zero-shot learning via joint latent similarity embedding[END_REF]. Performance is reported with 2 different features i.e. VGG-VeryDeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] and AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] for fair comparisons. As images of AwA are not public anymore it is only possible to use the features available for download. On the four datasets our model achieves above state-of-the-art performance (note : [START_REF] Zhang | Zero-shot learning via joint latent similarity embedding[END_REF] was published after our submission), with a noticeable improvement of more than 8% on aP&Y.

As explained in the previous section, our model is based on a multi-objective function trying to maximize metric discriminating capacity as well as attribute prediction. It is interesting to observe how the performance degrades when one of the two terms is missing. In Table 2, the 'Ours w/o ML' setting makes use of the Euclidean distance i.e. W A = I. The 'Ours w/o constraint' setting is when the attribute prediction term (Eq. 4) is missing in the criterion. This term gives a 4% improvement, on average.

Figure 1a shows the accuracy as a function of the embedding dimension. This projection maps the original data in a space in which the Euclidean distance is good for the task considered. It can be seen as a way to exploit and select the correlation structure between attributes. We experimented that the best performance is generally obtained when the dimension of this space less than 40% smaller than the size of the initial attribute space.

Few-Shot Learning

Few-shot learning corresponds to the situation where 1 (or more) annotated example(s) from unseen classes are available at test time. In this case our model is first trained using only the seen classes (same as with ZSL), and we introduced the examples from unseen class data one by one before fine-tuning the model parameters by doing a few more learning iterations using these new data only.

Figure 1b shows the accuracy evolution, given as a function of the number of additional images from the unseen classes. Please note that for the SUN dataset we have used a maximum of 10 additional examples as unseen classes contain only 20 images. We observed that knowing even a very few number of annotated examples significantly improves the performance. It is a very encouraging behavior for large-scale applications where annotations for a large number of categories are hard and expensive to get.

Zero-Shot Retrieval

The task of Zero-Shot image Retrieval consists in searching an image database with attribute-based queries. For doing this, we first train our model as for standard ZSL. We then take the attribute descriptions of unseen classes as queries, and rank the images from the unseen classes based on the similarity with the query. Table 3 reports the mean average precision on the 4 datasets. Our model outperforms the state-of-the-art SEE method [START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF] by more than 10% on average. Figure 2 shows the average precision for each class of the 4 datasets. In the aP&aYa dataset, the 'donkey', 'centaur' and 'zebra' classes have a very low average precision. This can be explained by the strong visual similarity between these classes which only differ by a few attributes.

Conclusion

This paper has presented a novel approach for zero-shot classification exploiting multi-objective metric learning techniques. The proposed formulation has the nice property of not requiring any ground truth at the category level for learning a consistency score between the image and the semantic modalities, but only requiring weak consistency information. The resulting score can be used with versatility on various image interpretation tasks, and shows close or above stateof-the-art performance on four standard benchmarks. The formal simplicity of the approach allows several avenues for future improvement. A first one would be to provide a better embedding on the semantic side of the consistency score ÂY (Y). A second one would be to explore more complex functions than the linear mappings tested in this work, and introduce, for instance, deep network architectures.

Fig. 1 :

 1 Fig. 1: (a) ZSL accuracy as a function of the dimensionality of the metric space; best results are obtained when the dimension of the metric embedding is less than 40% of the image space dimension. It also shows the improvement due to the attribute prediction term in the objective function. (b) Few-shot learning: Classification accuracy (%) as a function of the amount of training examples per unseen classes.

Fig. 2 :

 2 Fig. 2: Precision Recall curve for each unseen class by dataset. For CUB dataset we randomly choose 10 classes (best viewed on a computer screen).

Table 1 :

 1 Dataset statistics

	Dataset	#Training classes #Test classes #Instances #Attributes
	aPascal & aYahoo [23]	20	12	15,339	64
	Animals with Attributes [3]	40	10	30,475	85
	CUB 200-2011 [44]	150	50	11,788	312
	SUN Attributes [45]	707	10	14,340	102

Table 3 :

 3 Zero-Shot Retrieval task: Mean Average Precision (%) on the 4 datasets aP&Y AwA CUB SUN Av. Zhang et al. [11] 15.43 46.25 4.69 58.94 31.33 Ours (VGG features) 36.92 68.1 25.33 52.68 45.76

We use the letters A and A in our notations since we will focus on the space of attribute descriptions as the embedding space.

To make notations simpler, we do not rename or re-index from the original dataset the pairs of data for the similar and dissimilar cases.


Mateusz Koziński
email: mateusz.kozinski@unicaen.fr

Loïc Simon
email: loic.simon@ensicaen.fr

Frédéric Jurie
email: frederic.jurie@unicaen.fr

Semi-Supervised Training of Structured Output Neural Networks with an Adversarial Loss

We propose a method for semi-supervised training of structured-output neural networks. Inspired by the framework of Generative Adversarial Networks (GAN), we train a discriminator network to capture the notion of 'quality' of network output. To this end, we leverage the qualitative difference between outputs obtained on the labelled training data and unannotated data. We then use the discriminator as a source of error signal for unlabelled data. This effectively boosts the performance of a network on a held out test set. Initial experiments in image segmentation demonstrate that the proposed framework enables labelling two times less data than in a fully supervised scenario, while achieving the same network performance.

Introduction

The unprecedented power that neural networks offer when applied to vision problems comes at a cost of large volumes of annotated training data required from training. When the annotations are produced manually the process can be laborious and costly, especially for structured output problems like image segmentation. In this paper we propose an approach to semi-supervised training of structured output neural networks. The proposed approach allows to capitalize on large sets of unlabelled data. We show that the performance of a network trained in a fully supervised regime on a certain amount of labelled data can be matched by using a significantly smaller amount of labelled data, together with a sufficiently large volume of unlabelled data. In consequence, significant labelling effort can be saved. In technical terms, we generate a useful error signal for data for which no ground truth labels are available, by means of adversarial training. During training, both the labelled training data and the unlabelled data is forwarded through the network. The network produces qualitatively better output on the labelled images than on the unlabelled images. Much like in training a Generative Adversarial Network (GAN), we train a discriminator network to cap-ture this difference. The negative gradient of the discriminator with respect to its unlabelled input is used as the error signal for the unlabelled data. Our technical contribution consists in an adversarial learning approach for semi-supervised training of structured output neural networks. A particular advantage of our method is that it can be applied to any structured output problem, independently of the architecture of the applied predictor. Contrary to pre-training, the proposed method can be applied to a complete network, not just to its featureextracting part.

Related work

Our work is related to previous efforts to use unannotated data for training neural networks, including autoencoders, self-supervised learning and the use of GANs. A considerable research effort has been devoted to autoencoders [7,[START_REF] Masci | Stacked convolutional autoencoders for hierarchical feature extraction[END_REF] -neural networks that encode an image into a latent representation, from which they attempt to reconstruct the original image. Different regularization techniques are applied to impose useful properties on the hidden representation. The encoder of a trained autoencoder is considered a useful 'feature extractor'. In a pre-training scenario [START_REF] Zeiler | Adaptive deconvolutional networks for mid and high level feature learning[END_REF][START_REF] Ranzato | Unsupervised learning of invariant feature hierarchies with applications to object recognition[END_REF] the encoder of a trained autoencoder is incorporated as a feature extractor into another network, which is then fine-tuned for a particular task on labelled data. In a semi-supervised scenario [START_REF] Zhao | Stacked what-where auto-encoders[END_REF][START_REF] Zhang | Augmenting supervised neural networks with unsupervised objectives for large-scale image classification[END_REF] parameters are shared between an encoder of an autoencoder and a feature extractor of a supervised network, and both are trained simultaneously. Plain autoencoders attempt to encode all the visual information in the latent representation. It has been hypothesised that much of the information is irrelevant for particular vision tasks, and autoencoders that transfer some of it between the input and the output [START_REF] Ranzato | Unsupervised learning of invariant feature hierarchies with applications to object recognition[END_REF][START_REF] Zeiler | Adaptive deconvolutional networks for mid and high level feature learning[END_REF][START_REF] Zhang | Augmenting supervised neural networks with unsupervised objectives for large-scale image classification[END_REF], instead of encoding everything in the latent representation, produce more useful representations. A recent example of such architecture is the ladder network [START_REF] Valpola | From neural {PCA} to deep unsupervised learning[END_REF][START_REF] Rasmus | Semi-Supervised Learning with Ladder Networks[END_REF], where the critical information content that should be encoded in the latent representation is learnt in a semi-supervised setting. One drawback of autoencoders is that they constrain the ar-chitecture of the supervised network to be the same as that of the encoder. While the convolutional autoencoders [START_REF] Masci | Stacked convolutional autoencoders for hierarchical feature extraction[END_REF] with pooling layers [START_REF] Ranzato | Unsupervised learning of invariant feature hierarchies with applications to object recognition[END_REF][START_REF] Zeiler | Adaptive deconvolutional networks for mid and high level feature learning[END_REF][START_REF] Zhang | Augmenting supervised neural networks with unsupervised objectives for large-scale image classification[END_REF] match the architectures of contemporary image classification networks [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] well, they can only be matched to an initial part of a structured output network, for example one used for image segmentation [START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF][START_REF] Badrinarayanan | A deep convolutional encoderdecoder architecture for image segmentation[END_REF]. In consequence, the other part of a network does not benefit from the unlabelled data. This is consistent with the intuition that the 'final' part of such network, that upsamples feature maps, represents a correlation between the output variables. Such correlation cannot be learnt by an autoencoder that is never exposed to any ground truth annotations.

A number of 'self-supervised' methods of training feature extractors [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF][START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF] emerged recently. They consist in 'deconstructing' unlabelled images by removing some information, and training a neural network to reproduce the original image. The 'deconstruction' methods include masking image regions, or dividing an image into tiles and shuffling the resulting tiles. The corresponding reconstruction tasks include inpainting the masked regions based on the context [START_REF] Pathak | Context encoders: Feature learning by inpainting[END_REF] and guessing a relative position of two or more tiles [START_REF] Doersch | Unsupervised visual representation learning by context prediction[END_REF][START_REF] Noroozi | Unsupervised learning of visual representations by solving jigsaw puzzles[END_REF]. The reconstruction requires extracting high-level information from the image, which makes the obtained 'feature extractors' useful for other vision tasks. However, from the perspective of structured-output tasks, the 'self-supervised' methods suffer from the same drawbacks as autoencoders: they constrain the architecture of the trained network and are not suitable for capturing dependencies between output variables.

Our unsupervised objective is inspired by the Generative Adversarial Networks (GANs) [START_REF] Goodfellow | Generative adversarial nets[END_REF]. In GAN, a generator network is trained to transform a random vector originating from a simple sampling distribution to a sample from a complicated target distribution. The flagship application is to train a generator to yield realistically looking images from random vectors. The interesting property of GANs is that all that is required for training the generator is a collection of vectors originating from the target distribution. The error signal is backpropagated to the generator from a discriminator network that attempts to differentiate between vectors originating from the true target distribution and the ones generated by the trained network. The generator and the discriminator are trained in an alternating manner. Theoretically, GAN training has been shown to be an instance of a saddle point problem. GANs are difficult to train in practice, and some work has been devoted to improving their behaviour during training [START_REF] Salimans | Improved techniques for training gans[END_REF], identifying architectures that work well in GANs [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF], and generalizing the discriminator from a binary classifier to an energy function [START_REF] Zhao | Energy-based generative adversarial network[END_REF].

A number of attempts at using GANs for unsupervised learning has been made recently [START_REF] Salimans | Improved techniques for training gans[END_REF][START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF]. In the simplest case, the initial layers of a discriminator are used as a feature extractor [START_REF] Radford | Unsupervised representation learning with deep convolutional generative adversarial networks[END_REF]. In the task of image classification, the generated images can constitute a new class of input im-ages [START_REF] Salimans | Improved techniques for training gans[END_REF], augmenting the total number of training images. GANs can also be used for mapping directly between two domains of interest. In this case discrimination is performed between pairs of input and output. A recent work [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] showcased learning such a mapping between artistic depictions and photographs, images taken during day and night time, or segmentations and the corresponding images. The discriminator differentiates pairs of input and the corresponding ground truth output from pairs of input and output generated by the network. This new, learned cost function is shown to give more visually plausible results than the standard L2 reconstruction loss. However, it performed worse than the baseline loss on mappings with less output ambiguity, like the mapping from images to segmentation maps. The same type of loss has been demonstrated to boost segmentation results when combined with a standard cost function [START_REF] Luc | Semantic segmentation using adversarial networks[END_REF]. The methods are fully supervised -the adversarial criterion is evaluated for the labelled training data. In contrast, we use a discriminator specifically to generate a training signal for unlabelled data. Another body of research [START_REF] Donahue | Adversarial feature learning[END_REF][START_REF] Dumoulin | Adversarially learned inference[END_REF] proves that mappings between the latent space and the data space can be learnt with just samples from both domains, and that corresponding input-output pairs are not necessarily needed. Two networks are trained simultaneously: a generator g(), producing samples from a latent representation z, and an encoder f (), inferring the latent representation from data x coming from the target distribution. Discrimination is performed between two types of pairs: a pair of generated data and the corresponding latent vector (g(z), z), and a pair of data originating from the target distribution, and the inferred latent representation (x, f (x)). At optimality g() and f () are proven to be inverses. However, the methods have only been shown to work on low-dimensional inputs and latent representations. The discrimination-based approach can also be applied to domain adaptation [START_REF] Ganin | Domainadversarial training of neural networks[END_REF][START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF]. The discriminator is learning to differentiate between features obtained for samples from two different domains, like synthetic and real images, or images representing different seasons of year. The discriminator is a source of an error signal, that makes the network produce similar features on data coming from both domains. The method can be used for object detection and image classification [START_REF] Ganin | Domainadversarial training of neural networks[END_REF], and for semantic segmentation [START_REF] Hoffman | Fcns in the wild: Pixel-level adversarial and constraint-based adaptation[END_REF]. While the network architecture used in domain adaptation is similar to ours, the concept behind the methods is substantially different. The goal of domain adaptation is to make the network insensitive to certain shift in the input data. In contrast, our goal is to regularize the network with use of the unlabelled data.

Method description

We address the problem of training a structured output network f w , parametrised with a weight vector w, to produce predictions y on input data x. The x and y can take any form as long as they can be input and output by a neural network. We are targeting a scenario in which, in addition to a set of training examples x t , t ∈ T , with annotations λ t , a volume of unlabelled examples x u , for u ∈ U, is available. To handle the unlabelled data, our approach combines a classic supervised loss l (y t , λ t ), measuring the consistency of y t and λ t , with a novel and unsupervised one l adv (y u ). The new loss term is described in section 3.1. We define a global cost consisting of two components

δ x t -→ x u --→ λ t -→ y t -→ y u -→ ← -- l (y t ,λ t ) ← -- l adv (y u ) δ(y u ) --→ ← -- -1 δ(y u )
C tot (w) =C (w) + αC adv (w) =E [l (f (x t ), λ t )] + αE [l adv (f (x u )] , (1)
where α is a constant. Training consists in determining the optimal network parameter by solving

w * = arg min w C tot (w). (2)
We describe the training algorithm in section 3.2

Adversarial loss

Defining a loss function, that measures network performance based only on its outputs seems infeasible. On the other hand, given two image segmentations, output by a neural network at two sufficiently distant training epochs, a human can spot the perceptual difference corresponding to increasing accuracy. This suggests that there might exist a measure of 'output quality', applicable at least for a certain range of accuracy. In this section, we attempt to construct such a function. In a classical setting, training a network on the labelled training data (x t , λ t ), t ∈ T results in a qualitative difference between outputs y t = f w (x t ) and outputs produced for the unseen data y u = f w (x u ), u ∈ U. Ideally, x t and x u are identically distributed, so one might think that the same holds for f w (x t ) and f w (x u ). In practice, the dimensionality of x is typically large and the training set is not necessarily representative of the variability present in some unseen data. This biases f w towards better performance on the training examples. 1 We leverage this qualitative difference to define the unsupervised cost C adv as a regularisation term that tends to close this gap.

Inspired by the GAN framework, we propose to train a discriminator network δ v , parametrised by v, to capture the qualitative difference between f w (x t ) for t ∈ T , and f w (x u ), for u ∈ U. We use a binary discriminator. We interpret the scalar output δ v (y) as the likelihood that y has been obtained from an element of the labelled training set y = f w (x t ), t ∈ T , and we interpret 1 -δ v (y) as the likelihood of y originating from the unlabelled set y = f w (x u ), u ∈ U. The optimal parameter configuration of the discriminator v * = arg min v CE disc (v) is defined in terms of the cross-entropy

CE disc (v) = (-E [log(δ v (f (x t )))] -E [log(1 -δ v (f (x u )))]).
When the discriminator is trained to optimum, its negative logarithm can be used as a local 'quality measure' for image segmentations. This is because δ v * (y) is the likelihood that y has been generated on the training set, and the outputs on the training set are qualitatively better. We therefore define the unsupervised cost as

C adv (w) =E [-log(δ v * (f w (x u ))] . (3)
Minimising (3) with respect to w drives f w towards reducing the gap between performance on labelled training and unlabelled data. It is however important to do so in a oneway manner, so as to avoid making the performance of f w degrade on the labelled examples. Clearly, we want f w to perform as well on unlabelled examples as it does on labelled ones but not the other way around. Therefore, we apply the adversarial component of the cost function only to the unlabelled data.

Algorithm

The minimization can be performed with a gradient-based optimization routine, for example SGD. The gradient of the objective consists of two components and its estimate on a training batch T and unlabelled batch U can be denoted as

∇ w C T U tot (w) = ∇ w C T (w) + α∇ w C U adv (w). ( 4
)
The gradient can be computed by backpropagation. The flow of data and gradients forward and back through the networks is depicted in Figure 3. In practice, we train the network using algorithm for IterNum = 1 to k do 4:

T ← pickBatch T , batchSize 5:

U ← pickBatch U, batchSize 6: g ← -t∈T ∇ v log(δ v (f w (x t ))) - u∈U ∇ v log(1 -δ v (f w (x u ))) 7: v ← update v, g 8:
end for

9:

T ← pickBatch T , batchSize 10:

g T ← t∈T ∇ w l (f w (x t ))
11:

U ← pickBatch U, batchSize 12:

g U ← -u∈U ∇ w log(δ v (f w (x u ))) 13:
w ← update w, g T + αg U 14: end while [START_REF] Dumoulin | Adversarially learned inference[END_REF]

Experimental evaluation

The goal of the experiments is to compare semi-supervised training with the adversarial loss to fully supervised training. In particular, we are interested in the trade-off between the labelling effort and the performance of the trained network. The question we are asking is: given a collection of training input data and having labelled a part of it, what is the benefit, in terms of network performance, of labelling a certain portion of it with respect to using it in an unsupervised manner. Knowing the answer helps to take the decision whether the expense of manpower required to label a certain portion of data is worth the increase in performance that it can bring.

We run experiments according to the following pattern. We run the baseline method on the whole training set, and on the training set consisting of 1 2 , 1 4 and 1 8 images of the original training set. Then, we apply the proposed method, with We use the segnet-basic network [START_REF] Badrinarayanan | A deep convolutional encoderdecoder architecture for image segmentation[END_REF]. It has an encoderdecoder architecture, where the encoder consists of four blocks of architecture 64c(7)-BN-ReLU-MP(2), where Nc(K) denotes a layer of N convolutional filters of size K×K, applied with output stride of one pixel in both directions, BN denotes the batch normalization, ReLU denotes the Rectified Linear Unit and MP(K) denotes the max-pooling operation performed in windows of size K×K with the output stride of K pixels. Each encoder block effectively subsamples the feature map by a factor of 2 in both dimensions. The decoder consists of four blocks of MU(2)-64c(7)-BN, where MU(K) denotes a maxunpooling layer, with the output stride of K×K, where the unpooling indeces are transfered from the symmetric maxpooling layer in the encoder. Each block of the decoder effectively upsamples the feature map by a factor of 2 in both dimensions. The output is produced by a 1x1 convolutional layer with 11 filters and a stride of 1 in both directions. We refer the reader to the original work [START_REF] Badrinarayanan | A deep convolutional encoderdecoder architecture for image segmentation[END_REF] for a more detailed explanation of the encoder-decoder architecture with coupled pooling and unpooling layers. The discriminator consists of three blocks of 64c(3,2)-BN-LReLU, followed by a global average pooling layer and the final linear layer with a single output variable. By 64c(3,2) we denote a convolutional layer consisting of 64 filters of size 3×3 and an output stride of 2×2, and LReLU denotes a rectified linear unit with the slope of 0.2 for the 'deactivated' piece. When performing the experiments we found out that an important aspect of the discriminator architecture is the global pooling layer. Its effect is similar to per-pixel discriminator ground truth used in [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] and consists in preventing the discriminator from overfitting by memorizing the contents of labelled training and unlabelled images. The discriminator is binary and trained using a cross-entropy loss. We use the basic SGD algorithm for updating both the segmentation network and the discriminator. We use momentum of 0.9 and weight decay of 0.001. For both the baseline and the proposed approach we train the network for 10 4 iterations with a learning rate of 0.1, then for 4•10 3 iterations with a learning rate of 0.05, another 4 • 10 3 with a learning rate of 0.025 and finally for 2 • 10 3 iterations with a learning rate of 0.0125. We jitter the training images by shifting them by a random number of between 0 and 32 pixels in both directions. We perform the accuracy tests on the network defined by the weight vector after the last update in this procedure, instead of cherry-picking the model using a cross validation on the validation set. We found out that this strategy gave better results for both the baseline and the proposed algorithm. When using our method, we set k = 1, that is, we update the discriminator once per every update of the trained network. We use batches of 16 training images for the baseline, and batches of 8 training and 8 unlabelled images for the semi-supervised setting. We present numerical results in table 1 and in figure 2. The baseline attains an accuracy of 49.3% Intersection-over-Union (IoU), which exceeds the performance of 47.7% reported in the original paper [START_REF] Badrinarayanan | A deep convolutional encoderdecoder architecture for image segmentation[END_REF]. We suspect the increase comes from the differences in the training protocol, including jitter and a decreasing learning rate. Our method consistently outperforms the baseline. Besides for every ratio of supervision, the performance of our network is nearly as good as the baseline using twice as many labelled examples. For a ratio of 1 4 , our approach even improves on the baseline with 1 2 annotations used.

Conclusion

In this work, we propose a novel kind of regularization technique that can be applied in a semi-supervised context. Our approach attempts to learn a good regularization cost that is guided by an adversarial scheme. The rational be-hind this choice is that, as a standard learning procedure goes, a trained network tends to perform better on training than on test data. We can therefore train a classifier to discriminate between labelled and unlabelled output distributions. Then, the likelihood estimate produced by the discriminator can be used as a signal to improve the behaviour of the main network on the unlabelled population.

We have leveraged the aforementioned principle to derive a generic framework that can be applied to a large number of typical machine learning problems where a structured output is generated. In order not to hinder the exposition, we have focused our experiments on a single study case, namely semantic segmentation. Nonetheless, our approach can be adapted seamlessly to different tasks, such as depth or normal inference. In the considered scenario, we have evaluated our method on a standard benchmark and demonstrated that our regularization achieves substantial improvements over a baseline using only labelled data.

We have also studied the evolution of the performance in response to a varying ratio of supervision. One interesting observation of this study is that one can match the performance of a supervised baseline with our semi-supervised approach using half the amount of ground truth annotations.

Figure 1 :

 1 Figure 1: The flow of data and error signals when training a structured output network f with the proposed method, presented in algorithm 1. The discriminator update is not shown in the drawing. The green line denotes the flow of labelled training data and the corresponding gradients. The red line denotes the flow of unlabelled data and the corresponding gradients. By l (y, λ) we denote the partial derivative of the loss with respect to the prediction, l (y, λ) = ∇ y l (y, λ).

1 2 , 1 4 and 1 8

 28 of the training set used for the supervised subtask, and the remaining part of the training set used in an unsupervised manner. We use the CamVid dataset, in the version used by Badrinarayanan, Kendall and Cipolla [1]. It consists of images captured by a forward-looking vehicle-mounted camera, of the size of 360 × 480 pixels. The datasets contains of 367 training, 101 validationframe and 233 test images. The set of labels consists of 11 classes. The compact size of the dataset lets us run a number of experiments in reasonable time.

Figure 2 :

 2 Figure 2: The IoU attained by segnet-basic on the CamVid dataset with respect to the number of annotations used for training. Note that the proposed method almost matches the baseline with two times as many annotations.

  1. The update(w, g) procedure accepts the network weights w and a gradient of the cost function g and performs an update on w. While we used SGD with momentum, any update rule used for training neural networks is applicable. Instead of training the discriminator to optimality at each iteration, we perform k updates of the discriminator for a single update of the network f w itself. There is no guarantee of convergence of the algorithm. However, our experiments demonstrate its practical utility.

Algorithm 1 Training a structured output network with adversarial cost for unlabelled data 1: v, w ← randInit() 2: while not converged do 3:

Table 1 :

 1 Performance of segnet-basic on CamVid w.r.t. the fraction of used annotations. The baseline is a fully supervised SGD with momentum. The proposed method uses the reported fraction of annotated data and the remaining part of the training set without annotations. The performance measures: intersection over union (IoU), average per-class recall (C) and global precision (G). HINTON, G. E., AND SALAKHUTDINOV, R. R. Reducing the dimensionality of data with neural networks. Science 313, 5786 (July 2006), 504-507.

	subs. factor:	1		1 2		1 4		1 8
		IoU C	G	IoU C	G	IoU C	G	IoU C	G
	baseline	49.3 64.6 83.5 46.2 58.0 82.6 44.3 56.7 81.0 40.0 50.9 79.1
	ours	---	47.9 60.0 83.4 47.3 58.8 82.3 43.0 53.3 81.9
	[7]

Note that we are not referring to the phenomenon of overfitting, where a mismatch of model complexity to the size of the training set can cause a situation where decreasing the training objective results in increasing the error on a held out test data, but simply to the higher performance of the network on the training set.


Jean Ogier Du Terrail
email: jean.ogier-duterrail@unicaen.frfrederic.jurie@unicaen.fr

Frederic Jurie

ON THE USE OF DEEP NEURAL NETWORKS FOR THE DETECTION OF SMALL VEHICLES IN ORTHO-IMAGES

Keywords: detection, learning, aerial-imagery

. This work supports an extensive study of the problems one might face when applying deep neural networks with low resolution and scarce data and proposes some solutions. One of the contributions of this paper is a network severely outperforming the state-of-the-art while being much simpler to implement and a lot faster than competitive approaches. We also list the limitations of this approach and provide several new ideas to further improve our results.

INTRODUCTION AND RELATED WORK

Although automatic detection of objects in images is an old problem and benchmarks have been around for a while [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2012[END_REF], the computer vision community only recently began to put the focus on detecting small objects. Modern benchmarks like [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF] that favored small objects have proven to be surprisingly challenging and to this day no approach has clearly gained the upper-hand on the others. In spite of the progress of the technologies behind satellite optics it is likely that aerial imagery pictures will always be filled with small objects with relatively low resolution. Furthermore, as aforementioned, the vehicles to detect occupy a few hundreds pixels on often very large images (around a million pixels) so most pixels/windows on such images will have to be classified as belonging to a sink class that we will call background. Those very peculiar conditions explain why one cannot directly ap-ply winning methods on [START_REF] Everingham | The PASCAL Visual Object Classes Challenge 2012[END_REF] or [START_REF] Russakovsky | ImageNet Large Scale Visual Recognition Challenge[END_REF] like [START_REF] Shaoqing Ren | Faster R-CNN: towards real-time object detection with region proposal networks[END_REF] on such benchmarks.

Modern pipelines for the detection of objects in images can be roughly classified in 3 main groups. The first one contains grid-based regression methods. The user defines a grid on the image and regress bounding boxes on the images based on offsets w.r.t this grid [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF][START_REF] Szegedy | Scalable, High-Quality Object Detection[END_REF][START_REF] Liu | SSD: single shot multibox detector[END_REF]. The second one is region proposals based methods, with most of them consisting of a cascade starting with class agnostic region proposals [START_REF] Cheng | BING: Binarized normed gradients for objectness estimation at 300fps[END_REF][START_REF] Uijlings | Selective search for object recognition[END_REF][START_REF] Zitnick | Edge boxes: Locating object proposals from edges[END_REF] followed by a classifier [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF][START_REF] He | Spatial pyramid pooling in deep convolutional networks for visual recognition[END_REF][START_REF] Girshick | Fast R-CNN[END_REF]. This category also includes the recent development that began with [START_REF] Shaoqing Ren | Faster R-CNN: towards real-time object detection with region proposal networks[END_REF] and that consists of learning the region proposal part together with the classification (see e.g. [START_REF] Dai | R-FCN: object detection via region-based fully convolutional networks[END_REF]). The last one, which is not so popular anymore, contains sliding window based methods [START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF][START_REF] Szegedy | Deep neural networks for object detection[END_REF]. We argue that none of the methods could be applied without extensive modifications to the task of detecting small targets in large images. For the first class of methods the grid used would have to be excessively large with a close-knit network, which would require to work on pooling or upsampling the final output and that goes against the philosophy of grid-based methods, which is to be coarse to be fast. In the second class of methods, problem specific region proposal algorithms would have to be designed so that they could detect small low-resolution objects with high recall. For instance, we think that the work we propose could be used as a first step in such pipelines.

Vehicle detection on aerial imagery has been recently studied in [START_REF] Zhao | Car detection in low resolution aerial images[END_REF][START_REF] Leberl | Recognizing cars in aerial imagery to improve orthophotos[END_REF][START_REF] Eikvil | Classificationbased vehicle detection in high-resolution satellite images[END_REF][START_REF] Kembhavi | Vehicle detection using partial least squares[END_REF][START_REF] Ali | A real-time deformable detector[END_REF]. The most recent articles on the subject are [START_REF] Razakarivony | Apprentissage de variétés pour la Détection et Reconnaissance de véhicules faiblement résolus en imagerie aérienne[END_REF][START_REF] Gleason | Vehicle detection from aerial imagery[END_REF], based on handcrafted descriptors, and [START_REF] Chen | Vehicle detection in satellite images by parallel deep convolutional neural networks[END_REF][START_REF] Mundhenk | A large contextual dataset for classification, detection and counting of cars with deep learning[END_REF][START_REF] Audebert | On the usability of deep networks for object-based image analysis[END_REF] using convolutional neural networks. [START_REF] Chen | Vehicle detection in satellite images by parallel deep convolutional neural networks[END_REF] also uses a convolutional neural network with a similar architecture, but does not provide any strategy to deal with class imbalance, it uses a different and outdated cost function and is much much slower. [START_REF] Mundhenk | A large contextual dataset for classification, detection and counting of cars with deep learning[END_REF] introduces a new and large dataset and reports good performances but the context-based method it used is impractical in our case, furthermore, the metric used to measure performances is much less demanding in terms of precision of the localization.

This work concentrates on developing hard-mining strategies to deal with classes with few examples and is a lot faster than all former approaches due to the small sized network and the fully convolutional inference and is specifically tailored for the detection of small objects.

METHODOLOGY

Introduction

In contrast with all the previous detection results on VEDAI, we do not use any sliding a window over the image to do inference but get rid of this expensive step using fully convolutional networks, as proposed by [START_REF] Shelhamer | Fully convolutional networks for semantic segmentation[END_REF]. In terms of architectures, we have experimented several variants around the simple LeNet-5 architecture ( [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF]) . This architecture seems indeed a good candidate for our detection task as the LeNet5 network has only 2 max-pooling layers (non-overlapping). Consequently, when used as fully convolutional network the resolution of the output heat maps is 4 times smaller than the original image. In practice there is no need to upsample the heat map by unpooling with max-pooling switches or using transposed convolutions like in ( [START_REF] Badrinarayanan | Segnetme: A deep convolutional encoder-decoder architecture for image segmentation[END_REF] or [START_REF] Shelhamer | Fully convolutional networks for semantic segmentation[END_REF]), nor to use dilated convolutions as in [START_REF] Yu | Multi-scale context aggregation by dilated convolutions[END_REF] nor even to use the shiftand-stitch trick of [START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF]. One key difference of detection on ortho-images and detection on commonly used datasets like Ms COCO, is that, because the distance between the sensor/camera and the ground is known in advance, all targets of a same class share the same size in pixels, approximately, and this size can be estimated accurately. Therefore there is no need to adopt a multi-scale approach nor to regress the width and height of the bounding boxes of the vehicles. For this reason, we can set once and for all the size of the extracted patches each class having a different sized associated patch. By doing that we also set the size of the first fully connected layer. The imbalance between the sink class and the vehicle class being so overwhelming we had to use a multi-step approach to get rid of most of the backgrounds and simplify the classification as noticed by [START_REF] Audebert | On the usability of deep networks for object-based image analysis[END_REF] once the vehicles and the background are identified classification between the different classes is more straightforward.

Training

We first extract positive patches around the targets. The negative patches used are sampled uniformly from the images until we reach a ratio of 5 to 1 (which has been determined by cross-validation). Too much negative samples and the imbalance is too strong, too little and the heterogeneity of the background class is not fully captured. A critical point that is also investigated in section 5 is the imbalance factor applied to the cross-entropy cost. Then the network is trained using stochastic gradient descent (SGD), with a dropout of 0.5 [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF] in the fully connected layers. When it reaches convergence, after around 100 epochs, then we do inference on the images on the training set in a fully convolutional fashion. From 1064x1064 pixels image (images have been up-sampled to 1024x1024 and padded) we get 256x256 heat maps. To select the maximums we do traditional Non Maximum Suppression (NMS). We then consider the remaining maximums to be the result of our detection and we evaluate them on the ground truth in this process we do not estimate orientations for bounding boxes working with squared bounding boxes is sufficient to get good results even with an orientation dependent metric.

In the process we sample negative patches that have been misclassified and weak true positives. In fact we experimented with 4 different strategies of hard-mining results are presented in section 5. We repeat the process multiple times.

Hard-Mining strategies

Bootstrapping offers a lot of liberties on how the hard examples are chosen. One could for instance pick a limited number of false positives per image or one could fix a threshold and only pick a false positive if its score is superior than a fixed threshold (0.5 for instance). There is also the question of weak true positives whether to include them at all and if yes from which threshold should we pick them. The previous experiments used the thresholds (0.5,0.5) while limiting the number of examples in each image to be 25 at max. We tested the 4 following different hard-mining variants on our network :

• Strategy 1: the number of hard examples per image is set to be exactly N (with N=25 in our experiments) whatever their scores. No weak true positive are added.

• Strategy 2: Same strategy that 1 but with weak true positive examples whose scores are less than 0.5 (scores are normalized probabilities).

• Strategy 3: We chose only hard negative with scores superior to 0.5 and weak true positives with scores less than 0.5

• Strategy 4: Same as strategy 3 without weak true positives.

These 4 strategies are evaluated in Section 3.

Addressing class imbalance issues

One of the issues with detection is the imbalance between the few present vehicles (targets) on images and the large variety of backgrounds. This issue is even more present in aerial imagery because of the relative size of full images and small vehicles like cars. In this context, VeDAI is especially challenging as each image only contains a few vehicles. There are many techniques to effectively fight this. Controlling vehicles/background proportion in each batch, minority oversampling like [START_REF] Nitesh | Smote: synthetic minority oversampling technique[END_REF], or modifying the cost itself to give more weights to the misclassification of classes that are less present.

We chose the latter because of its simplicity. The cost function implemented is the weighted cross-entropy:

L(w) = N i=1 C * t i * log(f w (x i ))+(1-t i ) * log(f w (x i )) (1)
where x i is one of the N image-patches in the training set, f w (x) is the score given by the convolutional neural network to a patch x, t i is 1 when x i belongs to the class under test 0 otherwise, C is a scalar defined as the ratio of Negative (background) vs Positive (vehicles) examples in the batch.

Classification study: rotation invariance, etc.

This part aims at finding the weaknesses of our approach and further motivates a follow-up work on using this baseline network as the first part of a cascade. We created numerous classification sets using collected hard-negative samples. We looked at every possible combination of: contrast normalizing the patches (or not), angle normalizing the targets (meaning all targets have the same orientation)(or not) and shifting the patches from the targets (4 pixels apart in checkerboard distance)(or not). For simplicity we adopted the following code: S and ¡ S mean respectively with shifted target normalization or without, A and A means angle-normalized patches or all rotations included and C and C contrast-normalized patches or no contrast normalization used. These different options are experimented in Section 3

EXPERIMENTS

The VeDAI dataset The VeDAI dataset [START_REF] Razakarivony | Vehicle Detection in Aerial Imagery: A small target detection benchmark[END_REF] consists of 1200 images that come in two different resolutions 512x512 and 1024x1024. Every image is available in two versions either colored (RGB) or infrared. All experiments in the paper were conducted on the infrared version of the 512x512 images. The dataset is provided with 10 folds and 1340 cars in total, with an average of around 140 cars to detect per fold. The definition of what a positive detection is pretty different from the standard Intersection Over Union (IOU) criterion adopted by the Pascal VOC or MS COCO. A detection is considered correct if it lies within the ellipse centered on the ground truth and lying inside the edges of the target car (multiple hits on the same targets are counted as False Positives). We are interested in mainly two metrics namely the mAP which is measured across all folds and the recall at low FPPI (recall for a given rate of False Positive Per Image).

Results on VeDAI

We first report general results on VeDAI using an architecture inspired by the LeNet-5 network. On overall, the network contains 2 convolutional layers followed by three fully connected like layers, as detailed Table 1. All convolutions are VALID type convolutions (no padding) and all fully connected layers are implemented as 1x1 convolutions. In order to make the results as good as possible, we experimented with the following parameters : (i) size of the receptive field, (ii) amount of regularization (iii) depth of each layer. We selected optimal parameters on a validation set. As there is none validation set by default in VeDAI, we split the 10 folds provided with the dataset into ten subsets of equal length and chose our parameters by training the network on 9 of them and validating on the tenth.

The selected network reaches a mAP of 77.8±3.3 (see 1) which is a very large improvement (12%) from the previous state of the art, on the car category. As a comparison, the recent work of [START_REF] Razakarivony | Apprentissage de variétés pour la Détection et Reconnaissance de véhicules faiblement résolus en imagerie aérienne[END_REF] reports a mAP of 69.6 ±3.4 for the same class.

We used an L2 regularization of 0.0001 and a wide networks with respectively for each layer 96, 192, 384 and 84 neurones. The network was trained using classic SGD with a fixed learning rate of 0.001.

The Table 2 lists all the published results on VeDAI and method AP Recall@0.01FPPI DPM [START_REF] Razakarivony | Vehicle Detection in Aerial Imagery: A small target detection benchmark[END_REF] 60.5 ± 4.2 13.4 ± 6.8 SVM+HOG31 [START_REF] Razakarivony | Vehicle Detection in Aerial Imagery: A small target detection benchmark[END_REF] 55.4 ± 2.6 7.8 ± 5.5 SVM+LBP [START_REF] Razakarivony | Vehicle Detection in Aerial Imagery: A small target detection benchmark[END_REF] 51.7 ± 5.2 5.5 ± 2.2 SVM+LTP [START_REF] Razakarivony | Vehicle Detection in Aerial Imagery: A small target detection benchmark[END_REF] 60.4 ± 4.0 9.3 ± 3.7 SVM+HOG31+LBP [START_REF] Razakarivony | Vehicle Detection in Aerial Imagery: A small target detection benchmark[END_REF] 61.3 ± 3.9 8.3 ± 5.2 SVM Fusion AED (HOG) [START_REF] Razakarivony | Apprentissage de varietes pour la Detection et Reconnaissance de vehicules faiblement resolus en imagerie aerienne[END_REF] Compensating the translations, rotations, etc.. We experimented the 3 normalization strategies given Section 2.5. As said before, to make the evaluation of the performance faster, we experimented these alternatives on a classification task. For building this classification set, we used all the targets from fold 1 and added hard negative examples.

There is 50000 examples in the training set with a ratio of negative over positive of 70 it is 5 times more than in the course of hard-mining passes as we had to sample only one positive example per target present instead of 5 as we have when we start detection training. We measure the accuracy by which percentage of images is classified correctly (with a fixed threshold of 0.5) but as the number of backgrounds is 70 times the number of targets, the accuracy is actually not

CONCLUSIONS

We have presented state-of-the-art results on a challenging benchmark and insights on how to tackle detection in the difficult context of few examples learning and extreme imbal-ance. This study cuts across the many articles using transfer learning to avoid having to deal with training a network from scratch. We are also confident that having found in 3 that angle normalization has such a strong effect on performances we could improve a lot on these results by using a cascade.

Fig. 1 .

 1 Fig. 1. Precision-Recall curves given by our RPN network on the 10 folds of the VEDAI dataset. The mAP is of 0.78± 0.03.

Table 1 .

 1 Architecture of our network, inspired by LeNet-5

	Name	Type	Filter Size	Input	Output
			Stride	Size	Output
	Conv1	convolution	5x5/1	45x45x3	41x41x96
	Pool1	max-pooling	2x2/2	41x41x96	20x20x96
	Conv2	convolution	5x5/1	20x20x96	16x16x192
	Pool2	max-pooling	2x2/2	16x16x192	8x8x192
	fc1	convolution	8x8/1	8x8x192	1x1x384
	fc2	convolution	1x1/1	1x1x384	1x1x84
	fc3	convolution	1x1/1	1x1x84	1x1x2
	Softmax	Softmax	None	1x1x2	1x1x2

Table 2 .

 2 Comparisons with related works

					69.6 ± 3.4	20.4 ± 6.2
		Ours			77.80 ± 3.3	31.04 ± 11
	Strategy R1	R2	R3	R4	R5	R6
	S1	0.00 0.21 0.39 0.36 0.28 0.25
	S2	0.00 0.10 0.17 0.43 0.27 0.32
	S3	0.00 0.35 0.34 0.31 0.25 0.29
	S4	0.00 0.23 0.32 0.23 0.32 0.32

Table 3 .

 3 Recall@0.01FPPI w.r.t. the number of passes for the 4 strategies, on a validation set ours. We can see that in terms of Average Precision or recall the advantages of our method.[START_REF] Audebert | On the usability of deep networks for object-based image analysis[END_REF] do not report any detection results only classification so we cannot compare to it directly.

	Evaluation of the 4 hard-negative-mining strategies As
	said before, bootstrapping offers a lot of liberties on how the
	hard examples are chosen. We proposed in Section 2 four
	different hard-mining strategies we are now going to evalu-
	ate. Table 3 gives the Recall at 0.01 False Positive per Image
	(FPPI) for different rounds of hard mining. From this table
	we can make three observations: first, the best overall perfor-
	mance is obtained with S2 and should be preferred. Second, if
	one wants to limit the number of rounds of hard example min-
	ing, he should prefer S3 which gives 0.35 after only 2 rounds.
	Third, without hard mining, the performance of the network
	is very low.

Table 4 .

 4 Learning variant/invariant representations. See Section 3 for details. very informative. It is more helpful to have also the accuracy on the target class patches, which verifies that the classifier would not get away with classifying everything as background and still get a good accuracy. We added the recall for 0.001 FP meaning the recall of the target classes patch obtained for a given proportion of 0.001 false positive in the validation set for the same reason. The results are given in

		Config.		Acc.	Acc.	Acc.	Recall
					pos.	backgds	at 0.001 FP
		S	£ C	98.35	53.91	98.94	28.02
	£ A	£	C	98.94	43.21	99.68	25.59
		S	£ C	98.78	66.67	99.21	41.86
			C	99.01	66.67	99.44	50.33
		S	£ C	99.01	82.72	99.23	63.45
	A	£	C	99.10	86.42	99.27	71.68
		S	£ C	99.33	91.77	99.43	82.79
			C	99.41	92.59	99.50	86.58

Table 4 .

 4 It is also interesting to see what would the network that was trained on perfectly centered patches do on shifted targets (it often happens when the equivalent sliding window step is not 1). We get 16.5 accuracy on the positive examples (instead of 53.91) and a recall at 0.001 FPPI of 18.20 (instead of 28.02). There is a severe degradation of the performances. The same phenomenon is observed when we train on all orientations and test on angle normalized patches. Accuracy on positive examples drops to 52.67 (instead of 82.71). The recall at 0.001 FPPI is also impacted 55.8 (instead of 63.45). Therefore this network performs much better when the targets are centered and normalized in rotations. We ran additional experiments to see if the angle normalization step of the cascade was really necessary so we added a data-augmentation module which rotated the patches in the current batch by a random angle uniformly chosen between 0 and 360. The network that performed poorly (53.91 accuracy on positive examples and 28.02 recall at 0.001 FPPI) gave reasonable results with 88.77 accuracy on the positive examples. So if we can center and normalize detected results we could use this trained classifier on top and it would better the results.