Maxime Bucher

Stéphane Herbin

Frédéric Jurie

Hard Negative Mining for Metric Learning Based Zero-Shot Classification

Keywords: domain adaptation, zero-shot learning, hard negative mining, bootstrapping

, which proposed an approach to solve Zero-Shot classification problems (ZSC) by introducing a novel metric learning based objective function. This objective function allows to learn an optimal embedding of the attributes jointly with a measure of similarity between images and attributes. This paper extends their approach by proposing several schemes to control the generation of the negative pairs, resulting in a significant improvement of the performance and giving above state-of-the-art results on three challenging ZSC datasets.

Introduction

Among the different image interpretation methods exploiting some kind of knowledge transfer in their design, Zero Shot Classification (ZSC) can be considered as a domain adaptation problem where the new target domain is defined using an intermediate level of representation made of human understandable semantic attributes. The source domain is defined by an annotated image database expected to capture the relation between data and attribute based representation of classes.

Most of the recent approaches addressing ZSC [START_REF] Lampert | Attribute-Based Classification for Zero-Shot Visual Object Categorization[END_REF][START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF][START_REF] Romera-Paredes | An embarrassingly simple approach to zero-shot learning[END_REF][START_REF] Zhang | Zero-Shot Learning via Semantic Similarity Embedding[END_REF][START_REF] Zhang | Zero-shot learning via joint latent similarity embedding[END_REF][START_REF] Wang | Zero-shot visual recognition via bidirectional latent embedding[END_REF][START_REF] Xian | Latent embeddings for zero-shot classification[END_REF] rely on the computation of a similarity function in the semantic space. They learn the semantic embedding, either from data or from class description, and compare the embedded data using standard distance.

Recently, [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF] proposed to add a metric learning (ML) step to adapt empirically the similarity distance in the embedding space, leading to a multi-objective criterion optimizing both the metric and the embedding. The metric is learned using an empirical optimized criterion on random but equally sampled pairs of similar (positive) and dissimilar (negative) data.

In this paper, following observations from the active learning community (see e.g., [START_REF] Fu | A survey on instance selection for active learning[END_REF]), we show that a careful choice of the negative pairs combined with the multi-objective criterion proposed in [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF] leads to above state of the art results.

Improved ZSL by efficient hard negative mining

In a recent work, Bucher et al. [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF] introduced a metric learning step in their zero-shot classification pipeline. Their model is trained from pairs of data, where positive (resp. negative) pairs are obtained by taking the training images associated with their own provided attribute vector (resp. by randomly assigning attribute vector of another image) and are assigned to the class label '1' (resp. '-1'). The set of positive pairs are denoted D + in the following, and D -for the negatives. This paper investigates improved ways to select negative pairs.

Bucher et al. metric learning framework for zero-shot classification

This section summarizes Bucher et al. [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF] paper, to which the reader should refer for more details. Zero-shot classification problem is cast into an optimal framework of the form:

Y * = arg min Y∈Y S(X, Y),
where X is an image, Y a vector of attributes and S a parametric similarity measure. A metric matrix denoted as W A transforms the attribute embedding space into a space where the Euclidean distance can be used. The similarity between images and attributes is computed as:

S(X, Y) = ( ÂX (X) -Y) T W A 2 (1)
where ÂX embeds the X modality into the space of Y using a linear transformation combined with a ReLU type transfer function:

ÂX (X) = max(0, X T W X + b X ). (2)
The role of learning is to estimate jointly the two matrices W X and W A . The empirical learning criterion used is the sum of 3 terms:

(i) a term for the metric W A :

l H (X i , Y i , Z i , τ ) = max 0, 1 -Z i (τ -S(X i , Y i ) 2 ) . (3)
where Z i states that the two modalities are consistent (Z i = 1) or not (Z i = -1). τ is the threshold separating similar and dissimilar examples.

(ii) a quadratic loss for the linear attribute prediction W X (only applied to positive pairs):

l A (X i , Y i , Z i ) = max(0, Z i ). Y i -ÂX (X i ) 2 2 . (4)
(iii) a quadratic penalization to prevent overfitting:

R(W A , W X , b X ) = W X 2 F + b X 2 2 + W A 2 F (5)
The overall objective function can then now be written as the sum of the previously defined terms:

L(W A , W X , b X , τ ) = i l H (X i , Y i , Z i , τ ) + λ i l A (X i , Y i , Z i ) +µR(W A , W X , b X ) (6)
In this context, ZSC is achieved by finding the most consistent attribute description from a set of exclusive attribute class descriptors {Y * k } C k=1 given the image where k, is the index of a class:

k * = arg min k∈{1...C} S(X, Y * k ) (7)

Hard negative mining

In a metric learning problem, while the set of positive pairs D + is fixed and given by the training set with one pair per positive image, the set of negative pairs D -can be chosen more freely; indeed, as there are many more ways of being different than being equal, the number of negative and positive pairs may not be identical. Moreover, we will see that increasing the size of D -compared to that of D + by some factor n leads to better overall results.

In the following, we explore three different strategies to sample the distribution of negative pairs using several learning epochs: we first present a variant of the method of [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF] and then describe two iterative greedy schemes.

Random In [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF], negative pairs are obtained by associating a training image with an attribute vector chosen randomly among those of other seen classes, with one negative pair for each positive one. As a variant, we propose to generate randomly n negative pairs (instead of one) for each positive pair, chosen as in [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF], i.e., by randomly sampling the set of attribute vectors from the other classes. We include in the objective function a penalization to compensate for the unbalance between positive and negative pairs (see section 2.3).

Uncertainty This strategy is inspired by hard mining for object detection [START_REF] Fu | A survey on instance selection for active learning[END_REF][START_REF] Shrivastava | Training region-based object detectors with online hard example mining[END_REF][START_REF] Li | Bootstrapping visual categorization with relevant negatives[END_REF][START_REF] Canévet | Efficient sample mining for object detection[END_REF] and consists in selecting the most informative negative pairs and iteratively updating the scoring function given by Eq. ( 1). We denote by S t (X, Y) this score at time t. During training, each time step t corresponds to a learning epoch. At the first epoch, S 1 (X, Y) is learned using the random negative pairs of [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF]. At each time t each pair of training image X i and candidate annotation Y coming from different (but seen) classes is ranked according to the uncertainty score:

u t (Y|X i ) = exp(-(S t (X i , Y) -S t (X i , Y * ))) (8)
where Y * is the true vector of attributes of X i . The vector of attributes which are most similar to the actual one while coming from different classes are the most relevant for improving the model. We define a probability of generating the pair based on this similarity score and sample this distribution.

Uncertainty/Correlation We propose to improve the previous approach by taking into account the intra-class correlation. The underlying principle governing the selection is that the most correlated vectors of attribute, in a given class, are the most useful ones to consider. The correlation can be measured by:

q(Y) = exp -1 |Y k | Y ∈Y k Y -Y 2 (9)
where k is the true class index of Y and Y k is the set of attribute vector representations.

A trade-off between uncertainty and correlation is obtained globally by using the following scoring function:

p t (Y|X i ) = u t (Y|X i ) * q(Y) (10)
where each image attribute vector Y at epoch t has a score of p t to be associated with X i . The current set of negative pairs D -at epoch t is obtained by iteratively increasing the set with new data sampled according to Eq. ( 10).

Adaptation of the objective function

The original learning criterion [START_REF] Zhang | Zero-shot learning via joint latent similarity embedding[END_REF], such as defined in [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF], assumes that negative and positive pairs are evenly distributed. This is not the case in the proposed approach: the criterion must be adapted to compensate for the imbalance between positive and negative pairs, by weighting the positive and negative pairs according to their frequencies:

L(W A , W X , b X , τ ) = 1 |D + |   i∈D+ l H (X i , Y i , Z i , τ ) + λl A (X i , Y i , Z i )   + 1 |D -|   j∈D- l H (X j , Y j , Z j , τ )   + µR(W A , W X , b X ) (11)
This criterion is updated at each new epoch when learning the model.

Experiments

Datasets In this section we evaluate the proposed hard mining strategy on different challenging zero-shot learning tasks, by doing experiments on the 4 following public datasets: aPascal&aYahoo (aP&Y) [START_REF] Farhadi | Describing objects by their attributes[END_REF], Animals with Attributes (AwA) [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF], CUB-200-2011 (CUB) [START_REF] Wah | The Caltech-UCSD Birds-200-2011 Dataset[END_REF] and SUN attribute (SUN) [START_REF] Patterson | The SUN Attribute Database: Beyond Categories for Deeper Scene Understanding[END_REF] datasets. They have been designed to evaluate ZSC methods and contain a large number of categories (indoor and outdoor scenes, objects, person, animals, etc.) described using various semantic attributes (shape, material, color, part name etc.). To make comparisons with previous works possible, we used the same training/testing splits as [START_REF] Farhadi | Describing objects by their attributes[END_REF] (aP&Y), [START_REF] Lampert | Learning to detect unseen object classes by between-class attribute transfer[END_REF] (AwA), [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF] CUB and [START_REF] Jayaraman | Zero-shot recognition with unreliable attributes[END_REF] (SUN).

Table 1: Zero-shot classification accuracy (mean ± std) on 5 runs. We report results with VGG-verydeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] features. unc./cor. = Uncertainty/Correlation method. The unc./cor. method can't be apply to the AwA dataset since all images of the same class have the same attributes, contrarily to the aP&Y, CUB and SUN datasets. Image features For each dataset, we used the VGG-VeryDeep-19 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] CNN models, pre-trained on imageNet (without fine tuning) and extract the fully connected layer (e.g., FC7 4096-d) for representing the images.

Hyper-parameters To estimate the three hyper-parameters (λ, the dimensionality of the metric space (m) and µ) we apply a grid search validation procedure by randomly keeping 20% of the training classes. W A and W X are randomly initialized with normal distribution and optimized with stochastic gradient descent.

Zero-shot classification

The experiments follow the standard ZSC protocol: during training, a set of images from known classes is available for learning the model parameters. At test time, images from unseen classes have to be assigned to one of the possible classes. Classes are described by a vector of attributes. Performance is measured by mean accuracy and std over the classes. Tables 1 and2 show the performance given by our hard-mining approach, which outperforms previous methods on 3 of the 4 datasets by more than 3% on average (+9% on AwA). The smart selection of negative pairs plays a role on the decision boundaries especially where classes have close attribute descriptions. We did not compare our results with [START_REF] Akata | Evaluation of Output Embeddings for Fine-Grained Image Classification[END_REF] or [START_REF] Xian | Latent embeddings for zero-shot classification[END_REF] as they use different image features. The 2 alternatives explored in the paper (Uncertainty vs Uncertainty/Correlation) give similar performance, but, as shown in the next section Uncertainty/Correlation is faster. [1] configuration corresponds to random method with one negative example per positive one. Our new negative pair selection method have a strong impact on the performance with a noticeable mean improvement of 3%. Augmenting the ratio of negative pairs over positive ones has a positive influence on the accuracy.

Convergence

We also made experiments to evaluate the impact of the hard mining selection on the convergence during training. Figure 1 shows that Uncertainty/Correlation converges around 4 times faster than the Uncertainty and Random methods. This confirms the fact that more informative (negative) pairs are selected with this strategy. The negative/positive ratio has a (small) positive impact on the convergence.

Conclusions

This paper extended the original work of Bucher et al. [START_REF] Bucher | Improving Semantic Embedding Consistency by Metric Learning for Zero-Shot Classification[END_REF] by proposing a novel hard negative mining approach used during training. The proposed selection strategy gives close or above state-of-the-art performance on four standard benchmarks and has a positive impact on convergence.

Feat

Fig. 1 :

 1 Fig. 1: Evolution of the performance as a function of the number of epochs, on the aP&Y dataset, with a neg/pos ratio of 1 and 100.

3. 2

 2 Performance as a function of the ratio of positive/negative pair Table 2 give accuracy performances on aP&Y dataset for the three methods in function of the number of negative examples for each positive pair. Bucher et al.

Table 2 :

 2 Zero-shot classification accuracy (mean ± std) on aP&Y dataset as a function of the ratio of positive/negative pairs. 15±0.88 53.98±0.79 54.41±1.47 54.37±1.05 Uncertainty 55.47±1.00 55.84±1.09 55.48±1.37 56.01±0.58 Uncertainty/Correlation 56.08±0.41 56.05±0.54 56.69±1.78 56.77±0.75

	Method / #neg. pair	1	10	50	100
	Random	53.


C Farhat
email: cfarhat@stanford.edu

P Avery
email: pavery@stanford.edu

T Chapman

P Collins

K Washabaugh

M Zahr
email: mzahr@stanford.edu

Recent advances in parametric nonlinear model order reduction: treatment of shocks, contact and interfaces, structure-preserving hyper reduction, acceleration of multiscale formulations, and application to design optimization

Keywords: ECSW, Hyper reduction, Model order reduction, Multiscale modeling, Reducedorder model, Structure-preserving

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Parametric, projection-based, Model Order Reduction (MOR) is a mathematical tool for constructing a parametric Reduced-Order Model (ROM) by projecting a given parametric High Dimensional Model (HDM) onto a Reduced-Order Basis (ROB). It is rapidly becoming indispensable for a large number of applications including, among others, computational-based design and optimization, multiscale analysis, statistical analysis, uncertainty quantification, and model predictive control. It is also essential for scenarios where real-time simulation responses are desired. During the last two decades, linear, projection-based, parametric MOR has matured and made a major impact in many fields of engineering including electrical engineering, acoustics, and structural acoustics, to name only a few. By comparison, nonlinear, projectionbased, parametric MOR remains somehow in its infancy. Nevertheless, giant strides have been recently achieved in many of its theoretical, algorithmic, and offline/online organizational aspects. The main purpose of this lecture is to highlight some of these advances, discuss their mathematical and computer science underpinnings, and report on their impact for an important class of problems in aerodynamics, fluid mechanics, nonlinear solid mechanics and structural dynamics, failure analysis, multiscale analyis, uncertainty quantification, and design optimization.

To this effect, nonlinear, projection-based, parametric MOR will be first interpreted as a constrained semidiscretization on a subset of a compact Stiefel manifold, using a low-dimensional basis of global shape functions constructed a posteriori `-that is, after some knowledge about the response of the system of interest has been developed. Usually, such a knowledge is gathered using the given parametric HDM and an offline training procedure where the model parameters are sampled with a greedy strategy based on a costeffective ROM error indicator. Specifically, a set of problems related to the parametric problem of interest are solved at the sampled parameter points using the given HDM, and the computed solution snapshots are compressed to obtain the desired global ROB. Depending on the mathematical type of the governing equations underlying the given HDM, a dual ROB is also constructed and the sought-after nonlinear parametric ROM is constructed by Galerkin (or Petrov-Galerkin) projection of the HDM onto the global ROB (and its dual counterpart).

Unfortunately, despite its low dimension, the resulting nonlinear ROM does not necessarily guarantee computational feasibility. This is because the construction process described so far does not scale only with the size of the target ROM, but also with that of the underlying HDM. This caveat is remedied by equipping the nonlinear MOR tool with a rigorous scheme for approximating the resulting reduced operators whose computational complexity scales only with the small size of the ROM. Such a procedure is often referred to in the literature as Hyper Reduction (HR). Here, the Energy Conserving Sampling and Weighting (ECSW) technique introduced in [START_REF] Farhat | Dimensional reduction of nonlinear finite element dynamic models with finite rotations and energy-conserving mesh sampling and weighting for computational efficiency[END_REF] and analyzed in [START_REF] Farhat | Structure-preserving, stability, and accuracy properties of the energyconserving sampling and weighting (ECSW) method for the hyper reduction of nonlinear finite element dynamic models[END_REF] will be discussed, and its distinctive structure-preserving and numerical stability properties will be highlighted. In particular, different implementations of ECSW will be contrasted and discussed with respect to their potential for an online implementation.

For highly nonlinear problems characterized by different physical regimes (such as, for example, pre-shock, shock, and after-shock; or pre-buckling, bukcling, and post-buckling), keeping the dimension of the global ROB sufficiently small while maintaining accuracy is challenging. For such problems, a concept of local ROBs where locality refers a priori `neither to space nor to time, but to the region of the manifold where the solution lies at a given time, was recently introduced in [START_REF] Amsallem | Nonlinear model order reduction based on local reduced-order bases[END_REF]. In this lecture, the implementation of this concept using a preferred clustering technique, its coupling with the structure-preserving HR method ECSW, and its merit for nonlinear MOR will be discussed and demonstrated with the real-time solution of aerodynamics problems with shocks, and the real-time simulation of the collapse of metallic structures. Enabling the latter family of applications requires the reduction of contact constraints. To this effect, the approach recently developed in [START_REF] Balajewicz | Projection-based model reduction for contact problems[END_REF] for reducing the dimension of nonlinear contact models when the contact constraints are enforced by Lagrange multipliers will be first outlined. Then, new results pertaining to the stability analysis and performance of the HR of the Lagrange multiplier field in the context of local ROBs will be presented.

For highly nonlinear fluid-structure interaction problems with arbitrarily evolving domains and interfaces, HDMs are often based on an embedded (or immersed) boundary method for CFD (Computational Fluid Dynamics). In this case, the computational fluid domain typically contains both real and ghost regions, which complicates the collection and compression of solution snapshots. In [START_REF] Balajewicz | Reduction of nonlinear embedded boundary models for problems with evolving interfaces[END_REF], this issue was addressed by formulating the snapshot compression problem as a weighted low-rank approximation problem where the binary weighting identifies the evolving component of the individual computational snapshots. This simple approach will be revisited here with particular attention of computational efficiency in the presence of very large-scale HDMs.

For multidisciplinary optimization problems, a recently developed [START_REF] Zahr | Progressive construction of a parametric reduced-order model for PDEconstrained optimization[END_REF] adaptive approach for using ROMs as surrogate models will be also presented and illustrated. This approach distinguishes itself from alternatives by the fact that it breaks the traditional offline/online framework of MOR. It formulates a high-dimensional PDE (Partial Differential Equation)-constrained optimization problem as a sequence of reduced optimization problems constrained by an evolving ROM, and trains this ROM at a given instance of the sequence by sampling the parametric HDM at the solution points of some of the previous reduced optimization problems in the sequence. This adaptive approach also equips the reduced optimization problems with a nonlinear trust-region based on a residual error indicator in order to keep the optimization trajectory in a region of the parameter space where the ROM is accurate. The net result is a ROM that is progressively constructed along the trajectory of the optimization procedure, rather than in a global parameter domain sampled by a design of experiments approach. For a given objective function (or a small set of them), this approach delivers a superior computational efficiency.

All aforementioned ideas, concepts, and mathematical techniques have been integrated into a computational framework for nonlinear MOR and HR. The versatility of this framework will be illustrated with its application to multiscale HDMs constructed using the multilevel finite element method FE2 [START_REF] Feyel | A multilevel finite element method (FE2) to describe the response of highly non-linear structures using generalized continua[END_REF]. In particular, it will be shown that for such multilevel models, MOR is computationally feasible even for a single analysis at a single parameter point. Finally, the potential of this framework for practical engineering operations will be also demonstrated for challenging applications pertaining to underbody blast, underwater implosion, and aircraft shape design [START_REF] Washabaugh | On the use of discrete nonlinear reduced-order models for the prediction of steady-state flows past parametrically deformed complex geometries[END_REF]. In all cases, speedup factors ranging between two and five orders of magnitude will be demonstrated, depending on the problem complexity and its setup.

Powered by TCPDF (www.tcpdf.org)


Thierry Doré

Sylvain Plantureux

May Balabane

Guy Richard

The virtual university of agroecology project

teaching and research institutions in France or abroad, or from public or private research centers.

The Virtual University for Agroecology project www.ea.inra.fr/uvae

Project dynamics

The UVAE, which will also feed the Agreenium university on line, is now developing under a dual dynamics regarding production of modules and training based on them. This aims to make the UVAE a space for collective action beyond individual training achievements, towards restrengthening agroecology in agronomic research.

Context and stakes

New dynamics of agriculture is needed to face multidimensional challenges such as food security, climate change or environment protection. This implies changes in the way to analyze functioning of agroecosystems and to design agricultural systems. Improving the use of biological interactions is necessary. Agroecology is hence a priority of research and higher education institutions, as well as of national and international policies. In this context a consortium of French public research and higher education institutions in agronomy considered continuing training directed towards their own staff as an essential lever alongside specific recruitments. They decided to build up the project of Virtual University for Agroecology.

Objectives and plan UVAE is a distance learning, french speaking, facility which aims to allow staff in agricultural research and higher education institutions to acquire or perfect their knowledge and skills in agroecology. It can also be used for students training at master level. Its design combines an introductory module, delivered in 2014, and three-types modules, in progress. The introductory module enables learners to familiarise themselves with the diverse positioning of the agricultural, scientific and social dimensions of agroecology. It constitutes the entry point to modules that aim the appropriation of concepts, methods and theories, alongside modules allowing to acquire knowledge on agroecosystems functioning and agroecological engineering. The overall plan is not designed for a single learning path but rather to promote dialogue and feedback between engineering and concepts and to allow for diverse potential learning pathes.

( 1 )

 1 AgroParisTech, 16 rue Claude Bernard, 75231 Paris Cedex 05, France -thierry.dore@agroparistech.fr (2) Université de Lorraine -Inra -Ensaia, UMR Agronomie et Environnement, 2 avenue de la forêt de Haye, 54518 Vandoeuvre, France -sylvain.plantureux@univ-lorraine.fr (3) Inra, Département Environnement et Agronomie, UR 251, RD 10, 78026 Versailles, France -may.balabane@versailles.inra.fr (4) Inra, Département Environnement et Agronomie, 2163 avenue de la Pomme de Pin, 45075 Orléans, France -guy.richard@orleans.inra.fr Thierry DORÉ (1) , Sylvain PLANTUREUX (2) , May BALABANE (3) and Guy RICHARD (4)